Running Disaster Recovery Drills for an ECS + Aurora API Service with FIS

Are you running disaster recovery drills? In this post, I'll share the key points from running a drill on an API service built with ECS (Fargate) + Aurora (MySQL) using FIS.

Service Architecture

A straightforward API backend architecture:

• Public Subnet — NAT Gateway, ALB
• Private Subnet — ECS (Fargate)
• Isolated Subnet — Aurora (MySQL)

Choosing a Scenario (Failures That Threaten Availability)

When planning the drill, we considered various failure scenarios. Failures range from small (instance-level) to large (region-wide), but we chose a single-AZ failure scenario. Since we use ECS (Fargate), instance-level failures are automatically recovered, and a region-wide failure is too broad in scope—requiring cross-region service reconstruction with potentially no clear path to recovery. A single-AZ failure hits the sweet spot: realistic in scale and recoverable with the right procedures.

Key Takeaway

The best scenario for your product and organization varies widely. Without careful consideration, you risk hearing "Would that really happen?" or "Doesn't that recover automatically without human intervention?"—which defeats the purpose of the drill.

Using Fault Injection Service to Simulate an AZ Failure

Fault Injection Service (FIS) is a chaos engineering service provided by AWS that lets you inject controlled faults into near-production environments to validate system resilience. We configured it to simulate a single-AZ failure for the architecture described above.

# ============================================================
# FIS BCP Test: Single AZ (ap-northeast-1d) Failure Simulation
# ============================================================

# ------------------------
# IAM Role for FIS
# ------------------------

data "aws_iam_policy_document" "fis_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["fis.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
  }
}

resource "aws_iam_role" "fis_bcp_test" {
  name               = "fis-bcp-test-role"
  assume_role_policy = data.aws_iam_policy_document.fis_assume_role.json
}

data "aws_iam_policy_document" "fis_bcp_test" {
  # Network disruption
  statement {
    actions = [
      "ec2:CreateNetworkAcl",
      "ec2:CreateNetworkAclEntry",
      "ec2:DeleteNetworkAcl",
      "ec2:DescribeNetworkAcls",
      "ec2:DescribeSubnets",
      "ec2:DescribeVpcs",
      "ec2:ReplaceNetworkAclAssociation",
    ]
    resources = ["*"]
  }

  # RDS failover
  statement {
    actions = [
      "rds:FailoverDBCluster",
      "rds:DescribeDBClusters",
    ]
    resources = ["*"]
  }

  statement {
    actions = [
      "ec2:CreateTags",
      "ec2:DeleteTags",
    ]
    resources = ["*"]
    condition {
      test     = "StringEquals"
      variable = "ec2:CreateAction"
      values   = ["CreateNetworkAcl"]
    }
  }

  # CloudWatch
  statement {
    actions = [
      "cloudwatch:DescribeAlarms",
    ]
    resources = ["*"]
  }

  # Logs
  statement {
    actions = [
      "logs:CreateLogDelivery",
      "logs:GetLogDelivery",
      "logs:UpdateLogDelivery",
      "logs:DeleteLogDelivery",
      "logs:ListLogDeliveries",
      "logs:PutResourcePolicy",
      "logs:DescribeResourcePolicies",
      "logs:DescribeLogGroups",
    ]
    resources = ["*"]
  }
}

resource "aws_iam_role_policy" "fis_bcp_test" {
  name   = "fis-bcp-test-policy"
  role   = aws_iam_role.fis_bcp_test.id
  policy = data.aws_iam_policy_document.fis_bcp_test.json
}

# ------------------------
# CloudWatch Alarm (Stop Condition)
# ------------------------

resource "aws_cloudwatch_metric_alarm" "fis_bcp_stop_condition" {
  alarm_name          = "fis-bcp-stop-condition"
  alarm_description   = "BCP Test Stop Condition (Safety Lever): Automatically stop the FIS experiment if ALB 5XX errors exceed the threshold."
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "HTTPCode_ELB_5XX_Count"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Sum"
  threshold           = 100
  treat_missing_data  = "notBreaching"

  dimensions = {
    LoadBalancer = "<your-alb-arn-suffix>"
  }
}

# ------------------------
# FIS Experiment Template
# ------------------------

resource "aws_fis_experiment_template" "bcp_az_failure" {
  description = "BCP Test: ap-northeast-1d single AZ failure - network disruption + Aurora failover"
  role_arn    = aws_iam_role.fis_bcp_test.arn

  stop_condition {
    source = "aws:cloudwatch:alarm"
    value  = aws_cloudwatch_metric_alarm.fis_bcp_stop_condition.arn
  }

  # --- Targets ---
  target {
    name           = "az-1d-subnet"
    resource_type  = "aws:ec2:subnet"
    selection_mode = "ALL"

    resource_arns = [
      "<your-public-subnet-arn>",
      "<your-private-subnet-arn>",
      "<your-isolated-subnet-arn>",
    ]
  }

  target {
    name           = "aurora-cluster"
    resource_type  = "aws:rds:cluster"
    selection_mode = "ALL"

    resource_arns = [
      "<your-aurora-cluster-arn>",
    ]
  }

  # --- Actions ---
  action {
    name        = "disrupt-az-1d-connectivity"
    action_id   = "aws:network:disrupt-connectivity"
    description = "disrupt ap-northeast-1d subnet"

    parameter {
      key   = "scope"
      value = "all"
    }

    parameter {
      key   = "duration"
      value = "PT30M"
    }

    target {
      key   = "Subnets"
      value = "az-1d-subnet"
    }
  }

  # Aurora Failover
  action {
    name        = "failover-aurora"
    action_id   = "aws:rds:failover-db-cluster"
    description = "Aurora failover"

    target {
      key   = "Clusters"
      value = "aurora-cluster"
    }
  }

  # --- Logging ---
  log_configuration {
    log_schema_version = 2

    cloudwatch_logs_configuration {
      log_group_arn = "${aws_cloudwatch_log_group.fis_bcp_test.arn}:*"
    }
  }

  tags = {
    purpose = "bcp-test"
  }
}

resource "aws_cloudwatch_log_group" "fis_bcp_test" {
  name              = "fis-bcp-test"
  retention_in_days = 7
}

The configuration blocks all inbound and outbound network connectivity for ap-northeast-1d across all three subnets (public, private, and isolated) for 30 minutes. To cover the case where the Aurora Writer instance is in the affected AZ, we also trigger an Aurora failover simultaneously.

Key Takeaway

As I'll discuss later, FIS is still a simulation—it doesn't perfectly replicate a real failure. You'll need to pick the closest available fault pattern to your intended scenario. FIS capabilities are steadily expanding, so it's worth reviewing what's available when designing your experiments.

Execution and Findings

We used k6 to continuously call one read endpoint and one write endpoint while injecting the fault with FIS. When the network disruption and Aurora failover fired simultaneously, the error rate spiked sharply at the moment of failover, followed by intermittent errors that persisted until the FIS experiment ended.

The Aurora failover itself completes fine, but traffic continues to be routed to the now-unreachable instance, causing all requests directed to it to time out. In a real failure, the Aurora control plane might behave differently, but with FIS the situation is essentially "the AZ is down, yet the managed service still recognizes the instance as existing." Regardless, since you can't predict exactly what will happen in a real incident, you need to be able to recover from this case as well.

Key Takeaway

I recommend just running it once first. You'll almost certainly find things that don't work as expected or areas to improve. That's exactly why you should leverage FIS to make it easy to run drills repeatedly.

Remediation and Results

To exclude a specific Aurora instance, you can use a custom endpoint. Create an endpoint consisting only of healthy Reader instances, then point the application to it. In our architecture, we store the DB endpoint in Parameter Store and pass it to ECS task definitions as an environment variable. During a failure, we create a custom endpoint, update Parameter Store, and restart the application. The procedure is as follows:

• [Symptom] Failure occurs — the Writer instance in 1d fails over to a Reader role
• [Symptom] The Reader instance in 1d can no longer serve requests
• [Action] Add a Reader instance in a healthy AZ (1a or 1c) to compensate for lost capacity
• [Action] Create a custom endpoint composed only of Reader instances in 1a and 1c
• [Action] Update the Read endpoint in Systems Manager Parameter Store
• [Action] Force a new deployment of the ECS service (at this point, also exclude 1d from the network configuration to resolve any ECS capacity issues)

After performing these steps following the failure, the intermittent errors stopped entirely—achieving full recovery.

Key Takeaway

We arrived at this procedure through several rounds of trial and error, ultimately codifying it into a Runbook. Repeated practice builds familiarity, so you won't panic when it's the real thing. We've also designed the Runbook to be as command-driven as possible, ensuring anyone can execute the same recovery regardless of experience.

Conclusion

By establishing this Runbook, we can now expect to recover within one hour even if an entire AZ goes down. A Runbook isn't something you write once and forget—it only proves its worth through periodic validation and refinement. With FIS making it easy for anyone to simulate failures, we plan to continue running drills regularly.