Your Budget Alert Won't Save You: Building a Real Cloud Spend Circuit Breaker

You set up budget alerts. You get the email. You nod wisely. And then... you keep spending. The money is already gone by the time the alert arrives.

Budget alerts are reactive by design. They tell you after the fact. A circuit breaker is different — it proactively interrupts the flow before spend spirals out of control.

In this article, I'll show you how to build a real cloud spend circuit breaker using AWS Budgets + CloudWatch + SNS + Lambda, with Terraform, that can actually stop or throttle spend in near-real-time.

Why Budget Alerts Fail

Problem	Why It Matters
8-hour metric granularity	AWS Billing metrics are sampled every 8 hours. An alert at 100% threshold means the bill already exceeded your limit.
Forecasted spend is optimistic	Cost Explorer forecasts use historical averages and don't account for bursty, unpredictable workloads.
Email delays	Budget notifications arrive via SNS email subscription, which can take minutes to hours to be confirmed and delivered.
No enforcement mechanism	An alert is just a notification. There's no built-in way to stop spend at the infrastructure level.
Threshold blindness	Setting a single 100% threshold means you only learn you've overspent, with no warning lane.

The Circuit Breaker Architecture

The solution combines three AWS services into a closed-loop system:

1. AWS Budgets — Tracks actual and forecasted spend, fires alerts at configurable thresholds
2. CloudWatch Alarms — Monitors the budget state and triggers Lambda execution
3. Lambda + SNS — Executes remediation actions and notifies stakeholders

The Key Differentiator: Forecasted + Actual Dual Thresholds

Most people set one alert at 100%. That's too late. Instead, use three thresholds:

Threshold	Type	Purpose
80%	ACTUAL	Warning: "Hey, you're at 80% of budget. Keep an eye on it."
100%	ACTUAL	Danger: "You've hit your monthly limit. Time to review."
110%	FORECASTED	Proactive: "Forecast predicts you'll exceed budget mid-month. Act now."

The forecasted 110% threshold is the circuit breaker's trigger point — it fires before you actually overspend, while there's still budget runway to react.

The Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                   AWS BUDGETS                               │
│  ACTUAL spend → 80% warning │ 100% danger │ 110% forecast │
│                       ↓                                   ↓         │
│               CloudWatch Alarm → Lambda → SNS →            │
│               ┌─────────────────┐  ┌─────────────────┐ │
│               │  Throttle       │  │  Auto-shutdown  │ │
│               │  workloads      │  │  offending      │ │
│               └─────────────────┘  └─────────────────┐ │
│                       ↓                                   │ │
│               SNS notifications to Slack/PagerDuty/Lambda │
└─────────────────────────────────────────────────────────────┘

Building the Circuit Breaker with Terraform

Here's a complete Terraform configuration that sets up the full circuit breaker.

1. Budget Resource

module "spend_budget" {
  source  = "cloudposse/budgets/aws"
  version = "1.5.0"

  name         = "circuit-breaker-monthly"
  budget_type  = "COST"
  limit_amount = "500"   # $500 monthly limit
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  cost_filter = {
    Environment = "production"
  }

  cost_types = {
    include_credit           = false
    include_discount         = true
    include_other_subscription = true
    include_recurring        = true
    include_refund           = false
    include_subscription     = true
    include_support          = true
    include_tax              = true
    include_upfront          = true
    use_blended              = false
  }

  # Tiered alerting: 80% actual warning, 100% actual danger, 110% forecasted circuit breaker
  notifications = {
    actual = {
      comparison_operator = "GREATER_THAN"
      threshold           = 80
      threshold_type      = "PERCENTAGE"
      notification_type   = "ACTUAL"
    }
    danger = {
      comparison_operator = "GREATER_THAN"
      threshold           = 100
      threshold_type      = "PERCENTAGE"
      notification_type   = "ACTUAL"
    }
    forecast = {
      comparison_operator = "GREATER_THAN"
      threshold           = 110
      threshold_type      = "PERCENTAGE"
      notification_type   = "FORECASTED"
    }
  }

  notifications_enabled = true
  encryption_enabled     = true
}

2. Lambda Function: The Circuit Breaker Logic

The Lambda is the core of the circuit breaker. When triggered by a CloudWatch alarm, it can:

• Throttle workloads (e.g., scale down ASG, reduce instance counts)
• Terminate offending resources
• Post notifications to Slack/PagerDuty
• Return status to the caller

import json
import boto3
import os
import logging
import urllib.request
import urllib.error

logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Configuration from environment variables
SLACK_WEBHOOK = os.environ.get("SLACK_WEBHOOK_URL")
THROTTLE_PERCENT = float(os.environ.get("THROTTLE_PERCENT", "50"))  # Scale down to 50%
MAX_THROTTLE_ITERATIONS = int(os.environ.get("MAX_ITERATIONS", "3"))

# AWS clients
cloudwatch = boto3.client("cloudwatch")
autoscaling = boto3.client("autoscaling")
sns = boto3.client("sns")

def lambda_handler(event, context):
    """Main entry point for the circuit breaker Lambda."""

    logger.info(f"Received event: {json.dumps(event, indent=2)}")

    # Determine which budget triggered the alarm
    alarm_name = event.get("AlarmName", "")
    new_state = event.get("NewStateValue", "")
    old_state = event.get("OldStateValue", "")

    logger.info(f"Alarm {alarm_name} transitioned from {old_state} to {new_state}")

    # Parse the budget name from the alarm name
    budget_name = alarm_name.replace("SpendBudget-", "")

    if new_state == "ALARM":
        # Circuit breaker tripped - take action
        logger.warning(f"Circuit breaker tripped for budget: {budget_name}")

        # Take remediation actions
        actions_taken = []

        # 1. Scale down auto-scaling groups
        try:
            scaling_actions = throttle_autoscaling_groups(THROTTLE_PERCENT)
            actions_taken.append(f"ASG throttle: {scaling_actions}")
        except Exception as e:
            logger.error(f"ASG throttle failed: {e}")
            actions_taken.append(f"ASG throttle FAILED: {e}")

        # 2. Post to Slack
        if SLACK_WEBHOOK:
            try:
                post_to_slack(
                    f":warning: *Circuit breaker tripped* for budget `${budget_name}`. "
                    f"Budget is at {event.get('Trigger', {}).get('Value', '?')}%. "
                    f"Automated throttling actions initiated."
                )
            except Exception as e:
                logger.error(f"Slack notification failed: {e}")

        # 3. Send SNS notification (redundant with CloudWatch but good for audit)
        try:
            sns.publish(
                TopicArn=os.environ.get("CIRCUIT_BREAKER_SNS_TOPIC"),
                Message=json.dumps({
                    "budget": budget_name,
                    "state": "ALARM",
                    "actions": actions_taken,
                    "timestamp": event.get("StateChangeTime")
                }),
                Subject=f"[CIRCUIT BREAKER] {budget_name} exceeded threshold"
            )
        except Exception as e:
            logger.error(f"SNS notification failed: {e}")

    return {
        "statusCode": 200,
        "body": json.dumps({
            "budget": budget_name,
            "state": new_state,
            "actions_taken": actions_taken
        })
    }


def throttle_autoscaling_groups(percent_reduction):
    """Scale down all production ASGs by the given percentage."""
    asg_names = os.environ.get("PROD_ASG_NAMES", "").split(",")
    actions = []

    for asg_name in asg_names:
        asg_name = asg_name.strip()
        if not asg_name:
            continue

        # Get current ASG details
        asg = autoscaling.describe_auto_scaling_groups(
            AutoScalingGroupNames=[asg_name]
        )["AutoScalingGroups"][0]

        current_capacity = asg["DesiredCapacity"]
        new_capacity = max(1, int(current_capacity * (100 - percent_reduction) / 100))

        if new_capacity < current_capacity:
            autoscaling.set_desired_capacity(
                AutoScalingGroupName=asg_name,
                DesiredCapacity=new_capacity,
                HonorCooldown=False
            )
            actions.append(f"{asg_name}: {current_capacity} → {new_capacity}")

    return "; ".join(actions)


def post_to_slack(message):
    """Post a message to Slack via webhook."""
    payload = json.dumps({"text": message}).encode("utf-8")
    req = urllib.request.Request(
        SLACK_WEBHOOK,
        data=payload,
        headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=10) as response:
        return response.read()

3. CloudWatch Alarm Configuration

resource "aws_cloudwatch_metric_alarm" "budget_80_warning" {
  alarm_name          = "SpendBudget-80-warning"
  comparison_operator = " GreaterThanThreshold "
  evaluation_periods  = 1
  metric_name         = "BudgetType"
  namespace           = "AWS/Billing"
  period              = 300
  statistic           = "Maximum"
  threshold           = 80
  treat_missing_data  = "missing"

  alarm_actions = [module.lambda_circuit_breaker.arn]
  ok_actions        = [module.lambda_circuit_breaker.ok_arn]

  dimensions = {
    BudgetName = module.spend_budget.budget_name
  }
}

# Similar alarms for 100% and 110% thresholds...

4. SNS Topic and Subscription

resource "aws_sns_topic" "circuit_breaker_alerts" {
  name = "circuit-breaker-alerts"
}

# Allow Budgets to publish to this topic
resource "aws_sns_topic_policy" "budgets_access" {
  arn = aws_sns_topic.circuit_breaker_alerts.arn

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "AllowBudgetsPublish"
      Effect    = "Allow"
      Principal = { Service = "budgets.amazonaws.com" }
      Action    = "SNS:Publish"
      Resource  = aws_sns_topic.circuit_breaker_alerts.arn
    }]
  })

# Subscribe email (manual step required)
resource "aws_sns_topic_subscription" "email_alert" {
  topic_arn = aws_sns_topic.circuit_breaker_alerts.arn
  protocol  = "email"
  endpoint  = "[e-mail]"
}

The Human Element: What Actually Happens

I've seen this pattern play out in production:

1. Day 1-10: Everything looks normal. Budgets at 15-20%. No alerts.
2. Day 11: A new feature launches. Traffic spikes. Budget creeps to 40%.
3. Day 15: 80% actual threshold triggers. Team gets Slack notification. "Whoa, we're at 80%. Better check what's running."
4. Day 18: 100% actual threshold triggers. Budget hit. Alarm fires Lambda. ASGs get throttled 50%. Spend growth slows.
5. Day 20: 110% forecasted threshold would have triggered 3 days earlier if forecast alerts were enabled. "We could have prevented this."

The circuit breaker doesn't magically reduce costs — it creates the conditions for intervention before it's too late.

Common Mistakes & Gotchas

Mistake	Fix
Only setting 100% ACTUAL	Add 80% ACTUAL + 110% FORECASTED thresholds
No forecasted alerts	Forecasted alerts warn before you hit the limit
Lambda has no error handling	Budget alerts are "best effort" — always log failures
Forgetting SNS subscription confirmation	AWS sends a confirmation email — click it!
No cost filters	Scope budgets to specific environments/services to avoid noise
Throttling without safety valves	Always maintain a minimum capacity (e.g., max(1, ...))
Circuit breaker can't stop all spend	Budgets alert; they don't enforce hard caps. Pair with SCPs for enforcement.

The Mental Model

Think of a cloud spend circuit breaker like an electrical circuit breaker:

• Normal operation: Current flows normally, breaker is closed
• Warning state (80%): Current is getting high. Monitor closely. Don't panic yet.
• Danger state (100%): Current has reached the limit. Breaker starts to trip. Take preemptive action.
• Circuit breaker tripped (110% forecast): Current is excessive. Breaker opens automatically. Load is shed. System protected.

The key insight: You want the breaker to trip proactively, not reactively. A tripped breaker at 110% forecast means you had 10% budget buffer to react. A tripped breaker at 100% actual means the money is already gone.

Practical Next Steps

1. Enable forecasted alerts on your existing budgets — it's the single highest-impact change
2. Set three thresholds: 80% warning, 100% danger, 110% circuit breaker
3. Deploy a Lambda that can throttle your most expensive ASGs or workloads
4. Subscribe to SNS notifications and route to Slack/PagerDuty
5. Review monthly — adjust thresholds based on actual spending patterns

Conclusion

Budget alerts are necessary but insufficient. They tell you after the fact. A circuit breaker using AWS Budgets + CloudWatch + Lambda + SNS gives you proactive spend control — the kind that can actually stop or throttle spend before it becomes a crisis.

The three-threshold approach (80%/100%/110%) gives you a full visibility spectrum: warning, danger, and proactive intervention. The forecasted 110% threshold is the real circuit breaker — it fires before you overspend, while there's still budget runway to react.

Don't let your budget alerts become "set and forget" noise. Wire them into an actual remediation pipeline and take control of your cloud spend.

Further reading:

• AWS Budgets User Guide
• CloudWatch Metric Alarms
• Terraform AWS Budgets Module

Got a circuit breaker story from your own cloud experience? Drop a comment — I'd love to hear what's worked (or failed) for your team.