DEV Community

Siddharth Mishra
Siddharth Mishra

Posted on

Catching a Silent AWS Failure: A Serverless Daily Report for Suspended Auto Scaling Groups

#serverless #aws #monitoring #devops

Catching a Silent AWS Failure: A Serverless Daily Report for Suspended Auto Scaling Groups

How I built a zero-cost, zero-server monitoring alert with Lambda, EventBridge, and SES, and the two production bugs I hit along the way.

The silent failure nobody watches

Amazon EC2 Auto Scaling Groups (ASGs) have a feature that's incredibly useful during incidents and quietly dangerous afterward: you can suspend individual scaling processes like Launch, Terminate, HealthCheck, or AZRebalance.

Engineers suspend these all the time, to freeze capacity during a deployment, to stop instance churn while debugging, to hold a group steady during a migration. The problem is what happens next: someone forgets to resume them.

A group with Launch suspended won't add capacity when traffic spikes. A group with HealthCheck suspended won't replace unhealthy instances. Nothing alarms on this by default. You often discover it during the exact incident where you needed scaling to work.

I wanted a simple, reliable way to surface this every single day. Here's what I built.

The architecture

The whole thing is serverless and costs effectively nothing to run:

Amazon EventBridge (daily cron)
        |
        v
   AWS Lambda (Python + boto3)  --- scans all ASGs for suspended processes
        |
        v
   Amazon SES  --- emails a branded HTML report
        |
        v
     Your inbox
Enter fullscreen mode Exit fullscreen mode
  • EventBridge triggers the function on a daily schedule.
  • Lambda does the work: list every ASG across the configured regions, filter to those with suspended processes.
  • SES delivers a formatted HTML table so the report is readable at a glance, not a wall of JSON.

No servers, no agents, no cron box to maintain.

The core logic

The heart of the function is a paginated scan of Auto Scaling Groups, keeping only the ones with suspended processes:

import boto3

def find_suspended_asgs(region):
    client = boto3.client("autoscaling", region_name=region)
    paginator = client.get_paginator("describe_auto_scaling_groups")

    suspended = []
    for page in paginator.paginate():
        for asg in page.get("AutoScalingGroups", []):
            processes = asg.get("SuspendedProcesses", [])
            if processes:
                suspended.append({
                    "region": region,
                    "name": asg["AutoScalingGroupName"],
                    "process_names": [p["ProcessName"] for p in processes],
                    "desired": asg.get("DesiredCapacity"),
                    "min": asg.get("MinSize"),
                    "max": asg.get("MaxSize"),
                })
    return suspended
Enter fullscreen mode Exit fullscreen mode

Using the paginator matters, accounts with many ASGs return paged results, and a naive single call would silently miss groups.

The results are rendered into an HTML table and sent through SES. The email shows each affected ASG, which processes are suspended, and the min/desired/max capacity so you can judge the blast radius immediately. There's also a NOTIFY_WHEN_EMPTY toggle so you can choose between "only email me when something's wrong" and "email me a daily all-clear."

Deploying with just the AWS CLI

I kept the deployment dependency-free, no framework, just a bash script wrapping the AWS CLI. The essentials:

# Package the function
cd src && zip -q -r ../function.zip handler.py

# Create the Lambda (first deploy)
aws lambda create-function \
  --function-name asg-suspend-report \
  --runtime python3.12 \
  --role "$ROLE_ARN" \
  --handler handler.handler \
  --timeout 120 \
  --zip-file fileb://function.zip \
  --region ap-south-1
Enter fullscreen mode Exit fullscreen mode

EventBridge and the IAM role can be created the same way, or once manually. That's the whole footprint.

The two bugs (the fun part)

A tutorial where everything works first try teaches you nothing. Here's what actually happened.

Bug 1: Runtime.ImportModuleError: No module named 'lambda_function'

The first invocation failed immediately:

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function':
No module named 'lambda_function'
Enter fullscreen mode Exit fullscreen mode

The cause: Lambda's handler setting still pointed at the default lambda_function.lambda_handler, but my code lived in handler.py with a function named handler. The handler string must be <file>.<function>.

The fix:

aws lambda update-function-configuration \
  --function-name asg-suspend-report \
  --handler handler.handler \
  --region ap-south-1
Enter fullscreen mode Exit fullscreen mode

Lesson: the handler string is a file-and-function path, and it's easy for it to drift from your actual code, especially if the function was created with console defaults.

Bug 2: AccessDenied on autoscaling:DescribeAutoScalingGroups

Next invocation got further, then failed on permissions:

[ERROR] ClientError: An error occurred (AccessDenied) when calling the
DescribeAutoScalingGroups operation: ... is not authorized to perform:
autoscaling:DescribeAutoScalingGroups
Enter fullscreen mode Exit fullscreen mode

The execution role had a basic trust policy but was missing the actual permissions the code needs. The fix, a minimal inline policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "autoscaling:DescribeAutoScalingGroups",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["ses:SendEmail", "ses:SendRawEmail"],
      "Resource": "*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Note that DescribeAutoScalingGroups doesn't support resource-level restrictions, so it requires Resource: "*". SES can be scoped tighter to a verified identity ARN for stricter least privilege.

Lesson: least privilege means you'll hit AccessDenied, and that's a good thing. The errors tell you exactly which action to grant, one at a time, instead of over-provisioning with a wildcard.

What this small project demonstrates

It's not a huge system, but it exercises a surprising amount of real cloud engineering:

  • Serverless event-driven design (EventBridge → Lambda → SES)
  • The AWS SDK with correct pagination
  • IAM least-privilege debugging
  • Lambda packaging and configuration gotchas
  • Infrastructure delivered as a repeatable script

Those are exactly the day-to-day skills that keep production healthy.

Takeaways

  1. Suspended ASG processes are a silent risk. If your team uses them during incidents, monitor for ones left behind.
  2. Serverless is perfect for periodic checks, no infrastructure to babysit, effectively free.
  3. Read your error types. Runtime.ImportModuleError is a config problem; AccessDenied is an IAM problem. Each points straight at the fix.
  4. Small automations have outsized value. This one can prevent a genuinely bad day for a few lines of Python.

Thanks for reading. I write about AWS, serverless, and cloud operations. Connect with me if you're building similar things.

Top comments (1)

Collapse
 
raknaos profile image
Raknaos •

The handler-string bug is the one I'd have hit in the exact same order — creating the function from console defaults leaves lambda_function.lambda_handler in place, and the error names a module that doesn't exist rather than the mismatch, so you go looking for a packaging problem when the zip was fine.

What I like about the design is the NOTIFY_WHEN_EMPTY toggle: an all-clear email turns the report into a liveness probe for the monitor itself, which matters more than the check it wraps. The AccessDenied-as-a-feature point is right too — least privilege is only debuggable when the error names the one action to grant. Did you consider suspending AZRebalance specifically during migrations? That's the one process I've seen teams leave frozen for weeks because nothing visibly breaks.