How to Alert on Lambda Failures with CloudWatch Alarms and SNS

Originally published on kuryzhev.cloud

---

The scenario

A CloudWatch alarm Lambda errors SNS pipeline is exactly what was missing the night our order-processor function started throwing exceptions at 2am and nobody knew for six hours. The function processed SQS messages triggered by S3 uploads, and it had been failing silently — not crashing loud enough to page anyone, just quietly dropping orders one at a time. We only found out when a customer emailed support asking where their confirmation was.

Here's the thing nobody tells you when you first deploy Lambda: CloudWatch Logs capture everything, but nobody is tailing logs at 2am. Logs are for forensics after you already know something broke — they're useless as a detection mechanism on their own. You need something that watches the metrics Lambda already emits and screams the moment they cross a threshold.

That's the goal here: wire a CloudWatch alarm on the Errors metric to an SNS topic, so an email (or Slack, later) lands in someone's inbox within a minute or two of the first failure. Not a dashboard you have to remember to check. An actual page. We'll build this with Terraform since that's what survives past the first deploy, but the same logic maps directly to the AWS CLI or console if you're prototyping.

Prerequisites

Before touching any code, make sure you actually have the access and pieces in place — half the "why isn't this working" tickets I've debugged came down to missing IAM permissions or an unconfirmed subscription.

• AWS CLI v2.15.30 or newer, or Terraform 1.7+ with the hashicorp/aws provider pinned to ~> 5.40. Older provider versions handle composite alarms differently, so don't skip the version pin.
• Credentials with cloudwatch:PutMetricAlarm, sns:CreateTopic, sns:Subscribe, and enough Lambda permissions to invoke the function for testing.
• An existing Lambda function already emitting standard metrics — I'll use order-processor throughout. If it's invoked at least occasionally, it's already publishing Invocations, Errors, Throttles, and Duration to the AWS/Lambda namespace with zero extra config.
• A destination for the alert. Start with an email address. If you're planning to route to Slack later, that's a separate Lambda-forwarder or chatbot integration — mentioned below but not required to get this working today.

If you want the full reference on what CloudWatch alarms support, the AWS documentation on alarm actions is worth bookmarking — it covers composite alarms and anomaly-detection bands too, which we're not using here but might be worth a follow-up.

Step 1 — Create the SNS topic and subscription

Build the notification channel before the alarm, otherwise you'll be staring at an ALARM state with nowhere for it to go. Create the topic, subscribe an email endpoint, and remember the ARN format looks like arn:aws:sns:us-east-1:123456789012:lambda-failure-alerts.

Watch out: SNS email subscriptions require manual confirmation. The subscriber gets a "Confirm subscription" email with a link they have to click. Skip that step and your alarm will happily flip to ALARM forever while zero emails arrive — no error, no warning, just silence. This bit us on our first deploy; the alarm history showed three state transitions and the on-call engineer swore he never got paged. He hadn't confirmed the subscription three weeks earlier when it was set up.

Here's the Terraform for the topic, subscription, and the resource policy that restricts publishing to CloudWatch alarms only — an open topic policy means any principal in your account (or worse, a misconfigured cross-account role) can publish fake alerts and trigger a false page.

resource "aws_sns_topic" "lambda_alerts" {
  name = "lambda-failure-alerts"
}

resource "aws_sns_topic_subscription" "email_alert" {
  topic_arn = aws_sns_topic.lambda_alerts.arn
  protocol  = "email"
  endpoint  = "[e-mail]"
  # NOTE: subscriber must confirm via email before notifications flow
}

For scaling this pattern, we later added a second subscription — a Lambda function that reformats the SNS message and posts it to a Slack webhook. That's a separate build, but it plugs into the exact same topic with zero changes to the alarm logic.

Step 2 — Define the CloudWatch alarm on the Errors metric

Now the actual detection logic. The metric is AWS/Lambda / Errors, dimensioned by FunctionName=order-processor. Statistic is Sum, period is 60 seconds (the minimum granularity Lambda supports), evaluation periods is 1, threshold is >= 1.

Gotcha #1: use Sum, never Average. On a low-traffic function, one error out of five invocations in a period averages to 0.2 — comfortably under most thresholds, and your alarm never fires even though a customer just got a failed order. Sum counts the raw error, full stop.

Gotcha #2: set treat_missing_data to notBreaching. By default CloudWatch treats a period with zero data points as "insufficient data" and, depending on your comparison operator, can flip that into a breaching state. For functions that aren't invoked every 60 seconds — which is most functions — this generates noisy false alarms that train your team to ignore pages. We learned this the hard way after getting paged three nights in a row for a function that simply hadn't run.

resource "aws_cloudwatch_metric_alarm" "lambda_errors" {
  alarm_name          = "order-processor-errors"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 1
  metric_name         = "Errors"
  namespace           = "AWS/Lambda"
  period              = 60                # seconds — matches Lambda invocation metric granularity
  statistic           = "Sum"              # NOT Average — sum catches single failures
  threshold           = 1
  treat_missing_data  = "notBreaching"     # avoid false alarms on idle functions

  dimensions = {
    FunctionName = "order-processor"       # case-sensitive dimension key
  }

  alarm_actions = [aws_sns_topic.lambda_alerts.arn]
  ok_actions    = [aws_sns_topic.lambda_alerts.arn]

  alarm_description = "Fires when order-processor Lambda throws >=1 error in a 60s window"
}

One more detail: FunctionName is case-sensitive. I've hand-written enough raw JSON alarm definitions with a lowercase functionName to know CloudWatch won't complain — it just silently creates an alarm that matches nothing and never fires.

Step 3 — Wire the alarm action to the SNS topic and lock down the policy

Both alarm_actions and ok_actions point at the topic from Step 1, so the team hears about recovery too — not just the failure. Skipping ok_actions means people have to guess when it's safe to stop worrying, which is its own kind of alert fatigue.

The part people forget is the SNS resource policy. Without it, the alarm can be perfectly configured and still fail to notify anyone, because CloudWatch's service principal doesn't automatically have publish rights on your topic. The policy below restricts sns:Publish to cloudwatch.amazonaws.com, scoped with a condition on aws:SourceArn so only this specific alarm can use it.

resource "aws_sns_topic_policy" "allow_cloudwatch" {
  arn = aws_sns_topic.lambda_alerts.arn

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "AllowCloudWatchPublish"
      Effect    = "Allow"
      Principal = { Service = "cloudwatch.amazonaws.com" }
      Action    = "SNS:Publish"
      Resource  = aws_sns_topic.lambda_alerts.arn
      Condition = {
        ArnLike = {
          "aws:SourceArn" = aws_cloudwatch_metric_alarm.lambda_errors.arn
        }
      }
    }]
  })
}

While you're in here, add a near-identical alarm on the Throttles metric pointed at the same topic. Throttling is a distinct failure mode — the function isn't erroring, it's being rejected by concurrency limits — and it's the one people forget to monitor until a traffic spike takes down checkout for twenty minutes.

Step 4 — Reduce noise with a composite alarm (optional)

If you've added separate alarms for Errors, Throttles, and maybe Duration, you'll start getting duplicate pings for what's really one incident. Composite alarms have been GA since 2020 and are still underused — they let you combine multiple alarm states into a single OR/AND expression, so one Slack message fires instead of three.

aws cloudwatch put-composite-alarm takes an alarm rule expression like ALARM("order-processor-errors") OR ALARM("order-processor-throttles") and treats it as its own alarm object with its own actions. Add the composite alarm to a dashboard widget too — during an actual incident, having one visual panel that shows Errors, Throttles, and Duration side by side cuts triage time significantly compared to opening three separate metric graphs.

Verify and test

Don't trust this pipeline until you've forced it to fail and watched it recover. Invoke the function with a payload designed to throw, then check the alarm state directly rather than waiting on your inbox.

# 1. Force an invocation that raises an exception in the handler
aws lambda invoke \
  --function-name order-processor \
  --payload '{"forceError": true}' \
  --cli-binary-format raw-in-base64-out \
  response.json

cat response.json
# Expect: {"errorMessage": "...", "errorType": "..."}

# 2. Wait ~90s, then check alarm state
aws cloudwatch describe-alarms \
  --alarm-names order-processor-errors \
  --query 'MetricAlarms[0].StateValue'
# Expected output: "ALARM"

# 3. Confirm the transition and reason
aws cloudwatch describe-alarm-history \
  --alarm-name order-processor-errors \
  --history-item-type StateUpdate \
  --max-records 1

# Sample output snippet:
# "HistorySummary": "Alarm updated from OK to ALARM",
# "HistoryData": "{\"newState\":{\"stateValue\":\"ALARM\",...}}"

Expect a 60-120 second delay between the failed invocation and the state transition — that's the metric aggregation period plus alarm evaluation lag, not a bug. Don't expect sub-minute alerting no matter how tightly you tune the period.

Check the email inbox — and the spam folder, since first-time SNS notification emails get flagged by some providers more often than you'd expect. If nothing arrives but describe-alarms shows ALARM, go straight to describe-alarm-history; it's the fastest way to confirm whether the alarm fired correctly and the problem is actually an unconfirmed subscription or a bad SNS resource policy. One caveat worth noting: this synchronous invoke test won't exercise your Throttles alarm — testing that requires concurrent invocations exceeding reserved concurrency, which is a separate load test.

Closing

This is a baseline pattern, not a finished monitoring strategy — the moment it works for order-processor, turn it into a reusable Terraform module and apply it to every critical Lambda in the account, tuning evaluation periods and thresholds per function's actual traffic pattern so a chatty function doesn't drown out a quiet one in false positives. Separate SNS topics by severity — critical versus warning — rather than dumping everything into one channel, because a CloudWatch alarm Lambda errors SNS setup that pages the same channel for every blip trains people to ignore it within a week. Pair it with a CloudWatch Logs Insights query for the actual error message, since the alarm itself only tells you "N errors occurred" — not why. We keep a small collection of these patterns documented over at kuryzhev.cloud for exactly this reason: the second time you build it should take ten minutes, not an afternoon.

Related

• More AWS automation and Lambda monitoring patterns
• Terraform modules and IaC checklists for production infrastructure
• Alerting and observability setups that avoid false-positive fatigue