We cut a Lambda bill 96.5% by fixing a retry storm.

The invocation graph is the first thing I open on any serverless engagement, and this one looked wrong from across the room. 29.3K Lambda invocations per hour. For a workload that justified a few hundred.

Nobody had noticed, because nothing was down. That's the ugly property of a retry storm. The system stays green while it burns money. Every retry eventually succeeds, lands in a dead-letter queue, or quietly expires. Dashboards show traffic, and the only symptom is a bill that grows faster than the product does.

What a retry storm actually is

AWS retries on your behalf in more places than most teams can list. Async Lambda invocations retry twice by default. An SQS event source leaves failed messages on the queue, and after the visibility timeout they become visible again and are redelivered. EventBridge has its own retry policy. Step Functions too. Each layer is individually reasonable. Stacked together, they multiply.

The failure mode that bit this client was surprisingly simple. The handler wrapped everything in try/catch, logged the error, and re-threw. Everything. Including validation errors on malformed messages. A malformed message can never succeed, but the code told SQS, "please try again."

Without a redrive policy, SQS keeps redelivering a failing message until the retention period expires. The default retention period is four days. One poison message became four days of Lambda invocations. There were thousands of poison messages, and the function that consumed their output failed too, creating its own retry storm downstream.

The fix

Four changes, in order of impact.

1. Classify errors before throwing

Retryable and terminal errors are different animals. A network timeout deserves another attempt. A schema violation belongs in a parking lot, not in an infinite retry loop.

export const handler = async (event) => {
  const batchItemFailures = [];

  for (const record of event.Records) {
    try {
      await process(JSON.parse(record.body));
    } catch (err) {
      if (err instanceof ValidationError) {
        await parkMessage(record, err);
        continue;
      }

      batchItemFailures.push({
        itemIdentifier: record.messageId,
      });
    }
  }

  return { batchItemFailures };
};

This uses SQS partial batch responses, so only the failed messages are retried. Without this feature, one failed record can cause the entire batch to be retried, including messages that were already processed successfully.

2. Configure a dead-letter queue with a sensible maxReceiveCount

Three attempts, then move the message to a DLQ and alert on queue depth.

{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:eu-west-1:123456789:orders-dlq",
    "maxReceiveCount": 3
  }
}

The DLQ is not a trash can. It is a work queue for humans. Someone should periodically inspect it, fix the underlying issue, and decide whether those messages should be replayed.

3. Add a circuit breaker around flaky downstream services

When a dependency is down, ten thousand Lambda invocations hammering it will not make it recover any faster. Fail fast, back off, and let SQS absorb the burst.

One caveat: circuit breakers are awkward in Lambda because execution environments are ephemeral. The breaker state has to live somewhere external. We stored ours in a DynamoDB item with a TTL. It worked well enough, even if it was not the prettiest implementation.

4. Tune timeouts to reality

The function timeout was set close to the Lambda maximum "to be safe." Safe for whom?

Long timeouts increase the maximum cost of every failed invocation and delay retries. We set the timeout to our p99 execution time plus some headroom. That reduced the cost of failures while still giving healthy requests enough time to complete.

The result

Lambda invocations dropped from 29.3K per hour to 669. That's a 97.7% reduction. The Lambda line item fell by 96.5%.

Same product. Same traffic. Same features.

The only difference was that we stopped paying to retry work that could never succeed.

Don't forget idempotency

Retries only work safely if your processing is idempotent.

If processing the same order twice sends two emails, ships two packages, or charges the customer twice, retries become a business problem instead of an infrastructure problem.

Whenever you rely on retries, make sure your handlers can safely process the same message multiple times.

How to catch this in your own account

Spend ten minutes checking these today.

• Track the ratio of Errors / Invocations in CloudWatch. Alert when the error rate is consistently above your normal baseline.
• For every SQS-triggered Lambda, verify the queue has a redrive policy. Without a DLQ, poison messages keep cycling until the retention period expires.
• Compare Lambda invocations with real business events. If you process 10,000 orders per day but see 300,000 invocations, retries are probably inflating the numbers.
• Review MaximumRetryAttempts for asynchronous Lambda invocations. The default is two retries, but for some idempotency-sensitive workloads you may want zero.

Retries are a good default for a world where failures are often transient. They are a terrible default for deterministic failures. Your error handling is what tells AWS which kind it is.

If your AWS bill feels wrong and you'd rather not gamble on another generic cloud consultancy, I do this at reducecost.cloud on a pay-for-savings basis. Zero upfront. You only pay a share of what I actually save you.