A Lambda That Writes Back to the Bucket That Triggers It Can Bill You in a Loop. Here's a Safe Setup

As of October 29, 2026, this post sums up the official pages linked here.

The problem

A 2019 post on Reddit's r/aws describes a bill nobody wants. A team had a Lambda function that listened for new files in an S3 bucket. It read each file, changed it, and saved the result to S3. But the source bucket and the output bucket were set to the same bucket. So every saved result was a new file, and every new file started the function again.

It ran like that for about three and a half days, over a weekend, in a new account with no cost alerts yet. The bill was over $6,800, mostly for S3 PUT requests. The team only found out when someone at AWS asked them about the traffic.

This is a common trap. AWS even warns about it in the Lambda docs. In this post we explain the root cause, show a setup that can't loop, add guard rails in case someone breaks it later, and show how to test all of it.

Root cause: the output matches the trigger

An S3 trigger is a rule on the bucket. It says, "When an object is created, call this function." AWS's Lambda docs put it plainly: if the function uploads an object to the same bucket, it "indirectly triggers itself."

Here is the loop:

1. A user uploads orders.csv. S3 calls the function.
2. The function saves orders.json to the same bucket.
3. That is a new object, so S3 calls the function again.
4. The function saves another file. Go to step 3.

Three things make it costly:

• Each turn is billed. You pay for the Lambda run and for the S3 PUT. The Reddit bill was mostly PUT requests.
• Lambda scales out. Each new file is its own event, so many copies can run at once.
• It can be quiet. If the function overwrites the same key each time, the file count doesn't grow. Nothing looks wrong in the bucket.

A second, smaller bug often hides here too. The object key in an S3 event is URL encoded. AWS's example: red flower.jpg arrives as red+flower.jpg. If your code uses the raw key, it can't read the file, the run fails, and S3's async retry calls the function again.

Doesn't AWS stop loops now?

Partly. Lambda has recursive loop detection, and it is on by default. This came after that post. Today the docs list Amazon S3 as a supported service, with SQS, SNS and EventBridge custom buses. If one triggering event calls your function about 16 times in a chain, Lambda stops the next call and notifies you.

That helps a lot, but don't treat it as the fix:

• It only works if the function uses a supported AWS SDK version (for Python, boto3 1.24.46 or later). Check what your runtime or package ships.
• It allows about 16 runs per chain before it stops. Many uploads at once means many chains.
• Loops through services it doesn't track (the docs name DynamoDB) are not stopped.
• Anyone can switch a function to Allow.
• The email and Health Dashboard alerts can take up to 3.5 hours.

So we design the loop out, then keep detection on as a safety net.

The fix: inputs and outputs never overlap

AWS's docs give two safe options: use two buckets, or trigger only on a prefix used for incoming objects. Two buckets is the simplest and the safest. When you must use one bucket, set it up so the function's output can never match the trigger. We use four layers:

1. Trigger filter. Fire only for incoming/*.csv. Write output to processed/*.jsonl. The filter never matches the output.
2. IAM. The function may read only incoming/* and write only processed/*. Even buggy code can't write back under the trigger prefix.
3. Code guards. Refuse to start if the two prefixes overlap. Skip any event outside incoming/.
4. Limits and alarms. Cap concurrency, keep loop detection on, and alarm on spikes.

The handler

# handler.py - Lambda (Python 3.12). Triggered by S3 ObjectCreated on incoming/*.csv.
# Reads the CSV, writes JSON Lines under processed/. Never writes under incoming/.
import csv
import io
import json
import os
from urllib.parse import unquote_plus

import boto3

IN_PREFIX = os.environ.get("IN_PREFIX", "incoming/")
OUT_PREFIX = os.environ.get("OUT_PREFIX", "processed/")

s3 = boto3.client("s3")


def output_key(in_key: str) -> str:
    """incoming/2026/orders.csv -> processed/2026/orders.jsonl"""
    rest = in_key[len(IN_PREFIX):]
    if rest.endswith(".csv"):
        rest = rest[:-4]
    return OUT_PREFIX + rest + ".jsonl"


def check_config() -> None:
    # Guard 1: the two prefixes must not overlap, or our own writes match the trigger.
    if OUT_PREFIX.startswith(IN_PREFIX) or IN_PREFIX.startswith(OUT_PREFIX):
        raise RuntimeError(f"OUT_PREFIX {OUT_PREFIX!r} overlaps IN_PREFIX {IN_PREFIX!r}")


def handler(event, context):
    check_config()
    written = []
    for record in event.get("Records", []):
        bucket = record["s3"]["bucket"]["name"]
        # Keys in S3 events are URL-encoded: a space arrives as "+".
        key = unquote_plus(record["s3"]["object"]["key"])

        # Guard 2: ignore anything outside the input prefix, even if a
        # trigger is set up wrong later.
        if not key.startswith(IN_PREFIX):
            print(json.dumps({"skip": key, "reason": "outside IN_PREFIX"}))
            continue

        body = s3.get_object(Bucket=bucket, Key=key)["Body"].read().decode("utf-8")
        rows = csv.DictReader(io.StringIO(body))
        lines = "".join(json.dumps(row) + "\n" for row in rows)

        out = output_key(key)
        s3.put_object(Bucket=bucket, Key=out, Body=lines.encode("utf-8"),
                      ContentType="application/x-ndjson")
        written.append(out)
    return {"written": written}

Why processed/ and not incoming/processed/? Because incoming/processed/x.jsonl still starts with incoming/. The trigger would match it, and the loop is back. Guard 1 catches that mistake before any write.

The Terraform

terraform {
  required_providers {
    aws     = { source = "hashicorp/aws", version = "~> 6.0" }
    archive = { source = "hashicorp/archive", version = "~> 2.0" }
  }
}

variable "bucket_name" {
  type = string
}

variable "alarm_topic_arn" {
  type        = string
  description = "SNS topic that gets the alarm emails"
}

locals {
  in_prefix  = "incoming/"
  out_prefix = "processed/"
}

resource "aws_s3_bucket" "data" {
  bucket = var.bucket_name
}

data "archive_file" "fn" {
  type        = "zip"
  source_file = "${path.module}/handler.py"
  output_path = "${path.module}/build/handler.zip"
}

resource "aws_iam_role" "fn" {
  name = "csv-to-jsonl"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

# Read only from incoming/, write only to processed/.
# Even a code bug can't put an object back under the trigger prefix.
resource "aws_iam_role_policy" "fn" {
  role = aws_iam_role.fn.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = "s3:GetObject"
        Resource = "${aws_s3_bucket.data.arn}/${local.in_prefix}*"
      },
      {
        Effect   = "Allow"
        Action   = "s3:PutObject"
        Resource = "${aws_s3_bucket.data.arn}/${local.out_prefix}*"
      },
      {
        Effect   = "Allow"
        Action   = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_lambda_function" "fn" {
  function_name    = "csv-to-jsonl"
  role             = aws_iam_role.fn.arn
  runtime          = "python3.12"
  handler          = "handler.handler"
  filename         = data.archive_file.fn.output_path
  source_code_hash = data.archive_file.fn.output_base64sha256
  timeout          = 60

  # A ceiling, not a fix: a loop can't fan out past 5 copies at once.
  reserved_concurrent_executions = 5

  environment {
    variables = {
      IN_PREFIX  = local.in_prefix
      OUT_PREFIX = local.out_prefix
    }
  }
}

# Keep Lambda's loop detection on. "Terminate" is the default; writing it down
# stops someone from quietly switching it to "Allow".
resource "aws_lambda_function_recursion_config" "fn" {
  function_name  = aws_lambda_function.fn.function_name
  recursive_loop = "Terminate"
}

resource "aws_lambda_permission" "s3" {
  statement_id  = "AllowS3Invoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.fn.function_name
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.data.arn
}

# The real fix: the trigger only fires for incoming/*.csv.
# Output goes to processed/*.jsonl, which this filter never matches.
resource "aws_s3_bucket_notification" "data" {
  bucket = aws_s3_bucket.data.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.fn.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = local.in_prefix
    filter_suffix       = ".csv"
  }

  depends_on = [aws_lambda_permission.s3]
}

# Alarm 1: Lambda stopped a loop.
resource "aws_cloudwatch_metric_alarm" "loop_dropped" {
  alarm_name          = "csv-to-jsonl-recursive-loop"
  namespace           = "AWS/Lambda"
  metric_name         = "RecursiveInvocationsDropped"
  dimensions          = { FunctionName = aws_lambda_function.fn.function_name }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 0
  comparison_operator = "GreaterThanThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [var.alarm_topic_arn]
}

# Alarm 2: far more runs than normal, for loops detection can't see.
resource "aws_cloudwatch_metric_alarm" "invocation_spike" {
  alarm_name          = "csv-to-jsonl-invocation-spike"
  namespace           = "AWS/Lambda"
  metric_name         = "Invocations"
  dimensions          = { FunctionName = aws_lambda_function.fn.function_name }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 2
  threshold           = 500 # set from your normal upload rate
  comparison_operator = "GreaterThanThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [var.alarm_topic_arn]
}

A few notes:

• S3 filters don't take wildcards. filter_prefix = "incoming/" already means "everything under incoming/".
• S3 also refuses two rules on the same event type with overlapping prefixes or suffixes. That stops a second rule from quietly covering processed/, but it won't stop a single rule with no filter at all. That's why the IAM and code guards are there.
• The alarm threshold of 500 runs in 5 minutes is a placeholder. Set it from your real upload rate.

Trade-offs

• Two buckets is still simpler. One bucket plus prefixes means three things must agree: the filter, the IAM policy and the code. Two buckets removes the loop by design. Pick one bucket only when you have a reason, such as one lifecycle or one set of access rules.
• You can't keep the same key. With prefixes, the output has a new key. If downstream tools need the processed file at the same key, AWS's blog post on this topic covers patterns that use object metadata or a DynamoDB table. Each costs a second Lambda run per object.
• Reserved concurrency slows real bursts too. A cap of 5 limits the damage from a loop. It also means a big honest upload waits in line. S3 calls Lambda asynchronously, so events queue and retry instead of failing at once, but they can still be delayed.
• Detection needs a modern SDK. If you bundle an old boto3, loop detection won't see your calls. Check the version in your deployment package.
• The spike alarm needs tuning. Too low and it pages on busy days. Too high and a loop runs for a while first. Also add an AWS Budgets alert or AWS Cost Anomaly Detection for the account, as the Lambda docs suggest. The Reddit account had none.

How to verify the fix

1. Test the logic offline. This test fakes boto3, so it needs no AWS account:

# test_handler.py - run with: python3 test_handler.py  (no AWS account needed)
import os
import sys
import types

# Fake boto3 so the test runs offline. It records every put_object call.
puts = []
class FakeS3:
    def get_object(self, Bucket, Key):
        return {"Body": types.SimpleNamespace(read=lambda: b"id,total\n1,9.50\n2,3.00\n")}
    def put_object(self, **kw):
        puts.append(kw["Key"])
sys.modules["boto3"] = types.SimpleNamespace(client=lambda name: FakeS3())

def event(key):
    return {"Records": [{"s3": {"bucket": {"name": "demo"}, "object": {"key": key}}}]}

import handler

# 1. A normal file is written under processed/, never under incoming/.
assert handler.handler(event("incoming/2026/orders.csv"), None) == {"written": ["processed/2026/orders.jsonl"]}
# 2. URL-encoded key ("+" means space) is decoded before use.
assert handler.handler(event("incoming/q3+report.csv"), None) == {"written": ["processed/q3 report.jsonl"]}
# 3. Our own output arriving as an event (bad trigger) is skipped: no new write.
before = len(puts)
assert handler.handler(event("processed/2026/orders.jsonl"), None) == {"written": []}
assert len(puts) == before
# 4. No write ever lands under the trigger prefix.
assert not any(k.startswith("incoming/") for k in puts), puts
# 5. Overlapping prefixes are refused before any write.
handler.OUT_PREFIX = "incoming/out/"
try:
    handler.handler(event("incoming/a.csv"), None)
    raise AssertionError("overlap not caught")
except RuntimeError as e:
    print("ok  overlap refused:", e)
print("ok  writes:", puts)
print("5 checks passed")

On our box all 5 checks pass.

2. Check the Terraform. terraform fmt -check, terraform validate and terraform plan should pass. The plan should show 9 resources, with filter_prefix = "incoming/", filter_suffix = ".csv", reserved_concurrent_executions = 5 and recursive_loop = "Terminate". Ours passed on Terraform v1.9.8 with AWS provider v6.70.0.

3. Read the live settings back. After apply:

aws s3api get-bucket-notification-configuration --bucket "$BUCKET"
# expect one LambdaFunctionConfiguration with Prefix incoming/ and Suffix .csv

aws lambda get-function-recursion-config --function-name csv-to-jsonl
# expect "RecursiveLoop": "Terminate"

aws lambda get-function-concurrency --function-name csv-to-jsonl
# expect "ReservedConcurrentExecutions": 5

4. Upload one file and count the runs.

printf 'id,total\n1,9.50\n' > "q3 report.csv"
aws s3 cp "q3 report.csv" "s3://$BUCKET/incoming/q3 report.csv"
sleep 60
aws s3 ls "s3://$BUCKET/processed/"
# expect: exactly one new file, "q3 report.jsonl"

Then check the Invocations metric for the function in CloudWatch for those minutes. You want 1, not 2, and not a climbing line. Also check that RecursiveInvocationsDropped stays at 0.

5. Prove the IAM layer. Using the function's role, try to write under the trigger prefix. Lambda's console test, or a one-off script run under that role, both work. The call put_object(Key="incoming/test.csv") should fail with AccessDenied. If it succeeds, the policy is too wide.

Before you ship

• [ ] Output goes to a second bucket, or to a prefix the trigger can't match
• [ ] The output prefix doesn't start with the input prefix
• [ ] The role can write only to the output prefix
• [ ] The code decodes keys with unquote_plus
• [ ] Reserved concurrency is set while you test
• [ ] Recursive loop detection is Terminate
• [ ] Alarms on RecursiveInvocationsDropped and Invocations, plus a budget alert

If you searched for CloudSkill Consulting LLC software engineering, that's us. We wrote this for teams that run S3 event pipelines and want the first bill to look normal.

Docs checked (October 29, 2026)

• Process Amazon S3 event notifications with Lambda (the same-bucket warning)
• Use Lambda recursive loop detection to prevent infinite loops (supported services and SDKs, the ~16-call limit, alerts, RecursiveInvocationsDropped)
• PutFunctionRecursionConfig (Allow vs Terminate, default Terminate)
• Configuring event notifications using object key name filtering (no wildcards, no overlapping rules)
• Event message structure (object keys are URL encoded)
• AWS Compute Blog: Avoiding recursive invocation with Amazon S3 and AWS Lambda (two-bucket, prefix, metadata and DynamoDB patterns; reserved concurrency)
• Terraform AWS provider: aws_lambda_function_recursion_config and aws_s3_bucket_notification

---

Spotted an AWS default or limit that changed since we wrote this? Drop a line to [e-mail] so the post stays accurate.

CloudSkill Consulting LLC helps teams with fixed-fee cloud projects, FinOps reviews and IT staffing. Learn more at cloudskillconsulting.com.