Originally published on kuryzhev.cloud
We had a client whose ingestion pipeline "worked fine" for eight months. Then a partner started uploading larger batch files, retries kicked in, and roughly 3% of objects just vanished — no error in the logs, no alert, nothing in CloudWatch. The root cause wasn't a bug in their code. It was lambda s3 trigger error handling that nobody had actually designed, because the trigger had always quietly retried and succeeded before.
This is the part of AWS serverless architecture that people treat as plumbing you don't need to think about. You do. S3-to-Lambda triggers have a specific delivery model, specific retry semantics, and specific failure modes that will bite you the moment volume or file size changes.
What this actually does
When an object lands in S3, the bucket emits an event notification — either through the bucket's own notification configuration or through EventBridge. That delivery is asynchronous and at-least-once. S3 does not wait for your Lambda to finish, and it does not guarantee exactly-once delivery. Duplicate invocations for the same object are expected behavior, not an edge case.
Lambda doesn't receive the file. It receives metadata: bucket name, object key, size, eTag, and versionId if versioning is on. Your function has to call GetObject itself to get the actual bytes. People sometimes assume the event payload contains content — it never does.
The invocation is async, which matters a lot for retries. S3 doesn't retry anything — Lambda's async invocation layer does, with two automatic retries by default (configurable via MaximumRetryAttempts, 0–2). After that, if you've configured a failure destination or DLQ, the event goes there. If you haven't, it's discarded silently.
One more assumption that quietly breaks pipelines: people expect "one event, one file, one invocation." S3 can batch multiple records into a single invocation payload, especially under load. If your handler assumes event['Records'] has exactly one item, you'll drop data the first time it doesn't.
How people use it wrong
The most common failure I see is no idempotency check. Since S3 delivers at-least-once, the same object can trigger your function twice. Without a dedup mechanism, that means duplicate database rows, double-charged invoices, or duplicate downstream messages. Teams usually don't notice until someone in finance asks why a report double-counted revenue for one afternoon.
Second: nobody sets a DLQ. It feels optional because the happy path never needs it. Then a schema change causes every invocation to throw, Lambda retries twice, and the events are gone. There's no trace they ever existed. Watch out for this specifically — CloudWatch will show two failed invocations, then nothing, and you'll assume the problem "resolved itself."
Third is the recursive invocation trap. Someone writes the processed output back into the same prefix that triggers the function. Now every write triggers another invocation, which writes again, which triggers again. This can burn through account concurrency limits in minutes, and I've seen it generate a surprising AWS bill overnight.
Fourth: wrapping the entire handler in one try/except and raising on the first bad record. If a batch has five records and one is malformed, you lose all five — the good four get reprocessed unnecessarily or, worse, get lost if retries also fail.
Fifth: no reserved concurrency. A bulk S3 sync of 10,000 files spikes concurrent invocations instantly. Without limits, you either throttle account-wide or silently drop events because Lambda can't absorb the burst.
The correct approach
Structure the handler to iterate records independently and collect failures instead of raising on the first error. This is the single biggest change that fixes most of the problems above.
import json
import logging
import boto3
from botocore.exceptions import ClientError
s3 = boto3.client("s3")
dynamodb = boto3.resource("dynamodb").Table("processed-objects")
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def handler(event, context):
failures = [] # collect per-record failures instead of raising on first error
for record in event.get("Records", []):
bucket = record["s3"]["bucket"]["name"]
key = record["s3"]["object"]["key"]
etag = record["s3"]["object"]["eTag"]
try:
# idempotency guard: skip if this eTag was already processed
dynamodb.put_item(
Item={"pk": f"{bucket}/{key}/{etag}", "status": "processing"},
ConditionExpression="attribute_not_exists(pk)",
)
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
logger.info(f"Skipping duplicate delivery for {key}")
continue
raise
try:
# stream the object instead of loading it fully — matters for large files
obj = s3.get_object(Bucket=bucket, Key=key)
body = obj["Body"].read()
process(body)
except Exception as err:
logger.error(f"Failed processing {key}: {err}", exc_info=True)
failures.append({"itemIdentifier": record.get("responseElements", key)})
if failures:
# for SQS-backed sources, this triggers partial batch redelivery only
return {"batchItemFailures": failures}
def process(body: bytes):
data = json.loads(body)
# actual business logic here
The idempotency guard uses a DynamoDB conditional write on eTag — this is cheap, atomic, and doesn't require a separate locking layer. If the write fails with ConditionalCheckFailedException, you already processed this exact object version, so skip it instead of reprocessing.
Wire an SQS-based DLQ or an OnFailure destination on the event configuration, and actually alert on DLQ depth — a DLQ nobody watches is just a slower way to lose events. Better yet, decouple ingest from processing entirely: S3 → SQS → Lambda. This removes you from S3's fixed two-retry policy and gives you visibility, backoff control, and replay ability.
Here's what the raw event actually looks like, which is worth memorizing so you stop assuming it contains file content:
// Sample event payload delivered to the Lambda — note it's metadata only,
// and can contain multiple records in one invocation.
{
"Records": [
{
"eventTime": "2026-01-14T09:32:01.000Z",
"eventName": "ObjectCreated:Put",
"s3": {
"bucket": { "name": "ingest-raw-logs" },
"object": {
"key": "uploads/2026/01/14/log-4471.json.gz",
"size": 20481,
"eTag": "d41d8cd98f00b204e9800998ecf8427e",
"versionId": "3sL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY"
}
}
}
]
}
Add structured JSON logging with the bucket, key, and requestId as fields. When something goes wrong at 2am, a CloudWatch Logs Insights query filtered on the object key is the fastest way to trace its full retry history across invocations — far faster than scrolling raw log lines.
Advanced patterns
Once you're past basic correctness, a few patterns show up repeatedly in production systems that actually scale.
Fan-out via EventBridge. S3 bucket notifications only support one destination per event type per prefix/suffix combination. If two different teams need the same "object created" event, direct S3 notifications will fight each other. Route through EventBridge instead — one rule, multiple targets, no config collisions.
Batch failure isolation with SQS. When Lambda consumes from SQS, enable ReportBatchItemFailures in the event source mapping. Without it, one bad record in a batch of ten forces redelivery of the entire batch, including the nine that already succeeded — and now you're reprocessing work you already did, which reopens the idempotency question you thought you'd solved.
Large objects need a different shape entirely. If a file might exceed your memory or timeout budget, don't try to force it through a single Lambda invocation. Trigger a Step Functions workflow instead, and let the state machine handle chunking, retries, and long-running steps. See the AWS Step Functions documentation for the orchestration patterns that fit here.
Cross-account triggers get messy fast. If the consuming Lambda lives in a different account than the bucket, direct bucket-to-Lambda permissions require resource policies on both sides and tend to break during account migrations. Route through SNS or EventBridge instead — it's more moving parts, but the permission model is far more predictable.
Chaos-test your error paths before production does it for you. Upload malformed JSON, zero-byte files, and duplicate deliveries into a staging bucket. If your DLQ and idempotency logic haven't been exercised deliberately, you don't actually know they work — you're just hoping.
Performance notes
Memory allocation in Lambda scales CPU proportionally. Bumping from 128MB to 512MB often cuts processing time enough that total cost drops despite the higher per-millisecond rate — this shows up clearly on parsing- or compression-heavy workloads like gzip decompression of log files.
Cold starts matter more here than in typical API-driven Lambdas, because S3 events are bursty by nature. A bulk upload of thousands of files spikes concurrency instantly, and a wave of cold starts adds real latency across that spike. Provisioned concurrency helps for latency-sensitive pipelines, but it costs you even when idle — size it to expected burst, not average traffic.
Avoid loading entire objects into memory. For anything multi-gigabyte, use range requests and stream processing instead of a single GetObject().read() call — otherwise you're paying for memory you don't need and risking OOM kills mid-processing.
Buffering through SQS smooths bursty invocation spikes and decouples your Lambda concurrency from S3's raw upload rate. For high-frequency small-file triggers — think IoT logs landing every second — batching through SQS is meaningfully cheaper than invoking a Lambda per single PUT, and it reduces throttling-driven waste during traffic spikes.
None of this is exotic. It's the difference between a trigger that "usually works" and one that survives retries, bursts, and the inevitable bad file someone uploads at the worst possible time. If you're building this from scratch, start with the official AWS S3-to-Lambda trigger documentation and layer the idempotency and DLQ patterns above on top before you ship. We cover more of these serverless failure patterns over at kuryzhev.cloud if you want the broader AWS operations context.
Top comments (0)