S3 Event Notifications can lose events if you misconfigure the underlying destination, yet many engineers treat them as a reliable message queue. When your S3 bucket receives thousands of files per minute, a misconfigured downstream consumer—such as a Lambda function with insufficient concurrency limits—leads to silent event loss or massive retry storms that overwhelm your downstream databases.
This article explains the architectural differences between direct S3 Event Notifications and the EventBridge-based approach. You will learn how to shift from a direct, "fire-and-forget" notification model to a robust, event-driven pattern using Amazon EventBridge. This scope covers the configuration of both mechanisms and the implications for delivery reliability and observability; it does not cover S3 Batch Operations or S3 Inventory reports.
What you'll build
You will implement an event-driven architecture that captures S3 object creation events and routes them through EventBridge to a consumer.
- A source S3 bucket configured to publish events to the default EventBridge bus.
- An EventBridge rule to filter specific object-creation events.
- A target Amazon SQS queue to decouple the arrival event from the processor, providing buffering and retry capabilities.
- A Lambda function as an idempotent consumer.
flowchart LR
S3["Amazon S3 (bucket)"] --> EB["Amazon EventBridge (default bus)"]
EB --> SQS["Amazon SQS (queue)"]
SQS --> Lambda["AWS Lambda (processor)"]
Prerequisites
- An AWS account with access to the AWS CLI v2 and IAM permissions to create buckets, rules, and queues.
- AWS CLI configured with a profile that has
AdministratorAccessfor the purpose of this tutorial. - The tutorial assumes you are working in a single Region, such as
us-east-1.
Step 1: Configure S3 to publish to EventBridge
By default, S3 buckets do not send events to EventBridge. You must explicitly enable this via the bucket's notification configuration. Enabling this allows you to leverage EventBridge's routing capabilities instead of tight coupling to a single destination.
Run the following command to enable event notifications for your bucket:
aws s3api put-bucket-notification-configuration \
--bucket <your-unique-bucket-name> \
--notification-configuration '{
"EventBridgeConfiguration": {}
}'
Verify this configuration by running:
aws s3api get-bucket-notification-configuration \
--bucket <your-unique-bucket-name>
You should see "EventBridgeConfiguration": {} in the JSON output.
Step 2: Create a buffer queue
Directly invoking Lambda from S3 events is common, but using an SQS queue as an intermediary provides a safety net. If your processing logic fails or hits a concurrency limit, the events remain in the queue rather than being lost or forcing an immediate, potentially destructive retry.
aws sqs create-queue --queue-name s3-event-buffer
Capture the Queue URL from the output for use in the next step.
Step 3: Create an EventBridge rule
The rule acts as a filter. Instead of receiving every S3 event, we only want to trigger our pipeline when a file is uploaded to a specific prefix (e.g., uploads/).
Create a file named rule.json:
{
"source": ["aws.s3"],
"detail-type": ["Object Created"],
"detail": {
"bucket": {
"name": ["<your-unique-bucket-name>"]
},
"object": {
"key": [{ "prefix": "uploads/" }]
}
}
}
Now, create the rule and link it to your SQS queue:
aws events put-rule \
--name s3-to-sqs-rule \
--event-pattern file://rule.json
aws events put-targets \
--rule s3-to-sqs-rule \
--targets "Id"="target1","Arn"="arn:aws:sqs:<region>:<account-id>:s3-event-buffer"
To verify, upload a test file to the bucket:
touch test.txt
aws s3 cp test.txt s3://<your-unique-bucket-name>/uploads/test.txt
Check the SQS queue depth after a few seconds:
aws sqs get-queue-attributes \
--queue-url <your-queue-url> \
--attribute-names ApproximateNumberOfMessages
Step 4: Configure Lambda with an SQS trigger
Your Lambda function should read from the SQS queue. Because SQS allows for partial batch failures, ensure your code logic handles individual messages and only deletes them from the queue upon successful processing.
When writing your consumer, assume messages might arrive more than once. Use the S3 Object ETag or the event timestamp combined with the object key to ensure your data sink remains idempotent.
Design decisions
- EventBridge, not direct S3 notifications. S3 event notifications allow you to send events directly to SQS, SNS, or Lambda. However, EventBridge provides a centralized bus where you can add multiple consumers—such as a logging service, an alerting service, and your data pipeline—without modifying the S3 bucket configuration.
- SQS buffer, not direct Lambda invocation. Direct invocation is simpler but lacks a native retry buffer. If your Lambda function fails due to an upstream database timeout, direct invocations are harder to manage. SQS provides a Dead Letter Queue (DLQ) mechanism, which is critical for handling poison messages.
- Filtering at the bus, not at the processor. By using EventBridge rules to filter events, you save compute costs. You only pay for the events that trigger your downstream targets, rather than invoking a Lambda function just to have it check a string and exit.
Troubleshooting
- Symptom: Events are not appearing in the SQS queue. Check if the SQS queue policy allows the
events.amazonaws.comservice principal to performsqs:SendMessage. Without this, EventBridge cannot deliver the message. - Symptom: Lambda is not receiving events. Ensure the Lambda function is mapped to the SQS trigger and that the execution role has
sqs:ReceiveMessage,sqs:DeleteMessage, andsqs:GetQueueAttributespermissions. - Symptom: High latency in processing. Verify the batch size and batch window settings on your SQS-to-Lambda trigger. If the batch window is set to a high value, Lambda waits until that duration passes or the batch size is reached before executing.
Clean up
To avoid ongoing charges for these resources, delete the created components:
# Delete the rule and targets
aws events remove-targets --rule s3-to-sqs-rule --ids target1
aws events delete-rule --name s3-to-sqs-rule
# Delete the queue
aws sqs delete-queue --queue-url <your-queue-url>
# Remove bucket notification
aws s3api put-bucket-notification-configuration \
--bucket <your-unique-bucket-name> \
--notification-configuration '{}'
Conclusion
By using EventBridge as the intermediary for S3 file events, you gain the ability to decouple your data producers from your consumers and add multiple downstream listeners without operational complexity. Using an SQS queue between EventBridge and your processing logic provides the necessary buffer to handle surges and failures gracefully.
Try it: Modify the EventBridge rule pattern to trigger only on specific file extensions (like .parquet or .csv) to further refine which files enter your data pipeline.
References
Tags: #aws #s3 #eventdriven #dataengineering
About the examples: automated checks confirmed that the JSON block parses and that 9 AWS CLI commands match the AWS service API models (operation and parameter names). The examples have not been run end to end, so test them in a non-production environment before relying on them.
Cover photo by Dries Vanlerberghe on Unsplash.
This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.
Top comments (0)