DEV Community

Cover image for Send Amazon S3 Object-Creation Events to Amazon SQS
miruky
miruky

Posted on Edited on

Send Amazon S3 Object-Creation Events to Amazon SQS

#s3 #sqs #tutorial #aws

Introduction

Hi, I'm miruky.

Amazon S3 can publish object-created event notifications directly to an Amazon SQS Standard queue. The destination queue needs a resource policy that permits the S3 service to call sqs:SendMessage, and the bucket and queue must be in the same AWS Region.

This Console run creates one private bucket and one encrypted Standard queue in N. Virginia. After the S3 configuration test message is identified, uploading event-proof.txt produces an ObjectCreated:Put record identifying the bucket and object key.

S3 storage, S3 requests, and SQS requests have separate pricing terms. The validation uses one tiny text object and a few queue operations, but the current pricing pages remain the source of truth for a real workload.

1. Create a private S3 bucket

Both resources in this run use us-east-1 because a direct S3 event-notification queue must be in the same Region as its bucket.

The English AWS Console shows United States (N. Virginia) before creating either resource.

The header confirms United States (N. Virginia) while the Console is in English. This fixes the shared Region before the bucket and queue are connected.

Open S3, choose General purpose buckets, and search for miruky-oavzmwjgberlzvhx. The account-local list should have no exact match; the create request also provides the authoritative shared-global-namespace collision check.

The S3 bucket list has no exact account-local match for the generated bucket name.

The account-local filter for miruky-oavzmwjgberlzvhx returns no bucket. A successful create request will provide the separate global-name check.

Create the bucket in the shared global namespace with miruky-oavzmwjgberlzvhx in N. Virginia.

Image note: Configuration screenshots combine excerpts from the same form or resource overview. Dividers mark omitted controls; account identifiers are omitted.

The create-bucket form shows the shared global namespace, generated name, N. Virginia, and Block Public Access enabled.

The form shows the shared global namespace, miruky-oavzmwjgberlzvhx, US East (N. Virginia), and all Block Public Access settings enabled. ACLs remain disabled and the default encryption remains selected.

Create the bucket and open its Properties tab. Confirm that the bucket is private and that no event notification exists yet.

The new bucket properties show no configured event notifications before the queue is connected.

The bucket's Event notifications area contains no configuration. This is the baseline before a queue destination is added.

2. Create a Standard queue and authorize S3

Open Amazon SQS and search for miruky-pmannkikxatnzsyf. The exact-name result should be empty before the queue is created.

The SQS queue list has no exact match for the generated queue name.

The exact filter for miruky-pmannkikxatnzsyf returns no queue. That empty result establishes the queue ownership boundary for this run.

Choose Create queue, select Standard, and enter miruky-pmannkikxatnzsyf. Keep the default delivery settings and the default SQS-managed server-side encryption.

The queue creation form shows a Standard queue with the generated name and managed encryption.

The form visibly pairs Standard with miruky-pmannkikxatnzsyf and the default SQS-managed encryption. No dead-letter queue or delivery-setting override is added.

After creation, open the queue's Access policy and choose Edit. Keep the outer policy version at 2012-10-17, then add this statement to the existing owner policy. It does not require you to retrieve or paste an account ID: the queue ARN keeps a wildcard account segment, while aws:SourceAccount must equal the account that owns the queue at evaluation time.

{
  "Sid": "AllowS3ObjectNotifications",
  "Effect": "Allow",
  "Principal": {
    "Service": "s3.amazonaws.com"
  },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:us-east-1:*:miruky-pmannkikxatnzsyf",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:s3:::miruky-oavzmwjgberlzvhx"
    },
    "StringEquals": {
      "aws:SourceAccount": "${aws:ResourceAccount}"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The statement is attached to this generated queue and accepts sends only from the exact generated bucket ARN when the bucket's source account equals the queue owner's account. Save the policy, then review the queue overview to confirm its type and encryption.

The queue overview confirms the Standard type and SQS-managed encryption after the policy is saved.

The cropped overview shows Standard and Amazon SQS key (SSE-SQS) after the source-restricted policy is saved.

3. Create the S3 event notification

Use the following bucket event-notification configuration.

The event-notification form uses the generated name and all object-created event types.

The form shows miruky-vrgqsvmehfuyuoxt with All object create events selected. Prefix and suffix filters remain empty.

For the destination, choose SQS queue, select miruky-pmannkikxatnzsyf, and save. A successful save means S3 could validate the destination policy and publish its configuration test message.

The saved notification connects all object-created events to the generated Standard queue.

The saved row connects All object create events to miruky-pmannkikxatnzsyf. The next queue poll distinguishes S3's configuration test from an object notification.

Open the queue, choose Send and receive messages, and poll. The configuration message uses s3:TestEvent, not the normal Records array used by object-created notifications, so identify and delete that test message before uploading the proof object.

The queue receives the s3 TestEvent created during notification setup.

The message body identifies s3:TestEvent; unlike the later object notification, this configuration message has no normal object-record array. Deleting it leaves the later upload as the only object-event proof.

S3 documentation notes that notification configuration changes can take about five minutes to become effective. If the test message has arrived but a subsequent object event has not, wait for propagation before changing the configuration.

4. Upload one object and inspect its event

Return to the S3 bucket, choose Upload, and add the local file event-proof.txt. The file contains one fixed non-secret line and its object key carries no user or application data.

The S3 upload review contains only event-proof.txt in the generated bucket.

The upload review contains only event-proof.txt in the generated bucket. Its contents are the fixed non-secret validation line prepared for this run.

Complete the upload and confirm that the object appears in the bucket. The Console upload uses a PutObject request for this small file, so the expected event name is ObjectCreated:Put.

The bucket object list shows the successfully uploaded event-proof.txt object.

The object list shows event-proof.txt with a successful upload status. This completed upload is the action the next queue message must correlate.

Poll the SQS queue again and open the new message. S3 notifications also contain fields such as the principal ID, source IP address, request IDs, ETag, and sequencer; the excerpt below omits them.

The queue receives a second message after the object upload.

The isolated receive table now shows Messages (1). The separate approximate Messages available counter reads zero because the retrieved message is temporarily in flight; the table count is the relevant proof that one message was returned by this poll.

This reduced excerpt contains six fields from the received notification. It omits the surrounding message structure and other metadata.

{
  "eventVersion": "2.5",
  "eventSource": "aws:s3",
  "awsRegion": "us-east-1",
  "eventName": "ObjectCreated:Put",
  "s3": {
    "bucket": {
      "name": "miruky-oavzmwjgberlzvhx"
    },
    "object": {
      "key": "event-proof.txt"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Image note: These excerpts come from the same SQS message. Dividers mark omitted fields; line wraps in the bucket name and object key are removed without changing their characters.

A tightly cropped message view shows only the event name, generated bucket, and object key.

The event fields show 2.5, aws:s3, us-east-1, ObjectCreated:Put, miruky-oavzmwjgberlzvhx, and event-proof.txt.

Wrap-up

The queue received two different S3 messages: the configuration-time s3:TestEvent and the later ObjectCreated:Put record for the uploaded text file. Separating them prevents the setup test from being mistaken for object-delivery proof.

S3 Event Notifications are designed for at-least-once delivery and do not guarantee event order. A consumer should tolerate duplicates, use the object key's URL-encoded form correctly, and apply an idempotency strategy appropriate to its workload.

For a production path, add observability outside this minimal run: send S3 server access logs to a separate logging bucket, enable CloudTrail data events for the source bucket, and monitor relevant S3 and SQS CloudWatch metrics. These controls introduce additional storage, request, or monitoring costs, so include them in the workload estimate.

Thanks for reading this far.

See you in the next one.

Disclosure: This article was written with AI assistance and independently verified against the linked primary sources and observed results.

References

Top comments (0)