miruky

Posted on Sep 1

Send Amazon S3 Object-Creation Events to Amazon SQS

#s3 #sqs #tutorial #aws

Introduction

Hi, I'm miruky.

Amazon S3 can publish object-created event notifications directly to an Amazon SQS Standard queue. The destination queue needs a resource policy that permits the S3 service to call sqs:SendMessage, and the bucket and queue must be in the same AWS Region.

This Console run creates one private bucket and one encrypted Standard queue in N. Virginia. After the S3 configuration test message is identified, uploading event-proof.txt produces an ObjectCreated:Put record that names only the generated bucket and synthetic object key in the article evidence.

S3 storage, S3 requests, and SQS requests have separate pricing terms. The validation uses one tiny text object and a few queue operations, but the current pricing pages remain the source of truth for a real workload.

1. Create a private S3 bucket

Both resources in this run use us-east-1 because a direct S3 event-notification queue must be in the same Region as its bucket.

The header confirms United States (N. Virginia) while the Console is in English. This fixes the shared Region before the bucket and queue are connected.

Open S3, choose General purpose buckets, and search for miruky-oavzmwjgberlzvhx. The account-local list should have no exact match; the create request also provides the authoritative shared-global-namespace collision check.

The account-local filter for miruky-oavzmwjgberlzvhx returns no bucket. A successful create request will provide the separate global-name check.

Create the bucket in the shared global namespace with miruky-oavzmwjgberlzvhx in N. Virginia. The account regional namespace is outside this article because its name format can expose an account ID.

The next image vertically stacks two crops from the same create-bucket form. The divider marks the omitted middle controls; it does not join two different runs.

The form shows the shared global namespace, miruky-oavzmwjgberlzvhx, US East (N. Virginia), and all Block Public Access settings enabled. ACLs remain disabled and the default encryption remains selected.

Create the bucket and open its Properties tab. Confirm that the bucket is private and that no event notification exists yet.

The bucket's Event notifications area contains no configuration. This is the baseline before a queue destination is added.

2. Create a Standard queue and authorize S3

Open Amazon SQS and search for miruky-pmannkikxatnzsyf. The exact-name result should be empty before the queue is created.

The exact filter for miruky-pmannkikxatnzsyf returns no queue. That empty result establishes the queue ownership boundary for this run.

Choose Create queue, select Standard, and enter miruky-pmannkikxatnzsyf. Keep the default delivery settings and the default SQS-managed server-side encryption.

The next image vertically stacks the queue type and encryption portions of the same create-queue form.

The form visibly pairs Standard with miruky-pmannkikxatnzsyf and the default SQS-managed encryption. No dead-letter queue or delivery-setting override is added.

After creation, open the queue's Access policy and choose Edit. Keep the outer policy version at 2012-10-17, then add this statement to the existing owner policy. It does not require you to retrieve or paste an account ID: the queue ARN keeps a wildcard account segment, while aws:SourceAccount must equal the account that owns the queue at evaluation time.

{
  "Sid": "AllowS3ObjectNotifications",
  "Effect": "Allow",
  "Principal": {
    "Service": "s3.amazonaws.com"
  },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:us-east-1:*:miruky-pmannkikxatnzsyf",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:s3:::miruky-oavzmwjgberlzvhx"
    },
    "StringEquals": {
      "aws:SourceAccount": "${aws:ResourceAccount}"
    }
  }
}

The statement is attached to this generated queue and accepts sends only from the exact generated bucket ARN when the bucket's source account equals the queue owner's account. Save the policy, then review the queue overview without exposing its ARN or owner account number. The next image stacks the identity-excluding success and queue-summary portions of that same overview.

The cropped overview shows Standard and Amazon SQS key (SSE-SQS) after the source-restricted policy is saved. ARN and owner fields are excluded from the image.

3. Create the S3 event notification

Use the following bucket event-notification configuration.

The form shows miruky-vrgqsvmehfuyuoxt with All object create events selected. Prefix and suffix filters remain empty.

For the destination, choose SQS queue, select miruky-pmannkikxatnzsyf, and save. A successful save means S3 could validate the destination policy and publish its configuration test message.

The saved row connects All object create events to miruky-pmannkikxatnzsyf. The next queue poll distinguishes S3's configuration test from an object notification.

Open the queue, choose Send and receive messages, and poll. The configuration message uses s3:TestEvent, not the normal Records array used by object-created notifications, so identify and delete that test message before uploading the proof object.

The message body identifies s3:TestEvent; unlike the later object notification, this configuration message has no normal object-record array. Deleting it leaves the later upload as the only object-event proof.

S3 documentation notes that notification configuration changes can take about five minutes to become effective. If the test message has arrived but a subsequent object event has not, wait for propagation before changing the configuration.

4. Upload one object and inspect its event

Return to the S3 bucket, choose Upload, and add the local file event-proof.txt. The file contains one fixed non-secret line and its object key carries no user or application data.

The upload review contains only event-proof.txt in the generated bucket. Its contents are the fixed non-secret validation line prepared for this run.

Complete the upload and confirm that the object appears in the bucket. The Console upload uses a PutObject request for this small file, so the expected event name is ObjectCreated:Put.

The object list shows event-proof.txt with a successful upload status. This completed upload is the action the next queue message must correlate.

Poll the SQS queue again and open the new message. The complete body stays private because S3 notifications can contain an IAM principal ID, source IP address, request IDs, host ID, ETag, and sequencer.

The isolated receive table now shows Messages (1). The separate approximate Messages available counter reads zero because the retrieved message is temporarily in flight; the table count is the relevant proof that one message was returned by this poll. Its full body remains private until sensitive fields are excluded from the evidence crop.

For comparison, I retained six fields that do not identify the account or operator. The following is a deliberately reduced excerpt transcribed from the verified values, not a copy of the complete message body.

{
  "eventVersion": "2.5",
  "eventSource": "aws:s3",
  "awsRegion": "us-east-1",
  "eventName": "ObjectCreated:Put",
  "s3": {
    "bucket": {
      "name": "miruky-oavzmwjgberlzvhx"
    },
    "object": {
      "key": "event-proof.txt"
    }
  }
}

The screenshot below vertically stacks narrow crops from that same SQS message body. The dividers mark omitted sensitive fields, and the line-wrapped bucket name and object key are joined only at their original wrap points.

The retained fields show 2.5, aws:s3, us-east-1, ObjectCreated:Put, miruky-oavzmwjgberlzvhx, and event-proof.txt. Principal, network, request, host, checksum, and sequencer fields are outside the crop.

Wrap-up

The queue received two different S3 messages: the configuration-time s3:TestEvent and the later ObjectCreated:Put record for the uploaded text file. Separating them prevents the setup test from being mistaken for object-delivery proof.

S3 Event Notifications are designed for at-least-once delivery and do not guarantee event order. A consumer should tolerate duplicates, use the object key's URL-encoded form correctly, and apply an idempotency strategy appropriate to its workload.

For a production path, add observability outside this minimal run: send S3 server access logs to a separate logging bucket, enable CloudTrail data events for the source bucket, and monitor relevant S3 and SQS CloudWatch metrics. These controls introduce additional storage, request, or monitoring costs, so include them in the workload estimate.

Thanks for reading this far.

See you in the next one.

Disclosure: This article was written with AI assistance and independently verified against the linked primary sources and observed results.

DEV Community