DEV Community: kapil kumawat

Manage Amazon S3 Buckets Using Event Notifications With AWS SQS

kapil kumawat — Mon, 09 May 2022 05:35:53 +0000

Recently I came across a problem where the admin team should get notified when some user deletes any object from S3 object. If you also have a similar requirement where the team should get notification whenever an object is created, deleted, or modified, this is where event notifications comes into the picture.

You can use AWS S3 Event notifications feature to receive notifications to following destinations when certain events happen in the AWS S3 bucket.

Amazon Simple Notification Service (Amazon SNS) topics
Amazon Simple Queue Service (Amazon SQS) queues
AWS Lambda function

In this article, I will be configuring AWS SQL queues as a destination to track delete event from my S3 bucket.

Configure AWS SQL queues to track delete event from S3 bucket
Step 1

Step 2

Go to S3 bucket and create a new bucket if it does not exist. I have already created a bucket with name ‘mydemobucket198’.

Step 3

Go to Properties.

Step 4

Scroll down to the page until section Event Notification appears. Click on Create event notification button.

Step 5

A window appears to fill details to create a new event notification. I have put ‘s3_notify’ as the event name.

Step 6

A list will appear for events that are currently supported like Object creation, Object removal, etc. I checked Object Removal for this article so whenever any user deletes any S3 object I will get a notification.

Step 7

Go to Destination section where you can choose a destination to publish the event.

I have chosen SQL queues so notification will be sent to SQS queues that can be read later by a server and perform action accordingly.

Choose SQL queues if it's already created or create a new one. As I don't have any existing SQL queues, I will create a new one.

Step 8

Go to Amazon SQS service and click on Create queue.

Step 9

Enter name of SQS queue as ‘s3_delete_notify’ and choose all settings as default.

Step 10

Once the SQS queue is created, go to Access Policy.

Step 11

We need to modify the Access Policy to send a message from S3 bucket when any object gets deleted.

Step 12

Click on Edit and paste below Access Policy code -

{
"Version": "2012-10-17",
"Id": "Policy1651140347168",
"Statement": [
{
"Sid": "Stmt1651140341677",
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:ap-south-1:464473132183:s3_delete_notify"
}
]
}

Don't forget to replace the ARN value with your SQL queue.

Step 13

Refresh the page and select the SQS queue that we have just created in the above steps. Click on Save Changes.

Step 14

Go to S3 bucket and upload some files.

Step 15

Select any file and click on Delete button. I have selected photo-1.jpeg and deleted that file.

Step 16

Go to SQS queue ‘s3_delete_notify’ and click on Send and receive message button.

Step 17

Click on Poll for messages button to pull messages.

Step 18

As you can see, selected ID is related to S3 object that has been deleted.

There is another ID which was created by AWS to test initially when queue was created.

Step 19

After you click on Message ID, you can see the details in Message Body. In our case, we can see S3 bucket name with S3 object name photo-1.jpeg which was deleted.

Conclusion

Using S3 Event notification you can enable notification for events like creating, deleting, and modifying S3 objects using destinations Amazon SNS, Amazon SQS and AWS Lambda.

Migrating Limited Objects With Pre-Defined Prefix Using S3 Batch Replication

kapil kumawat — Fri, 29 Apr 2022 08:53:12 +0000

In my previous article, we learned about replicating existing objects between different AWS S3 buckets using S3 Batch Replication.

Replicate Existing Objects Using AWS S3 Batch Replication

In this article, we will learn how to migrate limited objects with a pre-defined prefix using S3 Batch Replication which can be done at the time of configuring the replication rule.

Taking a use case where I don't want to replicate all objects from existing S3 source bucket to new target bucket but need certain files with specific names only, that is also now possible with the new update of AWS S3 replication of existing objects.

How to get started
Step 1

From the AWS S3 source bucket, you like to migrate objects starting with name ‘house’ as shown below -

Step 2

Goto Management page and choose Create Replication Rule option.

Step 3

Enter Replication Rule name.

Step 4

Choose option ‘Limit the scope of this rule using one or more filters’.

Step 5

In the Prefix option, write the prefix value ‘house’ to limit the scope.

Step 6

Choose your destination bucket from S3 bucket list. You can replicate objects across different AWS Regions as Cross-Region Replication or can replicate in the same AWS Region as Same-Region Replication.

Step 7

Choose an IAM role. With default setting AWS S3 creates a new IAM role with sufficient permission to complete this task. I chose my existing IAM role which I used earlier for replication.

Step 8

Once you click on Save, a Batch Operation window will appear showing all details that you configured in previous steps.

Step 9

In Status, you will be asked for your confirmation to run this job.

Step 10

Click on Run Job button.

Step 11

Now the job will be in running state and you will see status as Completed once replication job is completed.

Step 12

Navigate to destination AWS S3 bucket and you will get files with specified prefix only.

In my case, I use the prefix ‘house’ and can only see those objects after replication.

Conclusion

Using S3 Batch replication we can also replicate a specific set of objects using the prefix option in configuring the replication rule.

Replicate Existing Objects Using AWS S3 Batch Replication

kapil kumawat — Thu, 17 Feb 2022 06:23:40 +0000

With new updates, you can replicate existing AWS S3 objects and synchronize AWS S3 buckets using new replication features.

Until today, AWS S3 replication did not support replicating existing objects but now you can do it using AWS S3 batch replication. This is different from live replication which continuously and automatically replicates new objects across different S3 buckets located in different AWS accounts or AWS regions.

Amazon Web Service S3 Replication is a low cost, fully managed feature that automatically replicates S3 objects between buckets in same AWS region using S3 Same-Region-Replication (SRR) or across different AWS Region by using S3 Cross-Region-Replication (CRR). S3 Batch Replication refills newly created buckets with existing objects, can migrate data across different accounts, retry objects that got failed or unable to replicate in existing replication run.

You can configure S3 Batch replication using AWS SDK’s, AWS S3Console or AWS Command Line Interface (CLI).

Here is the replication process diagram from AWS site-

AWS S3 Batch Replication can help to do -

Replicate Existing Objects - S3 Batch Replication can be used to replicate objects that were added to buckets before configuring any replication rules.
Replicate Objects that failed to replicate previously - You can retry to replicate objects that were previously failed to replicate due to any reasons.
Replicate Objects that were already replicated - You might need to store multiple copies of your data to separate AWS accounts or different AWS Regions. S3 Batch can replicate existing objects to newly added destinations.

How to get started
I have already created two AWS S3 buckets (replication-bucket1 and replication-bucket2) in the region us-east1 for this demo.

Step 1

Before configuring the replication rule, if Bucket Versioning is not enabled then you need to enable Bucket Versioning on source and target bucket else you will receive error message like this on replication configuration page,

Step 2

Go to the Properties page and enable Bucket Versioning.

Step 3

Go to S3 bucket list and select a source bucket (replication-bucket1) that contains objects for replication.

Step 4

Go to Management page and choose Create Replication Rule option.

Step 5

Enter Replication rule name

Step 6

After filling required details and creating rule, you will get a prompt asking if you want to replicate existing objects.

Step 7

If you click option Yes, then you will get redirected to a Create Batch operations job. Choose the default option to Automatically run the job when it's ready.

Step 8

You are recommended to choose the option Generate completion report which will contain results of replication job. If you don't have a required IAM role for this then keep the default setting and AWS S3 will create a new IAM role with sufficient permission to run this Batch operation.

Step 9

Once you Save this job, you can check status on the Batch Operations page. Job Status will keep changing from configuring -> in progress -> completion during this process.

Step 10

You can go to your destination bucket and confirm new objects has been replicated.

Conclusion
Using new replication feature it is easy to replicate existing S3 objects between different S3 buckets in same AWS region or different AWS Regions or account.