DEV Community

GargeeBhatnagar for AWS Community Builders

Posted on

1

Cross-Region Bucket Replication of Existing Data Using Amazon S3 Batch Operation

“ I have checked the documents of AWS to do cross-region bucket replication of existing data using amazon s3 batch operation. Pricing of S3 bucket as per storage size and S3 batch job replication as per $0.25 per job.”

An S3 Batch Operations is a data management functionality in Amazon S3 that allows you to handle billions of items at scale with only a few clicks in the Amazon S3 Management Console or a single API request. With S3 Batch Operations, you can make changes to object metadata and properties, as well as perform other storage management tasks such as copying or replicating objects between buckets, replacing object-tag sets, modifying access controls and restoring archived objects from S3 Glacier.

A single job can execute a specific operation on billions of objects carrying exabytes of data. It manages retries, tracks progress, sends notifications, generates completion reports and delivers events to AWS Cloudtrail for all changes made and tasks executed.

In this post, you will get to know how to do cross-region bucket replication of existing data using amazon s3 batch operation. Here I have created a s3 bucket, IAM role, S3 replication rule and s3 batch operation.

Prerequisites

You’ll need an Amazon Simple Storage Service for this post. Getting started with Amazon Simple Storage Service provides instructions on how to create a bucket in simple storage service.

Architecture Overview

Image description
The architecture diagram shows the overall deployment architecture with data flow, amazon s3 and iam role.

Solution overview

The blog post consist of the following phases:

  1. Create of IAM Replication Role and Replication Rule on Source Bucket
  2. Create of IAM Batch Role and Replication Job
  3. Output of S3 Batch Operation

I have s3 buckets as below →

Image description

Image description

Image description

Phase 1: Create of IAM Replication Role and Replication Rule on Source Bucket

  1. Open the IAM console and create a replication role with required permission. Also open the source bucket, create a replication rule under the management section.

Image description

Image description

Image description

Phase 2: Create of IAM Batch Role and Replication Job

  1. Open the IAM console and create a batch role with required permission. Also open the source bucket, create a replication job.

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Phase 3: Output of S3 Batch Operation

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Image description

Clean-up

Delete S3 Bucket and IAM Role.

Pricing

I review the pricing and estimated cost of this example.

Cost of S3 in N. Virginia = $0.232

Cost of S3 in Oregon = $0.232

S3 Batch Replication for Job = $0.25

Total Cost = $0.714

Summary

In this post, I showed “cross-region bucket replication of existing data using amazon s3 batch operation”.

For more details on Amazon S3 Batch Operations, Checkout Get started Amazon S3 Batch Operations, open the Amazon S3 Batch Operations console. To learn more, read the Amazon S3 Batch Operations documentation.

For more details on IAM, Checkout Get started IAM, open the IAM console. To learn more, read the IAM documentation.

Thanks for reading!

Connect with me: Linkedin
Image description

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more →

Top comments (0)

Best Practices for Running  Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK cover image

Best Practices for Running Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK

This post discusses the process of migrating a growing WordPress eShop business to AWS using AWS CDK for an easily scalable, high availability architecture. The detailed structure encompasses several pillars: Compute, Storage, Database, Cache, CDN, DNS, Security, and Backup.

Read full post