Setting up a Cross-Region Replication rule for S3 buckets through AWS CLI.

#aws #s3

S3 is an object-storage service provided by AWS. Object storage takes each piece of data and designates it as an object. Data is kept in separate storehouses versus files in folders and is bundled with associated metadata and a unique identifier to form a storage pool.

This S3 object-based storage is accessible via a Web API which makes it easy to use, against a block-storage type like EBS which has to be attached to an EC2 Instance in order to access it.

You don't have to pay for creating a S3 bucket but you have pay for the Objects! That means every file uploaded to the bucket as well as for how long decides your bill.

FAQ

What is a Replication Rule?
A replication rule is a rule to replicate objects from a bucket to another.
The rule is attached to a particular bucket (aka Source bucket) which has different parameters, but most importantly the Destination bucket.
The rule automatically replicates the objects to the destination bucket whenever we upload something to the source bucket.
What do we mean by cross-region?
Cross-region here means that our buckets are deployed in different regions. For e.g. I deployed a bucket in us-west-1 (N. California) but I want to the replicate the objects to another bucket deployed in ap-northeast-1 (Tokyo)
Why do we need a Replication between the buckets?
Mainly for Disaster Recovery (DR). Whenever the bucket hosted in our primary datacenter goes down, we can immediately and easily reroute to our secondary datacenter which can be in the same zone or some other zone/region.

So let's start setting up our Replication rule on a bucket!

But before using the CLI, we have to configure it. For the configuration we have to create a access key in our AWS acc.
aws configure is the command.

Here we gave the access key id and the secret access key obtained from the 'access key' we created.
I took us-west-1 as my main region and the output format is json which is the default one.

Creating Source and Destination buckets in the CLI

Let's create a source bucket and enable versioning in it.

Create bucket command -> aws s3api create-bucket --bucket <source-bucket-name> --region <region-code> --create-bucket-configuration LocationConstraint=<region-code>

Note: When deploying a S3 bucket in us-east-1 region, we don't have to give the LocationConstraint tag.

Enable Versioning command -> aws s3api put-bucket-versioning --bucket <source-bucket-name> --versioning-configuration Status=Enabled

Now let's create a destination bucket in another region and enable versioning in it.

Create bucket command -> aws s3api create-bucket --bucket <destination-bucket-name> --region us-east-1

Note: Since we deployed this destination bucket in the us-east-1 region, you can see that we don't have to use the LocationConstraint tag.

Enable Versioning command -> aws s3api put-bucket-versioning --bucket <destination-bucket-name> --versioning-configuration Status=Enabled

Ok, we can see that our source and destination buckets are deployed in different regions...

Creating an IAM role and Replication Rule

In order to create a Replication rule, we need an IAM role to attach the rule to the source bucket. For creating an IAM role, we need 2 JSON files. First for the trust-policy and the Second for the role-permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Save this code as s3-trust-policy.json

Then use the following command to create an IAM role and attach the trust-policy to it,
aws iam create-role --role-name S3replicationRole --assume-role-policy-document file://s3-trust-policy.json

Note: Save the json files in the directory where you've launched your CMD else you have to properly give the file location.

Now attach a Role Permission to the IAM role,

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectVersionForReplication",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging"
      ],
      "Resource": ["arn:aws:s3:::<source-bucket-name>/*"] 
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetReplicationConfiguration"],
      "Resource": ["arn:aws:s3:::source-bucket"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ReplicateTags"
      ],
      "Resource": "arn:aws:s3:::<destination-bucket-name>/*" 
    }
  ]
}

Replace and with your respective source and destination bucket names and save it as s3-role-perms.json

Now attach the file to the role with the following command,
aws iam put-role-policy --role-name S3replicationRole --policy-document file://s3-role-perms.json --policy-name S3replicationRolePolicy

Now let's create a Replication Config file which will set the Replication Rule to our Source bucket.

{
  "Role": "<Role-ARN>",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Filter": { "Prefix": "Documents" },
      "Destination": {
        "Bucket": "arn:aws:s3:::<destination-bucket-name>"
      }
    }
  ]
}

Replace with the ARN (Amazon Resource Number) of the IAM Role we created in the previous steps ...

... and with the name of the Destination bucket & save the file as replicationConf.json

Then with the help of the following command, attach this config file to the source bucket to set up the Replication Rule,
aws s3api put-bucket-replication --replication-configuration file://replicationConf.json --bucket <source-bucket-name>