DEV Community: Saswat Sambit

Object Replication of Azure Blobs.

Saswat Sambit — Tue, 23 Aug 2022 17:46:00 +0000

Azure Blob storage is Microsoft's object storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that doesn't adhere to a particular data model or definition, such as text or binary data.

FAQ

What is a Replication Rule?
A replication rule is a rule to replicate objects from a container in a storage account to another.
The rule is attached to the storage account which has different parameters, but most importantly the Destination storage account.
The rule automatically replicates the objects to the destination container whenever we upload something to the source container.
What do we mean by cross-region?
Cross-region here means that our containers in the storage accounts are deployed in different regions. Here I deployed a container in us-east but I want to the replicate the objects to another container deployed in australia-southeast
Why do we need a Replication between the storage accounts?
Mainly for Disaster Recovery (DR). Whenever the bucket hosted in our primary datacenter goes down, we can immediately and easily reroute to our secondary datacenter which can be in the same zone or some other zone/region.

Creating my Source Storage Account and a Source Container in it.

Create a Storage Account, name it something like 'sourcestorage-20YYMMDD'.
Now go to the 'Containers' in the Left Pane of the Source Storage Account and select the '+ Container'.
Name the container and select 'Create'.

Container is created!

Creating my Destination Storage Account and a Destination Container in it.

Create another Storage Account, I named it 'destination20220823' [Note: Character length restriction while naming] P.s. I created this Storage account in australia-southeast region while my source storage account was in us-east region.

Similar to the Source container creation, create a container in the Destination storage account. Destination container is created!

Creating a Replication rule.

Now we will create a replication rule which will replicate objects from the Source container in the Storage Storage Account to the Destination container in the Destination Storage Account.

Go to the Source Storage Account, and there proceed to Object replication option under the Data management section in the Left Pane. Then select 'Create Replication rules'
Now select your destination storage account and fill up the container pair details. In my case I paired my source container 'sourcecontainer-20220823' to my destination container 'destination20220823'.
As we can see that the replication rule is successfully attached.

Testing the Replication rule.

Uploading a file to the source container 'sourcecontainer-20220823'
We can see that the file is successfully uploaded to the source container.
In about 2~3mins, the file is successfully replicated to the destination container.

Overview of Azure Storage Accounts and their Redundancy.

Saswat Sambit — Tue, 23 Aug 2022 16:30:00 +0000

Disclaimer: Learning about Azure Storage Accounts is very ambiguous and overwhelming & quite confusing when coming straight from AWS storage services. (Personal Experience 😬)

What is an Azure Storage Account?

An Azure storage account contains all of your Azure Storage data objects, including blobs, file shares, queues, tables, and disks. The storage account provides a unique namespace for your Azure Storage data that's accessible from anywhere in the world over HTTP or HTTPS.

Types of Storage Account provided and recommended by Microsoft for most scenarios,

There are some Legacy storage accounts which aren’t recommended by Microsoft, but may be used in certain scenarios:

How is Storage in Azure any different from that of AWS?

In AWS platform, storage is primarily broken down into 3 services:

Simple Storage Service (S3) - Basic object storage that makes data available through an Internet accessible API.
Elastic Block Storage (EBS) - Block level storage intended for access by a single VM.
Elastic File System (EFS) - File storage meant for use as shared storage for up to thousands of EC2 instances.

In Azure Storage, subscription-bound storage accounts allow you to create and manage the following storage services:

Blob storage stores any type of text or binary data, such as a document, media file, or application installer. You can set Blob storage for private access or share contents publicly to the Internet. Blob storage serves the same purpose as both AWS S3 and EBS.
Table storage stores structured datasets. Table storage is a NoSQL key-attribute data store that allows for rapid development and fast access to large quantities of data. Similar to AWS' SimpleDB and DynamoDB services.
Queue storage provides messaging for workflow processing and for communication between components of cloud services.
File storage offers shared storage for legacy applications using the standard server message block (SMB) protocol. File storage is used in a similar manner to EFS in the AWS platform.

Here's a table briefly comparing AWS & Azure storage services,

Now here I've created a Storage Account where I can store blobs,

As we can see in the above image, file service along Queue & Table services are disabled in this account.
Note: I can store blobs (in Containers), files, queue or tables by accessing the respective option under the Data Storage section in the Left Pane.

Now I created another Storage Account where both Blob and File Service are enabled,

Redundancy of Storage Accounts

Azure Storage always stores multiple copies of your data so that it's protected from planned and unplanned events, including transient hardware failures, network or power outages, and massive natural disasters. Redundancy ensures that your storage account meets its availability and durability targets even in the face of failures.

Redundancy in the primary region.

Data in an Azure Storage account is always replicated three times in the primary region. Azure Storage offers two options for how your data is replicated in the primary region:

Locally redundant storage (LRS) copies your data synchronously three times within a single physical location in the primary region. LRS is the least expensive replication option, but isn't recommended for applications requiring high availability or durability.
Zone-redundant storage (ZRS) copies your data synchronously across three Azure availability zones in the primary region. For applications requiring high availability, Microsoft recommends using ZRS in the primary region, and also replicating to a secondary region.

Redundancy in a secondary region.

For applications requiring high durability, you can choose to additionally copy the data in your storage account to a secondary region that is hundreds of miles away from the primary region. If your storage account is copied to a secondary region, then your data is durable even in the case of a complete regional outage or a disaster in which the primary region isn't recoverable.

Azure Storage offers two options for copying your data to a secondary region:

Geo-redundant storage (GRS) copies your data synchronously three times within a single physical location in the primary region using LRS. It then copies your data asynchronously to a single physical location in the secondary region. Within the secondary region, your data is copied synchronously three times using LRS.
Geo-zone-redundant storage (GZRS) copies your data synchronously across three Azure availability zones in the primary region using ZRS. It then copies your data asynchronously to a single physical location in the secondary region. Within the secondary region, your data is copied synchronously three times using LRS.

Note: We can set up the Redundancy while creating a Storage Account -or- we can change it after creating an account acc. to our requirements.

Changing the Redundancy replication of a Storage Account from LRS to GRS

As we can see the in the details of the Storage Account, that it is set for LRS replication.
Proceed for Configuration in Left Pane of the Storage Account and there change it to GRS in Replication option and save it.
Now we can see that our Storage Account is set for GRS replication.

Setting up a Cross-Region Replication rule for S3 buckets through AWS CLI.

Saswat Sambit — Sat, 06 Aug 2022 13:06:00 +0000

S3 is an object-storage service provided by AWS. Object storage takes each piece of data and designates it as an object. Data is kept in separate storehouses versus files in folders and is bundled with associated metadata and a unique identifier to form a storage pool.

This S3 object-based storage is accessible via a Web API which makes it easy to use, against a block-storage type like EBS which has to be attached to an EC2 Instance in order to access it.

You don't have to pay for creating a S3 bucket but you have pay for the Objects! That means every file uploaded to the bucket as well as for how long decides your bill.

FAQ

What is a Replication Rule?
A replication rule is a rule to replicate objects from a bucket to another.
The rule is attached to a particular bucket (aka Source bucket) which has different parameters, but most importantly the Destination bucket.
The rule automatically replicates the objects to the destination bucket whenever we upload something to the source bucket.
What do we mean by cross-region?
Cross-region here means that our buckets are deployed in different regions. For e.g. I deployed a bucket in us-west-1 (N. California) but I want to the replicate the objects to another bucket deployed in ap-northeast-1 (Tokyo)
Why do we need a Replication between the buckets?
Mainly for Disaster Recovery (DR). Whenever the bucket hosted in our primary datacenter goes down, we can immediately and easily reroute to our secondary datacenter which can be in the same zone or some other zone/region.

So let's start setting up our Replication rule on a bucket!

But before using the CLI, we have to configure it. For the configuration we have to create a access key in our AWS acc.
aws configure is the command.

Here we gave the access key id and the secret access key obtained from the 'access key' we created.
I took us-west-1 as my main region and the output format is json which is the default one.

Creating Source and Destination buckets in the CLI

Let's create a source bucket and enable versioning in it.

Create bucket command -> aws s3api create-bucket --bucket <source-bucket-name> --region <region-code> --create-bucket-configuration LocationConstraint=<region-code>

Note: When deploying a S3 bucket in us-east-1 region, we don't have to give the LocationConstraint tag.

Enable Versioning command -> aws s3api put-bucket-versioning --bucket <source-bucket-name> --versioning-configuration Status=Enabled

Now let's create a destination bucket in another region and enable versioning in it.

Create bucket command -> aws s3api create-bucket --bucket <destination-bucket-name> --region us-east-1

Note: Since we deployed this destination bucket in the us-east-1 region, you can see that we don't have to use the LocationConstraint tag.

Enable Versioning command -> aws s3api put-bucket-versioning --bucket <destination-bucket-name> --versioning-configuration Status=Enabled

Ok, we can see that our source and destination buckets are deployed in different regions...

Creating an IAM role and Replication Rule

In order to create a Replication rule, we need an IAM role to attach the rule to the source bucket. For creating an IAM role, we need 2 JSON files. First for the trust-policy and the Second for the role-permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Save this code as s3-trust-policy.json

Then use the following command to create an IAM role and attach the trust-policy to it,
aws iam create-role --role-name S3replicationRole --assume-role-policy-document file://s3-trust-policy.json

Note: Save the json files in the directory where you've launched your CMD else you have to properly give the file location.

Now attach a Role Permission to the IAM role,

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectVersionForReplication",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging"
      ],
      "Resource": ["arn:aws:s3:::<source-bucket-name>/*"] 
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetReplicationConfiguration"],
      "Resource": ["arn:aws:s3:::source-bucket"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ReplicateTags"
      ],
      "Resource": "arn:aws:s3:::<destination-bucket-name>/*" 
    }
  ]
}

Replace and with your respective source and destination bucket names and save it as s3-role-perms.json

Now attach the file to the role with the following command,
aws iam put-role-policy --role-name S3replicationRole --policy-document file://s3-role-perms.json --policy-name S3replicationRolePolicy

Now let's create a Replication Config file which will set the Replication Rule to our Source bucket.

{
  "Role": "<Role-ARN>",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Filter": { "Prefix": "Documents" },
      "Destination": {
        "Bucket": "arn:aws:s3:::<destination-bucket-name>"
      }
    }
  ]
}

Replace with the ARN (Amazon Resource Number) of the IAM Role we created in the previous steps ...

... and with the name of the Destination bucket & save the file as replicationConf.json

Then with the help of the following command, attach this config file to the source bucket to set up the Replication Rule,
aws s3api put-bucket-replication --replication-configuration file://replicationConf.json --bucket <source-bucket-name>

Let us verify whether the Replication Rule is properly attached or not!

Use the command to check in the CLI,
aws s3api get-bucket-replication --bucket <source-bucket-name>

or we can check in the console,