DEV Community: Mariem

Cross-account S3 replication: encrypted and non-encrypted objects

Mariem — Mon, 11 Sep 2023 08:49:12 +0000

If you need to share S3 data between accounts, to meet compliant storage needs, or to bring S3 data closer to your end user, S3 Replication is a feature of the service S3 that could be your solution. S3 replication is for automatic, asynchronous copying of objects across buckets in the same or different AWS Regions, in the same or different AWS accounts. It retains all metadata, such as the original object creation time, object access control lists (ACLs), and version IDs.

For replication, both source and destination buckets must have versioning enabled, a replication rule established, and IAM permissions granted.

When setting up the replication of objects from a source bucket to a destination bucket using Amazon S3, there are several options to consider. You can :

Choose to replicate all the objects of the source buckets or only objects starting with a prefix or with a special tag
Set up a priority in the case of many replication rules for the same bucket
You must specify an IAM role that Amazon S3 assumes to replicate objects from the source bucket to the destination bucket.
Choose to replicate objects that are encrypted by server-side encryption.
Specify the storage class for the object replicas that can be different from the source bucket
Enable or disable the replication. The replication is applied for the uploaded objects after enabling the replication rule.

In the following, I will demonstrate some of the capabilities of S3 replication through illustrations and a hands-on that will be refined as we progress. We will begin by replicating non-encrypted objects between accounts, then move on to replicating delete markers between accounts, and finally, we will cover the replication of KMS-encrypted objects.

Thanks to Guille Ojeda's amazing newsletter for inspiring this article. I highly recommend following him on LinkedIn and subscribing to his newsletter.

A. S3 Replication of Non-encrypted Objects across accounts

To set up cross-account S3 replication, you'll need two separate accounts and buckets - one for the source and one for the destination. You need to enable versioning in the two buckets. Next, create a replication rule and specify your options. You'll also need to set permissions on both sides of the relationship:

In the source account, create a role that the S3 service can assume to replicate objects. This role must be authorized to read from the source bucket and replicate to the destination bucket.
In the destination account, make sure the destination bucket's policy allows the source account's S3 role to replicate objects.

Permissions can be tricky, so be careful😊!

Let’s try it!

1. Create a bucket in each of the two accounts, and make sure to choose meaningful names to avoid confusion 😁

2. Enable versioning for both buckets.

3. In the management part of the source bucket, create the replication rule.

We'll mainly use the default options.

Choose a Name for the replication rule

Choose to replicate all objects in the bucket

Choose to replicate on another account.
Provide the account ID and bucket name of the destination where you would like to transfer the data.

Choose the IAM role for S3 to be created automatically.

Keep the other options as default

After you confirm the creation of the replication rule, you will receive a message regarding your decision about the existing objects in the source bucket before enabling the replication.They are not replicated automatically.
This is particularly helpful when you already have objects in the source bucket!

That's not all! While we have the necessary permissions and configuration in the source account, the destination bucket needs to authorize the created S3 role to replicate objects. Therefore, we must copy the name of the role and update the bucket policy of the destination bucket.

Copy the S3 role name from the details of the replication rule

In the destination account, update the bucket policy of the destination bucket

We have everything required for replication. You can upload a file into the source bucket and await its appearance in the destination bucket. Objects replicate within 15 minutes.

B. What happens in the destination bucket when we delete an object from the source bucket?

Try it before reading the next part! Don’t forget to toggle on the show version!

When we enable versioning in a bucket, and we delete an object, the object isn’t deleted permanently. Instead, a delete marker (placeholder) is added as the current version of your object. The Delete Marker makes AWS S3 behave as if the object has been deleted.

Amazon S3 does not replicate the delete marker by default. However, you can add delete marker replication to non-tag-based rules. The delete marker replication is applied for the uploaded objects after its enablement. Depending on your case and your goal behind the replication, you choose to replicate delete markers or not.

In our case, the delete marker is added in the source and not replicated in the destination bucket. Let’s make the changes in our configuration to make it possible.

Update the replication rule of the source account to enable delete marker replication. That’s all you need to do

Now, when you delete an object from the source account, the delete marker is added to the source and replicated to the destination bucket

Remark!
If you delete an object version, Amazon S3 deletes that object version in the source bucket permanently. But it doesn't replicate the deletion in the destination buckets, even when you enable delete marker replication. This protects data from malicious deletions.

C. How can I update our configuration to enable the replication of KMS-encrypted objects across accounts (same region)?

AWS Key Management Service (AWS KMS) lets you create, manage, and control cryptographic keys across your applications and AWS services.

When we have to replicate encrypted objects, we need two KMS keys in the source and destination account and there are more permissions to set up than in the case of no encrypted objects.

The S3 role must now also be allowed to use and decrypt objects in the source bucket using the KMS SourceKey and to use and encrypt them in the destination bucket using the KMS DestinationKey

Let’s make the changes to be able to replicate KMS-encrypted objects with different KMS keys in the two accounts!

In this hands-on, I created two KMS keys in the source and destination account, but if you already have keys, you can use them and jump to step 3. You should only pay attention to keys permissions!

Create a KMS key in the source account

Give an alias (name) for your key and keep all other options as default.

You may need to update the key policy. For me, I kept the default policy key as I used an Adminstrator Access IAM user for this hands-on. This authorized me to allow S3 to encrypt objects using the created KMS key. If you have some permission troubles in the following steps, check the key policy and your permissions.

Create a KMS key in the destination account.

Give an alias (name) for your key, you must give the authorization to the source account (using console or JSON policy) to use it and keep all other options as default.

Update the source bucket properties to enable default encryption by the source KMS key

Or, you can specify the encryption using the KMS key when uploading an object to the source bucket, without updating the bucket's default encryption.

Update the replication rule
Allow replicating the encrypted objects and encrypt them in the destination bucket using the destination key

Update the S3 role
The role must have permission to decrypt with the source account KMS key and to encrypt with the destination account key. So, you need to add the following permissions to the S3 role

Congratulations👏! You have successfully configured everything you need to replicate encrypted objects within the same region across two accounts. You can verify the results!
(The image below is composed from four screenshots to show the final result)

Conclusion
S3 replication is a powerful feature that you can use when you need to replicate data between S3 buckets.

The hands-on I provided is done in different accounts within the same region. Try to find what changes to make in the configuration to make it possible in different regions for encrypted and non-encrypted objects.

Happy learning!

AWS Global Accelerator (Standard)

Mariem — Thu, 07 Sep 2023 13:10:10 +0000

AWS Global Accelerator is a network layer service that can be deployed in front of your applications to improve its availability and performance and improve the experience of globally distributed users. With AWS global accelerator, the traffic is automatically redirected to the edge location closest to the user and then directed through the AWS global network to reach your endpoints. The endpoints can be an Application Load Balancer (ALB), Network Load Balancer (NLB), EC2 instance, or elastic IP address, and can be located in one or more AWS regions.

AWS Global Accelerator provides two anycast IP addresses from the AWS edge network. Additionally, it assigns a DNS name for your accelerator. These anycast IP addresses act as a fixed entry point to your app making you independent of your endpoints’ IP addresses. So, you can easily move or change endpoints without having to update your DNS configuration or client-facing applications. This makes the global accelerator a good solution for use cases that require IP whitelisting and high availability.

AWS Global Accelerator routes traffic to the healthy best-fit endpoint and in case of failure, an instant failover is provided to the next best endpoint

With AWS Global Accelerator, you can control the percentage of traffic directed to different regions using the Traffic Dial option. Additionally, you can control the traffic between endpoints within the same region using the endpoint weight option.

By default, AWS Global accelerator is protected by AWS Shield Standard for DDoS Attacks and if you need more advanced protection you can enable AWS Shield Advanced (paid)

AWS Global Accelerator is always compared to CloudFront. Both services are global and designed to reduce latency and improve performance for global applications. However, AWS Global Accelerator does not serve users from edge location caches as CloudFront does. Instead, users are served directly from endpoints, with the edge location network used solely to reduce latency. Additionally, CloudFront is limited to HTTP use cases, while AWS Global Accelerator can also be used for non-HTTP use cases such as gaming (UDP), IoT (MQTT), and Voice over IP. This makes it a good solution for a variety of applications.

In the following, I will present the main components of the AWS Global Accelerator and a hands-on based on the AWS Global Accelerator workshop.

AWS Global Accelerator components

The resource that we create with the service AWS Global Accelerator is called an « Accelerator ». Each accelerator has one or more listeners that « listen » to traffic according to what you fixed as port and protocol. If there is a match between a traffic and a listener, the traffic is routed to one of the endpoints registered with this listener. The endpoints of a listener are grouped by region in an endpoint group. A listener can have one endpoint group if all endpoints are in the same region or many otherwise. Each endpoint group can have one or more endpoints. Once the accelerator is created, you receive its DNS name and two elastic IP addresses.

Let’s create an accelerator together and discover some of its features!

Hands-On — Based on the AWS Global Accelerator Workshop

I found the AWS Global Accelerator Workshop a great way to get started with AWS Global Accelerator. In this section, I share the main steps of the workshop and include some helpful images. If you are a visual person like me, I think this will be especially useful 😊.

The workshop starts with creating a basic app that does not have an accelerator. After that, the environment will be enhanced by adding an accelerator. Next, additional endpoints will be added, in the same region and in a different region. Finally, only the accelerator will be granted access to the endpoints.

Step 1. Create an app without Accelerator

The workshop provides a CloudFormation template for deploying an environment for a simple app without an AWS global accelerator. The template creates a VPC with two public subnets in two Availability Zones (AZ), a public ALB in the two newly created public subnets, and a lambda function registered in the target group of the ALB. Additionally, the template takes care of all the necessary permissions for the lambda and the ALB. After deployment, users can access the application using the DNS name of the ALB.

Step 2. Add an accelerator

In this step, you will create an AWS global accelerator to connect the user to the ALB. The accelerator has one listener that captures traffic on port 80 and directs it to an endpoint group containing the app created in Step 1 as an endpoint.

The result of this step is an accelerator with two IP elastic IP addresses and a DNS name that you can use to invoke the lambda function via the ALB

Step 3. Add another endpoint in the same region

In this step, the task is to duplicate the app in the same region, so the environment now has two endpoints to serve traffic through the accelerator. In this case, the accelerator will still listen on port 80 but will now distribute traffic between the two endpoints. Since both endpoints are in the same region, the second ALB is registered with the same endpoint group that was created in step 2. To adjust the traffic distribution between endpoints, you can use the “weight” option to specify the desired percentage.

Step 4. Add another endpoint in another region

In this stage, the application is replicated in a separate region to ensure continued service for users and to be ready for potential disaster or deployment changes. As a result, we now have an additional endpoint in a different region. To connect this to the same listener established in Step 2, we need to create another endpoint group for the accelerator. With the Traffic Dial option, you can control the percentage of traffic sent to each region.

As an exercise, delete an endpoint group or an endpoint from an endpoint group and tell me how the accelerator acts as a consequence!

Step 5. Make the Global Accelerator the single internet-facing access for the app

Up until now, the security groups for the ALBs of the three apps have been permissive, allowing the user to access them via the accelerator or the ALBs. In order to increase security, we should limit access to the ALBs to only the accelerator.

How it can be possible?

When using a global accelerator with an ALB endpoint, an ENI is created in each subnet where the ALB is deployed. The created ENIs have the same security group that only allows traffic to and from the accelerator. To ensure security for the ALB, simply update its security group so that the only inbound traffic allowed is from the accelerator security group.

You can see the different ENIs created in your environment

To make it more secure, Update The ALB security groups 😊

Conclusion

AWS Global Accelerator is a good choice for global apps to reduce latency and improve user experience.

All configurations done at the level of the accelerator are transparent for users or applications that continue using the same DNS name or IP address of your app.

Don’t forget to delete resources (The three stacks created by the CloudFormation template and the accelerator) when you finish the lab!

Happy learning!