DEV Community

Arseny Zinchenko
Arseny Zinchenko

Posted on • Originally published at rtfm.co.ua on

AWS: S3 Cross-Region Replication configuration and objects DeleteMarkers replication

#aws #replication

At this moment I’m configuring a new CDN for our project.

Will use CloudFront and Cloudflare here so need to create two dedicated buckets with different names – cdn.cfr.example.com => CloudFront and cdn.cfl.example.com => Cloudflare.

To avoid coping data each time to both buckets – an AWS S3 Cross-Region Replication can be used, so data from a bucket-1 will be copied to a bucket-2.

Documentation – Cross-Region Replication.

CRR prerequisites:

  1. both buckets must have S3 Versioning enabled (see. Using Versioning)
  2. both buckets must be created in different regions
  3. both buckets must have permissions for replication between them

CRR limitations:

  1. already existing files in a source-bucket will not be copied after CRR will be configured (i.e. only new files will be replicated)
  2. can not create a replication-chain, i.e. Bucket-A => Bucket-B => Bucket-C

AWS S3 Cross-Region Replication set up

Create two buckets:

For both enable Versioning:

In a source-bucket bttrm-crr-source go to the Management > Replication, click on the Add rule:

Set replicate all from this bucket:

Click Next, set the receiver-bucket name:

Next – permissions and IAM role.

Chose the Create new IAM role, set its name:

Save:

Replication testing

Upload a test file to the source bttrm-crr-source bucket:

$ touch testfile.txt
$ aws --profile bm-backend --region us-east-2 s3 cp testfile.txt s3://bttrm-crr-source
$ upload: ./testfile.txt to s3://bttrm-crr-source/testfile.txt
Enter fullscreen mode Exit fullscreen mode

Check if it is present:

$ aws --profile bm-backend --region us-east-2 s3 ls s3://bttrm-crr-source
2019-07-15 15:36:38          0 testfile.txt
Enter fullscreen mode Exit fullscreen mode

And check the destination bucket:

$ aws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt
Enter fullscreen mode Exit fullscreen mode

The file is present, all good.

Deleting files from S3 Cross-Region Replication and DeleteMarkers

if you’ll just delete a file from the current source-bucket – it will be deleted there, but not in the destination bucket.

Remove it in the source:

$ aws --profile bm-backend --region us-east-2 s3 rm s3://bttrm-crr-source/testfile.txt
delete: s3://bttrm-crr-source/testfile.txt
Enter fullscreen mode Exit fullscreen mode

Check it in the destination:

$ aws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt
Enter fullscreen mode Exit fullscreen mode

The file is still here.

To delete objects from under S3 Versioning – AWS adds a special marker called DeleteMarker (see. Working with Delete Markers):

$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-source
{
  "Versions": [
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      "Size": 0,
      "StorageClass": "STANDARD",
      "Key": "testfile.txt",
      "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
      "IsLatest": false,
      "LastModified": "2019-07-15T12:36:38.000Z",
      "Owner": {
      "DisplayName": "example.aws",
      "ID": "9365528c1a92cd01abba171eb7b95c352a330a40ceb28b420c9462fac5891bd2"
      }
    }
  ],
  "DeleteMarkers": [
    {
      "Owner": {
      "DisplayName": "example.aws",
      "ID": "9365528c1a92cd01abba171eb7b95c352a330a40ceb28b420c9462fac5891bd2"
},
      "Key": "testfile.txt",
      "VersionId": "PIr6ZOPcdi9oFP8y0rMXkRhrFuKpDQ4P",
     "IsLatest": true,
     "LastModified": "2019-07-15T12:39:24.000Z"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

But this marker will not be copied to the destination bucket, see the What Does Amazon S3 Replicate:

  • If you make a DELETE request without specifying an object version ID, Amazon S3 adds a delete marker. Amazon S3 deals with the delete marker as follows:

    • If you are using the latest version of the replication configuration, that is, you specify the Filter element in a replication configuration rule, Amazon S3 does not replicate the delete marker.
    • If you don’t specify the Filter element, Amazon S3 assumes that the replication configuration is a prior version V1. In the earlier version, Amazon S3 handled replication of delete markers differently. For more information, see Backward Compatibility .

  • If you specify an object version ID to delete in a DELETE request, Amazon S3 deletes that object version in the source bucket, but it doesn’t replicate the deletion in the destination bucket. In other words, it doesn’t delete the same object version from the destination bucket. This protects data from malicious deletions.

Check the file in the destination bucket:

$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-dest
{
  "Versions": [
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "testfile.txt",
      "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
      ...
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

No DeleteMarkers present.

Check the replication settings:

$ aws --profile bm-backend --region us-west-1 s3api get-bucket-replication --bucket bttrm-crr-source  --query 'ReplicationConfiguration.Rules[*].DeleteMarkerReplication' --output text

Disabled
Enter fullscreen mode Exit fullscreen mode

DeleteMarkerReplication – "Status": "Disabled", okay.

IAM s3:ReplicateDelete

At first – check the IAM role used (see the Setting Up Permissions for Cross-Region Replication).

Find its name:

And find this role in the IAM:

Check it – the "s3:ReplicateDelete" in the Actions must be present:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::bttrm-crr-source",
                "arn:aws:s3:::bttrm-crr-source/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::bttrm-crr-dest/*"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

S3 Bucket Policy

Now – export existing S3 policy:

$ aws --profile bm-backend --region us-west-1 s3api get-bucket-replication --bucket bttrm-crr-source > bttrm-crr-source-origin.json
Enter fullscreen mode Exit fullscreen mode

Check its content:

{
    "ReplicationConfiguration": {
        "Role": "arn:aws:iam::534***385:role/service-role/s3crr_role_for_bttrm-crr-source_to_bttrm-crr-dest",
        "Rules": [
            {
                "ID": "bttrm-s3-crr-replica",
                "Priority": 1,
                "Filter": {},
                "Status": "Enabled",
                "Destination": {
                    "Bucket": "arn:aws:s3:::bttrm-crr-dest"
                },
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }
            }
        ]
    }
}
Enter fullscreen mode Exit fullscreen mode

Now remove ReplicationConfiguration, "Filter": {}, "Priority": 1 DeleteMarkerReplication parameters and add "Prefix": "", so this policy will be like Version 1:

{
        "Role": "arn:aws:iam::534***385:role/service-role/s3crr_role_for_bttrm-crr-source_to_bttrm-crr-dest",
        "Rules": [
            {
                "ID": "bttrm-s3-crr-replica",
                "Prefix": "",
                "Status": "Enabled",
                "Destination": {
                    "Bucket": "arn:aws:s3:::bttrm-crr-dest"
                }
            }
        ]
}
Enter fullscreen mode Exit fullscreen mode

The problem is that CRR config version 2 still can't do DeleteMarkers replication at this moment, the AWS Support reply was:

I did have a look at your blog and the suggestions provided by you seem correct. In fact, this would have been my suggestion as well. The internal team is working on bringing feature of Delete replication in V2 as well but it may take some time for that to happen. It is possible that the way V2 has been implemented in back-end is little different from V1 and using this feature directly may have been causing issues. Unfortunately, I don't have any ETA on when this would finish. Till then, I'm sure your blog would be able to help AWS Users :)

Apply this config:

$ aws --profile bm-backend --region us-west-1 s3api put-bucket-replication --bucket bttrm-crr-source --replication-configuration file://bttrm-crr-source.json
Enter fullscreen mode Exit fullscreen mode

Upload some new file to the source bucket:

$ aws --profile bm-backend --region us-east-2 s3 cp test.file s3://bttrm-crr-source
Enter fullscreen mode Exit fullscreen mode

Delete it:

$ aws --profile bm-backend --region us-east-2 s3 rm s3://bttrm-crr-source/test.file
Enter fullscreen mode Exit fullscreen mode

Check the source bucket now:

$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-source
{
  "Versions": [
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "test.file",
      "VersionId": "0mkfsrIRdkGCgL.zKgMyMXjo\_UydigIt",
      ...
    },
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "testfile.txt",
      "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
      ...
    }
  ],
  "DeleteMarkers": [
    {
      ...
      "Key": "test.file",
      "VersionId": "1kpKJlgOM7xpFnH4qsEp0EzqhCE1HFMX",
      ...
    },
    {
      ...
      "Key": "testfile.txt",
      "VersionId": "PIr6ZOPcdi9oFP8y0rMXkRhrFuKpDQ4P",
      ...
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Great – both files found, both have DeleteMarkers.

And check the destination bucket:

$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-dest
{
  "Versions": [
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "test.file",
      "VersionId": "0mkfsrIRdkGCgL.zKgMyMXjo\_UydigIt",
      ...
    },
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "testfile.txt",
      "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
     ...
    }
    ],
    "DeleteMarkers": [
      {
        ...
        "Key": "test.file",
        "VersionId": "1kpKJlgOM7xpFnH4qsEp0EzqhCE1HFMX",
        ...
      }
   ]
}
Enter fullscreen mode Exit fullscreen mode

There is still no DeleteMarker for the testfile.txt – it’s OK, as it was deleted before we made changes in the DeleteMarkers behavior.

But for the test.file – DeleteMarker already present, it was replicated from the source, so this file is “deleted” now as well:

$ aws --paws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt
Enter fullscreen mode Exit fullscreen mode

No test.file found.

Done.

Similar posts

Top comments (2)

Collapse
 
david_j_eddy profile image
David J Eddy •

Nice post Arsney. I like that you went in to detail with regards to how the delete marker works with cross-region replication. Bookmarking this one for future referance.

Collapse
 
setevoy profile image
Arseny Zinchenko •

Hi, @david .
Thanks.
In fact - I didn't need to delete objects, but tried it just out of curiosity and found, that AWS's documentation is absolutely unclear and messed about that (surprise, as usually, AWS docs are perfect).
So I did a bit investigation for myself to find a way to delete objects.

Read next

eyalestrin profile image

Unpopular opinion about "Moving back to on-prem"

Eyal Estrin -

jiripik profile image

Why You Should Never Join the AWS Community Builders Program

Jiri Pik -

giftedlane profile image

Second Time at AWS re:Invent: Here's What I'm Doing Differently

GIFTED LANE® -

enna profile image

Top 5 Best Practices for AWS Cost Allocation Tags in 2024

Ena Vaghela -