DEV Community

Abdallah Deeb
Abdallah Deeb

Posted on • Originally published at deeb.me on

Sync S3 objects across AWS account?

How do I copy S3 objects/files/etc from another AWS account?
How do I migrate an S3 bucket to another AWS account? [playbook below + extra preliminary step, create the bucket]

Well, you can do it manually following the notes in the AWS KB article I linked at the end of the article.
Honestly though, I am always confused by those KB articles. They are concise ans usually well written. I guess it's my dislexya ;)
So I have to read them a few times before I’m able to apply the steps correctly. That's not a bad thing really ... better than blindly copy/paste commands (which I NEVER DO, right).

Recently, I’ve been going the other way around: whenever there are steps to follow, I try to translate them into an Ansible playbook. This way I make sure I got it right, AND I get to reuse the playbook, which makes things a lot faster the second time around. Oh and as an added bonus, I can share the playbook and have someone else take care of the task!

Enough introductions, here’s what you came here for: an ansible playbook for syncing files between S3 buckets in different accounts:

# copy-bucket.yml
- hosts: localhost
  vars:
    source_bucket: my-source-bucket  
    dest_bucket: my-dest-bucket
    dest_user_arn: arn:aws:iam::ACCOUNTID:user/USERNAME
    dest_user_name: USERNAME
    source_profile: src_profile
    dest_profile: dest_profile
  tasks:
    - name: Attach bucket policy to source bucket
      s3_bucket:
        name: "{{ source_bucket }}"
        policy: "{{ lookup('template','bucket-policy.json.j2') }}"
        profile: "{{ source_profile }}"

    - name: Attach an IAM policy to a user in dest account
      iam_policy:
        iam_type: user
        iam_name: "{{ dest_user_name }}"
        policy_name: "s3_move_access"
        state: present
        policy_json: "{{ lookup( 'template', 'user-policy.json.j2') }}"
        profile: "{{ dest_profile }}"
      register: user_policy

    - name: Sync the files 
      shell: aws s3 sync s3://{{ source_bucket }}/ s3://{{ dest_bucket }}/ --profile {{ dest_profile }}

You will also need the following json templates

bucket-policy.json.j2

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::{{ source_bucket }}",
                "arn:aws:s3:::{{ source_bucket }}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::{{ dest_bucket }}",
                "arn:aws:s3:::{{ dest_bucket }}/*"
            ]
        }
    ]
}

user-policy.json.j2

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "{{ dest_user_arn }}"},
            "Action": ["s3:ListBucket","s3:GetObject"],
            "Resource": [
                "arn:aws:s3:::{{ source_bucket }}/*",
                "arn:aws:s3:::{{ source_bucket }}"
            ]
        }
    ]
}

run: ansible-playbook -v copy-bucket.yml

make sure the profile names are setup on your machine correctly, and the IAM user is there.

Ref: AWS KB: copy-s3-objects-account

Discussion (0)