Using AWS with Ansible but from Google Cloud

What is the problem?

Ansible on AWS can use server roles to very conveniently run playbooks with access to AWS resources
Or if it’s from a “home” account, AWS creds in files work just fine.
But ansible running remotely on non AWS (ie on prem, GCP, Oracle Cloud etc) then getting the credentials loaded can be a problem

How it works

Step one, ensure local creds in file

Some credentials should already exist on your local host in the ~/.aws/credentials file. All you have to do is convert those credentials into a format that ansible can load.

Ansible AWS module is based on boto3 and so will honour environment variables AWS_SECRET_KEY_ID and AWS_ACCESS_KEY_ID. Ansible can load variable files into environment variables: this is path I’ve taken

The file format in ~/.aws/credentials is like this

    [default]
    aws_access_key_id = AKIAXEX5OPTQQLXUAVL5
    aws_secret_access_key = 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

    [admin-account]
    aws_secret_access_key = mzTh1Sj9Btjxucbq//X6Ece1z6U6XE3Cysb+Ntdm
    aws_access_key_id = ASIAXEX3PUBILNYIJHAY
    aws_session_token = sXiACeiyqiPpd5z2wAHvEtmTR7/GGf88yVCJ/epz3StU2cU0y1SJUKo6+gch
    2YSsldTlsZzDEBSDubzGKtYur6rmzYZfq+v+KwrUbGobETwn1gvaNd4vg+kEaCg6eNhICisCMRHjrLfK
    YdaikcP9SbUSqQPHwew/F5rtzyWwU5pEle+8uHfYycwY1sSHWOn4bnBA9IpiiJDonTKdnPjWdwuep2E9
    pf7A3fxaPOCb+jss9CoI9YfzPLQ9CQSztqfdhqN4PxYvpXnLSms3Z6SB7cS

The format needed for ansible is

    ---
    env_vars:
    AWS_SECRET_ACCESS_KEY: mzTh1Sj9Btjxucbq//X6Ece1z6U6XE3Cysb+Ntdm
    AWS_ACCESS_KEY_ID: ASIAXEX3PUBILNYIJHAY
    AWS_SESSION_TOKEN: sXiACeiyqiPpd5z2wAHvEtmTR7/GGf88yVCJ/epz3StU2cU0y1SJUKo6+gc
    h2YSsldTlsZzDEBSDubzGKtYur6rmzYZfq+v+KwrUbGobETwn1gvaNd4vg+kEaCg6eNhICisCMRHjrLf
    KYdaikcP9SbUSqQPHwew/F5rtzyWwU5pEle+8uHfYycwY1sSHWOn4bnBA9IpiiJDonTKdnPjWdwuep2E
    9pf7A3fxaPOCb+jss9CoI9YfzPLQ9CQSztqfdhqN4PxYvpXnLSms3Z6SB7cS

AWScre.py in my github will do this conversion

Step 2, run the AWScre

To use the program call it with 3 parameters
The credentials file path, usually $HOME/.aws/credentials
An output path to be used later by ansible, for example awsvars.yml
The account that is being used, for example ‘admin-account’

The output path should be in the ansible directory so that it is available on the remote host

Step 3, use in Ansible playbooks

Next, add a vars_file directive to load the file you just made. Add this to the playbook that needs to use AWS from GCP, like this

    ---

    - hosts: all
    gather_facts: yes
    become: yes
    become_user: root
    vars_files:
        - ../awsvars.yml
    environment: "{{ env_vars }}"

The python script I’ve written to setup the AWS credentials also makes a tweak to the sudo settings on the remote to ensure that the variables are exported during the “become_user”. So the output file that is used by ansible actually looks like this

    ---
    env_vars:
    ANSIBLE_BECOME_FLAGS: '-H -S -n -E'
    AWS_SECRET_ACCESS_KEY: mzTh1Sj9Btjxucbq//X6Ece1z6U6XE3Cysb+Ntdm
    AWS_ACCESS_KEY_ID: ASIAXEX3PUBILNYIJHAY
    AWS_SESSION_TOKEN: sXiACeiyqiPpd5z2wAHvEtmTR7/GGf88yVCJ/epz3StU2cU0y1SJUKo6+gc
    h2YSsldTlsZzDEBSDubzGKtYur6rmzYZfq+v+KwrUbGobETwn1gvaNd4vg+kEaCg6eNhICisCMRHjrLf
    KYdaikcP9SbUSqQPHwew/F5rtzyWwU5pEle+8uHfYycwY1sSHWOn4bnBA9IpiiJDonTKdnPjWdwuep2E
    9pf7A3fxaPOCb+jss9CoI9YfzPLQ9CQSztqfdhqN4PxYvpXnLSms3Z6SB7cS

How well does it work? Transferring software artifacts held in s3 from AWS to GCP was as fast as you’d expect, a few seconds for hundreds of MB