More Automation for Your AWS Resources, More Coffee Time for You!

Hi everyone! It's been a while since my latest post (2 months ago) but today I have an interesting topic to discuss. Actually, I already talked about this topic at the AWS UG Indonesia event but the session was in Indonesian (here's the video) so I thought about turning it into a blog post in English.

Alright! You may see that mostly I talked about Ansible for AWS on this blog, but at this time I'll talk on the other side about where we can combine them. They can actually be "friends" and shouldn't always be comparisons. These works can help you to do more automation for your AWS resources which means that you will have more time to enjoy your coffee :) I mean everybody loves it, right?

Prerequisites:

1. AWS CLI and set at least one credential;

2. Ansible;

3. Ansible collection for AWS by running ansible-galaxy collection install amazon.aws and ansible-galaxy collection install community.aws.

1. Dynamic Inventory For EC2 Instances

Yes, we usually manage our server inventory manually which we input the IP addresses or hostnames manually like below:

ec2:
  hosts:
    108.136.226.235:
    108.136.226.232:
    108.136.226.238:

But Ansible has a plugin that enables us to manage our EC2 instances and meets our needs. It's very easy to do, simple and you will get what you want in a short time.

To use this plugin, below are the steps:

• Create a new YAML file and input the code below:

plugin: aws_ec2
regions:
- ap-southeast-3
filters:
availability-zone: ap-southeast-3b

It will enable us to get the EC2 details and create a group for our instances based on the filter we use. For example above, I need to manage EC2 instances in the Jakarta region and I added a filter AZ because I just need to manage the instances in the AZ B. Any filter we can use is just the same as we use the EC2 filter by AWS CLI (if we can use the filter using AWS CLI EC2, then we can use it too on this plugin).

• After creating the file, run the command:

$ ansible-inventory -i aws_ec2.yml --list

Here's the sample output:

"aws_ec2": {
  "hosts": [
    "ec2-108-136-56-235.ap-southeast-3.compute.amazonaws.com"
  ]
}

• If we just wanna see the grouping, run the command:

$ ansible-inventory -i aws_ec2.yml --graph

Here's the sample output:

@all
  |--@aws_ec2:
  |  |--ec2-108-136-56-235.ap-southeast-3.compute.amazonaws.com
  |--@ungrouped:

• In case we wanna run some command or playbook to our instances for example to install an application on them, we can mention the inventory just like we use Ansible as usual. Below is a sample by running ansible ad-hoc and we will see the output when it's done.

$ ansible -i aws_ec2.yml -u ubuntu -m shell -a "sudo apt install apache2 -y"

Now we all see, how easy is for us to manage our EC2 instances. More about this plugin, you can find it here.

2. Running AWS CloudFormation using Ansible

In this concept, we automate the AWS IaC tool. As we all know, CloudFormation gives us an easy way for infrastructure provisioning on AWS into a file, just by pushing a template. That sounds great, right? But do you know that Ansible has a module for that?

Yup! Ansible has and we can get some benefits by combining them. As I mentioned before, they shouldn't always be comparisons but it will be better to run them together.

What can we do with them?

• Create multiple stacks at once

In case we wanna create some stacks with the same resources along with their own values, we can do that with Ansible. We don't need to copy-paste the same template just for the same resources. All we need to do is just turn it into a variable. Let's see how it works!

Here's the sample task that we can use. The key is the template_body argument which allows us to use the jinja2 template. In this case, I wanna create a bucket in two stacks and each stack has its own stack name and the bucket name as well. So in the stack1, I wanna create a bucket named nuruls3 and for the stack2, the bucket name is dhonas3.

    - name: create cloudformation stacks
      amazon.aws.cloudformation:
        stack_name: "{{ item.name }}"
        state: present
        template_body: "{{ lookup('template', 'cf-template.yml.j2') }}"
      loop:
        - { name: stack1, bucket: nuruls3 }
        - { name: stack2, bucket: dhonas3 }

What does the template look like? It's just as simple as this. We turn the bucket name which the data type is string into a variable.

Resources:
  S3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: "{{ item.bucket }}"

So when we wanna define different values for different stacks using one (same) template, we just need to change the variables under the loop action.

Here's the output:

TASK [create a cloudformation stack] *******************************************
changed: [localhost] => (item={'name': 'stack1', 'bucket': 'nuruls3'})
changed: [localhost] => (item={'name': 'stack2', 'bucket': 'dhonas3'})

and the result:

$ aws s3 ls
2022-10-30 11:08:19 dhonas3
2022-10-30 11:07:51 nuruls3

Note: We can also update and delete as well as the creation. More about this module, click here!

• Create a stack that contains resources in different accounts (in case you have more than one).

This is a sample from the Ansible documentation:

- name: Create a stack set with instances in two accounts
  community.aws.cloudformation_stack_set:
    name: my-stack
    description: Test stack in two accounts
    state: present
    template_url: https://s3.amazonaws.com/my-bucket/cloudformation.template
    accounts: [1234567890, 2345678901]
    regions:
    - us-east-1

On this step, I'm not able to reproduce it because I just have one account and I think I don't need to create a new one just to do this :) it's enough for me to have one because it's just for personal use but in case you need to use it because you have more than one account, please let me give you some tips. For me, instead of using template_url which you have to upload on the S3 bucket, you can use template_body. Why? Because you will be able to do something more flexible. For example, you can use the jinja2 template where you can freely customize the parameters.

3. Running Ansible from AWS Systems Manager

SSM is a great thing to use to manage our EC2 instances and the good thing is we can run the Ansible playbook from SSM.

All we need to do is just to attach IAM role on the EC2 instances you want to manage that allows them to communicate with SSM or we can say to make the SSM-Agent installed on the EC2 instances work. Yup! For the latest AMI, SSM-Agent has already been installed by default so we don't need to install it again (check the full information here).

One more thing we need to do is install Ansible on our EC2 instances because it will be running on your instances which acts as an Ansible host, that's why it should be installed. (This action can be done by SSM as well as by SSM RunCommand but in this case we use Shell).

Here we will use RunCommand with the document RunAnsiblePlaybook from Systems Manager and AFAIK it's free to do RunCommand so it can be one of your alternatives to run Ansible instead of using your localhost.

Here's a sample of the playbook. You can directly write it into the column provided or use the URL where you store your playbook.

In case you wanna run it using AWS CLI, below is the sample:

$ aws ssm send-command --document-name "AWS-RunAnsiblePlaybook" --document-version "1" --targets '[{"Key":"InstanceIds","Values":[your-instances-ids]}]' --parameters '{"playbook":["hosts: localhost\ntasks:\n - name: install web server\n apt: name=apache2"],"playbookurl":[""],"extravars":["SSM=True"],"check":["False"],"timeoutSeconds":["3600"]}' --timeout-seconds 600 --max-concurrency "50" --max-errors "0" --region ap-southeast-3

Alright! Those are three cool things we can do with Ansible to automate our AWS resources. I hope they can help you extend your time enjoying the coffee :) instead of spending more time working for your system (just let them work for you).

That's it for now! Thank you for coming and I'm looking forward to your feedback. Follow me to get notified when my new post is published!