How to Automate Instance Management with AWS SDK for Python (Boto3)

INTRODUCTION:

As many of us know, DevOps focuses on automating tasks to save valuable time in the workplace. In this project, we’ll explore how Python can help streamline the management of AWS resources.

AWS Cloud9 is a cloud-based Integrated Development Environment (IDE) that enables you to write, run, and debug code directly in the cloud. It’s especially useful for collaborative projects, as multiple developers can work on the same codebase simultaneously.

Boto3 is a Python library that provides a simple and intuitive interface for interacting with AWS services such as EC2, S3, and DynamoDB, making automation easier and more efficient.

Prerequisites:

• AWS CLI and Boto3 installed
• AWS account with I AM user access, NOT root user
• Basic AWS command line knowledge
• Basic Python programming language
• Basic knowledge of AWS Interactive Development Environment (IDE)

The Project

PART I

Our DevOps team frequently uses a development lab to test new releases of our application. However, management has raised concerns about the rising costs of maintaining the lab. To reduce expenses, we need to stop all (for this example) 3 EC2 instances once all engineers have finished work.

Task: Create a Python script that can be executed to stop all EC2 instances.

PART II – Advanced

To avoid affecting production workloads, we want to ensure that only development instances are stopped. Enhance your script to include logic that stops only the running instances that have the tag Environment: Dev, leaving all other instances untouched.

References about AWS Cloud9 and Boto3 can be found on the official AWS website: Official AWS and Cloud9 documentation

Step 1: Install boto3
First of all, we need to install Boto3 in the IDE environment. This allows Boto3 to interact with our instances.

and then we also need to install AWS CLI by running the code:

Now we already have a repo “gold-member” setup from our previous project we’d be creating a branch on this repo for our project. Follow the steps below to create a new branch “Week14ProjectBranch”

Go to Cloud9 > Click Main > click create new branch > Name the branch > Week14ProjectBranch” > Then hit the Enter key on your keyboard to switch automatically to this new branch

On cloud9 CLI clone the newly created repository by running the code below

As you can see above, our repo have now been cloned from our remote GitHub to our local cloud9 IDE.

Next we need to create a new branch to work with and then a python file. Follow the steps below:

Go to cloud9 > Select File > New From Template > Python File > Save As… and give it a name. Make sure you KEEP the extension .py.

We need to switch to our working directory for this project that is “gold-member” using the code below

Step 2: Create a python script that you can run to build, start and stop all Instances.

Our objective is to create a python script that will create and stop all 3 EC2 instances. To achieve this we’ll use the code below:

To explain how each line of the code works see below

Line 1: import boto3

This line is used to import the Boto3 library, which is a Python library for interacting with AWS services programmatically.

Line 2: ec2 = boto3.resource(‘ec2’)

This line of code is for creating EC2 resource object by calling the resource method of the Boto3 EC2 client with the ec2 variable holding this resource object. Moreover the EC2 service in AWS is needed as it is used to interact with our EC2 instances.

Lines 3 to 8: ec2.create_instances()

instance = ec2.create_instances(
    ImageId='ami-09c5c62bac0d0634e',
    InstanceType='t2.micro',
    MinCount=3,
    MaxCount=3,
    TagSpecifications=[{'ResourceType':'instance','Tags':[{'Key':'Name','Value':'EmmanueEnv'}]}]
)

The function “instance = ec2.create_instances()” as shown above holding several parameters allows us to launch one or more EC2 instances as required by specifying the following details as needed

**ImageId: **This is the Amazon Machine Image (AMI) we want to use for our instance. We will be using a Linux AMI with the ID ami-09c5c62bac0d0634e.

InstanceType: This is the type of EC2 instance we want to launch. In this case we’re using the t2.micro fee tier eligible

MinCount: This is the minimum number of instances we want to launch.

MaxCount: This is the maximum number of instances we want to launch.

TagSpecifications: This indicates the tags on our instance. Adding a tag with the key Name and the value to the instance allows us to name our instances.

Line 9: print(instance)

Finally we print out the instance object using the print(instance) statement giving us the output of our newly created instance, including its ID, state etc….

Now let’s create our EC2 Instances. For the purpose of this project we are going to create 3 instances for production and another 3 for development.

Create Developments Instances

We will use the same code above for both sets of Instances. The only difference will be the TagSpecifications. The value Production specifying our production EC2 and Development specifying our development EC2. For the Production instances use the code below:

import boto3


ec2 = boto3.resource('ec2')


dev = ec2.create_instances(
    ImageId='ami-09c5c62bac0d0634e', #Image ID is specidic to your account
    InstanceType='t2.micro',
    MaxCount= 3,
    MinCount= 3,
    TagSpecifications=[
        {
            'ResourceType': 'instance',
            'Tags': [
                {'Key': 'Name','Value': 'Development'},#Change the ID as required
                {'Key': 'ENV','Value': 'Development'}

            ]
        }
    ]
)

print(dev)

Create Production Instances

To create the production Instances we use the same code as above and just change the tagspecification from Development to Production as shown below:

import boto3

ec2 = boto3.resource('ec2')


dev = ec2.create_instances(
    ImageId='ami-09c5c62bac0d0634e', #Image ID is specidic to your account
    InstanceType='t2.micro',
    MaxCount= 3,
    MinCount= 3,
    TagSpecifications=[
        {
            'ResourceType': 'instance',
            'Tags': [
                {'Key': 'Name','Value': 'Production'},#Change the ID as required
                {'Key': 'ENV','Value': 'Production'}

            ]
        }
    ]
)

print(dev)

Script to Stop all three Development Instances
To stop all three development instances use the script below

Push Python Script Code to GitHub

Awesome! Now that completes the first part of the project! Now ensure to push your code to to GitHub as it is always a good practice. With our code on GitHub we can always access it wherever and whenever we need to. Click here to see how to push our project to GitHub.

Challenges Encountered

When I initially tried to run the code, I experienced the error massage as shown below:

traceback (most recent call last): file "/home/ec2-user/environment/gold-member/week14projectfile.py", line 1, in <module> import boto3 modulenotfounderror: no module named 'boto3'

Solution:

From research online I observe that this happened because boto3 was not installed in the same location as our file path “/gold-member/Week14ProjectFile.py”. Go to the path to your Python environment (location of the python.exe file) through the command line and then reinstall boto3 by running the code below:

I hope you have also learnt something about the Automation of EC2 Instance management using Boto3.

if this was insightful to you, please give my blog a follow