DEV Community: Rimpal Johal

Understanding AWS Instance Metadata Service: A Closer Look

Rimpal Johal — Sun, 21 Jan 2024 22:05:59 +0000

In the realm of cloud computing, particularly within the Amazon Web Services (AWS) ecosystem, there exists a service that, while invisible to many, plays a crucial role in the efficient management of virtual machines. This is the AWS Instance Metadata Service (IMDS). Below, we unravel what IMDS is, its significance in AWS, where you can find it within the AWS Management Console, and why it needs to be secured.

What is the AWS Instance Metadata Service?

AWS Instance Metadata Service is a specialized service that allows AWS Elastic Compute Cloud (EC2) instances to access instance-specific data, which is pivotal for the configuration or management of the running instance. The service is reachable at a special URL that is only accessible from the EC2 instance itself: http://169.254.169.254/latest/meta-data/.

This metadata includes details such as the instance's IP address, the IAM role it is using, information about the AMI (Amazon Machine Image) used to launch the instance, and much more.

Why Does AWS Use IMDS?

IMDS serves as a bridge between the EC2 instance and the AWS environment. It allows instances to bootstrap themselves with the necessary configuration without the need to hard-code sensitive data into the instance or AMI. This can include dynamically setting environment variables, configuring services at runtime, or even obtaining temporary credentials to access other AWS services securely.

Security Implications of IMDS

While IMDS is undeniably useful, it can also pose a security risk if not properly secured. Here are the few potential threats to highlight

If an attacker gains access to your EC2 instance, they could potentially retrieve IAM credentials and other sensitive data from the IMDS.
AWS introduced IMDSv2, which requires session-oriented requests, to mitigate the risk of unauthorized retrieval of metadata. IMDSv1, on the other hand, allows any process that can access http://169.254.169.254/latest/meta-data/ to retrieve instance metadata.

What is IMDSv1 and How it is different from IMDSv2

IMDSv1 stands for "Instance Metadata Service version 1" in AWS (Amazon Web Services). It's a service provided by Amazon EC2 (Elastic Compute Cloud) to allow EC2 instances to retrieve instance-specific data, such as the instance ID, public IP, and user data, among other metadata.
The Instance Metadata Service (IMDS) allows EC2 instances to access metadata about themselves. This metadata includes information like the instance type, security groups, and more. Applications or scripts running on the EC2 instance can use this data to make decisions or to configure themselves accordingly.

Endpoint: IMDS is available at a specific IP address (169.254.169.254) that can be accessed from within the instance. This is a link-local address, which means it's only accessible from the instance itself and not from outside. 
IMDSv1 vs IMDSv2: AWS introduced IMDSv2 as an enhanced version of IMDSv1. The primary difference is in the security model:

IMDSv1: Allows direct retrieval of metadata with a simple HTTP GET request to the aforementioned IP.

IMDSv2: Introduces session-based requests. Before fetching metadata, you need to create a session by making a PUT request. This session returns a token, which must be provided in subsequent GET requests for metadata.

Security Considerations: IMDSv1 can be vulnerable to certain types of attacks, especially if applications on the instance do not properly validate external input. For example, a server-side request forgery (SSRF) attack could trick the server into retrieving instance metadata. IMDSv2 was introduced to help mitigate this type of risk by requiring session tokens. 
Usage: To use IMDSv1, one would typically use a tool like curl to fetch data from the endpoint. For example: 
curl http://169.254.169.254/latest/meta-data/instance-id

AWS recommendation and more details about IMDS

AWS recommends using IMDSv2 due to its enhanced security features. If you're using EC2 instances and rely on instance metadata, it's a good idea to evaluate and potentially migrate to IMDSv2.
In summary, IMDSv1 is the original version of the Instance Metadata Service provided by AWS EC2, allowing instances to access metadata about themselves. However, due to security improvements, AWS introduced IMDSv2, which is now the recommended version for accessing instance metadata.

Both IMDSv1 and IMDSv2 are enabled by default on all EC2 instances. This means that you can make requests using either version without any additional configuration.
Although both versions are available, AWS recommends using IMDSv2 due to its enhanced security features. IMDSv2 uses session-oriented requests, which can help protect against certain vulnerabilities, such as server-side request forgery (SSRF) attacks.
You can modify the IMDS settings for an EC2 instance. For example, if you want to enforce the use of IMDSv2 for security reasons, you can configure the instance to disable IMDSv1. This is done through EC2 instance metadata options when launching a new instance or modifying an existing one.
While new instances have both versions enabled, it's worth noting that very old EC2 instances that have been running since before the introduction of IMDSv2 might only have IMDSv1 available. However, this would be rare, as IMDSv2 has been available for several years.
In summary, by default, both IMDSv1 and IMDSv2 are enabled on all EC2 instances. AWS recommends using IMDSv2 for its security benefits, and you have the option to configure which versions are enabled on your instances.

If you're trying to determine which versions of IMDS are enabled on a running EC2 instance, you'd typically check the instance's metadata options or use AWS Management Console, AWS CLI, or SDKs. AWS provide multiple ways to determine the version of IMDS, details are as given below:

AWS Management Console: Checking for IMDSv1-Enabled Instances: In the AWS Management Console, it's possible to determine which instances have IMDSv1 enabled by using the IMDSv2 attribute column on the Amazon EC2 page. To view the IMDSv2 attribute column, follow these steps:

Navigate to the Amazon EC2 console and select 'Instances'.
Click on the settings gear icon located in the upper-right corner.
Find 'IMDSv2' in the list and activate it by moving the slider to the on position.
Click 'Confirm' to apply this setting.

This action will display the IMDS status for your instances. If the status is marked as 'optional', it indicates that IMDSv1 is enabled for that instance. Conversely, a status of 'required' suggests that IMDSv1 is disabled.

Identifying IMDSv1-Enabled Instances via AWS CLI
To ascertain if your instances have IMDSv1 enabled, utilize the AWS Command Line Interface (AWS CLI). This is achievable by executing the aws ec2 describe-instances command and examining the HttpTokens value. This particular value is pivotal in determining the enabled version of IMDS. A setting of optional signifies that both IMDSv1 and IMDSv2 are enabled, whereas required implies that only IMDSv2 is in use. In essence, when the status is optional, it indicates the activation of IMDSv1 on the instance, and required denotes that IMDSv1 is deactivated.
Identifying IMDSv1-Enabled Instances Using Security Hub
Within Security Hub, there's a specific control for Amazon EC2 that leverages the AWS Config rule, named ec2-imdsv2-check, to verify whether the instance metadata version is set to IMDSv2. Should the HttpTokens setting be configured as optional, the rule will flag as NON_COMPLIANT, indicating that the EC2 instance is operating with IMDSv1 enabled.

Below is a Python code snippet using Boto3, the AWS SDK for Python, that will list all EC2 instances (both running and stopped) in an AWS account and determine which versions of IMDS are enabled on each:

import boto3

# Initialize the EC2 client
ec2_client = boto3.client('ec2')

def check_imds_versions():
    # Paginate through all EC2 instances
    paginator = ec2_client.get_paginator('describe_instances')
    for page in paginator.paginate():
        for reservation in page['Reservations']:
            for instance in reservation['Instances']:
                instance_id = instance['InstanceId']
                state = instance['State']['Name']

                # Get IMDS version
                imds_version = "Unknown"
                if 'MetadataOptions' in instance:
                    if instance['MetadataOptions']['HttpTokens'] == 'required':
                        imds_version = "IMDSv2"
                    else:
                        imds_version = "IMDSv1 & IMDSv2"

                print(f"Instance ID: {instance_id}, State: {state}, IMDS Version: {imds_version}")

if __name__ == "__main__":
    check_imds_versions()

The AWS IMDSv1 service uses a request/response access method and the AWS IMDSv2 service uses a session-oriented method. Both services are fully secure, but v2 provides additional layers of protection for four types of vulnerabilities that could be used to try to access IMDS.

Implementing IMDSv2 on EC2 Instances

Upon identifying instances operating with the default IMDSv1, we can proceed to adopt recommended practices provided by AWS to transition these instances from IMDSv1 to the more secure IMDSv2 protocol.
For Launching New Instances:
To ensure new instances utilize IMDSv2, disable IMDSv1 and enable IMDSv2 through the metadata-options parameter in the run-instances CLI command:

aws ec2 run-instances \
    --image-id <ami-xxxxxxxxxxx> \
    --instance-type m4.large \
    --metadata-options "HttpEndpoint=enabled,HttpTokens=required"

To Update Running Instances:
Modify the metadata options of an already running instance to enforce the use of IMDSv2:

aws ec2 modify-instance-metadata-options \
    --instance-id <instance-xxxxxxx> \
    --http-tokens required \
    --http-endpoint enabled

For Configuring New AMIs:
When registering a new Amazon Machine Image (AMI), set it to support IMDSv2 by default:

aws ec2 register-image \
    --name <private-image> \
    --root-device-name /dev/xdga \
    --block-device-mappings DeviceName=/dev/xvda,Ebs={SnapshotId=<snap-xxxxxxx>} \
    --imds-support v2.0

Launching Instances via the AWS Console with IMDSv2:
When initiating the launch of instances through the AWS Console, proceed by selecting 'Launch Instance'. After reaching the instance configuration stages, navigate to the 'Advanced details' tab. Scroll to the 'Metadata version' section and ensure to select 'V2 only (token required)' from the available options. This setting ensures that the instance will exclusively utilize IMDSv2.

Implementing IMDSv2 with EC2 Launch Templates:
EC2 launch templates serve as configurable blueprints that Amazon Auto Scaling groups leverage for the deployment of EC2 instances. When crafting these launch templates through the AWS Console, it's possible to designate the Metadata version. To enhance security, select 'V2 only (token required)' in the Metadata version settings. This choice ensures that any EC2 instances launched using this template will automatically use the more secure IMDSv2.

Configuring IMDSv2 in EC2 Launch Templates via AWS CloudFormation:
When utilizing AWS CloudFormation to create an EC2 launch template, it's crucial to set up the MetadataOptions property to enforce the use of IMDSv2. This is achieved by specifying HttpTokens as required. With this configuration, the process of fetching AWS Identity and Access Management (IAM) role credentials will only yield IMDSv2 credentials, effectively making IMDSv1 credentials inaccessible.
To implement this, include the following properties within your CloudFormation template:

{
  "HttpEndpoint": "<String>",
  "HttpProtocolIpv6": "<String>",
  "HttpPutResponseHopLimit": <Integer>,
  "HttpTokens": "required",
  "InstanceMetadataTags": "<String>"
}

This setup ensures that any EC2 instances launched using the CloudFormation template are configured with the enhanced security standards of IMDSv2.

Leveraging IMDSv2 in EC2 Instances with AWS Cloud Development Kit (CDK):
For those employing the AWS Cloud Development Kit (AWS CDK) to orchestrate and deploy EC2 instances, the requireImdsv2 property presents a streamlined way to enhance instance security. By setting requireImdsv2 to true, you effectively disable IMDSv1 and enable IMDSv2, ensuring a higher level of security compliance.In your AWS CDK script, configure the EC2 instance as follows:

new ec2.Instance(this, 'Instance', {
    // ...include other necessary parameters
    requireImdsv2: true,
});

AWS IMDSv2 provides additional protection against the following types of vulnerabilities compared to IMDSv1:

Server-Side Request Forgery (SSRF): SSRF vulnerabilities occur when a malicious actor can cause a server to make a request to an unintended location, such as the instance metadata service. IMDSv2 mitigates this risk by requiring a session token that is obtained via a PUT request. This token must be included in subsequent requests to access the metadata, which is something an SSRF vulnerability typically cannot perform since it can often only induce the server to make simple GET requests.
Open Firewalls: In some configurations, the firewall might be set to allow all traffic to reach the instance metadata service. This can be risky, especially if the services running on the instance have vulnerabilities or are misconfigured. IMDSv2 requires the use of a secret token, which is only known to the instance and AWS, thus reducing the risk from an open firewall.
Insecure Instance Roles: Instance roles are used to grant permissions to EC2 instances to access other AWS services. If these roles are overly permissive, they can be exploited. IMDSv2 reduces the risk by making it more difficult for a potential attacker to retrieve the credentials associated with the role since they would need to obtain a valid session token first.
Misconfigured HTTP Proxies: HTTP proxies within an EC2 instance can inadvertently allow access to the IMDS if not properly secured. Since IMDSv1 only requires a simple HTTP GET request, any process that can send such requests through the proxy can potentially access the metadata service. IMDSv2's use of a session-oriented approach requires a PUT request to get a token, which is typically not allowed through an HTTP proxy by default, thus mitigating this risk.

While IMDSv1 is considered secure under many circumstances, IMDSv2 adds these layers of security to protect against specific attack vectors that have been identified as potential risks. AWS recommends using IMDSv2 whenever possible to leverage these additional security benefits.

AWS Lambda Permissions: Resource-Based Policies vs. IAM Roles

Rimpal Johal — Sat, 04 Nov 2023 18:08:22 +0000

When it comes to managing access permissions for your AWS Lambda functions, AWS provides two primary methods: resource-based policies and IAM roles. Each method has its own set of use cases, benefits, and limitations. In this article, I'll delve into the details of Lambda resource-based policies and IAM roles, compare them, provide examples of each, and offer guidance on selecting the best approach in line with AWS best practices and the AWS Well-Architected Framework.

What is a Lambda Resource-Based Policy?

A Lambda resource-based policy is an access policy attached directly to a Lambda function. This policy defines which entities (principally AWS accounts and other AWS services) can invoke your Lambda function and what specific actions they can perform. It's a straightforward way to grant access to your Lambda function without having to manage separate IAM (Identity and Access Management) policies or roles.

Example of a Lambda Resource-Based Policy:
Imagine you want to allow another AWS account to invoke your Lambda function. Here's a snippet of what that resource-based policy might look like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:us-west-2:987654321098:function:my-function"
    }
  ]
}

Note that this policy can be viewed and edited directly in the AWS Lambda console under the 'Permissions' tab of your Lambda function.

What is an IAM Role?

An IAM role is an entity within AWS that defines a set of permissions that dictate what the identity can and cannot do in AWS. IAM roles can be assumed by trusted entities, such as IAM users, applications, or AWS services, including AWS Lambda. Roles are a secure way to grant permissions that applications can assume when they need to perform actions on your behalf.

Example of an IAM Role Policy for Lambda:

Consider an IAM role that grants permission to invoke any Lambda function in your account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "lambda:InvokeFunction",
      "Resource": "*"
    }
  ]
}

You can create and manage IAM roles in the IAM console, under the 'Roles' section.

Resource-Based Policy vs. IAM Roles: The Differences

The key differences between the two are:

Attachment: Resource-based policies are attached directly to a Lambda function, while IAM roles are standalone entities.
Management: Resource-based policies are managed in the Lambda console, whereas IAM roles are managed in the IAM console.
Scalability: Resource-based policies are less scalable due to their size limit. IAM roles are more scalable as they don't have this limitation.
Flexibility: IAM roles are more flexible, as they can be assumed by many different accounts and services without modifying the policy on the resource itself.

Best Practices and the AWS Well-Architected Framework

According to AWS best practices and the Well-Architected Framework, the choice between using a Lambda resource-based policy and IAM roles largely depends on the scale of your operation and the need for flexibility.

For simple scenarios with a few resources, a resource-based policy is adequate and easier to manage. However, as your environment grows, IAM roles are recommended. They offer greater scalability and flexibility, which align with the AWS Well-Architected Framework's principle of granting least privilege—providing only the necessary permissions to perform a task.

IAM roles are particularly beneficial when dealing with:

Cross-account access.
Applications that run on Amazon EC2 or other AWS services that need to perform actions on your Lambda functions.
Environments where functions are invoked by many different principals.

Cross-Account Access via IAM Roles

To enable cross-account access, you must create an IAM role with the necessary permissions and then allow other AWS accounts to assume that role. Here's how you can achieve this:

Step 1: Create an IAM Role

First, you create an IAM role in the account that owns the Lambda function. You define a trust policy that allows entities from the other account to assume the role.

Trust Policy for IAM Role (Trust Relationship):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::OTHER_ACCOUNT_ID:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step 2: Attach Permissions to the IAM Role

Next, you attach a permissions policy to the IAM role that grants the lambda:InvokeFunction permission.

Permissions Policy for IAM Role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME"
    }
  ]
}

Replace REGION, ACCOUNT_ID, and FUNCTION_NAME with your specific details.

Step 3: Assume the IAM Role from the Other Account

The entity in the other account must then use the AWS Security Token Service (STS) to assume the IAM role. Here's how you could do that programmatically also using AWS SDK for Python (Boto3).

Concluding the blog here, I would like to say while resource-based policies offer a quick and easy way to grant access to your Lambda functions, they come with limitations, especially concerning policy size and scalability. IAM roles, on the other hand, provide a more robust and flexible solution that can grow with your AWS environment.

When designing your cloud architecture, consider using IAM roles for their scalability and adherence to the principle of least privilege. By doing so, you'll ensure your AWS infrastructure remains secure, efficient, and well-aligned with AWS best practices.

Managing Amazon S3 Objects with Python

Rimpal Johal — Tue, 28 Mar 2023 05:48:05 +0000

Amazon Simple Storage Service (S3) is a popular cloud storage service that provides scalable and secure object storage for various types of data. Managing S3 objects can be a daunting task, especially when dealing with large datasets. In this blog post, we'll explore how to use Python and the boto3 library to manage S3 objects and list the objects created after a specific date and time.

To get started, we'll need to install the boto3 library, which is the Amazon Web Services (AWS) SDK for Python. Once installed, we can create an S3 client and set the region name and bucket name that we want to work with. We'll also set the timezone to Melbourne, Australia, using the pytz library.

Next, we'll set the start date for listing the S3 objects. In this example, we'll set the start date to March 5, 2023, at 12:00 AM in Melbourne time. We'll use the datetime and tzinfo modules to define the start date and timezone.

We'll then use a paginator to iterate over all objects in the S3 bucket and filter the objects that were created after the specified start date. The paginator will help us handle large datasets and ensure that we don't exceed any API rate limits.

We'll convert the UTC LastModified time of each object to Melbourne timezone using the astimezone() method and format it as a string that includes the timezone name and offset. We'll also convert the size of each object from bytes to megabytes and store the filtered objects in a list.

Finally, we'll print and write the list of objects to a file named 's3_objects.txt' using the pprint and open() methods. The output file will include the object key, LastModified time, and size in megabytes.

Here's the complete Python script:

import boto3
import datetime
import pprint
import json
import pytz

# Create an S3 client
s3 = boto3.client('s3', region_name='ap-southeast-2')

# Set the S3 bucket name
bucket_name = 'myuats3bucket'
tz = pytz.timezone('Australia/Melbourne')
# Set the start date for listing objects
start_date = datetime.datetime(2023, 3, 5, tzinfo=tz)
start_after = start_date.strftime('%Y-%m-%d %H:%M:%S')
print (start_after)

# Set the prefix for filtering objects
prefix = ''

# Initialize the list of objects
object_list = []

# Use a paginator to iterate over all objects in the bucket
paginator = s3.get_paginator('list_objects_v2')
try:
    for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix, StartAfter=start_after):
    # If there are no more objects, stop iterating
        if 'Contents' not in page:
            break

    # Iterate over each object in the current page of results
    for obj in page['Contents']:
        # Add the object to the list if it was created after the start date
        if obj['LastModified'] >= start_date:
            # Convert UTC LastModified time to Melbourne timezone
            melbourne_time = obj['LastModified'].astimezone(tz)
            obj['LastModified'] = melbourne_time.strftime('%Y-%m-%d %H:%M:%S %Z%z')
            # Convert size to MB
            obj['Size'] = round(obj['Size'] / (1024 * 1024), 2)
            object_list.append(obj)
except Exception as e:
    print("Error:", e)

# Print the list of objects
pprint.pprint(object_list)

# Open the file for writing
with open('s3_objects.txt', 'w') as f:
    # Write the formatted output to the file
    for obj in object_list:
        f.write(f"Key: {obj['Key']}, Last Modified: {obj['LastModified']}, Size: {obj['Size']} MB\n")

In summary, using Python and the boto3 library to manage Amazon S3 objects is a powerful way to automate and streamline your data management workflows. With the ability to filter and format S3 objects based on specific criteria, you can save time and reduce errors in your data processing pipeline. Whether you're dealing with small or large datasets, this approach can help you manage your S3 objects more efficiently.

Thank you for reading this blog post, and we hope that you found it helpful. If you have any questions or feedback, please feel free to leave a comment below.

Restore Postgresql Database in a Docker Container

Rimpal Johal — Mon, 27 Mar 2023 07:09:11 +0000

Introduction

In this blog post, we will discuss a common challenge faced by many developers when setting up and configuring a PostgreSQL database in a Docker container. The problem we encountered was the need to create a PostgreSQL database with the PostGIS extension enabled and to import data from an existing SQL backup file. This process involves a series of steps, such as initializing the database, starting the PostgreSQL server, creating the database, enabling the PostGIS extension, and importing the data from the backup file.

To address this challenge, we'll use Docker, a platform that allows us to easily build, ship, and run applications in containers. Containers provide an isolated environment for running applications, which helps streamline the development and deployment process. In our case, we'll create a Docker container with PostgreSQL, a powerful and widely-used open-source relational database management system. Additionally, we'll be using the PostGIS extension, which adds support for geographic objects, allowing location queries to be run in SQL.

In this blog post, we will walk you through the entire process of creating a Docker container with a PostgreSQL database and PostGIS extension enabled. We'll also demonstrate how to import data from an existing SQL backup file into the newly created database. This tutorial will serve as a comprehensive guide for developers who need to set up a PostgreSQL database with the PostGIS extension in a Docker container and import data from a backup file.

Setting up the PostgreSQL Docker container

In this section, we'll guide you through the process of setting up a PostgreSQL Docker container with the PostGIS extension enabled. We'll start by creating a custom Dockerfile, which will define the necessary instructions and configurations for our container.

Here's the Dockerfile we used to create our PostgreSQL container:

# Base image with PostgreSQL and PostGIS
FROM postgis/postgis

# Set up environment variables
ENV PGDATA=/var/pgdata
ENV POSTGRES_USER=postgres
ENV POSTGRES_PASSWORD=mysecretpassword
ENV POSTGRES_DB=mypostgresDB

# Create a directory to store PostgreSQL data and logs
RUN mkdir -p ${PGDATA} /tmp /var/log/postgresql && chown -R postgres:postgres ${PGDATA} /tmp /var/log/postgresql

WORKDIR /data

# Expose the PostgreSQL port
EXPOSE 5432

# Set the user to run the container
USER postgres

# Copy the entrypoint script to the container
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

# Run the entrypoint script
CMD ["/entrypoint.sh"]

In this Dockerfile, we start with the postgis/postgis base image, which already includes PostgreSQL and the PostGIS extension. We then set up environment variables for the PostgreSQL data directory, the default PostgreSQL user, password, and database name. These variables will be used later in our entrypoint script.

We create directories to store PostgreSQL data and logs, ensuring that the postgres user has the necessary permissions to access these directories. The PostgreSQL port (5432) is exposed to allow connections from outside the container.

Finally, we copy our custom entrypoint.sh script into the container and set it as the command to be executed when the container starts. The entrypoint script is responsible for initializing the PostgreSQL data directory, starting the PostgreSQL server, creating the database with the PostGIS extension, and importing data from the SQL backup file.

By using this Dockerfile, we're able to create a customized PostgreSQL container with the PostGIS extension enabled, tailored to our specific requirements. This setup ensures that our database is configured and ready to use, with the necessary data imported, as soon as the container starts running.

Creating the entrypoint.sh script

The entrypoint.sh script is an essential part of our PostgreSQL Docker container setup, as it handles the initialization and configuration of the database. In this section, we'll walk you through the contents of the entrypoint.sh script, explaining each step in detail.

Here's the content of the entrypoint.sh script:

#!/bin/bash

# Initialize the PostgreSQL data directory
initdb -D ${PGDATA}

# Start PostgreSQL in the background
pg_ctl -D ${PGDATA} -l /var/log/postgresql/logfile start

# Wait for PostgreSQL to start
wait_postgresql() {
  while ! pg_isready -q; do
    echo "Waiting for PostgreSQL to start..."
    sleep 1
  done
}
wait_postgresql

# Create the Postgres database named "mypostgresDB" with the "template_postgis" template
createdb mypostgresDB --template=template_postgis

# Extract the schema and data from the backup file
pg_restore -f /tmp/schema_new.sql -s /data/mypostgresDB-backup.sql 2>&1 | tee /var/log/postgresql/schema_extract.log
pg_restore -f /tmp/new_data.sql -a /data/mypostgresDB-backup.sql 2>&1 | tee /var/log/postgresql/data_extract.log

# Import the schema and data into the new database
psql --dbname=mypostgresDB -f /tmp/schema_new.sql 2>&1 | tee /var/log/postgresql/schema_import.log
psql --dbname=mypostgresDB -f /tmp/new_data.sql 2>&1 | tee /var/log/postgresql/data_import.log

# Keep PostgreSQL running
tail -f /var/log/postgresql/logfile

Initialize the PostgreSQL data directory: The initdb command initializes the data directory for the PostgreSQL server, setting up the necessary files and folders.
Start PostgreSQL in the background: The pg_ctl command starts the PostgreSQL server in the background, with logs being written to the specified logfile.
Wait for PostgreSQL to start: The wait_postgresql function checks if the PostgreSQL server is ready to accept connections. It uses the pg_isready command to query the server status and waits until the server is ready.
Create the PostgreSQL database: The createdb command creates a new PostgreSQL database named mypostgresDB using the template_postgis template, which includes the PostGIS extension.
Extract schema and data from the backup file: The pg_restore command is used to extract the schema and data from the SQL backup file. We create two separate files: schema_new.sql for the schema and new_data.sql for the data. The extraction process logs are written to /var/log/postgresql/.
Import schema and data into the new database: The psql command imports the schema and data from the extracted files into the newly created mypostgresDB database. The import process logs are written to /var/log/postgresql/.
Keep PostgreSQL running: The tail -f command ensures that the PostgreSQL server keeps running by continuously monitoring the logfile.

By using this entrypoint.sh script, we ensure that our PostgreSQL container is fully configured and ready to use when it starts, with the necessary data imported from the backup file.

Running the Docker container and mounting the backup file

In this section, we'll cover the steps to run the Docker container and mount the backup file during the process. This allows the entrypoint.sh script to access the backup file and perform the required operations to restore the database schema and data.

First, build the Docker image using the Dockerfile created earlier. Navigate to the directory containing the Dockerfile and run the following command:

docker build -t postgres-postgis .

This command builds a new Docker image with the tag postgres-postgis.

Download the SQL backup file: Before running the Docker container, ensure that you have downloaded the SQL backup file from the S3 bucket to your local system. This file will be mounted as a volume in the Docker container to restore the database.

Run the Docker container: To run the Docker container and mount the backup file, use the following command:

docker run -d --name my-postgres -p 5432:5432 -v /path/to/your/local/backupfile.sql:/data/mypostgresDB-backup.sql postgres-postgis

Replace /path/to/your/local/backupfile.sql with the path to the downloaded SQL backup file on your local system. This command does the following:

Runs the Docker container in detached mode (-d) with the name my-postgres.
Maps the local port 5432 to the container's port 5432 (-p 5432:5432), allowing you to connect to the PostgreSQL server running inside the container.
Mounts the backup file as a volume in the container at /data/mypostgresDB.sql. This allows the entrypoint.sh script to access the backup file during the container startup.

Once the container is up and running, the entrypoint.sh script will initialize the PostgreSQL server, create the database, extract the schema and data from the backup file, and import them into the new database. You can then connect to the PostgreSQL server using your preferred client and verify that the database has been restored correctly.

By following these steps, you have successfully created a Docker container for a PostgreSQL server with the PostGIS extension and restored a database using an SQL backup file.

Conclusion

In this blog post, we have demonstrated how to set up a PostgreSQL Docker container with the PostGIS extension and restore a database using an SQL backup file. We walked through the process of creating a Dockerfile to define the container, implementing an entrypoint.sh script to manage the database restoration process, and running the Docker container while mounting the backup file.

By leveraging Docker and PostgreSQL, we have created an easily deployable and scalable solution for managing and restoring geospatial databases. This approach not only simplifies the database restoration process but also ensures a consistent environment for your database server across different stages of development, testing, and production.

With this knowledge, you can now confidently use Docker and PostgreSQL to manage your geospatial databases, and restore them from backup files as needed.

AWS Multi-AZ FSX

Rimpal Johal — Sat, 22 Oct 2022 01:31:50 +0000

Introduction

Amazon FSx for Windows File Server is a file system that provides the same features as NTFS. The file system can be deployed across multiple Availability Zones in the AWS Region, which will help ensure high availability and durability of your file system in the event of a catastrophic failure or disaster.

What is AWS Multi-AZ?

AWS Multi-AZ is a feature of AWS that provides high availability and durability for your file system in the event of a catastrophic failure or disaster. For example, if you have an on-premises Windows File Server (WFS) and you want to use the WFS with AWS, then AWS Multi-AZ can be used to provide high availability and durability for your file system by replicating data between multiple Availability Zones.

What is FSx for Windows File Server

Amazon FSx for Windows File Server provides fully managed Microsoft Windows file servers, backed by a fully native Windows file system. FSx for Windows File Server has the features, performance, and compatibility to easily lift and shift enterprise applications to the AWS Cloud.You can use it to store your data and share it with other application services running on the same instance or applications running in other AWS Regions. Amazon FSx for Windows File Server offers the benefits as

High availability and durability
Resilience to common operating system failures through consistency groups, which make sure every region has replicas of your file system data. If one region fails—for example because of an electrical problem or hardware failure—the other regions continue serving files from their own replicas without interruption.

What is AWS Multi-AZ FSx?

AWS Multi-AZ FSx is a file system that can be deployed across multiple Availability Zones in the AWS Region. This will help ensure high availability and durability of your file system in the event of a catastrophic failure or disaster.

How does AWS Multi AZ Fsx benefit me?

You can't afford to have your business data stored in a single-region location, so Multi-AZ Fsx is a perfect fit. It'll keep your data safe from any catastrophic events that might happen elsewhere, whether it's an earthquake or a cyber attack.
You can also expect to see more performance as well as cost efficiency with Multi-AZ Fsx, since there are two copies of data in different availability zones that can be used simultaneously by your applications. This helps ensure that no matter what happens in one zone, there will still be access to all of the needed information for transactions and other operations to complete successfully.

Also as Multi-AZ FSx is hybrid-enabled, it helps me to migrate and synchronize large data sets from data centre to AWS and further facilitate to immediately available the data to a broad set of integrated AWS Services.

How do I create an AWS Multi-AZ deployment for Fsx?

One can deploy the Multi-AZ FSx both from AWS Console or from CloudFormation Template. I have used the CloudFormation template to create an FSx file system across multiple availability zone.

AWS Multi-AZ FSx deployment Cloudformation snippet for reference. Please note this need to change as per your business requirements. In the below given code snippet, I have created three Multi-AZ FSx:

DRBSQLDataFileSystemMAZ Size 40 GB
DRBSQLLogsFileSystemMAZ Size 32 GB
DRBSQLDBFileSystemMAZ Size 32 GB



Resources:

DRBSQLDataFileSystemMAZ:

Type: 'AWS::FSx::FileSystem'

Properties:

FileSystemType: WINDOWS

StorageCapacity: 40

StorageType: SSD

SubnetIds:


!Ref DRBWorkloadSubnetID1
!Ref DRBWorkloadSubnetID2
SecurityGroupIds:
!FindInMap
!Ref Environment
DRBSg
sgSQLFsx
Tags:
Key: Name
Value: !Sub DRB-SQL-Data-Fsx-Maz
Key: OS
Value: WINDOWS
Key: !Ref Environment
Value: Prod
Key: AppName
Value: SQLWindowsFileShareMaz
WindowsConfiguration:
ThroughputCapacity: 32
AutomaticBackupRetentionDays: 30
WeeklyMaintenanceStartTime: '6:17:30'
DailyAutomaticBackupStartTime: '17:45'
CopyTagsToBackups: false
DeploymentType: MULTI_AZ_1
PreferredSubnetId: !Ref DRBWorkloadSubnetID1
SelfManagedActiveDirectoryConfiguration:
DnsIps:
!Select
0
!Split
','
!Ref DomainControllerIps
DomainName: 'example.com'
UserName: !Ref DRBSQLDBServiceAccountUName
Password: !Ref DRBSQLDBServiceAccountPwd


DRBSQLLogsFileSystemMAZ:

Type: 'AWS::FSx::FileSystem'

Properties:

FileSystemType: WINDOWS

StorageCapacity: 32

StorageType: SSD

SubnetIds:


!Ref DRBWorkloadSubnetID1
!Ref DRBWorkloadSubnetID2
SecurityGroupIds:
!FindInMap
!Ref Environment
DRBSg
sgSQLFsx
Tags:
Key: Name
Value: !Sub DRB-SQL-Logs-Fsx-Maz
Key: OS
Value: WINDOWS
Key: !Ref Environment
Value: Prod
Key: AppName
Value: SQLWindowsFileShareMaz
WindowsConfiguration:
ThroughputCapacity: 32
AutomaticBackupRetentionDays: 30
WeeklyMaintenanceStartTime: '6:17:00'
DailyAutomaticBackupStartTime: '17:15'
CopyTagsToBackups: false
DeploymentType: MULTI_AZ_1
PreferredSubnetId: !Ref DRBWorkloadSubnetID1
SelfManagedActiveDirectoryConfiguration:
DnsIps:
!Select
0
!Split
','
!Ref DomainControllerIps
DomainName: 'example.com'
UserName: !Ref DRBSQLDBServiceAccountUName
Password: !Ref DRBSQLDBServiceAccountPwd


DRBSQLDBFileSystemMAZ:

Type: 'AWS::FSx::FileSystem'

Properties:

FileSystemType: WINDOWS

StorageCapacity: 32

StorageType: SSD

SubnetIds:


!Ref DRBWorkloadSubnetID1
!Ref DRBWorkloadSubnetID2
SecurityGroupIds:
!FindInMap
!Ref Environment
DRBSg
sgSQLFsx
Tags:
Key: Name
Value: !Sub DRB-SQL-TempDB-Fsx-Maz
Key: OS
Value: WINDOWS
Key: !Ref Environment
Value: Prod
Key: AppName
Value: SQLWindowsFileShareMaz
WindowsConfiguration:
ThroughputCapacity: 32
AutomaticBackupRetentionDays: 30
WeeklyMaintenanceStartTime: '6:16:30'
DailyAutomaticBackupStartTime: '16:45'
CopyTagsToBackups: false
DeploymentType: MULTI_AZ_1
PreferredSubnetId: !Ref DRBWorkloadSubnetID1
SelfManagedActiveDirectoryConfiguration:
DnsIps:
!Select
0
!Split
','
!Ref DomainControllerIps
DomainName: 'example.com'
UserName: !Ref DRBSQLDBServiceAccountUName
Password: !Ref DRBSQLDBServiceAccountPwd

Security and Permission for Multi-AZ FSx

AWS recommends following windows file server ports to open as the mandatory requirement for the FSx to work in the new deployment. We have created defined these ports in the security group and attached that security group to Multi-AZ FSx. However, one port detail which is missing in AWS document is port 464 which is required for an inbound rule for both TCP and UDP traffic on domain controller instances. We have added port 464(tcp/udp) in the security group attached to the domain controller instances.

Conclusion

AWS Multi-AZ Fsx provides added protection for file systems. The concept of this deployment is similar to a database cluster: it keeps copies of your data in multiple locations so that if one location fails, there are still backups available. In addition, AWS Multi-AZ FSx provides a way to create file shares across multiple Availability Zones that can be managed by the same administrator or group of administrators through a single console.

Migration Evaluator install and Configuration

Rimpal Johal — Sun, 14 Aug 2022 06:41:36 +0000

Migration Evaluator which is formerly known as TSO Logic is the AWS provided tool which can facilitate access to insights on data center assets(mainly the workload earmarked for migration)and accelerate business decision-making for migration to AWS at no cost. Following data collection from data center(on-premise) you will quickly receive an assessment including a projected cost estimate and savings of running your on-premises workloads in the AWS Cloud.

To gather the required information, we need to deploy the agentless Migration Evaluator data collector to monitor VMware, Hyper-V, and bare-metal infrastructure. AWS Migration Evaluator leverages TSO Logic technology as part of the assessment.

TSO- Preinstall checklist

Has an account been created on https://console.tsologic.com? Please contact your Migration Evaluator specialist if you have not received an invitation request.
Need a dedicated windows server to install the migration evaluator (TSO) software with following requirements.

How to Install TSO on Windows Server

To Install TSO software, login to https://console.tsologic.com and download the below tools

Download and save the TSOBootstapper.exe from https://console.tsologic.com/discover/tools
onto the new designated server for TSO installation
Download and save the Migration Evaluator Collector software MSI from
https://console.tsologic.com/discover/tools onto the new designated server
Download and save the collector specific encryption certificate from
https://console.tsologic.com/discover/collectors onto the new designated server
Ensure you are logged in as a local Administrator and meet the server hardware requirements

Select Install and wait while the packages are installed. Once done, select the Close button to complete the process.
Once the bootstrapper installed, then run the TSOCollector_.msi. Select the certificate file (-.crt) previously downloaded.
Select to run the collector under a local system account or the service account created prior to install.
Select the Grant rights automatically checkbox and click OK to proceed. Click test Credentials to ensure you have required permissions to run the software.
Select HTTPS for communication with the IIS application.
Select Yes to automatically start the collection service once the install sequence finishes
Once installation has completed, your next step will be to create your local account for managing the collector. This is not the account used on https://console.tsologic.com
Access the Migration Evaluator Collector software by clicking on the newly created desktop shortcut, or by opening your browser at: https://localhost. Enter your desired credentials, and click Create Account
Take note of your recovery key. This will allow you access to the Migration Evaluator Collector if you forget your password.

Once open you will see the screen as below

How to configure OS collection VMware Servers

If on-premise servers are identified as a virtual machine in Datacenter then configure the collection as per OS credentials via WMI Protocol.

Before configuring the collection, test if WMI protocol is open from the TSO Server. Use the below command to check for one of the data center server. If the command is success, it will return the name else we will get an error as "Access Denied" which means that WMI port is not opened for that server. It is advisable to contact your organisation system administrator to gain access to the server via WMI.



wmic /user:/<"user name"> /node:"10.28.x.x" CSPRODUCT GET NAME

Once get the confirmation on WMI connectivity, proceed to the configuration bit.

Configure OS Credentials

As per the official migration evaluator documentation, If we have servers identified as virtual machines then we need to configure Operating system credentials . Go to Configuration tab -> Global -> OS credentials tab -> Select WMI protocol -> and click on “Next”

In the above screen, select the "WMI" protocol and provide name "VMware Virtual Platform", domain as your organisation domain and username and password as service account credentials which are provided you by your administrator and then click on "save".

Prepare the CSV file

To add all the servers’ details in Data providers section, Prepare the CSV file with Name and IP and save the file to your local system.
If all the servers in your organisation are identified as a virtual machine in Datacenter. Prepare the csv file with Name and IP all servers for Operating system metrics collection from Migration Evaluator (TSO).

**Upload the CSV to Migration Evaluator

In the migration evaluator go to Global settings --> Data Providers --> Select "AWS Migration Evaluator" and click "Next"

In the next screen provide the name as "VMware" and browse to the location of the CSV file from your computer.

Check the collection for all Servers

Go to Device settings --> Under Bare_metal list --> Operating systems, you can see all the servers configured for collection.

This is stage where you need to "Test Colection"

Select any one of the server which you can see from your data center and click on "Test collection", if it return the status as "Success" that means your Data Center Server is ready for the collection.

Here you may have the possibility for some server which have difficulty in connection. Troubleshoot the faulty servers before you proceed to the next step of OS Data collection.

OS Collection

The final step is initiate the data collection using the configured Migration Evaluator Tool
You may need to add few servers manually which are having the issue with credentials.

For adding servers individually, go to configuration -->Global-->OS Credentials and add each server individually with appropriate credentials.

Once all the servers which are having issue with credentials are added , then go to “OS Collection” tab

Global Settings-> OS Collection -> Click on “Scan all Adhoc Devices for OS Utilization”

Migration Evaluator will take minimum two weeks of time to generate the report with complete assessment of client infrastructure with OS level utilisation and Cost estimates.

VMC on AWS Overview

Rimpal Johal — Sun, 22 Aug 2021 12:13:07 +0000

VMware Cloud on AWS is a cloud service jointly developed by VMware and AWS. This offering brings the same technologies enterprise customers use on premises (e.g., VMware vSphere® ESXi, vSAN, NSX) into the cloud.

VMware Cloud on AWS (in short VMC) helps customers deploy and accelerate migration of VMware vSphere-based workloads to the cloud while providing robust capabilities and hybrid cloud solutions. VMware Cloud on AWS allows organizations to take immediate advantage of the scalability, availability, security, and global reach of the AWS infrastructure.

Also this gives customers capability to access native AWS services from the VMWare Cloud on AWS SDDC. The access to AWS native services from SDDC is without incurring any ingress or egress charges.

A key advantage of VMware Cloud on AWS is that it runs ESXi hosts on bare metal EC2 instances on the AWS infrastructure. This helps end users to establish a VMware Cloud on AWS
SDDC and quickly begin deploying workloads to the cloud.

Use cases for VMC on AWS are

Cloud Migration: This helps customers to migrate to cloud without converting or re-architecting their existing application stacks.
Data Center Extension: VMC is the most viable solution for customers who are looking for expanding data center capacity in a cost-effective way.
Disaster Recovery: VMC can be considered as a DR solution for a customers who want combine VMWare disaster recovery with AWS Cloud.
Application Modernization: Customers can liverage private access to AWS native services to modernize application with the use of feature rich AWS native services.

The first step for VMC on AWS is to - Deploy a SDDC. Below are the steps to deploy a SDDC.

Open https://console.cloud.vmware.com/ - login with your credentials.
Under my services, click on the VMWare Cloud on AWS tile.
Click on Create SDDC.
The first step when creating a new SDDC is to define properties relating to the AWS region you wish to deploy your SDDC to, the deployment and host types for the SDDC, and to give your SDDC a name. Enter the following parameters and click Next:

Property
Cloud <>
AWS Region << Your AWS Region>>
Deployment << Mult-Host>>
Host Type << I3 ( local SSD)
SDDC Name << Custom Name>>
Number of Hosts << 2 >>
In this step, connect to the AWS account(Note this is the customer AWS account - defined/named as sidecar account). Select the AWS account and click next
For VPC and subnet, select the VPC and choose the subnet.
For the management subnet, define the CIDR or use the default.
Review and acknowledge and click deploy SDDC.
Once the SDDC is deployed , you can see your new SDDC in the list of SDDCs.

That completes the first section of deploying the SDDC.

Virtual Private Gateways association to Direct Connect Gateways in AWS

Rimpal Johal — Tue, 17 Aug 2021 12:37:22 +0000

In this blog, I am explaining how to associate multiple virtual private gateways to single direct connect gateway in an AWS account.

Direct connect gateway gives you the option of associating multiple Virtaul Private Gateways(VGW) in an account to one direct connect gateway. When the direct connection established, it need to be consumed at AWS console either by VPG or by direct connect gateway. If you have multiple VPCs in the account and have multiple associated virtual gateways which need to be facilitated by one direct connect connection then direct connect gateway is the best option to manage this. Detailing below the steps to associate multiple VGWs to single direct connect gateway.

Create a Virtual Private Gateway.
After you create the Virtual Private Gateway, it will be in the detach state, attach it or associate it with your VPC.
After Virtual Gateway is attached to the VPC, create the direct connect gateway. Navigate to Direct Connect > Direct connect gateways and click on create direct connect gateways
Mention some name for the direct connect gateway and provide Amazon side ASN with in the given rage. This range need to define between 64512–65534
After direct connect gateway is created in your AWS account, it will show in the available state. Click on the gateway id and click the second tab gateway associations to associate your virtual gateway to the direct connect gateway.
Click on the associate gateway and attach the virtual private gateway which you have created earlier and associated with VPC.
Initially the page will show the status of associating and after 3–4 minutes, the state will change to associated.

As given above in the steps, you can associate multiple virtual private gateway to single direct connect gateway in an account. Please note this association can happen only with in the account and not cross-accounts.

CloudFormation stack creation using Python

Rimpal Johal — Thu, 12 Aug 2021 04:25:34 +0000

CloudFormation stack can be created from AWS Console, AWS CLI or using many other ways. We can also automate the creation of the CloudFormation stack using AWS CLI, CodePipeline etc. In this section of my blog, I am going to introduce how to use an AWS SDK for Python to automate the CloudFormation Stack creation. Here I am assuming that you know the basic Python and have understanding of the AWS SDK for Python. I would suggest to test the below scripts with small stack first and further you can customise the Python script for your own requirements.

Follow the steps below to automate the Cloudformation stack creation.

Make sure you have the latest Python installed on your box where you are intending to run the Python Script. In case of Mac it comes with default installed Python. I have upgraded it to the latest version.
```
rimpaljohal@MacBook-Pro Blog-Contents % Python --version
Python 3.9.6
```
Get hold of the AWS SDK for Python and installed it on the box where you are going to execute the Python Script.

AWS SDK for Python (Boto3)
You need two files for automate creation of CloudFormation stack.
CFNStackCreation.py --> Your Python Script
Parameter.json --> Your Parameter file
Keep these two files in the same directory path from where you are going to execute the Python script. For example
```
 rimpaljohal@MacBook-Pro Rimpal % ls CFNStackCreation.py Parameter.json 
CFNStackCreation.py Parameter.json
```

Python Script below for reference

#-- Import modules
import sys
import os.path
import json
import time
import boto3

 #-- Functions
def check_status( cfn_client_ss, cfn_stack_name_ss ):
stacks_ss = cfn_client_ss.describe_stacks(StackName=cfn_stack_name_ss)["Stacks"]
stack_ss_val = stacks_ss[0]
status_cur_ss = stack_ss_val["StackStatus"]
print ("Current status of stack " + stack_ss_val["StackName"] + ": " + status_cur_ss)
for ln_loop in range(1, 9999):
    if "IN_PROGRESS" in status_cur_ss:
        print ("\rWaiting for status update(" + str(ln_loop) + ")...",)
        time.sleep(5) # pause 5 seconds

        try:
            stacks_ss = cfn_client_ss.describe_stacks(StackName=cfn_stack_name_ss)["Stacks"]
        except:
            print (" ")
            print ("Stack " + stack_ss_val["StackName"] + " no longer exists")
            status_cur_ss = "STACK_DELETED"
            break

        stack_ss_val = stacks_ss[0]

        if stack_ss_val["StackStatus"] != status_cur_ss:
            status_cur_ss = stack_ss_val["StackStatus"]
            print (" ")
            print ("Updated status of stack " + stack_ss_val["StackName"] + ": " + status_cur_ss)
    else:
        break

return status_cur_ss
#-- End Functions

 #-- Main program
def main(access_key_ss, secret_key_ss, param_file_ss):

#-- Confirm parameters file exists
if os.path.isfile(param_file_ss):
    json_data_ss=open(param_file_ss).read()
else:
    print ("Parameters file: " + param_file_ss + " is invalid!")
    print (" ")
    sys.exit(3)

print ("Parameters file: " + param_file_ss)
parameters_data_ss = json.loads(json_data_ss)
region_ss = parameters_data_ss["RegionId"]

#-- Connect to AWS region specified in parameters file
print ("Connecting to region: " + region_ss)
cfn_client_ss = boto3.client('cloudformation', region_ss, aws_access_key_id=access_key_ss, aws_secret_access_key=secret_key_ss)

#-- Store parameters from file into local variables
cfn_stack_name_ss = parameters_data_ss["StackName"]
print ("You are deploying stack: " + cfn_stack_name_ss)
#-- Check if this stack name already exists
stack_list_ss = cfn_client_ss.describe_stacks()["Stacks"]
stack_exists_ss = False
for stack_ss_cf in stack_list_ss:
    if cfn_stack_name_ss == stack_ss_cf["StackName"]:
        print ("Stack " + cfn_stack_name_ss + " already exists.")
        stack_exists_ss = True

#-- If the stack already exists then delete it first
if stack_exists_ss:
    user_response = input ("Do you want to delete or update the stack")
    print ("Calling Delete Stack API for " + cfn_stack_name_ss)
    cfn_client_ss.delete_stack(StackName=cfn_stack_name_ss)

    #-- Check the status of the stack deletion
    check_status(cfn_client_ss, cfn_stack_name_ss)

print (" ")
print ("Loading parameters from parameters file:")
fetch_stack_parameters_ss = []
for key_ss in parameters_data_ss.keys():
    if key_ss == "TemplateUrl":
        template_url_ss = parameters_data_ss["TemplateUrl"]
    elif key_ss == "StackName" or key_ss == "RegionId":
        # -- Do not send as parameters
        print (key_ss + " - "+ parameters_data_ss[key_ss] + " (not sent as parameter)")
    else:
        print (key_ss + " - "+ parameters_data_ss[key_ss])
        fetch_stack_parameters_ss.append({"ParameterKey": key_ss, "ParameterValue": parameters_data_ss[key_ss]})

#-- Call CloudFormation API to create the stack   TemplateBody='', 
print (" ")
print ("Calling CREATE_STACK method to create: " + cfn_stack_name_ss)

status_cur_ss = ""

result_ss = cfn_client_ss.create_stack(StackName=cfn_stack_name_ss, DisableRollback=True, TemplateURL=template_url_ss, Parameters=fetch_stack_parameters_ss, Capabilities=["CAPABILITY_NAMED_IAM"])
print ("Output from API call: ")
print (result_ss)
print (" ")

#-- Check the status of the stack creation
status_cur_ss = check_status( cfn_client_ss, cfn_stack_name_ss )

if status_cur_ss == "CREATE_COMPLETE":
    print ("Stack " + cfn_stack_name_ss + " created successfully.")
else:
    print ("Failed to create stack " + cfn_stack_name_ss)
    sys.exit(1)

#-- Call Main program
if __name__ == "__main__":
if len(sys.argv) < 4:
    print ("%s:  Error: %s\n" % (sys.argv[0], "Not enough command options given"))
    print ("Argument 1 (required): AWS Access Key ")
    print ("Argument 2 (required): AWS Secret Access Key ")
    print ("Argument 3 (required): Stack Parameters JSON file ")
    print (" ")
    sys.exit(3)
else:
    access_key_ss = sys.argv[1]
    secret_key_ss = sys.argv[2]
    param_file_ss = sys.argv[3]

main(access_key_ss, secret_key_ss, param_file_ss)

Parameter json file below for reference. This is the template parameter json file and one can customise it as per the cloudformation build requirement. I am calling master stack here through this parameter file. Master stack is having the nested stacks to create VPC, Subnets(private and public, Database Subnets), application ec2 instances with autoscaling and Load Balancer.

 {
"RegionId": "us-east-1",
"TemplateUrl": "https://blogbucket.s3.amazonaws.com/masterstackv1.yml",
"StackName": "nonprodstack",
"EnvironmentType": "dev",
"VpcCIDR": "10.192.0.0/16",
"PublicSubnet1CIDR": "10.192.10.0/24",
"PublicSubnet2CIDR": "10.192.11.0/24",
"PublicSubnet3CIDR": "10.192.12.0/24",
"PrivateSubnet1CIDR": "10.192.20.0/24",
"PrivateSubnet2CIDR": "10.192.22.0/24",
"PrivateSubnet3CIDR": "10.192.24.0/24",
"DatabaseSubnet1CIDR": "10.192.21.0/24",
"DatabaseSubnet2CIDR": "10.192.23.0/24",
"DatabaseSubnet3CIDR": "10.192.25.0/24",
"LatestAmiID": "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2",
"ALBAccessCIDR": "0.0.0.0/0",
"ServerInstanceType": "t2.micro",
"WebAsgMax": "4",
"WebAsgMin": "3"
}

Once the required two files(Python script and the Parameter file) are saved in one directory then open the terminal and navigate/go to the directory which is having two mentioned files.

Execute the python scripts as given below

   rimpaljohal@MacBook-Pro Rimpal % python CFNStackCreation.py AU************* SL*********************** Parameter.json

Here you need to give AWS Access Key and AWS Secret Access Key of your AWS account.

As the script progress, you can see the progress on the Mac terminal.

Above defined steps will help you to deploy the CloudFormation stack using python script.

Happy Coding!!

Install Python AWS SDK Boto3

Rimpal Johal — Sun, 18 Jul 2021 05:48:53 +0000

This post is about installing AWS SDK boto3 on Mac. Python comes pre-installed on Mac OS so it is easy to start using it. However, to take advantage of the latest versions of Python, you will need to download and install newer versions alongside the system ones. Note that Python installers are available for the latest Python 3 and Python 2 releases that will work on all Macs that run Mac OS X 10.5 and later.

Follow the steps below to install Python AWS SDK boto3 on MacBook.

Check the version of Python on your Mac machine first

  @MacBook-Pro ~ % python --version
  Python 2.7.16

To install AWS SDK boto3 , it is recommended to upgrade the Python to latest version
Since the Python is system installed here so we need to install the latest version of Python and not upgrade the Python. Note that any attempt to upgrade the Python will give the error as " Error: Python not installed"

command to use is
```
 brew install python
```

Once the latest version is installed on your Mac, check the installed python by the command as below

@MacBook-Pro ~ % ls -ltra /usr/local/bin/python*

One can see the output as below from the above command

lrwxr-xr-x  1 rimpaljohal  admin  38 16 Jul 09:27 /usr/local/bin/python3 -> ../Cellar/python@3.9/3.9.6/bin/python3
lrwxr-xr-x  1 rimpaljohal  admin  45 16 Jul 09:27 /usr/local/bin/python3-config -> ../Cellar/python@3.9/3.9.6/bin/python3-config
lrwxr-xr-x  1 rimpaljohal  admin  40 16 Jul 09:27 /usr/local/bin/python3.9 -> ../Cellar/python@3.9/3.9.6/bin/python3.9
lrwxr-xr-x  1 rimpaljohal  admin  47 16 Jul 09:27 /usr/local/bin/python3.9-config -> ../Cellar/python@3.9/3.9.6/bin/python3.9-config
lrwxr-xr-x  1 rimpaljohal  admin  24 16 Jul 10:03 /usr/local/bin/python -> /usr/local/bin/python3.9

After confirming the python version, change the default python symlink to the version one want to use from above.

Command below will create the symbolic link for pointing the default Python path to the latest installed version of the Python

  ln -s -f /usr/local/bin/python3.9 /usr/local/bin/python

Close the terminal and reopen the new session. Check the version of Python now
```
  Python --version

  @MacBook-Pro ~ % Python --version
  Python 3.9.6
```
Here, I have updated my Mac to the latest Python Version. Now, I will move to install Boto3 on my Mac

Install the pip in the first place

sudo easy_install pip

Searching for pip
Best match: pip 21.1.3
Processing pip-21.1.3-py2.7.egg
pip 21.1.3 is already the active version in easy-  install.pth
Installing pip script to /usr/local/bin
Installing pip2.7 script to /usr/local/bin
Installing pip2 script to /usr/local/bin

Once the pip is installed then install boto3

@MacBook-Pro ~ % pip3 install boto3
Successfully installed boto3-1.18.0 botocore-1.21.0    jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.0 six-1.16.0 urllib3-1.26.6

After boto3 is installed then install awscli

@MacBook-Pro ~ % pip3 install awscli

Successfully installed PyYAML-5.4.1 awscli-1.20.0 colorama-0.4.3 docutils-0.15.2 pyasn1-0.4.8 rsa-4.7.2

In the final step configure awscli with and you are all ready to use Python AWS SDK boto3 on your Mac super machine!!

Some quick commands

@MacBook-Pro ~ % which pip
/usr/local/bin/pip

@MacBook-Pro ~ % pip3 --version
pip 21.1.3 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)

Happy Coding!!