DEV Community: Chandrashekar Y M

Automating Snowflake Resource Deployment using Terraform and GitHub Actions

Chandrashekar Y M — Sat, 18 May 2024 12:57:51 +0000

Lately at work, I have been using Terraform for our Infrastructure as Code (IaC) requirements for AWS workloads. As part of this learning journey, I also acquired Terraform Associate certification.

I wanted to explore Terraform for non-AWS use cases. At work, we are building a unified data platform for our data needs using Snowflake. So, I thought I will try to automate Snowflake resource deployments using Terraform.

Snowflake is defined as a cloud native, data platform offered as a SaaS. Lately, in the ever-evolving world of data platforms, Snowflake has emerged as a leading cloud-based data warehousing solution.

Manual provisioning of Snowflake resources like Databases, Schemas, Tables, Grants, Warehouses etc, is time consuming and prone to errors. This is where Infrastructure as Code (IaC) tools like Terraform and CI/CD pipelines using GitHub Actions makes life easier.

Terraform is an open-source, cloud agnostic IaC tool that allows us to define and provision cloud infrastructure using a very high level configuration language. Terraform implements this using plugins called providers.

GitHub Actions enables us to create efficient CI/CD pipelines based on code in GitHub repository.

This blog post demonstrates a step-by-step guide on how to deploy resources to Snowflake using Terraform and GitHub Actions, leveraging our repository cicd-with-terraform-for-snowflake. We will deploy a Database, a Schema, Grants and a Table onto two different environments (DEV and PROD) on the a Snowflake instance. We will use release based deployment pipelines to deploy to PROD environment.

Some pre-requisites and assumptions:

An AWS Account with an S3 bucket and DynamoDB table already provisioned - we will be using these for Terraform remote backend and State locking.
AWS credentials (Access Key and Secret Access Key) for the above AWS account configured as GitHub repository secrets.
A Snowflake instance, a user with ACCOUNTADMIN permissions and related Key-pair authentication setup. Related private key configured as GitHub repository secret.
A Snowflake role TF_READER pre-created in the Snowflake instance. We will be deploying grants for this role using Terraform resources.

Setting up repository:

Clone the repository to your local machine:

git clone https://github.com/shekar-ym/cicd-with-terraform-for-snowflake.git
cd cicd-with-terraform-for-snowflake

Repository Structure:

.github/workflows/: Contains the GitHub Actions workflow files that automate the deployment process.
dev and prod folders contain the Terraform files for development and production environment respectively.
module folder contains the Terraform module definition which will be used for provisioning Snowflake resources like Database, Schema and Tables.

Terrform Modules:

Modules are containers for multiple resources that are used together. Modules are used to package and reuse resource configurations with Terraform.

In our case, we will be using modules to define Snowflake database, schema, grants, table and warehouse resources configuration. This module will be reused to create resource for development and production environments.

For example, below is the module resource configuration for database and schema:

resource "snowflake_database" "tf_database" {
  name                        = var.database
  comment                     = "Database for ${var.env_name}"
  data_retention_time_in_days = var.time_travel_in_days

}

resource "snowflake_schema" "tf_schema" {
  name     = var.schema
  database = snowflake_database.tf_database.name
  comment  = "Schema for ${var.env_name}"
}

Refer to the Github repository for other module resource configurations.

GitHub Actions Workflow:

The workflow will trigger a deployment to DEV environment when you merge any code changes to main branch using a pull request.

The workflow also includes a step for infrastructure code scan to scan Terraform code. This uses Checkov action against infrastructure-as-code, open source packages, container images, and CI/CD configurations to identify misconfigurations, vulnerabilities, and license compliance issues.

    security-scan-terraform-code:
        name: Security scan (terraform code)
        runs-on: ubuntu-latest
        steps:
          - name: Checkout repo
            uses: actions/checkout@v4

          - name: Run Checkov action
            id: checkov
            uses: bridgecrewio/checkov-action@master
            with:
              directory: .
              soft_fail: true
              download_external_modules: true
              framework: terraform

There is a preview step, when you create a pull request - this preview step performs a terraform plan to give you an overview what resources will be deployed or changed.

When you create a release/* branch from main branch, this triggers a deployment to PROD environment.

Deploying the resources to DEV:

Let us make some changes to the Terraform code, push the changes to GitHub repo and create a pull request (PR). Below is how the deployment pipeline looks:

And below are the steps performed as part of preview:

Let us merge our pull request to main branch.

Here is the output of terraform apply step:

Let us verify the resources on Snowflake. As you can see, the deployment pipeline created a database(TASTY_BYTES_DEV) and schema(RAW_POS) and a table (MENU)

A new warehouse was also provisioned.

Deploying the resources to PROD:

Let us create a release branch from the main branch. This will trigger a deployment to PROD environment.

As mentioned earlier, there will be an preview step which performs a terraform plan to give you an overview what resources will be deployed or changed.

Since, I have configured environment protection rules, the pipeline stops for a manual approval, before triggering a deploy to PROD.

Approving this will trigger a deploy to PROD.

Here is the output of terraform apply step (for PROD):

Completed pipeline:

Let us verify the resources on Snowflake for PROD environment. As you can see, the deployment pipeline created a database(TASTY_BYTES_PROD) and schema(RAW_POS) and a table (MENU)

A new warehouse for PROD was also provisioned.

Conclusion:

Automating the deployment of Snowflake resources using Terraform and GitHub Actions streamlines the process, reduces the potential for errors, and ensures that infrastructure is managed consistently. This setup not only saves time but also enhances the reliability and reproducibility of deployments. By following the steps outlined in this guide, you can leverage the power of IaC and CI/CD to manage your Snowflake infrastructure efficiently.

Thanks for reading. Please let me know your feedback in comments section.

Harnessing Managed GitHub Action Runners on AWS CodeBuild for Efficient DevOps Workflows

Chandrashekar Y M — Fri, 10 May 2024 12:37:48 +0000

Few weeks back, AWS announced a new feature involving AWS CodeBuild, that allows you to configure self-hosted GitHub action runners in CodeBuild containers to process GitHub Action workflow jobs. This feature allows CodeBuild projects to receive GitHub Actions workflow job events and run them on CodeBuild ephemeral hosts.

What is AWS CodeBuild?
AWS CodeBuild is a robust, managed continuous integration service that automates code compilation, testing, and artifact production without requiring the management of underlying servers.

Traditionally, our approach involved using EC2 Spot Instances with custom AMIs in a scheduled auto-scaling setup to accommodate the fluctuating demands of GitHub Action runners. This method, while effective, often led to bottlenecks due to time-zone variances across our DevOps teams, resulting in delayed CI/CD pipelines.

The introduction of managed GitHub Action runners by AWS offers a promising alternative, integrating seamlessly with AWS services like IAM, Secrets Manager, CloudTrail, and VPC for enhanced security and operational efficiency.

Let us explore this feature step by step, by connecting one of my GitHub repositories to a CodeBuild project, see how it can pick up the queued workflow job and performs the GitHub actions configured in the workflow job. The workflow is quite simple one - it builds a container image based on a Docker file, tags it and pushes this to an existing Amazon Elastic Container Registry (ECR).

Step 1- CodeBuild project:

Let us navigate to CodeBuild console and create a project with name github-action-runners

Select the source provider as GitHub and Connect using OAuth. We can also us PAC (personal access tokens) to connect to GitHub. To keep things simple, let us stick to OAuth:

Authorize aws-codesuite to access your GitHub repositories:

Once the connection is successful, you should be able to select your repository from the list:

For Webhook, let us select Rebuild every time a code change is pushed to this repository and Event type as WORKFLOW_JOB_QUEUED

There are following event types available - PUSH, PULL_REQUEST_CREATED, PULL_REQUEST_UPDATED, PULL_REQUEST_REOPENED, PULL_REQUEST_MERGED, PULL_REQUEST_CLOSED, WORKFLOW_JOB_QUEUED,RELEASED, PRERELEASED.

You can also add additional conditions for "Start build" and "Don't start build", if needed.

Next, let us choose the Compute Environment for the CodeBuild. CodeBuild offers On-demand and Reserved Capacity options to choose from. For image options you can choose CodeBuild managed images or a custom Docker image. In case of custom image option, you can choose an image from your Amazon ECR (in your account or from another account). Or you can choose an image hosted in an external Docker registry.

Let us stick to default values for our use case.

Note that, we can always override these options by using a label in our GitHub Actions workflow file. More on this in later sections.

CodeBuild can create a service role with required permissions for you or you can create and choose your own custom role for the project.

Next is Buildspec yaml file. In our case, Buildspec will be ignored when we use CodeBuild to run GitHub Actions workflow jobs. Instead, CodeBuild will override it to use commands that will setup the self-hosted runner.

Let us go-ahead and create the CodeBuild project. Once the creation is successful, you can see that a webhook has been created on your GitHub repository. Navigate to Settings --> Webhooks section to see this:

Step 2: GitHub Actions Workflow Configuration

Below is the workflow file:

name: HelloWorld app
on:
  repository_dispatch:
    types: [webhook_triggered]
  pull_request:
    branches:
      - main
  push:
    branches:
      - main
env:
  AWS_REGION: 'us-east-1'

jobs:
  build:
    name: Build Docker Image
    runs-on: codebuild-github-action-runner-${{ github.run_id }}-${{ github.run_attempt }}-al2-5.0-small
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set outputs
        id: vars
        run: echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Setup AWS ECR Details
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{env.AWS_REGION}} 

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build, tag, and push image to Amazon ECR
        id: build
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          ECR_REPOSITORY: ${{ secrets.AWS_ECR_REPO }}
          IMAGE_TAG: v1.0.0.${{ steps.vars.outputs.short_sha }}
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
          echo "image_tag=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT
          echo "${{ github.event.action }}"
    outputs:
      image_tag: ${{ steps.build.outputs.image_tag }}

From above, you can see that I have AWS credentials and ECR private repository name, configured as repository secrets and referred in the workflow. Refer to my repository here for Docker file and other scripts.

Note that runs-on label has the value in format

codebuild-<project-name>-${{ github.run_id }}-${{ github.run_attempt }}-<image>-<image-version>-<instance-size>

where

project-name = Name of the CodeBuild project we created in step 1 above.
image-image-version-instance-size = al2-5.0-small, which indicates I am overriding the values configured in Environment section of the CodeBuild project.

Refer to the Supported compute images table in this page for the list of compute images that CodeBuild provides.

When you push any changes or create a PR to the main branch, workflow will be triggered.

The webhook associated with this repository will notify the CodeBuild project, which is now our GitHub actions runner picks up this job, as shown below:

If you navigate to CodeBuild project on AWS console, you can see that CodeBuild run is in progress, executing the GitHub actions:

Also, on GitHub Settings page, you can see an ephemeral GitHub actions runner powered by CodeBuild is in works.

Below is snapshot of the container image that was built and pushed to an existing ECR repository, as part of the GitHub workflow.

Looking Ahead
This integration not only simplifies our operations but also potentially reduces costs and improves performance compared to our previous setups. I plan to further explore and optimize this feature, considering even custom runner images to align with our security standards.

I'm excited to see how this feature evolves and look forward to sharing more insights, including a detailed cost comparison among various runner options.

Please let me know what you think by adding your comments.

Teslamate data logger, AWS and Raspberry PI

Chandrashekar Y M — Mon, 18 Dec 2023 01:02:03 +0000

In this blog post, I want to share how I installed and configured Teslamate - A powerful, self-hosted data logger for my Tesla car on a Raspberry PI running Ubuntu. I will also list other AWS Cloud Options I considered to host this data logger and the challenges I faced.

I own and drive a Tesla Model 3 here in Australia. For your car, Tesla provides an API which can be polled to get the logging data and store it in a local database. This data can be further used to build visualization dashboards and perform any kind data analysis using Grafana.

Teslamate

Before going further, few details on Teslamate. This is an open source software developed by Adrian Kumpf / others and is distributed under MIT License. More details about this project can be found here.

There are many installation options, I will be using advanced option with Traefik, Let's Encrypt & HTTP Basic Auth.

I have a few Raspberry PIs running on my home network, which I use for my learning and experimentation. I had written a blog here, on how I manage them using AWS Systems Manager.

Before ending up using Raspberry PI for hosting Teslmate, with cost being priority, I explored few options on AWS:

AWS EC2 Instance: This was the simplest option. I could have used the EC2 options eligible under AWS Free tier + an Elastic IP address (NOT FREE). However, it would cost me after free tier expires.
Deploy applications on Amazon ECS using Docker Compose - I tried this option using ECS Fargate. However, I ran into issues like ECS Fargate not supporting bind mounts. Also, this solution creates and deploys a CloudFormation stack with Amazon EFS resources which in-turn will add up-to the cost for hosting a simple personal data-logger.
Automated software delivery using Docker Compose and Amazon ECS - I tried this, which is an extension of above solution with continuous delivery using AWS CodePipeline, AWS CodeBuild, and Amazon ECR. It worked fine. However, the stack includes provisioning resources like Load Balancer and CI CD Pipelines which again for me, was an overkill for a simple personal data-logger. Also, major changes to Teslamate open source software is also not too often.

So, I zeroed in on hosting Teslmate on one of my Raspberry PIs (for which i have already payed for) and use my personal domain (hosted on AWS Route53) for accessing the Teslamate Web App and Dashboards.

Steps to install and configure Teslmate on Raspberry PI.

First, let us update the available list of packages and upgrade any out-of-date packages for Ubuntu.

sudo apt-get update && sudo apt-get upgrade

I already have docker and docker-compose installed on my Raspberry PI. You can use the following commands to install the same:

sudo apt install docker-compose

For current user to interact with Docker, we must add it to the “docker” group. You can do the same using below command:

sudo usermod -aG docker $USER

You need to logout and log back in for the changes to take effect.

Next, we create a new directory telsamate and navigate to this folder.

mkdir teslamate

cd ~/teslamate

Follow the steps in teslamate advanced installation guide to create 3 files inside teslamate folder you created above.

docker-compose.yml

.env

.htpasswd

With this advanced installation option, few things to note are:

We use a reverse proxy (Traefik) which terminates HTTPS traffic. Publicly accessible TeslaMate and Grafana sit behind this reverse proxy.

Let's Encrypt certificate is automatically acquired by Traefik.

TeslaMate service is protected by HTTP Basic Authentication.

Grafana is configured to require a login

Now, we can start the stack using:

docker-compose up -d

This will download the latest version of Teslamate along with Grafana, Postgres and Mosquitto.

As mentioned earlier, I have my own personal domain (chandra-ym.com) registered and managed on Amazon Route53. I created an A record on the related hosted zone to direct the traffic for teslmate.chandra-ym.com to the IP address of my Raspberry PI.

When I first logged into https://teslamate.chandra-ym.com/, which is protected by HTTP Basic Authentication, I need to enter the username and password used to generate the token for .htpasswd file created in step 4.
Next, you are presented the login screen, where you can enter the Tesla API credentials, which uses Access Token and Refresh token. I use iOS app Auth for Tesla to generate these tokens, based on my actual Tesla account credentials. These are temporary tokens with expiration time.

Once you login with the tokens, you can see the Teslamate web interface with vehicle summary:

Some of the visualisations built into Teslmate:

Overview:

States:

Projected Range:

Drive Details:

Charging Stats:

Drive Stats:

Drives:

Efficiency:

Charge Details:

Charges:

Conclusion

At this point of time, Raspberry Pi seemed to be the cheapest option to host this data logger. I know there many free and paid apps available out there for the same purpose. I will continue to explore cost effective options on AWS to host Teslamate and come up with related blog posts in future.

Thanks for reading. Please provide any feedback, in the comments section.

DynamoDB import from S3

Chandrashekar Y M — Wed, 24 Aug 2022 07:49:35 +0000

I recently attended AWS ANZ Database Roadshow 2022 at Sydney AWS office. One of the session that I was looking forward to was about DynamoDB. As expected, this session delivered by one of the Solutions Architect was the highlight of this event.

And every year, I look forward to Jeff Bar's blog about Amazon Prime Day and the stats/metrics related to various AWS services which powers the Amazon site to support huge traffic on this day. DynamoDB powers multiple high-traffic Amazon properties and systems including Alexa, the Amazon.com sites, and all Amazon fulfilment centers. Over the course of Prime Day, these sources made trillions of calls to the DynamoDB API. DynamoDB maintained high availability while delivering single-digit millisecond responses and peaking at 105.2 million requests per second.

So I diligently follow all the updates related to DynamoDB and its features. One such feature which was recently announced was DynamoDB import from S3. This is a fully managed feature that doesn’t require writing code or managing infrastructure. I wanted to explore this feature and get some hands on.

Before this feature was announced, there were very limited options available for bulk importing data into DynamoDB. Such pipelines needed building and operating of custom data loaders on a fleet of virtual instances, monitoring and exception handling.

DynamoDB import from S3 helps you to bulk import terabytes of data from Amazon S3 into a new DynamoDB table with no code or servers required. The data in S3 should be in CSV, DynamoDB JSON or ION format with GZIP or ZSTD compression, or no compression. Each record is S3 data should have a Partition Key and a Sort Key (optional) to match the key schema of the target table.

Any errors encountered during parsing of the data or during import, a log entry is created for each error in Amazon CloudWatch Logs. If number of errors exceeds 10000 - the logging stops but the import continues.

Another important thing to note here is, DynamoDB import from S3 feature does not consume any write capacity units. So you don't need to provision additional capacity when creating a new table.

For testing this feature, I downloaded a dataset from Kaggle. This dataset consists of listings of all the movies and tv shows available on Netflix, along with details such as - cast, directors, ratings, release year, duration, etc. and the dataset has close to 9000 records.

Now let us try to use this feature to import the above dataset into a DynamoDB table. I already have an S3 bucket called dynamodb-import-s3-demo and the dataset CSV file is uploaded in the folder path /netflix-shows-movies as shown below:

From the dataset, I will be using the columns title and show_id as Partition Key and Sort Key respectively, for the DynamoDB table. Below is the snapshot of the dataset being used.

Step 1: Next, from the AWS Console, I choose Imports from S3 option under DynamoDB service.

Step 2: Click on Import from S3 button to navigate to Import options.

In the S3 URL, enter the path to the source S3 bucket and the prefix in URI format.
Select** This AWS account** as the bucket owner
Select the remaining fields as shown in the below image and click on Next:

Step 3: On the next screen: Destination table - new table

Table Name - Enter a name for the DynamoDB table.
Partition key - as mentioned above, enter title.
Sort key - as mentioned above, enter show_id
For Table Settings, leave the Default settings selected. DynamoDB table will be created with default RCUs and WCUs. As mentioned earlier, the import process will not consume any of the table's capacity.
Choose Next

Step 4: Review the details and click Import

Step 5: An import job is created. It takes sometime to complete the import. Monitor the status of the job to move to Complete

The dataset has 8808 records and 8807 of those were successfully imported. One record failed to import and the same was logged in CloudWatch Log groups, as shown below:

Below are the records which was imported as DynamoDB table as items.

Common errors that we might encounter can be syntax errors, formatting issues and records which are missing Partition Key and Sort key. Please refer to the Validation errors section in the Developer Guide for more details.

One limitation I see with this feature is that data can only be imported into a new table that will be created during the import process. Already existing DynamoDB tables cannot be used as part of the import process.

Cost wise, DynamoDB import from S3 feature costs much less than normal write costs for loading data manually using custom solutions.

Thanks for reading this blog. Please share your comments and feedback. It helps me to learn and grow.

AWS Systems Manager to manage Raspberry Pi running Ubuntu server

Chandrashekar Y M — Sun, 17 Oct 2021 21:22:17 +0000

As part of this blog, I wanted to share my learnings towards AWS Systems Manager(previously AWS Simple Systems Manager - SSM) and how I configured Systems Manager to manage and perform auto patching on a hybrid environment. The hybrid environment here is a simple Raspberry Pi running one my home network, with Ubuntu server on it.

What is AWS Systems Manager?

AWS Systems Manager gives you visibility and control of your infrastructure on AWS. Systems Manager provides a unified user interface so you can view operational data from multiple AWS services, and then automate operational tasks across your AWS resources. Systems Manager helps you maintain security and compliance by scanning your managed instances and reporting on (or taking corrective action on) any policy violations it detects.

Task

When your environment consists of servers / VMs running on AWS cloud, on-premises data centers and on computers like Raspberry Pi, it will be difficult manage them separately on multiple tools/interfaces. Having a single interface to manage both cloud and non-cloud servers would reduce a lot of admin overhead and streamline the process.

Solution

AWS Systems Manager provides a single interface to manage, administer and get operational insights from servers running in AWS cloud, on-premise / hybrid environments.

SSM Agent

Systems Manager makes use of SSM agent installed on the servers to achieve this. By default SSM agent is pre-installed on instances created using certain AMIs on AWS cloud. For on-premise servers and VMs in hybrid environment, the agent needs to installed and configured manually.

The solution involves following steps:

Create an Hybrid Activation on AWS Systems Manager. This activation also creates an IAM role and grants AssumeRole permission to the SSM service.
Install and configure SSM agent on Raspberry Pi running the Ubuntu server 20.04 to enable SSM service to communicate with the server.
Optional. Setup an Inventory association on AWS Systems Manager to collect information about software and settings for a target set of managed instances.
Configure Patch Manager on AWS Systems Manager to automate patching of the managed instances on a pre-configured schedule.

Raspberry Pi 4 and Ubuntu Server 20.04

Since I am learning Linux administration, I have a Raspberry Pi set up with Ubuntu server 20.04 on my home Wifi for this purpose. If you are interested to set up the same, you can purchase a Raspberry Pi from here and use the step by step instructions here to install Ubuntu server 20.04. I also setup the Ubuntu server to connect to my home Wifi. Since my home internet router has DHCP reservation by default, the Ubuntu server always gets static IP when it connects to the WiFi network. So I need not worrry about setting up a static IP separately.

Solution Step 1. Hybrid Activation on AWS Systems Manager

Login to your AWS Console and Navigate to AWS Systems Manager. Click on Hybrid Activations. And then Create an Activation.

Enter the Activation description and Instance Limit. The Activation also creates an IAM role AmazonEC2RunCommandForManagedInstances which uses IAM policy AmazonSSMManagedInstanceCore and grants AssumeRole permission to SSM service.

Successful creation of Activation provides an Activation Code and Activation ID. Please make a note of these 2 values as these will be used at later step to configure the SSM agent on the server.

Solution Step 2. Install and configure SSM agent on Ubuntu Server

ssh into the Ubuntu server with your credentials and run the following set of commands to install SSM agent

ubuntu@ubuntu:/$ mkdir /tmp/ssm

ubuntu@ubuntu:/$ curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_arm64/amazon-ssm-agent.deb -o /tmp/ssm/amazon-ssm-agent.deb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29.6M  100 29.6M    0     0   675k      0  0:00:44  0:00:44 --:--:-- 1179k

Note: Since Raspberry Pi comes with ARM architecture, you need to use the corresponding version of the SSM agent.

ubuntu@ubuntu:/$ sudo dpkg -i /tmp/ssm/amazon-ssm-agent.deb

Stop the SSM agent and register the agent using the Activation code and Activation ID that you noted down in previous step.

ubuntu@ubuntu:/$ sudo service amazon-ssm-agent stop

ubuntu@ubuntu:/$ sudo amazon-ssm-agent -register -code "ACTIVATION_CODE" -id "ACTIVATION_ID" -region "ap-southeast-2"
Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelog.xml: no such file or directory
Initializing new seelog logger
New Seelog Logger Creation Complete
2021-10-16 22:54:14 WARN Could not read InstanceFingerprint file: InstanceFingerprint does not exist.
2021-10-16 22:54:14 INFO No initial fingerprint detected, generating fingerprint file...
2021-10-16 22:54:15 INFO Successfully registered the instance with AWS SSM using Managed instance-id: mi-001e234567890dd12

Note: You can ignore the warning and error. Make sure you receive a message that your instance/server has been registered with SSM.

Start the SSM agent

ubuntu@ubuntu:/$ sudo service amazon-ssm-agent start

Now you can see the registered instance / server on AWS Systems Manager --> Fleet Manager (which was earlier referred to as Managed instances)

Solution Step 3. [Optional] Setup an Inventory association

AWS Systems Manager Inventory Association enables to collect information about your instances and the software installed on them, helping you to understand your system configurations and installed applications.

From the AWS Systems Manager page, navigate to Inventory section and then Setup Inventory

Leave the default settings.

Once the Inventory association is setup, it runs every 30 mins to gather all the inventory from the managed instances. This makes use of AWS Systems Manager document AWS-GatherSoftwareinventory. You can verify the same from State Manager section.

Once the status changes to Success, you can view more details from the Resources tab in State Manager section.

Information collected about the software and settings on the managed Ubuntu server are displayed in Inventory section.

Solution Step 4. Configure Patch Manager

AWS Systems Manager patch manager helps you select and deploy operating system and software patches automatically across large groups of Amazon EC2 or on-premises instances.

Using patch baselines, you can configure to auto approve a select categories of patches to be installed like OS or high severity patches.

From the AWS Systems Manager page, navigate to Patch Manager section and then Configure patching

Then you can define a maintenance window for patches so that they are only applied during preset times.

As you can see, AWS uses the AWS-UbuntuDefaultPatchBaseline for the patching the Ubuntu servers/instances. This is the default patch baseline for Ubuntu provided by AWS.

System Manger's Maintenance window acts like glue for all the components in Patch Manager.

In the background, Systems Manger use RUN COMMAND to perform the patching task.

Patching task is auto-executed at the preset time and details of the same can be verified in the History section.

Click on the View Output to see the task execution details.

*** Conclusion
Using AWS Systems Manager's Patch Manager feature, I was able to successfully patch my Ubuntu server on my home WiFi and also setup a maintenance window to do the same activity at preset time.

Apart from Patch Manager and Inventory, AWS Systems Manager also provides features like Incident Manager, Parameter Store, Automation, Run Command and OpsCenter which I would like to explore in my future blogs.

Thanks for reading my blog. Please share your comments and feedback.

AWS CloudFormation - Retry Stack Operations

Chandrashekar Y M — Sun, 05 Sep 2021 11:34:32 +0000

As I continue my study for AWS Certified Solutions Architect Professional (SAP-C01) certification exam, I was practising the use of AWS CloudFormation service which provides an easy way to to model a collection of AWS and 3rd party resources, provision them quickly and consistently, and manage them through their lifecycles.

As part of CloudFormation service, we create a template that describes all the AWS resources that we need to create and manage, upload the template, and CloudFormation service takes care of provisioning and configuring the resources and their dependencies as a stack.

During the provisioning process of a stack, it might fail for different reasons like errors in template, typos or invalid values specified for the parameters and also due to issues outside the template like IAM permission errors. When such errors occur, CloudFormation rolls back the stack to previous stable state. If this error was part of stack creation, then CloudFormation deletes all the resources that it created up-to the point of the error. This roll back process can take a lot of time depending on the complexity of the template, number of resources and their dependencies involved.

On 30-Aug-2021, AWS announced a new CloudFormation feature which allows us to disable the automatic rollback, keep the resources which were successfully created or updated before the error occurs, and retry the stack operations from the point of failure. Details of this new feature can be read here. This new feature helps save a lot of time by allowing us to fix the errors and retry the creation or update of the stack.

I wanted to explore this new feature by trying out a sample CloudFormation template that AWS provides in their documentation. This blog is about my learning and hands-on experience of this new feature. The sample CloudFormation template I used - installs and deploys a WordPress onto a single EC2 instance with a local MySQL database for storage. The template can be downloaded here.

Since I wanted the stack creation to fail in order to test the new feature, I edited the CloudFormation template and set the default value for EC2 instance type to invalid value of "t22.small".

I want to create a stack from this template. On the CloudFormation console, I uploaded the edited template.

Then, I entered the name of the stack and fill the parameter values. One of the parameter in the template is to choose the web server EC2 instance type. Since I had defaulted this value to "t22.small", the same was set, as shown below.

Now, on the next screen in Stack failure options, I see a new option to select Preserve successfully provisioned resources to keep the resources, in case of errors, the resources that have already been created. Failed resources are always rolled back to the last known stable state.

I then review the chosen configurations and click Create stack button.

The stack creation process starts and after few mins it fails because of an error. The creation of the WebServer EC2 instance failed as I had selected an invalid EC2 instance type. The details can be viewed on Events tab.

Since I chose the configuration to preserve the provisioned resources, the WebServerSecurityGroup that got created before the error are not rolled back and still present. On the Resources tab, you can see it's status to be CREATE_COMPLETE. While the status of WebServer is in the CREATE_FAILED status.

The rollback is paused and I get the following options to proceed:

Retry – To retry the stack operation without any change. This option is useful if a resource failed to provision due to an issue outside the template. I can fix the issue and then retry from the point of failure.
Update – To update the template or the parameters before retrying the stack creation. The stack update starts from where the last operation was interrupted by an error.
Rollback – To roll back to the last known stable state. This is similar to default CloudFormation behaviour.

Fixing the error

Since I know what caused the error (selecting invalid instance type for parameter InstanceType), I choose Update.

I don't need to upload a modified template to fix this. In Parameters, I choose a valid InstanceType to fix the error.

In the Change set preview, the update will modify the EC2 Instance which is in CREATE_FAILED state and tries to provision the EC2 Instance with the updated Instance type. Then, I choose Update stack option.

This time, the creation of stack was successful with status UPDATE_COMPLETE. Below is the screenshots of Events and Resources tabs.

The expected output for the template that was used was a WordPress Website, which can be seen on Outputs tab.

The template I chose for my learning was quite simple and the new feature of retrying stack operations from the point of failure still saved me time. With more complex templates, I can imagine the amount of time that can be saved with this new capability of retrying stack operations.

Thanks for reading the blog post. I wanted to write down my understanding of this new feature of CloudFormation. I am sure many AWS developers would appreciate this really helpful feature.

Please point out any mistakes and provide your feedback.

AWS Lambda Layers

Chandrashekar Y M — Sun, 22 Aug 2021 04:36:51 +0000

As part of my preparation towards AWS Certified Solutions Architect Professional (SAP-C01) certification exam, I am currently studying the details of AWS Lambda included in Serverless section of the exam blue print. Exam blue print or exam guide can be found here.

I want to share my learnings on Lambda layers as part of this blog. As you might be aware, AWS Lambda is a compute service that lets us to run our code without provisioning any virtual machines or compute infrastructure. It is a Function-as-a-Service offering from AWS. Lambda functions uses a runtime (example: Python) and runs in a runtime environment when it is invoked. As a service, you are billed for the duration that the functions runs.

When a Lambda function is created, the code is packaged as a deployment package into .zip file. As you add libraries and other dependencies - which is referred by the Lambda function, creating and uploading the deployment package can slow down the development and execution.

In November 2018, AWS introduced Lambda Layers. A Lambda layer is a .zip file archive that contains additional code, data, libraries, custom runtime and configuration files. The .zip file archive can be loaded to Lambda layer from an S3 bucket or from your local machine. Any layer which is used are actually extracted into /opt folder inside the runtime environment. Use of Lambda Layers promote code sharing and separation of responsibilities which enables faster development.

Lambda Layers offers following advantages over not using them in first place:

Layers allow us to use new runtimes which are not currently supported by default with Lambda.
Layers also allows libraries to be externalised which means contained in a separate package, enforcing separation of concerns if any, between dependencies and actual function code.
Enables faster deployments, because code that needs to be packaged and uploaded is smaller.
Layers can be reused by other Lambda functions within an AWS account and shared between AWS accounts or shared publicly with developer communities.

I also tested the example provided by AWS to see how layers work in real. Related AWS blog can be found here.

I created a Lambda function using the following piece of Python code with Python 3.8 as runtime.

import numpy as np
from scipy.spatial import ConvexHull

def lambda_handler(event, context):

    print("\nUsing NumPy\n")

    print("random matrix_a =")
    matrix_a = np.random.randint(10, size=(4, 4))
    print(matrix_a)

    print("random matrix_b =")
    matrix_b = np.random.randint(10, size=(4, 4))
    print(matrix_b)

    print("matrix_a * matrix_b = ")
    print(matrix_a.dot(matrix_b))
    print("\nUsing SciPy\n")

    num_points = 10
    print(num_points, "random points:")
    points = np.random.rand(num_points, 2)
    for i, point in enumerate(points):
        print(i, '->', point)

    hull = ConvexHull(points)
    print("The smallest convex set containing all",
        num_points, "points has", len(hull.simplices),
        "sides,\nconnecting points:")
    for simplex in hull.simplices:
        print(simplex[0], '<->', simplex[1])

Screenshot of the Lambda function:

After I deployed my Lambda function and tested the same with invoking it, I received the following error:

The error was because there was no module named NumPy. We haven't included their use in the Lambda function code. The default runtime we chose Python 3.8 does not include the NumPy module. To resolve the error, we either have to create a deployment package including this module and upload to Lambda OR use Lambda Layers.

To use Layers, I create a Layer from the corresponding section of the Lambda function:

AWS provides following 3 options to configure a layer. I choose to use AWS provided layers.

I selected the layer AWSLambda-Python38-ScyPy1x which provides the required module NumPy and also the SciPy module which can be used for advanced spatial algorithms.

Now, again I invoked the Lambda function. This time, the Lambda function was executed successfully by performing a matrix multiplication.

To conclude, I used a Lambda Layer to provide the required modules for the Lambda function. These modules were not uploaded manually as we used the externalized set of libraries provided by AWS.

Thanks for reading this blog. I wanted to write down my understanding of the Lambda Layers feature.

Please point out any mistakes and provide your feedback.

Cloud Resume Challenge - My Journey

Chandrashekar Y M — Mon, 04 Jan 2021 07:49:47 +0000

My Story

I'm a Systems Administrator and Configuration Management specialist as part of my day job. I have a minimum experience with coding/scripting. But I do know, how to read/understand a piece of code (JavaScript, Python) and what it is doing.

As part of my journey through learning AWS and getting certified, I came across Cloud Resume Challenge floated online (Linked/Reddit) by Forrest Brazeal (Cloud Architect, AWS Serverless Hero). Details of this challenge can be found here.

I know I am too late to join this challenge (last date set by Forrest Brazeal was 31-July-2020 to get the code review) due to my preparation towards AWS certifications, but I still wanted to take it up and give it a try. I joined the Discord channel run by Forrest, to see how others have completed this challenge and take some help from that wonderful community. I would like to thank Chris Nagy, as I referred his blog and git hub repos, whenever I needed help.

Now, the Challenge

AWS Certification

Being completely new to AWS world, I achieved AWS Certified Cloud Practitioner Certification in Feb 2019. Then I went on to achieve AWS Certified Solutions Architect Associate (March 2020), AWS Certified SysOps Administrator (July 2020), AWS Certified Developer Associate (Oct 2020), Currently, I am preparing for AWS Certified Solution Architect Professional and would like to ace asap.

Front end- HTML / CSS

I chose a template from Colorlib and modified it by removing additional pages and links. I tried to keep it as minimalistic as possible. HTML/CSS experience from early days of my career + google helped with editing the template. The HTML page would also include a JavaScript snippet which will update and fetch the visitor count from the "back end".

Static S3 Website

Based on the knowledge I acquired during preparation of AWS certifications, hosting a static website on S3 and using CloudFront for content distribution was easy. I purchased a domain name using AWS Route 53 and configured it to use the CloudFront distribution. Also, I made use of AWS Certification Manager(ACM) to procure an SSL certificate for the site.

Back-end

Backend infrastructure and logic was needed to update and retrieve the visitor count from a database table. This involved use of AWS resources like API Gateway, Lambda and DynamoDB.

DynamoDB: DynamoDB is AWS's fast and flexible NoSQL Database service offering for any scale. I created a simple DynamoDB table with one item to store and update the visitor count. Atomic Counter feature of DynamoDB comes in handy here. Atomic Counter is a numeric attribute that is incremented, unconditionally, without interfering with other write requests. With an atomic counter, the updates are not idempotent. In other words, the numeric value increments each time you call UpdateItem operation. This operation is implemented in the Lambda function (more details below).
Lambda: AWS Lambda is a compute service that lets you run code without provisioning or managing servers. I created a python based Lambda function, which queries the DynamoDB table and updates the visitor count item. As mentioned earlier, I utilised the update_item operation on the DynamoDB table to increment the numeric value. Since, I don't have Python scripting experience, I referred to the Lambda function code from git hub repos of Chris Nagy and Bansi Mendapara.
API Gateway: Amazon API Gateway provides the option to create and manage APIs to back-end systems running on Amazon EC2, AWS Lambda, or any publicly addressable web service. In our case, the API Gateway exposes a REST API Endpoint, which will be called by Javascript snippet embedded in the front-end html page, on every page visit/refresh - to update and fetch the visitor count from a DynamoDB table, through Lambda function. Enabling CORS (Cross-Origin Resource Sharing) on the API Gateway resource is mandatory to fetch the response back, when it is called.

Back end - Infrastructure as Code

I initially created back-end components like DynamoDB, Lambda and API Gateway separately using AWS Console and configured them to work together to update and provide the visitor count for the front end html. But the requirement was to make use of AWS SAM (Serverless Application Model) template to define these back end resources as Infrastructure as Code (IaC) and deploy them using SAM commands. I did have basic understating of SAM while preparing for AWS Certified Developer certification. Again, this blog post from Chris Nagy helped me to better understand use of SAM for serverless applications build and deployment. Next step, as an improvement, I want to create and deploy even the front-end sources (S3 bucket, CloudFront Distribution, Route53 records configuration) as a SAM template (IaC).

Here is designer view of the CloudFormation template built and deployed by SAM:

Front end - CI/CD

Another requirement is to use GitHub repositories to store the front-end / back-end code and make use of GitHub Actions to achieve continuous integration and deployment, or CI/CD. I never used GitHub Actions earlier, so it was new learning for me. GitHub Actions simplifies and automates many steps in creating/updating and deployment of the resources. For front-end CI/CD, I used GitHub Actions to configure AWS credentials, deploy the changes to S3 bucket (which stores html/css/js/images content) and then invalidating the CloudFront distribution. GitHub secrets were used to securely store environment variables like AWS Login Keys.

Back end - CI/CD

A separate GitHub repo was created to store the back-end code, which included the Lambda function, SAM Template and Python tests. Corresponding GitHub action is created to configure AWS credentials, run python tests, SAM build and SAM deploy commands. Again, GitHub secrets were used to securely store environment variables like AWS Login Keys.

Here is the final (not really!!) version of my one page resume: https://www.chandraym.com/