DEV Community: Thakur Rishabh Singh

AWS Step Functions workflow for an ETL Job on COVID-19 and deploying it with Terraform (#CloudGuruChallenge Series) (Part 2/3)

Thakur Rishabh Singh — Mon, 26 Apr 2021 03:00:45 +0000

This Post is about creating an AWS Step Functions Workflow on AWS to process an ETL Job on daily COVID-19 count and deploying the infrastructure with Terraform

Project Overview
Architecture
Step Functions Workflow
- What is AWS Step Functions?
- Creating Step Functions Workflow
Deploying Infrastructure with Terraform
Conclusion

1. Project Overview

This project is Second part in the series #CloudGuruChallenge – Event-Driven Python on AWS. Here we deploy an AWS Step Functions Workflow along with various other components required for error handling. The AWS Step Functions Workflow will process an ETL job on daily COVID-19 count which will be demonstrated in the next/final part of this series. Note:- Appropriate permissions must be configured in IAM for the below code to work.
To automate the process Terraform is used for IaC (Infrastructure as Code) and AWS CodePipeline is used for CI/CD. The details about setting up Terraform and CodePipeline has been discussed in detail in Part-1 of the series.

2. Architecture

The above architecture works as follows:

An event bridge rule triggers the AWS Step Functions Workflow once daily at 1pm.
The Step Functions Workflow upon success sends an Email to the owner through Amazon SNS.
Upon Failure the another event is triggered by the event bridge which takes the failed event and sends it to SNS, SQS and cloudwatch logs. A notification of this is sent to the owner by Email through SNS.
An event bridge rule triggers a lambda function once daily at 7am. This function retrieves the failed events from SQS. It then triggers the step functions workflow with an input containing all the failed dates.

3. Step Functions Workflow

What is AWS Step Functions?

It is a serverless orchestration service where different serverless services can be made to interact based on events while exchanging data in JSON format.

Creating Step Functions Workflow

The above workflow works as follows:

Each rectangular block represents a lambda function task except for CheckETLStatus and Notify which are Choice state and SNS task respectively.
The workflow starts with retrieving the ETL status which can be either initial load or update.
If it is initial load, all the data is retrieved as a batch from the source.
If it is update, only a row of data is retrieved from the source (which is the case count for the current day).
The retrieved data is transformed and loaded through the transform and load tasks respectively.
Finally an Email is sent to the owner notifying him about the success of the ETL job.

To Create the above workflow the code is given below.

{
  "StartAt": "GetETLStatus",
  "States": {
    "GetETLStatus": {
        "Type": "Task",
        "Resource": "YOUR-LAMBDA-ARN",
        "Next": "CheckETLStatus",
        "TimeoutSeconds": 3,
        "ResultPath": "$.result"
    },
    "CheckETLStatus": { 
        "Type": "Choice",
        "Choices": [
          {
            "Variable": "$.result.status",
            "StringEquals": "InitialLoad",
            "Next": "InitialLoad"
          },
          {
            "Variable": "$.result.status",
            "StringEquals": "Update",
            "Next": "Update"
          }
        ],
        "Default": "InitialLoad"
     },
    "InitialLoad": {
        "Type": "Task",
        "Resource": "YOUR-LAMBDA-ARN",
        "Next": "Transform",
        "TimeoutSeconds": 3,
        "ResultPath": "$.result"
    },
    "Update": {
        "Type": "Task",
        "Resource": "YOUR-LAMBDA-ARN",
        "Next": "Transform",
        "TimeoutSeconds": 3,
        "ResultPath": "$.result"
    },
    "Transform": {
        "Type": "Task",
        "Resource": "YOUR-LAMBDA-ARN",
        "Next": "Load",
        "TimeoutSeconds": 3,
        "ResultPath": "$.result"
    },
    "Load": {
        "Type": "Task",
        "Resource": "YOUR-LAMBDA-ARN",
        "TimeoutSeconds": 3,
        "ResultPath": "$.result",
        "Next": "Notify"
    },
    "Notify": {
        "Type": "Task",
        "Resource": "arn:aws:states:::sns:publish",
        "Parameters": {
          "TopicArn": "YOUR-TOPIC-ARN",
          "Message.$": "$"
        },
        "End": true
    }
  }
}

4. Deploying Infrastructure with Terraform

The code to deploy the above infrastructure with terraform is shown below. For more details about how to set up terraform and code pipeline visit Part-1 of the series. I have omitted the code for the other two scheduled events as I would be showing them in next part of this series.

state_machine.tf

resource "aws_sfn_state_machine" "sfn_state_machine" {
  name     = "YOUR-STATE-MACHINE-NAME"
  role_arn = "YOUR-ROLE-ARN"

  definition = <<EOF
{
  "StartAt": "GetETLStatus",
  "States": {
    "GetETLStatus": {
      "Type": "Task",
      "Resource": "YOUR-LAMBDA-ARN",
      "Next": "CheckETLStatus",
      "TimeoutSeconds": 3,
      "ResultPath": "$.result"
    },
    "CheckETLStatus": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.result.status",
          "StringEquals": "InitialLoad",
          "Next": "InitialLoad"
        },
        {
          "Variable": "$.result.status",
          "StringEquals": "Update",
          "Next": "Update"
        }
      ],
      "Default": "InitialLoad"
    },
    "InitialLoad": {
      "Type": "Task",
      "Resource": "YOUR-LAMBDA-ARN",
      "Next": "Transform",
      "TimeoutSeconds": 3,
      "ResultPath": "$.result"
    },
    "Update": {
      "Type": "Task",
      "Resource": "YOUR-LAMBDA-ARN",
      "Next": "Transform",
      "TimeoutSeconds": 3,
      "ResultPath": "$.result"
    },
    "Transform": {
      "Type": "Task",
      "Resource": "YOUR-LAMBDA-ARN",
      "Next": "Load",
      "TimeoutSeconds": 3,
      "ResultPath": "$.result"
    },
    "Load": {
      "Type": "Task",
      "Resource": "YOUR-LAMBDA-ARN",
      "TimeoutSeconds": 3,
      "ResultPath": "$.result",
      "Next": "Notify"
    },
    "Notify": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "YOUR-SNS-TOPIC-ARN",
        "Message.$": "$"
      },
      "End": true
    }
  }
}
EOF

depends_on = [
  aws_sns_topic.ETLJobStatus,
  aws_lambda_function.state_machine_lambdas
]
}

sqs.tf

resource "aws_sqs_queue" "Events_DLQ" {
  name = "Events_DLQ_T"
}

sns.tf

resource "aws_sns_topic" "ETLJobStatus" {
  name = "ETLJobStatus_T"
}

resource "aws_sns_topic" "ETLErrorMessages" {
  name = "ETLErrorMessages_T"
}

resource "aws_sns_topic_subscription" "ETLJobStatus_target" {
  topic_arn = aws_sns_topic.ETLJobStatus.arn
  protocol  = "email"
  endpoint  = "YOUR-EMAIL-ID"

  depends_on = [
    aws_sns_topic.ETLJobStatus
  ]
}

resource "aws_sns_topic_subscription" "ETLErrorMessages_target" {
  topic_arn = aws_sns_topic.ETLErrorMessages.arn
  protocol  = "email"
  endpoint  = "YOUR-EMAIL-ID"

  depends_on = [
    aws_sns_topic.ETLErrorMessages
  ]
}

cloudwatch.tf

#EventBridge Events

resource "aws_cloudwatch_event_rule" "state_machine_events_failed" {
  name        = "state_machine_events_failed_t"
  description = "This event is triggered when the state machine fails."

  event_pattern = <<EOF
{
  "source": ["aws.states"],
  "detail-type": ["Step Functions Execution Status Change"],
  "detail": {
    "status": ["FAILED"],
    "stateMachineArn": ["${aws_sfn_state_machine.sfn_state_machine.arn}"]
  }
}
EOF

depends_on = [
  aws_sfn_state_machine.sfn_state_machine
]
}

#EventBridge Event Targets

resource "aws_cloudwatch_event_target" "sns" {
  rule      = aws_cloudwatch_event_rule.state_machine_events_failed.name
  target_id = "SendToSNS"
  arn       = aws_sns_topic.ETLErrorMessages.arn

  depends_on = [
    aws_cloudwatch_event_rule.state_machine_events_failed,
    aws_sns_topic.ETLErrorMessages
  ]
}

resource "aws_cloudwatch_event_target" "sqs" {
  rule      = aws_cloudwatch_event_rule.state_machine_events_failed.name
  target_id = "SendToSQS"
  arn       = aws_sqs_queue.Events_DLQ.arn

  depends_on = [
    aws_cloudwatch_event_rule.state_machine_events_failed,
    aws_sqs_queue.Events_DLQ
  ]
}

resource "aws_cloudwatch_event_target" "cloudwatch_logs" {
  rule      = aws_cloudwatch_event_rule.state_machine_events_failed.name
  target_id = "SendToCloudwatchLogs"
  arn       = aws_cloudwatch_log_group.log_group.arn

  depends_on = [
    aws_cloudwatch_event_rule.state_machine_events_failed,
    aws_cloudwatch_log_group.log_group
  ]
}

#Cloudwatch Log Group

resource "aws_cloudwatch_log_group" "log_group" {
  name = "state_machine_events_failed_t"
}

5. Conclusion

In this post we have seen how to build step functions workflow for an ETL job and deploy it with terraform. I know that AWS Glue could be a better approach but still I wanted to explore step functions and this challenge was modified to accommodate it. In the next/final part of this series I'll be combining everything to complete the #CloudGuruChallenge – Event-Driven Python on AWS. I'll be performing ETL job on daily COVID-19 cases and display it in AWS Quicksight.

Deploying Infrastructure on AWS with Terraform and AWS CodePipeline (#CloudGuruChallenge Series) (Part 1/3)

Thakur Rishabh Singh — Sun, 11 Apr 2021 03:15:57 +0000

This Post is about creating a CI/CD pipeline on AWS using CodePipeline which deploys Infrastructure on AWS using Terraform.

Project Overview
Setting up Terraform
Setting up AWS CodePipeline
- Source stage
- Terraform Plan step
- Manual Approval step
- Terraform Apply stage
- Deploy stage
Final View Of the Pipeline
Conclusion

1. Project Overview

This project is First part in the series #CloudGuruChallenge – Event-Driven Python on AWS. Here we deploy an s3 buckets and a lambda function. The lambda function will be part of an AWS Step Functions Workflow which will be developed in the next part of this series and the S3 bucket is used to store the lambda deployment.
To automate the process Terraform is used for IaC (Infrastructure as Code) and AWS CodePipeline is used for CI/CD.

2. Setting up Terraform

The following are the required steps to start working with Terraform on AWS:

Create an S3 Bucket which will store the terraform state file.
Create a dynamodb table with on demand capacity with a primary key of LockID.

The above steps will configure terraform with S3 as the backend. The provider.tf and backends.tf file is shown below.

provider.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "us-east-1"
}

backends.tf

terraform {
  backend "s3" {
    bucket = "YOUR-BUCKET-NAME"
    key    = "terraform.tfstate"
    region = "YOUR-REGION-NAME"
    dynamodb_table = "terraform-state-lock"

  }
}

Create a sample lambda_function.py and zip it in the same directory as the .tf files with name lambda_function_payload.zip. After which the IaC for S3 and lambda could be written as shown below:

S3.tf

resource "aws_s3_bucket" "lambda_s3_buckets" {
    bucket = "YOUR-BUCKET-NAME"
    acl    = "private"
    force_destroy = true

}

resource "aws_s3_bucket_object" "object" {
    bucket = "YOUR-BUCKET-NAME"
    key    = "YOUR-PATH-TO-STORE-LAMBDA-DEPLOYMENT"
    source = "lambda_function_payload.zip"

    depends_on = [
    aws_s3_bucket.lambda_s3_buckets,
    ]
}

Lambda.tf

resource "aws_lambda_function" "state_machine_lambdas" {
    function_name = "YOUR-FUNCTION-NAME"
    role          = "ROLE-ARN"
    handler       = "lambda_function.lambda_handler" 
    s3_bucket = "BUCKET-NAME_WITH-LAMBDA-DEPLOYMENT"
    s3_key    = "PATH-TO-LAMBDA-DEPLOYMENT"

    runtime = "python3.8"
    depends_on = [
    aws_s3_bucket_object.object,
    ]
}

Store all the .tf files in a folder named terraform.

3. Setting up AWS CodePipeline

The AWS CodePipeline will be used for CI/CD (Continuous Integration/Continuous Delivery). The AWS free tier allows 1 free pipeline per month. Our pipeline consists of five stages viz source ,build (Terraform Plan), build (Manual Approval step), Terraform Apply and Deploy.

Source Stage

Here Github is used as a source repository for the pipeline. The configuration for this stage is shown below.

Stage 1: Source Configuration

Github version 2 is selected as the source code repository. You can choose others such as AWS codecommit etc. For github you need to connect to it which is a straight forward process through the console.
The repository name and branch name must be set up which will trigger the pipeline whenever there is a code change in that specific branch in the repository.

Build Stage(Terraform Plan Step)

This stage consists of two builds, a manual approval step and a deploy step. The first build is for terraform plan. AWS Code Build is used for creating the build projects. To set this up, in the pipeline stages add a new action group under the build stage as shown below

Adding a build action to CodePipeline

The project must be AWS Code Build Project which can be created by clicking on create project which will open the wizard as shown below.

CodeBuild Project For Terraform Plan

The important configuration above is

selecting 3gb memory with 2cpus (which is included in free tier)
The service role arn (which gives terraform the permission to provision AWS resources). It must contain all permissions which terraform needs such as access to S3, lambda etc.
The env variable TF_COMMAND_P which will be used in buildspec file.
The path to buildspec.yml file which contains the build commands.

After Configuring the above the git repository it must contain a buildspec file which contains the necessary commands. It is shown below

Terraform_Plan.yml

version: 0.1

phases:

  install:
    commands:
      - "apt install unzip -y"
      - "wget 
https://releases.hashicorp.com/terraform/0.14.10/terraform_0.14.10_linux_amd64.zip"
      - "unzip terraform_0.14.10_linux_amd64.zip"
      - "mv terraform /usr/local/bin/"
  pre_build:
    commands:
      - terraform -chdir=Terraform init -input=false

  build:
    commands:
      - terraform -chdir=Terraform $TF_COMMAND_P -input=false -no-color

  post_build:
    commands:
      - echo terraform $TF_COMMAND_P completed on `date`

Manual Approval Step

Create a manual approval build action after the Terraform plan action as shown below.

Manual Approval Build Stage

You must create an SNS topic with a subscription to your email address to recieve notifications about approval and rejections.

Terraform Apply Stage

Follow a similar process as Terraform plan for Terraform Apply build stage with the following buildspec file.

Terraform_Apply.yml

version: 0.1

phases:

  install:
    commands:
      - "apt install unzip -y"
      - "wget https://releases.hashicorp.com/terraform/0.14.10/terraform_0.14.10_linux_amd64.zip"
      - "unzip terraform_0.14.10_linux_amd64.zip"
      - "mv terraform /usr/local/bin/"
  pre_build:
    commands:
      - terraform -chdir=Terraform init -input=false

  build:
    commands:
      - terraform -chdir=Terraform $TF_COMMAND_A -input=false -auto-approve

  post_build:
    commands:
      - echo terraform $TF_COMMAND_A completed on `date`

Deploy Stage

Now the final stage would be deploy where code build is used to deploy code to lambda via s3 bucket. Create a folder in your repo called lambda_code and store your lambda_function.py in it. Create a new code build project. All the steps for code build project will be the same. The buildspec file for the project is shown below.

deploy_state_machine_code.yml

version: 0.1

phases:

  pre_build:
    commands:
      - mkdir -p ./lambda_code/zipped
      - zip -r -j lambda_code/zipped/lambda-function-payload.zip lambda_code/*

  build:
    commands:
      - aws s3 sync ./lambda_code/zipped s3://YOUR-BUCKET-NAME/YOUR-S3-KEY --delete

      - aws lambda update-function-code --function-name YOUR-FUNCTION-NAME --s3-bucket YOUR-BUCKET-NAME --s3-key YOUR-S3-KEY-TO-FILE

  post_build:
    commands:
      - echo state_machine_code was deployed to lambda from S3 bucket on `date`

4. Final View Of the Pipeline

After the above steps the pipeline should look something like the one shown below.

Final View Of the Pipeline

Pushing to github will trigger the pipeline and the build process can be viewed through the details option on each build action.

5. Conclusion

In this post I have covered how to deploy infrastructure on AWS with terraform through AWS CodePipleine. In the next part of the series I'll show how to set up AWS Step Functions workflow which is triggered through a cloudwatch event. Stay Tuned!

Portfolio/Resume Serverless Website (Cloud Resume Challenge)

Thakur Rishabh Singh — Sat, 20 Mar 2021 23:08:03 +0000

This post is about building a serverless website for your resume and hosting it on AWS.

Disclaimer: If you want to finish the challenge on your own DO NOT read this post. It'll spoil the fun ;)

About the challenge

This website has been done as part of the cloud resume challenge by @forrestbrazeal.

Certification

This is a requirement of the challenge and it was easy for me as I already obtained the AWS Certified Solutions Architect Associate Certification in December 2020 and pursued this challenge to gain hands on experience on the AWS cloud.

AWS Solutions Architect Associate

1. Why Serverless?

The primary reasons are:
1.Cost
2.Scalability
3.Availability
4.Performance

The serverless paradigm offers a way to pay only for what you use. Moreover, the developer need not provision anything beforehand, amazon takes care of that. With low costs it still offers scalability to varying workloads on demand. This makes it very suitable to experiment with cloud technologies out of an enterprise organization and also meet real world challenges. I literally pay 0$ a month for the entire website. My only cost is a custom domain (1.5$/year) and hosted zone on route53 (0.5$/month).

The only downside of it is the infrastructure gets increasingly complex and becomes difficult to manage. However, it is still growing and the future may hold something promising. There is also a possibility that it may succumb to the containerization paradigm (highly debatable).

2. Website Architecture

The above architecture works as follows:

Users request the webpages to the browser.
The browser sends request to Route53 which resolves the DNS to reach nearest cloudfront edge location.
CloudFront forwards the request to S3 bucket which contains the website files and retrieves its contents.
The S3 bucket with frontend code is protected with Origin Access Identity (OAI) which prevents direct access to the bucket.
The javascript code in the frontend sends GET and POST requests to API Gateway to retrieve the number of visitors stored in the database.
The API Gateway forwards the request to lambda as JSON.
Lambda identifies the type of request and performs a get/put operation on DynamoDB to store/retrieve the number of visitors.
The visitor count is then displayed on the website.
A git repository on Github provides code version control and CI/CD through Github actions.
Terraform deploys the AWS infrastructure.

3. Choosing a source control

The version control system used for this project is github. The development workflow consists of two branches master and dev.

4. Choosing a CI/CD mechanism

The continuous integration and continuous delivery (CI/CD) is achieved using github actions. It consists of yml files which are used to automate the build, test and deploy phases of the development process.

5. Infrastructure as code with Terraform

The infrastructure required to host the website on AWS is built using Terraform.

Before getting started it is important to take care of the following aspects:

Security: Terraform needs permission to deploy infrastructure on aws. Therefore, a user is created who has access only to STS (Secure Token Service) and nothing else. A role is created which has the IAM permissions to perform actions on the required AWS services.
Terraform State: A remote backend is configured using an s3 bucket to store the Terraform state file and dynamodb is used to store the state lock. This ensures that the infrastructure declared in the tf files and the actual infrastructure deployed is always the same.

The following infrastructure components are deployed using Terraform:

Private S3 bucket which hosts the frontend code of website.
A new table is created in dynamodb with on demand capacity and a primary key ID. A default item is created to store the value of the number of visitors.
A lambda function with python runtime. The code is retrieved from s3 bucket.
An API Gateway configured as lambda proxy is created which sends GET and POST requests to lambda in the form of JSON and the lambda function also responds in JSON.

6. CI/CD for terraform

The Terraform code must be pushed to dev. Creating a pull request results in the triggering of a github action which generates a plan and posts a comment as shown below

Terraform plan triggered by creating a pull request

Terraform plan displayed as a comment

Merging the pull request to master applies the plan and deploys the infrastructure as shown below.

Terraform apply triggered by merging of pull request

7. Implementing backend with python

The backend code is written in python. This code is deployed to lambda through CI/CD. The backend code uses Boto3 SDK to communicate with DynamoDB. The Lambda function has an IAM Role to perform actions on DynamoDB.

Dealing with CORS: The lambda response contains the header Access-Control-Allow-Origin for * to allow cross origin requests for GET, POST and OPTIONS requests.

8. CI/CD for backend

A github action is configured to do the following when code is pushed to /backend on the master branch:

Run tests on the python code to report bugs.
Zip the python code
Upload the zip file to s3 bucket
Update the lambda function code by retrieving the zip from the S3 bucket.

This process is shown below:

CI/CD for backEnd

9. Building the frontend

The frontend code is built using HTML, bootstrap and Javascript. This code resides on a private S3 bucket acting as origin to cloudfront.

10. CI/CD for frontend

A github action is configured to do the following when code is pushed to /frontend on the master branch:

Upload the code to S3 bucket.
Invalidate the cloudfront cache to get the latest contents from the bucket.

This process is shown below:

CI/CD for frontend

11. Conclusion

It has been an amazing experience completing this challenge. I have learnt a ton from Terraform to CI/CD to AWS. It is definitely worth it for anyone who is serious about a career and cloud. Thanks to @forrestbrazeal for this amazing challenge. As for me I'm on to the next challenge, Stay tuned for another Blog post on a serverless project to automate an ETL pipeline on AWS using python.

DEV Community: Thakur Rishabh Singh

AWS Step Functions workflow for an ETL Job on COVID-19 and deploying it with Terraform (#CloudGuruChallenge Series) (Part 2/3)

This Post is about creating an AWS Step Functions Workflow on AWS to process an ETL Job on daily COVID-19 count and deploying the infrastructure with Terraform

Contents

1. Project Overview

2. Architecture

3. Step Functions Workflow

What is AWS Step Functions?

Creating Step Functions Workflow

4. Deploying Infrastructure with Terraform

5. Conclusion

Deploying Infrastructure on AWS with Terraform and AWS CodePipeline (#CloudGuruChallenge Series) (Part 1/3)

This Post is about creating a CI/CD pipeline on AWS using CodePipeline which deploys Infrastructure on AWS using Terraform.

Contents

1. Project Overview

2. Setting up Terraform

3. Setting up AWS CodePipeline

Source Stage

Build Stage(Terraform Plan Step)

Manual Approval Step

Terraform Apply Stage

Deploy Stage

4. Final View Of the Pipeline

5. Conclusion

Portfolio/Resume Serverless Website (Cloud Resume Challenge)

This post is about building a serverless website for your resume and hosting it on AWS.

Disclaimer: If you want to finish the challenge on your own DO NOT read this post. It'll spoil the fun ;)

About the challenge

Certification

Contents:

1. Why Serverless?

2. Website Architecture

3. Choosing a source control

4. Choosing a CI/CD mechanism

5. Infrastructure as code with Terraform

6. CI/CD for terraform

7. Implementing backend with python

8. CI/CD for backend

9. Building the frontend

10. CI/CD for frontend

11. Conclusion