Create AWS Step Function and Lambda using Terraform and GitLab CI

Shubham Pawar — Thu, 25 Aug 2022 08:09:38 +0000

Install Terraform

Download Terraform binary and unzip the package.
Set terraform path to unzipped binary location. Terraform runs as a single binary named terraform. Any other files in the package can be safely removed and Terraform will still function.
Verify the setup by using command terraform version in CLI.

Install AWS CLI

Download and run the AWS CLI MSI installer for Windows (64-bit):
To confirm the installation, open the Start menu, search for cmd to open a command prompt window, and at the command prompt use the aws --version command.

Build a infrastructure

Create a directory for your configuration mkdir terraform
Change the directory and create a file main.tf.

Initialize the directory with terraform init.

Run terraform plan to see any changes are required for infrastructure.
Format and validate the configuration
Apply the configuration now with the terraform apply command.

Configuring Gitlab CI for Terraform

Create a file on the root of your repository called .gitlab-ci.yml
Get your access key ID and secret access key.
Set aws variables go to - gitlab project → settings → CI/CD → Variables

Sample GitLab pipeline

stages:
  - validate
  - plan
  - apply
  - destroy
image:
  name: hashicorp/terraform:light
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
before_script:
  - export AWS_ACCESS_KEY=${AWS_ACCESS_KEY_ID}
  - export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
  - rm -rf .terraform
  - cd terraform/
  - terraform --version
  - terraform init

validate:
  stage: validate
  script:
    - terraform validate
  only:
    - branches

plan:
  stage: plan
  script:
    - terraform plan -out "planfile"
  dependencies:
    - validate
  artifacts:
    paths:
      - planfile

apply:
  stage: apply
  script:
    - terraform apply -input=false "planfile"
  dependencies:
    - plan
  when: manual
  only:
    - main

destroy:
  stage: destroy
  script:
    - echo "Destroying resources"
    - terraform destroy -state=$STATE --auto-approve
  dependencies:
    - apply
  when: manual
  only:
    refs:
      - main

Sample terraform code for lambda and step function

What is AWS CDK & Why to use it?

Shubham Pawar — Wed, 22 Jun 2022 06:38:03 +0000

What Is AWS CDK?

The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework for defining cloud infrastructure as code (IaC) and provision it through AWS CloudFormation. It was introduced in July 2019.It has all the advantages of CloudFormation and IaC can be provisioned using modern programming languages like Typescript, C# (C-Sharp), Java, and Python.

Why to use CDK

In AWS CDK reusable components can be designed that meet organizational security policy, compliance needs and other requirements, also as any other software library this can be shared easily around the organization, enabling bootstrapping of new projects with best practices by default.

CDK use a template file to create and delete a collection of resources together as a single unit (a stack).
Use the AWS CDK to define your cloud resource in a familiar programing language which CDK supports.
It provides high-level components that preconfigure cloud resources with proven defaults, helping you build on AWS without needing to be an expert
You can use your favorite IDE and can take the advantage of existing productivity tools and testing frameworks.

The two main building blocks of AWS CDK

Stacks
Stacks are basic units of deployment in AWS CDK that are composable. Single or multiple stacks can be used in a project. A stack can share values by consuming resources from another stack. Behind the scenes, CDK uses the Cloudformation import value to pass around different Cloudformation stacks.

Constructs
Constructs are the basic building blocks of AWS CDK apps. A construct represents a "cloud component" and encapsulates everything AWS Cloudformation needs to create the component. A construct can be an Amazon Simple Storage Service (S3) bucket, or any other related AWS resources. AWS CDK have a collection of constructs which is called AWS construct library. Constructs code can be reused throughout infrastructure project.

Setting up CDK-

Install or update the AWS CDK CLI from npm (requires Node.js ≥ 14.15.0)
```
npm i -g aws-cdk
```

Initialize a project:

mkdir CDK_Demo
cd CDK_Demo
cdk init app --language=python

This will create a sample project.

Use cdk ls / cdk list to verify if stack is created also see a list of the IDs of the stacks in your AWS CDK application.

Directory structure of app will be like this –

Overview of created files –

.venv - The python virtual environment created in the process.
cdk_demo_stack.py— A custom CDK stack construct for use in your CDK application.
tests — Contains all tests.
- unit — Contains unit tests.
- test_cdk_demo_stack.py- Test cases for the constructs created in the cdk package. This is mainly to demonstrate how tests can be hooked up to the project.
app.py — The entry point for the application.
cdk.json — A configuration file for CDK that defines what executable CDK should run to generate the CDK construct tree.
README.md — The introductory README for this project.
requirements.txt/requirements-dev.txt —This file is used by pip to install all of the dependencies for your application.
source.bat — To activate the virtual environment
The cdk init command also initializes the project as a Git repository, along with the .gitignore file.

You can find a working example at cdk-demo,follow the README for further steps.

Block diagram for cdk-demo app.

This example uses AWS services like S3 bucket, lambda function and DynamoDB, whenever objects are dumped in s3 bucket it triggers lambda function and writes the file metadata to DynamoDB table.

Some CDK commands to interact with project.

cdk deploy: It deploys the stack using AWS CloudFormation to the configured account.
cdk synth: This generates a cdk.out file, containing a YAML-formatted template, with the resources defined in the stack converted to the equivalent AWS CloudFormation template.
cdk diff: compares your app with the deployed stack.
cdk destroy:Destroy the stack(s) name provided
cdk --help:You can see general help about the utility and a list of the provided subcommands.

For a detailed walkthrough, see the tutorial in the AWS CDK Developer Guide.

Drawbacks/limitations of using AWS CDK:

Due to large number of constructs available it becomes messy to pick the right one as per requirement.
As mentioned earlier a programming language is used to create CDK Code so a standard must be maintained. Without standardization the code can become very complex and challenging to manage in future implementation.

CloudFormation vs CDK:

CDK allows us to use loops and condition statements which reduces the code size as compared to CloudFormation.
In CDK we can use our familiar language instead of using YAML /JSON.
Reusable code can be created and tested locally in CDK.
CDK is easy to integrate with our CI/CD process Nothing is the best, it depends on your requirement, team, project and client demand.

Conclusion :

AWS CDK is a code-first approach to defining cloud application infrastructure. In this post I described the advantages of CDK over Cloudformation and why one should use it also demonstrated how cloud applications can be created using CDK python libraries.

Go clean as you code 😊

Disclaimer: This is a personal [blog, post, statement, opinion]. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.

DEV Community: Shubham Pawar