DEV Community: Sandeep Yaramchitti

Building a Unified Notification System with GitHub Actions

Sandeep Yaramchitti — Sun, 06 Aug 2023 20:26:12 +0000

Explore the power of GitHub actions actions by integrating with multiple notification services including Slack, Chime, Teams, AWS SNS and more..

Introduction

In this blog, we will explore the process of building a GitHub actions for notifications services to publish actions workflow execution results for Slack, Discord, Teams, Chime, SNS. Also, I will share the steps to publish this to Github Actions market place.

You can find this action available in Github Actions Market place

Uninotify Actions Marketplace: https://github.com/marketplace/actions/uninotify

GitHub Repo: https://github.com/OSCloudysky/UniNotify

The Magic of GitHub Actions

GitHub Actions is an API for cause and effect on GitHub. You can orchestrate any workflow, based on any event, while GitHub manages the execution, provides rich feedback, and secures every step along the way.

The Power of Unified Notifications

The idea behind this implementation is to ability to use an action in any actions workflow to publish execution details to several notification services as shown in the diagram. Notification plays a big role in the CI/CD process and this allows teams to focus on building their workflow efficiently and not worry about integrations to their notification services.

Diving into the Implementation

The first step was to set up GitHub Actions repo and I have used the following template to bootstrap my github actions. This is highly recommended if you are planning to build custom actions using Typescript.

Actions Typescript github template: https://github.com/actions/typescript-action

Please find the repo link for my custom action to publish messages to multiple notification services. I have created a custom action in TypeScript that utilizes Octokit, a GitHub REST API client, to fetch details about the workflow, commit, and repository etc. UniNotify action is responsible to format the message including details such as workflow name, event name, run URL, commit messages etc. I have included support to send default messages and as part of next release, you will have the ability to send custom messages as well.

UniNotify repo : https://github.com/OSCloudysky/UniNotify

Now, let’s dive into how this works and here is the entry point for the Action in main.ts file. As you can see here, it leverages Octokit, GitHub REST API Client. to fetch details about workflow, commit, and repo etc.

Then, it basically calls respective services which I have created as each component for better code management and reusable code.

import * as core from '@actions/core'
import * as github from '@actions/github'
import * as service from './services'
import {Octokit} from '@octokit/rest'
import fetch from 'node-fetch'

async function run(): Promise<void> {
  const messageType = core.getInput('messageType')
  const githubToken = core.getInput('githubToken')

  // Get more details information of the job
  const context = github.context
  const octokit = new Octokit({auth: `token ${githubToken}`, request: {fetch}})

  const commit = await octokit.repos.getCommit({
    owner: context.repo.owner,
    repo: context.repo.repo,
    ref: context.sha
  })

  const runUrl = `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`

  const message = `
    Workflow Name: ${context.workflow}
    Event: ${context.eventName}
    Run URL: ${runUrl}
    Triggered By: ${context.actor}
    Commit URL: https://github.com/${context.repo.owner}/${context.repo.repo}/commit/${context.sha}
    Commit Message: ${commit.data.commit.message}
  `

  try {
    const token = core.getInput('slackToken')
    if (messageType === 'slack') {
      await service.sendSlackMessage(token, context, commit)
    } else if (messageType === 'discord') {
      const webhookUrl = core.getInput('discordWebhookUrl')
      await service.sendDiscordMessage(webhookUrl, context, commit)
    } else if (messageType === 'chime') {
      const webhookUrl = core.getInput('chimeWebhookUrl')
      await service.sendChimeMessage(webhookUrl, context, commit)
    } else if (messageType === 'teams') {
      const webhookUrl = core.getInput('teamsWebhookUrl')
      await service.sendTeamsMessage(webhookUrl, message)
    } else if (messageType === 'sns') {
      core.info('Sending SNS message')
      const snsParams = {
        awsAccessKeyId: core.getInput('awsAccessKeyId'),
        awsSecretAccessKey: core.getInput('awsSecretAccessKey'),
        awsRegion: core.getInput('awsRegion'),
        snsTopicArn: core.getInput('snsTopicArn')
      }

      if (
        snsParams.awsAccessKeyId &&
        snsParams.awsSecretAccessKey &&
        snsParams.awsRegion &&
        snsParams.snsTopicArn
      ) {
        await service.sendSNSMessage(context, commit, snsParams)
      }
    } else {
      core.setFailed(
        'Unsupported messageType. Supported types are "slack" and "discord"'
      )
    }
  } catch (error) {
    if (error instanceof Error) {
      core.setFailed(error.message)
    } else {
      core.setFailed(`Unexpected error: ${JSON.stringify(error)}`)
    }
  }
}

run()

Let’s look at one of the service and we will pick Slack for reference. Slack integration is done by token and you will find few more services that work with Webhook Urls to publish messages.

Here in slack service, I am basically defining the type for Context and Commit data that I am using. Slack offers a nice library to publish messages and i have used embedded content to set a style for the message.

Prior to this, follow the Slack documentation on how to create new slack app and create bot user with token here

import {WebClient} from '@slack/web-api'
type Context = {
  workflow: string
  eventName: string
  runId: number
  actor: string
  sha: string
  repo: {
    owner: string
    repo: string
  }
}

type Commit = {
  data: {
    commit: {
      message: string
    }
  }
}

export async function sendSlackMessage(
  token: string,
  context: Context,
  commit: Commit
): Promise<void> {
  // Styling the message
  const runUrl = `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`
  const message = {
    text: 'GitHub Actions Workflow Execution Details',
    blocks: [
      {
        type: 'divider' // divider at the top
      },
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `*Workflow Name:* ${context.workflow}`
        }
      },
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `*Event:* ${context.eventName}`
        }
      },
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `*Run URL:* <${runUrl}>`
        }
      },
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `*Triggered By:* ${context.actor}`
        }
      },
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `*Commit URL:* <https://github.com/${context.repo.owner}/${context.repo.repo}/commit/${context.sha}>`
        }
      },
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `*Commit Message:* ${commit.data.commit.message}`
        }
      },
      {
        type: 'divider' // divider at the top
      }
    ]
  }

  const web = new WebClient(token)
  await web.chat.postMessage({
    channel: '#general',
    text: message.text,
    blocks: message.blocks
  })
}

You will also need to define the input fields in action.yml file and here is a code snippet for the entire yml to support all the notification services listed above.

name: 'Uninotify'
description: 'Send notification to all messaging platforms'
author: 'Sandeep Yaramchitti'
inputs:
  githubToken:
    description: 'Github API token'
    required: true
  slackToken:
    description: 'Slack API token'
    required: false
  messageType:
    description: 'Type of message to send (slack, teams, chime)'
    required: true
  message:
    description: 'Message to send'
    required: false
  discordWebhookUrl:
    description: 'Webhook URL for discord'
    required: false
  chimeWebhookUrl:
    description: 'Webhook URL for chime'
    required: false
  teamsWebhookUrl:
    description: 'Webhook URL for teams'
    required: false
  awsAccessKeyId:
    description: 'AWS Access Key ID'
    required: false
  awsSecretAccessKey:
    description: 'AWS Secret Access Key'
    required: false
  awsRegion:
    description: 'AWS Region'
    required: false
  snsTopicArn:
    description: 'AWS SNS Topic ARN'
    required: false
runs:
  using: 'node16'
  main: 'dist/index.js'

Now, time to create a workflow to test this integration. Here is a workflow example on how to use UniNotify Action for slack integration.

Check out the Uninotify actions documentation for more information on required parameters.

Uninotify Github Actions: https://github.com/marketplace/actions/uninotify

name: 'build-test'
on: # rebuild any PRs and main branch changes
  pull_request:
  push:
    branches:
      - main
      - 'releases/*'
  workflow_dispatch:

jobs:
  notify:
    runs-on: ubuntu-latest
    steps:
    - name: Send Notification to slack
      uses: OSCloudysky/UniNotify@v1
      with:
        githubToken: ${{ secrets.GITHUB_TOKEN }}
        messageType: 'slack' 
        slackToken: ${{ secrets.SLACK_API_TOKEN }}

Once you run the above workflow, you should see the following slack message in the channel you directed the messages to be sent.

Challenges

The biggest challenge was handling different message formatting requirements for each platform. I had to read through each platform’s API documentation to understand how to format messages effectively.

Another challenge was managing tokens securely. I decided to use GitHub’s secret management capabilities to securely store our tokens.

Please do check out the action and let me in case. you have any feedback.

Discord Bot for AWS Blogs: Leveraging Terraform, Ansible, EC2, and GitHub Actions

Sandeep Yaramchitti — Mon, 24 Jul 2023 12:59:58 +0000

Build a discord bot that brings AWS devops blogs right into your Discord server. Starting from setting up our infrastructure using Terraform, we leverage Ansible to configure EC2 instances precisely as per our needs and use GitHub Actions, which facilitates continuous integration and deployment (CI/CD), ensuring that our bot is always running the latest code.

Introduction

**
In this blog post, we will create a dynamic discord bot specifically to fetch AWS DevOps blogs and bring them right into your discord server with ChatOps as the Discord Bot is a form of ChatOps working within your chat environment to keep you updated on AWS blogs.

Check out the Github code for reference : Discord BOT

Discord.js documentation: Discord.js

Setting up the Infrastructure

**
While there are a few examples of setting this up manually, lets dive into how this can be achieved through Infrastructure as Code with Terraform for provisioning EC2, associated AWS resources and we will leverage Ansible to configure the EC2 instance.

Terraform
In this blog, we will use the power of Terraform to provision our resources with expressive HCL syntax to define infrastructure’s desired state. This way, our infrastructure becomes easily replicable and version-controlled, promoting consistency and minimizing potential errors that can occur with manual configurations.

Here is the project structure that I have currently and lets define the resources required in each blocks.

main.tf
Let us add the following in the main.tf file and please find the explanation for the code block right after the code snippet.

# Add provider
provider "aws" {
  region = "us-east-1"
}
# VPC
resource "aws_vpc" "cloudysky-vpc" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
  tags = {
    Name = "cloudysky-vpc"
  }
}
# Subnet
# Refer http://blog.itsjustcode.net/blog/2017/11/18/terraform-cidrsubnet-deconstructed/
resource "aws_subnet" "cloudysky-subnet" {
  vpc_id                  = aws_vpc.cloudysky-vpc.id
  cidr_block              = cidrsubnet(aws_vpc.cloudysky-vpc.cidr_block, 3, 1)
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = true
}
# Security Group
resource "aws_security_group" "cloudsky-ingress-all" {
  name   = "cloudsky-ingress-allow-all"
  vpc_id = aws_vpc.cloudysky-vpc.id
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  // Terraform requires egress to be defined as it is disabled by default..
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
# EC2 Instance for testing
resource "aws_instance" "cloudysky-ec2" {
  ami                    = var.ami_id
  instance_type          = "t2.micro"
  key_name               = var.ami_key_pair
  subnet_id              = aws_subnet.cloudysky-subnet.id
  vpc_security_group_ids = [aws_security_group.cloudsky-ingress-all.id]
  tags = {
    Name = "cloudysky-ec2"
  }
}
# To access the instance, we would need an elastic IP
resource "aws_eip" "cloudysky-eip" {
  instance = aws_instance.cloudysky-ec2.id
  vpc      = true
}
# Route traffic from internet to the vpc
resource "aws_internet_gateway" "cloudysky-igw" {
  vpc_id = aws_vpc.cloudysky-vpc.id
  tags = {
    Name = "cloudysky-igw"
  }
}
# Setting up route table
resource "aws_route_table" "cloudysky-rt" {
  vpc_id = aws_vpc.cloudysky-vpc.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.cloudysky-igw.id
  }
  tags = {
    Name = "cloudysky-rt"
  }
}
# Associate the route table with the subnet
resource "aws_route_table_association" "cloudysky-rt-assoc" {
  subnet_id      = aws_subnet.cloudysky-subnet.id
  route_table_id = aws_route_table.cloudysky-rt.id
}

Provider: This block tells Terraform to use the AWS provider and to operate in the “us-east-1” region. The AWS provider is responsible for creating and managing resources on AWS.

VPC: A VPC (Virtual Private Cloud) is created, which is essentially a private network in AWS where you can launch your resources. The CIDR block “10.0.0.0/16” determines the IP range for the VPC. The parameters enable_dns_hostnames and enable_dns_support are set to true to enable DNS resolution and hostname support within the VPC.

Subnet: A subnet is created within the VPC. The subnet is assigned a subset of the IP range defined in the VPC (determined by the cidrsubnet function). The subnet is set to be in the “us-east-1a” availability zone and will automatically assign public IP addresses to instances launched within it.

Security Group: A security group acting as a virtual firewall for the EC2 instance is created. This security group is configured to allow incoming traffic on ports 22 (SSH), 80 (HTTP), and 443 (HTTPS) from any IP address (0.0.0.0/0). It also allows all outbound traffic.

EC2 Instance: An EC2 instance (a virtual server) is created with the specified AMI (Amazon Machine Image) and instance type. It’s launched within the created subnet and associated with the security group. You can SSH into this instance using the specified key pair.

Elastic IP (EIP): An Elastic IP address is a static, public IPv4 address, which is allocated to your AWS account. In this case, an EIP is associated with the EC2 instance to provide a fixed public IP address, useful if you restart your instance or it fails for some reason.

Internet Gateway: An internet gateway is a horizontally scaleable, redundant, and highly available VPC component that allows communication between your VPC and the internet. It’s attached to your VPC.

Route Table: A route table contains rules, called routes, that determine where network traffic from your subnet or gateway is directed. In this case, a route table is created which sends all traffic (0.0.0.0/0) to the Internet Gateway.

Route Table Association: Finally, the route table is associated with the subnet, which means that the routing rules defined in the route table will now apply to the subnet.

provider.tf

**
At the time of writing this blog, please refer the version that was used.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.8.0"
    }
  }
  cloud {
    organization = "cloudysky"
    workspaces {
      name = "terraform-sandeep-aws"
    }
  }
}

This code block is mainly about setting up your Terraform project with the necessary provider (AWS in this case) and configuring it to use Terraform Cloud for storing state and running operations.

required_providers: This block specifies which providers are necessary for your Terraform project. A provider is a plugin that Terraform uses to manage resources. In this case, it specifies that the AWS provider is required. The source attribute indicates that the AWS provider should be downloaded from the HashiCorp registry and the version attribute specifies that version 5.8.0 of the AWS provider should be used.

cloud: This block is used to configure the backend for your Terraform state. The state is a necessary component of Terraform that tracks the resources it has created. In this case, it is being configured to use Terraform Cloud, which is a service provided by HashiCorp for storing your Terraform state and running your Terraform operations. The organization attribute is used to specify the name of the Terraform Cloud organization to use.

workspaces: This block is used to specify which workspace in Terraform Cloud to use. A workspace in Terraform Cloud is a place where Terraform runs are executed. The name attribute is used to specify the name of the workspace. In this case, it specifies that the workspace “terraform-sandeep-aws” should be used.

variables.tf

This Terraform code defines two variables: ami_id and ami_key_pair. Here is a walkthrough of what each piece does:

variable "ami_id" {
  type    = string
  default = "ami-06ca3ca175f37dd66"
}
variable "ami_key_pair" {
  type    = string
  default = "terraform-ec2"
}

variable “ami_id”: This block declares a variable named ami_id. Variables in Terraform are a way to provide customizable, user-defined values that can be used within Terraform code. Please specify the AMI ID you would like to use.

variable “ami_key_pair”: Similarly, this block declares another variable named ami_key_pair. This variable is also of type string, and its default value is set to terraform-ec2. This terraform code expects you to create EC2 key pair manually from the AWS console and save the private key information securely.

output.tf

This Terraform code block defines an output value. In Terraform, outputs are like return values for a Terraform module. Once you run terraform apply, the outputs are displayed on the command line as a convenient way for users to retrieve and use these values.

output "instance_public_ip" {
  value = aws_instance.cloudysky-ec2.public_ip
}

Let’s break it down:

output “instance_public_ip”: This line is declaring an output named instance_public_ip. The string in the quotes is the identifier for this output, which you’ll use when referencing the output value.

value = aws_instance.cloudysky-ec2.public_ip: This is assigning a value to the output. In this case, the value is the public_ip attribute of the aws_instance resource named cloudysky-ec2

If you have followed so far, then you should have an fair idea on the terraform resources defined. We will use actions workflow to provision these resources, also store the terraform state in terraform cloud workspace.

Ansible

Ansible is an open-source software provisioning, configuration management, and application deployment tool. In this case, we will be using Ansible playbook to configure our EC2 instance which will be created by terraform. Primarily, we will be using playbook to install Nodejs 16 on EC2 and followed by PM2 installation process manager for production Node.js applications.

Here is walkthrough of the code after the code snippet.

- name: Install Node.js on EC2 instance
  hosts: ec2_instance
  gather_facts: no
  become: yes
  vars:
    ansible_ssh_private_key_file: /home/runner/.ssh/id_rsa
    ansible_python_interpreter: /usr/bin/python3
    ansible_user: ec2-user
    ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
  tasks:
    - name: Wait for SSH to come up
      wait_for:
        host: "{{ inventory_hostname }}"
        port: 22
        delay: 60
        timeout: 900
        state: started
    - name: Gathering Facts
      setup:
    - name: Update all packages
      dnf:
        name: '*'
        state: latest
    - name: Install tar and xz for unpacking Node.js binaries
      dnf:
        name:
          - tar
          - xz
        state: present
    - name: Download Node.js binary tarball
      get_url:
        url: https://nodejs.org/dist/v16.14.2/node-v16.14.2-linux-x64.tar.xz
        dest: /tmp/node-v16.14.2-linux-x64.tar.xz
    - name: Extract Node.js binary tarball
      command:
        cmd: tar -xf /tmp/node-v16.14.2-linux-x64.tar.xz -C /usr/local --strip-components 1
    - name: Verify Node.js installation
      command:
        cmd: node -v
      register: node_version
    - name: Display Node.js version
      debug:
        msg: 'Node.js version is {{ node_version.stdout }}'
    - name: Install PM2 globally
      npm:
        name: pm2
        global: yes

Let’s break down this ansible playbook

The following tasks of the playbook are carried out on hosts belonging to the group ec2_instance.

Wait for SSH to come up: This task uses the wait_for module to pause the playbook execution until SSH is available on the host.

Gathering Facts: The setup module is used here to gather facts about the host. This was explicitly requested as a task, as gather_facts: no was specified at the beginning of the play.

Update all packages: The dnf module is used to update all installed packages on the host.

Install tar and xz: The dnf module is again used to install the specified packages (tar and xz), which will later be used for unpacking the Node.js binaries.

Download Node.js binary tarball: The get_url module is used to download the Node.js binaries from the specified URL to the temporary location /tmp.

Extract Node.js binary tarball: The command module is used to extract the downloaded Node.js binaries to /usr/local.

Verify Node.js installation: The command module is used again to verify the installed Node.js version, with the output registered into the node_version variable.

Display Node.js version: The debug module is used to display the installed Node.js version, retrieved from node_version.stdout.

Install PM2 globally: Finally, the npm module is used to install PM2 (a process manager for Node.js applications) globally on the host.
This playbook, therefore, provides an automated way to install and configure Node.js on an EC2 instance.

Creating the Discord Bot

Now let’s look at how to create discord bot using discord.js (a popular nodejs library to build discord bots )

We will be using the bot to fetch AWS blogs specifically devops blogs and for easier understanding, I have kept it simple to fetch top 5 devops blogs when you send the request in the ChatOps but there are several enhancements that can be made to this.

const Discord = require('discord.js');
const client = new Discord.Client();
const Parser = require('rss-parser');
let parser = new Parser();
require('dotenv').config();
const AWS_DEVOPS_BLOGS_RSS_URL =
  'https://aws.amazon.com/blogs/devops/category/devops/feed/';
client.on('ready', () => {
  console.log(`Bot is ready and logged in as ${client.user.tag}!`);
});
client.on('message', async (msg) => {
  if (msg.content === '!devops') {
    try {
      let feed = await parser.parseURL(AWS_DEVOPS_BLOGS_RSS_URL);
      feed = feed.items.slice(0, 5);
      feed.forEach((item) => {
        const embed = new Discord.MessageEmbed()
          .setColor('#0099ff')
          .setTitle(item.title)
          .setURL(item.link)
          .setAuthor('AWS DevOps Blog', 'https://aws.amazon.com/favicon.ico')
          .setDescription(item.contentSnippet)
          .setThumbnail('https://aws.amazon.com/favicon.ico')
          .setTimestamp(new Date(item.pubDate))
          .setFooter('AWS DevOps Blog', 'https://aws.amazon.com/favicon.ico');
        msg.channel.send(embed);
      });
    } catch (error) {
      console.log(error);
    }
  }
});
client.login(process.env.DISCORD_BOT_TOKEN);

Set Up: The necessary packages and modules are imported, including the Discord.js for interacting with the Discord API, and rss-parser to parse the RSS feed from the AWS DevOps blog. An instance of the Discord client and RSS Parser are created.

Initialization: The bot logs in using a token from the environment variables and indicates it’s ready in the console.

Listening for Messages: The bot starts listening for messages on any server it’s connected to. It reacts specifically to a message containing “!devops”.

Processing the “!devops” Command: Upon receiving the “!devops” command, the bot fetches the RSS feed from the AWS DevOps blog, parses it and slices the first five items (the five most recent blog posts).

Creating and Sending Message Embeds: For each of these five blog posts, it generates a rich content message, known as an ‘embed’, that contains the blog post’s title, URL, description, and other details. These embeds are then sent to the same channel where the “!devops” command was used.

Error Handling: If there’s an error at any point during the “!devops” command processing, it’s caught and logged to the console.

Github Actions

I have used Github Actions to build the discord bot application code, upload to github artifacts and then download the artifacts and deploy it to target host (EC2)

Prior to the that, I have also used Github actions to provision the required infrastructure, configuration.

Infrastructure Provisioning and Configuration

Lets look at the Github Actions workflow and pretty long workflow :) The idea was to share one of the approaches to do it and it can be improved with reusable workflows for better organization of workflow.

name: Provision EC2
on: workflow_dispatch
jobs:
  runterraform:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: List out directory
        run: ls
      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
          terraform_wrapper: false
      - name: Terraform init
        run: terraform init -upgrade
      - name: Terraform apply
        id: terraform
        run: terraform apply -auto-approve
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - name: Output Instance Public IP
        id: ip
        run: |
          IP=$(terraform output -raw instance_public_ip)
          echo "::set-output name=PUBLIC_IP::$IP"
          echo "The public IP is $IP"
        env:
          TF_API_TOKEN: ${{ secrets.TF_API_TOKEN }}
    outputs:
      public_ip: ${{ steps.ip.outputs.PUBLIC_IP }}

  ansible:
    needs: runterraform  # Make this job depend on the `terraform` job
    runs-on: ubuntu-latest
    steps:
      - name: Set up SSH
        run: |
          mkdir -p ~/.ssh/
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Use the IP
        run: |
          echo "The public IP from the terraform job is ${{ needs.runterraform.outputs.public_ip }}"
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: 3.x
      - name: Install Ansible
        run: python -m pip install --upgrade pip && pip install ansible
      - name: Create Ansible Inventory
        run: |
          echo '[ec2_instance]' > inventory.ini
          echo '${{ needs.runterraform.outputs.public_ip }}' >> inventory.ini
      - name: Run Ansible Playbook
        run: ansible-playbook -i inventory.ini playbook.yml
        env:
          ANSIBLE_HOST_KEY_CHECKING: False

There are 2 workflow jobs in this workflow template and one is for Terraform used in Infrastructure provisioning and 2nd job is for Ansible to infrastructure configuration.

Terraform job is responsible to checkout the source code from github repo mentioned above and then use setup-terraform actions to plan and apply the changes to provision the required AWS resources. It uses Terraform Teams token and it was created in the terraform cloud console and I have stored it in the github secrets. Refer this terraform docs for team token — Terraform token for CI / CD It also outputs the public IP that was assigned on the EC2 instance is created.

Ansible Job is responsible to create a SSH connection to the target EC2 instance, install ansible on github runner, create ansible inventory for EC2 host and then run the playbook on target to perform required configuration.

Discord BOT application build and deployment

Let’s look at how to use Github actions to build this NodeJs code and deploy to EC2 and leverage PM2 — Advanced Node Process Manager to start the BOT.

name: Build and Deploy to EC2
on: workflow_dispatch
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v3
    - name: Setup Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '16' 
    - name: Install dependencies
      run: |
        cd src
        npm ci
    - name: Build project
      run: |
        cd src
        npm run build
    - name: Upload build files
      uses: actions/upload-artifact@v3
      with:
        name: build
        path: src/dist/
  deploy:
    name: Deploy
    needs: build
    runs-on: ubuntu-latest
    steps:
    - name: Download build files
      uses: actions/download-artifact@v3
      with:
        name: build
    - name: Deploy to EC2
      run: |
        echo "${{ secrets.SSH_PRIVATE_KEY }}" > id_rsa
        chmod 600 id_rsa
        ls
        ssh -o StrictHostKeyChecking=no -i id_rsa ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }} "echo export DISCORD_BOT_TOKEN=${{ secrets.DISCORD_BOT_TOKEN }} >> ~/.bashrc && source ~/.bashrc"
        ssh -o StrictHostKeyChecking=no -i id_rsa ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }} 'mkdir -p /home/ec2-user/discord-bot'
        scp -o StrictHostKeyChecking=no -i id_rsa -r dist/* ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }}:/home/ec2-user/discord-bot
        ssh -o StrictHostKeyChecking=no -i id_rsa ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }} 'cd /home/ec2-user/discord-bot && npm ci --production && pm2 kill && pm2 start main.js'
      shell: bash

This worklfow has couple of jobs called Build and deploy. Build job is responsible to checkout source code, build nodejs application, upload artifacts to github artifacts. Deploy job is responsible for deploying the application to EC2 using SSH using private key that I have stored in Secrets. Alternatively, you can use AWS CodeDeploy to deploy to target EC2 instance but I have gone with SSH route.

Once everything is all set, You should see the Discord Bot online and get a response for every command you send that has !DevOps

Watch out for more interesting blogs and happy reading!

Provisioning and Accessing Cloud9 Development Environment with a Public-Facing URL using Terraform and Github actions

Sandeep Yaramchitti — Fri, 07 Jul 2023 19:35:33 +0000

AWS Cloud9 IDE provides a powerful and collaborative environment for software development. Provisioning Cloud9 with Terraform and automating the process with GitHub Actions can streamline the setup, configuration, and maintenance of your development environment. In this blog post, we will explore how to leverage Terraform and GitHub Actions to provision and manage Cloud9 IDE, enabling faster and more efficient development workflows.

Setting up Terraform for Cloud9 provisioning

Follow the Terraforms official guide _Get Started - AWS _for quick start and installation process for Terraform, configuring it to link your AWS account.

Once, you have the required prerequisites - Lets start writing the terraform code to provision Cloud9 instance.

resource "aws_cloud9_environment_ec2" "cloudysky_cloud9_instance" {
  name                        = "cloudysky_cloud9_instance"
  instance_type               = "t2.medium"
  automatic_stop_time_minutes = 30
  subnet_id                   = data.aws_subnet.existing_subnet.id

  tags = {
    Environment = "dev"
    Owner       = "CloudySky"
  }
}

Check out the official Terraform documentation for the available arguments for this resource aws_cloud9_environment_ec2

In my example above, I have included name, instance type, automatic stop time minutes to shut down the instance after 30 minutes since last used and fetch subnet id dynamically. Also, i have included couple of tags for this resource.

To get the subnet ID you can use various approaches and for easier understanding, I have used data block so that terraform reads from the given source which can be exported and referred with the local name.

data "aws_subnet" "existing_subnet" {
  filter {
    name   = "vpc-id"
    values = ["vpc-048c23df4f54370a4"] # Replace with the ID of the VPC
  }

  filter {
    name   = "tag:Name"
    values = ["project-subnet-public1-us-east-1a"] # Replace with the desired subnet name
  }
}

In addition to this, if you would like to make the Cloud9 URL with public access - please add the following ingress SG rule as well. You could also use AWS Cognito user pool to authenticate users. (Watch out for a blog in that soon :) )

To allow public access, first get the security group associated with the EC2 instance dynamically and then add a SG rule to allow traffic from public internet.

data "aws_security_group" "cloud9_secgroup" {
  filter {
    name = "tag:aws:cloud9:environment"
    values = [
      aws_cloud9_environment_ec2.cloudysky_cloud9_instance.id
    ]
  }
}
# Allow public access to port 8080
resource "aws_security_group_rule" "cloud9_inbound" {
  type              = "ingress"
  from_port         = 8080
  to_port           = 8080
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = data.aws_security_group.cloud9_secgroup.id
}

In addition to the above, define the following output variables for EC2 ID, ARN and Cloud9 URL.

# EC2 ID
output "id" {
  value = aws_cloud9_environment_ec2.cloudysky_cloud9_instance.id
}

# EC2 ARN
output "arn" {
  value = aws_cloud9_environment_ec2.cloudysky_cloud9_instance.arn
}

# cloud9 url
output "cloud9_url" {
  value = "https://${var.region}.console.aws.amazon.com/cloud9/ide/${aws_cloud9_environment_ec2.cloudysky_cloud9_instance.id}"
}

Automating Cloud9 Provisioning with GitHub Actions

GitHub Actions provides a powerful automation platform that integrates seamlessly with your GitHub repositories. By leveraging GitHub Actions, you can automate the provisioning of Cloud9 environments, ensuring consistent and reliable setups for your development workflows. Here's how you can automate Cloud9 provisioning using GitHub Actions.

Create a Workflow File:

In your GitHub repository, create a new workflow file in the .github/workflows directory. You can name it something like cloud9-provisioning.yml. This file will define the steps and triggers for the Cloud9 provisioning workflow. _ _

Define Workflow Triggers:

The idea here is to run the terraform plan for any PR request created and update the pull request comment with the plan details and terraform apply --auto-approve when the pull request is merged.

Configure Workflow Steps:

Inside the workflow file, define the steps required to provision the Cloud9 environment. Here is a quick snippet on how I have done this.

name: "Terraform Cloud9 Provisioning"

on:
  push:
    branches:
      - master
  pull_request:

jobs:
  terraform:
    name: "Terraform Cloud9 Provisioning"
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1
        with:
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}

      - name: Terraform Format
        id: fmt
        run: terraform fmt -check

      - name: Terraform Init
        id: init
        run: terraform init

      - name: Terraform Validate
        id: validate
        run: terraform validate -no-color

      - name: Terraform Plan
        id: plan
        if: github.event_name == 'pull_request'
        run: terraform plan -no-color -input=false
        continue-on-error: true

      - name: Update Pull Request
        uses: actions/github-script@v6
        if: github.event_name == 'pull_request'
        env:
          PLAN: ${{ steps.plan.outputs.stdout }}
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const output = `#### Terraform Format and Style 🖌\`${{ steps.fmt.outcome }}\`
            #### Terraform Initialization ⚙️\`${{ steps.init.outcome }}\`
            #### Terraform Validation 🤖\`${{ steps.validate.outcome }}\`
            #### Terraform Plan 📖\`${{ steps.plan.outcome }}\`

            <details><summary>Show Plan</summary>

            \`\`\`terraform\n
            ${process.env.PLAN}
            \`\`\`

            </details>

            *Pushed by: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output
            })

      - name: Terraform Plan Status
        if: steps.plan.outcome == 'failure'
        run: exit 1

      - name: Terraform Apply
        if: github.ref == 'refs/heads/master' && github.event_name == 'push'
        run: terraform apply -auto-approve -input=false

Please make sure to configure _TF_API_TOKEN _in github secrets and following the steps mentioned on official documentation for Terraform and Github.

Test and Deploy Workflow:

Commit and push the workflow file to your repository. GitHub Actions will automatically pick up the workflow configuration and start running it based on the specified triggers. Monitor the workflow runs and verify that the Cloud9 environment is provisioned successfully.

**Check out the source code here : **https://github.com/SandeepKumarYaramchitti/terraform-ec2-cloud9ide

By automating the Cloud9 provisioning process with GitHub Actions, you can ensure consistent and reproducible setups for your development environments. This automation saves time and effort, enabling developers to focus on writing code rather than manually setting up environments.

Provisioning OpenSearch as Infrastructure as Code (IaC) and Securing Data Access with API Gateway and Lambda

Sandeep Yaramchitti — Sun, 25 Jun 2023 20:05:48 +0000

This blog is a continuation of my blog series on "How I built my Cloudysky portal from ground up using some of the modern tech stacks. Please have a look at the following blogs to get the overview of the architecture.

In this blog, I will focus on provisioning OpenSearch as Infrastructure as Code (IaC) through CloudFormation Template automate the deployment and configuration of an OpenSearch cluster while ensuring secure access to the cluster's data through API Gateway and Lambda functions.

Provisioning OpenSearch Cluster

In this example, I will be using aws cli to provision OpenSearch Cluster with the CloudFormation template.

Create a CloudFormation template in YAML or JSON format. This template will define your OpenSearch cluster's infrastructure.
Include the necessary resources, such as an AWS::OpenSearchService::Domain resource, to provision the OpenSearch cluster.
Configure the desired properties for the OpenSearch cluster, such as instance types, storage options, and network settings.
Add any additional resources or configurations required for your specific use case.

_Remember to review the OpenSearch EngineVersion you would be using. Also, review the charges for OpenSearch services to avoid unnecessary charges for the service usage. Also, as a best practice keep the Opensearch domain endpoint username and password secured. In this example, I have kept them in Secrets Manager for my reference only. _

AWSTemplateFormatVersion: '2010-09-09'

Parameters:
  DomainName:
    Type: String
    Description: A name for the Amazon OpenSearch domain
  MasterUsername:
    Type: String
    Description: Master usernames must be between 1 and 16 characters.
  MasterPassword:
    Type: String
    Description: Master password must be at least 8 characters long and contain at least one uppercase letter, one lowercase letter, one number, and one special character.
    NoEcho: true

Resources:
  OpenSearchServiceDomain:
    Type: AWS::OpenSearchService::Domain
    Properties: 
      DomainName: !Ref DomainName
      EngineVersion: 'OpenSearch_2.3'
      ClusterConfig:
        DedicatedMasterEnabled: true
        InstanceCount: '2'
        ZoneAwarenessEnabled: true
        InstanceType: 'r6g.large.search'
        DedicatedMasterType: 'r6g.large.search'
        DedicatedMasterCount: '3'
      EBSOptions:
        EBSEnabled: true
        Iops: '0'
        VolumeSize: '20'
        VolumeType: 'gp2'
      AccessPolicies:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            AWS: '*'
          Action:
          - 'es:*'
          Resource: !Sub 'arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${DomainName}/*'
      EncryptionAtRestOptions:
        Enabled: true
      NodeToNodeEncryptionOptions:
        Enabled: true
      DomainEndpointOptions:
        EnforceHTTPS: true
      AdvancedSecurityOptions:
        Enabled: true
        InternalUserDatabaseEnabled: true
        MasterUserOptions:
          MasterUserName: !Ref MasterUsername
          MasterUserPassword: !Ref MasterPassword
  MySecretMasterUsername:
    Type: 'AWS::SecretsManager::Secret'
    Properties:
      Name: os-username
      Description: Master username for Amazon Opensearch cluster
      SecretString: !Ref MasterUsername
  MySecretMasterPassword:
    Type: 'AWS::SecretsManager::Secret'
    Properties:
      Name: os-password
      Description: Master password for Amazon Opensearch cluster
      SecretString: !Ref MasterPassword
Outputs:
  DomainEndpoint:
    Value: !GetAtt OpenSearchServiceDomain.DomainEndpoint

Step 1: Upload the CloudFormation Template to S3

If you haven't already, create an S3 bucket where you can upload the CloudFormation template.
Use the AWS CLI to upload the template file to your S3 bucket. For example:

aws s3 cp <local_template_file> s3://<your_bucket_name>/<template_file_name>

Step 2: Create the CloudFormation Stack

Use the AWS CLI to create the CloudFormation stack, specifying the template URL and desired stack parameters. For example:

aws cloudformation create-stack --stack-name <stack_name> --template-url https://s3.amazonaws.com/<your_bucket_name>/<template_file_name> --parameters ParameterKey=<parameter_name>,ParameterValue=<parameter_value> --capabilities CAPABILITY_NAMED_IAM

Replace with a unique name for your CloudFormation stack.
Update the --template-url parameter with the appropriate URL for your S3 bucket and template file.
Provide the necessary parameters and their values using the --parameters option. Adjust the parameters according to your CloudFormation template.
Wait for the CloudFormation stack creation to complete. You can use the following command to check the stack status:

aws cloudformation describe-stacks --stack-name <stack_name> --query "Stacks[0].StackStatus"

Step 3: Access the Provisioned OpenSearch Cluster

Once the stack creation is complete, use the AWS Management Console or AWS CLI to retrieve the endpoint URL of your provisioned OpenSearch cluster.

Data Ingestion into OpenSearch

In this example, I have used the sanity CMS data coming from webhook event to ingest into OpenSearch and based on your use case, you could modify it.

Here is a sample Lambda code that ingests data into OpenSearch Cluster.

I found opensearch-py client library has good support to interact with Opensearch cluster in python.
As a first step, you would connect to the OpenSearch domain endpoint using the client library using the secrets stored in SecretsManager and then perform insert and update operation conditionally.

import json
import boto3
from opensearchpy import OpenSearch, helpers


def lambda_handler(event, context):

    domain_endpoint = "YOUR_OPENSEARCH_DOMAIN_ENDPOINT"
    sanity_blog_playload = event['Payload']
    sanity_blog_title = sanity_blog_playload['title']
    sanity_blog_description = sanity_blog_playload['description']
    sanity_blog_body = sanity_blog_playload['body']
    sanity_blog_id = sanity_blog_playload['id']
    sanity_blog_categories = sanity_blog_playload['categories']
    sanity_blog_created_at = sanity_blog_playload['createdAt']
    sanity_blog_updated_at = sanity_blog_playload['updatedAt']

    categories = []
    for category in sanity_blog_categories:
        categories.append(category['title'])

    sanity_cms_data = [
        {
            "id": sanity_blog_id,
            "title": sanity_blog_title,
            "description": sanity_blog_description,
            "body": sanity_blog_body,
            "categories": categories,
            "createdAt": sanity_blog_created_at,
            "updatedAt": sanity_blog_updated_at
        }
    ]

    # Get Secret from secret manager

    def get_secrets(secret_name):
        region_name = "us-east-1"
        session = boto3.session.Session()
        client = session.client(
            service_name='secretsmanager',
            region_name=region_name
        )
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name)
        return get_secret_value_response

    # Get connection string
    def get_connection_string():
        user = get_secrets(secret_name="os-username")['SecretString']
        password = get_secrets(secret_name="os-password")['SecretString']
        connection_string = "https://{}:{}@{}:443".format(
            user, password, domain_endpoint)
        return connection_string

    # Insert Documents
    def insert_documents(data, client, index):
        def gendata():
            for document in data:
                id = document["id"]
                yield {
                    "_id": id,
                    "_index": index,
                    "_source": document
                }
        response = helpers.bulk(client, gendata())
        print("\nIndexing Documents")
        print(response)

    # update Document
    def update_documents(connection_string, data, client):
        def gendata():
            for document in data:
                index_id = document["id"]
                # Delete 'id' column because we don't want to index it
                yield {
                    "_op_type": "update",
                    "_id": index_id,
                    "_index": index,
                    "doc": document,
                }
        response = helpers.bulk(client, gendata(), request_timeout=60)
        print('\nUpdating document')
        print(response)

    # Execution of the index data
    connection_string = get_connection_string()
    client = OpenSearch([connection_string])
    # Search for the document
    document_exists = client.exists(index=index, id=sanity_blog_id)

    if document_exists:
        print("Document already exists")
        update_documents(connection_string, sanity_cms_data, client)
    else:
        print("Document does not exist")
        insert_documents(sanity_cms_data, client, index)

You can verify the data ingestion by logging into the OpenSearch domain endpoint URL.

Secure access to the cluster's data through API Gateway and Lambda functions

To access the data securely, you could use various strategies and the following uses API gateway and Lambda and this is used by my Cloudysky web app to make calls to OpenSearch to get search results.

Here is the CloudFormation template for the API gateway with custom route53 domain and also, uses API key to secure the API access. Also, it has the Lambda function resource which will used to search the data in OpenSearch cluster.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  cloudysky-web-opensearch-api

  Sample SAM Template for cloudysky-web-opensearch-api

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3
    MemorySize: 128
    Tracing: Active
  Api:
    TracingEnabled: true

Resources:
  ApiCertificate:
    Type: AWS::CertificateManager::Certificate
    Properties:
      DomainName: !Sub api-prod.cloudysky.link
      ValidationMethod: DNS

  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: prod
      Cors: "'*'"
      Domain:
        DomainName: !Sub CUSTOM_DOMAIN
        CertificateArn: !Ref ApiCertificate
        EndpointConfiguration: EDGE
        Route53:
          HostedZoneId: HOSTED_ZONE
      Auth:
        ApiKeyRequired: true # sets for all methods
        UsagePlan:
          CreateUsagePlan: PER_API
          # Quota:
          #   Limit: 1000
          #   Period: DAY
          # Throttle:
          #   RateLimit: 5
      # CacheClusterEnabled: true
      # CacheClusterSize: '0.5'
      # MethodSettings:
      #   - ResourcePath: /search
      #     HttpMethod: GET
      #     CachingEnabled: true
      #     CacheTtlInSeconds: 300    

  OpenSearchAPIFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: opensearch_api/
      Handler: app.lambda_handler
      Runtime: python3.9
      Architectures:
      - x86_64
      Events:
        OpenSearchAPI:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /search
            Method: get
            RestApiId:
              Ref: ApiGatewayApi
      Policies:
        - AmazonOpenSearchServiceFullAccess 
        - CloudWatchPutMetricPolicy: {}
        - SecretsManagerReadWrite
        - AmazonSSMReadOnlyAccess

  ApplicationResourceGroup:
    Type: AWS::ResourceGroups::Group
    Properties:
      Name:
        Fn::Join:
        - ''
        - - ApplicationInsights-SAM-
          - Ref: AWS::StackName
      ResourceQuery:
        Type: CLOUDFORMATION_STACK_1_0
  ApplicationInsightsMonitoring:
    Type: AWS::ApplicationInsights::Application
    Properties:
      ResourceGroupName:
        Fn::Join:
        - ''
        - - ApplicationInsights-SAM-
          - Ref: AWS::StackName
      AutoConfigurationEnabled: 'true'
    DependsOn: ApplicationResourceGroup
Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: API Gateway endpoint URL for Prod stage for Hello World function
    Value: !Sub "https://${ApiGatewayApi}.execute-api.${AWS::Region}.amazonaws.com/prod/"
  OpenSearchAPIFunction:
    Description: Hello World Lambda Function ARN
    Value: !GetAtt OpenSearchAPIFunction.Arn
  OpenSearchAPIFunctionIamRole:
    Description: Implicit IAM Role created for Hello World function
    Value: !GetAtt OpenSearchAPIFunctionRole.Arn

Here is the snippet for the actual Lambda function that gets invoked based on API gateway event. You could use various query DSL strategies to perform a full text search and in this case, match_phrase_prefix query worked fine for me.

import json
import boto3
from opensearchpy import OpenSearch, helpers


def lambda_handler(event, context):

    domain_endpoint = 'YOUR_DOMAIN_ENDPOINT'
    index = 'cloudysky-cms-data'
    queryString = event['queryStringParameters']['q']

    # Get Secrets from Secrets Manager
    def get_secrets(secret_name):
        region_name = "us-east-1"
        session = boto3.session.Session()
        client = session.client(
            service_name='secretsmanager',
            region_name=region_name
        )
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name)
        return get_secret_value_response
    # Get Connection String
    def get_connection_string():
        user = get_secrets(secret_name="os-username")['SecretString']
        password = get_secrets(secret_name="os-password")['SecretString']
        connection_string = "https://{}:{}@{}:443".format(
            user, password, domain_endpoint)
        return connection_string

    # Search Documents within OpenSearch
    def search_documents(data):
        connection_string = get_connection_string()
        client = OpenSearch([connection_string])
        response = client.search(index=index, body={
            "query": {
                "match_phrase_prefix": {
                    "title": {
                        "query": data,
                        "slop": 3,
                        "max_expansions": 10,
                    }
                }
            }       
        })
        return response

    response_data = search_documents(queryString)   
    hits = response_data['hits']['hits']

    response = {
        "statusCode": 200,
        "headers": {
            "Access-Control-Allow-Origin": '*'
        },
        "isBase64Encoded": False
    }

    response['body'] = json.dumps(hits)
    return response

    # return {
    #     "statusCode": 200,
    #     "body": json.dumps({
    #         "message": queryString,
    #     }),
    # }

If you have followed above steps correctly, you would end up with a API URL like mine :- https://**.**/api/opensearch-results?q=DevOps

This integrated approach offers several benefits. It allows you to provision and manage your OpenSearch cluster and associated resources in a consistent and repeatable manner. API Gateway provides a secure and scalable interface for accessing the OpenSearch data, ensuring fine-grained control over access permissions and enforcing security measures. Lambda functions offer flexibility and extensibility, enabling you to implement custom business logic or data transformations as needed.

By combining IaC, API Gateway, and Lambda, you can streamline the provisioning of OpenSearch clusters, enhance security, and provide a scalable and controlled API interface for accessing and managing your data.

Automating AWS Step Functions: Programmatically Initiating Workflows for Seamless Orchestration

Sandeep Yaramchitti — Fri, 23 Jun 2023 20:04:47 +0000

Recently, I came across a scenario that required using a programmatic approach, such as invoking Step Functions directly from a Lambda function over triggering Step Functions via API Gateway directly.

While developing the Cloudysky portal, I came across a scenario that basically processes the Sanity CMS webhook data and initiate a state machine workflow to cross post and publish my blogs into various blogging and logging platform such as medium, Devto, Hashnode and AWS opensearch for logging. Initial intent was to just trigger and pass the data from API gateway directly to the step function workflow and that didn't really scale as I ended up doing same repeated data transformation in each state machine lambda functions. So, I thought of adding a Lambda function between the API gateway and State Machine which can do that data transformation and initiate state machine workflow by passing the transformed data.

Here is how I was able to build this solution and I have used Cloudformation to build the required infrastructure.

The below code snippet creates the API gateway resource

  SanityEventApiGateway: 
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      GatewayResponses:
        DEFAULT_2XX:
          ResponseParameters:
            Headers:
              Access-Control-Allow-Origin: "'*'"
          StatusCode: 200
          ResponseTemplates:
            application/json: '{ "message": "Sucessfully processed the event" }'

Here is the step function initiator lambda function that runs when any webhook event is received from Sanity CMS once configured. Also, note that StepFunctionInitiatorFunction will require _StepFunctionsExecutionPolicy to allow permission to kick of the workflow. _In addition, I have also provided SecretsManagerReadWrite which is used to fetch required credentials for data transformation.

  StepFunctionInitiatorFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: stepfunction-initiator-lambda/
      Handler: app.lambdaHandler
      Runtime: nodejs16.x
      Architectures:
      - x86_64
      Policies:
        - SecretsManagerReadWrite
        - StepFunctionsExecutionPolicy:
            StateMachineName: !GetAtt StateMachine.Name
      Environment:
        Variables:
          StateMachineArn: !GetAtt StateMachine.Arn
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            RestApiId: !Ref "SanityEventApiGateway"
            Path: /webhook
            Method: post

Also, refer the state machine and StatesExecutionRole defined in the Cloudformation template. State machine will require the reference to Amazon State Language file that defines the workflow and has StatesExecutionRole that allows _lambda:InvokeFunction for each of the lambda functions within the workflow. _

StateMachine:
    Type: AWS::Serverless::StateMachine # More info about State Machine Resource: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-statemachine.html
    Properties:
      DefinitionUri: statemachine/stateMachine.asl.json
      DefinitionSubstitutions:
        MediumArticleLambdaFunctionArn: !GetAtt MediumArticleLambdaFunction.Arn
        DevtoArticleLambdaFunctionArn: !GetAtt DevtoArticleLambdaFunction.Arn
        HashnodeArticleLambdaFunctionArn: !GetAtt HashnodeArticleLambdaFunction.Arn
        OpenSearchLoadArticleFunctionArn: !GetAtt OpenSearchLoadArticleFunction.Arn
      Role:
        Fn::GetAtt: [ StatesExecutionRole, Arn ]
      Logging:
        Destinations:
          - CloudWatchLogsLogGroup:
              LogGroupArn: !GetAtt StateMachineLogGroup.Arn
        IncludeExecutionData: false
        Level: 'ALL'  
  StatesExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - !Sub states.${AWS::Region}.amazonaws.com
            Action: "sts:AssumeRole"
      Path: "/"
      Policies:
        - PolicyName: LambdaExecute
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "lambda:InvokeFunction"
                Resource: 
                  - !GetAtt MediumArticleLambdaFunction.Arn
                  - !GetAtt DevtoArticleLambdaFunction.Arn
                  - !GetAtt HashnodeArticleLambdaFunction.Arn
                  - !GetAtt OpenSearchLoadArticleFunction.Arn
              - Effect: Allow
                Action:
                  - "cloudwatch:*"
                  - "logs:*"
                Resource: "*"

Now, lets see the code which does the programatic call to start the workflow. This is just a snippet and can be refactored based on the need.

The below snippet for Lambda function receives the Sanity CMS webhook event data and trims down the data to required ones.
For easier reference, I have excluded the data transformation section and this is basically to show that any type of data manipulation can be done here.
Also, fetches additional data from sanity connecting through sanity client and gets required credentials from secrets manager.
Finally, it creates an instance to connect to Step Functions using AWS-SDK.
Last step of this flow is to call the stepfunctions.startExecution function with the data you would want to pass it to the step function workflow.

let response;
let blogData;
const AWS = require('aws-sdk');
const moment = require('moment');
const { createClient } = require('@sanity/client');
const secretManagerClient = new AWS.SecretsManager();

exports.lambdaHandler = async (event, context) => {
  // Step Function execution section
  const sfnArn = process.env.StateMachineArn;
  const rightNow = moment().format('YYYYMMDD-hhmmss');

  let blog = JSON.parse(event.body);

  // Write data transformation logic

  // Get the secret value for sanity
  const secret_name = 'sanity-cms';
  let response;
  try {
    response = await secretManagerClient
      .getSecretValue({
        SecretId: secret_name,
      })
      .promise();
  } catch (error) {
    throw error;
  }
  const secret = JSON.parse(response.SecretString);
  const SANITY_PROJECT_ID = secret.SANITY_PROJECT_ID;
  const SANITY_DATASET = secret.SANITY_DATASET;

  // Create a sanity client
  const config = {
    dataset: SANITY_DATASET,
    projectId: SANITY_PROJECT_ID,
    apiVersion: '2022-11-16',
    useCdn: false,
  };
  const client = createClient(config);

  const query = `*[_type == 'post' && _id == '${blogID}']{
    categories[] -> {
      title,
      slug
    }
  }`;

  // Get the categories
  const categories = await client.fetch(query);

  const data = {
    id: blog._id,
    title: blog.title,
    description: blog.description,
    body: blog.body,
    categories: categories[0].categories,
    createdAt: blog._createdAt,
    updatedAt: blog._updatedAt,
    mainImage: blog.mainImage,
  };

  const params = {
    stateMachineArn: sfnArn,
    input: JSON.stringify(data),
    name: blogID + '-' + rightNow,
  };

  const stepfunctions = new AWS.StepFunctions();
  try {
    response = await stepfunctions.startExecution(params).promise();
  } catch (err) {
    console.error(err);
  }

  return response;
};

There are several scenarios where using a programmatic approach, such as invoking Step Functions directly from a Lambda function, may be preferable over triggering Step Functions via API Gateway:
Fine-grained control: When you need precise control over the invocation of your Step Functions workflow and want to perform custom logic or transformations on the input data before triggering the workflow, a programmatic approach provides more flexibility.
Complex data transformations: If you have complex data transformations or data preprocessing requirements before triggering the workflow, using a Lambda function allows you to handle these tasks efficiently. You can perform format conversions, or data validation as needed within the Lambda function before passing the processed data to the Step Function.
Orchestration with multiple services: If your workflow involves coordination and interaction with multiple AWS services or external systems, a programmatic approach can be beneficial. The Lambda function can act as the central orchestrator, integrating with various services and systems, making API calls, retrieving data, and applying business logic to decide when and how to trigger the Step Functions workflow.

Watch out for more blogs in this space….

How I built my Cloudsky portfolio and leveraged AWS services for infrastructure and automation

Sandeep Yaramchitti — Thu, 22 Jun 2023 12:38:38 +0000

Are you interested in building your own portfolio application from ground up using modern tech stack? Here is a blog explaining the process I followed to build my Cloudysky web app.

My inspiration for this app is to have the ability to create blog posts, cross post to multiple blogging platforms in a fully automated way, organize and manage my blog posts at one consolidated place in admin studio, logging events, analytics, service uptime calculator and AWS resources cost calculator for my portal, showcase some of the modern features in tech, integration with external services and in the process stay current with tech.

Now, lets talk about the tech stack in details and implementation process.

Web application in Next.Js

Next.js is a powerful and popular framework for building production ready applications with cutting edge features and few of them used on this app are server side rendering, Incremental Static Regeneration, optimize website performance by pre-rendering pages, enabling faster initial load times, automatic code splitting etc. I have used Tailwindcss for styling and micro link for link previews etc. to name a few.

Check out the github repo for more information :- https://github.com/SandeepKumarYaramchitti/cloudysky-web-app

Composable CMS with Sanity

Sanity CMS is a flexible and composable headless content management system that allows to structure and manage content for the websites or applications. Why i went with Sanity for CMS ?

Using the composable CMS capabilities of Sanity in conjunction with Next.js greatly enhanced the content management and development workflow for my application. With Sanity Studio, its powerful content editing interface, I was able to create customized and intuitive editing experiences tailored to my specific project requirements. The studio provides a user-friendly interface for content authors to create, edit, and organize content in a structured manner.

I also used Next.js provided server-side rendering and static site generation (ISR) capabilities, allowing to fetch data from Sanity during the build process and pre-render pages with the latest content. This ensures that my application always displays up-to-date content without compromising performance.

AWS Amplify Web hosting and Route 53

Amplify Web Hosting is a feature of AWS Amplify that allows to deploy and host web applications, static websites, and single-page applications (SPAs) with ease. It provides a managed environment for building, deploying, and scaling your applications while offering features like continuous deployment, automatic SSL certificate management, and CDN (Content Delivery Network) integration for fast content delivery.

These are some of the features I really like with Amplify

**Build and Deploy Settings: **Within the Amplify console, I have defined the build and deployment settings for my web application. This includes specifying the build commands, environment variables, secrets, branch mappings, and other configuration options. Above all, this is logging that is next level for a SSR / ISR application specially as it is NextJs based app. Also, configurations are slightly different for nextjs and checkout Amplify.yml for nextjs build configuration.
**Pull Request Preview: **This is one of my all time favorite feature on Amplify and takes effect when a pull request is created in my web application github repo, Amplify automatically detects it and triggers a preview build. The preview build creates a temporary environment that reflects the changes made in the pull request along with a preview URL.
**Route 53 domain URL: **Amplify works well with Route 53 URLs out of the box as I could do the DNS Configuration in Amplify console after domain registration, Amplify provides automatic provisioning and management of SSL/TLS certificates using AWS Certificate Manager (ACM). After that, Amplify will build my application and create a deployment bundle, which is then distributed and served by AWS infrastructure. In summary, Amplify leverages Route 53 for domain registration, DNS configuration, and routing of traffic to my web application - https://cloudysky.link/. It provides an integrated workflow, making it easier to manage custom domains and their associated DNS settings within the Amplify ecosystem

Sanity Webhook and AWS API Gateway Integration

Sanity provides a webhook functionality as part of its content management system. A webhook in Sanity is a mechanism that allows to receive real-time notifications or trigger actions in external systems when certain events occur within the CMS. These events can include content updates, deletions, creations, or other actions within the CMS. When the specified event occurs, Sanity will automatically send an HTTP request to the configured webhook URL, providing relevant data about the event.

I have set up the API gateway in AWS using Cloudformation Iac that will receive the sanity webhook events on CMS content change and will forward that to a step function initiator lambda to kick of data processing.

  SanityEventApiGateway: 
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      GatewayResponses:
        DEFAULT_2XX:
          ResponseParameters:
            Headers:
              Access-Control-Allow-Origin: "'*'"
          StatusCode: 200
          ResponseTemplates:
            application/json: '{ "message": "Sucessfully processed the event" }'

Once the sanity webhook event is received, API gateway triggers a Lambda function to initiate the step function workflow.

  StepFunctionInitiatorFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: stepfunction-initiator-lambda/
      Handler: app.lambdaHandler
      Runtime: nodejs16.x
      Architectures:
      - x86_64
      Policies:
        - SecretsManagerReadWrite
        - StepFunctionsExecutionPolicy:
            StateMachineName: !GetAtt StateMachine.Name
      Environment:
        Variables:
          StateMachineArn: !GetAtt StateMachine.Arn
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            RestApiId: !Ref "SanityEventApiGateway"
            Path: /webhook
            Method: post

AWS State machine to transform and publish blog posts

An AWS State Machine, also known as AWS Step Functions, is a serverless workflow service provided by Amazon Web Services (AWS). It enables to build and coordinate highly scalable and resilient applications by orchestrating multiple AWS services and integrating them with custom code or business logic. In this scenario, the step function orchestrated the lambda function executions in parallel to transform data required to publish to each of the blogging platforms such as Medium, DevTo, Hashnode and also, lambda to ingest data into opensearch for data logging.

Watch out this space and my Cloudsky portal for more details on this. At somepoint, I will make the github source code public as well once I refactor and review for any security concerns etc.

Upstash for serverless database for caching

I have also used serverless data platform Upstash to caching data when data in ingested to AWS opensearch. This just helps to avoid repeated API calls to opensearch for data query. More to come in this and snippets on how this integration is done.

I still have some work to do on my portal however I thought of sharing my experience on building the portal as a side project. Let me know if this helps to get an idea on how to design, build, deploy your own idea into production.

Watch out for more updates in this....