DEV Community: Camille He

Host a Vue Project in AWS Amplify using Terraform

Camille He — Fri, 26 Jul 2024 07:12:45 +0000

In the blog, I will introduce how to scaffolding a Vue based frontend project in AWS Amplify from scratch. The template project is setup using Vite, AWS Amplify infrastructure is provisioned using Terraform. Both Vue source code and Terraform infrastructure code are managed and versioned in GitHub repository. Besides, GitHub Actions workflows are created to provision and manage Cloud infrastructure. New commits to GitHub repository will trigger a new deployment in Amplify App accordingly. In the end, you should know:

How to setup a Vue based frontend template project from scratch.
How to define and manage AWS infrastructure using Terraform.
How to provision and manage AWS infrastructure via GitHub Actions workflow.

Prerequisites
Preparation
- 1. Create PAT in GitHub Account
- 2. Create Terraform Backend S3 Bucket
- 3. Create a New GitHub Repository
- 4. Install and Authorize the Amplify GitHub App for Deployment
Scaffolding Vue Project
Setup Terraform Infrastructure
Provision Terraform Infrastructure using CLI
- Option 1. From Local Machine
- Option 2. Via GitHub Actions
Demo Project
Next Step
Summary

The Demo Repo: https://github.com/camillehe1992/amplify-vue-app

Prerequisites

A GitHub account
- permission to create an empty repository
- permission to create and manage personal access token
- permission to create and manage GitHub Actions workflow
An AWS account
- administrative permissions for AWS services, such as Amplify, IAM, S3, etc.
Desktop:
- a local machine with npm, nodejs installed.
- VSCode IDE (Optional, but best to have)

Preparation

Here are some manual preparation work.

1. Create PAT in GitHub Account

Personal access tokens are intended to access GitHub resources on behalf of yourself. With personal access token, you can manage your GitHub source code using Git CLI from local machine. Follow the official tutorial to generate a fine-grained personal access token.

Copy the token and save it somewhere else as the token will gone if the browser is refreshed. The token will be used in the variable ACCESS_TOKEN of Terraform infrastructure, and in Git commands.

2. Create Terraform Backend S3 Bucket

As our application is provisioned using Terraform, we should create a S3 bucket manually from AWS console as the Terraform backend to store state files.
From AWS S3 console. Click on "Create Bucket" button, enter a bucket name with the AWS account id and region to make it unique in the global, for example terraform-state-{aws-account-id}-{aws-region}. The bucket will be used in the environment variable of STATE_BUCKET of Terraform CLI.

3. Create a New GitHub Repository

Create a GitHub repository from GitHub console https://github.com/new. In the demo, I created a repository in my personal GitHub account with name amplify-vue-app as below.

4. Install and Authorize the Amplify GitHub App for Deployment

Before you deploy a new app to Amplify from existing code in a GitHub repo, follow instructions to install and authorize the GitHub App. After completed, Amplify will be granted all necessary permissions to interact with GitHub repository you configured. For example, pull source code, listen on GitHub events, etc.
From GitHub -> Settings -> Integrations -> Applications, a GitHub App as below is installed successfully. You can view and update the Amplify GitHub App from here as needed.

Notes:
If you missed the step, you will be asked to install and authorize the Amplify GitHub App from AWS Amplify console when your Amplify app firstly launched.
If the step is already completed, but the deployment fail with error "[ERROR]: !!! Unable to assume specified IAM Role. Please ensure the selected IAM Role has sufficient permissions and the Trust Relationship is configured correctly", double check if your new repository is in the list of Repository access of above GitHub App.

Scaffolding Vue Project

Step 1. Clone your new repository to local machine. You may be asked for the username and password. The username is the GitHub account username, password is the personal access token (PAT) that generated in Prep 1.



git clone https://github.com/camillehe1992/amplify-vue-app.git
cd amplify-vue-app

Step 2. Scaffolding your Vue project using Vite. See the detailed information from https://vitejs.dev/guide/



npm create vite@latest . -- --template vue
npm install
npm run dev

The Vue project is listening on http://localhost:5173/ as below.

Step 3. Commit the code change and push to remote repository.



git add .
git commit -m "scaffolding vue project"
git push

Now we have setup a simple Vue project.

Setup Terraform Infrastructure

We use Terraform to provision and manage AWS infrastructure, including Amplify App. All Terraform infrastructure related code is under terraform directory. You can find the detailed description of Terraform infrastructure from repository documentation.



.

├── .env.sample              # Environment variables for Terraform

├── .terraform-docs.yaml     # terraform-docs configuration file

├── .terraform.lock.hcl      # Auto generated The dependency lock file       

├── Terraform.md

├── amplify.yaml             # The build specification for an Amplify app

├── main.tf                  # Terraform resources definition

├── outputs.tf               # Terraform outputs

├── tf_dev.tfvars            # Terraform variables for dev env

├── tf_prod.tfvars           # Terraform variables for prod env

├── variables.tf             # Terraform input variable definition

└── versions.tf              # Terraform providers and backend configuration

Provision Terraform Infrastructure using CLI

Terraform provides CLI to manage its infrastructure. See basic Terraform CLI features from official documentation. With Terraform CLI, you can provision AWS infrastructure from local machine or via GitHub Actions workflows.

Option 1. From Local Machine

To provision Amplify infrastructure from local machine, follow the tutorial to setup below configurations:

Install Terrafrom CLI
Install AWS CLI
Configurate AWS Credentials
Execute Make Commands

After executing make quick-deploy command, an Amplify App named dev-amplify-vue-app is created in AWS Amplify. This Amplify app is for development only, and there is another Amplify app named prod-amplify-vue-app for production environment that will be created using GitHub Actions workflow later.

As you can see, there are two Amplify apps here, which I called Multi-Apps architecture. The architecture simulates the multiple environments in a real project. The production branch in dev-amplify-vue-app is develop, which targets to develop branch in GitHub repository. The production branch in prod-amplify-vue-app is main, which targets in main branch accordingly. In Amplify, each branch has an environment, that maintains in the backend.
In the Multi-Apps architecture solution, when new commits are pushed to the GitHub repository develop branch, a new deployment on the develop branch in Amplify App dev-amplify-vue-app will be triggered automatically. AWS Amplify pulls source code from repository develop branch, then build and deploy it to development environment. After deployment completed successfully, changes in new commits will be available via Amplify app dev-amplify-vue-app default domain url.
Developers can validate the new changes in development environment. If everything works as expected, developers can create a pull request from develop to main branch in repository.
A dedicated branch with prefix pr created for each pull request in Amplify app in the target branch. For example, a pull request from develop to main branch will create a PR branch named pr-2 in prod-amplify-vue-app. No.2 is the order number in GitHub -> Pull Request. Developers can validate the change from the domain url before merging the pull request.

The deployment result and domain url for pull request information is appended in the pull request conversion as below.

Amplify app prod-amplify-vue-app is for production environment, and we disabled the automate build, you should trigger a new deployment from AWS console manually. After completed, the code changes are available in production environment.
Below diagram illustrate the development and deployment workflow in the whole lifecycle.

The benefits of Multi-Apps architecture:

A dedicated Amplify app for each environment.
Configuration flexibility on Amplify app for different environments.
Easy to maintain, easy to scale out (add a new environment, such as testing, or staging).
Control management, for example, secure Amplify app for production with restricted access if only allowed people or role can manual trigger a new deployment in production environment.

Option 2. Via GitHub Actions

Now, you know how to provision Amplify infrastructure from local machine, and the workflow between GitHub and AWS Amplify. Next, let's create GitHub Actions workflows to automate the Terraform infrastructure provision and management process.
You should be familiar with the concept of Infrastructure of Code (IoC). IaC is the ability to provision and support your computing infrastructure using code instead of manual processes and settings. With Terraform and GitHub Actions, we can provision and manage Amplify infrastructure automatically.

Follow the documentation to provision AWS infrastructure via GitHub Actions workflows.

Demo Project

After provisioning Amplify App prod-amplify-vue-app for production via GitHub Actions workflow, you should get two Amplify App in AWS. Here is the domain url for each environment.

Next Step

Now, we have setup a basic Amplify Multi-Apps architecture solution following infrastructure as code (IaC). You can do some enhancements on top of it, for example,

Add an Amplify Domain Association resource on Amplify app.
Add backend resources on Amplify app.
Add notification process using SNS, CloudWatch Event Bridge, etc, or enable Amplify WebHook.
Refactor the Single-Account Multi-Apps solution as needed, for example, a dedicated AWS account for Amplify app for specific development. With the Multi-Accounts solution as below, you can grant different permissions for users or roles for best practice. It's suitable to setup an central platform that provisions and manages extensible, Full-Stack Web and Mobile Apps powered by AWS Amplify service in enterprise scope, and integrates with other frameworks or tools to provide one-stop service.

Summary

In the end, you have learned:

How to setup a Vue based frontend template project from scratch.
How to define and manage AWS infrastructure using Terraform.
How to provision and manage AWS infrastructure via GitHub Actions workflow.

Thanks for reading and looking forward to your ideas and comments.
Happy learning, and happy coding!

Scaffolding Serverless Web Application on AWS

Camille He — Mon, 22 Jul 2024 07:32:57 +0000

In this article, I'm going to setup a serverless based web application on AWS Cloud. Firstly I will explain its AWS cloud architecture and infrastructure that used in the application. Then, dive into the details of AWS services. Finally, go through the source code, and guide you to setup your own application from scratch. In the end, you should know:

How to design a serverless web application using API Gateway, Lambda and DynamoDB.
How to provision and organize cloud infrastructure using Terraform.
How to use Lambda PowerTools to organize and fulfill HTTP/HTTPS requests/responses.
How to use PynamoDB to interact with DynamoDB using an ORM-like interface.

Demo Application
AWS Infrastructure
- Core Layer - API Gateway & Lambda
- Database Layer - DynamoDB
- Access Control Layer - IAM
Source Code
- Terraform
- Lambda Code
- Others
Development & Deployment
Summary
References

Let's get started!

Demo Application

The demo application provides a simple “to-do list” backend interface that supports CURD operations. Here is the screenshot of Swagger UI.

AWS Infrastructure

The serverless web application is built with several AWS services together. Below diagram shows how these AWS services integrate with each other. It contains four layers:

Core Layer: includes API Gateway and Lambda function, layers.
Database Layer: DynamoDB tables.
Access Control Layer: IAM roles for Lambda function.
Monitoring Layer: CloudWatch Log groups for API Gateway and Lambda function.

Core Layer - API Gateway & Lambda

Create a Restful API in API Gateway, and a function and layers in Lambda.
API Gateway provide APIs that act as the "front door" for applications to access data, business logic, or functionality from your backend services.

From the screenshot, each API method has a Lambda integration . Any GET /todos request flows into integrated Lambda function. The function fulfills the request and get todo items from DynamoDB table.

Database Layer - DynamoDB

Amazon DynamoDB is a serverless, NoSQL database service that enables you to develop modern applications at any scale. Two tables are created.

Access Control Layer - IAM

With AWS Identity and Access Management (IAM), you can specify who or what can access services and resources in AWS, centrally manage fine-grained permissions, and analyze access to refine permissions across AWS.
Two IAM roles are created: API Gateway logging role and Lambda function execution role.

API Gateway Logging Role

In order to collect logs from API Gateway, I grant API Gateway permission to read and write logs to CloudWatch in the account. The role takes effect on the account level, so I created manually from AWS console before the demo. Follow the official tutorial to create the role.

Lambda Function Execution Role

A Lambda function's execution role is an AWS IAM role that grants the function permission to access AWS services and resources. For example, you might create an execution role that has permission to send logs to Amazon CloudWatch. In the demo, we should grant DynamoDB permissions to the role.

Monitoring Layer - CloudWatch

As we grant API Gateway and Lambda function permissions to write logs to CloudWatch Logs, we still need to create CloudWatch Log groups for them. You can find the logs for investigation or debugging.

Next section, I'll introduce the source code of Lambda function, and AWS infrastructure. You can find the source code from https://github.com/camillehe1992/scaffolding-serverless-project-on-aws.

Source Code

As I mentioned in the beginning, although we don't need to provision and manage servers, we still need to create AWS infrastructure manually from AWS console, or manage infrastructure as code (IaC) using CDK, CloudFormation or Terraform, etc. I use Terraform to define all AWS infrastructure and deploy them into AWS via Terraform CLI in GitHub Actions workflow. Except for Terraform code, the Lambda function source code is important as well.
I will introduce Terraform and Lambda function code structure and python dependencies I used. These dependencies make the code readable, maintainable and be organized. Greatly reduce the workload of development in a real project of your work.

Terraform

Terraform is an infrastructure as code tool that lets you build, change, and version infrastructure safely and efficiently. Terraform code is in the terraform directory.



.
├── deployments     # Define AWS resources and data with Terraform scripts     
│   ├── api
│   ├── common_infra
│   └── dynamodb
├── modules        # Define Terraform modules for AWS resources
│   ├── README.md
│   ├── api_gateway
│   ├── dynamodb
│   ├── iam
│   ├── lambda_function
│   ├── lambda_layer
│   └── vpc_endpoint
└── settings.     # Define Terraform variables for each environment
    ├── dev
    └── prod

I won't dive into the details of Terraform scripts here, which are not in the scope. The key point I want to share, from my experience, you should organize Terraform code according to the cloud architecture. For example, separate AWS infrastructure into several groups, so that they can be managed and provisioned individually, especially when there is a bunch of infrastructure in the cloud architecture. Use Terraform modules to reduce code redundancy and operational and maintenance cost.

Terraform Modules are containers for multiple resources that are used together in a configuration. https://developer.hashicorp.com/terraform/language/modules#modules

Lambda Code



# File structure in src directory
.
├── __init__.py
├── portal                 # Lambda function source code
│   ├── __init__.py
│   ├── app
│   └── requirements.txt
└── tests                  # All test related source code
    ├── e2e
    ├── local
    ├── thunder
    └── unit

The core lambda logic code is in src.portal.app directory.



# In src.portal
.
├── __init__.py
├── app
│   ├── __init__.py
│   ├── database
│   ├── enum.py
│   ├── logging.py
│   ├── main.py
│   ├── models
│   └── routers
└── requirements.txt

The main Python dependencies I use in the serverless web application are Lambda PowerTools and PynamoDB.

Lambda PowerTools

https://docs.powertools.aws.dev/lambda/python/latest/
Lambda PowerTools provides many amazing features, such as validation, logging, event handler, enable swagger ui etc. It makes the serverless code work as web application framework that you are more familiar with, for example Flask or FastAPI. The entry of the application is lambda_handler function in app.main.py file.

PynamoDB

https://github.com/pynamodb/PynamoDB
A pythonic interface to Amazon's DynamoDB, that provide an ORM-like interface with query and scan filters. It supports many features which makes it more comfortable to interact with DynamoDB API.

Others

Except for Lambda and Terraform source code, there are some development configurations.

cloudformation/infra.yaml: Define the S3 bucket and DynomoDB lock table for Terraform backend configuration.
Makefile: Define make script to simplify your local deployment using shell scripts.
.pylinrc, .pytest.ini, .pre-commit-config.yaml etc : Testing, linting and formatting configuration for python code.
.github/actions: automate workflows to deploy/destroy infrastructure to/from AWS account.

Development & Deployment

For local development, you can follow the Development documentation to setup environment on your local machine.
You can also deploy the serverless web application from your local machine or via GitHub Actions following Deployment documentation.

Summary

In the end, you should learned:

Design a serverless web application architecture using API Gateway, Lambda and DynamoDB.
Provision and organize cloud infrastructure using Terraform.
Use Lambda PowerTools to fulfill core logic.
Use PynamoDB to interact with DynamoDB using an ORM-like interface.

Besides, there are tips you'd better to know.

AWS officially provides Lambda layers for Lambda PowerTools https://docs.powertools.aws.dev/lambda/python/2.26.1/#lambda-layer
API Gateway API methods and resource can be generated from swagger.yaml. https://github.com/camillehe1992/scaffolding-serverless-project-on-aws/blob/main/terraform/deployments/api/swagger.yaml

References

Thanks for reading and looking forward to your ideas and advices.

AWS S3 Bucket Website Hosting using Terraform

Camille He — Thu, 27 Jun 2024 03:59:47 +0000

In previous blog Deploy Terraform resources to AWS using GitHub Actions via OIDC, I explained how to configure OpenID Connect within GitHub Actions workflows to authenticate with AWS, then demonstrated the process using a very simple Actions workflow that lists all buckets in my AWS account. As I mentioned, the common use case in real world is we define AWS infrastructure as code, provision AWS resources automatically and manage these cloud infrastructure in GitHub.

In this article, I’ll go one step further. Use GitHub Actions workflow to provision and manage a S3 website using S3 website static hosting feature. The S3 bucket related AWS infrastructure is defined using Terraform. The website content and Terraform code are saved in GitHub. Any code change will trigger a new workflow build automatically to sync its infrastructure status in AWS. After done, your website can be available through bucket static website endpoint from browser.

The whole process includes three sections:

Infrastructure as Code: Define AWS infrastructure as code using Terraform.
Website Static Content: Create index.html and 404.html files as the website static content.
GitHub Actions workflow: Create workflow to provision AWS infrastructure to AWS.

Let's get started.

Prerequisites

1. Select Tools

Terraform: I choose Terraform to provision and manage AWS infrastructure. You can use any other tools as you want, such as CloudFormation, CDK etc. The core concept is the same. Define your cloud infrastructure as code.
GitHub: Manage source code in a version control system, such as GItHub, Bitbucket, AWS CodeCommit, or Gitlab, etc.
GitHub Actions: CICD pipeline management tool, such as GitHub Actions, Jenkins, Bitbucket pipeline, AWS CodeBuild, etc.

2. Create Terraform Backend S3 Bucket

As Terraform uses persisted state data to keep track of the resources it manages, we use a backend to store state remotely. S3 bucket is commonly used to save the state of Terraform infrastructure in AWS. You can create a S3 bucket manually from AWS console, Click on Create bucket button, enter a meaningful bucket name, to make it simple, just keep all configuration as default, and click Create bucket.

The bucket name will be used in Step 3 workflow environment variable TF_BACKEND_S3_BUCKET.

3. Attach Policy on Deployment IAM Role

Remember we created a dedicated IAM role for deployment named GitHubAction-AssumeRoleWithAction in previous blog. In order to provision all AWS infrastructure that we used in this demo, you should add polices on the role. The easiest way is to attach an AWS managed policy AmazonS3FullAccess which grants full access to all buckets in your AWS account.

After done, move to coding part. You can find the sample code from GitHub repo: https://github.com/camillehe1992/demo-for-aws-deployment-via-oidc

Step 1. Infrastructure as Code

I use Terraform to define all AWS infrastructure as code, so that AWS resources can be provisioned automatically through Actions workflow and be managed in GitHub. I won’t dive into the Terraform code, as that's not in the scope. All terraform related files are in terraform directory as below.

└── terraform
    ├── local.tf
    ├── main.tf
    ├── mime.json
    ├── outputs.tf
    ├── prod.tfvars
    ├── providers.tf
    └── variables.tf

Step 2. Website Static Content

Now, let’s prepare our demo website content. In public directory, create index.html, 404.html and image files. All the files in the directory are uploaded to S3 bucket as the website static content.

├── public
│   ├── 404.html
│   ├── images
│   │   ├── coffee.jpg
│   │   └── dogs.jpg
│   └── index.html

Step 3. GitHub Actions Workflow

I created a new Actions workflow named deploy.yaml in .github/workflows directory.

├── .github
│   └── workflows
│       ├── deploy.yaml
│       └── get-started.yaml
...

Comparing with get-started.yaml workflow, Here are main updates:

Add new environment variables in env block and configure these variables in GitHub Settings -> Secrets and variables -> Variable -> repository variable.

env:
  ...
  TF_BACKEND_S3_BUCKET: ${{ vars.TF_BACKEND_S3_BUCKET }}
  ENVIRONMENT: prod
  NICKNAME: demo-for-aws-deployment-via-oidc

TF_BACKEND_S3_BUCKET: the S3 bucket name of Terraform state files.
ENVIRONMENT: is part of Terraform backend S3 object key. Meanwhile, it's used as the suffix of website bucket name.
NICKNAME: is part of Terraform backend S3 object key.

Add three steps in job block after authentication:

      - name: Terraform init
        working-directory: terraform
        run: |
          terraform init -reconfigure \
            -backend-config="bucket=$TF_BACKEND_S3_BUCKET" \
            -backend-config="region=$AWS_REGION" \
            -backend-config="key=$NICKNAME/prod/$AWS_REGION/terraform.tfstate"
      # An exit code of 0 indicated no changes, 1 a terraform failure, 2 there are pending changes.
      - name: Terraform plan
        id: tf-plan
        working-directory: terraform
        run: |
          export exitcode=0

          terraform plan \
            -var-file=$ENVIRONMENT.tfvars -detailed-exitcode -no-color -out tfplan || export exitcode=$?

          echo "exitcode=$exitcode" >> $GITHUB_OUTPUT

          if [ $exitcode -eq 1 ]; then
            echo Terraform Plan Failed!
            exit 1
          else
            exit 0
          fi
      # Apply the pending changes
      - name: Terraform apply
        if: ${{ steps.tf-plan.outputs.exitcode == 2 }}
        working-directory: terraform
        run: |
          terraform apply -auto-approve tfplan -no-color

Steps:
Terraform init: Run terraform init CLI with backend configuration
Terraform plan: Run terraform plan CLI to generate a plan.
Terraform apply: Run terraform apply CLI to apply the plan if there is pending change on it.

Commit and push to remote. A new workflow appears in Actions -> workflow named Deploy Static Website with running build.

Here is the detail steps:

All content in S3 bucket:

You can find the website_endpoint from the end of Terraform apply logs. Or go to AWS console, find your website bucket -> Properties. Scroll to the bottom of the page, then you will find the Bucket website endpoint. Depending on your Region, your Amazon S3 website endpoint follows one of these two formats.

You can visit your website from browser now. Add path after the endpoint that doesn't exist, an 404 page is returned.

Correct URL:

Incorrect URL:

Summary

You should know how to provision and manage AWS infrastructure using GitHub and Terraform. Now you can provision more AWS services or even other Cloud infrastructure as you want following the same methodology.

References

https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html

Thanks for reading!

Deploy Terraform resources to AWS using GitHub Actions via OIDC

Camille He — Thu, 27 Jun 2024 01:56:51 +0000

The article explains how to configure OpenID Connect within your GitHub Actions workflows to authenticate with AWS, so that the workflow can access AWS resources. The common use case is define AWS infrastructure as code, using CloudFormation, CDK or Terraform, etc, then sync the infrastructure update in AWS through workflows on each code change commit. As we only focus on the IODC setup here, to make it simple, the demo workflow authenticates to AWS account firstly, then list all S3 buckets in that account.

So, what is OpenID Connect?

OpenID Connect (OIDC) is an identity authentication protocol that is an extension of open authorization (OAuth) 2.0 to standardize the process for authenticating and authorizing users when they sign in to access digital services. OIDC provides authentication, which means verifying that users are who they say they are.

Let's talk about how OpenID Connect works with identity provider and federation.

In AWS, when managing user identities outside of AWS, you can use identity providers instead of creating IAM users in AWS account. With an identity provider (IdP), you can give these external user identities permissions (defined in IAM role) to use AWS resources in your account. An external IdP provides identity information to AWS using either OpenID Connect (OIDC) or SAML 2.0. Identity providers help keep your AWS account secure because you don't have to distribute or embed long-term security credentials, such as access keys, in your application.

In the demo, GitHub is an external identity provider for AWS. GitHub Actions workflows can be treated as external user identities.

The core process is to authenticate with AWS using temporary credentials within your GitHub Actions workflows. It contains the following steps:

Firstly, establish trust between AWS account and GitHub by adding GitHub identity provider in AWS IAM service.
Create IAM role that allows to be assumed by the new added identity provider.
GitHub actions workflow assumes the IAM role:
- Workflow retrieves a JWT token from GitHub;
- Workflow makes an AssumeRoleWithWebIdentity call to AWS STS service with JWT token.
- AWS STS service validates the trust relationship, and returns temporary credentials in AWS that map to the IAM role with permissions to access specific resources in AWS account
Workflow access AWS resources.

Here is a diagram that displays the authentication process.

Prerequisites

An AWS account with permission to create OIDC identity provider, role, attach policy in AWS IAM service.
An GitHub account to create a repository and workflows.

Solution Overview

Step 1 & 2 Add GitHub IdP & Create IAM Role in AWS Account

You can follow the process and description from the blog just like what I did. I won't repeat the process here because the blog is clear and understandable.

After completed, you will have an identity provider named token.actions.githubusercontent.com with OpenID Connect type, and an IAM role named GitHubAction-AssumeRoleWithAction with trust relationship.

Identity Provider

IAM Role -> Trust Relationship

Step 3. Create GitHub Actions Workflow

In your GitHub repository, create a workflow yaml file named get-started.yaml in .github/workflows directory with below code.

Update and with the real values. Update the branches in the code if you want to trigger the workflow from other branches.



# This is a basic workflow to help you get started with Actions
name: Get Started
# Controls when the action will run. Invokes the workflow on push events but only for the main branch
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
env:
  AWS_REGION: <YOUR_AWS_ACCOUNT_REGION>
  ROLE_TO_ASSUME: arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:role/GitHubAction-AssumeRoleWithAction
  ROLE_SESSION_NAME: GitHub_to_AWS_via_FederatedOIDC
# Permission can be added at job level or workflow level
permissions:
  id-token: write # This is required for requesting the JWT
  contents: read # This is required for actions/checkout
jobs:
  AssumeRoleAndCallIdentity:
    runs-on: ubuntu-latest
    steps:
      - name: Git clone the repository
        uses: actions/checkout@v4
      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ env.ROLE_TO_ASSUME }}
          role-session-name: ${{ env.ROLE_SESSION_NAME }}
          aws-region: ${{ env.AWS_REGION }}
      # Hello from AWS: WhoAmI
      - name: Sts GetCallerIdentity
        run: |
          aws sts get-caller-identity

Commit and push to remote. A build is triggered automatically. Below screenshot shows the workflow assumes the IAM role ROLE_TO_ASSUME successfully.

At this moment, the IAM role doesn't attach any policy, which means your workflow has no permissions for AWS resources.

Next, I'm going to use the workflow to list all buckets in my AWS account. To enable it, we need to attach an IAM policy with necessary permission on IAM role GitHubAction-AssumeRoleWithAction.

From AWS IAM Console, find role, Add permissions -> create inline policy.

Select a service: S3
Action allowed: ListAllMyBuckets
Resources: All
Next
Policy name: AllowListAllMyBuckets
Create Policy

Please be noted, to list all AWS buckets, we choose action s3:ListAllMyBuckets, not s3:ListBucket.

Now, you have attached an inline policy on IAM role. Then add a new step after step Sts GetCallerIdentity to list all buckets using AWS CLI. The step name is List All S3 Buckets, it executes shell script "aws s3 ls" to list all buckets from your AWS account.



...
- name: List All S3 Buckets
  run: aws s3 ls

Commit and push change to remote. A new build is triggered automatically.

Following the least privilege principle, I only granted s3:ListAllMyBuckets permission to the role. The policies assigned to the role determine what the federated users are allowed to do in AWS. You should follow the principle for best practice according to your needs in the daily work.

Others...

Since we are taking about best practice, let's enhance our workflow by replacing hard-coded or sensitive environment variables with GitHub Secrets and variables. I'm going to save these environment variables in Settings -> Secrets and variables -> Actions. Add repository variables.

Finally replace the hard-coded environment variables in workflow.

Hard-coded:



AWS_REGION: xxxxxx
ROLE_TO_ASSUME: arn:aws:iam::xxxxxxxxxxx:role/GitHubAction-AssumeRoleWithAction
ROLE_SESSION_NAME: GitHub_to_AWS_via_FederatedOIDC

Retrieved from GitHub repository variables



AWS_REGION: ${{ vars.AWS_REGION }}
ROLE_TO_ASSUME: ${{ vars.ROLE_TO_ASSUME }}
ROLE_SESSION_NAME: ${{ vars.ROLE_SESSION_NAME }}

Commit and push to remote. A new build is triggered automatically. The workflow will retrieve these variables from GitHub Secrets and variables at the beginning of job.

You can find the source code from GitHub repo: https://github.com/camillehe1992/demo-for-aws-deployment-via-oidc

Summary

You have learned how to add GitHub as an identity provider and create an IAM role with correct trust relationship in AWS. Besides, you created a GitHub Actions workflow that authenticates to AWS and list all S3 buckets in the AWS account within the build.

As I mentioned, the common use case is we define AWS infrastructure as code, these AWS resources are provisioned and managed through GitHub repositories and Actions workflows, a.k.a CICD pipelines. All changes are traceable and controllable, easy to repeat and recover with the power of IoC (Infrastructure as Code) and CICD tools.

References

https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/
https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers.html
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc.html

Thanks for reading!

Mask Sensitive Data using Python Built-in Logging Module

Camille He — Tue, 04 Jun 2024 07:19:12 +0000

Logging is an essential and common topic in software development, which play a vital role in monitoring, debugging and troubleshooting applications. If logs are not properly secured or managed, they can become a target for hackers and other malicious actors who may attempt to gain access to this sensitive data. By keeping sensitive data out of logs, you can help protect users' privacy and reduce the risk of data breaches or other security incidents.

There are lots of best practices for logging for different programming languages, and this article only focus on masking sensitive data in python using python built-in module - logging.

Here is the demo source code in GitHub.

Let's get started.

Initialize Logging Configuration

First of all, we create a file named log.py with a log configuration as below. The outputs are formatted according to the formatter argument. The default is "console".



# import modules
import re
import time
import logging
import logging.config

def init_logging(
    log_level: str = "DEBUG", formatter: str = "console"
) -> logging.Logger:
 
    LOG_CONFIG = {
        "version": 1,
        "handlers": {
            "stdout": {
                "class": "logging.StreamHandler",
                "stream": "ext://sys.stdout",
                "formatter": formatter,
            }
        },        "formatters": {
            "json": {
                "format": (
                    '{"msg":"%(message)s","level":"%(levelname)s",'
                    '"file":"%(filename)s","line":%(lineno)d,'
                    '"module":"%(module)s","func":"%(funcName)s"}'
                ),
                "datefmt": "%Y-%m-%dT%H:%M:%SZ",
            },
            "console": {
                "format": "%(asctime)s %(levelname)s : %(message)s",
                "datefmt": "%Y-%m-%dT%H:%M:%SZ",
            },
        },
        "root": {"handlers": ["stdout"], "level": log_level},
    }
    logging.Formatter.converter = time.gmtime
    logging.config.dictConfig(LOG_CONFIG)
    logger = logging.getLogger(__name__)
 
    return logger

Then, create a file named main.py with below code.



from log import init_logging

LOG = init_logging()
 
if __name__ == "__main__":
    test_case = (
        "John [513-84-7329] made a payment with credit card 1234-5678-9012-3456."
    )
    LOG.info(test_case)

Outputs:



2024-06-03T06:45:38Z INFO : John [513-84-7329] made a payment with credit card 1234-5678-9012-3456.

Obviously, there are sensitive data in logs. 513-84-7329 is U.S. social security number, 1234-5678-9012-3456 is a credit card number. These data are highly confidential and should be masked or redacted with a series of asterisks or replace them with a hash value.

Although filters are used primarily to filter records based on more sophisticated criteria than levels, they get to see every record which is processed by the handler or logger they’re attached to: this can be useful if you want to do things like counting how many records were processed by a particular logger or handler, or adding, changing or removing attributes in the LogRecord being processed.

From logging.Filter https://docs.python.org/3/library/logging.html#filter-objects

According to the above description, I use logging.Filter to mask sensitive data in logs.

Setup logging.Filter in Logger Configuration

Create a filter and configure it in log configuration. Add below code in log.py file.




# 1. Define the regex patterns of sensitive data
regex_patterns = [
    # U.S. Social Security numbers
    r"\d{3}-\d{2}-\d{4}",
    # Credit card numbers
    r"\d{4}-\d{4}-\d{4}-\d{4}",
]
 
# 2. Define a filter class to mask sensitive data that match the pre-defined regex patterns
class SensitiveDataFilter(logging.Filter):
    patterns = regex_patterns
    def filter(self, record):
        record.msg = self.mask_sensitive_data(record.msg)
        return True
    def mask_sensitive_data(self, message):
        for pattern in self.patterns:
            message = re.sub(pattern, "******", message)
        return message

# 3. Add filter configuration in LOG_CONFIG
LOG_CONFIG = {
        "version": 1,
        "handlers": {
            "stdout": {
                ...
                # setup filters
                "filters": ["sensitive_data_filter"],
            }
        },
        # add filters in configuration
        "filters": {
            "sensitive_data_filter": {
                "()": SensitiveDataFilter,
            }
        },
        "formatters": {...},
        "root": {...},
    }

Replace main.py with below code. We print message to console using three string formatting methods for comparison. All work.

Read Python String Formatting Best Practices if you are interested in string formatting in python.



from log import init_logging

LOG = init_logging()

if __name__ == "__main__":
    test_case = (
        "John [513-84-7329] made a payment with credit card 1234-5678-9012-3456."
    )
    LOG.debug(f"use f-string: {test_case}")
    LOG.debug("use str.format: {}".format(test_case))
    LOG.debug("use string modulo method: %s" % (test_case))

Outputs:



2024-06-03T07:05:26Z DEBUG : use f-string: John [******] made a payment with credit card ******.
2024-06-03T07:05:26Z DEBUG : use str.format: John [******] made a payment with credit card ******.
2024-06-03T07:05:26Z DEBUG : use string modulo method: John [******] made a payment with credit card ******.

Updates:

Define the regex patterns of sensitive data
Define a filter class SensitiveDataFilter to mask sensitive data that match the regex patterns
Add and setup filter configuration in LOG_CONFIG

The masking process is clear. We define the regex patterns for sensitive data, and replace them with asterisks.

Mask sensitive data in the values of dictionary

In above examples, the sensitive data have regex patterns that can be defined easily. However, sometimes our sensitive data is a random string, for example, we need to print test_case as below and mask the password only. We cannot define a regex pattern for password string 'fFwpUd!CJT4'.



test_case = {'username': 'John Doe', 'password': 'fFwpUd!CJT4'}

After diving into logging module, I found two solutions to mask sensitive data in a dictionary. They may not be the best, but they did solve my problem.

Option 1. Mask sensitive data in record.msg matching regex pattern

The first solution use regex pattern as well, but this time the sensitive data is located by it's key in dictionary.

Let's update class SensitiveDataFilter. Add below code in log.py file.




# Define a list of keys that values are sensitive data
sensitive_keys = (
    "headers",
    "credentials",
    "Authorization",
    "token",
    "password",
)


# mask sensitive data in record.msg
class SensitiveDataFilter2(logging.Filter):
    patterns = regex_patterns
    sensitive_keys = sensitive_keys

    def filter(self, record):
        record.msg = self.mask_sensitive_msg(record.msg)
        return True

    def mask_sensitive_msg(self, message):
        for pattern in self.patterns:
            message = re.sub(pattern, "******", message)
        # replace sensitive data with asterisks
        for key in self.sensitive_keys:
            pattern_str = rf"'{key}': '[^']+'"
            replace = f"'{key}': '******'"
            message = re.sub(pattern_str, replace, message)
        return message

Updates:

Define a list of keys that values are sensitive data. These keys are used to locate the sensitive data in dictionary.
Mask sensitive data in dictionary.values() by processing record.msg. In function mask_sensitive_msg, we define the pattern string '{key}': '[^']+'. And replace the match string with '{key}': ******.

Outputs:



2024-06-03T07:33:15Z DEBUG : use f-string: {'username': 'John Doe', 'password': '******'}
2024-06-03T07:33:15Z DEBUG : use str.format: {'username': 'John Doe', 'password': '******'}
2024-06-03T07:33:15Z DEBUG : use string modulo method: {'username': 'John Doe', 'password': '******'}

In the solution, the dictionary object is treated as a part of log message essentially. The only difference is the regex patterns. However, it's not stable from my perspective, causes errors for special scenarios. So I tried another solution.

Option 2. Mask sensitive data in record.args

When going through the logging module documentation, I found that, for a LogRecord, the primary information is passed in msg and args, which are combined using msg % args to create the message attribute of the record. With args, we can process the sensitive data before being merged into msg. Here is the description of msg and args fields in a LogRecord object.

msg (Any) – The event description message, which can be a %-format string with placeholders for variable data, or an arbitrary object (see Using arbitrary objects as messages).
args (tuple | dict[str, Any]) – Variable data to merge into the msg argument to obtain the event description.

Interesting, right? Let's update class SensitiveDataFilter as below to make it work.



class SensitiveDataFilter(logging.Filter):
    patterns = regex_patterns
    sensitive_keys = sensitive_keys
    def filter(self, record):
        try:
            record.args = self.mask_sensitive_args(record.args)
            record.msg = self.mask_sensitive_msg(record.msg)
            return True
        except Exception as e:
            return True
    def mask_sensitive_args(self, args):
        if isinstance(args, dict):
            new_args = args.copy()
            for key in args.keys():
                if key in sensitive_keys:
                    new_args[key] = "******"
                else:
                    # mask sensitive data in dict values
                    new_args[key] = self.mask_sensitive_msg(args[key])
            return new_args
        # when there are multi arg in record.args
        return tuple([self.mask_sensitive_msg(arg) for arg in args])
    def mask_sensitive_msg(self, message):
        # mask sensitive data in multi record.args
        if isinstance(message, dict):
            return self.mask_sensitive_args(message)
        if isinstance(message, str):
            for pattern in self.patterns:
                message = re.sub(pattern, "******", message)
            for key in self.sensitive_keys:
                pattern_str = rf"'{key}': '[^']+'"
                replace = f"'{key}': '******'"
                message = re.sub(pattern_str, replace, message)
        return message

Replace main.py with below code.



from log import init_logging

LOG = init_logging()

if __name__ == "__main__":
    test_case = {"username": "John Doe", "password": "xyz"}
    LOG.debug("use args: %s", test_case)
    LOG.debug("use multi args: %s %s", test_case, test_case)

Outputs:



2024-06-03T09:41:02Z DEBUG : use args: {'username': 'John Doe', 'password': '******'}
2024-06-03T09:41:02Z DEBUG : use multi args: {'username': 'John Doe', 'password': '******'} {'username': 'John Doe', 'password': '******'}

Let's go through the updates.

I created a new function named mask_sensitive_args to process variable data in record.arg:
When args is dictionary: Iterate all key, value pair of dict. replace the sensitive value if the key in the sensitive_keys.
When args is a tuple: Iterate tuple. The item of each tuple may be string or dict.
- If item is a string, mask sensitive data using regex patterns
- If item is a dict, use mask_sensitive_args

Modify existing function mask_sensitive_msg in order to support multiple args scenarios.
Wrap the filter function with a try-catch block. We catch all exception when masking data, and print the data without masking if there is any error occur during masking.

Summary

When masking sensitive data in logs, choose regex pattern if the data match common pattern, for example SSN, or credit card.
When masking sensitive data (random string) in a dictionary, utilize sensitive key name to locate sensitive value or leverage record.args.
Always put an eye on your logs.

References:
Keep sensitive information out of logs

Thanks for reading. Looking forward to your comments and ideas.

Setup containerized Application in AWS ECS - Part 3/3

Camille He — Fri, 08 Dec 2023 05:38:46 +0000

In the previous Part 2/3, I introduced how to setup a ECS Cluster in AWS ECS. Now, we will dive into next and also the final topic - Setup ECS Service related resources in AWS ECS.

⚓ Amazon ECS Concepts

The AWS resources in green box are created in this part, which include:

ECS Task Definition & ECS Service

A task definition is a blueprint for your application. It is a text file in JSON format that describes the parameters and one or more containers that form your application.

ECS service is used to run and maintain a specified number of instances of a task definition simultaneously in an Amazon ECS cluster.

ECS Task definition is the core component for your containerization application. All the container's related parameters are defined in container_definitions parameter of resource aws_ecs_task_definition. For example, the docker image, resource (CPU, Memory) allocation, log configuration, container environment variables, etc.

ALB & Target Group & Listener

A load balancer serves as the single point of contact for clients. The load balancer distributes incoming application traffic across multiple targets.

A listener checks for connection requests from clients, using the protocol and port that you configure.

Each target group routes requests to one or more registered targets, such as EC2 instances, using the protocol and port number that you specify.

Now we have some containers running in the ECS Cluster. You can access a specific container via the public IP address of the EC2 instance that the container locates with the host port, however it's not recommended. Instead, we use an ALB to route traffic to the containers. All containers for a specific ECS service are managed by a target group. With Load Balancer Listener, Load Balancer can forward traffic to target group. Here is the diagram of relationship among load balancer, listener and target group.

Application Auto Scaling Target & Policy

Automatic scaling is the ability to increase or decrease the desired count of tasks in your Amazon ECS service automatically. Amazon ECS leverages the Application Auto Scaling service to provide this functionality. Amazon ECS Service Auto Scaling supports the following types of automatic scaling: Target tracking, Step and Scheduled. In the project, I chose Target tracking.

With target tracking scaling policies, you select a metric and set a target value. In the project, I chose metric ECSServiceAverageCPUUtilization and scalable dimension ecs:service:DesiredCount with target value 75. Which means when the average CPU utilization of AWS ECS service is greater than 75%, new tasks will be launched (scale out) to meet the requirements, and vice versa. The automate scaling is monitored by CloudWatch alarm. If you go to the CloudWatch Alarm console, you will find two alarms are created as below.

You provide the peak point (75) and AWS calculates the valley point (67.5) according to peak point.

Project Source Code

GitHub source code https://github.com/camillehe1992/containerized-app-in-aws-ecs

Terraform

All Terraform related source code is in terraform directory. Go through the README documentation for the details. Setup local environment if you want to deploy these AWS resources from local machine, or you can use GitHub Actions workflows.

Currently you can use the Docker image https://hub.docker.com/repository/docker/camillehe1992/strapi/general. However you can build your own Docker image for the Strapi application and push to registry following the GitHub README.

Strapi

https://strapi.io/

Validate the task is running successfully from AWS ECS console.

Then, you can access the Strapi application via ALB DNS name, like http://strapi-prod-xxxxxxxxx.aws-region.elb.amazonaws.com/. Here is the portal of Strapi application.

And you can use admin panel as below to do some funny things after sign-up.

📚 References

Always appreciate for your ideas and comments. Thanks for reading! 😄

Setup containerized Application in AWS ECS - Part 2/3

Camille He — Thu, 07 Dec 2023 08:57:50 +0000

In the previous Part 1/3, I introduced how to setup a MYSQL database in AWS RDS. Now, we will dive into next topic - Setup ECS Cluster in AWS ECS.

Amazon Elastic Container Service (Amazon ECS) is a fully managed container orchestration service that helps you easily deploy, manage, and scale containerized applications. The article will focus on the ECS services and how them work together to manage your containerized application in the Cloud.

⚓ Amazon ECS Concepts

Here is the architecture diagram of the entire project.

The AWS resources in green box are created in this part, which include:

ECS Cluster & ECS Capacity

An Amazon ECS cluster is a logical grouping of tasks or services. There are two capacity options: EC2 and Fargate. In this project, I setup ECS Cluster on EC2 type. You can find a article that talks about the differences between ECS and Fargate at the bottom of article.

Capacity Provider

A capacity provider defines the cluster capacity that Amazon ECS scales up and down of the infrastructure you specify. For Amazon ECS on Amazon EC2, a capacity provider consists of a capacity provider name, an Auto Scaling group.

Amazon EC2 Auto Scaling

As we use Amazon EC2 instances for ECS capacity, I use Auto Scaling groups to manage the Amazon EC2 instances registered to ECS cluster. Auto Scaling ensure that we have the correct number of Amazon EC2 instances available to handle the load for our application.

Launch Template

A launch template contains the configuration information to launch an instance in EC2 Auto Scaling, including the ID of the Amazon Machine Image (AMI) and provides full functionality for Amazon EC2 Auto Scaling.

SNS Topic & Lambda Function

On the right top of the diagram, we setup a SNS topic and Lambda function to automate container instance draining in Amazon ECS. It's not mandatory for the whole project, but good to have in a real world project. I won't dive into the details here, but go through the blog if you're interested the topic.

Terraform Project

Same as before, the AWS resources are defined using Terraform. Here is the GitHub source code https://github.com/camillehe1992/aws-terraform-examples/tree/main/ecs-cluster-on-ec2

Walk through the README documentation in above GitHub repository, and setup local environment if you want to deploy these AWS resources from local machine, or you can use GitHub Actions workflows.

After done, validate the setup is successful from ECS Console. Your ECS Cluster must have 1 registered container instance as the below picture shows.

Next to Setup containerized Application in AWS ECS - Part 3/3

📚 References

Always appreciate for your ideas and comments. Thanks for reading! 😄

Setup containerized Application in AWS ECS - Part 1/3

Camille He — Thu, 07 Dec 2023 08:44:29 +0000

In the article, I'll dive into the details that how to setup a containerized application in AWS Cloud. The AWS services leveraged in the project include ECS, RDS, ALB, IAM etc. All source code is stored in GitHub that publicly available.

Before getting started, I assume you that:

Have basic knowledge about AWS services: ECS, RDS, ALB, IAM etc and hands-on experience on AWS.
Have basic understanding about containerization and the benefits of containerization.
Experience on Terraform and CICD.
Love coding and be eager to learn.

As there are lots of topics and knowledge in the article, I separate it into three sections, which include:

Build MySQL Database in AWS RDS
Build ECS Cluster in AWS ECS
Build containerized application in AWS ECS

Something you'd better know:

I'm working on MacBook M1, so some scripts in the source code may not work in other OS. Besides, you should configure the environment if you want to debug, develop and deploy the application on the machine.
As the project is intended for demo purpose, which means it's not a best practice from the security and stability perspective. At the end of the article, I'll share some ideas about how to improve the project in real world.

⚓ Project Background

In the demo, I will build a containerized application in AWS. The application is hosted in EC2 as containers. Users can access the application via an internet ALB (Application Load Balancer). A RDS instance will be launched for data persistence. The application is built based on Strapi, which is an open-source headless CMS. The Strapi application provides an admin panel and makes it easy to build standard backend API. Meanwhile, it integrates with most of popular database engines smoothly, for example MYSQL, Postgres and sqlite. All the features makes it the best fit in my demonstration. You can dive into Strapi via its official website portal if interested.

The architecture diagram in AWS shows as below.

All AWS resources in above diagram will be created in the project, except for network things. I will use the default VPC, Subnets and Security Groups in the AWS account. Well it's unsafe but easier to setup a project just for demo purpose.

🏡 Setup Database in AWS RDS

Now, let's focus on the RDS database setup (AWS resources in green box). Find the source code from https://github.com/camillehe1992/aws-terraform-examples/tree/main/rds-mysql-instance

The source code contains two parts, the Terraform resources and Lambda function source code. The AWS resources created in the project include:

AWS RDS Instance & Secrets

A RDS instance with custom parameter group is created. The database secret (username and password) is managed in Secrets Manager. The secret is auto-generated by AWS and will be used when interacting with database.

AWS Lambda Function & Lambda Execution IAM Role & CloudWatch Logs Group

A Lambda function and CloudWatch Logs group for function logs persistence. The function is used to initialize database, such as create a database or tables. A Lambda function execution IAM role with appropriate permissions.

Replace the default SQL script in src/script.sql with below script. I need to create a database name strapi after the instance is available.



CREATE DATABASE IF NOT EXISTS strapi;

You can invoke the Lambda function via AWS CLI from local or console directly.

Go to the README documentation in above GitHub repository to setup local environment if you want to deploy AWS resources from local machine or via GitHub Actions workflows.

After done, you can get three environment variables that will be used in next part.



# the endpoint of RDS database (retrieved from RDS)
DATABASE_HOST
# the username of RDS database (retrieved from Secrets Manager)
DATABASE_USERNAME
# the password of RDS database (retrieved from Secrets Manager)
DATABASE_PASSWORD

Next to Setup containerized Application in AWS ECS - Part 2/3

📚 References

Always appreciate for your ideas and comments. Thanks for reading! 😄

Using AWS S3 and Hugo to Create and Host a Static Website

Camille He — Sat, 07 Oct 2023 09:20:02 +0000

In this blog, I'm going to guide you how to create a static website using AWS S3 and Hugo. The demo application is deployed using Terraform and I assume you have basic knowledge about Terraform, Hugo, Github Actions, and AWS S3 Static Website Hosting, etc.

What is Terraform?

Terraform is an infrastructure as code tool that lets you build, change, and version cloud and on-prem resources safely and efficiently.

What is Hugo?

Hugo is a fast and modern static site generator written in Go, and designed to make website creation fun again.

What is Docsy?

Docsy is a Hugo theme module for technical documentation sites, providing easy site navigation, structure, and more. In the demo, I use Docsy theme component as a Hugo module. Repo Static WebSite using Docsy in AWS S3 is generated from official docsy-example.
You can clone/copy the docsy-example and edit it with your own content, or use it as an example.

What is GitHub Actions?

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can create workflows that build and test every pull request to your repository, or deploy merged pull requests to production.

Hosting a static website using Amazon S3

You can use Amazon S3 to host a static website. On a static website, individual webpages include static content. They might also contain client-side scripts.

The blog covers two sections:

Deploy S3 Bucket using Terraform.
Deploy Static Website Content using Hugo.

Deploy S3 Bucket using Terraform

Repo: Terraform S3 Static Website

As we use AWS S3 bucket for static website hosting, in this demo, I use Terraform to define and create AWS S3 bucket related resources. There are two options for the deployment.

Deploy from Local

Follow the terraform-examples to setup local environment to deploy Terraform resources, including

Install Terraform CLI
Install AWS CLI
Setup AWS Credentials

Then, use make commands as below for deployment.



# Create a Terraform plan named `tfplan`
make plan

# Apply the plan `tfplan`
make apply

# Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
# 
# Outputs:
# 
# website_endpoint = "<static_bucket_name>.s3-website-<aws_region>.amazonaws.com"

At this point, you will get 404 Not Found if access the website via website_endpoint as above. After static content get uploaded to bucket, you should get the rendered content after refresh the page.

Deploy from GitHub Actions

Or, you can follow the Setup GitHub Environment for GitHub Actions Workflows to use GitHub Actions to deploy Terraform resources with deploy-static-content.yml workflow.

Deploy Static Website Content using Hugo

Repo: Static WebSite using Docsy in AWS S3

As mentioned before, the above source code repo is generated from docsy-example. I did two main changes after repo created.

Configure Hugo Deployment

Add below configuration in hugo.toml file. Replace <static_bucket_name> with the static_bucket_name variable you provided in Terraform project source. You should create your own bucket.



# Hugo deployment configuration

[deployment]
[[deployment.targets]]
name = "dev"
url = "s3://<static_bucket_name>"

[[deployment.targets]]
name = "prod"
url = "s3://<static_bucket_name>"

Then, create dedicated Github Actions workflows to build Hugo project and upload the static website content to S3 bucket in specific environments.

You should configure AWS credentials secrets from GitHub console following document Setup GitHub Environment for GitHub Actions Workflows, then update STATIC_BUCKET_NAME in env block in workflow yaml file.

The workflow is automatically triggered. After a successful build, access the static website via http://.s3-website-.amazonaws.com

In this demo, access below website via http://docsy-portal-prod.s3-website-ap-southeast-1.amazonaws.com.

Advanced Configuration

In a real project for static website hosting using AWS S3, you should think about more complex configurations.

Setup a CDN using AWS CloudFront to secure and improve the performance following https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started-cloudfront-overview.html.
Create a staging/pre-production environment in order to validate the change of content before delivering the final content to users/customers.

Done. Thanks for reading.

Deploy Github Secrets to AWS Secrets Manager using Terraform and Github Actions

Camille He — Thu, 21 Sep 2023 09:49:54 +0000

In this article, I'm going to guide you how to save the project secure tokens in Github Secrets and deploy them into AWS Secrets Manager using Terraform and Github Actions workflow.

Using sensitive information, such as api key, password, secure tokens in a real project is very common. For example, you are working on a backend web server that needs to interacts with database. You must have database user/password configured programmatically no matter which programming language you select. These secure tokens are normally persisted in a secret management tool, and injected into environment variables when the server is up.

Secrets management tools help companies securely store, transmit, and manage sensitive digital authentication credentials such as passwords, SSH keys, API keys, database passwords, certificates like TLS/SSL certificates or private certificates, tokens, encryption keys, privileged credentials, and other secrets.

Some companies use third-party tools, such as AWS Secrets Manager, Azure Key Vault, and HashiCorp Vault, etc. Some CICD tools also provide secret management features to allow your pipeline retrieves these secrets easily before deployment. Jenkins has Credentials and Github Actions has Secrets.

What I implemented:

Create a Terraform project that defines a aws_secretsmanager_secret resource using AWS official provider.
Create an Environment named dev and add secrets and variables in the environment.
Create a Github Actions workflow to deploy Terraform resources to AWS account.

Find the demo source code from https://github.com/camillehe1992/aws-terraform-examples/tree/main/secret-manager

Project Structure

The project source code structure as follows.



.
├── Makefile                    # a markfile for terraform commands 
├── README.md
├── main.tf                     # define the specs of secretsmanager module
├── modules
│   └── secretsmanager          # secretsmanager module
│       ├── locals.tf
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── outputs.tf                  # outputs of terraform
├── terraform.tfvars            # variables of terraform
├── variables.tf                # variables definition of terraform
└── versions.tf                 # terraform backend configuration and provider versions

Define a variable named database_password in variables.tf.



variable "database_password" {
  type        = string
  description = "The password of database"
}

Define a Terraform module named secretsmanager and create a bucket of secrets.

Modules are containers for multiple resources that are used together.



module "secrets" {

  source = "./modules/secretsmanager"

env      = var.env

  nickname = var.nickname

  tags     = var.tags

secret_specs = {

    database_password = {

      description   = "A sample secure. e.g password"

      secret_string = var.database_password

    }

  }

}

Create Environment and Secrets

Create an environment named dev in Github Settings, and add secrets and variables in it as follows.

Github Actions Workflow

Define an environment variable name DATABASE_PASSWORD in env block in Github Actions workflow file and inject the environment variable DATABASE_PASSWORD dynamically when running command terraform plan via parameter -var.



env:

  # Secret tokens injected into Terraform infra

  DATABASE_PASSWORD: "${{ secrets.TF_VAR_DATABASE_PASSWORD }}"

...

  <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Terraform Plan</span>
    <span class="na">if</span><span class="pi">:</span> <span class="s">${{ inputs.destroy }} == </span><span class="kc">false</span>
    <span class="na">id</span><span class="pi">:</span> <span class="s">tf-plan</span>
    <span class="na">working-directory</span><span class="pi">:</span> <span class="s">${{ env.WORKING_DIRECTORY }}</span>
    <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
      <span class="s">export exitcode=0</span>
      <span class="s">terraform plan -var-file $(pwd)/tf_$ENVIRONMENT.tfvars \</span>
      <span class="s">-var="database_password=$DATABASE_PASSWORD" \</span>
      <span class="s">-detailed-exitcode -no-color -out tfplan || export exitcode=$?</span>

Trigger Deployment Manually

Choose the target environment to deploy. Currently only dev is available.
Check True to destroy checkbox if you want to destory the resources.
Check True to force checkbox if you want to apply refresh the secure tokens. Useful when there is no change in Terraform infra, but there is a variables or secrets update in Github Settings.

Done. I'm always appreciating your comments and advice.

Stop/Start RDS Instances Automatically Using System Manager for Cost Optimization

Camille He — Tue, 12 Sep 2023 01:15:56 +0000

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the AWS Cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks. But there are some things that users should be manage like availability of the database, choosing the right size of database engine, maintaining backups, cost optimization and so on.

In this post, we're going to take about Cost Optimization with the runtime of the database. For example, If you want to run your database instance only a certain time like 9 am to 6 pm in working days. But manually start the database in morning 9 am and stop it in evening 6 pm is boring. Some times you forget to start the database and it will lead the application downtime in the working hours.

But if you setup an automation process which is stop and start the RDS database daily, at the time which you specified. Could that be amazing? AWS provides a service called System Manager, which is we are gonna use to stop and start our database. Let’s get into the details now.

What is AWS System Manager

AWS Systems Manager is the operations hub for your AWS applications and resources and a secure end-to-end management solution for hybrid and multi-cloud environments that enables secure operations at scale. In the post, we will use one of capabilities of System Manager, which named State Manager. State Manager is a secure and scalable service that automates the process of keeping managed nodes in a hybrid and multi-cloud infrastructure in a state that you define.

In our example, RDS instance is a type of nodes that can be managed using automate process. To enable the automation, you should do the following steps:

Create IAM role and Policy for System Manager
Create SSM Associations for Stop/Start RDS Instance

Create IAM role and Policy for System Manager

Firstly, you need to create an automation IAM role which grants start/stop RDS instance permissions to SSM. You can do it from AWS Console or any other IaC tools. Here are the details for the IAM role.

Role Name
StartStopRebootRDS

Trust Relationships



{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ssm.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Inline Policy



{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "rds:Describe*",
                "rds:Start*",
                "rds:Stop*",
                "rds:Reboot*"
            ],
            "Resource": "*"
        }
    ]
}

Create SSM Associations for Stop/Start RDS Instance

In the example, I'm going to implement: start RDS instance at 8:30 AM and stop it at 17:30 PM from Monday to Friday. As there is a limitation for the scheduled expression as below. We have to create a specific association for each day.

Schedule expression cron(0 30 08 ? * MON-FRI *) is currently not accepted. Supported expressions are every half, 1, 2, 4, 8 or 12 hour(s), every specified day and time of the week. Supported examples are: cron(0 0/30 * 1/1 * ? ), cron(0 0 0/4 1/1 ? ), cron (0 0 10 ? SUN ), cron (0 0 10 ? * *)

That's a batch of repeated manual stuff if creating associations from AWS console. I use AWS CLI to simplify the creation process.

Run below shell script to create SSM associations to start target RDS instances (billing-test) at 00:30 AM (UTC+8) from Monday to Friday.

My local time is UTC+0800, so the start instance cron expression for 08:30 AM is 00:30. So does the stop instance cron expression.



#!/bin/sh

WORKDAY="MON TUE WED THU FRI"

for day in $WORKDAY; do
    aws ssm create-association \
        --name AWS-StartRdsInstance \
        --schedule-expression "cron(30 0 ? * $day *)" \
        --association-name StartRDSInstance_$day\
        --parameters InstanceId=billing-test,AutomationAssumeRole=arn:{aws::partition}:iam::{aws:accountid}/role/StartStopRebootRDS \
        --profile your-credentials
done

Run below shell script to create SSM associations to stop target RDS instances (billing-test) at 17:30 PM (UTC+8) from Monday to Friday.



#!/bin/sh

WORKDAY="MON TUE WED THU FRI"

for day in $WORKDAY; do
    aws ssm create-association \
        --name AWS-StopRdsInstance \
        --schedule-expression "cron(30 9 ? * $day *)" \
        --association-name StopRDSInstance_$day\
        --parameters InstanceId=billing-test,AutomationAssumeRole=arn:{aws::partition}:iam::{aws:accountid}:role/StartStopRebootRDS \
        --profile your-credentials
done

Here is a screenshot of the associations I created using AWS CLI.

References

https://docs.aws.amazon.com/cli/latest/reference/ssm/create-association.html
https://www.easydeploy.io/blog/automate-rds-instance-using-system-manager/#Introduction
https://docs.aws.amazon.com/systems-manager/latest/userguide/state-manager-about.html

Build Scheduled AWS Batch Job Infrastructure using Terraform

Camille He — Sun, 10 Sep 2023 06:22:56 +0000

In this post, I will walk you through how to build a scheduled AWS Batch job infrastructure using Terraform. It focuses on the Terraform infrastructure and modules definition, and how they work together to build the entire workflow. It doesn't cover the advanced features that AWS Batch provided, and only use Terraform aws official provider
and basic Terraform features. In short, this post helps you setup a basic AWS Batch job infrastructure quickly, and you can add more features and enhance the structure as needed.

Prerequisite

AWS CLI (V2) is installed on the local machine.
AWS credentials setup. We use the credentials to deploy Terraform resources to the target AWS Account. In this demo, I use the same AWS profile for S3 remote backend configuration and Terraform apply.
Terraform CLI (1.3.4) is installed in the local machine. You can loosen the Terraform version restriction in versions.tf to use other close versions, however I'm not sure if the code works as expected without verification.
I'm working on Mac (MacOS Monterey) with Apple M2 Chip. So the aws provider installed is .terraform/providers/registry.terraform.io/hashicorp/aws/5.0.0/darwin_arm64/terraform-provider-aws_v5.0.0_x5. If you are in other operating system, you should remove .terraform.lock.hcl file from source code, and let Terraform CLI install the well-matched aws provider version according to your OS.
In the demo source code, I use default VPC, subnets and security group for EC2 instances. You can use customized network resources, however you should take care of the network availability if your Batch needs internet access, for example, the docker image that is used is hosted in public docker registry.

You can find the demo source code from https://github.com/camillehe1992/on-demand-job-in-aws-batch

Terraform Structure

The Terraform structure contains several components shows as below.

.
├── config.tfbackend  # Remote backend config file
├── data.tf           # File for Terraform data source 
├── main.tf           # The reference of modules
├── outputs.tf        # The outputs of Terraform resources
├── terraform         
│   ├── dev           # Environment specified variables
│   └── modules       # Terraform modules
├── variables.tf      # Terraform input variables that should be passed to the arch module
└── versions.tf       # Defines the versions of Terraform and providers

Remote Backend Configuration (S3)

For best practice, I save the Terraform state files in the remote location, which is S3 bucket here. The configuration below specifies the S3 bucket name, region, and profile used to access S3 bucket. Another parameter is key, which we

# config.tfbackend
region         = "cn-north-1"
bucket         = "tf-state-756143471679-cn-north-1"
profile        = "service.app-deployment-dev-ci-bot"

Another parameter is key that is provided via -backend-config command in terraform init as there is a ENVIRONMENT specific path and we have to inject it dynamically instead of hard-coding in config.tfbackend file. You can use other configurations settings as needed.

terraform init -reconfigure \
    -backend-config=$BACKEND_CONFIG \
    -backend-config="key=$NICKNAME/$ENVIRONMENT/$AWS_REGION/terraform.tfstate"

Terraform Main File

The main.tf in the root directory is the core component and entrance in Terraform structure. I define all resources here, including IAM roles, Batch, CloudWatch Events (EventBridge), and SNS Topic. You can separate them into specific files, for example, iam.tf, batch.tf, events.tf, sns.tf, etc. I keep them in one file as the project is not that complex, and it's easy to understand the relationships between resources/modules. You should make a better decision according to your project structure.

In the main.tf file, I define 8 resources, 3 IAM roles, 1 secrets group, 1 batch, 2 CloudWatch events, 1 SNS topic. All of them compose the AWS architecture as the diagram shows

Workflow steps:

User creates a docker image, uploads the image to the Amazon ECR or another container registry (for example, DockerHub), and creates a job definition, compute environment and job queue in AWS Batch. In this repo, we use an AWS official image public.ecr.aws/amazonlinux/amazonlinux:latest for demo purpose.
Batch job is submitted using job definition into the job queue in AWS Batch by CloudWatch Event regularly as scheduled.
AWS Batch launches an EC2 instance in computing environment, pulls the image from the image registry and create an container.
The container should implement some tasks on your behave. An email notification will be triggered if the job is failed.
After done, the container will be stopped and removed. EC2 instance is shutdown automatically by AWS Batch.

Terraform Modules

For well-architecting and organizing Terraform structure, I define modules for each resource group. Modules is a key feature in Terraform which helps users manage their own resources efficiently. Let's dive into the details of each module in this section.

Batch Module: Defines resources in Batch service, including computing environment, job queue, job definition.
EventBridge Module: Defines resources in EventBridge, including event rule, rule target. One named submit_batch_job_event is used to submit Batch job as scheduled, another named capture_failed_batch_event is used to send out an alert email if a Batch job is failed.
IAM Module: Defines IAM resources, including roles, policies, instance profile. These roles are used by AWS Batch resources and EventBridge rules.
SecretManager Module: Defines secret token that may be used in Job container. It's not required in the demo project, but for your reference.
SNS Module: Defines resource in SNS, including topic and subscription.

Apply/Destroy Terraform Resources

I create a Makefile and shell script to simplify the apply/destroy process in one command. You can find the shell script from /scripts/apply.sh in demo code.

# apply Terraform infrastructure
make apply

# destroy Terraform infrastructure
make destroy

Submit Batch Job Manually

After applying Terraform resources successfully using make apply, you can submit a Batch job for testing the entire workflow. The Batch job is submitted/triggered by CloudWatch Event (EventBridge) per day regularly as scheduled. However, you are allowed to submit a job manually via AWS CLI as below. Or from AWS Console directly.

Don't forget to update job definition revision in --job-definition if you have a new revision created. Only the latest revision is ACTIVE.

# Setup AWS_PROFILE with permission to submit batch job
export AWS_PROFILE=service.app-deployment-dev-ci-bot

# Submit a job using CLI
aws batch submit-job \
  --job-name triggered-via-cli \
  --job-definition arn:aws-cn:batch:cn-north-1:756143471679:job-definition/dev-helloworld-jd:1 \
  --job-queue arn:aws-cn:batch:cn-north-1:756143471679:job-queue/dev-helloworld-jq

After submitted successfully, go to AWS Console -> Batch -> Jobs. Select the target job queue from the dropdown list, then your new submitted job will be listed on the top. It will spend a few minutes for a job to complete, according to the job processing time, and whether you allocate an EC2 instance resource in advance by giving variable desired_vcpus a number greater than 0 or not. If the job failed, an email notification will be sent out to the Topic subscribers you provided in variable notification_email_addresses.

For cost saving, I set the desired_vcpus as 0 as default, which means a new EC2 instance will be launched when a new job is submitted and shut down immediately after completed. The screenshot below shows the latest job that submitted by CloudWatch Event (EventBridge) at 04:00 AM (UTC).

Logging

The logging data is saved to CloudWatch Logs automatically. You can find the logs on the bottom of the job details (some delay to sync logs from CloudWatch Logs). In the job details view page, it also provides a link to the log stream of current job. AWS creates a CloudWatch Logs group named /aws/batch/job automatically when you submit a Batch job at the first time in the same region.

Notes:

In the compute_resources block of resource aws_batch_compute_environment, the instance_role argument is the ARN of the IAM role profile, not the IAM role.
You may met a ClientException that failed to delete Computing Environment as it has a relationship with Job Queue. This is a well-known issue in Terraform aws provider and lots of people raised on the Internet. A few workarounds came up with, and you can find one that fits your needs if you google it.
Don't forget to accept the email subscription request when you first deploy the SNS subscription, then your subscribed email address is able to receive failed job alert.
For secret token, NEVER EVER checkin the secret token in your source code. For demo purpose, I export the token in Makefile using export TF_VAR_my_secret=replace_me, but you should save the secret tokens in a better place, for example GitHub Secrets, and inject them as environment variables in runtime.

Reference

Done. I'm always appreciating your commands and ideas. Happy learning!