DEV Community: Luthfi Anandra

Gatekeep CodeCatalyst Workflow using Approval

Luthfi Anandra — Wed, 15 May 2024 17:26:13 +0000

In some use cases, our CI/CD pipeline or workflow may need to be validated before code can be deployed.

Refer to this announcement "Workflow approvals for Amazon CodeCatalyst", Amazon CodeCatalyst, now has released feature called approval gates which can be utilized so that workflow can be paused until user with privilege can validate the workflow and whether workflow is approved and can be processed or not.

More details regarding Approval in CodeCatalyst can be read on this documentation: Approval Gate.

Scenario
Prerequisites
Configuration Steps
- Directory Structure
- CodeCatalyst Workflow File Configurations
- Example of Approval Process in CodeCatalyst Workflow
Summary

Scenario

This blog is related with my previous blog "CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst". In my previous blog, in workflow/pipeline, when terraform plan has finished and will continue to terraform apply, there is a process in between where pipeline will wait for 1 minute to check the terraform plan output and if the plan is not expected, we need to cancel the workflow manually. The difficulty that faced back then is because there was no out of the box solution provided by CodeCatalyst to restrict whether shifting between one action to other action can be cancelled if there is unexpected result.

In this blog, i will make an update which between terraform plan and terraform apply action, there will be an approval action. The goal if the plan is not expected, we can cancel the workflow and terraform apply will not be processed.

Here is the diagram that used in this blog:

Prerequisites

Here is the prerequisite that needed to follow configurations discussed in this blog:

CodeCatalyst space and project that has been connected with AWS account and source repository. In this blog, I use GitHub as source repository. For configuration reference, please check my other blog “Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst”

Configuration Steps

Directory Structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sandbox-sbx0-vpc.yml
└── sandbox/
    └── sbx0/
        └── vpc/
            ├── backend.tf  
            ├── main.tf
            ├── terraform.tfvars
            └── variables.tf

CodeCatalyst Workflow File Configurations

Here is the configuration of CodeCatalyst workflow file that written in file ./codecatalyst/workflows/tf-sandbox-sbx0-vpc.yml

Name: tf-sandbox-sbx0-vpc
SchemaVersion: "1.0"

Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox\/sbx0\/vpc\/.*

Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.8.2
      Steps:
      - Run: cd sandbox/sbx0/vpc
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2
  wait-for-approval:
    Identifier: aws/approval@v1
    DependsOn:
      - terraform-plan
    Configuration:
      ApprovalsRequired: 1
  terraform-apply:
    DependsOn:
      - wait-for-approval
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.8.2
      Steps:
      - Run: cd sandbox/sbx0/vpc
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Next, I will explain more detail each sections that defined on workflow file above:

Define workflow name. Then define how workflow will be ran. Workflow will be ran automatically if there is a push to repository, specifically to directory sandbox/sbx0/vpc

Name: tf-sandbox-sbx0-vpc
SchemaVersion: "1.0"

Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox\/sbx0\/vpc\/.*

Define action. First action will run workflow terraform plan. In this section, action uses identifier aws/build@v1. Then action will be ran on sandbox environment that has been connected with AWS account and also has IAM role associated with that account. In Configuration section, has been defined that action will be ran on top of container using specific version of terraform public image that pulled from DockerHub. Also has defined working directory and several terraform commands which include terraform plan. Lastly, define compute type which is EC2

Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.8.2
      Steps:
      - Run: cd sandbox/sbx0/vpc
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2

Next action is wait-for-approval. Goal of this action is to verify workflow can be ran to next action if there is an approval from privileged user(s). This action is using identifier aws/approval@v1 and depends on previous action which is terraform-plan. This action need at least 1 privileged user approval so that next action can be ran.
```
wait-for-approval:
    Identifier: aws/approval@v1
    DependsOn:
      - terraform-plan
    Configuration:
      ApprovalsRequired: 1
```

Next action that defined is action to run terraform apply. This action more or less similar with action terraform-plan, the difference is just terraform command that declared on configuration. In this section, action will run commands that needed for terraform apply. This action is depends on previous action which is wait-for-approval

terraform-apply:
    DependsOn:
      - wait-for-approval
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.8.2
      Steps:
      - Run: cd sandbox/sbx0/vpc
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Example of Approval Process in CodeCatalyst Workflow

I have defined Terraform codes that will be used as example in this blog. Files can be found in this GitHub path: https://github.com/lanandra/mock-aws-infrastructure/tree/master/sandbox/sbx0/vpc

Existing main.tf configuration:

#######
# VPC #
#######


data "aws_availability_zones" "available" {}

locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    "environment" = "${var.env_name}"
    "managedBy"   = "terraform"
  }
}

module "sbx0_apse1" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.8.1"

  name = var.env_name
  cidr = var.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k + 3)]

  enable_nat_gateway = false
  single_nat_gateway = true

  tags = local.tags
}

Here are the configuration steps:

Change Terraform configuration. Enable NAT gateway in VPC
```
enable_nat_gateway = true
```
As defined in workflow file, all changes inside directory sandbox/sbx0/vpc will trigger workflow run. Go to CodeCatalyst web console to verify. Go to space and project where workflow ran. Go to CI/CD menu, then choose Workflows. Verify there is new runs. Go to that runs
Go to run details by clicking name or run. Next, we will be redirected to overview page of that run. Click action terraform-plan, on the right side will appear tray box that display all steps that configured on that action. If we want to see details of terraform plan, go to tab log, then click step terraform plan -no-color -input-false. Verify action is succeeded
Next, action wait-for-approval will be ran and waiting input from privileged user, whether workflow run can be approved or not
Choose approve then click submit
If pop-up approval appeared, click Confirm to continue workflow
We also can see the status of approval, whether it has been approved or rejected
After approval has been approved, next action will be ran where in this blog example is terraform apply. Verify terraform apply is succeeded

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

In some use cases, maybe we will find condition where we will need approval before deployment in CI/CD pipeline can be done
CodeCatalyst have approval feature that can gatekeep deployment in CI/CD

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

Schedule Access Key Rotation Using CodeCatalyst

Luthfi Anandra — Sun, 03 Mar 2024 08:16:49 +0000

To interact with services/resources in Amazon Web Services (AWS), commonly We need security credentials. Security credentials are needed for authentication and authorization. There are two types of security credentials, long term and temporary. Both have its own use cases, but temporary credentials is more recommended.

Despite using temporary security credentials is more recommended, maybe in some use cases We still need or have to use long term credentials such as IAM user access key and secret key. To prevent security hole, We can and encouraged to rotate access key and secret key periodically.

More details about AWS security credentials: [+]

Scenario
Prerequisites
Configuration Steps
- Directory Structure
- Access Key Rotation Script Configurations
- CodeCatalyst Workflow File Configurations
- Example of Automate Access Key Rotation in CI/CD CodeCatalyst
Summary

Scenario

In this blog, I will create CI/CD pipeline in Amazon CodeCatalyst or in CodeCatalyst also called as workflow, where this workflow will run scheduled automated IAM user access key rotation. First, CodeCatalyst will delete old access key, then it will create new access key. In the process, CodeCatalyst workflow will run shell scripts that has been configured to run aws cli commands that needed for each steps.

Prerequisites

Here are some prerequisites that needed to follow configurations discussed in this blog:

CodeCatalyst space and project that has been connected with AWS account and source repository. In this blog, I use GitHub as source repository. For configuration reference, please check my other blog “Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst”
IAM user

Configuration Steps

Directory Structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── rotate-access-key.yml
└── sandbox2/
    └── iam/
        ├── create-access-key.sh
        └── delete-access-key.sh

Access Key Rotation Script Configurations

Configuration started with create shell scripts that will be used for automate access key rotation.

Create shell script delete-access-key.sh that will be used to check whether IAM user has access key and if user has, script will delete that access key. In this script, We need to export environment variable value IAM_USER that used to define which IAM user will be checked

#!/usr/bin/env bash

set -euo pipefail

IAM_USER=$IAM_USER

ACCESS_KEY=$(aws iam list-access-keys \
  --user-name $IAM_USER \
  --query 'AccessKeyMetadata[].AccessKeyId' \
  --output text \
  --no-cli-pager)

if [ -z "$ACCESS_KEY" ]; then
  echo "No access key found on user $IAM_USER"
else  
  echo "Found access key $ACCESS_KEY on user $IAM_USER"

  DELETE_KEY=$(aws iam delete-access-key \
    --user-name $IAM_USER \
    --access-key-id $ACCESS_KEY)

  echo "Access key $ACCESS_KEY deleted"
fi

Create shell script create-access-key.sh that will be used to create/generate new access key. Similar with delete-access-key.sh, We need to export enviroment variable value IAM_USER. Output of access key creation will be saved into file called access-key.csv

#!/usr/bin/env bash

set -euo pipefail

IAM_USER=$IAM_USER

CREATE_KEY=$(aws iam create-access-key \
  --user-name $IAM_USER\
  --output text)

echo "Creating access key for user $IAM_USER"
echo "$CREATE_KEY" > ./access-key.csv
echo "Access key created. Please find the key values on access-key.csv"

CodeCatalyst Workflow File Configurations

After created shell script for automate access key rotation, then We need to define workflow file that will be used to schedule automation of access key rotation.

I configured that workflow in file .codecatalyst/workflows/rotate-access-key.yml. Here is the full content of workflow file:

Name: rotate-access-key
SchemaVersion: "1.0"

Triggers:
  - Type: SCHEDULE
    Expression: "30 12 ? * THU *"
    Branches:
      - master

Actions:
  delete-access-key:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
      Variables:
        - Name: IAM_USER
          Value: "tf-admin"
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: cd sandbox2/iam
      - Run: ./delete-access-key.sh
    Compute:
      Type: EC2
  create-access-key:
    DependsOn:
      - delete-access-key
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
      Variables:
        - Name: IAM_USER
          Value: "tf-admin"
    Outputs:
      Artifacts:
        - Name: ACCESSKEY
          Files:
            - sandbox2/iam/access-key.csv
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: cd sandbox2/iam
      - Run: ./create-access-key.sh
    Compute:
      Type: EC2

Next I will explain more details each section inside workflow file.

Define workflow name. Then define how workflow run. In this blog, as an example, workflow will be ran as scheduled on Thursday at 12:30 UTC and only will be ran on master branch. Please adjust schedule following cron expression guide that used by CodeCatalyst. CodeCatalyst use six-field syntax and separated with space, more details please check this documentation: CodeCatalyst workflow triggers expression & EventBridge cron expressions reference
```
Name: rotate-access-key
SchemaVersion: "1.0"

Triggers:
  - Type: SCHEDULE
    Expression: "30 12 ? * THU *"
    Branches:
      - master
```

Define actions. Action will be divided into 2 sections. First section will run shell script delete-access-key.sh in action delete-access-key. As mentioned above, we need to export environment variable value IAM_USER. In this blog, I will use example user called tf-admin. Action will be ran in sandbox environment that has been connected with AWS account and also has IAM role associated with that account. In configuration section, I defined steps how to run shell script delete-access-key.sh from directory where the file located

Actions:
  delete-access-key:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
      Variables:
        - Name: IAM_USER
          Value: "tf-admin"
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: cd sandbox2/iam
      - Run: ./delete-access-key.sh
    Compute:
      Type: EC2

Next section will run shell script create-access-key.sh in action create-access-key. This action will depends on previous action which is delete-access-key and this action will not run if previous action is failed. More or less this action will be the same with action delete-access-key. We need to export environment variable value IAM_USER. Similar with previous action, i will input example user called tf-admin. Action will be ran in sandbox environment that has been connected with AWS account and also has IAM role associated with that account. In configuration section, there will be different steps, I defined steps to run shell script create-access-key.sh from directory where the file located. Beside that, this action will generate output artifact called ACCESSKEY, this artifact consist of file access-key.csv where information releted access key and secret key stored
```
create-access-key:
    DependsOn:
      - delete-access-key
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
      Variables:
        - Name: IAM_USER
          Value: "tf-admin"
    Outputs:
      Artifacts:
        - Name: ACCESSKEY
          Files:
            - sandbox2/iam/access-key.csv
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: cd sandbox2/iam
      - Run: ./create-access-key.sh
    Compute:
      Type: EC2
```

Example of Automate Access Key Rotation in CI/CD CodeCatalyst

After configure CodeCatalyst workflow file, next we verify scheduled automation access key rotation is working as expected.

Go to CodeCatalyst web console, choose space and project where workflow resided. Go to menu CI/CD — Workflows. Find workflow name. In section Recent runs, verify there is a successful scheduled run
Click action delete-access-key. In the right will appear window to check Logs related to that action. Verify shell script delete-access-key.sh is run and successfully found existing access key in user and delete that key
Click action create-access-key. In the right will appear window to check Logs related to that action. Verify shell script create-access-key.sh is run and successfully create access key
Go to Artifacts menu in related run. Verify artifact is successfully created and verify file can be downloaded
Open csv file inside the artifact, verify information related access and secret key. Verify user is match with what defined in workflow environment variable
For validation, go to AWS web console. Go to IAM user menu and find related user. Verify access-key is match with what defined inside csv file

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

There are two types of security credentials in AWS, long term and temporary
Temporary credentials is more recommended. But in some cases or environments, we may need or have to use long term credentials such as access key
To prevent security hole, please ensure to rotate access key periodically
CodeCatalyst can be used as tool to schedule automation of access key rotation

Please check out my GitHub repository to see source code example that used in this blog:

Shell scripts used for access key rotation: rotation scripts
Amazon CodeCatalyst workflow file: codecatalyst workflow

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst

Luthfi Anandra — Wed, 22 Nov 2023 04:25:20 +0000

Terraform workflow can be ran using several methods. One of them is running Terraform workflow inside CI/CD pipeline. Running Terraform workflow inside CI/CD pipeline can have several benefits, such as: automate creation or provision resources, simplify collaboration between engineers/developers, etc.

Scenario
Prerequisites
Configuration Steps
- Directory Structure
- Terraform Backend Configurations
- CodeCatalyst Workflow File Configurations
- Example of Terraform Workflow Process in CodeCatalyst CI/CD
Summary

Scenario

In this blog, I will explain how to run Terraform workflow inside CI/CD pipeline which in this blog is Amazon CodeCatalyst. GitHub is used as source code repository and has been connected with CodeCatalyst. Next in CodeCatalyst pipeline/workflow, We will provision resources via Terraform. In this blog, Terraform is using S3 backend. Here is the topology used in this blog:

This blog is using reference from tutorial “Bootstrapping your Terraform automation with Amazon CodeCatalyst” written by @Cobus Bernard with some adjustment in configuration.

If You want to know more details about CodeCatalyst, please check CodeCatalyst documentation.

Prerequisites

Before starting configuration, if you have not got AWS Builder ID account, please sign up first
After sign up, go to Amazon CodeCatalyst web console to start project and configuration. Apply initial configuration that needed by CodeCatalyst such as:
- Creating space
- Connect CodeCatalyst space with AWS account
- Create and connect IAM role that will be used for running CodeCatalyst workflow/pipeline
- Connect GitHub source code repository
- Define environment that will be used by workflow/pipeline
- Please check my blog “Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst” to find reference for configuration above. Please check section Amazon CodeCatalyst Pipeline/Workflow Configuration
Prepare S3 bucket and DynamoDB lock table (optional) that will be used by Terraform backend

Configuration Steps

Directory Structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sbx2-vpc-apse1.yml
└── sandbox2/
    ├── vpc/
    │   └── ap-southeast-1/
    │       ├── main.tf
    │       ├── resources.tf
    │       └── variables.tf
    └── tf-backend/
        ├── main.tf
        ├── resources.tf
        └── variables.tf

Terraform Backend Configurations

Configuration start with provisioning resources that will be needed by Terraform backend such as S3 bucket and DynamoDB table. Beside that, We also need to prepare IAM role that will be needed to provision resources from CodeCatalyst workflow/pipeline.

At first, We haven’t had any resource for storing Terraform state in S3, so that we need to run and store terraform state on our local, for example: laptop. Later, We will move that state from local to S3. We start by defining configuration in sandbox2/tf-backend/main.tf
```
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.aws_region
}
```

Define variable region in file sandbox2/tf-backend/variables.tf

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

Define resources S3, DynamoDB and IAM that needed by CodeCatalyst in file sandbox2/tf-backend/resources.tf. In this blog, IAM role CodeCatalyst has administrator access just for demo purposes. Please bear in mind to always use least privilege method on your production to prevent any security breaches

######
# S3 #
######

resource "aws_s3_bucket" "my_sandbox2_tfbucket" {
  bucket = "my-sandbox2-tfbucket"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "my_sandbox2_tfbucket" {
  bucket = aws_s3_bucket.my_sandbox2_tfbucket.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "my_sandbox2_tfbucket" {
  bucket = aws_s3_bucket.my_sandbox2_tfbucket.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "my_sandbox2_tfbucket" {
  bucket                  = aws_s3_bucket.my_sandbox2_tfbucket.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

############
# DynamoDB #
############

resource "aws_dynamodb_table" "my_sandbox2_tflocks" {
  name         = "my-sandbox2-tflocks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

#######
# IAM #
#######

data "aws_iam_policy_document" "codecatalyst_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type = "Service"
      identifiers = [
        "codecatalyst.amazonaws.com",
        "codecatalyst-runner.amazonaws.com"
      ]
    }
  }
}

resource "aws_iam_role" "codecatalyst_admin" {
  name               = "codecatalyst-admin"
  assume_role_policy = data.aws_iam_policy_document.codecatalyst_assume_role_policy.json
}

resource "aws_iam_role_policy_attachment" "codecatalyst_admin" {
  role       = aws_iam_role.codecatalyst_admin.name
  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}

Refer AWS keys that needed for running terraform workflow commands, for example by exporting environment variables as explained in this document
Run terraform cli commands that needed for provisioning resources
```
terraform init
terraform plan
terraform apply
```
Verify terraform apply is successful and resources successfully provisioned

Back to file sandbox2/tf-backend/main.tf, add additional configurations related to S3 backend

###########################
# Terraform Configuration #
###########################

terraform {
  backend "s3" {
    bucket         = "my-sandbox2-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-sandbox2-tflocks"
    encrypt        = true
  }
}

Run terraform init command again. Terraform will detect changes related to S3 backend and will move terraform state to S3 bucket that has been defined. Choose yes if asked for input

terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Releasing state lock. This may take a few moments...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Go to Amazon Web Services (AWS) web console, go to S3 service menu. Verify terraform.tfstate file has been copied to defined S3 bucket/path

CodeCatalyst Workflow File Configurations

After configurations for terraform backend has been finished as described above, next we continue configure CodeCatalyst workflow/pipeline.

Before starting workflow configuration, please make sure to configure all configurations that needed by CodeCatalyst as mentioned in Prerequisites.

As mentioned in document Build, test, and deploy with workflows in CodeCatalyst, CodeCatalyst use workflow definition file as reference workflow/pipeline configuration and this file is saved in directory ~/.codecatalyst/workflows/.

In this blog, I will create basic resources that needed by VPC on AWS. As mentioned on Directory Structure above, pipeline will be ran when there is changes inside directory sandbox2/vpc/ap-southeast-1.

Below is full content of workflow file codecatalyst/workflows/tf-sbx2-vpc-apse1.yml that used as example in this blog.

Name: tf-sbx2-vpc-apse1
SchemaVersion: "1.0"

Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox2\/vpc\/ap-southeast-1\/.*

Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2
  wait-period:
    DependsOn:
      - terraform-plan
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: echo "Please wait for a while before terraform apply"
      - Run: echo "If you wish to cancel terraform apply, please stop this run"
      - Run: sleep 60
    Compute:
      Type: EC2
  terraform-apply:
    DependsOn:
      - wait-period
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Next, I will explain more detail each sections that defined on workflow file above.

Define workflow name. Then define how workflow will be ran. In this blog, workflow will be ran when there is a push to master branch but only ran when changes is under directory sandbox2/vpc/ap-southeast-1/ (regular expression or regex is used to detect changes)
```
Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox2\/vpc\/ap-southeast-1\/.*
```

Define action. Action will be divided into 3 sections. The first one is used to run workflow terraform plan. In this section, action uses identifier aws/build@v1. Then action will be ran on sandbox environment that has been connected with AWS account and also has IAM role associated with that account. In Configuration section, has been defined that action will be ran on top of container using specific version of terraform public image that pulled from DockerHub. Also has defined working directory and several terraform commands which include terraform plan. Lastly, define compute type which is EC2

Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2

Next action that defined is called wait-period. This action is defined so that there is an interlude between terraform plan and terraform apply. In case there is something that unwanted in terraform plan, so that we have a time to not continue to next action or process which is terraform apply. This action is workaround because by the time this blog is released, there is no out of the box solution provided by CodeCatalyst, for example manual approval. So in case there is something unwanted, we can abort the workflow. In this action, I created a logic to pause the workflow for 60 seconds and this action is depends on previous action which is terraform plan
```
wait-period:
    DependsOn:
      - terraform-plan
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: echo "Please wait for a while before terraform apply"
      - Run: echo "If you wish to cancel terraform apply, please stop this run"
      - Run: sleep 60
    Compute:
      Type: EC2
```

terraform-apply:
    DependsOn:
      - wait-period
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Example of Terraform Workflow Process in CodeCatalyst CI/CD

After configure CodeCatalyst workflow file, next I will create example of Terraform codes. In this blog, I will create VPC resources using public module terraform-aws-modules/vpc/aws as reference. Here is the initial configuration/codes that defined:

sandbox2/vpc/ap-souhteast-1/main.tf

###########################
# Terraform Configuration #
###########################

terraform {
  backend "s3" {
    bucket         = "my-sandbox-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-sandbox-tflocks"
    encrypt        = true
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.aws_region
}

sandbox2/vpc/ap-souhteast-1/variables.tf

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

variable "env_name" {
  type    = string
  default = "sbx2"
}

sandbox2/vpc/ap-souhteast-1/resources.tf

#######
# VPC #
#######


data "aws_availability_zones" "available" {}

locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    "myTag:environment" = "sbx2"
    "myTag:managedBy"   = "terraform"
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.1.2"

  name = var.env_name
  cidr = "10.255.8.0/21"

  azs             = local.azs
  private_subnets = ["10.255.8.0/24", "10.255.9.0/24", "10.255.10.0/24"]
  public_subnets  = ["10.255.15.0/24", "10.255.14.0/24", "10.255.13.0/24"]

  enable_nat_gateway = false
  single_nat_gateway = true

  tags = local.tags
}

To conduct testing of Terraform workflow in CI/CD, I will change 1 line of code inside file sandbox2/vpc/ap-souhteast-1/resources.tf. I will activate NAT gateway with changing this line of code that current value is false
```
enable_nat_gateway = true
```
As defined in workflow file, all changes inside directory sandbox/vpc/ap-southeast-1 will trigger workflow run. To verify, go to CodeCatalyst web console, go to space and project where codes resided. Then go to CI/CD menu, choose Workflows. Go to tab Runs, verify there is new run that currently running.
Go to run details by clicking name/id of run. Then We will be redirected to overview page of run. Click action name terraform-plan, on the right will be showed tray box that display details configuration/steps of action. For example We want to see details of terraform plan, go to tab log, then click step terraform plan -no-color -input-false. Verify action succeeded
Verify terraform plan show expected result. Terraform will create resources releated to NAT gateway as mentioned in point number 1
As defined in workflow file, after terraform-plan action ran successfully, next workflow will proceed next action which is wait-period to give some interlude whether terraform plan will be executed to next action which is terraform-apply or apply will be aborted. If plan has met expectation and We want to proceed to terraform apply, no action needed, just wait 60 seconds and then verify action succeeded
But, if want to abort workflow and don’t want continue to terraform apply, click stop button when action still in progress
Let’s say We want to continue to terraform apply process, then terraform-apply action will be ran. We can see the details of the action inside tray box similar with previous actions. To verify resources has been provisioned via terraform apply, click step terraform apply -auto-approve -no-color -input=false
Verify terraform apply has been expected and action is succeeded
Go to VPC service page in AWS web console to verify NAT gateway has been provisioned successfully

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

Amazon CodeCatalyst can act as alternative for CI/CD engine/tools that can be used to run Terraform workflow
Amazon CodeCatalyst use IAM role to interact with AWS services. By using this method, Terraform doesn’t need to inject static credentials such as AWS Access Key and AWS Secret Key into the pipeline. This can’t help prevent security breaches
Amazon CodeCatalyst can give seamless experience if We want to deploy application to AWS environments
But some workflow or logic haven’t been provided by CodeCatalyst by the time this blog is released. Maybe there will be some improvement in the future

Please check out my GitHub repository to see source code example that used in this blog:

Terraform backend configuration: tf backend
Amazon CodeCatalyst workflow file: codecatalyst workflow
Example of Terraform AWS VPC configuration: tf aws vpc

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst

Luthfi Anandra — Sat, 16 Sep 2023 05:20:25 +0000

In this blog, I will discuss how to build and release container image to Amazon Elastic Container Registry (ECR) using service that by the time this blog is released, the service is quite new which is Amazon CodeCatalyst.

Before continue further, I would like to explain briefly about Amazon CodeCatalyst. As referred from Amazon CodeCatalyst documentation:

Amazon CodeCatalyst is an integrated service for software development teams adopting continuous integration and deployment practices into their software development process. CodeCatalyst puts the tools you need all in one place. You can plan work, collaborate on code, and build, test, and deploy applications with continuous integration/continuous delivery (CI/CD) tools.

Scenario

In this blog, I will discuss how to build and release container image with GitHub as source code repository. Then Amazon CodeCatalyst will run pipeline/workflow used for build and release container image, then later container image will be sent to ECR. The topology would be like this:

Prerequisites

Before starting configuration, if you have not got AWS Builder ID account, please sign up first
After sign up, go to Amazon CodeCatalyst web console to start project and configuration
Create repository for container image in Amazon ECR

Configuration Steps

Amazon ECR Repository Configuration

In this blog, I created ECR repository using Terraform with remote backend (Terraform Cloud). ECR repository created with reference from terraform public module terraform-aws-modules/terraform-aws-ecr.

Below is file or directory structure used by Terraform codes:

└── ecr/
    └── region-name/
        ├── files/
        │   └── policies/
        │       └── expire-untagged-images.json
        ├── main.tf
        ├── resources.tf
        └── variables.tf

Here are the configuration steps:

Define Terraform configuration inside file main.tf. In this configuration, I declared remote backend (Terraform Cloud), workspace used by Terraform, provider and its version and also AWS region that declared from variable

###########################
# Terraform Configuration #
###########################

terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "my-organization"

    workspaces {
      name = "my-ecr-workspace"
    }
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.aws_region
}

Define variable inside file variables.tf

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

Define ECR repository creation using public module reference as mentioned above. I also defined repository lifecycle policy so that untagged image will be expired after period amount of time. This configuration is defined in file resources.tf

####################
# ECR Repositories #
####################

module "my_repository" {
  source  = "terraform-aws-modules/ecr/aws"
  version = "1.6.0"

  repository_name             = "my-repository"
  repository_lifecycle_policy = file("${path.module}/files/policies/expire-untagged-images.json")

  tags = {
    Name             = "my-repository"
    "my:environment" = "my-environment"
  }
}

Policy file reference expire-untagged-images.json is contain this instruction

{
    "rules": [
        {
            "rulePriority": 1,
            "description": "Expire images older than 14 days",
            "selection": {
                "tagStatus": "untagged",
                "countType": "sinceImagePushed",
                "countUnit": "days",
                "countNumber": 14
            },
            "action": {
                "type": "expire"
            }
        }
    ]
}

Next commit and push terraform codes to the repository. After that run Terraform plan and Terraform apply from Terraform Cloud workspace so that infrastructures or resources can be provisioned. This activity will not be explained in detail and I will only show the simulation. If you are interested in how to configure Terraform Cloud workspace and how to run Terraform plans and apply for the provision of AWS resources, please check out my other blog post here.

Amazon CodeCatalyst Pipeline/Workflow Configuration

Before starting configuration, please make sure space on Amazon CodeCatalyst has been created. More details on space, please check Spaces in CodeCatalyst
Connect Amazon CodeCatalyst space with AWS account. In space page, go to menu Settings > AWS accounts. Click Add an AWS account button
Input required values such as AWS account ID, then display name. Next click Associate AWS account button
Next We will be redirected to Amazon CodeCatalyst page in AWS web console. Insert token that has been generated by CodeCatalyst (image for this is not displayed in this blog). Verify space has been connected
Back to AWS account page in CodeCatalyst web console, verify account has been connected as well
Add IAM role that will be used to run workflow. Go to account details page by clicking account name in previous point. Then click Manage roles from AWS Management Console button
In this blog, I created administrator IAM role because I just need for development only. On production environment, please bear in mind to always use least privileged method for security concern. Next, set role name and click Create development role button
Verify IAM role has been created
Go back to CodeCatalyst space page and create project by clicking Create project button
In this blog, I created project with bring your own code method. The codes has been stored in GitHub. Choose GitHub account that will be connected, then choose GitHub repository name. After that, set project name used by CodeCatalyst. If all set, click Create project button
Verify GitHub repository has been connected. In project page, go to menu Code > Source repositories.
Create environment that will be used by CI/CD workflow/pipeline. Go to menu CI/CD > Environment. Then click Create environment button
In Create environment page, input environment name, environment type and AWS account connection (optional). If all set, click Create environment button
Define CI/CD workflow/pipeline. CodeCatalyst need workflow definition file for pipeline creation. This file can be created by several methods, for example create via CodeCatalyst web console or using code/text editor. Further details about workflow in CodeCatalyst, please check CodeCatalyst documentation Workflows concepts
Before creating workflow file, I would like to inform that there are several pre-defined workflow definition files that has been provided. To check those files, go to menu Create workflow > in Actions section, find template file that suitable with your needs. In this blog, for example I will use pre-defined template for workflow push to Amazon ECR. This pre-defined template will help me to simplify build and push container image to Amazon ECR
We can see the details of each pre-defined template file. For example documentation about how to use, YAML preview etc. To see the details, click pre-defined template name and next will be redirected to detail page
We also can see the preview of YAML and also the details for example variables used by the template
Next You can adjust with your preference, whether you want to create configuration using visual editor on web console or using code/text editor

In this blog, I created configuration using code/text editor and I will explain the details of workflow file

# - I created workflow name build-container
# - Workflow created using reference from pre-defined template `codecatalyst-labs/push-to-ecr@v1.0.1` with source from WorkflowSource which in this blog is GitHub repository
# - Workflow will run in sandbox environment with AWS account reference as explained above and IAM role that associated within that account
# - Define ECR repository name, image tag and AWS region

Name: build-container
SchemaVersion: "1.0"

Actions:
  build-and-push-to-ecr:
    # Identifies the action. Do not modify this value.
    Identifier: codecatalyst-labs/push-to-ecr@v1.0.1

    # Specifies the source and/or artifacts to pass to the action as input.
    Inputs:
      # Required
      Sources:
        - WorkflowSource # This specifies that the action requires this Workflow as a source

    # Required; You can use an environment, AWS account connection, and role to access AWS resources.
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: CodeCatalystWorkflowDevelopmentRole-lanandra-sandbox

    # Defines the action's properties.
    Configuration:
      # Required; type: string; description: The name to use for the repository. 
      # The repository name may be specified on its own (such as nginx-web-app) 
      # or it can be prepended with a namespace to group the repository into a category (such as project-a/nginx-web-app)
      RepositoryName: steampipe-container-agent
      ImageTag: v0.1.0
      AWSRegion: ap-southeast-1

Run workflow. In this blog, I did not configured trigger event for workflow that can make workflow run automatically. Thus, I need to run the workflow manually. Go to menu CI/CD > Workflows. In tab Workflows, choose GitHub repository name, branch name and workflow name. Then in Action drop down list, choose Run
Next workflow will be running. Verify it by choosing workflow name as mentioned in previous point, then choose tab Runs
Go to workflow run. Verify run has been running and succeeded. Details of run can be seen inside tab Logs
Verify container image has been created and uploaded to Amazon ECR. Go to ECR service page, then go to repository used or defined in workflow

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

Amazon CodeCatalyst is service by AWS that provide Continues Integration/Continues Delivery or Deployment (CI/CD) in one place
CodeCatalyst can be used as alternative for CI/CD engine/tools that have seamless experience if we want to deploy our application to AWS environment
Currently there are some integration or template that can simplify user for creating CI/CD pipeline

Please check out my GitHub repository to see source code example that used in this blog:

Terraform code for ECR creation: ecr
Amazon CodeCatalyst workflow: codecatalyst

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

Alternative Way Provision AWS Resource via Terraform (Using IAM Assume Role Reference)

Luthfi Anandra — Wed, 01 Mar 2023 14:55:43 +0000

When provisioning or creating Amazon Web Services (AWS) resources via Terraform, maybe the most common method used is using AWS credentials reference that includes AWS Access Key ID and AWS Secret Access Key.

In this blogpost, I will explain one of the alternative for provisioning AWS resource via Terraform. I reference IAM assumed role during provision.

For context, before we discussed the configuration in detail, below is the scenario or architecture that I used in this blogpost:

In this blogpost, I used Terraform with backend remote or Terraform Cloud
In most common usage of Terraform, we reference AWS Access Key ID and AWS Secret Access Key for communication or interaction to AWS API. In this blogpost, I used IAM role that associated with IAM policy that needed for provision AWS resource
But in this case, We still need IAM user or if I may call that intermediary user that act as middle man when Terraform interact with AWS API. And this IAM user still need AWS Access Key ID and AWS Secret Access Key, although this IAM user does not associated with IAM policy at all.
IAM role will run assume role to IAM intermediary user before it can send API call that used for provision AWS resource
The idea is we only concern about AWS keys that used by IAM intermediary user. And after that, We only need to create IAM role with privilege or IAM policy that needed, then IAM role will assume role to IAM intermediary user

Prerequisites

As mentioned on scenario above, to provision from terraform via IAM assume role, we need some IAM resource. Here are some of them:

IAM user that act as intermediary user
IAM role
Associate IAM policy to IAM role

In this blogpost, I will create IAM resource using terraform code. Here are the steps:

Create IAM user that act as intermediary user. This IAM user will be associated to IAM group. IAM user and IAM group will be provisioned using public terraform aws module iam. Here is example of code:

#############
# IAM Users #
#############

module "tf_assume_user" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"
  version = "5.9.2"

  name                          = "tf-assume-user"
  force_destroy                 = true
  create_iam_access_key         = false
  create_iam_user_login_profile = false

  tags = {
    Name             = "tf-assume-user"
    "my:environment" = "my-environment"
  }
}

##############
# IAM Groups #
##############

module "tf_assume_group" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-group-with-policies"
  version = "5.9.2"

  name                              = "tf-assume-group"
  attach_iam_self_management_policy = false

  group_users = [
    module.tf_assume_user.iam_user_name
  ]

  custom_group_policy_arns = [
  ]

  tags = {
    Name             = "tf-assume-group"
    "my:environment" = "my-environment"
  }
}

Here is the example of IAM group and IAM user when verified from AWS Web Console. Create new AWS credentials (AWS Access Key ID and AWS Secret Acces Key) that will be used by IAM intermediary user.

Create IAM role that will assign IAM intermediary user above as trusted entity and will run sts:AssumeRole. Then in this blgpost, I will associate that IAM role with customer managed policy that can provision Amazon Lightsail. Policy that used for provision Lightsail is just for demo purpose so that it is not least privileged, please consider to use least privileged policy in production environment. IAM role and IAM policy will be provisioned using public terraform module as well. Here is the example of code:

#############
# IAM Roles #
#############

module "compute_admin_role" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role"
  version = "5.11.1"

  trusted_role_arns = [
    "arn:aws:iam::12digitsawsid:user/tf-assume-user",
  ]

  trusted_role_services = [
    "lightsail.amazonaws.com"
  ]

  create_role       = true
  role_name_prefix  = "compute-admin-sbx"
  role_requires_mfa = false

  custom_role_policy_arns = [
    module.lightsail_full_access.arn
  ]
}

################
# IAM Policies #
################

module "lightsail_full_access" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-policy"
  version = "5.11.1"

  name        = "lightsail-full-access"
  path        = "/"
  description = "Provides full access to Amazon Lightsail"
  policy      = file("${path.module}/files/policies/lightsail/lightsail-full-access.json")

  tags = {
    Name             = "lightsail-full-access"
    "my:environment" = "my-environment"
  }
}

Here is the example of IAM role when verified from AWS Web Console.

Configuration Steps

In this section, I will explain configuration steps that needed for provisioning AWS resource. As mentioned above, I used remote backend or Terraform Cloud. So I will explained a little bit about configuration on Terraform Cloud as well. In this blogpost, I provisioned Amazon Lightsail Instance as example.

In Terraform Cloud workspace that used for provision AWS resource, on variables section I associated AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY that used by IAM intermediary user. I referenced those variables using variable sets. Variable sets configuration will not be explained more detail in this blogpost, please visit this HashiCorp blog for more details.
Next in Terraform Code, I created file main.tf. In this file, I declared terraform configuration that refers to terraform cloud (app.terraform.io) as a remote backend. I also declared the organization and workspace used by Terraform code. Next in this file, I declared the provider used by Terraform code which is Hashicorp/AWS, and versions related to it. Last, I declared the AWS region refer to the variable aws_region and IAM role arn refer to variable role_arn , which both are configured in file variables.tf which I will explain later.
```
###########################
# Terraform Configuration #
###########################

terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "my-organization"

    workspaces {
      name = "my-workspace"
    }
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.53.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.aws_region

  assume_role {
    role_arn = var.role_arn
  }
}
```

Next, I defined variables used by the Terraform in file variables.tf.

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

variable "role_arn" {
  type        = string
  description = "IAM role ARN"
  default     = "arn:aws:iam::12digitsawsid:role/compute-admin-sbx"
}

variable "az" {
  type        = list(string)
  description = "AWS Availability Zone"
  default     = ["ap-southeast-1a", "ap-southeast-1b", "ap-southeast-1c"]
}

variable "key_pair_name" {
  type        = string
  description = "Name of Lightsail key pair"
  default     = "my-key-pair"
}

Next I created example code for provisioning Amazon Lightsail Instance.

######################
# Lightsail Instance #
######################

resource "aws_lightsail_instance" "my_instance" {
  name              = "my-instance"
  availability_zone = var.az[0]
  blueprint_id      = "ubuntu_20_04"
  bundle_id         = "micro_2_0"
  key_pair_name     = var.key_pair_name

  tags = {
    "Name"           = "my-instance"
    "my:environment" = "my-environment"
    "my:owner"       = "my-owner"
  }
}

resource "aws_lightsail_instance_public_ports" "my_instance_ports" {
  instance_name = aws_lightsail_instance.my_instance.name

  port_info {
    protocol  = "tcp"
    from_port = 22
    to_port   = 22
  }

  port_info {
    protocol  = "tcp"
    from_port = 443
    to_port   = 443
  }
}

Next commit and push terraform codes to the repository. After that run Terraform plan and Terraform apply from Terraform Cloud workspace so that infrastructures or resources can be provisioned. This activity will not be explained in detail and I will only show the simulation. If you are interested in how to configure Terraform Cloud workspace and how to run Terraform plans and apply for the provision of AWS resources, please check out my other blog post here.
After resource provisioning via Terraform Cloud has been finished, then we can verify resources have been successfully created via the AWS web console. I switched to Lightsail service page and verified that instance has been provisioned.

Conclusion

So We have reached the last section of this article. There are some key takeaways that I want to point out:

Beside using most common method which is using IAM user that associated with AWS Credentials (AWS Access Key ID and AWS Secret Access Key) and IAM policy, we can provision AWS resource via Terraform using IAM role reference (IAM assume role)
The idea is We only need to create IAM role with certain privilege and We don’t need create multiple IAM user that need AWS Credentials (AWS Access Key ID and AWS Secret Access Key)
But by the time this blogpost is released, I found that there is still some limitation with this IAM assume role method. Because We still need IAM user that act as intermediary user and this IAM user need AWS Credentials (AWS Access Key ID and AWS Secret Access Key). Although this IAM user is not associated with any IAM policy at all and just IAM role that associated with IAM policy

Please check my GitHub repository to see source code example used in this blogpost. For IAM resource configuration, please check this iam directory and for example how to apply this Terraform assume role configuration, please refer to this lightsail directory.

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

Centralizing IAM Access Key Used by IAM Machine User Using AWS Secrets Manager

Luthfi Anandra — Thu, 19 Jan 2023 15:08:41 +0000

In Amazon Web Services (AWS) environment, if We want to interact with AWS API, we can use AWS API key/AWS credentials for programmatic access which usually consist of AWS Access Key ID and AWS Secret Access Key. Usually this API key associated with AWS Identity and Access Management (IAM) such as IAM user. IAM user also can be classified to IAM user for human-to-machine interaction and IAM user for machine-to-machine interaction. Example usage of machine-to-machine interaction is when We created IAM machine user for CI/CD integration, then We save API key credential of IAM machine user in CI/CD engine environment variable so that the engine can interact with AWS. For example when it is used to push container image from CI/CD to AWS.

Sometimes it is common that those API key need to share with developers/engineers that in need and has privileged to access the key. For example Infrastructure Team want to share API key information with one of developer in service product team A. If the API key just shared or sent via csv file, probably there will be a problem in the future. For example that developer forgot where the API key is saved, or if the developer/engineer has resigned and forgot to share that API key with other team member or that API key might be shared or leaked to unprivileged user or bad actor and will open a security hole.

In this blogpost, I want to share some idea or solution regarding problem that mentioned above. The solution is We keep the API key used by IAM machine user in centralized way using AWS service called AWS Secrets Manager.

Scenario

Configuration of storing API key will be initialized with Infrastructure as Code (IaC) method using Terraform
Terraform configuration is using remote backend via Terraform Cloud

Directory Structure

In this article, the file structure used by Terraform codes is described below:

.
└── environment-name/
    └── secrets-manager/
        └── region-name/
            ├── main.tf
            ├── resources.tf
            └── variables.tf

Configuration

I started with creating file main.tf. In this file, I declared terraform configuration that refers to terraform cloud (app.terraform.io) as a remote backend. I also declared the organization and workspace used by Terraform code. Next in this file, I declared the provider used by Terraform code which is Hashicorp/AWS, and versions related to it. Last, I declared the AWS region used by Terraform code to refer to the variable aws_region that is configured in file variables.tf which I will explain later.
```
###########################
# Terraform Configuration #
###########################

terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "my-organization"

    workspaces {
      name = "my-secretsmanager-workspace"
    }
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.48.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.aws_region
}
```

Next, I defined variables used by the Terraform in file variables.tf.

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

variable "waiting_deletion_period" {
  type        = number
  description = "Waiting period in days before secret deleted. Set to 0 for force destroy"
  default     = 0
}

Refer to Terraform AWS provider document regarding resource aws_secretsmanager_secret_version. If We want to create secret with key-value pairs, we can get those value with several methods and one in common is from variable. So that, still in file variables.tf, I created variable that consist key-value pairs used by IAM machine user. And because AWS_SECRET_ACCESS_KEY contain sensitive value, I did not type the real value in the code but I changed manually from AWS Web Console to prevent repository contain any sensitive value. Example:
```
variable "my_machine_credentials" {
  type        = map(string)
  description = "Key value pairs consist of AWS credentials used by my machine user"
  default = {
    "AWS_ACCESS_KEY_ID"     = "AKIAWE1AB2CD34FG5I767"
    "AWS_SECRET_ACCESS_KEY" = "change_me"
  }
}
```

Next in file resources.tf, I declared secret with value, which value is referred from variable that described in point number 3.

###########
# Secrets #
###########

resource "aws_secretsmanager_secret" "dummy_machine_sbx" {
  name                    = "dummy-machine-sbx"
  recovery_window_in_days = var.waiting_deletion_period

  tags = {
    "Name"                 = "dummy-machine-sbx"
    "my:environment"       = "sandbox"
    "my:owner"             = "Luthfi"
  }
}

resource "aws_secretsmanager_secret_version" "dummy_machine_sbx" {
  secret_id     = aws_secretsmanager_secret.dummy_machine_sbx.id
  secret_string = jsonencode(var.dummy_machine_user_sbx_credentials)
}

Next commit and push terraform codes to the repository. After that run Terraform plan and Terraform apply from Terraform Cloud workspace so that infrastructures or resources can be provisioned. This activity will not be explained in detail and I will only show the simulation. If you are interested in how to configure Terraform Cloud workspace and how to run Terraform plans and apply for the provision of AWS resources, please check out my other blog post here.
After resource provisioning via terraform has been finished, then we can verify resources have been successfully created via the AWS web console. Go to Secrets Manager service page, then go to menu Secrets. Verify secret that defined in Terraform has been successfully created.
Next go to details configuration of the secret by clicking secret name mentioned in previous step. In this page, We can see details configuration of secret such as: secret name, secret ARN, encryption key used by secret and secret tags.
In this blogpost, I don’t use automatic rotation for the secret.
To verify the value of sercet, click Retrieve secret value button as seen in point number 7. Then click Edit button to change the value of AWS_SECRET_ACCESS_KEY.
Change the value of AWS_SECRET_ACCESS_KEY then click Save.
Verify value has been updated.
Next please always make sure that only privileged user that can access the secret.

Conclusion

So We have reached the last section of this article. There are some key takeaways that I want to point out:

To prevent some potential issue regarding security hole and shared secret of API key used by IAM machine, We can keep the API key in centralized way using AWS Secrets Manager.
Secret configuration in AWS Secrets Manager can be declared using the Infrastructure as Code (IaC) method. So that We can have a state file of infrastructure that has been created.

Please check my GitHub repository to see source code example used in this blogpost.

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

What’s Alternative Way to Find AWS IAM Key that has not been rotated for ages? Let’s Check Out This Tool

Luthfi Anandra — Sat, 24 Dec 2022 09:27:52 +0000

AWS IAM Key is one of tool that used to interact with AWS API. In AWS, it’s common for user that used for human-to-machine interaction or machine-to-machine interaction to have AWS Key that consists of AWS Access Key and AWS Secret Key.

For security reason, IAM key should be rotated for example every 90 days. The goal is to prevent security holes/leaks, for example we accidentally expose our IAM key. Because with this IAM key, every person that has access to the key can access AWS API in accordance with privilege given to that key and we don’t want any bad actor have access to our AWS environment.

In this blogpost, I want to introduce a tool that can be used to check or notify if any our AWS IAM key has exceeded the threshold. This tool name kuncen-aws-iam

Prerequisites

To run kuncen-aws-iam, you must follow these prerequisites:

Have Docker Engine installed or run on your environment
Pull lanandra/steampipe-container-agent container image. This container will utilize Steampipe for querying AWS IAM API
Have AWS API credentials (such as: AWS Access Key and AWS Secret Key) that have privilege to AWS IAM service

How to Use

Clone kuncen-aws-iam source code

From your terminal, export AWS credentials (AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY)

export AWS_ACCESS_KEY_ID="AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="AWS_SECRET_ACCESS_KEY"

Go to directory where kuncen-aws-iam resided. Then run the script
```
cd path/to/kuncen-aws-iam/

./kuncen-aws-iam.sh
```
Or if you want to run kuncen-aws-iam anywhere in your terminal, please create symlink to this script. Example:
```
sudo ln -s path/to/kuncen-aws-iam/kuncen-aws-iam.sh 
/usr/local/bin/kuncen-aws-iam
```
Example output of kuncen-aws-iam

Notes

As mentioned above, kuncen-aws-iam will check and send output of IAM key that age has exceed 90 days. But if you need to adjust the interval days, you can set it by set this environment variable below:
```
export KUNCEN_AWS_IAM_INTERVAL=int

example:
export KUNCEN_AWS_IAM_INTERVAL=180
```
As mentioned above, kuncen-aws-iam utilize Steampipe for querying AWS IAM API. By default Steampipe will output a query with table format. But Steampipe offers some output format such as: line, csv, json or table. If you need kuncen-aws-iam send output with different output other than table, you can set it by set this environment variable below:
```
export KUNCEN_AWS_IAM_OUTPUT=string

example:
export KUNCEN_AWS_IAM_OUTPUT=json
```
example output:

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you.

Backup MySQL Database That Running on Docker Container

Luthfi Anandra — Fri, 15 Jan 2021 13:36:14 +0000

In this article, i want to share some information how to backup MySQL database that running on Docker Container.

The environment that i use is Ubuntu Linux running Docker container. MySQL is running on top of Docker container and mysql backup is using mysqldump.

Here are few steps, how to backup MySQL that running on Docker container:

Check your MySQL container name
Backup your database using this command:

docker exec [mysql_container_name] /usr/bin/mysqldump -u [mysql_username] --password=[mysql_password] [database_name] > [destination_path]
Verify your backup destination path. Make sure backup sql file has been added to destination

Extend EBS Volume

Luthfi Anandra — Mon, 14 Dec 2020 17:28:51 +0000

In this article, i want to share some information how to extend your aws ebs volume. In this example, i use ec2 instance with ubuntu operating system and mount gp2 ebs volume to that ec2.

EC2 instance is using non Nitro System or non NVMe volume

The scenario is, i have existing ebs volume with size 10 GB then i want to extend the volume to 20 GB

Here are few configuration steps to extend linux ebs volume:

Verify existing disk usage. Type df -hT
Verify existing block device. Type lsblk
Based on example existing configuration, disk space is configured with 10 GB in size.
Go to aws console. Go to menu EC2 > Elastic Block Store > Volumes. Click on the block store that want to be extended
To prevent any unwanted issues, you can optionally create a snapshot for that ebs. Go to menu Actions > Create Snapshot
Configure snapshot description and tag (optional). After that, click Create Snapshot
Wait until create snapshot request succeeded, then click Close
Verify snapshot has been successfully created. Go to menu Elastic Block Store > Snapshot. Then verify
Then, you can modify your EBS volume. Go to menu Elastic Block Store > Volumes. Click on the volume you want to extend. Then click Actions > Modify Volume
Modify volume size, after that click Modify
Modify volume confirmation page will be prompted. Click Yes to continue
Wait until modify volume request succeeded
Verify volume has been modified
SSH to your EC2 instance. Type this command:

sudo growpart [block_number]

in this example, the command should be:

sudo growpart /dev/xvda 1
Verify block device size has been extended
Resize your block device to new size. Type this command:

sudo resize2fs [block_number]

in this example, the command should be:

sudo resize2fs /dev/xvda1
Verify your disk space has been extended

For further reference you can also check aws documentation on this link:

Link

DEV Community: Luthfi Anandra

Gatekeep CodeCatalyst Workflow using Approval

Table of Contents

Scenario

Prerequisites

Configuration Steps

Directory Structure

CodeCatalyst Workflow File Configurations

Example of Approval Process in CodeCatalyst Workflow

Summary

Schedule Access Key Rotation Using CodeCatalyst

Table of Contents

Scenario

Prerequisites

Configuration Steps

Directory Structure

Access Key Rotation Script Configurations

CodeCatalyst Workflow File Configurations

Example of Automate Access Key Rotation in CI/CD CodeCatalyst

Summary

CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst

Table of Contents

Scenario

Prerequisites

Configuration Steps

Directory Structure

Terraform Backend Configurations

CodeCatalyst Workflow File Configurations

Example of Terraform Workflow Process in CodeCatalyst CI/CD

Summary

Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst

Scenario

Prerequisites

Configuration Steps

Amazon ECR Repository Configuration

Amazon CodeCatalyst Pipeline/Workflow Configuration

Summary

Alternative Way Provision AWS Resource via Terraform (Using IAM Assume Role Reference)

Prerequisites

Configuration Steps

Conclusion

Centralizing IAM Access Key Used by IAM Machine User Using AWS Secrets Manager

Scenario

Directory Structure

Configuration

Conclusion

What’s Alternative Way to Find AWS IAM Key that has not been rotated for ages? Let’s Check Out This Tool

Prerequisites

How to Use

Notes

Backup MySQL Database That Running on Docker Container

Extend EBS Volume