DEV Community: Tarlan Huseynov

Terraform Your AWS AgentCore

Tarlan Huseynov — Tue, 24 Mar 2026 10:49:45 +0000

There is no shortage of content on agentic workflows and AI today. But most of it stops at the concept. Today we focus on what actually matters in production: the right platform to host your non-deterministic workloads and the right way to ship it. Enter Amazon Bedrock AgentCore and Terraform.

Introduction

Hey Folks! Today we are going all-in on Amazon Bedrock AgentCore and deploying it the right way, with Terraform.

Amazon Bedrock AgentCore brings together everything you need to run production AI agents on AWS: managed runtimes, a unified tool Gateway, persistent memory, Agentcore Policy: Cedar-based policy enforcement, and identity management. When we set out to build a reference implementation that exercises every one of these capabilities, the deployment choice was obvious. Terraform has been shaping cloud infrastructure for over a decade and in the age of GenAI, it's still the right answer. One monorepo, one terraform apply: three AgentCore runtimes, a Gateway connecting 21 tools, Cognito M2M auth, Cedar-based policy enforcement, and persistent session memory, all wired together through a single dependency graph.

This post covers the architecture, what that command actually orchestrates under the hood, and where the hashicorp/aws provider still has gaps on AgentCore coverage and how we bridged them without stepping outside the IaC boundary.

What is AgentCore?

Before we get into the deployment story, let's briefly cover what we're deploying on top of, because AgentCore is more than just a runtime.

Amazon Bedrock AgentCore is AWS's managed platform for running production AI agents. Instead of stitching together Lambda functions, API Gateway routes, DynamoDB tables, and custom auth flows just to give your agent somewhere to live, AgentCore handles that layer for you. Here's what it brings:

AgentCore Runtime - a secure, serverless hosting environment for your agent code, purpose-built for long-running agentic workloads (up to 8 hours), ARM64-based, scales automatically
AgentCore Gateway - a fully managed service that converts APIs, Lambda functions, and MCP servers into tools accessible by your agent, with a single unified endpoint, handling both ingress and egress authentication
AgentCore Memory - persistent session context across invocations, with short-term (within session) and long-term (across sessions) memory, and summarization strategies for older interactions
Policy in AgentCore - Cedar-based access control enforced at the Gateway boundary, controlling which tools an agent can call, deterministic, outside the model, pre-call
AgentCore Identity - credential and identity management for automated workloads, supporting both user-delegated OAuth and machine-to-machine client credentials flows

What We Built

Full source for reference github.com/tarlan-huseynov/agentcore-monorepo

The premise was straightforward: what if you could describe the infrastructure you wanted in plain English?

"Create an SQS queue called order-events in eu-central-1 with a 5-minute visibility timeout."

The agent figures out the CloudFormation schema, generates the desired state, explains what it's about to create, waits for your confirmation, and calls the Cloud Control API, which supports over 1,100 AWS resource types. Ask it what it's costing you and it queries Cost Explorer. Ask it what's in the logs and it searches CloudWatch.

The result is an Infrastructure Bootstrapper Agent, a Strands-based AI agent running on Amazon Bedrock AgentCore that manages real AWS infrastructure, analyzes costs, and searches CloudWatch Logs.

The architecture spans three AgentCore runtimes connected through a Gateway:

User Query
    |
Main Runtime          Strands Agent + AgentCore Memory
    |
AgentCore Gateway     Unified tool connectivity + Policy enforcement
    |-- CCAPI Runtime [MCP]         Cloud Control API (14 tools)
    |-- Cost Explorer Runtime [MCP]  AWS Cost Explorer (7 tools)

Standing this up from scratch involves: 3 agent runtimes, 1 Gateway, 2 Gateway targets, 5 IAM roles, 1 Cognito User Pool with M2M client, 1 OAuth2 credential provider, 1 Memory resource with summarization strategy, 3 CloudWatch log groups, 1 S3 bucket, and a Cedar policy engine.

The runtimes:

Main Orchestrator with Strands SDK ( main Agent )
AgentCore runtime for CCAPI MCP
AgentCore runtime for Cost Explorer MCP

Architecture

Why Not Console or CLI?

The console works fine for a spike. The moment you need to reproduce it, different region, colleague onboarding, teardown after a demo, you're reconstructing from screenshots and memory.

AWS CLI v3 is scriptable but stateless. You run aws bedrock-agentcore create-agent-runtime, it works, you move on. A week later you have no idea whether the policy engine is still attached to the Gateway. terraform plan tells you. aws bedrock-agentcore describe-* tells you one resource at a time.

The real problem with a multi-resource setup like this is ordering and dependency. The Gateway needs the runtimes before targets can be registered. The policy engine needs the Gateway. The main runtime needs the Gateway URL injected as an env var at creation time. Getting this right with scripts means maintaining your own dependency logic. With Terraform, you declare the references and the graph sorts it out. 😎

What One `terraform apply` Actually Does

The entry point uses the standard hashicorp/aws provider:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.32"
    }
  }
}

From there, terraform apply runs in dependency order:

S3 bucket created
null_resource.build cross-compiles Python to ARM64, uploads three ZIPs
IAM roles, Cognito User Pool, OAuth2 credential provider created in parallel
All three runtimes created, pointing at their S3 ZIPs
AgentCore Gateway created
null_resource.gateway_targets registers runtimes as Gateway targets
AgentCore Memory + summarization strategy created
null_resource.policy_setup creates the Cedar policy engine and attaches it to the Gateway

No manual steps. No README instructions that someone skips.

The `_CODE_VERSION` Trick

AgentCore caches the S3 ZIP when a runtime is first created. Uploading a new ZIP to S3 does nothing on its own, the runtime won't pick it up unless its config changes.

The fix is injecting the source hash as an environment variable. When code changes, the hash changes, the env var changes, and AgentCore detects a config update and re-fetches from S3:

resource "aws_bedrockagentcore_agent_runtime" "main" {
  environment_variables = {
    # When code changes -> hash changes -> env var changes ->
    # AgentCore detects config update and re-fetches ZIP from S3
    _CODE_VERSION = null_resource.build.triggers.source_hash
    GATEWAY_URL   = aws_bedrockagentcore_gateway.main.gateway_url
  }
}

Each MCP runtime has its own hash. Change ccapi_entrypoint.py and only the CCAPI runtime redeploys.

The Gaps and Their Real Costs

The hashicorp/aws provider is mature and battle-tested. AgentCore is a new service and full provider coverage hasn't caught up yet. Two resource types required CLI workarounds, and one required a lifecycle hack. 😊

Gap 1: Gateway Targets, Missing `grantType`

The Gateway uses Cognito Bearer tokens to authenticate outbound calls to MCP runtimes. That requires grantType: CLIENT_CREDENTIALS on the target credential config. As of ~> 6.32, the provider drops this field silently, no error, just doesn't work.

Workaround: manage targets via a null_resource (last resort solution) that calls an AWS CLI script, triggered by content hashes:

resource "null_resource" "gateway_targets" {
  triggers = {
    ccapi_arn  = aws_bedrockagentcore_agent_runtime.ccapi.agent_runtime_arn
    cost_arn   = aws_bedrockagentcore_agent_runtime.cost_explorer.agent_runtime_arn
    ccapi_code = filesha256("${path.module}/../mcp_servers/ccapi_entrypoint.py")
    cost_code  = filesha256("${path.module}/../mcp_servers/cost_entrypoint.py")
  }

  provisioner "local-exec" {
    command     = "bash scripts/setup_targets.sh"
    working_dir = "${path.module}/.."
    environment = {
      GATEWAY_ID              = aws_bedrockagentcore_gateway.main.gateway_id
      CCAPI_RUNTIME_ARN       = aws_bedrockagentcore_agent_runtime.ccapi.agent_runtime_arn
      CREDENTIAL_PROVIDER_ARN = aws_bedrockagentcore_oauth2_credential_provider.gateway_m2m.credential_provider_arn
      SCOPES                  = "mcp/invoke"
      REGION                  = local.region
    }
  }
}

The cost: null_resource outputs are opaque to terraform plan. You can't preview what the script will do before it runs. Debugging means reading shell output in the apply log. The script itself has to be idempotent, check-before-create logic you write and maintain.

Gap 2: Policy in AgentCore, No Resource Exists

There is no aws_bedrockagentcore_policy_engine resource in the provider at all. Same approach, null_resource with a shell script that calls bedrock-agentcore-control via AWS CLI:

resource "null_resource" "policy_setup" {
  triggers = {
    policy_hash = filesha256("${path.module}/policies/safety.cedar")
    gateway_id  = aws_bedrockagentcore_gateway.main.gateway_id
  }

  provisioner "local-exec" {
    command     = "bash scripts/setup_policy.sh"
    working_dir = "${path.module}/.."
    environment = {
      GATEWAY_ID  = aws_bedrockagentcore_gateway.main.gateway_id
      GATEWAY_ARN = aws_bedrockagentcore_gateway.main.gateway_arn
    }
  }

  lifecycle {
    replace_triggered_by = [
      aws_bedrockagentcore_gateway.main,
      null_resource.gateway_targets,
    ]
  }
}

That replace_triggered_by is easy to miss. Any Gateway update silently detaches the policy engine, Terraform won't flag it, the next terraform plan shows nothing wrong, and your Cedar safety rules are quietly gone until you notice.

The cost: terraform destroy doesn't clean up the policy engine or targets. You need separate teardown scripts or manual cleanup.

Gap 3: Gateway Drift on Two Fields

description and protocol_configuration aren't read back from the API after creation. Every terraform plan shows them as changed. Without the ignore_changes below, each apply would re-create the Gateway and silently detach the policy engine every single time:

lifecycle {
  ignore_changes = [description, protocol_configuration]
}

The cost: those two fields are now unmanaged by Terraform. Console changes to them won't be caught by terraform plan.

Why the Workarounds Are Still Worth It

Every script is hash-triggered, idempotent, and wired into the same dependency graph as the rest of the stack. They run at the right time, in the right order, automatically, as part of terraform apply, not after it.

The state blindspot is real but narrow: two resources that change rarely and have simple, easy-to-verify state. The alternative, a README with post-apply manual steps, is the worse trade-off in practice because those steps get skipped.

When the provider catches up, the null_resource blocks become clean resource declarations and the shell scripts get deleted. 🚀

Bonus: The ARM64 Gotcha

AgentCore runtimes run on Graviton (ARM64). Build your Python dependencies on an x86 machine and you'll get silent import errors at runtime. The packaging script uses uv with explicit platform targeting:

uv pip install \
  --python-platform aarch64-manylinux2014 \
  --python-version "3.12" \
  --target="$BUILD_DIR" \
  --only-binary=:all:

--only-binary=:all: is the critical flag. Without it, packages without ARM64 wheels fall back to compiling from source on your host architecture.

One more thing: AgentCore extracts code to /var/task, which is read-only. One of the upstream MCP packages writes a schema cache to its own package directory at import time, instant PermissionError on startup. The fix is a patched file that redirects the cache to /tmp, applied during packaging before the ZIP is built.

Farewell 😊

We covered what AgentCore brings to the table, why Terraform is still the right deployment story for it, and where the hashicorp/aws provider currently has gaps on AgentCore coverage and how to bridge them without leaving the IaC boundary.

Keep building, keep automating, and let the dependency graph do the ordering! 🚀

AWS Multi-Account IaC Sandwich: Terragrunt, Terraform, CloudFormation

Tarlan Huseynov — Wed, 12 Mar 2025 06:05:40 +0000

Introduction

Hey Folks! Today we have "IaC sandwich" on the menu, and we will look into a guide for ultimate centralized and secure AWS multi-account management with a layered strategy that ensures the best of all worlds—leveraging AWS-native capabilities, infrastructure as code flexibility, and streamlined multi-account governance.

Back in the day, while diving deep into the Terraform/Terragrunt duo and its incredible power, I found myself asking: How can I further automate infrastructure provisioning while ensuring security, scalability, and maintainability across multiple AWS accounts, while also achieving short-lived, secure authorization management (role-based) that remains seamless and controlled? I wanted a solution that would streamline access management, enforce security best practices, and provide a centralized approach to managing infrastructure.

While Terraform and Terragrunt duo already offer fantastic Infrastructure-as-Code (IaC) capabilities, there's one AWS-native service that often gets overlooked—CloudFormation StackSets. Although CloudFormation is AWS-specific and lacks the flexibility of Terraform providers, it has a significant advantage: organization-wide deployments at scale. This made me realize that CloudFormation StackSets could be a powerful addition to a Terraform/Terragrunt-based workflow, addressing the challenge of managing IAM roles securely and efficiently—if added as the first layer of my sandwich during the pre-provisioning phase—where I need an easy "init-hook" to get all the needed roles and permissions in place even before beginning the actual deployment of infrastructure.

So, in this article, I’ll walk you through how I designed a fully automated, secure, and scalable infrastructure management approach using Terraform, Terragrunt, and AWS CloudFormation StackSets—leveraging the best of each tool to create an ironclad AWS setup.

The Challenge

In multi-account AWS environments, access management and security are paramount. When using Terraform and Terragrunt, you need IAM roles that allow infrastructure automation while maintaining strict security controls. However, these roles must be pre-provisioned before Terraform can even begin managing infrastructure. This raises an important question:

How can we centrally create IAM roles across all AWS accounts in a secure and automated way?

Here's what we needed:

A centralized approach for IAM role provisioning across all AWS accounts.
A secure way to assume roles with least-privilege access.
Seamless integration with Terraform and Terragrunt for managing infrastructure.
Short-lived credentials for increased security via AWS Identity Center (SSO).

The Solution Overview

All Terraform state will be stored in the Shared-Services account. This ensures centralized state management, improving security and consistency across environments.

The Main Terraform execution role and GitHub Actions OIDC Role will also reside in the Shared-Services account. These roles will be able to assume Account-level Terraform execution roles alongside with Management account admins - so the entity that can assume this main role will be able to controle multiple accounts with iac approach.

Account-Based Terraform Execution Roles: Each AWS account (Development, Staging, Production) has its own Terraform execution role, which can be assumed when provisioning infrastructure within that specific account. Highlighting that each individual role per env has to be tailored based on least-privilege principle, yet for demo purposes our roles will have AdministratorAccess policy attached.

GitHub Actions OIDC Role: This role enables secure CI/CD automation by allowing GitHub Actions workflows to assume it for deployments.

To maintain strict security controls, we need to limit the identities that can assume the Shared-Services Terraform Execution Role. In our case:

The Master Account can assume both the Shared-Services Terraform Execution Role and the individual account-based roles directly.
Only authorized users from the Master Account will have permission to assume roles within the Shared-Services account, ensuring controlled access.

So, ultimately, the identities that should be able to assume Terraform execution roles should be very limited. In our case, these entities are:

AWS Management account Administrators
Shared-Services "terraform-execution-role"
GitHub Actions OIDC federated role

The only entities that apply anything on the infrastructure-org path—which stores the organization-level configurations that go through the management account—are AWS Management Account Administrators. Therefore, we are not providing CI automation for this section and are limiting access to this part to very few (Master Account admins).

So, getting back to the pre-provisioning phase to prepare these roles before the actual provisioning—how do we achieve that?
The answer? Leverage AWS CloudFormation StackSets to pre-provision IAM roles across all AWS accounts before using Terraform/Terragrunt against infrastructure-live. In simple terms, we will have specific management modules utilized under the infrastructure-org path before switching to infrastructure-live.

Two-Phase Deployment

Phase 1: Use CloudFormation StackSets to pre-provision IAM roles across AWS accounts.
Phase 2: Terraform/Terragrunt assumes these pre-created roles to deploy infrastructure on managed accounts.

GitOps Folder-Based Environment Approach

We also want to leverage a GitOps-style approach, structuring environments using a folder/directory per environment model. This will be combined with Terragrunt functions for seamless mapping of common variables (e.g., dynamically mapping account IDs based on environment folder names).

Closer look

Now that we have established the core concepts, let’s explore the structure of our Infrastructure-as-Code (IaC) project in detail.

All referenced example code-base is stored in GitHub.

.
├── common.hcl
├── infrastructure-live
│   ├── development
│   ├── production
│   ├── shared-services
│   │   └── gha-oidc
│   ├── staging
│   └── terragrunt.hcl
├── infrastructure-org
│   ├── root
│   │   ├── cfstacksets
│   │   └── organization
│   └── terragrunt.hcl
└── modules
    ├── cfstacksets
    ├── gha-oidc
    └── organization

For simplicity, we have only the skeleton defined for infrastructure-live, but there is no actual provisioning per environment except for the gha-oidc module in shared-services.

Key Aspects of Our IaC Structure

infrastructure-org (Organization-Level Management) - This path represents the root (management) account and delegated administrator accounts. It includes CloudFormation StackSets (cfstacksets) and AWS organizatins setup - OUs, SCPs etc. (organization). Since it manages organization-wide resources, access to this path must be strictly limited.
infrastructure-live (Environment-Specific Infrastructure) - This path contains per-environment configurations (development, staging, production, etc.). It includes shared-services, which hosts the gha-oidc module for GitHub Actions OIDC setup.
common.hcl (Shared Variables and Configuration) - This file contains shared variables used by both infrastructure-org and infrastructure-live. Common configurations such as account IDs, main region, and global settings|inputs|locals are defined here.

Terragrunt Pathing Pattern

As highlighted earlier, our Terragrunt pathing follows a structured pattern

infrastructure-path/account|env/modules

Parent terragrunt.hcl lives under the infrastructure-path, and child hcl files live on module level and source the parent hcl.

infrastructure-org

Here is a sample parent file for org path intended for organization-level management.

skip = true

terraform {
  source = "${get_repo_root()}/modules/${basename(get_terragrunt_dir())}"
}

locals {
  common_vars   = read_terragrunt_config(find_in_parent_folders("common.hcl"))
  global_prefix = local.common_vars.locals.global_prefix
  env           = "root"
  profile       = get_env("AWS_PROFILE_ROOT", "${local.global_prefix}-root-sso")
  region        = local.common_vars.inputs.region
}

inputs = merge(
  local.common_vars.inputs,
  {
    env        = local.env
    region     = local.region
    account_id = local.common_vars.inputs.org_account_ids[local.env]
  }
)

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }

  config = {
    bucket         = "${local.global_prefix}-terraform-state-root"
    key            = "${local.global_prefix}/${get_path_from_repo_root()}/terraform.tfstate"
    region         = local.region
    encrypt        = true
    dynamodb_table = "root-tfstate-lock-table"
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOF
    provider "aws" {
      region = "${local.region}"
      allowed_account_ids =["${local.common_vars.inputs.org_account_ids[local.env]}"]

      default_tags {
        tags = {
          Environment  = "${local.env}"
          ManagedBy    = "terraform"
        }
      }
    }
EOF
}

generate "versions" {
  path      = "versions.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 5.74"
        }
      }
    }
EOF
}

Provisioning the Root Account Modules

To apply the root account (call it management account) modules (cfstacksets and organization), we assume a management account administrator role and run the following command:

export AWS_PROFILE=your_management_admin_profile
terragrunt run-all apply --terragrunt-working-dir infrastructure-org

`cfstacksets` Module

The cfstacksets module provisions key IAM roles across different AWS Organizational Units (OUs). Let's examine the main components:

`stacks-sdlc.tf` (IAM Role Provisioning for SDLC Accounts)

resource "aws_cloudformation_stack_set" "terraform_role_sdlc" {
  permission_model = "SERVICE_MANAGED"
  name             = "${var.tf_role_name}-sdlc"

  auto_deployment {
    enabled = true
  }

  capabilities = ["CAPABILITY_NAMED_IAM"]

  template_body = jsonencode({
    AWSTemplateFormatVersion = "2010-09-09",
    Description              = "AWS CloudFormation Template to create an IAM Role named '${var.tf_role_name}' and attach the 'AdministratorAccess' AWS managed policy.",
    Resources = {
      OrgRole = {
        Type = "AWS::IAM::Role",
        Properties = {
          RoleName = "${var.tf_role_name}",
          AssumeRolePolicyDocument = {
            Version = "2012-10-17",
            Statement = [
              {
                Effect = "Allow",
                Principal = {
                  AWS = ["arn:aws:iam::${var.shared_services_id}:root"]
                },
                Action = ["sts:AssumeRole", "sts:TagSession"],
                Condition = {
                  StringLike = {
                    "aws:PrincipalArn" = [
                      "arn:aws:iam::${var.shared_services_id}:role/${var.tf_role_name}",
                      "arn:aws:iam::${var.shared_services_id}:role/${var.gha_role_name}"
                    ]
                  }
                }
              },
              {
                Effect = "Allow",
                Principal = {
                  AWS = ["arn:aws:iam::${var.root_account_id}:root"]
                },
                Action = ["sts:AssumeRole"],
              }
            ]
          },
          ManagedPolicyArns = [
            "arn:aws:iam::aws:policy/AdministratorAccess"
          ]
        }
      }
    }
  })

  lifecycle {
    ignore_changes = [administration_role_arn]
  }
}

resource "aws_cloudformation_stack_set_instance" "terraform_role_sdlc" {
  stack_set_name = aws_cloudformation_stack_set.terraform_role_sdlc.name
  deployment_targets {
    organizational_unit_ids = [var.org_ou_ids["sdlc"], var.org_ou_ids["production"], var.org_ou_ids["sandbox"]]
  }
}

`stacks-shared-services.tf` (IAM Role Provisioning for Shared Services)

This stack provisions an IAM role for shared services that can assume Terraform roles across multiple OUs.

resource "aws_cloudformation_stack_set" "terraform_role_shared" {
  permission_model = "SERVICE_MANAGED"
  name             = "${var.tf_role_name}-shared"

  auto_deployment {
    enabled = true
  }

  capabilities = ["CAPABILITY_NAMED_IAM"]

  template_body = jsonencode({
    AWSTemplateFormatVersion = "2010-09-09",
    Description              = <<EOT
AWS CloudFormation StackSet template to create an IAM Role named '${var.tf_role_name}' on Shared-Services
account and attach the 'AdministratorAccess' AWS managed policy. The role can be assumed by an external account with
a matching condition. Exclusively this role itself is able to assume '${var.tf_role_name}'s across the SDLC and
Production OUs. Note: Root Administrators are also able to assume target '${var.tf_role_name}'s across the SDLC
and Production OUs.
EOT
    Resources = {
      OrgRole = {
        Type = "AWS::IAM::Role",
        Properties = {
          RoleName = var.tf_role_name,
          AssumeRolePolicyDocument = {
            Version = "2012-10-17",
            Statement = [
              {
                Effect = "Allow",
                Principal = {
                  AWS = [
                    "arn:aws:iam::${var.shared_services_id}:root",
                    "arn:aws:iam::${var.root_account_id}:root"
                  ]
                },
                Action = ["sts:AssumeRole"]
              }
            ]
          },
          ManagedPolicyArns = [
            "arn:aws:iam::aws:policy/AdministratorAccess"
          ]
        }
      }
    }
  })

  lifecycle {
    ignore_changes = [administration_role_arn]
  }
}

resource "aws_cloudformation_stack_set_instance" "terraform_role_shared" {
  stack_set_name = aws_cloudformation_stack_set.terraform_role_shared.name
  deployment_targets {
    organizational_unit_ids = [var.org_ou_ids["core"]]
    account_filter_type     = "INTERSECTION"
    accounts                = [var.shared_services_id]
  }
}

This ensures that the terraform_role_shared is provisioned only in the Shared-Services account and can assume roles across different OUs securely.

infrastructure-live

Now that the organization-wide IAM roles and policies are in place, we move on to the environment-specific infrastructure provisioning under infrastructure-live. This is where Terraform/Terragrunt dynamically maps account-specific configurations, enabling seamless role assumption and execution.

Common Configuration (`common.hcl` in the root of the project)

The common.hcl file defines global configurations, including account mappings, environment inference, and shared settings.

locals {
  env_regex = "infrastructure-live/([a-zA-Z0-9-]+)/"
  env       = try(regex(local.env_regex, get_original_terragrunt_dir())[0], "shared-services")

  sdlc_account_ids = {
    development = "XXXXXXXXXXXXX"
    staging     = "XXXXXXXXXXXXX"
    production  = "XXXXXXXXXXXXX"
  }

  core_account_ids = {
    shared-services = "XXXXXXXXXXXXX"
    backups         = "XXXXXXXXXXXXX"
  }

  management_account_id = {
    root = "XXXXXXXXXXXXX"
  }

  sandbox_account_id = {
    sandbox = "XXXXXXXXXXXXX"
  }

  global_prefix = "XXXXXXXXXXXXX"
}

inputs = {
  global_prefix      = local.global_prefix
  sdlc_account_ids   = local.sdlc_account_ids
  core_account_ids   = local.core_account_ids
  org_account_ids    = merge(local.sdlc_account_ids, local.core_account_ids, local.management_account_id, local.sandbox_account_id)
  shared_services_id = local.core_account_ids["shared-services"]
  backups_id         = local.core_account_ids["backups"]
  root_account_id    = local.management_account_id["root"]
  org_units          = ["SDLC", "Production", "Core", "Sandbox"]
  tf_repo            = "XXXXXXXXXXXXX/terragrunt-infrastructure"
  tf_role_name       = "terraform-execution-role"
  gha_role_name      = "gha-role"
  gha_oidc_enabled   = true
  repo_root_path     = get_repo_root()
}

Terragrunt Configuration for `infrastructure-live` (`terragrunt.hcl`)

Each environment directory (e.g., development/, staging/, production/, etc.) will reference a common terragrunt.hcl file, ensuring consistent execution policies and automatic role assumption.

skip                          = true
terragrunt_version_constraint = ">= 0.66"
terraform_version_constraint  = ">= 1.9.0"
retryable_errors              = ["(?s).*failed calling webhook*"]
retry_max_attempts            = 2
retry_sleep_interval_sec      = 30

dependencies {
  paths = ["${get_repo_root()}/infrastructure-org/root/cfstacksets"]
}

terraform {
  source = "${get_repo_root()}/modules/${basename(get_terragrunt_dir())}"
}

locals {
  common_vars   = read_terragrunt_config(find_in_parent_folders("common.hcl"))
  region        = local.common_vars.inputs.region
  env_regex     = local.common_vars.locals.env_regex
  env           = local.common_vars.locals.env
  global_prefix = local.common_vars.locals.global_prefix
}

inputs = merge(
  local.common_vars.inputs,
  {
    env        = local.env
    region     = local.region
    account_id = local.common_vars.inputs.org_account_ids[local.env]
  }
)

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }

  config = {
    bucket         = "${local.global_prefix}-terraform-state-shared-services"
    key            = "${local.global_prefix}/${get_path_from_repo_root()}/terraform.tfstate"
    region         = local.region
    encrypt        = true
    dynamodb_table = "shared-services-tfstate-lock-table"

    assume_role = {
      role_arn = "arn:aws:iam::${local.common_vars.inputs.org_account_ids["shared-services"]}:role/${local.common_vars.inputs.tf_role_name}"
    }
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOF
    provider "aws" {
      region              = "${local.region}"
      allowed_account_ids = ["${local.common_vars.inputs.org_account_ids[local.env]}"]

      assume_role {
        role_arn = "arn:aws:iam::${local.common_vars.inputs.org_account_ids[local.env]}:role/${local.common_vars.inputs.tf_role_name}"
      }

      default_tags {
        tags = {
          Environment   = "${local.env}"
          ManagedBy     = "terraform"
          DeployedBy    = "terragrunt"
        }
      }
    }
EOF
}

generate "versions" {
  path      = "versions.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 5.74"
        }
      }
    }
EOF
}

Dynamic Mapping of Environments and Role Assumption

One of the key benefits of Terragrunt's DRY (Don't Repeat Yourself) approach is that we dynamically infer account configurations based on folder structure.

Each environment folder (infrastructure-live/development, staging, production, etc.) automatically determines its AWS account ID and region.
Local profile or CI/CD execution will dynamically assume the appropriate Terraform execution role for the target environment.

How It Works

env_regex captures the environment name from the path.
The inputs block maps the environment name to its corresponding AWS account ID.
remote_state ensures each environment uses its own Terraform state stored in the Shared-Services account.
The provider configuration automatically assumes the appropriate IAM role for infrastructure provisioning.

Executing Terraform/Terragrunt in `infrastructure-live`

With this setup, running Terraform/Terragrunt becomes straightforward. Whether executed locally or via GitHub Actions OIDC, the appropriate role is automatically assumed.

export AWS_PROFILE=your_profile # This can be Shared-Services Terraform Exec Role or Management Admin
terragrunt run-all apply --terragrunt-working-dir infrastructure-live

What Happens?

Terragrunt reads the environment directory structure (infrastructure-live/development, staging, production).
It dynamically assumes the correct IAM role for Terraform execution.
State files are stored centrally in the Shared-Services account.
The appropriate infrastructure is provisioned, following AWS best practices for multi-account security.

Farewell 😊

We've navigated the intricacies of multi-account AWS infrastructure automation using Terraform, Terragrunt, and AWS CloudFormation StackSets. By layering pre-provisioned IAM roles, dynamic environment mapping, we've inspected a secure, scalable, and streamlined approach to managing AWS at scale.

I hope this guide provides practical insights and a solid foundation. Keep refining, keep automating, and embrace the power of infrastructure as code! 🚀

Terraform, Ansible, and AWS SSM Pipeline: A Powerful Trio for seamless AWS Infrastructure Automation

Tarlan Huseynov — Mon, 29 Jul 2024 12:41:47 +0000

There are numerous materials comparing tools like Terraform vs Ansible. However, today is not about comparing them. Today is the day when we effectively combine these powerful utilities in AWS Cloud Realm!

Introduction
Prerequisites
Review
Quickstart
Building a CI/CD Pipeline
Farewell

Introduction

Hey Folks! Today, we're diving into a powerful trio for AWS infrastructure automation: Terraform, Ansible, and AWS Systems Manager (SSM). If you've ever managed cloud infrastructure, you know the challenges of provisioning, configuring, and maintaining resources efficiently.

Originally, Ansible leverages SSH connections for configuration management. While this works well, handling SSH keys and passwords for EC2 instances can be cumbersome and pose security risks. Enter AWS Systems Manager (SSM), a game-changer that simplifies and secures the management of your instances. SSM Session Manager allows you to access your EC2 instances without needing SSH, making the process smoother and more secure.

We'll start by setting up Terraform to provision our infrastructure. Then, we'll use Ansible, leveraging SSM sessions, to handle configuration tasks. This includes patching our EC2 instances, installing necessary tools, and even managing application installations—all done idempotently.

Prerequisites

We'll need a few tools and setups. You can find the entire codebase for this project on GitHub.

Terraform: The cornerstone of our infrastructure provisioning. You can install Terraform from the official site. Install Terraform.
AWS CLI: Necessary for interacting with AWS services. Install AWS CLI and configure it with your AWS credentials.
Ansible: Alongside with the king of CaC - we will use Ansible's SSM plugin for secure and efficient management of our EC2 instances. You can find more about the plugin and installation instructions here.
AWS SSM Plugin: This plugin allows us (and Ansible plugin) to connect to EC2 instances using SSM sessions, avoiding the need for SSH authorization. Download and install the AWS SSM session manager plugin from the official AWS documentation.

Docker Quickstart

For a seamless setup, you can use the Docker image built from the Dockerfile provided in the GitHub repository. This image contains all necessary tools and configurations, allowing you to start without individual installations.
Docker: You can install Docker from the official site. Install Docker.

Review

The backend.sh script in our project initializes the backend for Terraform. It sets up an S3 bucket to store the Terraform state files and a DynamoDB table to manage state locking and consistency. This ensures that our infrastructure state is safely stored and prevents simultaneous operations that could corrupt the state.

The s3.tf file sets up an S3 bucket for storing files related to our project, such as Ansible playbooks and configuration files. This bucket is essential as it provides a centralized location for these artifacts, which can be accessed by our EC2 instances during their configuration process. The AWS SSM plugin manages the transfer of these files seamlessly in the background, ensuring that all necessary files are available to the instances without manual intervention.

resource "aws_s3_bucket" "main" {
  bucket = "ansible-bucket-4-demo"

  tags = merge({
    Name = "ansible-bucket-4-demo",
  }, local.default_tags)
}

It’s important to ensure that the bucket name is unique across all of AWS.

The main.tf file contains the configuration for our AWS EC2 instances and all the supplementary resources. It defines the necessary AWS resources, such as subnets, AMIs, IAM roles, security groups, and the EC2 instance itself. Notably, it uses the existing default subnet, an Ubuntu 22.04 image, and a simple security group setup - for demo purposes.

The ansible.tf file integrates Ansible with our Terraform setup. It uses Terraform's local-exec provisioner to run Ansible playbooks for configuring the EC2 instances. This setup allows us to automate the configuration and management of instances using Ansible, leveraging SSM sessions for secure and seamless connections.

resource "time_sleep" "wait_60_seconds" {
  depends_on      = [aws_instance.main]
  create_duration = var.revision == 1 ? "60s" : "0s"
}

resource "null_resource" "ansible_os_patch" {
  depends_on = [time_sleep.wait_60_seconds]

  provisioner "local-exec" {
    command = "ansible-playbook ${path.root}/playbooks/os_patch.yaml"

    environment = {
      SSM_BUCKET_NAME = aws_s3_bucket.main.id
      SSM_INSTANCE_ID = aws_instance.main.id
    }
  }

  triggers = {
    playbook_update = filesha256("${path.root}/playbooks/os_patch.yaml")
    ssm_instance_id = aws_instance.main.id
    revision        = var.revision
  }
}

resource "null_resource" "ansible_app_install" {
  depends_on = [
    time_sleep.wait_60_seconds,
    null_resource.ansible_os_patch
  ]

  provisioner "local-exec" {
    command = "ansible-playbook ${path.root}/playbooks/app_install.yaml"

    environment = {
      SSM_BUCKET_NAME = aws_s3_bucket.main.id
      SSM_INSTANCE_ID = aws_instance.main.id
    }
  }

  triggers = {
    playbook_update = filesha256("${path.root}/playbooks/app_install.yaml")
    ssm_instance_id = aws_instance.main.id
    revision        = var.revision
  }
}

The time_sleep resource ensures a delay before provisioning starts, with the duration dependent on the revision variable. This is crucial for the first launch when we need to wait for initial status checks and instance start, and also additional adjustments might be needed. The null_resource triggers ensure that changes in the playbook or instance ID trigger a re-run of the playbooks, maintaining idempotency.

And ofc Playbooks to be used by terraform local-exec, already adjusted to leverage ssm for connection: ansible_connection: community.aws.aws_ssm

Quickstart

In this section, we'll build our Docker image and mount the current directory to interactively go through the process. Follow these steps after cloning the repository:

Build and verify the Docker Image:

   cd terraform-ansible-ssm
   docker build -t terraform-ansible-ssm . && docker run -it terraform-ansible-ssm

Run and attach the Docker Container:

   docker run --rm -it -v $PWD:/app terraform-ansible-ssm  /bin/sh
   cd /app

Configure AWS CLI: Configure the AWS CLI by setting up a default profile or exporting environment variables:

   aws configure
   #Alternatively, export the necessary environment variables:
   export AWS_REGION=<your-region>
   export AWS_ACCESS_KEY_ID=<your-access-key-id>
   export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>

For more details, refer to the AWS CLI configuration documentation.

Initialize Terraform Backend & Validate & plan: Run the backend.sh script to set up the Terraform backend, add a randomizer string to make the state bucket name unique:

   RANDOM_STRING=<your-random-string> ./backend.sh

The backend.sh script will populate the backend configuration in backend.tf. Initialize Terraform with the following command:

   terraform init && terraform validate && terraform plan

It's the terraform apply moment!

   terraform apply

Verify the Provisioned EC2 Instance: After applying the configuration, grab the output to check your freshly provisioned EC2 instance with all installed dependencies and applications. For example:

   Instance_public_dns = "ec2-18-191-254-165.us-east-2.compute.amazonaws.com"

Boom and it works!

Building a CI/CD Pipeline

To set up a CI/CD pipeline (or name it Continuous Provisioning pipeline 😎) using GitHub Actions, we'll create a workflow that builds our Docker image, runs Terraform, and applies our infrastructure changes automatically.

Tagging and Pushing Docker Image

First we will make sure to tag and push the Docker image to a public repository, as we will use our image as the runtime for our pipeline:

docker tag terraform-ansible-ssm wardove/terraform-ansible-ssm
docker push wardove/terraform-ansible-ssm

Workflow Configuration

We have a workflow file in the repository at .github/workflows/main.yml with the following content:

name: Terraform CI/CD
run-name: "Terraform CI/CD | triggered by @${{ github.actor }}"

on:
  push:
    branches:
      - 'main'

jobs:
  terraform-apply:
    runs-on: ubuntu-latest
    container:
      image: wardove/terraform-ansible-ssm:latest
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_REGION: us-east-2

    steps:
      - uses: actions/checkout@v4

      - name: Terraform Init
        run: terraform init

      - name: Terraform Validate
        run: terraform validate

      - name: Terraform Apply
        run: terraform apply -auto-approve

Adding Secrets to GitHub

To allow GitHub Actions to interact with your AWS environment, you'll need to add your AWS credentials to GitHub secrets:

Go to your repository on GitHub.
Navigate to Settings -> Secrets -> New repository secret.
Add the following secrets:
- AWS_ACCESS_KEY_ID: Your AWS access key ID.
- AWS_SECRET_ACCESS_KEY: Your AWS secret access key.

Note: There are more secure alternatives which provide access to GHA, so consider this approach only for demo purposes! and ofc we would do PRs instead of direct pushes on prod 😋

Once everything is in place, we commit and push the changes to the main branch of your repository. This will trigger the GitHub Actions workflow, which will run the Terraform scripts to apply our infrastructure changes!

And success again! But no infrastructure changes yet 🤓

There is also an alternative approach (without ansible plugin usage) with SSM Run-command and aws-cli to leverage Ansible with SSM, a flow that is very professionally described in this AWS Blog by Andres Silva.

Farewell 😊

We’ve journeyed through an effective method of AWS infrastructure automation using Terraform, Ansible, and AWS Systems Manager. By integrating these powerful tools, we've simplified and automated the management of cloud resources. Thank you for following along with this article. I hope it has provided you with valuable insights and serves as a useful reference for your future projects. Keep exploring, keep learning, and continue refining your cloud automation practices!

One tool to rule them all - Terraform: EKS Golang Client & E2E AWS Lambda CI/CD via IaC

Tarlan Huseynov — Thu, 04 Jul 2024 22:33:59 +0000

Introduction
Prerequisites
Lambda RBAC permissions
Farewell

Introduction

Hey Folks! In this article, we delve into the seamless integration of Terraform for deploying compiled Lambda functions. During the process I will showcase how AWS Lambda can be effectively used as a Kubernetes client. And as final layer of our sandwich - E2E Terraform CI/CD with GitHub Actions to facilitate continuous infrastructure provisioning and software delivery all in one!

So, here is some backstory. Previously, I wondered how to facilitate a more reactive and event driven interaction with my EKS cluster in a cloud-native environment. And naturally when I think of event-driven approach in AWS context - lambdas and EventBridge are one of the first things that come to mind, and for a good reason 😁 right? Combination of these 2 can provide endless number of solutions for various problems - where an action is needed based on some event or some specific schedule.
We will be looking at one of them, particularly - "How to scale down EKS workloads on a scheduled basis".

We will be using my humble EKS-downscaler app. As a client, it provides similar functionality to an operator with the same name. This app is conceptual and serves as an example of how AWS Lambdas can interact with Kubernetes. With our solution, we will specify namespaces and cron expressions, and Terraform coupled with downscaler lambda will handle everything. This article will showcase Terraform from a perfect angle - as it will be our main tool to manage continuous configuration changes, provision and manage cloud resources, and deliver seamless code-to-Lambda deployment. This approach can be used as a standalone terraform project or integrated into more complex IaC projects by passing inputs directly (e.g., "var.cluster_name" that is required as input could have been passed via remote state or directly from root/child module where we create the eks cluster).

Prerequisites

As you might have guessed, we have some prerequisites:

Terraform
AWS CLI
Go runtime
EKS cluster

You can provision and create and manage your EKS cluster in any way you prefer. It might be App of Apps, eksctl, or manual provisioning, as it is very individual. However, I will be showcasing example kubernetes resources with Terraform code examples.

So, as mentioned earlier - with our EKS cluster managed via Terraform, this project can be a part of it as a separate module or a standalone project decoupled from main code basis. In this article, it will be separate.

Lambda RBAC permissions

To enable our Lambda function to interact with the EKS cluster, we need to grant it specific permissions using Role-Based Access Control (RBAC). This involves defining a Kubernetes group, by creating a cluster role, and establishing a cluster role binding. I am creating those using my main EKS Terraform project. You can go ahead and do the same manually 🙃

Further, We can associate this group with our Lambda role by creating an access entry in the separate project where we will provision the Lambda itself or right in the main eks project or once again go and do it manually via console. For now let's just create RBAC components:

resource "kubernetes_cluster_role" "lambda" {
  metadata {
    name = "lambda-clusterrole"
  }
  rule {
    api_groups = ["*"]
    resources  = ["deployments", "deployments/scale", "statefulsets", "daemonsets", "jobs"]
    verbs      = ["get", "list", "watch", "create", "update", "patch", "delete"]
  }
}

resource "kubernetes_cluster_role_binding" "lambda" {
  metadata {
    name = "lambda-clusterrolebinding"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.lambda.metadata[0].name
  }
  subject {
    kind      = "Group"
    name      = "lambda-group"
    api_group = "rbac.authorization.k8s.io"
  }
}

EKS Lambda Client

Now, let's dive into the heart of our project: the EKS Lambda Client. You'll find all the relevant code to deploy our Lambda function using Terraform in this GitHub repo.

eks-downscaler
├── .github
│   └── workflows
│       └── terraform.yml
├── lambdas
│   └── downscaler
│       ├── go.mod
│       └── main.go
│       
├── modules
│   └── downscaler
│       ├── iam.tf
│       ├── lambda.tf
│       ├── locals.tf
│       ├── scheduler.tf
│       └── variables.tf
├── backend.sh
├── backend.tf
├── main.tf
├── readme.md
├── terraform.tfvars
└── variables.tf

Our repository contains both the Golang Lambda code and the Terraform scripts that manage its deployment. We'll be exploring each component step-by-step, starting with the Terraform S3 backend bootstrapper script. As always this is where the magic begins, setting up the backend for our infrastructure.

backend.sh

#!/bin/bash
set -euo pipefail

PROJECT_NAME=$(basename "$(dirname \"${PWD}\")")
AWS_REGION=${AWS_REGION:-us-east-2}
AWS_PROFILE=${AWS_PROFILE:-default}
AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text --profile ${AWS_PROFILE})"
export AWS_PAGER=""

echo -e "Bootstraping terraform backend...\n"
echo PROJECT_NAME: "${PROJECT_NAME}"
echo AWS_REGION: "${AWS_REGION}"
echo AWS_PROFILE: "${AWS_PROFILE}"
echo AWS_ACCOUNT_ID: "${AWS_ACCOUNT_ID}"
echo BUCKET NAME: "terraform-tfstate-${PROJECT_NAME}"
echo DYNAMODB TABLE NAME: terraform-locks
echo -e "\n"

aws s3api create-bucket \
    --region "${AWS_REGION}" \
    --create-bucket-configuration LocationConstraint="${AWS_REGION}" \
    --bucket "terraform-tfstate-${PROJECT_NAME}" \
    --profile "${AWS_PROFILE}"

aws dynamodb create-table \
    --region "${AWS_REGION}" \
    --table-name terraform-locks \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 \
    --profile "${AWS_PROFILE}"

cat <<EOF > ./backend.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.57"
    }
  }

  required_version = ">=1.9.0"

  backend "s3" {
    bucket         = "terraform-tfstate-${PROJECT_NAME}"
    key            = "${PROJECT_NAME}"
    region         = "${AWS_REGION}"
    dynamodb_table = "terraform-locks"
  }
}

provider "aws" {}
EOF

echo -e "\nBackend configuration created successfully!\n"
cat ./backend.tf

This script bootstraps a Terraform backend for managing state files. It captures the project name, AWS region, profile, and account ID. Then, it creates an S3 bucket and a DynamoDB table for state file storage and locking, respectively. Finally, it generates a backend.tf configuration file, linking Terraform to the newly created S3 bucket and DynamoDB table, ensuring secure and organized state management. So to initialize first you will need to define your Access credentials either by using AWS_PROFILE or AWS_SECRET_ACCESS_KEY & AWS_ACCESS_KEY_ID.

Main Configuration File

It would be redundant to go over every file in our repo, as the hcl + go code is defined in idiomatic way, and all the nitty-gritty part should be clear already (iam role and policies for lambda to describe our cluster, EventBridge scheduler resource and its permissions.. all the extras and bla bla bla..)

Our main.tf is the main spot where we define all the configuration for our client. This file references the downscaler module, specifying the EKS cluster name, Lambda source path, scaling schedules, and namespaces to manage. It's like the conductor of an orchestra, ensuring every component works in perfect harmony.

module "downscaler_lambda_client" {
  source             = "./modules/downscaler"
  eks_cluster_name   = var.cluster_name
  ci_env             = var.ci_env
  lambda_source      = "${path.root}/lambdas/downscaler"
  scale_out_schedule = "cron(00 09 ? * MON-FRI *)"
  scale_in_schedule  = "cron(00 18 ? * MON-FRI *)"
  eks_groups         = ["lambda-group"]
  namespaces         = ["development", "test"]
}

Lambda Deployment Process

Now, let's talk about how we build, zip, and deploy our Lambda function in modules/downscaler/lambda.tf.

resource "null_resource" "lambda_build" {
  provisioner "local-exec" {
    working_dir = var.lambda_source
    command     = "go mod tidy && GOARCH=amd64 GOOS=linux go build -o bootstrap main.go"
  }

  triggers = {
    ci_env    = var.ci_env
    file_hash = md5(file("${var.lambda_source}/main.go"))
  }
}

data "archive_file" "lambda_zip" {
  depends_on  = [null_resource.lambda_build]
  type        = "zip"
  source_file = "${var.lambda_source}/bootstrap"
  output_path = "${var.lambda_source}/main.zip"
}

resource "aws_lambda_function" "downscaler_lambda" {
  filename         = data.archive_file.lambda_zip.output_path
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  function_name    = var.project_name
  handler          = "main"
  runtime          = "provided.al2023"
  role             = aws_iam_role.lambda_role.arn

  environment {
    variables = { CLUSTER_NAME = var.eks_cluster_name }
  }
}

resource "aws_eks_access_entry" "lambda" {
  cluster_name      = var.eks_cluster_name
  principal_arn     = aws_iam_role.lambda_role.arn
  kubernetes_groups = var.eks_groups
  type              = "STANDARD"
}

Building the Lambda: We begin by checking for changes in our main.go file. Using a hash function, we detect any modifications and trigger a rebuild of the Lambda function. This ensures our deployment is always up-to-date with the latest code changes.
We also have a cheeky ci_env = var.ci_env trigger here, which identifies if we are running the project locally or from CI. Motive is to rebuild our application every time if we are applying in Github Actions CI Context.
Zipping the Lambda: After building the Lambda, we zip the compiled executable. This zipped file becomes the core package for our Lambda function, ready for deployment.
Deploying the Lambda: With Terraform, we deploy the Lambda function using the zipped file. Terraform handles all the heavy lifting, ensuring the Lambda is correctly set up and configured to interact with our EKS cluster.
Assigning Permissions: We also need to attach the necessary permissions (previously created RBAC resources) to local Lambda role by using Access Entries.
All AWS sided permissions -> iam resources - roles, necessary policies are defined in iam.tf file

This elegant process, managed by Terraform, ensures that our Lambda function is built, zipped, and deployed seamlessly.

Lambda client code

package main

import (
    "context"
    "encoding/json"
    "github.com/aws/aws-lambda-go/lambda"
    eksauth "github.com/chankh/eksutil/pkg/auth"
    log "github.com/sirupsen/logrus"
    autoscalingv1 "k8s.io/api/autoscaling/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "os"
)

type Payload struct {
    ClusterName string   `json:"clusterName"`
    Namespaces  []string `json:"namespaces"`
    Replicas    int32    `json:"replicas"`
}

func main() {
    if os.Getenv("ENV") == "DEBUG" {
        log.SetLevel(log.DebugLevel)
    }

    lambda.Start(handler)
}

func handler(ctx context.Context, payload Payload) (string, error) {
    cfg := &eksauth.ClusterConfig{
        ClusterName: payload.ClusterName,
    }

    clientset, err := eksauth.NewAuthClient(cfg)
    if err != nil {
        log.WithError(err).Error("Failed to create EKS client")
        return "", err
    }

    scaled := make(map[string]int32)

    for _, ns := range payload.Namespaces {
        deployments, err := clientset.AppsV1().Deployments(ns).List(ctx, metav1.ListOptions{})
        if err != nil {
            log.WithError(err).Errorf("Failed to list deployments in namespace %s", ns)
            continue
        }

        for _, deploy := range deployments.Items {
            if err := scaleDeploy(clientset, ctx, ns, deploy.Name, payload.Replicas); err == nil {
                scaled[ns+"/"+deploy.Name] = payload.Replicas
            }
        }
    }

    scaledJSON, err := json.Marshal(scaled)
    if err != nil {
        log.WithError(err).Error("Failed to marshal scaled deployments to JSON")
        return "", err
    }

    log.Info("Scaled Deployments: ", string(scaledJSON))
    return "Scaled Deployments: " + string(scaledJSON), nil
}

func scaleDeploy(client *kubernetes.Clientset, ctx context.Context, namespace, name string, replicas int32) error {
    scale := &autoscalingv1.Scale{
        ObjectMeta: metav1.ObjectMeta{
            Name:      name,
            Namespace: namespace,
        },
        Spec: autoscalingv1.ScaleSpec{
            Replicas: replicas,
        },
    }

    _, err := client.AppsV1().Deployments(namespace).UpdateScale(ctx, name, scale, metav1.UpdateOptions{})
    if err != nil {
        log.WithError(err).Errorf("Failed to scale deployment %s in namespace %s", name, namespace)
    } else {
        log.Infof("Successfully scaled deployment %s in namespace %s to %d replicas", name, namespace, replicas)
    }
    return err
}

Our Lambda client code adds the last piece of logic behind scaling operations in the EKS cluster. It starts by defining a Payload structure, which includes the cluster name, namespaces, and desired replicas. The main function sets up the Lambda handler, which initiates the scaling process. And obviously we are picking the name of the cluster as an environment variable - which is originally propagated via terraform resource.

The handler creates an EKS client, lists deployments in the specified namespaces, and scales each deployment to the desired number of replicas. The scaled deployments are then logged and returned as a JSON response. This ensures our deployments are dynamically scaled based on the defined schedules, providing efficient resource management.

Naturally, all logs flow directly to CloudWatch Logs, where we can observe all the details regarding invocations and it's output.

Terraform CI/CD

It's continuous delivery and continuous provisioning time! 🚀🚀🚀
Our CI/CD workflow is defined in the terraform.yml file within the .github/workflows directory. This workflow ensures that our Terraform configurations are automatically applied whenever changes are pushed to the main branch.

name: Terraform CI/CD
run-name: "Terraform CI/CD | triggered by @${{ github.actor }}"

on:
  push:
    branches:
      - 'main'

jobs:
  terraform-apply:
    runs-on: ubuntu-latest

    env:
      TF_VAR_cluster_name: ${{ secrets.CLUSTER_NAME }}
      TF_VAR_ci_env: true
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_REGION: us-east-2

    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.0

      - name: Terraform Init
        run: terraform init

      - name: Terraform Validate
        run: terraform validate

      - name: Terraform Apply
        run: terraform apply -auto-approve

The CI/CD pipeline kicks off by checking out the code from the repository and setting up Terraform. It then initializes Terraform, validates the configuration, and applies the changes to deploy our infrastructure.

It is always better to have extra linting/testing/scanning in terraform CI, another topic for another day maybe😉

Secrets and Environment Variables

Key secrets and environment variables are passed into the workflow to ensure secure and proper configuration. We use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for authenticating with AWS, and CLUSTER_NAME to specify the EKS cluster name. These secrets are securely stored in GitHub's repository settings.

As you may already know TF_VAR_ prefix, is handy to override/define values via environment variables - especially in CI environment. This ensures that even if variables are defined elsewhere or gitignored (in our case tfvars files are gitignored), our CI/CD pipeline uses the CI values. For example, TF_VAR_ci_env is set to true in the CI environment, enforcing rebuilds (via local-provisioner trigger) and ensuring changes are accurately reflected in the deployment.

Farewell 😊

In this article, we’ve explored a comprehensive method for deploying and managing AWS resources using Terraform, AWS Lambda, and EventBridge. We delved into the seamless integration of Terraform for deploying compiled Lambda functions, showcased how AWS Lambda can effectively interact with EKS, and implemented full-scale automation with GitHub Actions. These three pearls highlight the power of Terraform in creating a cohesive and dynamic deployment strategy.

Thank you for following along. I hope this guide has provided you with valuable insights and can serve as a reference for your future projects. Keep exploring, keep learning, and continue refining your cloud practices!

"One tool to rule them all, one tool to deploy them, One tool to automate 'em, and with terraform apply them; in the cloud where serverless hides"

EKS Secret Management — with Golang, AWS ParameterStore and Terraform

Tarlan Huseynov — Sun, 09 Jun 2024 12:08:38 +0000

Introduction
InitContainer with GO binary
OIDC Federated Access for EKS Pods
Farewell

Introduction

Hey Folks! In this article, we are going to delve into a robust approach to Kubernetes secret management by utilizing the efficiency of Golang, the security and flexibility of AWS ParameterStore, the authentication power of OIDC, and the infrastructure-as-code advantages of Terraform. We will explore ways to enhance your cloud-based applications and significantly bolster your security posture, providing you with a comprehensive understanding of this innovative strategy for revolutionizing your secret management processes.

Keep in mind that we have a few prerequisites. To fully engage with the material and examples we provide, you’ll need an AWS Account, an EKS Cluster, and a configured Terraform project with the AWS provider.

SSM Parameters

To kick things off, let’s explore how we can manage secrets using AWS Systems Manager (SSM) Parameter Store. This service from AWS provides secure, hierarchical storage for configuration data management and secrets. Leveraging the Parameter Store can significantly enhance the security posture of your applications by segregating and securing sensitive information like database passwords, license codes, and API keys.

Let’s consider a Terraform script to create these SSM parameters, starting with a locals block. This block includes the projects in which we want to manage the secrets and the keys that need to be stored securely.

locals {
  projects = {
    demo-project = {
      team_id                 = "demo_team"
      namespace               = "demo"
      platform                = "fargate"
      fqdn                    = "www.huseynov.net"
      ssm_secret_keys         = ["AMQ_USER", "AMQ_PASS"]
      deployment_grace_period = 60
      vpa_update_mode         = "Initial"
      svc_config              = {
        service_port   = 8080
        target_port    = 80
        type           = "NodePort"
        lb_access_mode = "internet-facing"
        alb_group      = "demo"
      }
      hpa_config = {
        min           = 1
        max           = 3
        mem_threshold = 80
        cpu_threshold = 60
      }
    }
  }

  ssm_secret_keys = { for v in flatten([
    for project_name, parameters in local.projects : [
      for key in try(parameters.ssm_secret_keys, []) : {
        key          = key,
        namespace    = parameters.namespace,
        project_name = project_name,
        team_id      = try(parameters.team_id, var.cluster_namespace)
      }
    ]
    ]) : "${v.namespace}.${v.project_name}.${v.key}" => v
  }
}


resource "aws_ssm_parameter" "project_ssm_secrets" {
  for_each = local.ssm_secret_keys
  name     = "/eks/${var.cluster_name}/${each.value.namespace}/${each.value.project_name}/${each.value.key}"
  type     = "SecureString"
  value    = "placeholder"
  tags = merge(local.default_tags, {
    "eks.namespace" = each.value.namespace
    "eks.project"   = each.value.project_name
    "eks.team_id"   = each.value.team_id
    }
  )
  lifecycle {
    ignore_changes  = [value]
    prevent_destroy = true
  }
}

Please note that in our locals block, several parameters are of particular importance for this tutorial:

ssm_secret_keys: This specifies the secrets that we need to securely manage. These are essentially the names of the secrets we are storing in the AWS ParameterStore.
namespace: This identifies the Kubernetes namespace where our resources will be located.
team_id: It's used to denote the team that owns the particular resource. It is optional, but further it can be used for ABAC for AWS.
project_name: This is the name of the project under which the resources will fall.

These are used in our aws_ssm_parameter block to create secure parameters, which are initially set with placeholder values.

The remaining parameters, such as platform, fqdn, deployment_grace_period, svc_config, and hpa_config, although not explicitly used in our SSM parameters creation script, can be utilized within the same Terraform project to create various resources with AWS and Kubernetes providers. These could include load balancers, Horizontal Pod Autoscalers, and other vital components for our cloud infrastructure, and they contribute to the flexibility and comprehensiveness of the system we are setting up.

The aws_ssm_parameter block creates the SSM parameters. Each SSM parameter is given a placeholder value to initialize it. The actual values will be input later by a dedicated team member, such as a developer or a DevOps engineer, with sufficient permissions. This can be done via the AWS console or command-line interface (CLI).

It’s important to note that storing these values directly from the Terraform project is not advisable because they would end up in the Terraform state. This is a situation we want to avoid for security reasons, as we don’t want sensitive information like secrets stored in the Terraform state.

InitContainer with GO binary

Moving on to the next crucial step, we need to prepare the init container Golang binary. This binary will fetch our secrets from the AWS Parameter Store and write them into a file. Kubernetes will then load these secrets from the file into environment variables for use by our applications.

For this, we’ll write a small program in Go. The script’s operation can be summarized as follows: It creates an AWS session, uses the SSM client to fetch the secrets we stored earlier, writes the secrets into an environment file (.env), and then stores this file in a specific path (/etc/ssm/). The environment variables stored in this file will be available to other containers in the pod once loaded.

main.go

package main

import (
 "bufio"
 "fmt"
 "github.com/aws/aws-sdk-go/aws"
 "github.com/aws/aws-sdk-go/aws/session"
 "github.com/aws/aws-sdk-go/service/ssm"
 "os"
 "path"
)

func main() {
 sess, err := session.NewSession(&aws.Config{
  Region: aws.String(os.Getenv("AWS_REGION")),
 })

 if err != nil {
  fmt.Println("Failed to create session,", err)
  return
 }

 ssmSvc := ssm.New(sess)

 eks_cluster := os.Getenv("EKS_CLUSTER")
 namespace := os.Getenv("NAMESPACE")
 ci_project_name := os.Getenv("CI_PROJECT_NAME")

 ssmPath := fmt.Sprintf("/eks/%s/%s/%s", eks_cluster, namespace, ci_project_name)

 paramDetails := &ssm.GetParametersByPathInput{
  Path:           aws.String(ssmPath),
  WithDecryption: aws.Bool(true),
 }

 resp, err := ssmSvc.GetParametersByPath(paramDetails)
 if err != nil {
  fmt.Println("Failed to get parameters,", err)
  return
 }

 file, err := os.Create("/etc/ssm/.env")
 if err != nil {
  fmt.Println("Failed to create file,", err)
  return
 }
 defer file.Close()

 writer := bufio.NewWriter(file)
 for _, param := range resp.Parameters {
  name := path.Base(*param.Name)
  value := *param.Value
  writer.WriteString(fmt.Sprintf("export %s=%s\n", name, value))
 }

 err = writer.Flush()
 if err != nil {
  fmt.Println("Failed to write to file,", err)
 }
 fmt.Println("env file created successfully")
}

Dockerfile

FROM golang:1.20-alpine as BUILD
WORKDIR /app
COPY . .
RUN go build -o main

FROM alpine:3.16 AS RUNTIME
WORKDIR /app
COPY --from=BUILD /app/main .
CMD ["./main"]

In this Go script:

We start by creating a new AWS session and initializing an SSM client using the aws-sdk-go package.
We retrieve the environment variables for the EKS cluster, the namespace, and the project name. These will be used to construct the path to our secrets stored in the SSM Parameter Store.
With the SSM client, we fetch the secrets from the Parameter Store using the GetParametersByPath method. This method retrieves all parameters within the provided path.
We then create a .env file and write the fetched parameters into this file. Each line in the file will contain one secret, with the syntax export SECRET_NAME=secret_value.

Great, with the init container’s Go script ready, the next step is building this into a Docker image and pushing it to the Amazon Elastic Container Registry (ECR). Ideally, this should be part of your CI/CD process. Here’s a condensed guide to help you achieve this manually:

# Build the image, authenticate Docker to ECR, and push the image
docker build -t ssm-init-container . 
aws ecr get-login-password | docker login --username AWS --password-stdin your-account-id.dkr.ecr.region.amazonaws.com
aws ecr create-repository - repository-name ssm-init-container
docker tag ssm-init-container:latest your-account-id.dkr.ecr.region.amazonaws.comssm-init-container:latest 
docker push your-account-id.dkr.ecr.region.amazonaws.com/ssm-init-container:latest

Please replace ‘your-account-id’ and ‘region’ with your AWS account ID and your region, respectively.

OIDC Federated Access for EKS Pods

Before we proceed to actual deployments, we need to ensure the IAM roles are correctly associated with the Kubernetes service accounts. This is vital for the secure operation of our applications, allowing them to access necessary AWS resources.

Our Terraform scripts will take care of this association using the concept of IAM Roles for Service Accounts (IRSA) in AWS EKS, facilitated by OpenID Connect (OIDC) federation. It’s a recommended best practice to follow to ensure fine-grained access control to AWS services, directly from within the EKS environment. This eliminates the need to provide broad access permissions at the node level and greatly enhances our application’s security.

For enhanced security, AWS EKS allows IAM roles to be assigned to Kubernetes Service Accounts through a feature called IAM Roles for Service Accounts (IRSA). This mechanism uses OpenID Connect (OIDC) federation to drive the mapping between Kubernetes service accounts and AWS IAM roles. This section shows how to implement it using Terraform.

The following guides from AWS EKS are recommended reading to gain a deeper understanding of the topic:

Here’s the Terraform code that sets up the IAM roles, policies and associates them with Kubernetes service accounts:

# Assume Role Policy for service accounts
data "aws_iam_policy_document" "service_account_assume_role" {
  for_each = local.projects
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc_provider_sts.url, "https://", "")}:aud"
      values   = ["sts.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc_provider_sts.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:${each.value.namespace}:${each.key}"]
    }
    principals {
      identifiers = [aws_iam_openid_connect_provider.oidc_provider_sts.arn]
      type        = "Federated"
    }
  }
}

# SSM Access permissions
resource "aws_iam_role" "service_account_role" {
  for_each           = local.projects
  assume_role_policy = data.aws_iam_policy_document.service_account_assume_role[each.key].json
  name               = "project-${each.key}-service-account-role"
  tags               = local.default_tags
}

# IAM Policy for SSM secrets
data "aws_iam_policy_document" "ssm_secret_policy" {
  for_each = local.projects
  statement {
    effect = "Allow"
    actions = ["ssm:GetParametersByPath", "ssm:GetParameters"]
    resources = [
      "arn:aws:ssm:${local.region}:${local.account_id}:parameter/eks/${var.cluster_name}/${each.value.namespace}/${each.key}*"
    ]
  }
}

resource "aws_iam_policy" "ssm_secret_policy" {
  for_each    = local.projects
  name        = "project-${each.key}-ssm-access"
  description = "Policy to allow EKS pods/projects to access respective SSM parameters"
  policy      = data.aws_iam_policy_document.ssm_secret_policy[each.key].json
  tags        = local.default_tags
}

resource "aws_iam_role_policy_attachment" "service_account_role_ssm" {
  for_each   = local.projects
  role       = aws_iam_role.service_account_role[each.key].name
  policy_arn = aws_iam_policy.ssm_secret_policy[each.key].arn
}

This code sets up an IAM role for each service account, granting it permissions to access specific SSM parameters. The service account is then mapped to the IAM role using OIDC. It’s a great approach to securely handle permissions and access secrets in EKS environments.

Remember to replace the placeholders with actual values before running the script. Make sure you also have the appropriate AWS access and permissions to execute these commands.

With this setup, your applications running in Kubernetes can securely and efficiently access the resources they need to function, all while adhering to the principle of least privilege.

Application Deployment

With our Go-based init container built and securely stored in ECR, let’s move on to deploying a demo application. For this, we’ll need an entrypoint.sh shell script, a Dockerfile for our application, and a Kubernetes Deployment manifest.

entrypoint.sh

FROM golang:1.20-alpine as BUILD
WORKDIR /build
COPY . .
RUN go mod tidy
RUN go build -o main
FROM alpine:3.16 AS RUNTIME
WORKDIR /app
COPY - from=BUILD /build .
RUN chmod +x entrypoint.sh
ENTRYPOINT ["./entrypoint.sh"]

To give you a head start, here is a simple Go application you can deploy for testing purposes: Go Resume Demo

Lastly, we have our Kubernetes Deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ${NAMESPACE}
  name: ${PROJECT_NAME}
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ${PROJECT_NAME}
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ${PROJECT_NAME}
        app.kubernetes.io/environment: ${EKS_CLUSTER}
        app.kubernetes.io/owner: Devops
    spec:
      serviceAccountName: ${PROJECT_NAME}
      initContainers:
        - name: secret-gopher
          image: ${ECR_REGISTRY}/<INIT_CONTAINER_NAME>:latest
          env:
            - name: EKS_CLUSTER
              value: ${EKS_CLUSTER}
            - name: NAMESPACE
              value: ${NAMESPACE}
            - name: CI_PROJECT_NAME
              value: ${PROJECT_NAME}
          volumeMounts:
            - name: envfile
              mountPath: /etc/ssm/
              subPath: .env
      containers:
        - image: ${BUILD_IMAGE}:${BUILD_TAG}
          volumeMounts:
            - name: envfile
              mountPath: /etc/ssm/
              subPath: .env
          imagePullPolicy: Always
          name: ${PROJECT_NAME}
          ports:
            - containerPort: 80
      volumes:
        - name: envfile
          emptyDir: {}

This sets us up for a Kubernetes deployment, which uses the init container we built previously to populate our environment variables from AWS SSM, securely managing our application’s secrets.

Here’s how it works:

When the Kubernetes Pod starts, the init container is the first to run. It fetches the secret data from AWS SSM and writes it to a file named .env located in the /etc/ssm/ directory.
This directory is a shared volume (emptyDir) that's accessible to all containers in the Pod. The emptyDir volume is created when a Pod is assigned to a Node, and it exists as long as that Pod is running on that Node. The data in emptyDir is safe across container crashes and restarts.
Once the init container successfully writes the .env file, it exits, and the main application container starts.
The main application container reads the .env file from the shared volume, thus having access to the secret data. The entrypoint.sh script in the main container sources the .env file to set the environment variables. Then, it removes the .env file for added security, ensuring the secrets do not persist in the file system once they've been loaded into the application's environment.
The main application then continues to run with the environment variables securely set, all without the secrets having to be explicitly stored or exposed outside the application’s memory.

The following block of code can be added to your application to debug and verify that you are retrieving the secret parameters correctly from the AWS Systems Manager (SSM) Parameter Store.

// Debug ssm fetching 
key1 := os.Getenv("AMQ_USER")
key2 := os.Getenv("AMQ_PASS")

filePath := "/app/ssm-vars"

file, err := os.Create(filePath)
if err != nil {
    log.Fatalf("Failed to create file: %v", err)
}
defer file.Close()

content := fmt.Sprintf("KEY1=%s\nKEY2=%s\n", key1, key2)
err = os.WriteFile(filePath, []byte(content), 0644)
if err != nil {
    log.Fatalf("Failed to write file: %v", err)
}

This piece of code reads the values of two environment variables, AMQ_USER and AMQ_PASS, which have been populated from the SSM Parameter Store. It then writes these values to a file for debugging purposes.

It’s important to understand that in a production environment, writing secrets to a file may expose them to additional security risks. This should only be used for debugging purposes and removed from the production code.

We achieved this secure management of secrets by following these steps:

Storing Secrets in AWS SSM Parameter Store: We stored our secrets securely in the AWS Systems Manager Parameter Store, which is a managed service that provides a secure and scalable way to store and manage configuration data.
Fetch Secrets with an Init Container: We used an init container in our Kubernetes pod, which runs before our main application starts. This init container runs a Go program that fetches the secrets from the AWS SSM Parameter Store and writes them to an environment file.
Populate Environment Variables: In the main container where our application runs, we used an entrypoint script that sources the environment file, thereby populating the environment variables with the secrets.
Removal of .env file: To ensure that our secrets are not written to disk in the main container, the entrypoint script removes the .env file after sourcing it.
Secrets in Memory: As a result of these steps, the secrets are only present in the memory of the running application process and nowhere else. They are not written to disk in the main container, and are not included in any container layers. This is a robust way to keep secrets secure.
Security Best Practices: We also ensured security best practices such as setting appropriate IAM roles for access to AWS SSM Parameter Store, and used Kubernetes RBAC for restricting access to Kubernetes

Farewell 😊

We’ve explored an effective method of Kubernetes secret management using Golang, AWS ParameterStore, OIDC, and Terraform. Thank you for reading this article. I hope it has provided you with valuable insight and can serve as a handy reference for your future projects. Keep exploring, keep learning, and continue refining your security practices!

EKS ZeroScaler

Tarlan Huseynov — Sun, 09 Jun 2024 11:09:49 +0000

Scheduled auto-scaling to zero with Lambda GO client

A seamless integration of Golang Lambda kubernetes client, EventBridge, and EKS, all efficiently deployed with Terraform, for dynamic, cost-effective auto-scaling in development environments.

Introduction
Building and Syncing to S3
Deploying with Helm Chart and Kubernetes Job
Farewell

Introduction

In this insightful piece, we will delve into a cutting-edge solution designed to enhance DevOps efficiency and reduce costs in development environments. Leveraging the power of AWS technologies, we will explore how a Golang-based Lambda function, orchestrated by an EventBridge scheduler, can dynamically scale down EKS Fargate namespace deployments to zero. This strategy proves highly effective not only with Fargate but also when paired with autoscaler/karpenter solutions, offering a versatile approach to cost reduction and resource optimization. Significantly lowering resource utilization during non-peak hours, this method is both cost-effective and efficient. Also worth to highlight that we will touch a streamlined way of deploying this Lambda function using Terraform, showcasing an infrastructure-as-code approach that ensures consistency and ease of management.

The Golang Lambda Function - A Deep Dive

The heart of this solution is a simple Golang-based Lambda function designed to dynamically scale down deployments in specific namespaces of an EKS cluster. Here’s a closer look at how this function operates.

Our Lambda function is crafted in Go, a language known for its efficiency and suitability for concurrent tasks, making it a perfect choice for cloud-native applications.

package main

import (
 "context"
 "encoding/json"
 "os"

 eksauth "github.com/chankh/eksutil/pkg/auth"

 autoscalingv1 "k8s.io/api/autoscaling/v1"
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"

 "github.com/aws/aws-lambda-go/lambda"
 log "github.com/sirupsen/logrus"
)

type Payload struct {
 ClusterName string   `json:"clusterName"`
 Namespaces  []string `json:"namespaces"`
 Replicas    int32    `json:"replicas"`
}

func main() {
 if os.Getenv("ENV") == "DEBUG" {
  log.SetLevel(log.DebugLevel)
 }

 lambda.Start(handler)
}

func handler(ctx context.Context, payload Payload) (string, error) {
 cfg := &eksauth.ClusterConfig{
  ClusterName: payload.ClusterName,
 }

 clientset, err := eksauth.NewAuthClient(cfg)
 if err != nil {
  log.WithError(err).Error("Failed to create EKS client")
  return "", err
 }

 scaled := make(map[string]int32)

 for _, ns := range payload.Namespaces {
  deployments, err := clientset.AppsV1().Deployments(ns).List(ctx, metav1.ListOptions{})
  if err != nil {
   log.WithError(err).Errorf("Failed to list deployments in namespace %s", ns)
   continue
  }

  for _, deploy := range deployments.Items {
   if err := scaleDeploy(clientset, ctx, ns, deploy.Name, payload.Replicas); err == nil {
    scaled[ns+"/"+deploy.Name] = payload.Replicas
   }
  }
 }

 scaledJSON, err := json.Marshal(scaled)
 if err != nil {
  log.WithError(err).Error("Failed to marshal scaled deployments to JSON")
  return "", err
 }

 log.Info("Scaled Deployments: ", string(scaledJSON))
 return "Scaled Deployments: " + string(scaledJSON), nil
}

func scaleDeploy(client *kubernetes.Clientset, ctx context.Context, namespace, name string, replicas int32) error {
 scale := &autoscalingv1.Scale{
  ObjectMeta: metav1.ObjectMeta{
   Name:      name,
   Namespace: namespace,
  },
  Spec: autoscalingv1.ScaleSpec{
   Replicas: replicas,
  },
 }

 _, err := client.AppsV1().Deployments(namespace).UpdateScale(ctx, name, scale, metav1.UpdateOptions{})
 if err != nil {
  log.WithError(err).Errorf("Failed to scale deployment %s in namespace %s", name, namespace)
 } else {
  log.Infof("Successfully scaled deployment %s in namespace %s to %d replicas", name, namespace, replicas)
 }
 return err
}

Key Components

Dependencies and Libraries: The function uses the eksauth package for authenticating with EKS, and the Kubernetes client-go library for interacting with the cluster. AWS Lambda Go SDK is employed for Lambda functionalities, whilelogrus provides robust logging capabilities.
Payload Structure: We define a Payload struct to encapsulate the input, including the target cluster’s name, the namespaces to scale, and the desired number of replicas.
The Main Function: It initializes the log level based on an environment variable and launches the Lambda handler, demonstrating a simple yet effective setup.
Lambda Handler: This function takes the Payload, authenticates with the EKS cluster, and iterates over namespaces to scale down deployments. It’s a demonstration of how straightforward interactions with Kubernetes can be orchestrated in a serverless environment.
Scaling Logic with scaleDeploy function: Here, we actually adjust the scale of each deployment. This function showcases the practical application of the Kubernetes API in managing cluster resources.

For seamless integration and management, we’ll be placing our Lambda function within a specific directory structure in our Terraform project. This approach not only keeps our project organized but also simplifies the deployment process.

We’ll store the Lambda function in files/lambdas/pod_zeroscaler (some fancy naming won’t hurt 😉) . This dedicated directory within our Terraform project ensures that our Lambda code is logically separated and easily accessible in a location where we can edit our code, build it and deploy it as a lambda function by applying terraform continuously ( we will look into how-to do that later ). Let’s initially set up the lambda function’s code basis.

Initialize the Go Module

Navigate to the pod_zeroscaler directory and run go mod init pod_zeroscaler followed by go mod tidy. This commands set up the necessary Go module files, helping manage dependencies and versioning.

Build the Lambda Function

After initializing the module, we compile our Lambda function using GOOS=linux go build -o main

Note: JetBrains Goland IDE - For managing projects that involve both Go code and Terraform modules, I highly recommend using JetBrains Goland IDE. Its integrated environment simplifies handling multiple aspects of the project, from coding to deployment. The IDE’s robust features enhance coding efficiency, debugging, and version control management, making it an invaluable tool for any DevOps engineer working in a cloud-native ecosystem. You can get the mentioned Terraform plugin here.

Deploying and Orchestrating the EKS Auto-Scaling Solution with Terraform

In this section, we delve into the infrastructure setup for our auto-scaling solution, using Terraform to deploy and orchestrate the components. This setup is a crucial part of our strategy to manage resources efficiently in a cloud-native DevOps environment.

Pre-requisites and Setup

Before we dive into the specifics, let’s establish our foundational pre-requisites:

Existing Terraform Project: Our setup assumes that there’s an existing Terraform project with either a remote or local state setup.
Pre-configured EKS Cluster: An EKS cluster is already in place, and our solution is designed to integrate with it seamlessly.
Modular Code Format: The provided Terraform code is in a scratch format, ready to be modularized as per the project’s requirements.

 locals {

  cluster_name = "my-eks-cluster"

  pod_zeroscaler = {
    enabled            = true
    scale_out_schedule = "cron(00 09 ? * MON-FRI *)" # At 09:00:00 (UTC), Monday through Friday
    scale_in_schedule  = "cron(00 18 ? * MON-FRI *)" # At 18:00:00 (UTC), Monday through Friday
    namespaces         = ["default", "development"]
  }

  default_tags = {
    Environment       = var.env
    Owner             = "Devops"
    Managed-by        = "Terraform"
  }  
 } 

   data "aws_iam_policy_document" "lambda_logging_policy" {
    statement {
      actions = [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ]
      effect    = "Allow"
      resources = ["*"]
    }
  }

  resource "aws_iam_policy" "lambda_logging_policy" {
    name   = "${local.cluster_name}-lambda-logging"
    policy = data.aws_iam_policy_document.lambda_logging_policy.json
    tags   = local.default_tags
  }

  data "aws_iam_policy_document" "describe_cluster_lambda_policy" {
    statement {
      actions   = ["eks:DescribeCluster"]
      resources = [aws_eks_cluster.eks_cluster.arn]
    }
  }

  resource "aws_iam_policy" "describe_cluster_lambda_policy" {
    name   = "describe-cluster-policy"
    policy = data.aws_iam_policy_document.describe_cluster_lambda_policy.json
    tags   = local.default_tags
  }

  resource "aws_iam_role" "eks_lambda_role" {
    name = "${local.cluster_name}-lambda-role"
    tags = local.default_tags
    assume_role_policy = jsonencode({
      Version = "2012-10-17"
      Statement = [
        {
          Action = "sts:AssumeRole"
          Effect = "Allow"
          Principal = {
            Service = ["lambda.amazonaws.com"]
          }
        },
      ]
    })
  }

  resource "aws_iam_role_policy_attachment" "describe_cluster_policy" {
    role       = aws_iam_role.eks_lambda_role.name
    policy_arn = aws_iam_policy.describe_cluster_lambda_policy.arn
  }

  resource "aws_iam_role_policy_attachment" "lambda_logging_policy" {
    count      = local.pod_zeroscaler["enabled"] ? 1 : 0
    role       = aws_iam_role.eks_lambda_role.name
    policy_arn = aws_iam_policy.lambda_logging_policy.arn
  }


  data "archive_file" "pod_zeroscaler_lambda_zip" {
    count       = local.pod_zeroscaler["enabled"] ? 1 : 0
    type        = "zip"
    source_file = "${path.root}/files/lambdas/pod_zeroscaler/main"
    output_path = "/tmp/main.zip"
  }

  resource "aws_lambda_function" "pod_zeroscaler_lambda" {
    count            = local.pod_zeroscaler["enabled"] ? 1 : 0
    filename         = data.archive_file.pod_zeroscaler_lambda_zip[0].output_path
    source_code_hash = data.archive_file.pod_zeroscaler_lambda_zip[0].output_base64sha256
    function_name    = "${local.cluster_name}-zeroscaler"
    handler          = "main"
    runtime          = "go1.x"
    role             = aws_iam_role.eks_lambda_role.name
    environment {
      variables = { CLUSTER_NAME = local.cluster_name }
    }
  }

  // AWS EventBridge Scheduler
  data "aws_iam_policy_document" "pod_zeroscaler_invoke_policy" {
    count = local.pod_zeroscaler["enabled"] ? 1 : 0
    statement {
      effect    = "Allow"
      actions   = ["lambda:InvokeFunction"]
      resources = [aws_lambda_function.pod_zeroscaler_lambda[0].arn]
    }
  }

  resource "aws_iam_policy" "pod_zeroscaler_invoke_policy" {
    name        = "${local.cluster_name}-zeroscaler-scheduler-policy"
    count       = local.pod_zeroscaler["enabled"] ? 1 : 0
    description = "Allow function execution for scheduler role"
    policy      = data.aws_iam_policy_document.pod_zeroscaler_invoke_policy[0].json
  }

  data "aws_iam_policy_document" "pod_zeroscaler_invoke_role" {
    count = local.pod_zeroscaler["enabled"] ? 1 : 0
    statement {
      actions = ["sts:AssumeRole"]
      principals {
        type        = "Service"
        identifiers = ["scheduler.amazonaws.com"]
      }
    }
  }

  resource "aws_iam_role" "pod_zeroscaler_invoke_role" {
    count              = local.pod_zeroscaler["enabled"] ? 1 : 0
    name               = "${local.cluster_name}-zeroscaler-scheduler-role"
    assume_role_policy = data.aws_iam_policy_document.pod_zeroscaler_invoke_role[0].json
  }

  resource "aws_iam_role_policy_attachment" "pod_zeroscaler_invoke_role" {
    count      = local.pod_zeroscaler["enabled"] ? 1 : 0
    policy_arn = aws_iam_policy.pod_zeroscaler_invoke_policy[0].arn
    role       = aws_iam_role.pod_zeroscaler_invoke_role[0].name
  }

  resource "aws_scheduler_schedule" "pod_zeroscaler_scale_in" {
    count      = local.pod_zeroscaler["enabled"] ? 1 : 0
    name       = "${local.cluster_name}-zeroscaler-scale-in"
    group_name = "default"

    flexible_time_window {
      mode = "OFF"
    }

    schedule_expression = local.pod_zeroscaler["scale_in_schedule"] #"cron(0 8 * * ? *)"

    target {
      arn      = aws_lambda_function.pod_zeroscaler_lambda[0].arn
      role_arn = aws_iam_role.pod_zeroscaler_invoke_role[0].arn
      input = jsonencode({
        "clusterName" = local.cluster_name
        "namespaces"  = local.pod_zeroscaler["namespaces"]
        "replicas"    = 0
      })
    }
  }

  resource "aws_scheduler_schedule" "pod_zeroscaler_scale_out" {
    count      = local.pod_zeroscaler["enabled"] ? 1 : 0
    name       = "${local.cluster_name}-zeroscaler-scale-out"
    group_name = "default"

    flexible_time_window {
      mode = "OFF"
    }

    schedule_expression = local.pod_zeroscaler["scale_out_schedule"]

    target {
      arn      = aws_lambda_function.pod_zeroscaler_lambda[0].arn
      role_arn = aws_iam_role.pod_zeroscaler_invoke_role[0].arn
      input = jsonencode({
        "clusterName" = local.cluster_name
        "namespaces"  = local.pod_zeroscaler["namespaces"]
        "replicas"    = 1
      })
    }
  }

Terraform Infrastructure Overview

Local Variables and Configuration
Cluster Name and Lambda Settings: We define key local variables, including the name of the EKS cluster and configurations for the Lambda function, such as its schedule and target namespaces.
Default Tags: These tags are applied to all created resources for easy tracking and management.
IAM Policies and Roles
Lambda Logging and EKS Access: Policies are set up to allow the Lambda function to log activities and interact with the EKS cluster ( we will configureaws-auth configmap in a separate section to provide a complete granting necessary access to our lambda ).
Policy Attachments: We attach these policies to the Lambda function’s IAM role, ensuring it has the necessary permissions.
Lambda Function Deployment
Function Packaging: The Lambda function, developed in Go and placed in files/lambdas/pod_zeroscaler, is packaged into a zip file.
Deployment: This package is then deployed as a Lambda function in AWS, complete with environment variables and IAM role associations.
EventBridge Scheduler
Invoke Policy and Role: We create an IAM policy and role that allows EventBridge to invoke our Lambda function.
Scheduling Resources: Two EventBridge schedules are set up — one for scaling down (scaling in) and the other for scaling up (scaling out) — each targeting our Lambda function with specific payloads.

This Terraform setup not only deploys our Lambda function but also orchestrates its operation, aligning with our goal of automating and optimizing resource utilization in a Kubernetes environment. By utilizing Terraform, we ensure a reproducible and efficient deployment process, encapsulating complex cloud operations in a manageable codebase.

Access Configuration for Lambda in EKS

The final step in our setup involves configuring the AWS EKS cluster to grant the necessary permissions to our Lambda function. This is done by editing the aws-auth ConfigMap in the kube-system namespace and setting up appropriate Kubernetes roles and bindings using Terraform. This section ensures our Lambda function can interact effectively with the EKS cluster by using the lambda role attached to it.

Reference: Enabling IAM principal access to your cluster

We need to update the aws-auth ConfigMap in Kubernetes. This ConfigMap is crucial for controlling access to your EKS cluster. Here’s a sample snippet where the lambda role arn is added to a specific RBAC group named lambda-group ( we will create this group and role binding ).

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::<ACCOUNT_ID>:role/my-eks-cluster-development-lambda-role
      groups:
        - lambda-group
    - groups:
      - system:bootstrappers
      - system:nodes
      - system:node-proxier
      rolearn: arn:aws:iam::<ACCOUNT_ID>:role/my-eks-cluster-development-fargate-pod-execution-role
      username: system:node:{{SessionName}}
    - groups:yam
        - system:bootstrappers
        - system:nodes
      rolearn: arn:aws:iam::<ACCOUNT_ID>:role/my-eks-cluster-development-node-pod-execution-role
      username: system:node:{{EC2PrivateDNSName}}

In this ConfigMap, we add a new role entry for our Lambda function’s IAM role. This entry assigns the Lambda role to a group, lambda-group, which we will reference in our Kubernetes role and binding.

Terraform codebase

# Cluster Role and Cluster Role Binding for Lambda
resource "kubernetes_cluster_role" "lambda_cluster_role" {
  count      = local.pod_zeroscaler["enabled"] ? 1 : 0
  metadata {
    name = "lambda-clusterrole"
  }
  rule {
    api_groups = ["*"]
    resources  = ["deployments", "deployments/scale"]
    verbs      = ["get", "list", "watch", "create", "update", "patch", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["pods"]
    verbs      = ["get", "list", "watch", "create", "update"]
  }
}

resource "kubernetes_cluster_role_binding" "lambda_cluster_role_binding" {
  count  = local.pod_zeroscaler["enabled"] ? 1 : 0
  metadata {
    name = "lambda-clusterrolebinding"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.lambda_cluster[0]_role.metadata[0].name
  }
  subject {
    kind      = "Group"
    name      = "lambda-group"
    api_group = "rbac.authorization.k8s.io"
  }
}

We define a Kubernetes Cluster Role and a Cluster Role Binding in Terraform to grant our Lambda function the necessary permissions within the cluster.

The kubernetes_cluster_role resource defines the permissions our Lambda function needs. We grant it a wide range of verbs on deployments and pods, ensuring it can perform its scaling actions.

The kubernetes_cluster_role_binding resource binds the above role to the lambda-group. This binding ensures that any IAM role in the lambda-group (which includes our Lambda function’s IAM role) gets the permissions defined in the cluster role.

Automating ConfigMap Update with Terraform and Bash script

To streamline the process of updating the aws-auth ConfigMap in Kubernetes, we can automate this step using Terraform coupled with a local provisioner and a Bash script. This automation not only saves time but also reduces the potential for manual errors.

We’ll use a Bash script, aws_auth_patch.sh, located in files/scripts, to modify the aws-auth ConfigMap. This script checks if the necessary permissions are already in place and, if not, adds them.

#!/bin/bash
set -euo pipefail

# Script to update the aws-auth ConfigMap

AWS_AUTH=$(kubectl get -n kube-system configmap/aws-auth -o yaml)

if echo "${AWS_AUTH}" | grep -q "${GROUP}"; then
  echo "Permission is already added for ${GROUP}"
else
  NEW_ROLE="    - rolearn: ${ROLE_ARN}\n      groups:\n        - ${GROUP}"
  PATCH=$(kubectl get -n kube-system configmap/aws-auth -o yaml | awk "/mapRoles: \|/{printf \"%s\n%s\n\", \$0, \"${NEW_ROLE}\";next}1")
  kubectl patch configmap/aws-auth -n kube-system --patch "${PATCH}"
fi

This script retrieves the current aws-auth ConfigMap, checks if the required role is already added, and if not, it patches the ConfigMap with the new role.

In our Terraform setup, we’ll use a null_resource with a local-exec provisioner to execute this script automatically.

resource "null_resource" "patch_aws_auth" {
  count = local.pod_zeroscaler["enabled"] ? 1 : 0

  triggers = {
    api_endpoint_up      = aws_eks_cluster.eks_cluster.endpoint
    lambda_bootstrapped = aws_iam_role.eks_lambda_role.id
    script_hash          = filemd5("${path.cwd}/files/scripts/aws_auth_patch.sh")
  }

  provisioner "local-exec" {
    working_dir = "${path.cwd}/files/scripts"
    command     = "./aws_auth_patch.sh"
    interpreter = ["bash"]
    environment = {
      ROLE_ARN = aws_iam_role.eks_lambda_role.arn
      GROUP    = kubernetes_cluster_role_binding.lambda_cluster_role_binding.subject[0].name
    }
  }
}

This Terraform resource triggers the script execution whenever there are changes to the EKS cluster endpoint, the Lambda IAM role, or the script itself. The script is executed in the appropriate directory, and necessary environment variables are passed to it.

Alternatively we could use eksctl + local provisioner instead of bash script + kubectl.

resource "null_resource" "patch_aws_auth_lambda_role" {
  depends_on = [
    aws_eks_fargate_profile.eks_cluster_fargate_kubesystem,
    data.utils_aws_eks_update_kubeconfig.bootstrap_kubeconfig
  ]

  provisioner "local-exec" {
    working_dir = "../files/scripts"
    command     = <<-EOT
      eksctl get iamidentitymapping \
        --cluster ${local.cluster_name} \
        --region ${local.region} \
        --profile ${var.kubeconfig_profile} | grep -q "${aws_iam_role.lambda_role.arn}" || \
        eksctl create iamidentitymapping \
          --cluster ${local.cluster_name} \
          --region ${local.region} \
          --arn ${aws_iam_role.lambda_role.arn} \
          --group ${kubernetes_cluster_role_binding.lambda_cluster_role_binding.subject[0].name} \
          --username system:node:{{EC2PrivateDNSName}} \
          --profile ${var.kubeconfig_profile}
    EOT
  }
}

Farewell 😊

Thank you for taking the time to read this article. I trust it has offered you valuable insights and practical knowledge to implement in your own DevOps endeavors. As we continue to navigate the ever-evolving landscape of cloud infrastructure and serverless architectures, remember the power of automation and efficient resource management. Keep pushing the boundaries, keep learning, and always strive for excellence!

GitOps for CloudFront and S3

Tarlan Huseynov — Sun, 09 Jun 2024 09:51:24 +0000

Front-end deployments in GitOps style

This strategy uses ArgoCD, EKS, and AWS services like CloudFront and S3 to streamline deployments, improve performance, and maintain best practices.

Introduction
Building and Syncing to S3
Deploying with Helm Chart and Kubernetes Job
Conclusion

In the world of modern web development, deploying front-end applications efficiently and reliably is a key challenge. As teams adopt GitOps strategies to streamline and automate deployments, certain complexities arise, particularly when integrating with AWS services like CloudFront and S3.

So let’s consider that ideally all our workloads are containerized and all run on Kubernetes (EKS) platform, we have all security checks, automations, tests and pipelines in place, and already have leveraged ArgoCD and supplementary tools for deployments.

Now one common dilemma is deciding how to manage front-end deployments consistently in GitOps style while maintaining the benefits of using CloudFront for caching and performance optimization. Some teams consider moving front-end assets to containers for consistency, but this can introduce unnecessary complexity and deviate from best practices.

When employing a centralized GitOps strategy, it’s crucial to keep the deployment process consistent and manageable. However, front-end applications often require specific considerations:

Caching and Performance: CloudFront provides a robust solution for caching and delivering static assets, ensuring high performance and low latency.
Artifact Management: Synchronizing build artifacts to the correct S3 paths while managing different versions can be challenging.
Deployment Automation: Automating the deployment process while ensuring the correct paths and versions are updated in CloudFront.
Consistency and Reproducibility: Maintaining consistent and reproducible deployments across environments.
Easy and rapid Rollbacks — if possible ofc. 😌

Introduction

In this article, I will share a solution I implemented to address these challenges. This approach leverages ArgoCD, EKS, AWS CloudFront, and S3, integrating them seamlessly into a GitOps workflow. By using a Kubernetes job with AWS CLI, we can manage CloudFront paths dynamically, ensuring our front-end application is always up-to-date and efficiently delivered.

Release branch is merged to main
New release tag created from main
GHA is triggered on release to test and build the code
Generated artifacts are tagged and synced to s3 with respective path
Developer creates a pull request to pass the new version to GitOps repo
PR is merged and values file is updated with new version
ArgoCD picks up the changes after being triggered via Webhook or by polling
Values diff triggered a new job creation by ArgoCD
Kubernetes Job sends api call to CF to swap the origin path based on the new version

Building and Syncing to S3

To deploy our front-end application, we use GitHub Actions to handle the build and deployment process. The workflow triggers on new version tags, checks out the repository, sets up build environment, and configures AWS credentials. It retrieves secrets, installs dependencies, runs tests, builds the application, and syncs the output to an S3 bucket. We can of-course have multiple parallel workflows for each environment and sync to different s3 buckets with dedicated path that reflects release tag, or even sync to the same s3 bucket with dedicated path that reflects target environment + release tag ( I like the 1st option better for total segregation ).

For instance, if we’re deploying a version tagged v1.0.0 to the production environment, the path in S3 would be s3://frontend-production-artifacts/production/v1.0.0.

name: Publish Release Artifact Version
run-name: "Production Release ${{ github.ref_name }} | Publishing Artifact Version | triggered by @${{ github.actor }}"

on:
  push:
    tags:
      - 'v[0-9]+.[0-9]+.[0-9]+'

env:
  ARTIFACTS_BUCKET: frontend-production-artifacts
  AWS_REGION: us-east-2
  ENVIRONMENT: production

jobs:
  build-and-publish:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key:  ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Get Secrets
        uses: aws-actions/aws-secretsmanager-get-secrets@v2
        with:
          secret-ids: frontend-secrets-${{ env.ENVIRONMENT }}
          parse-json-secrets: true

      - name: Install dependencies
        run: yarn install

      - name: Run tests
        run: yarn test

      - name: Build
        run: yarn build

      - name: Sync files to Artifacts bucket
        run: aws s3 sync build/ s3://${{ env.ARTIFACTS_BUCKET }}/${{ env.ENVIRONMENT }}/${{ github.ref_name }} --delete

Deploying with Helm Chart and Kubernetes Job

To automate the deployment process further, we can use a Helm chart that defines a Kubernetes job. This job handles updating the CloudFront origin path for the new version of our application using a Docker image with AWS CLI installed.

We have a values file that provides parameters like the application name, version, Docker image, S3 bucket name, and AWS region:

global:
  region: "us-east-2"

app:
  name: "frontend-app"
  version: v1.0.10
  backOffLimit: "4"
  jobImage: "amazon/aws-cli:2.16.1"
  originS3: "frontend-production-artifacts"

The Kubernetes job uses these values to dynamically set its configuration. It includes the job name, the container image, and environment variables for the S3 bucket, origin path, and AWS region.

When the job runs, it installs jq for JSON processing, retrieves the CloudFront distribution ID based on the S3 bucket name, fetches the current CloudFront configuration, updates the origin path to the new version, and invalidates the CloudFront cache to ensure the latest version is served to users. Of course you can always build your own lightweight docker image with all dependencies already installed (aws-cli and jq), or even build your own solution by leveraging AWS SDK directly.

apiVersion: batch/v1
kind: Job
metadata:
  name: swap-cf-origin-path-{{ .Values.app.name }}-{{ .Values.app.version }}
spec:
  template:
    spec:
      serviceAccountName: {{ .Values.app.name }}
      containers:
        - name: aws-cli
          image: {{ .Values.app.jobImage }}
          env:
            - name: S3_BUCKET_NAME
              value: {{ .Values.app.originS3 }}
            - name: ORIGIN_PATH
              value: /{{ .Release.Namespace }}/{{ .Values.app.version }}
            - name: AWS_REGION
              value: {{ .Values.global.region }}
          command: ["/bin/sh","-c"]
          args:
            - |
              set -e
              yum install jq -y

              CF_DIST_ID=$(aws cloudfront list-distributions --query "DistributionList.Items[?contains(Origins.Items[].DomainName, '${S3_BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com')].Id | [0]" --output text)

              OUTPUT=$(aws cloudfront get-distribution-config --id $CF_DIST_ID)
              ETAG=$(echo "$OUTPUT" | jq -r '.ETag')
              DIST_CONFIG=$(echo "$OUTPUT" | jq '.DistributionConfig')

              UPDATED_CONFIG=$(echo "$DIST_CONFIG" | jq --arg path "${ORIGIN_PATH}" '.Origins.Items[0].OriginPath = $path')

              aws cloudfront update-distribution --id $CF_DIST_ID --if-match $ETAG --distribution-config "$UPDATED_CONFIG"

              aws cloudfront create-invalidation --distribution-id $CF_DIST_ID --paths "/*"
      restartPolicy: Never
  backoffLimit: {{ .Values.app.backOffLimit }}

To allow our Kubernetes job to interact with AWS services like CloudFront and S3, we need to grant it the necessary permissions to the job’s service account. We can achieve this by using IAM Roles for Service Accounts (IRSA) or Pod identities. Here’s how you can configure IRSA option using Terraform. This setup allows the Kubernetes job to securely perform actions required for updating the CloudFront origin path and invalidating the cache.

data "aws_iam_policy_document" "service_account_assume_role" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc_provider_sts.url, "https://", "")}:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc_provider_sts.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:${var.namespace}:frontend-app"]
    }

    principals {
      identifiers = [aws_iam_openid_connect_provider.oidc_provider_sts.arn]
      type        = "Federated"
    }
  }
}

resource "aws_iam_role" "service_account_role" {
  assume_role_policy = data.aws_iam_policy_document.service_account_assume_role.json
  name               = "frontend-app-sa-role-${var.namespace}"
  tags               = local.default_tags

  lifecycle {
    create_before_destroy = false
  }
}

resource "aws_iam_policy" "frontend_app_swap_origin_policy" {
  name        = "frontend-app-policy-${var.namespace}"
  path        = "/"
  description = "IAM policy for frontend-app job service account"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = [
          "cloudfront:GetDistribution",
          "cloudfront:UpdateDistribution",
          "cloudfront:CreateInvalidation",
          "s3:ListBucket",
          "s3:GetObject"
        ]
        Effect   = "Allow"
        Resource = "*"
      }
    ]
  })
}

# Attach policies to the service account role
resource "aws_iam_role_policy_attachment" "service_account_role" {
  depends_on = [
    aws_iam_role.service_account_role,
    aws_iam_policy.frontend_app_policy
  ]

  role       = aws_iam_role.service_account_role.name
  policy_arn = aws_iam_policy.frontend_app_swap_origin_policy.arn
}

Conclusion

At this point, all we need to do is push the new version after building and syncing it to S3. The job will handle updating the CloudFront origin path and invalidating the cache, ensuring that users always get the latest version of our front-end application. For an even more cosmetically satisfying approach, we could implement an additional Continuous Deployment (CD) solution on top of ArgoCD, such as OctopusDeploy. However, that is a topic for another day and another discussion 😉 Farewell Folks! 😊

Endpoint Health Monitoring with AWS Services and Terraform

Tarlan Huseynov — Thu, 27 Jul 2023 12:31:56 +0000

Introduction
Implementation via console
IaC and CI/CD

Introduction

In an era where businesses rely on the digital world more than ever, it’s imperative to have robust and reliable systems monitoring our applications’ health. An unexpected outage or system failure can not only affect the user experience but can also lead to revenue loss and damage the company’s reputation. Hence, automated, efficient, and robust health checks and notifications are paramount.

Amazon’s Route53 Health Checks offer an excellent solution to this problem. By sending regular requests to your application endpoints, Route53 is designed to monitor the health of your applications continuously. If the application doesn’t respond within a specified period or the response doesn’t meet certain conditions, Route53 considers the endpoint unhealthy. But identifying the problem is only half the battle; the other half lies in efficient reporting and notification. This is where integrating CloudWatch alerts, SNS, Lambda, and communication tools like Slack and Microsoft Teams become incredibly valuable.

Architecture diagram

By setting up this kind of architecture, we ensure that we are notified the moment our systems detect any issues. We won’t have to wait for a user to report an issue or for a developer to stumble upon it during their work. This kind of proactive monitoring can save a lot of time and resources.

Furthermore, by automating the whole process using a modular Terraform project, we bring in immense scalability and consistency to our setup. As the infrastructure grows, the ability to manage resources as code becomes a significant advantage. It allows us to quickly deploy similar resources, maintain consistency across environments, and more efficiently manage and track infrastructure changes. We will add an extra layer of automation by implementing Terraform pipelines using GitLab CI. This will enable us to plan and deploy changes to our infrastructure automatically, making the maintenance and extension of our systems even more efficient.

In this article, we’ll explore how we can turbocharge our endpoint monitoring by integrating AWS Route53 health checks with AWS CloudWatch, AWS SNS, AWS Lambda as our backbone and use Slack, Microsoft Teams, and email for notifications — with automated deployments and configuration changes via Terraform + GitLab CI.

Step by step implementation via console

To fully appreciate the level of automation we’re aiming for, it’s essential to first walk through the manual steps required in the process. This will provide the context needed to understand the tasks we’re targeting to automate.

Setting up Route53 Health Checks

The first step is to create health checks for the application endpoints using Route53. These health checks monitor the health of the endpoints by sending regular requests and analyzing the responses based on pre-defined conditions.

AWS Developer Guide: Creating and updating health checks

Creating CloudWatch Alarms

For each health check, a corresponding CloudWatch alarm is created. These alarms monitor the health check status metric and trigger when the endpoint’s status changes (i.e., when it becomes unhealthy).

AWS Developer Guide: Monitoring health checks using CloudWatch

Setting up SNS Topic

An Amazon Simple Notification Service (SNS) topic is created for the alarms. When an alarm changes state, it publishes a message to this topic. The topic has subscribers who will receive these messages, like our AWS Lambda function and any email addresses for direct email notifications.

AWS Developer Guide: Creating an Amazon SNS topic

Building AWS Lambda Function

The Lambda function, which subscribes to the SNS topic, is built to handle the incoming alarm messages. This function parses the alarm data and prepares customized messages for Slack and Microsoft Teams.

AWS Developer Guide: Getting started with Lambda

lambda code [runtime Python 3.7]

import json
import os
import requests

def send_slack_message(message):
    slack_webhook_url = os.getenv('SLACK_WEBHOOK_URL')
    headers = {'Content-type': 'application/json'}
    response = requests.post(slack_webhook_url, headers=headers, data=json.dumps({'text': message}))

    if response.status_code != 200:
        raise ValueError(f'Request to Slack returned an error {response.status_code}, the response is:\n{response.text}')

def send_teams_message(message, color):
    teams_webhook_url = os.getenv('TEAMS_WEBHOOK_URL')
    headers = {'Content-type': 'application/json'}
    message = {
        "@type": "MessageCard",
        "@context": "http://schema.org/extensions",
        "themeColor": color,
        "summary": "Health Check Status",
        "sections": [{
            "activityTitle": "Health Check Status",
            "text": message
        }],
        "potentialAction": [{
            "@type": "OpenUri",
            "name": "Details",
            "targets": [{"os": "default", "uri": "https://console.aws.amazon.com/route53/healthchecks/home#"}]
        }]
    }
    response = requests.post(teams_webhook_url, headers=headers, data=json.dumps(message))

    if response.status_code != 200:
        raise ValueError(f'Request to Teams returned an error {response.status_code}, the response is:\n{response.text}')


def handler(event, context):
    alarm_message = json.loads(event['Records'][0]['Sns']['Message'])
    endpoint = alarm_message['AlarmDescription']  # Assumes the endpoint URL is in the AlarmDescription

    if alarm_message['NewStateValue'] == 'ALARM':
        formatted_message_slack = f"*Endpoint:* {endpoint}\n" \
                            f"*State:* :elmofire: endpoint health check failed :warning:\n" \
                            f"<https://console.aws.amazon.com/route53/healthchecks/home#|Details>"
        formatted_message_teams = f"{endpoint} health check failed!"
        teams_color = "FF0000"  # Red
    elif alarm_message['NewStateValue'] == 'OK':
        formatted_message_slack = f"*Endpoint:* {endpoint}\n" \
                            f"*State:* :baby-yoda-soup: endpoint recovered :white_check_mark:\n" \
                            f"<https://console.aws.amazon.com/route53/healthchecks/home#|Details>"
        formatted_message_teams = f"{endpoint} health check is ok!"
        teams_color = "00FF00"  # Green

    slack_webhook_url = os.getenv('SLACK_WEBHOOK_URL')
    teams_webhook_url = os.getenv('TEAMS_WEBHOOK_URL')

    if slack_webhook_url and slack_webhook_url.strip():
        send_slack_message(formatted_message_slack)
    if teams_webhook_url and teams_webhook_url.strip():
        send_teams_message(formatted_message_teams, teams_color)

Our Lambda function acts as a message interceptor and formatter for health checks. Here’s a brief description of its operation:

Environment Variables: Our function utilizes two environment variables: ‘SLACK_WEBHOOK_URL’ and ‘TEAMS_WEBHOOK_URL’. These variables hold the webhook URLs of our Slack and Teams channels, respectively.
Event Handler: The main entry point of our Lambda function is the handler method. This method receives an event object which contains the message from Amazon SNS, informing about the health status of our endpoints.
Message Processing: Our handler method processes the incoming SNS message, parsing it as JSON and extracting relevant information such as the endpoint (stored in AlarmDescription) and the new state of the alarm (NewStateValue).
Notification Formatting: Depending on the state of the alarm, it constructs custom messages for both Slack and Teams. For Slack, it uses markdown syntax and includes relevant emojis. For Teams, it creates a simpler message and determines the color of the Teams card (red for ‘ALARM’ state, green for ‘OK’ state).
Message Sending: If the respective webhook URL environment variable is set and not empty, it dispatches the corresponding formatted message to Slack or Teams using the send_slack_message or send_teams_message methods respectively.
Sending Slack Message: The send_slack_message method sends a HTTP POST request to the Slack webhook URL with the constructed message as payload. If the request fails for any reason, it raises an exception with a detailed error message.
Sending Teams Message: The send_teams_message method also sends a HTTP POST request, but to the Teams webhook URL, with a different payload structure suited for Teams.

Through this Lambda function, we’re able to transform a raw health check alert from AWS into a rich, custom, human-readable notification in our communication channels.

Configuring Slack and Microsoft Teams Webhooks

Webhooks in Slack and Microsoft Teams are set up to receive incoming messages from our AWS Lambda function. Each platform’s webhook URL is saved as an environment variable for the Lambda function to use.

Slack documentation: Sending messages using Incoming Webhooks
MS Teams documentation: Create Incoming Webhooks

Here is the slack channel view after few notifications 😅

IaC automation with Terraform and GitLab CI

Terraform Setup: Now that the individual components are ready, they need to be managed efficiently. Terraform scripts will automate the creation and management of these resources.

Root Module

module "health-checks" {
  source              = "./modules/healthcheck"
  endpoints           = {
    endpoint-1 = {
      fqdn = "huseynov.net"
      port          = 443
      path          = "healthcheck"
      search_string = "status:ok"
    },
    endpoint-2 = {
      fqdn = "google.az"
    }
  }
  notification_emails = ["tarlan@huseynov.net"]
  slack_webhook_url   = "https://<Webhook_url>"
  teams_webhook_url   = "https://<Webhook_url>"
}

This root module brings together all the elements needed to set up and manage health checks for specified endpoints, send alerts and enable notifications.

You provide inputs to the module through variables:

endpoints: This variable holds the list of endpoints that need to be monitored. Each endpoint is specified as a map with fqdn as the key for endpoint URL. If an endpoint includes the search_string key, the module creates a special kind of health check which not only checks the endpoint availability but also whether the response body contains the specified string.
notification_emails: A list of emails that should receive notifications about the health status of endpoints.
slack_webhook_url: The URL for the Slack channel where alerts will be posted.
teams_webhook_url: The URL for the Microsoft Teams channel where alerts will be posted.

Once configured with appropriate inputs, this module takes care of creating Route53 health checks for the listed endpoints, setting up CloudWatch alarms to monitor these health checks, configuring SNS topics for notifications, and connecting the alerts to your Slack and Microsoft Teams channels as well as your email addresses. All of this complexity is hidden behind a simple and elegant interface, making your health checks management a breeze.

HealthCheck Module

locals {
  endpoints_with_str_check   = { for name, parameters in var.endpoints : name => parameters if contains(keys(parameters), "search_string") }
  endpoints_with_https_check = { for name, parameters in var.endpoints : name => parameters if !contains(keys(parameters), "search_string") }
  route53_health_checks      = merge(aws_route53_health_check.https_check, aws_route53_health_check.https_str_check)
  lambda_alerts_enabled      = var.slack_webhook_url != "" || var.teams_webhook_url != ""
}

# Route 53 health-checks
resource "aws_route53_health_check" "https_check" {
  fqdn              = each.value.fqdn
  reference_name    = each.key
  port              = each.value.port
  type              = "HTTPS"
  resource_path     = each.value.path
  failure_threshold = "3"
  request_interval  = "30"

  tags = {
    Name = "${each.key}-hc"
  }
}

resource "aws_route53_health_check" "https_str_check" {
  for_each          = local.endpoints_with_str_check
  fqdn              = each.value.fqdn
  reference_name    = each.key
  port              = each.value.port
  type              = "HTTPS_STR_MATCH"
  search_string     = each.value.search_string
  resource_path     = each.value.path
  failure_threshold = "3"
  request_interval  = "30"

  tags = {
    Name = "${each.key}-hc"
  }
}

# Cloudwatch alarm
resource "aws_cloudwatch_metric_alarm" "endpoint_hc_alarm" {
  for_each            = local.route53_health_checks
  alarm_name          = "${each.value.reference_name}-hc-alarm"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = "1"
  metric_name         = "HealthCheckStatus"
  namespace           = "AWS/Route53"
  period              = "60"
  statistic           = "Minimum"
  threshold           = "1"
  alarm_actions       = [aws_sns_topic.alarm_sns_topic.arn]
  ok_actions          = [aws_sns_topic.alarm_sns_topic.arn]
  alarm_description   = each.value.fqdn # This is passed to lambda as endpoint value
  dimensions = {
    HealthCheckId = each.value.id
  }
}

data "archive_file" "notifications_lambda_zip" {
  type        = "zip"
  source_file = "${path.root}/files/lambdas/healthcheck.py"
  output_path = "/tmp/healthcheck.zip"
}

data "aws_iam_policy_document" "lambda_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "notifications_lambda_role" {
  count              = local.lambda_alerts_enabled ? 1 : 0
  name               = "health-check-notifications-lambda-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role_policy.json
}

resource "aws_lambda_function" "notifications_lambda" {
  count            = local.lambda_alerts_enabled ? 1 : 0
  filename         = data.archive_file.notifications_lambda_zip.output_path
  source_code_hash = data.archive_file.notifications_lambda_zip.output_base64sha256
  function_name    = "health-check-notifications"
  # handler - <python filename>.<handler function name>
  handler = "healthcheck.handler"
  runtime = "python3.7"
  role    = aws_iam_role.notifications_lambda_role[0].arn

  environment {
    variables = {
      TEAMS_WEBHOOK_URL = var.teams_webhook_url
      SLACK_WEBHOOK_URL = var.slack_webhook_url
    }
  }

  lifecycle {
    ignore_changes = [source_code_hash, last_modified]
  }
}

resource "aws_lambda_permission" "lambda_permission" {
  count         = local.lambda_alerts_enabled ? 1 : 0
  statement_id  = "AllowExecutionFromSNS"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.notifications_lambda[0].function_name
  principal     = "sns.amazonaws.com"
  source_arn    = aws_sns_topic.alarm_sns_topic.arn
}

data "aws_iam_policy_document" "lambda_policy" {
  count = local.lambda_alerts_enabled ? 1 : 0
  statement {
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    effect    = "Allow"
    resources = ["*"]
  }
}

resource "aws_iam_policy" "lambda_policy" {
  count  = local.lambda_alerts_enabled ? 1 : 0
  name   = "health-check-notifications-lambda"
  policy = data.aws_iam_policy_document.lambda_policy[0].json
}

resource "aws_iam_role_policy_attachment" "lambda_role_attached_policy" {
  count      = local.lambda_alerts_enabled ? 1 : 0
  role       = aws_iam_role.notifications_lambda_role[0].name
  policy_arn = aws_iam_policy.lambda_policy[0].arn
}


# SNS Topic for healthcheck alerts
resource "aws_sns_topic" "alarm_sns_topic" {
  name                             = "endpoint-health-check-alarm-topic"
  lambda_failure_feedback_role_arn = aws_iam_role.delivery_feedback_role.arn
  lambda_success_feedback_role_arn = aws_iam_role.delivery_feedback_role.arn
}

resource "aws_sns_topic_subscription" "lambda_topic_subscription" {
  count     = local.lambda_alerts_enabled ? 1 : 0
  topic_arn = aws_sns_topic.alarm_sns_topic.arn
  protocol  = "lambda"
  endpoint  = aws_lambda_function.notifications_lambda[0].arn
}

resource "aws_sns_topic_subscription" "email_topic_subscription" {
  for_each  = toset(var.notification_emails)
  topic_arn = aws_sns_topic.alarm_sns_topic.arn
  protocol  = "email"
  endpoint  = each.value
}

resource "aws_sns_topic_subscription" "sms_topic_subscription" {
  for_each  = toset(var.notification_mobile)
  topic_arn = aws_sns_topic.alarm_sns_topic.arn
  protocol  = "sms"
  endpoint  = each.value
}

# Feedback role
data "aws_iam_policy_document" "feedback_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["sns.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "delivery_feedback_role" {
  name               = "SNSFeedbackRole"
  assume_role_policy = data.aws_iam_policy_document.feedback_assume_role_policy.json

  inline_policy {
    name = "SNSFeedbackPolicy"

    policy = jsonencode({
      "Version" : "2012-10-17",
      "Statement" : [
        {
          "Effect" : "Allow",
          "Action" : [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents",
            "logs:PutMetricFilter",
            "logs:PutRetentionPolicy"
          ],
          "Resource" : [
            "*"
          ]
        }
      ]
    })
  }
}

The health-check module is a comprehensive health check solution. It splits the provided endpoints into two categories based on the presence of a search_string. It generates Route53 health checks, with HTTPS type for endpoints without search_string and HTTPS_STR_MATCH for those with search_string.

The module also creates CloudWatch alarms for each endpoint, monitoring health check status and triggering SNS topic notifications when a status change occurs. The SNS topic is linked to Lambda function for alerts on Teams or Slack, and also sends email notifications.

If either Teams or Slack webhook URLs are provided, a Lambda function is deployed. It has the necessary IAM role and permissions for execution and logging. The module also handles SNS topic subscriptions for both email and Lambda function.

Finally, the module provides an IAM role for SNS to handle delivery feedback.

The entire module is neatly encapsulated and parameterized, ready to be utilized in any Terraform project.

Integrating with GitLab CI: The final step is to automate the Terraform workflows using GitLab CI. GitLab pipelines are created for planning and deploying changes to the infrastructure. With each git push, a pipeline is triggered that runs Terraform commands to apply the changes to the infrastructure.

Requirement: Gitlab managed terraform state
_
gitlab-ci.yml_

#################################################################################################################
variables:
  TF_ROOT: ${CI_PROJECT_DIR}
  TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/tfstate
  ################ AWS Credentials ####################
  TF_VAR_region: $AWS_REGION_DEV
  TF_VAR_access_key: $AWS_ACCESS_KEY_ID
  TF_VAR_secret_key: $AWS_SECRET_ACCESS_KEY
##################################################################################################################
before_script:
  - cd ${TF_ROOT}
  - echo "SYSTEM INFO:"
  - cat /etc/os-release
  - cat /etc/hostname
##################################################################################################################
stages:
  - init
  - validate
  - plan
  - apply
##################################################################################################################

##################################################################################################################
default:

  image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest

  cache:
    key: ${CI_PROJECT_ID}-terraform-cache

    paths:
      - ${TF_ROOT}/.terraform
##################################################################################################################

##################################################################################################################
INITIALIZE:

  stage: init

  script:
    - gitlab-terraform init

  only:
    - merge_requests
    - master
##################################################################################################################

##################################################################################################################
VALIDATE:

  stage: validate

  script:
    - gitlab-terraform validate

  only:
    - merge_requests
    - master  
##################################################################################################################

##################################################################################################################
PLAN:

  stage: plan

  script:
    - gitlab-terraform plan
    - gitlab-terraform plan-json

  artifacts:
    name: plan
    paths:
      - ${TF_ROOT}/plan.cache
    reports:
      terraform: ${TF_ROOT}/plan.json

  only:
    - merge_requests
    - master      
##################################################################################################################

##################################################################################################################
APPLY:

  stage: apply

  environment:
    name: production

  script:
    - gitlab-terraform apply

  dependencies:
    - PLAN

  when: manual

  only:
    - master
##################################################################################################################

This GitLab CI/CD pipeline is configured to utilize a branch-based workflow with merge requests. It consists of four stages: init, validate, plan, and apply, that get triggered based on the branch and the action performed.

Here is the breakdown:

INITIALIZE stage: This is the first stage that runs gitlab-terraform init. It initializes the Terraform working directory, downloading necessary providers and modules, and sets up the backend. It’s executed for both merge requests and changes to the master branch.
VALIDATE stage: This stage runs gitlab-terraform validate which validates the Terraform configuration files for syntax and internal consistency. This also runs for both merge requests and changes to the master branch.
PLAN stage: This stage generates an execution plan with gitlab-terraform plan and outputs it as JSON with gitlab-terraform plan-json. The plan is saved as an artifact for inspection. It is triggered for merge requests and changes to the master branch.
APPLY stage: This stage is triggered manually and only on the master branch. It runs gitlab-terraform apply to apply the proposed changes from the execution plan. It depends on the PLAN stage.

Note that this pipeline configuration relies on GitLab’s managed Terraform state. The state is stored on GitLab’s infrastructure and is managed by GitLab, providing a secure and reliable backend for Terraform. This is specified by the TF_ADDRESS variable in the configuration, which points to GitLab’s API endpoint for managing Terraform state.

In this comprehensive article, we delved into the depths of infrastructure as code (IaC) through Terraform, focusing on deploying and managing health checks for a system architecture, aided by GitLab’s CI/CD pipeline. By utilizing AWS services like Route53, CloudWatch, SNS, and Lambda, we constructed a robust system capable of monitoring server status, carrying out HTTP string matching health checks, and notifying stakeholders via multiple channels when incidents occur.

Through this exploration, we hope you’ve gained valuable insights into how you can leverage these tools to enhance your systems’ reliability and maintainability. As always, keep exploring, learning, and innovating!

Back to Table of Contents

AWS Journey: Standalone Server to Scalable Golang App with CI/CD

Tarlan Huseynov — Sun, 16 Jul 2023 11:35:18 +0000

An Evolutionary Path from Standalone Server to Scalable

Architecture and implementation of AWS Developer Tools for continuous integration & continuous delivery

Introduction
References
GitHub Repositories
Part 1: Standalone Application
Part 2: Migration to RDS, User Pool Creation, Secrets Management and Logging
Part 3: Transitioning to a Scalable and Secure Infrastructure
Part 4: Implementing Continuous Integration/Continuous Deployment (CI/CD)

Introduction

In today's rapidly evolving technological landscape, designing scalable and resilient software applications is essential. The journey of scaling an application's architecture involves various stages, from single-tier to multi-tier, with each tier bringing a unique set of complexities and advantages. In this blog, we're going to delve into the process of transitioning a simple Golang application from a single-tier architecture to a more complex, but efficient, three-tier design.

Moreover, to ensure the stability and reliability of the application, a robust Continuous Integration and Continuous Delivery (CI/CD) pipeline becomes paramount. We'll also discuss the implementation of an effective CI/CD system for our upgraded Golang application.

Join me in this exciting expedition, as we navigate the intricacies of application scaling and pipeline development, providing you with practical insights and methodologies for your own software engineering journeys.

As we undertake this transformation journey for our demo Golang application, we're not working alone. We'll be leveraging a host of powerful tools and services from Amazon Web Services (AWS).

For our user management, we'll utilize AWS Cognito, which will act as our user pool, ensuring we can effectively manage, authenticate, and authorize our users. Handling secrets securely is a fundamental part of any application, for which we'll employ the AWS Systems Manager Parameter Store.

Our data persistence layer will be anchored on AWS Relational Database Service (RDS) using the robust Postgres engine, providing us with scalable and reliable database services. To handle our application deployment, we'll implement AWS CodePipeline, CodeBuild, and CodeDeploy, forming our Continuous Integration and Continuous Delivery (CI/CD) pipeline. These services will streamline our development process by automating build, test, and deployment phases.

Monitoring and logging are crucial aspects of maintaining the health of our application. Here, AWS CloudWatch steps in, offering us comprehensive visibility over our application and resources.

Our application will be hosted on EC2 instances, providing secure and resizable compute capacity in the cloud. To ensure our application can handle varying loads efficiently, we'll incorporate AWS Auto Scaling groups. This service will dynamically adjust our EC2 instance count based on real-time demands.

Finally, to secure and route end-user requests to our application, we'll leverage AWS Certificate Manager (ACM) and Route 53. ACM will take care of the SSL/TLS certificate needs for our application, while Route 53, AWS's scalable Domain Name System (DNS) service, will handle the routing of end-user requests to our application hosted on EC2 instances.

In essence, we are going to fully immerse our application in a cloud-native environment, harnessing the power of AWS services to ensure scalability, reliability, and operational efficiency. Follow along as we detail this evolutionary journey of our application, providing valuable insights into the intricacies of multi-tier architecture and the practical application of AWS services.

References:

For further reading and deeper insights, feel free to check out the following resources:

Amazon Web Services Documentation:
Instance Metadata and User Data: Provides a comprehensive guide on how to retrieve instance metadata.
AWS Database Migration Service: Explains how to manage PostgreSQL databases using AWS RDS.
Connecting to an Amazon RDS or Amazon EC2 Database Instance Remotely: A blog post that guides you through securely connecting to a remote database instance.
Installing the CloudWatch Agent Using AWS Systems Manager: Guides you through the installation process of the CloudWatch agent using AWS Systems Manager.

GitHub Documentation:
Commit Status API: An API reference on how to check and set commit statuses.

PostgreSQL Documentation:
pg_dumpall: Documentation on how to use the pg_dumpall utility.

GitHub Repositories:

In the course of our transformation journey, we will be referencing two key GitHub repositories:

AWS Go Demo: This repository plays a central role in our demonstration, featuring an AWS Golang application as a sample. Different branches of this repository will serve as references during various stages of our architectural evolution, providing you with practical examples and insights.

Commit Status Lambda: As we reach the final stages of our transformation, the implementation of this Lambda function becomes essential. This repository helps us set commit statuses on GitHub, a crucial component in our CI/CD pipeline that enhances transparency and facilitates better tracking of code changes.

These repositories will serve as a roadmap guiding us through our exploration. Their practical implementations offer invaluable insights and further deepen your understanding of the architectural transition and CI/CD pipeline establishment.

Happy learning!

Part 1: Standalone application

Our transformation journey begins from a simple standalone server architecture. A standalone server, essentially a single-tier architecture, is the most basic form of a computing system. Here, the user interface, application logic, and data storage all reside in the same environment.

Architecture diagram

Step-by-step Implementation

In the first part of our application's transformation journey, we'll lay down the foundation of our infrastructure and deploy the initial version of our Golang application on an AWS EC2 instance. Let's go through the steps:

Create and configure a Virtual Private Cloud (VPC) in your AWS environment.
Enable DNS hostnames for your VPC. This is essential for our instances to communicate with each other using DNS hostnames.
Create an Internet Gateway (IGW) and attach it to the VPC. This gateway allows our VPC to communicate with the internet.
Set up Route Tables for your VPC to control the routing of traffic.
Configure public and private subnets within your VPC.
Create a Security Group for your EC2 instance. This will serve as a virtual firewall to control inbound and outbound traffic.
Generate or import a key-pair. This is crucial for secure SSH connections to your instance.
Launch an EC2 instance using the Ubuntu 20.04 LTS AMI and attach an Elastic IP (EIP) to it.
Connect to the instance via SSH over EIP and create a directory for your application:

  $ sudo mkdir /etc/demo-app
  $ sudo -i
  $ echo APPDIR=/etc/demo-app >> /etc/environment
  $ exit

Database & Application

Install Postgres stable version on the instance and configure it:

$ sudo apt update && sudo apt install -y postgresql postgresql-contrib
$ sudo systemctl start postgresql.service
$ sudo -u postgres createuser demouser
$ sudo -u postgres createdb awsgodemo
$ sudo -u postgres psql
postgres=# alter user demouser with encrypted password 'demopass';
postgres=# grant all privileges on database awsgodemo to demouser;
postgres=# \l
postgres=# \q

Check out the demo-1 branch from the AWS Go Demo repository. Build the application and deploy the executable main (make sure it's executable) to the APPDIR you created.

Run and test the application:

$ sudo chmod +x main
$ ./main

Configure Route53 records for your domain and add user entries via the “/log” route of your application.

By following these steps, you'll have successfully deployed the initial version of your Golang application within a robust, secure AWS environment, setting the stage for further enhancements and the upcoming transition to a three-tier architecture.

AWS Service Documentation references for Part 1.

What is EC2 ?
What is VPC ?
What is Route53 ?

Part 2: Migration to RDS, User Pool Creation, Secrets Management and logging

In the second part of our journey, we shift gears towards a more resilient and scalable configuration by migrating our standalone database to AWS RDS. Additionally, we'll enhance our user management and security by creating a user pool via AWS Cognito and transferring our secrets to the AWS Systems Manager (SSM) Parameter Store. Let's delve into these enhancements:

Migrating Database to AWS RDS: As a managed service, AWS RDS offers scalability, reliability, and ease of management, which is why we'll migrate our standalone Postgres database to RDS. The process involves creating an RDS instance, exporting data from our standalone database, and importing it into the RDS instance.
User Pool Creation with AWS Cognito: User management is a key aspect of any application. AWS Cognito provides an easy-to-use user identity and data synchronization service that helps us manage user pools. We'll create a user pool for our application, allowing us to handle user registration, authentication, and account recovery seamlessly.
Secrets Management with SSM Parameter Store: For improved security and centralized management, we'll transfer our application secrets (such as database credentials) to the AWS SSM Parameter Store. It provides secure, hierarchical storage for configuration data management and secrets management.
Deploying Application as a Linux Service: To ensure our application is always up and running, we'll deploy it as a Linux service. This allows the application to start automatically upon system boot, restart after a crash, and benefit from various other systemd features.
Logging with AWS CloudWatch: For efficient monitoring and debugging, we'll set up our application to write logs to AWS CloudWatch. This service collects and tracks metrics, collects and monitors log files, sets alarms, and automatically reacts to changes in AWS resources.

Stay tuned for the detailed steps of this part, as we further enhance the robustness and security of our application by harnessing the power of AWS services. By the end of this part, we'll be one step closer to a full-fledged three-tier architecture, providing a scalable and reliable platform for our evolving Golang application.

Architecture diagram

Step-by-step Implementation

Now that we have outlined our strategy, let's delve into the step-by-step implementation of each aspect.

Migrate Database to RDS:

Start by creating a subnet group, security group, and a Multi-AZ RDS Database with Postgres Engine v 12.14.
Utilize the PostgreSQL native tools pg_dump and pg_restore to migrate the database:

 # Backup your database
 $ pg_dump -Fc -b -v -f awsgodemo.sql -d awsgodemo
 # Connect to your RDS instance and set up your database and user
 $ psql -h <rds endpoint> -U postgres -d postgres
 postgres=> create database awsgodemo;
 postgres=> create role demouser;
 postgres=> alter user demouser with login;
 postgres=> alter user demouser with encrypted password 'demopass';
 postgres=> grant all privileges on database awsgodemo to demouser;
 # Restore your database to your RDS instance
 $ pg_restore -v -h <rds endpoint> -U <username> -d awsgodemo awsgodemo.sql
 # Add new upcoming column to our table
 $ psql -h <rds endpoint> -U demouser -d awsgodemo
 awsgodemo=> alter table userlog add column email varchar(255);
 awsgodemo=> update userlog set email = 'NOEMAIL' where email is null;

Remove the local database from the instance:

 $ apt list --installed | grep postgres
 $ apt remove postgresql*
 $ sudo apt-get autoremove
 $ rm -rf /etc/postgresql
 $ rm -rf /var/lib/postgresql
 $ rm -rf /etc/postgresql-common/

Create Cognito user pool:
- Set up email and password authentication, enable the "forgot password" feature and email confirmation.
- Generate a client secret and add authentication flows.
- Note down the Client ID and Client Secret for future use.

Create Instance Role:

Attach the following IAM Policy to allow our instance to access SSM Parameters:

  {
     "Version": "2012-10-17",
     "Statement": [
        {
           "Sid": "AllowSSM",
           "Effect": "Allow",
           "Action": [
              "ssm:GetParameter"
           ],
           "Resource": [
              "arn:aws:ssm:<region>:<account_id>:parameter/demo-app/*"
           ]
        }
     ]
  }

Create SSM Parameters:
- Store your application secrets (like database credentials and Client ID/Secret from Cognito) in the SSM Parameter Store as SecureString.

NOTE: SESSION_ENCRYPTION_SECRET should be a random hashed string The sessionEncryptSecret in the NewCookieStore function is used as a secret key for encrypting and authenticating the session data stored in the cookie. This ensures that the session data cannot be tampered with or read by unauthorized parties. It's important to keep this secret key secure and not expose it in your codebase or version control.

Deploy the application:
- Check out the demo-2 branch from the AWS Go Demo repository, build and deploy the application.
```
  $ sudo chmod +x main
  $ ./main
```

Create a SystemD service for the application:

Set up a new service file for your application and manage it with systemd:

  $ sudo vi /etc/systemd/system/demo-app.service

  [Unit]
  Description=My GO Service
  After=network.target

  [Service]
  User=root
  ExecStart=/etc/demo-app/main
  WorkingDirectory=/etc/demo-app
  Restart=always
  StandardOutput=file:/etc/demo-app/main.log
  StandardError=file:/etc/demo-app/main.log

  [Install]
  WantedBy=multi-user.target

  $ sudo systemctl daemon-reload
  $ sudo systemctl enable (--now) demo-app
  $ sudo systemctl start demo-app

Install and configure CloudWatch Logs with AWS Systems Manager:
- Follow the official AWS documentation to install and configure CloudWatch Logs with AWS Systems Manager. Official AWS documentation link is provided in References.
- Check the ssm-agent version and install the CloudWatch agent, configure it, and store the configuration in the Parameter Store. Finally, verify your setup by checking the CloudWatch log group and stream.

After following these steps, your Golang application is now running on a resilient and scalable setup, logging to CloudWatch, and using AWS Cognito for user management and SSM Parameter Store for secrets management. We've successfully achieved the second part of our journey to a three-tier architecture. Stay tuned for the final transformation!

Part 3: Transitioning to a Scalable and Secure Infrastructure

In this part of our journey, we'll adjust our networking resources for enhanced security and scalability, introduce Auto Scaling groups (ASGs) for resilience and elasticity, and implement secure HTTPs communication with an Application Load Balancer (ALB). This step will see our application evolve into a scalable, three-tier architecture.

Architecture diagram

Step-by-step Implementation

Adjust Networking Resources: Create Network Address Translation (NAT) gateways and Elastic IPs (EIPs). These will allow instances in the private subnets to connect to the internet or other AWS services. Adjust the routing tables of the private subnets to direct their default routes to these NAT gateways.
Capture Amazon Machine Image (AMI) of the EC2 Instance: Create an Amazon Machine Image (AMI) of our EC2 instance. An AMI is like a blueprint of the running instance, which includes the instance's operating system, the applications running on it, and their configurations.
Create Launch Template from the AMI: Use the AMI to create a launch template. A launch template is a specification that contains the settings and configurations to launch instances. You can define settings such as the instance type, the AMI ID, the key pair, and many others.
Create Auto Scaling Group: Create an Auto Scaling Group (ASG) using the launch template. ASGs enable you to ensure that you have the correct number of Amazon EC2 instances available to handle the load for your application. You create collections, or groups, of EC2 instances, and then set policies that control when instances are launched or terminated.
Migrate Workload to Private Subnet 1: For enhanced security, migrate your workload to the first private subnet. Private subnets are not exposed directly to the internet, which helps reduce the potential attack surface for your application.
Create ASG Target Group for ALB: Create a target group for the Application Load Balancer (ALB) and associate it with the ASG. The ALB will distribute incoming traffic across the EC2 instances that are registered with these target groups.
Create ACM Certificate: Create an Amazon Certificate Manager (ACM) certificate for your domain and verify it. ACM handles the complexity of creating and managing public SSL/TLS certificates for your AWS based websites and applications.
Set Up the Application Load Balancer (ALB): Create the ALB, ALB listeners, and a dedicated security group for it. Adjust the security group attached to the instances accordingly. The ALB listener should be set up for HTTP (port 80) and redirect to HTTPS (port 443). Another listener for HTTPS (port 443) should target the target group with the ASG.
Update Route 53 Record: Adjust an A Record in Route 53 to point to the ALB as an alias. This allows the domain to resolve to the ALB, and therefore to the instances that are serving our application.

Part 4: Implementing Continuous Integration/Continuous Deployment (CI/CD)

Architecture diagram

In this section, we will set up a CI/CD pipeline using AWS CodeBuild, CodeDeploy, and CodePipeline. The pipeline will automatically build and deploy our application whenever changes are pushed to the GitHub repository.

To manage and visualize the whole workflow, we'll leverage AWS CodePipeline, and to notify GitHub about the state of the deployment process, we will use a Lambda function.

Step-by-Step Implementation:

Step 1: Setup AWS CodeBuild

CodeBuild will be used to build our application. We will instruct CodeBuild on how to handle the building of our application via a buildspec file. The file will list the commands to be executed during the build phase and specify the artifacts to be created. Here is the buildspec file that we'll use:

version: 0.2

phases:
  build:
    commands:
      - echo Build started on `date`
      - go build -o main

  post_build:
    commands:
      - echo Build completed on `date`

artifacts:
  files:
    - main
    - templates/**/*
    - scripts/**/*
    - appspec.yml
    - .env

Step 2: Setup AWS CodeDeploy

We will use AWS CodeDeploy to deploy our application. CodeDeploy will use the build artifacts created by CodeBuild and deploy them to the application servers managed by an Auto Scaling group. The deployment instructions are defined in the appspec.yml file in the demo-4-cicd-integration branch of our GitHub repository.

Step 3: Setup AWS CodePipeline

AWS CodePipeline will manage the overall workflow. It will listen for changes to the demo-4-cicd-integration branch of the GitHub repository, trigger a build on CodeBuild whenever changes are pushed, and if the build is successful, it will trigger a deployment on CodeDeploy. Note that demo-4 branch is already merged to main, and we are triggering deployments on push to main.

We will use AWS CodeStar connections to connect our GitHub repository to CodePipeline. This connection allows CodePipeline to get the source code from the GitHub repository using the GitHub version 2 source action.

Step 4: Setup AWS Lambda for GitHub Commit Status updates

The AWS Lambda function will listen to the state change events of CodePipeline, format the received data to be compatible with GitHub's Commit Status API, and send the status updates to GitHub.

For this, we will use the code from the Commit Status Lambda repository. Make sure to provide the necessary permissions for the Lambda function to interact with GitHub and CodePipeline. Analyze the readme carefully.

tarlan-huseynov / commit-status-lambda

Description

This lambda written in Golang interacts with "Commit Status REST API" to update pipeline status visually in Github.

Required IAM policies for lambda

  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "codepipeline:ListActionExecutions",
        "codepipeline:GetPipelineExecution"
      ],
      "Resource": "*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters"
      ],
      "Resource": "arn:aws:ssm:region:account_id:parameter/parameter_name"
    }
  ]
}

Note: default labmda policies for logging are also needed

SNS Topic policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codestar-notifications.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:eu-west-1:166733594871:githubCommitNotifier"
    }
  ]
}

ParameterStore secret

// Github classic token with

…

View on GitHub

By following these steps, you can establish a CI/CD pipeline for your application. This pipeline will enable you to automatically build and deploy your application while also providing status updates directly to GitHub.

Remember that CI/CD pipelines are a fundamental part of modern application development practices. They allow for quicker, more reliable deployments and help maintain high standards of code quality. By integrating our pipeline with GitHub, we can ensure that our application remains up-to-date and ready for deployment at any time.

AWS Service Documentation references for Part 4.

What is CodeBuild?
What is CodeDeploy?
What is CodePipeline?
Getting started with CodeDeploy
Appspec file structure
CodeDeploy Lifecycle hooks
CodeDeploy with ASG
CodeDeploy Agent

Back to Table of Contents

DEV Community: Tarlan Huseynov

Terraform Your AWS AgentCore

Introduction

What is AgentCore?

What We Built

Architecture

Why Not Console or CLI?

What One terraform apply Actually Does

The _CODE_VERSION Trick

The Gaps and Their Real Costs

Gap 1: Gateway Targets, Missing grantType

Gap 2: Policy in AgentCore, No Resource Exists

Gap 3: Gateway Drift on Two Fields

Why the Workarounds Are Still Worth It

Bonus: The ARM64 Gotcha

Farewell 😊

AWS Multi-Account IaC Sandwich: Terragrunt, Terraform, CloudFormation

Introduction

The Challenge

The Solution Overview

Two-Phase Deployment

GitOps Folder-Based Environment Approach

Closer look

Key Aspects of Our IaC Structure

Terragrunt Pathing Pattern

infrastructure-org

Provisioning the Root Account Modules

cfstacksets Module

stacks-sdlc.tf (IAM Role Provisioning for SDLC Accounts)

stacks-shared-services.tf (IAM Role Provisioning for Shared Services)

infrastructure-live

Common Configuration (common.hcl in the root of the project)

Terragrunt Configuration for infrastructure-live (terragrunt.hcl)

Dynamic Mapping of Environments and Role Assumption

How It Works

Executing Terraform/Terragrunt in infrastructure-live

What Happens?

Farewell 😊

Terraform, Ansible, and AWS SSM Pipeline: A Powerful Trio for seamless AWS Infrastructure Automation

Table of Contents

Introduction

Prerequisites

Docker Quickstart

Review

Quickstart

Building a CI/CD Pipeline

Tagging and Pushing Docker Image

Workflow Configuration

Adding Secrets to GitHub

Farewell 😊

One tool to rule them all - Terraform: EKS Golang Client & E2E AWS Lambda CI/CD via IaC

Table of Contents

Introduction

Prerequisites

Lambda RBAC permissions

Main Configuration File

Lambda Deployment Process

Lambda client code

Terraform CI/CD

Secrets and Environment Variables

Farewell 😊

EKS Secret Management — with Golang, AWS ParameterStore and Terraform

Table of Contents

Introduction

SSM Parameters

InitContainer with GO binary

OIDC Federated Access for EKS Pods

Application Deployment

EKS ZeroScaler

Scheduled auto-scaling to zero with Lambda GO client

Table of Contents

Introduction

The Golang Lambda Function - A Deep Dive

Initialize the Go Module

Build the Lambda Function

Deploying and Orchestrating the EKS Auto-Scaling Solution with Terraform

Pre-requisites and Setup

Terraform Infrastructure Overview

Access Configuration for Lambda in EKS

Terraform codebase

What One `terraform apply` Actually Does

The `_CODE_VERSION` Trick

Gap 1: Gateway Targets, Missing `grantType`

`cfstacksets` Module

`stacks-sdlc.tf` (IAM Role Provisioning for SDLC Accounts)

`stacks-shared-services.tf` (IAM Role Provisioning for Shared Services)

Common Configuration (`common.hcl` in the root of the project)

Terragrunt Configuration for `infrastructure-live` (`terragrunt.hcl`)

Executing Terraform/Terragrunt in `infrastructure-live`