DEV Community: globart

S3 native state locking in Terraform

globart — Fri, 29 Nov 2024 21:04:36 +0000

Since the Terraform 0.5.0 release from May 2015th we've been able to store our state on S3 buckets.

But in order to ensure it's consistency, we've had to use state locking using DynamoDB table. Although the DynamoDB-associated costs are negligable, it is still nonetheless another resource, that you have create manually, which could prove to be pretty annoying if you are managing large infrastructure and/or many projects at once.

The DynamoDB was needed due to the fact that S3 only had eventual consistency, meaning that if you read a file some time after it was written, you would get its latest version. But if you attempted to read a file immediately after it was written, it wasn't guaranteed.
This was addressed when AWS introduced strong read-after-write consistency for S3 in December 2020

However, this didn't solve the issue entirely, because there was no built-in mechanism to check if an object exists, before creating it.
Fortunately, after another 4 years, Amazon introduced support for conditional writes in S3 in August 2024

These changes made it possible to start work on state locking without DynamoDB, which doesn't require any additional resources, apart from the bucket itself.
After a couple of months, S3 native state locking was introduced in Terraform 1.10.0 in November 2024
While similar discussion exists in OpenTofu repo since September 2023, at the time of writing this article, there was no equivalent solution developed. But I hope it will be created in the coming months.

So, while previously your S3 backend configuration looked like this:

terraform {
  backend "s3" {
    bucket         = "example-bucket"
    key            = "path/to/state"
    region         = "us-east-1"
    dynamodb_table = "example-table"
  }
}

Now it can look like this:

terraform {
  backend "s3" {
    bucket       = "example-bucket"
    key          = "path/to/state"
    region       = "us-east-1"
    use_lockfile = true
  }
}

While terraform apply/destroy is going on, key.tflock file will be created in S3 bucket, which contains lock information, including a unique lock ID and other metadata.

If another user tries to do terraform apply at the same time, Terraform will see that the key.tflock file already exists, so apply will fail.
After apply is completed, key.tflock file will be deleted.

Currently, while this feature is still experimental, use_lockfile argument is optional and defaults to false.
To support migration from older versions of Terraform which only support DynamoDB-based locking, it can be configured simultaneously with dynamodb_table argument. In a future minor version the DynamoDB locking mechanism will be removed.

How to create AWS ASG with HTTPS ALB using Terraform modules

globart — Wed, 03 Jan 2024 18:37:35 +0000

General settings
VPC
AMI
Additional parameters
ALB with HTTPS
ASG

This is basically a slighty modified version of a tutorial by Anton Putra, which I've changed to use modules (with pinned versions for future compatibility) instead of plain resources, so it is easier to read. I've also added the code for creating and assigning appropriate IAM Role and Instance Profile to ASG instances, so you can manage them with SSM.

General settings

First of all, you'll have to set the region, which you would like the resources to be deployed in:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.31.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

VPC

Then, you'll create the VPC and all of its resources (security groups are dependant on eachother, so they are created as resources first and then rules are created as modules)

locals {
  ...
  vpc_cidr      = "10.0.0.0/16"
  azs           = slice(data.aws_availability_zones.available.names, 0, 2)
  tags = {
    ManagedBy = "Terraform"
  }
 ...
}

data "aws_availability_zones" "available" {}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.4.0"

  name               = "alb-vpc"
  cidr               = local.vpc_cidr
  azs                = local.azs
  private_subnets    = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k)]
  public_subnets     = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 4)]
  enable_nat_gateway = true
  single_nat_gateway = true
  tags               = local.tags
}

resource "aws_security_group" "ec2" {
  name   = "ec2-sg"
  vpc_id = module.vpc.vpc_id
  tags   = local.tags
}

resource "aws_security_group" "alb" {
  name   = "alb-sg"
  vpc_id = module.vpc.vpc_id
  tags   = local.tags
}

module "alb-sg-rules" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "5.1.0"

  create_sg           = false
  security_group_id   = aws_security_group.alb.id
  ingress_cidr_blocks = ["0.0.0.0/0"]
  ingress_rules       = ["http-80-tcp", "https-443-tcp"]
  egress_with_source_security_group_id = [
    {
      from_port                = 8080
      to_port                  = 8080
      protocol                 = "tcp"
      description              = "App port"
      source_security_group_id = aws_security_group.ec2.id
    },
    {
      from_port                = 8081
      to_port                  = 8081
      protocol                 = "tcp"
      description              = "Full healthcheck"
      source_security_group_id = aws_security_group.ec2.id
    }
  ]
}

module "ec2-sg-rules" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "5.1.0"

  create_sg         = false
  security_group_id = aws_security_group.ec2.id
  ingress_with_source_security_group_id = [
    {
      from_port                = 8080
      to_port                  = 8080
      protocol                 = "tcp"
      description              = "App port"
      source_security_group_id = aws_security_group.alb.id
    },
    {
      from_port                = 8081
      to_port                  = 8081
      protocol                 = "tcp"
      description              = "Full healthcheck"
      source_security_group_id = aws_security_group.alb.id
    }
  ]
  egress_cidr_blocks = ["0.0.0.0/0"]     // these are needed for instance to be able to initiate a connection to SSM
  egress_rules       = ["https-443-tcp"]
}

AMI

After this, you'll have to create launch template for your ASG. I'll use Packer to create the AMI, as provided in Anton's post. You'll have to create the following files:
files/my-app.service

[Unit]
Description=My App
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
ExecStart=/home/ubuntu/go/bin/my-app

User=ubuntu

Environment=GIN_MODE=release

Restart=always
RestartSec=1

[Install]
WantedBy=multi-user.target

scripts/bootstrap.sh

#!/bin/bash

set -e

sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt-get update
sudo apt-get install -y golang-go
go install github.com/antonputra/tutorials/lessons/127/my-app@main

my-app.pkr.hcl

packer {
  required_plugins {
    amazon = {
      version = "v1.2.9"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "my-app" {
  ami_name      = "my-app-{{ timestamp }}"
  instance_type = "t3a.small" // this isn't the instance type of ASG, it's the type of temp instance Packer will use to build an AMI off of
  region        = "eu-west-1" // change this to the region of your resources
  subnet_id     = "subnet-074a32b171778af28" // change this to any public subnet in the specified region, for example, you can use the one from default VPC

  source_ami_filter {
    filters = {
      name                = "ubuntu/images/*ubuntu-jammy-22.04-amd64-server-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = ["099720109477"]
  }

  ssh_username = "ubuntu"

  tags = {
    Name = "My-App",
    ManagedBy = "Packer"
  }
}

build {
  sources = ["source.amazon-ebs.my-app"]

  provisioner "file" {
    destination = "/tmp"
    source      = "files"
  }

  provisioner "shell" {
    script = "scripts/bootstrap.sh"
  }

  provisioner "shell" {
    inline = [
      "sudo mv /tmp/files/my-app.service /etc/systemd/system/my-app.service",
      "sudo systemctl start my-app",
      "sudo systemctl enable my-app"
    ]
  }
}

Then, run these commands to initialize and build your AMI using Packer:

packer init my-app.pkr.hcl
packer build my-app.pkr.hcl

Additional parameters

After this, you'll have to create/import keypair and decide on the instance type, you'd like to use for your ASG. You'll also have to create Route53 public hosted zone and update nameservers for your domain with the ones provided in NS record, which will be automatically created in the Route53 zone. Then, you'll add these values to locals:

locals {
  ...
  domain_name   = "https-alb.pp.ua"
  keypair_name  = "devops"
  instance_type = "t3a.micro"
  ami_id        = "ami-06f69317847054bb5"
  ...
}

ALB with HTTPS

You can now also create ALB with support for HTTPS and a target group, which will be attached to ASG:

module "acm" {
  source  = "terraform-aws-modules/acm/aws"
  version = "5.0.0"

  domain_name       = local.domain_name
  zone_id           = data.aws_route53_zone.public.zone_id
  validation_method = "DNS"
  tags              = local.tags
}

module "alb" {
  source  = "terraform-aws-modules/alb/aws"
  version = "9.4.0"

  name                       = "alb"
  vpc_id                     = module.vpc.vpc_id
  subnets                    = module.vpc.public_subnets
  security_groups            = [aws_security_group.alb.id]
  enable_deletion_protection = false
  target_groups = {
    asg = {
      name              = "asg-tg"
      port              = 8080
      protocol          = "HTTP"
      vpc_id            = module.vpc.vpc_id
      create_attachment = false

      health_check = {
        enabled             = true
        port                = 8081
        interval            = 30
        protocol            = "HTTP"
        path                = "/health"
        matcher             = "200"
        healthy_threshold   = 3
        unhealthy_threshold = 3
      }
    }
  }
  listeners = {
    http-https-redirect = {
      port     = 80
      protocol = "HTTP"
      redirect = {
        port        = "443"
        protocol    = "HTTPS"
        status_code = "HTTP_301"
      }
    },
    https = {
      port            = 443
      protocol        = "HTTPS"
      ssl_policy      = "ELBSecurityPolicy-2016-08" // set to allow most clients, should be change to newer one
      certificate_arn = module.acm.acm_certificate_arn
      forward = {
        target_group_key = "asg"
      }
    }
  }
  tags = local.tags
}

resource "aws_route53_record" "alb" {
  name    = local.domain_name
  type    = "A"
  zone_id = data.aws_route53_zone.public.zone_id
  alias {
    name                   = module.alb.dns_name
    zone_id                = module.alb.zone_id
    evaluate_target_health = false
  }
}

ASG

And, finally, you'll create the ASG itself:

resource "aws_iam_service_linked_role" "autoscaling" {
  aws_service_name = "autoscaling.amazonaws.com"
  description      = "A service linked role for autoscaling"
  custom_suffix    = "ssm"

  provisioner "local-exec" {
    command = "sleep 10"
  }
  tags = local.tags
}

module "asg" {
  source  = "terraform-aws-modules/autoscaling/aws"
  version = "7.3.1"

  name                             = "asg"
  use_name_prefix                  = false
  vpc_zone_identifier              = module.vpc.private_subnets
  min_size                         = 1 // automatically set as desired
  max_size                         = 3
  launch_template_name             = "my-app"
  launch_template_use_name_prefix  = false
  update_default_version           = true
  image_id                         = local.ami_id
  instance_type                    = local.instance_type
  key_name                         = local.keypair_name
  security_groups                  = [aws_security_group.ec2.id]
  create_traffic_source_attachment = true
  traffic_source_identifier        = module.alb.target_groups["asg"].arn
  service_linked_role_arn          = aws_iam_service_linked_role.autoscaling.arn
  service_linked_role_arn          = aws_iam_service_linked_role.autoscaling.arn
  create_iam_instance_profile      = true
  iam_instance_profile_name.       = "ssm-instance-profile"
  iam_role_name                    = "ssm-role"
  iam_role_path                    = "/ec2/"
  iam_role_description             = "SSM role example"
  iam_role_tags                    = local.tags
  iam_role_policies = {
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }
  block_device_mappings = [
    {
      # Root volume
      device_name = "/dev/xvda"
      no_device   = 0
      ebs = {
        delete_on_termination = true
        encrypted             = true
        volume_size           = 1
        volume_type           = "gp3"
      }
    }
  ]
  scaling_policies = {
    avg-cpu-policy-greater-than-80 = {
      policy_type               = "TargetTrackingScaling"
      estimated_instance_warmup = 300
      target_tracking_configuration = {
        predefined_metric_specification = {
          predefined_metric_type = "ASGAverageCPUUtilization"
        }
        target_value = 80.0
      }
    }
  }
  tags = local.tags
}

You can optionally add output, to see complete healthcheck URL after all of the resources are created:

output "custom_domain" {
  value = "https://${module.acm.distinct_domain_names[0]}/ping" // will output only first domain supplied
}

DynamoDB Tips and Tricks

globart — Mon, 10 Jul 2023 10:15:44 +0000

Terminology
Limits
Best practices
Examples
Sources

Terminology

Table and its elements

Each DynamoDB table contains items, which are rows of data. Each item is composed of two parts. The first one is the primary key. It can contain only partition key, or be composite - containing both primary key and sort key. The second one is the attributes, which are columns that have a name and a value associated with it

There is also a concept of item collection. An item collection is one or more items, which correspond to the same partition key.
Streams

You’ve probably already dealt with or at least heard about streams: a stream gets data from producers and passes it to consumers. In our case, the producer is DynamoDB and consumers are different AWS services. The prominent one is Lambda, with the help of which you can do some post-processing on your data or aggregate it.
Secondary indexes

What makes DynamoDB so much more than just a simple Key-Value store is the secondary indexes. They allow you to quickly query and lookup items based on not only the primary index attributes, but also attributes of your choice. Secondary Indexes, unlike primary keys, are not required, and they don't have to be unique. Generally speaking, they allow much more flexible query access patterns. In DynamoDB, there are two types of secondary indexes: global secondary indexes and local secondary indexes.

LSIs must be created at the same time the table is created and they must use the same partition key as the base table but they allow you to use a different sort key. They also limit you to 10 GBs of data per partition key and they share their throughput with base table - if you query for data using LSI, the usage will be calculated against capacity of the underlying table and its base index

GSIs can be created at any time after table creation and may use any attributes from the table as partition and sort keys, so two items can have the same partition and sort key pair on a GSI. Unlike LSIs, they don’t limit you in amount of data per partition key and they also don’t share their throughput. Each of the GSIs is billed independently, and, as a consequence, throttling is also separated. Their only downside is that they only offer eventual consistency, while LSIs offer both eventual and strong consistency.

Limits

https://dynobase.dev/dynamodb-limits/

Item size
DynamoDB's limit on the size of each item, which is row, is 400 KB. To avoid hitting it, you should store large objects (e.g images, videos, blobs etc) in S3 and just store a link to them in DynamoDB or, if the item is too large by itself, break it up into multiple items.

Indexes
DynamoDB Indexes allow you to create additional access patterns. GSIs (Global Secondary Index), aka indexes that use a different attribute as partition key, are limited to 20 per table. However, that limit can be increased by asking the support.
On the other hand, LSIs (Local Secondary Index) are hard-capped to 5 indexes. Usage of LSIs adds yet another, often overlooked limitation - it imposes a 10 GB size limit per partition key value.
For that reason, you should probably always favor GSIs over LSIs.

Scan and Query operations
These two operations have not only similar syntax - both of them can also return up to 1 MB of data per request. If the data you're looking for is not present in the first request's response, you'll have to paginate through the results - call the operation once again but with NextPageToken set to LastEvaluatedKey from the previous one.

Transactions and Batch Operations
Transactional and Batch APIs allow you to read or write multiple DynamoDB items across multiple tables at once.
For transactions:
TransactWriteItems is limited to 25 items per request
TransactReadItems is limited to 25 items per request
For batch operations:
BatchWriteItem is limited to 25 items per request
BatchGetItem is limited to 100 items per request

Partition Throughput
DynamoDB Tables are internally divided into partitions. Each partition has its own throughput limit, it is set to 3,000 RCUs (Read Capacity Units) and 1,000 WCUs (Write Capacity Units) per partition key.
Unfortunately, it is not always possible to distribute that load evenly. In cases like that, "hot" partitions (the ones that receive most of the requests), will use adaptive capacity for a limited period of time to continue operation without disruptions or throttling. This mechanism works automatically and is completely transparent to the application.

Others
Throughput Default Quotas per table - 40,000 RCUs and 40,000 WCUs
Partition Key Length - from 1 byte to 2KB
Sort Key Length - from 1 byte to 1KB
Table Name Length - from 3 characters to 255
Item's Attribute Names - from 1 character to 64KB long
Item's Attribute Depth - up to 32 levels deep
ConditionExpression, ProjectionExpression, UpdateExpression & FilterExpression length - up to 4KB
DescribeLimits API operation should be called no more than once a minute.
There's also a bunch of reserved keywords.

Best practices

https://dynobase.dev/dynamodb-best-practices/

Queries

If you’re new to DynamoDB, you can use SQL-like query language, called PartiQL, to work with DynamoDB, instead of it’s native language
Use BatchGetItem for querying multiple tables - you can get up to 100 items identified by primary key from multiple DynamoDB tables at once
Use BatchWriteItem for batch writes - you can write up to 16MB of data or do up to 25 writes to multiple tables with a single API call. This will reduce overhead related to establishing HTTP connection.
If you need consistent reads/writes on multiple records - use TransactReadItems or TransactWriteItems
Use Parallel Scan to scan through big datasets
Use AttributesToGet to make API responses faster - this will return less data from DynamoDB table and potentially reduce overhead on data transport
Use FilterExpressions to refine and narrow your Query and Scan results on non-indexed fields.
Because DynamoDB has over 500 reserved keywords, use ExpressionAttributeNames always to prevent from ValidationException
If you need to insert data with a condition, use ConditionExpressions instead of Getting an item, checking its properties and then calling a Put operation. This way takes two calls, is more complicated and is not atomic.
Use DynamoDB Streams for data post-processing with Lambda Instead of running expensive queries periodically for e.g. analytics purposes, use DynamoDB Streams connected to a Lambda function. It will update the result of an aggregation just-in-time whenever data changes.
To avoid hot partitions and spread the load more evenly across them, make sure your partition keys have high cardinality. You can achieve that by adding a random number to the end of the partition key values.
If you need to perform whole-table operations like SELECT COUNT WHERE, export your table to S3 first and then use Athena or any other suitable tool to do so.

Monitoring and Security

Use IAM policies for security and enforcing best practices this way you can restrict people from doing expensive Scan operations
Use Contributor Insights to identify most accessed items and most throttled keys which might cause you performance problems.
Use On-Demand capacity mode to identify your traffic patterns. Once discovered, switch to provisioned mode with auto scaling enabled to save money.
Remember to enable PITR (point-in-time-recovery), so there’s an option to rollback your table in case of an error
Add createdAt and updatedAt attributes to each item. - - - Moreover, instead of removing records from the table, simply add deletedAt attribute. It will not only make your delete operations reversible but also enable some auditing.

Storage and Data Modelling

Store large objects (e.g images, videos, blobs etc) in other places, like S3, and store only links to them inside the Dynamo
You can also split them into multiple rows sharing the same partition key. A useful mental model is to think of the partition key as a directory/folder and sort key as a file name. Once you're in the correct folder, getting data from any file within that folder is pretty straightforward.
Use common compression algorithms like GZIP before saving large items to DynamoDB.
Add a column with epoch date format (in seconds) to enable support for DynamoDB TTL feature, which you can use to filter the data or to automatically update or remove it
If latency to the end-user is crucial for your application, use DynamoDB Global Tables, which automatically replicate data across multiple regions. This way, your data is closer to the end-user. For the compute part, use Lambda@Edge functions.
Instead of using Scans to fetch data that isn't easy to get using Queries, use GSIs to index the data on required fields. This allows you to fetch data using fast Queries based on the attribute values of the item.
Use generic GSI and LSI names so you can avoid doing migrations as they change.
Leverage sort keys flexibility - you can define hierarchical relationships in your data. This way, you can query it at any level of the hierarchy and achieve multiple access patterns with just one field.

Examples

Composite primary key

This primary key design makes it easy to solve four access patterns:

Retrieve an Organization. Use the GetItem API call and theOrganization’s name to make a request for the item with a PK of ORG# and an SK of METADATA#.
Retrieve an Organization and all Users within the Organization.Use the Query API action with a key condition expression of PK = ORG#. This would retrieve the Organization and allUsers within it, as they all have the same partition key.
Retrieve only the Users within an Organization. Use the QueryAPI action with a key condition expression of PK = ORG# AND begins_with(SK, "USER#"). The use of thebegins_with() function allows us to retrieve only the Userswithout fetching the Organization object as well.
Retrieve a specific User. If you know both the Organizationname and the User’s username, you can use the GetItem API call with a PK of ORG# and an SK of USER# to fetch the User item.

Sources

“The DynamoDB Book” by Alex DeBrie: https://www.dynamodbbook.com/
Dynobase - Professional GUI Client for DynamoDB: https://dynobase.dev/

How to debug running CodeBuild builds in AWS Session Manager

globart — Mon, 10 Jul 2023 09:44:56 +0000

Prerequisites
Pause the build
Start the build
Connect to the build container
Resume the build

This is basically the guide from AWS with added screenshots of the process

Note
This feature is not available in Windows environments.

Prerequisites

To allow Session Manager to be used with the build session, you must enable session connection for the build. There are two prerequisites:

CodeBuild Linux standard curated images already have the SSM agent installed and the SSM agent ContainerMode enabled.

If you are using a custom image for your build, do the following:

Install SSM Agent. For more information, see this guide. SSM Agent version must be 3.0.1295.0 or later.
Copy this file to the /etc/amazon/ssm/ directory in your image. This enables Container Mode in the SSM agent.

Note
Custom images would require most updated SSM agent for this feature to work as expected.

The CodeBuild service role must have the following SSM policy:

{
  "Effect": "Allow",
  "Action": [
    "ssmmessages:CreateControlChannel",
    "ssmmessages:CreateDataChannel",
    "ssmmessages:OpenControlChannel",
    "ssmmessages:OpenDataChannel"
  ],
  "Resource": "*"
}

You can have the CodeBuild console automatically attach this policy to your service role when you start the build. Alternatively, you can attach this policy to your service role manually.

If you have Auditing and logging session activity enabled in Systems Manager preferences, the CodeBuild service role must also have additional permissions. The permissions are different, depending on where the logs are stored.

CloudWatch Logs

If using CloudWatch Logs to store your logs, add the following permission to the CodeBuild service role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:DescribeLogGroups",
      "Resource": "arn:aws:logs:<region-id>:<account-id>:log-group:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:<region-id>:<account-id>:log-group:<log-group-name>:*"
    }
  ]
}

Amazon S3

If using Amazon S3 to store your logs, add the following permission to the CodeBuild service role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetEncryptionConfiguration",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>",
        "arn:aws:s3:::<bucket-name>/*"
      ]
    }
  ]
}

For more information, see Auditing and logging session activity in the AWS Systems Manager User Guide.

Pause the build

To pause the build, insert the codebuild-breakpoint command in any of the build phases in your buildspec file. The build will be paused at this point, which allows you to connect to the build container and view the container in its current state.
For example, add the following to the build phases in your buildspec file:

phases:
  pre_build:
    commands:
      - echo Entered the pre_build phase...
      - echo "Hello World" > /tmp/hello-world
      - codebuild-breakpoint

Start the build

Go to your project’s pipeline, and click on “AWS CodeBuild” link in "Build" stage. This will take you to CodeBuild project, corresponding to your pipeline:

Click “Start build with overrides”:

Click “Advanced build overrides”:

By default, “AWS CodePipeline” will be chosen as Source provider, we’ll have to change it:

It should look like this, where “Source version” is the name of your branch:

Also, check “Enable session connection” in "Environment" section:

Connect to the build container

After all of this, you can scroll to the bottom and click "Start Build". After some time, link to connect to the build container will appear. Click it and a terminal session will open that allows you to browse and control the build container:

Resume the build

After you finish examining the build container, issue the codebuild-resume command from the container shell: