DEV Community: Carlo Mencarelli

AWS Amplify Through An Infrastructure Lens

Carlo Mencarelli — Wed, 19 Apr 2023 04:20:33 +0000

Every company wants an app these days, and creating one is getting easier every day. Amazon has specific solutions for this, advertising that you can "Build full-stack web and mobile apps in hours." And it's "Easy to start, easy to scale."

I spent a few weeks toying with Amplify. I wanted to learn how easy it was to build a mobile app. I'm not a front-end developer; I've spent the last ten years doing systems and infrastructure engineering. My programming experience is limited to scripting or side projects with no audience beyond myself.

In this write-up, I present my thoughts on Amplify as a platform for building mobile apps.

The Project

I chose to create a water consumption tracking app, for no real reason besides that it was something I could require authentication for, I needed data persistence, and I could get away with a minimal GUI.

I chose React Native somewhat on a whim. Some of my friends in the industry feel that it's a good solution, and I didn't have experience; plus, it is cross-platform, so I could theoretically build an iOS or Android app if I was going to release this to the public.

Documentation

In my opinion, the Amplify docs are a cut above the rest of the AWS documentation. They have a modern feel and are easy to consume, with panels for important callouts, code blocks with highlighting, and up-to-date UI screenshots.

The Development Process

When creating an Amplify project, I was impressed with the CLI that the Amplify team maintains. You can add storage, APIs, databases, and more using amplify add. After creating your project, you start a new Amplify project with amplify init, which walks you through several options for your app. The CLI does its best to figure out some values. For example, it knew I was building a React Native app and selected the appropriate framework.

First, I added authentication to the app. Using amplify add auth, I am given a short questionnaire to follow. An excellent quality of life feature in the Amplify CLI has an option for many of the wizard questions: "I want to learn more." This gives the user a brief explanation of the question and the possibilities.

Speaking of authentication, the Amplify library makes connecting Cognito super easy. Include the library and wrap App in my App.tsx: export default withAuthenticator(App);

After adding various components using the add command, you deploy them with amplify push. This pushes your configuration to AWS, which deploys via CloudFormation. It's generally a smooth process and simplifies the Cloud infrastructure components so an app developer can focus on building their app using their language of choice.

I am not generally impressed with the AWS Console experience. It's inconsistent, dated, and buggy. My go-to solution is usually infrastructure as code or the AWS CLI. However! The Amplify Studio is an outstanding example of what the console could be. It raises the bar in how an AWS service team can construct and manage the console. In a lot of ways, it feels like a completely different experience. This makes sense, given the target audience that Amplify has.

The Infrastructure

So what about the infrastructure that's created? Does it follow best practices and well-architected pillars? Using the Amplify CLI, I made a Cognito user pool, S3 buckets, a DynamoDB table, and a GraphQL API with AppSync.

The S3 bucket is created to hold user-uploaded content, public or private. It manages this separation using an IAM policy that allows access to a specific set of app users using the cognito-identity.amazonaws.com:sub IAM condition key. The bucket doesn't have versioning or logging enabled and still uses the default S3 KMS key. This is fine but should be a consideration depending on what the app's intention is.

When creating the DynamoDB table, the Amplify CLI allows you to select your columns and partition key, but also any sort keys, global secondary indexes, and Lambda triggers which is a nice touch when creating it in the first place. It's made using the AWS-owned KMS key for DynamoDB, which like S3, may not be an issue. A nice thing about the table that's created is that it is created with the "On-Demand" capacity mode, which is excellent for unknown loads.

AppSync is deployed with a simple configuration. No logging, WAF, or XRay enabled. However, when creating the API, you can have Amplify create all of the operations for the API. This was great for speeding up development time.

Security

Looking at the infrastructure created with a more security-minded lens, I used one of my favorite tools: Prowler and SecurityHub, for a second opinion. They both generally found the same problems.

Regarding S3, a lot of what you'd expect alerted: Missing MFA delete, versioning was disabled, SSE-KMS was not used, and access logging, to name a few. None of these are deal breakers, and they are easy enough to enable and understand what you are doing with little to no experience. I'd argue that KMS is the most complicated to get right, and as stated above, the use case would need to be right for it to be a required facet to enable.

Beyond this, Prowler and SecurityHub caught some misconfigurations with Cloudwatch logging and IAM. This last one has no relation to Amplify, but if I created an account for an app, I'd still have some security work here to do.

What I'd Like To See More Of

Security

Security is paramount in today's Internet. Some intelligent defaults are in place, but there's no easy CLI way to enable features such as the WAF or better IAM and account-level practices with a single click. Amazon knows what these practices are since they alert in SecurityHub. Allowing an easy adoption of these via Amplify makes a lot of sense in protecting the app and the developers from potential compromises.

That's not to say Amazon should be more responsible, but it's clear that Amplify is aimed at audiences that may not have a robust Cloud or Security background.

More Service Tie-Ins

Amplify has a rich tie-in with multiple standard services, including AppSync, DynamoDB, and Cognito. It can also hook up to some of Amazon's ML offerings, including Rekognition, Textract, Translate, and Pinpoint. I would love to see Amplify expand to connect to more services. What if I need to connect to my legacy API using the API Gateway or to an Elasticache cluster? Sure, nothing stops me from doing that, but Amplify loses some of its charm this way.

Conclusion

Amplify is an excellent platform for single or small teams of app developers. It's a novel app development approach that breaks down many barriers and allows you to get started very fast. I'd be curious to find some case studies of large, million-user apps that leverage Amplify just to see how they approach development, security, and operations.

If you are a developer with an idea or even someone who has never touched a line of code in their lives, you could get started reasonably quickly, which is impressive.

How To Make AWS Budgets Work For You

Carlo Mencarelli — Sun, 21 Aug 2022 14:03:07 +0000

TLDR

By spending a few minutes of your time now, you can prevent a billing catastrophe in the future!
There’s a lot of flexibility in what you can alert on and how that lets you fit it to your needs.

Why Use AWS Budgets?

A few days ago, I was speaking with a colleague who was toying with AWS Textract and was surprised when he had gotten contacted by an AWS rep trying to sell him business-tier support a few days later. When he logged in to his account, he saw that he managed to accumulate a $16,000 AWS bill due to an oversight in how he set up his code. Instead of testing Textract against a few documents, he tried it against thousands, each with a hundred pages. This got me thinking about a post I had written not too long ago that centered around keeping up with technology trends and how not doing so caused a massive spike in KMS costs.

Both of these costly mistakes could have been prevented with an easy setup of the AWS Budgets service.

Creating a Budget

Creating a basic budget is pretty straightforward. There’s a workflow that can be followed in the AWS console. There are different options for types of budgets, but the primary one is the Cost Budget. That’s not to say the others aren’t useful, but the cost budget lets you convert usage into actual spend and notify on what’s important: “How much of your money are you spending?”

While not reporting and alerting on usage, the cost budget has more options to filter and set the budget scope on than the usage budget, which is an interesting choice by the AWS team. The cost budget scope contains many of the same dimensions in the Cost Explorer.

After selecting the cost budget, you’re presented with a simple web form asking for the budget name, amount, and scope. Putting name aside, let’s talk about the Budget Amount panel.

The Budget Amount panel

This panel has a few settings. None are overly complex, but I’ll briefly go into each one.

The period selection is pretty simple: How often do you want the budget to reset? You can select daily, monthly, quarterly, or annually. I’ve found monthly to be a good sweet spot, but daily spending is suitable for personal accounts. Quarterly and annually are likely ones you may encounter for enterprise or financial planning budgets.

The renewal type is also straightforward. Do you want the budget to recur or expire at some time in the future? Then you’ll select the start (and end if you pick an expiring budget) dates.

Budgeting method is the most complex option on this part of the form. Fixed is the easiest. Set a fixed amount for the budget period that was selected above. Planned will let you set a value for each month or autogenerate it based on percentage growth. Auto-adjusting is a newer option that leverages forecasted values or historical data to generate your budget value. It’s an interesting approach for those who want to remove any budget maintenance that may come from growth or optimization occurring monthly.

Budget Scope selection

Next is the scope selection. Simple on the surface, but it contains a lot of potential for a very detailed selection of filters. This is also where you can exclude certain types of charges, such as credits, taxes, support, etc. If you need a simple, account-wide budget, then this is fine to leave alone, but what if you wanted to set up a budget to monitor your EC2 spot usage or your data transfer costs out of Lambda for US-West-2 and US-East-2? Well, you can!

Configuring Alerts

What good are these budgets if they don’t notify anyone when triggered? You can create numerous alerts on both forecasted and actual values. You can send an email, SNS, or AWS Chatbot alerts. I’d recommend setting up one for actual values and one for forecasted values. AWS cost forecasting isn’t always the most accurate, but it can be an additional layer of detection against things going awry in your account.

Configuring Actions

Budget actions are fascinating; from one perspective, it’s incredibly interesting that based on alerts configured in the previous step, you can apply a Service Control Policy (SCP) or even shut down RDS or EC2 instances. However, it’s also very limited. You need to select the instance ids when wanting to shutdown EC2 instances; this doesn’t work if you have a dynamic environment with instances being terminated and recreated. The actions also don’t allow for anything more complex than that. If, for example, you could trigger a lambda or step function for a Budget alert being triggered, you would be able to take many different types of actions.

Scaling Budget Creation (Using Terraform)

If you only have one account, you may only need one budget. However, that’s not always the case. Budget creation and management are fully compatible with Terraform. In fact, for some, it may be simpler to understand and put together budgets in Terraform than through the Console wizard.

Terraform has two related resources: aws_budgets_budget (Doc Link) and aws_budgets_budget_action (Doc Link). Below is a brief example similar to the example built in the GUI above.

resource "aws_budgets_budget" "account" {
  name              = "AccountWide"
  budget_type       = "COST"
  limit_amount      = "5000"
  limit_unit        = "USD"
  time_unit         = "MONTHLY"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 95
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = ["me@carlo.cloud"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 110
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["me@carlo.cloud"]
  }
}

Wrapping Up

That’s all there is to it; this is a simple but often overlooked service in AWS. It can give some peace of mind, so you never need to receive that call from finance or AWS asking if you want to buy business-level support due to your prolific usage of a service you never intended to use.

References

AWS - Budgets Documenation

Linux’s Cloud Init — Benefits, Quirks, and Drawbacks

Carlo Mencarelli — Mon, 13 Jun 2022 16:55:20 +0000

Originally created by Canonical for Ubuntu on AWS EC2, it’s now the de facto early boot configuration method

TLDR

Cloud Init is an invaluable resource for Cloud Engineers and Software Developers alike.
It's a straightforward service on the surface but is highly customizable to whatever needs an org may have the case for.
Cloud Init isn't only AWS EC2 user data; it does network configuration, vendor configuration, and provides metadata services.

You're probably using Cloud Init and don't even realize it. Created by Canonical in the early days of EC2, it helped revolutionize how we treat our servers and how runtime initialization is conducted. Since its inception, it has been one of the primary methods of early configurations for our infrastructure. It's also run in the stacks of every major public cloud provider and many private cloud environments like LXD, KVM, and OpenStack.

Cloud Init allows engineers to reduce or even eliminate package installs or configurations during application deployment. "Why should I need to install ImageMagick on every single Rails deployment?" Similarly, Cloud Init can provide breathing room between OS image builds since you can do any security patching as a part of Cloud Init so you can rotate your AMI on a more manageable basis, such as a weekly cadence.

How Does Cloud Init Work

Cloud Init Stages

Cloud Init works in a couple of different stages.

First, for systemd machines, is the Generator stage. If you're unfamiliar with systemd, a generator is a binary executed early in the boot process to dynamically generate unit files, symlinks, and more. Cloud Init's generator determines if the rest of the Cloud Init process should continue. If so, Cloud Init is included in the list of boot goals for the system.

Next is the Local phase. This phase runs the cloud-init-local.service systemd service and runs as early as possible. Essentially its entire purpose is to locate data sources and generate (or apply) networking configurations for the system. It's worth noting that this phase blocks much of the boot process, including the network initialization.

The Network phase continues the Cloud Init boot. This phase relies on networking being up (and, by association, the Local phase). This stage will run any cloud_init modules found. These might be things such as mount and bootcmd options.

After the Network phase is the Config phase, this is the phase that runs the modules that don't affect any other stages. Specifically, it runs the cloud_config modules in the Cloud Init config directory. runcmd is included in this step.

Cloud Init closes out with the Final phase. Running any cloud_final modules, this phase runs as late as possible. It is the stage that includes any user data scripts and configuration management tooling (Puppet, Chef, etc.).

Instance Metadata

Each server using Cloud Init also has a collection of data that Cloud Init uses to configure the instance. This includes what we generally think of as instance metadata on EC2 instances but also more.

Some providers will create or attach a config drive containing metadata service information files. OpenStack is an example of one such provider.

While we interact with user data, Cloud providers can also implement vendor data. The idea here is the same as user data; it exists to allow the cloud provider to customize the image at runtime. Some potential vendor data tasks might involve setting the instance's hostname or configuring package repository paths. Vendor data can be disabled if desired. It's also worth mentioning that user data overwrites vendor data when Cloud Init determines the final configuration.

Getting Started with Cloud Init

Cloud Init can be instrumented in two ways: a shell script or a YAML formatted cloud-config file. Both approaches are pretty straightforward:

#!/bin/sh

sudo yum --assumeyes --security update-minimal

Or, the equivalent cloud-config:

#cloud-config

runcmd:
 - [ sudo yum --assumeyes --security update-minimal ]

The script option is pretty easy to understand. As mentioned above, it's executed in the Final phase. The cloud-config option is more interesting since you can set up modules to run in the different phases, such as the bootcmd option. Check out the module reference page for a complete list of available modules. There is also a great list of example configurations on the cloud-config examples page.

Disabling Cloud Init

If for some reason, you want to, you can prevent Cloud Init from running. This can be accomplished in a couple of different ways. The easiest is to add a file during the AMI build time:

touch /etc/cloud/cloud-init.disabled

You can also add a parameter to /proc/cmdline:

cloud-init=disabled

It's also possible to disable user data by setting the allow_userdata parameter in /etc/cloud/cloud.cfg:

allow_userdata: false

Troubleshooting Cloud Init

Logs

Occasionally, you may want to dig deeper into Cloud Init. Maybe your user data isn't executing how you expect or possibly taking longer than expected. Fortunately, Cloud Init tracks a lot of details for debugging.

The main logs are:

/var/log/cloud-init.log
/var/log/cloud-init-output.log

These logs can interact with the cloud-init command with the analyze sub-command. This can help parse the logs into a more usable format.

There are also logs in the /run/cloud-init directory. These logs are more related to some of the inner workings and decisions of Cloud Init.

Data Files

The /var/lib/cloud/ directory is where the data files are kept. A handy file in this directory is the status.json file. This includes the stages ran and the start/finish times for each one (in epoch format).

[ec2-user@ip-10-0-0-60 data]$ cat /var/lib/cloud/data/status.json
{
 "v1": {
  "datasource": "DataSourceEc2",
  "init": {
   "errors": [],
   "finished": 1655096178.478916,
   "start": 1655096152.503821
  },
  "init-local": {
   "errors": [],
   "finished": 1655096151.389412,
...File snipped for brevity

Configuration Files

Config files are kept in /etc/cloud/cloud.cfg and the /etc/cloud/cloud.cfg.d/ directory.

Useful Cloud Init Commands to Know

Systems equipped with Cloud Init come with a binary used to interact with it. The command to use is cloud-init.

One of the most useful commands is cloud-init status which returns the status of the Cloud Init run. An optional --long flag grants more detail:

[ec2-user@ip-10-0-0-41 ~]# sudo cloud-init status
status: running
[ec2-user@ip-10-0-0-41 ~]# sudo cloud-init status --long
status: done
time: Mon, 13 Jun 2022 04:47:45 +0000
detail:
DataSourceEc2

The cloud-init status command also has another great flag: --wait. This flag waits until Cloud Init is completed before returning. It's helpful if you are using AWS CodeDeploy or a configuration management system that phones home on startup but isn't tied to Cloud Init for some reason. There is a very real chance that your CodeDeploy may start up before Cloud Init is finished which means any configuration, binaries, or environment variables set by your user data script would not be available.

[ec2-user@ip-10-0-0-41 ~]$ sudo cloud-init status --wait
..................
status: done

Another useful command is cloud-init query which references the cached instance metadata that was captured by Cloud Init:

[ec2-user@ip-10-0-0-41 ~]$ sudo cloud-init query cloud_name
aws
[ec2-user@ip-10-0-0-41 ~]$ sudo cloud-init query availability_zone
us-west-2b

Wrap Up

Knowing more about Cloud Init and how to properly leverage it can be extremely advantageous to multiple facets of an org. It can make Cloud Engineers and System Administrators' lives easier by reducing the need for configuration tooling and AMI rotations. It can also speed up application deployments.

The documentation for Cloud Init is pretty in-depth and a valuable resource. It has great details on many of the cloud providers' implementations of the metadata service. The documentation also has information about creating custom modules that can be injected and executed just like runcmd or mounts.

References

Reducing KMS Costs with S3 Bucket Encryption

Carlo Mencarelli — Mon, 18 Apr 2022 13:29:33 +0000

A tale of why keeping up in the industry is critical

TLDR

When encrypting at scale, customer-managed keys in KMS can be very costly. Using bucket keys will reduce the calls to KMS, which will reduce costs.
The implementation of bucket keys changed the cost from $1,500 per day in just KMS requests to $300 per day ($36,000 per month!)

I want to share a story that might help others from a technical perspective but also could encourage a thought that just because everything is running, there is still reason to continue to improve on existing infrastructure all of the time:

You get an alert from your finance team. Last month's AWS bill had an unexpected spike in cost. You pull up cost explorer and immediately see the increase. When you dig in, you see a spike in the expenses to KMS. It crept up from the average of a few hundred to a few thousand dollars per day.

It's not the end of the world, but you need to figure it out. You'll investigate it when you have time between different projects.

A few days later, you get a message in Slack: "We're hitting our KMS limit in production."

If you've never looked, the default rate limit for symmetric cryptographic requests in KMS is 50,000 requests/second.

Your cost issue just became a production incident though.

The incident itself isn't too significant, but the discovery of what was happening was.

What was making 50,000 KMS requests/second?

The KMS monitoring isn't great. Cloudwatch doesn't even split the operations, so you can't get the rates of just kms:decrypt calls. The monitoring overview page of the KMS developer guide briefly hints at this:

AWS KMS API activity for data plane operations. These are cryptographic operations that use a KMS key, such as Decrypt, Encrypt, ReEncrypt, and GenerateDataKey.

So when looking at CloudWatch you see a CryptographicOperationsSymmetric metric under All > Usage > By AWS Resource.

Fortunately, we had a timeframe: the spikes were happening every hour, on the hour. With nothing standing out in the applications, we turned to Cloudtrail logs. The logs showed us an abnormally large amount of requests from our Quicksight service role to KMS, with the encryption context being our data warehouse.

That presented a wrinkle, we needed to maintain the freshness of our SPICE dataset, but we weren't given too much in the way of options. So the team was able to perform some mitigation by spreading out the refreshes; we shouldn't hit the rate limit anymore. We were still seeing hundreds of millions of requests to KMS daily, which is expensive based on current prices:

500,000,000 requests/day * $0.03 per 10,000/requests = $1,500 per day

Enter S3 Bucket Keys

In December of 2020, Amazon introduced S3 Bucket Keys. Advertised as a method to reduce requests to KMS (and the associated costs).

The implementation is simple. In AWS, it's a simple check box:

In Terraform, it's equally simple using the aws_s3_bucket_server_side_encryption_configuration resource:

resource "aws_s3_bucket_server_side_encryption_configuration" "example" {
  bucket = aws_s3_bucket.mybucket.bucket

  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.mykey.arn
      sse_algorithm     = "aws:kms"
    }
    bucket_key_enabled = true
  }
}

The above solved requests for new objects in the future; however, the nature of our data warehouse was that we were fetching the entire data set on each refresh and the bucket contained terabytes of data encrypted with the old key. This revelation meant we were still seeing millions of KMS requests. AWS calls this out in the S3 Bucket Key documentation:

When you configure an S3 Bucket Key, objects that are already in the bucket do not use the S3 Bucket Key. To configure an S3 Bucket Key for existing objects, you can use a COPY operation

So we used an S3 Batch Operation to re-encrypt the data within a few hours. The effect was noticeable immediately. Looking back, it's even more distinct:

The Results

With essentially a one-line code change and a straightforward batch operation, we reduced our KMS requests from, on average, 500M requests per day to 38M requests per day. The bill showed similar results, $1,500 per day back to $300 per day in KMS requests:

This Story is Never Finished

The production incident was resolved, accounting would be happy, but the story isn't finished.

The story is never finished.

We live in a cycle of continuous improvement.

As long as Amazon, Microsoft, and the rest of the industry continue to iterate and improve, it will fall onto the engineering staff of companies to keep their knowledge relevant and constantly improve. Our implementation and feedback to the industry feed the next big release.

It's easy to fall into feature factory mode, where you only track new code and features going into the environment. It's essential to take a step back and consider the larger picture sometimes. How has the landscape changed since you created that Cloudfront distribution? What new features can be leveraged to improve engineering productivity and customer experience? It depends on the engineering leadership to guide and mentor the rest of the staff in these thoughts and approaches. It won't only benefit the company but also you, your engineers, and others with whom you share the learnings.

References

Multi-Region S3 Strategies

Carlo Mencarelli — Fri, 01 Apr 2022 13:51:13 +0000

TLDR

Whether starting a fresh project or enhancing a project that has a lot of mileage, the data layer should be one of the first ones tackled with respect to multi-region and disaster recovery.
Greenfield projects are straightforward to configure and manage. It can be as few as just 2 or 3 more Terraform resources. For more flexibility at the cost of simplicity, we can add bi-directional replication and multi-region access points.
Existing projects are a little more complicated but can be accomplished with not too much more work.
Check out the sample repos here:

Setting the Stage

Whether you are starting a project greenfield or working with a bucket that is celebrating its thirteenth birthday, you should consider what your data layer looks like through a multi-region and disaster recovery lens. Of course, S3 touts its durability (99.999999999% — 11 9’s!), and through its multiple availability zone design, there is very high availability; however we have certainly seen regional S3 outages.

In this article, I’ll explore what implementing multi-region S3 looks like for both existing and new buckets.

New Buckets

Starting with the easy scenario first:

Assume you are at a company based out of a single region, us-east-2. The company has been successful in a single region and has weathered most major Amazon outages with minimal reputational damage. The company is starting up a new web project, and you’re tasked with creating the S3 buckets for the static assets. The catch is that your manager asked you to design it with disaster recovery in mind in case us-east-2 ever has an outage.

The Approaches

At the start the solution seems simple: Create two buckets and just deploy to both of them.

In reality, that’s not exactly true. There’s a decision that needs to be made before any buckets are created. Do you augment your deployment pipeline to deploy to both buckets, or do you leverage AWS native S3 bucket replication? There are definitely benefits and drawbacks to each approach; for example, what if the deployment pipeline breaks halfway through the deployment of the change when uploading to the bucket in the DR region, or what if a manual change is made in us-east-2 and replicated to us-west-2? I won’t discuss the pros and cons too much since adoption ultimately falls to the design paradigms of you and the business at the end of the day.

Setting the stage for the examples below, each one will have a pair of versioned, encrypted buckets that have public access disabled. Additionally, a KMS key for each region will be created.

Deploying to Multiple Buckets with Your CI/CD Pipeline

Leveraging your CI/CD pipeline to push to both buckets simultaneously.

This is probably the more straightforward system to set up. In most deployment systems, you can add additional targets to deploy your artifact. The two buckets in AWS are treated as separate and do not interact with each other in any way.

The Terraform is equally simple. It’s only two buckets. You can see the sample here: mencarellic/terraform-aws-s3-multi-region/day-0-dual-deployment

Using S3 MRAP and Bi-Directional Replication

Letting AWS do the replication across region for you.

This strategy is a little more complex, but I think it has the added benefit of not needing your CI/CD tool to implement data resiliency. Let AWS and S3 worry about that; after all, they are the professionals at it.

Your artifact building pipeline pushes the artifact to two (or more) buckets that are tied together with a multi-region access point. The buckets use bi-directional replication, and from there, your deployment tool uses the same access point to deploy the code. This method has the additional safety of continued operations even if a region’s S3 service is down.

The Terraform for this is a little more complex. Mainly in the form of the inclusion of the aws_s3control_multi_region_access_point resource and the replication configuration to support bi-directional replication of the buckets.

First the multi-region access point resource. This is a straight-forward resource, just probably not common yet since it has a pretty narrow use case and is relatively new (re:Invent 2021). You can find the Terraform docs for the resource here: aws_s3control_multi_region_access_point.

resource "aws_s3control_multi_region_access_point" "app-artifact" {
  details {
    name = "app-artifact"region {
      bucket = aws_s3_bucket.app-artifact-east-2.id
    }region {
      bucket = aws_s3_bucket.app-artifact-west-2.id
    }
  }
}

Next, the replication configuration. The configuration exists in both buckets since we’re doing bi-directional replication. The replication also needs an IAM role for the replication to occur.

Note: In versions before the 4.X refactor of the AWS provider this was much more difficult to achieve since to apply bi-directional replication, you would either have to create the buckets first. Then add the configuration or accept that the first Terraform plan/apply would fail since there will be a race condition of both buckets having a replication configuration that requires the destination bucket to exist already before it could complete.

The configuration for the replication resource is straightforward as well (found here: aws_s3_bucket_replication_configuration). The catches come from the IAM policy and the KMS encryption.

In your IAM policy you’ll want to ensure you’re granting permission for the S3 actions for both buckets and their children objects:

statement {
    sid = "CrossRegionReplication"
    actions = [
      "s3:ListBucket",
      "s3:GetReplicationConfiguration",
      "s3:GetObjectVersionForReplication",
      "s3:GetObjectVersionAcl",
      "s3:GetObjectVersionTagging",
      "s3:GetObjectRetention",
      "s3:GetObjectLegalHold",
      "s3:ReplicateObject",
      "s3:ReplicateDelete",
      "s3:ReplicateTags",
      "s3:GetObjectVersionTagging",
      "s3:ObjectOwnerOverrideToBucketOwner"
    ]
    resources = [
      aws_s3_bucket.app-artifact-east-2.arn,
      aws_s3_bucket.app-artifact-west-2.arn,
      "${aws_s3_bucket.app-artifact-east-2.arn}/*",
      "${aws_s3_bucket.app-artifact-west-2.arn}/*",
    ]
}

If you’re doing KMS/SSE, you can enforce it with the following conditions in the above statement:

    condition {
      test     = "StringLikeIfExists"
      variable = "s3:x-amz-server-side-encryption"
      values   = ["aws:kms", "AES256"]
    }
    condition {
      test     = "StringLikeIfExists"
      variable = "s3:x-amz-server-side-encryption-aws-kms-key-id"
      values = [
        aws_kms_key.bucket-encryption-west-2.arn,
        aws_kms_key.bucket-encryption-east-2.arn
      ]
    }

Which, speaking of encryption, you need to remember to include kms:Encrypt and kms:Decrypt permissions in your IAM policy:

statement {
    sid = "CrossRegionEncryption"
    actions = [
      "kms:Encrypt",
      "kms:Decrypt"
    ]
    resources = [
      aws_kms_key.bucket-encryption-west-2.arn,
      aws_kms_key.bucket-encryption-east-2.arn
    ]    condition {
      test     = "StringLike"
      variable = "kms:ViaService"
      values = [
        "s3.${aws_s3_bucket.app-artifact-east-2.region}.amazonaws.com",
        "s3.${aws_s3_bucket.app-artifact-west-2.region}.amazonaws.com"
      ]
    }    condition {
      test     = "StringLike"
      variable = "kms:EncryptionContext:aws:s3:arn"
      values = [
        "${aws_s3_bucket.app-artifact-east-2.arn}/*",
        "${aws_s3_bucket.app-artifact-west-2.arn}/*"
      ]
    }
  }

AWS S3 MRAP Showing two-way replication between our application buckets. Success!

The full set of Terraform code can be found at mencarellic/terraform-aws-s3-multi-region/day-0-bucket-replication

Existing Buckets

Now the interesting scenario:

The new application you helped build to be multi-region from day 0 has taken off, and now you’ve been asked to help add disaster recovery to the legacy apps. Sounds easy, except the existing legacy bucket has terabytes of data that would need to be replicated.

So, realistically, an application might not have terabytes of data that is absolutely required for complete disaster recovery, but also who would want to sift through countless files in a bucket that is almost as old as S3 to find the relevant files?

Your existing applications infrastructure looks pretty standard: S3 bucket, public access block, KMS encryption. The first thing we need to do is stand up the secondary region’s bucket. We should probably enable replication so that anything going forward from today is replicated. This is done exactly as it was done above: with a aws_s3_bucket_replication_configurationresource.

Next, you’ll want to get an inventory of everything in the source bucket. We’ll do this with the aws_s3_bucket_inventory resource (documentation here: aws_s3_bucket_inventory). You’ll notice that our destination is actually a whole separate bucket. When taking on this endeavor, you’ll probably want to have all of your inventories go into the same place so you’ll know where they all are.

resource "aws_s3_bucket_inventory" "app-artifact-east-2" {
  bucket = aws_s3_bucket.app-artifact-east-2.id
  name   = "EntireBucketDaily"  included_object_versions = "Current"  schedule {
    frequency = "Daily"
  }  destination {
    bucket {
      format     = "CSV"
      bucket_arn = aws_s3_bucket.inventory.arn
    }
  }provider = aws.east-2
}

Once you have an inventory file, you can use an S3 batch operation to copy the files in the inventory file from the legacy bucket to the new bucket.

Note: The COPY batch operation is new as of February 8, 2022. You can read more about it in the AWS News post here: NEW — Replicate Existing Objects with Amazon S3 Batch Replication. Before this release, there were two options, and neither were very good: Manually copy the files using a script that takes time and costs money, or open a support ticket and hope they can get to it. When my team opened a request for this before batch job support, we were told it would be several weeks before they could get to it.

Regrettably, we can’t configure batch operations via Terraform just yet. There is an open issue in the Terraform provider though: Feature Request: Support for S3 Batch Operations. We can, however, configure the IAM role for the batch operation.

The IAM role is pretty easy:

An assume role policy for batchoperations.s3.amazonaws.com
s3:GetObject* for the source bucket
s3:PutObject* for the destination bucket
s3:GetObject* for the inventory bucket
If you are configuring a job report, you’ll want to allow s3:PutObject for an S3 destination too. I use the inventory bucket for this
kms:Encrypt and kms:Decrypt for encryption and decryption if there is encryption enabled

After the buckets and role are correctly configured, you’ll want to navigate to the S3 Batch Operations console.

If desired, you could change tags, metadata, or storage class. But if we use this for an active/active configuration, we’ll just leave everything as the default value.

After the job is created, it will run through a series of checks and then wait for your confirmation before executing. After confirmation, the job starts, and you’re able to monitor the status from the job status page. It runs pretty quickly. In my sample, I copy 1,000 objects in 14 seconds. Admittedly they are essentially empty objects, but it is still pretty fast.

The job status page with an overview and high-level status

Regarding other options for replicating existing buckets, as stated above, alternate options would be to set up replication between the source and destination buckets and set up either a lambda or batch job that touches each file to trigger bucket replication from the original bucket to the new bucket. You can do the same in an EC2 instance or even from a local laptop though those options are less recommended.

There are some negatives with this approach, mainly in that you’re recreating something that exists natively in AWS already, and I’m sure you have better things to be doing than recreating the wheel.

References

Terraform Docs

The Case for the Terraform Seed Workspace

Carlo Mencarelli — Mon, 21 Mar 2022 18:30:35 +0000

Why not use Terraform to manage your Terraform?

TLDR

Terraform collaboration tools such as Terraform Cloud, Scalr, and SpaceLift offer the ability to be managed by Terraform.
Managing your infrastructure automation tool, while more overhead, brings in many benefits ranging from consistency to security.
Terraform Cloud Demo repo: mencarellic/terraform-cloud-workspace - Scalr Demo repo: mencarellic/terraform-scalr-workspace
Spacelift Demo repo: mencarellic/terraform-spacelift-workspace

When working with a production Terraform configuration, there will inevitably be a time when you outgrow the single workspace pattern. Whether you split workspaces by account, region, service, or something else, soon you'll find that you need to break your Terraform configuration apart.

An aside on the term workspace

The word workspace in this post follows the Terraform Cloud and Scalr definition instead of the Terraform CLI definition. Hashicorp has a blurb here: Terraform Cloud vs. Terraform CLI Workspaces, but it boils down to this:

Terraform CLI workspaces are infrastructure as code split into different directories and have different state files. Terraform Cloud and Scalr workspaces are similar in that they have individual state files; however, these tools also provide more features like specific role-based access control and variable configurations.

Also worth noting is that in Spacelift, the equivalent of a workspace is a stack.

The Terraform configuration I work with daily consists of 31 active workspaces and 36,747 resources as of writing this. Two of those workspaces have over 9,000 resources a piece, and as you can imagine they take quite a bit of time to run. Logically breaking these workspaces down into smaller chunks will help speed those Terraform plans and applies up. It also helps unlock safe developer inclusion since I can allow an application team access to a workspace that contains only their infrastructure. All of a sudden, one of my 9,000 resource workspace turns into 12 workspaces that have 750 resources each with the added benefit of my engineering teams being able to iterate faster and independently of each other.

Of course, if I'm managing fifty or even thirty workspaces, I probably have repeated variables across them. The seed workspace lets me manage all of these variables (including sensitive ones) via code. I also can manage teams, tags, VCS configuration, and more.

The pattern lets me leverage Terraform patterns to maintain consistency and state on a critical layer of my tooling. The three demo repos I created are all very straightforward and should be easy to comprehend. The primary difficulty ends up being the use of the providers isn't as thoroughly tested as some of the primary Terraform providers like AWS or AzureRM so the documentation ends up having quirks. For example, in the Terraform Cloud demo I originally created teams for a more fleshed out sample, but the provider documentation for the tfe_team resource doesn't mention that the capability isn't available in the tier I was using.

In my Terraform Cloud Workspace demo, you can see I define four key values. The values for the keys ultimately live in Terraform Cloud's variable section which I can lock down using the built-in access controls. But since those values are a part of my seed workspace, I can assign them to the workspaces I create programmatically which makes rotation a breeze as well.

The fifth variable I define is actually a global Terraform version that I apply to each of my workspaces. This is just an example. I could do the same with branch name, SSH key, etc. This pattern is all about keeping consistency in a repeatable and secure method.

variable "terraform-version" {
  type        = string
  description = "Global Terraform version to use"
  default     = "1.1.4"
}
variable "azure-dev-key" {
  type        = string
  description = "Development key for Azure"
  default     = ""
}
variable "azure-prod-key" {
  type        = string
  description = "Production key for Azure"
  default     = ""
}
variable "aws-dev-key" {
  type        = string
  description = "Development key for AWS"
  default     = ""
}
variable "aws-prod-key" {
  type        = string
  description = "Production key for AWS"
  default     = ""
}

Managing variables for your Terraform tooling with Terraform ends up carrying a lot of overhead. Hashicorp has introduced something named variable sets to Terraform Cloud. In Scalr, you can do something similar with shell variables in your account dashboard or in your environment configuration. It's a different way of doing the same thing and works just as well as what I did above.

The Cost

There is a downside to this pattern. If you are just getting started or only need a fast prototype or change, why would you want to go through opening up a PR against your seed workspace and running it through Terraform? It's certainly much faster to create a new workspace and wire it up with the existing VCS connection. The only answer I can provide is consistency. If you are a team of yourself or just two or three, this pattern makes less sense. But once the use of Terraform is growing and you begin to have tiers of access and experience levels with your Terraform usage, perhaps this pattern makes sense.

References

Terraform Module Testing with LocalStack and GitHub Actions

Carlo Mencarelli — Mon, 21 Mar 2022 18:09:44 +0000

Testing your infrastructure as code is just as important as testing your application code. And it doesn’t have to be a nightmare!

TLDR

With a combination of LocalStack and GitHub Actions, you can do easy and effective testing for most of your Terraform code
This method reduces external dependencies on libraries and knowing best practices for programming languages like Ruby and GoLang.
Given this is essentially a mocked AWS service, there are limitations in the service offering.
See demo repo here: mencarellic/terraform-aws-module

When I first saw production Terraform code, it was a single repository split into three directories: one for the production account, one for a development account, and one for the modules. Fast forward a few years, and I’m now with a new company with a similar format, except we use two clouds and have 18 different environments across those clouds.

When I checked earlier, our modules sub-directory was 38,312 lines across 57 modules. Needless to say, we’re in the midst of moving from using a single repo to using a repo per module and leveraging Terraform Cloud’s Private Registry. Now the question is: when I push changes to a module, how do I ensure that module will validate, plan, and apply successfully without too much interruption in the release workflow. The validate and plan are pretty straightforward; I can test it locally with a CLI validate and plan. Applies are trickier, and no one wants to debug Terraform failures while the rest of the team is trying to publish their changes.

There are plenty of patterns, such as using sandbox or testing accounts and Terraform test frameworks like Kitchen or Terratest. But I don’t want to introduce another account and I definitely don’t want to introduce more code to manage. That’s where LocalStack fits in.

LocalStack

LocalStack self-describes in their GitHub repo as: “A fully functional local AWS cloud stack…” It is a feature-rich AWS service that runs in a container with pretty wide coverage. The full list of services can be found here: AWS Service Feature Coverage. The community version contains a reasonable number of services that can cover a lot of different modules, the Pro feature set contains some major parts of the AWS stack (notably API Gateway v2, CloudFront, and RDS).

The setup for Localstack is pretty straight-forward. Pull the docker image and run it. The development team released a CLI tool that helps with orchestration, so what we end up running is:

$ pip install localstack
$ docker pull localstack/localstack
$ localstack start -d

For setting up Terraform to work with LocalStack, the team over there has an integration page with details, though I’ll cover some here as well.

The main thing that you need to know is that you’ll be setting some custom endpoints in your provider configuration to tell Terraform to reach out to localhost instead of AWS for the plan and apply steps.

GitHub Action

The GitHub Action workflow file really only needs to do three things:

Installs Terraform
Installs LocalStack CLI and starts the Docker container (See above)
Runs terraform apply -auto-approve

The workflow file I have does a couple other things based on personal preference, but you can really boil the file down to less than twenty lines if you really want to.

Some of the additional things I do (and why) are:

Ignore the main branch since I don’t want this to run on pushes to that branch. Only feature branches
Use ASDF and a .tool-versions file to manage my Terraform version. You could also use the GitHub Action hashicorp/setup-terraform if desired
Run a terraform plan before the apply to make sure I can see in the logs whether a failure is a plan or apply error

Bringing It All Together

Once you get your GitHub Action workflow file setup, you just need to add a directory where you can place your tests. Inside the directory, you can have a single file where you define your provider and module block.

You’ll need to force the provider to target your Localstack endpoints instead of the live AWS endpoints. You can do that with the endpoints block:

endpoints {
    apigateway     = "http://localhost:4566"
    apigatewayv2   = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    ec2            = "http://localhost:4566"
    es             = "http://localhost:4566"
    elasticache    = "http://localhost:4566"
    firehose        = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    kms            = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    rds            = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    route53        = "http://localhost:4566"
    s3             = "http://s3.localhost.localstack.cloud:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"

After that you can open up some PRs to test your positive and negative test cases.

DEV Community: Carlo Mencarelli

AWS Amplify Through An Infrastructure Lens

The Project

Documentation

The Development Process

The Infrastructure

Security

What I'd Like To See More Of

Security

More Service Tie-Ins

Conclusion

How To Make AWS Budgets Work For You

TLDR

Why Use AWS Budgets?

Creating a Budget

The Budget Amount panel

Budget Scope selection

Configuring Alerts

Configuring Actions

Scaling Budget Creation (Using Terraform)

Wrapping Up

References

Linux’s Cloud Init — Benefits, Quirks, and Drawbacks

Originally created by Canonical for Ubuntu on AWS EC2, it’s now the de facto early boot configuration method

TLDR

How Does Cloud Init Work

Cloud Init Stages

Instance Metadata

Getting Started with Cloud Init

Disabling Cloud Init

Troubleshooting Cloud Init

Logs

Data Files

Configuration Files

Useful Cloud Init Commands to Know

Wrap Up

References

Reducing KMS Costs with S3 Bucket Encryption

A tale of why keeping up in the industry is critical

TLDR

What was making 50,000 KMS requests/second?

Enter S3 Bucket Keys

The Results

This Story is Never Finished

References

Multi-Region S3 Strategies

TLDR

Setting the Stage

New Buckets

Existing Buckets

References

The Case for the Terraform Seed Workspace

Why not use Terraform to manage your Terraform?

TLDR

The Cost

References

Terraform Module Testing with LocalStack and GitHub Actions

TLDR

LocalStack

GitHub Action

Bringing It All Together

References and Docs I Used