DEV Community: Maciej Raszplewicz

Testing in Infrastructure as Code and why Terraform may not be the best option

Maciej Raszplewicz — Thu, 07 Jan 2021 06:44:02 +0000

You don’t need any special tool to automatically test your IaC code, you can use any programming language and unit testing framework you like.

You should test your Infrastructure as Code because it is code - that is probably obvious. Most likely, you want to do it automatically. Here I will share our approach to testing IaC and how it drove our technology decisions for the DevOpsBox platform (https://www.devopsbox.io/).

The Test Pyramid

Let’s start with some theory. There is an old concept of the test pyramid:

There are often different names on the diagram but the concept is:

You should have a lot of fast robust tests, which are often unit tests. It is important they execute in milliseconds and do not depend on any external services.
There should be a moderate number of integration tests. These tests often depend on external services, but are not testing the whole system. They are much slower and more fragile than unit tests, probably executed in seconds or minutes, and can sometimes fail e.g. because of the network problems.
You should only have a few tests of your whole system. They are really slow and fragile. It is sometimes very hard to convince management and business to that, but this is the reality.

It is important to note that nothing will check how code works, as fast as the unit tests do. You should write them for yourself to check your code. However, there are some misconceptions about what the unit is and how to write good tests. In my opinion, you should always think about what you want to achieve with your code and test this behavior - sometimes you will test a single function/method, sometimes something bigger. You shouldn’t call external dependencies in your unit tests and should write your tests in a way that will not require you to change them after a refactoring. Ports and adapters architecture can help with that. If you have good unit tests for a part of your application, you can assume that it works well, like you often assume that some external program works (e.g. AWS CLI). Then, you don’t have to test every variant in your integration or system tests.

Infrastructure as Code tests examples

Often, when we talk about Infrastructure as Code tools, Terraform comes to our minds. It is great for maintaining your infrastructure state and talking to your cloud provider’s API, but you have to write code in HCL, which is not a real programming language. Is it bad? Sometimes yes, especially when it comes to writing unit tests. Let’s check how we can test the Terraform code and what we can use instead, and have real unit tests!

Our example code under test will create an S3 bucket maintaining naming conventions:

Template of the bucket name will be <company name>-<env name>-<app name>-<bucket purpose> (e.g. acme-dev-orders-pictures) if it does not exceed 63 characters (maximum for an s3 bucket)
If it does exceed 63 characters, it will be a hash of the name. We will use a substring of a sha256.

Prerequisites

You will need several tools:

GO (tested with 1.15.3)
Terraform (tested with 0.14.3)
AWS CDK (tested with 1.71.0)
Java JDK (tested with openjdk 11.0.9)
NodeJS (required by AWS CDK, tested with v12.18.3)
Docker (tested with 18.09.5)
cdklocal (https://github.com/localstack/aws-cdk-local, tested with 1.65.2)

An AWS account with proper credentials is also required. The code will probably work with other versions too.

Terratest

We will test this Terraform code https://github.com/devopsbox-io/example-iac-test/blob/master/terraform/s3/main.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

locals {
  requested_bucket_name = "${var.company_name}-${var.env_name}-${var.app_name}-${var.bucket_purpose}"
  bucket_name = length(local.requested_bucket_name) > 63 ? substr(sha256(local.requested_bucket_name), 0, 63) : local.requested_bucket_name
}

resource "aws_s3_bucket" "bucket" {
  bucket = local.bucket_name
}

and variables https://github.com/devopsbox-io/example-iac-test/blob/master/terraform/s3/variables.tf

variable "aws_region" {
  type = string
}

variable "company_name" {
  type = string
}

variable "env_name" {
  type = string
}

variable "app_name" {
  type = string
}

variable "bucket_purpose" {
  type = string
}

Nothing special here, just the implementation of our example in Terraform.

Tests in Terratest are quite easy to write https://github.com/devopsbox-io/example-iac-test/blob/master/test/s3_module_test.go

func TestS3BucketCreated(t *testing.T) {
    t.Parallel()

    envName := strings.ToLower(random.UniqueId())
    awsRegion := "eu-west-1"

    tests := map[string]struct {
        terraformVariables map[string]interface{}
        expectedBucketName string
    }{
        "short name": {
            terraformVariables: map[string]interface{}{
                "aws_region":     awsRegion,
                "company_name":   "acme",
                "env_name":       envName,
                "app_name":       "orders",
                "bucket_purpose": "pictures",
            },
            expectedBucketName: "acme-" + envName + "-orders-pictures",
        },
        "long name": {
            terraformVariables: map[string]interface{}{
                "aws_region":     awsRegion,
                "company_name":   "acme",
                "env_name":       envName,
                "app_name":       "orders",
                "bucket_purpose": "pictures12345678901234567890123456789012345678901234567890",
            },
            expectedBucketName: sha256String("acme-" + envName + "-orders-pictures12345678901234567890123456789012345678901234567890")[:63],
        },
    }

    for name, testCase := range tests {
        // capture range variables
        name := name
        testCase := testCase
        t.Run(name, func(t *testing.T) {
            t.Parallel()

            terraformModuleDir, err := files.CopyTerraformFolderToTemp("../terraform/s3", "terratest-")
            if err != nil {
                t.Fatalf("Error while creating temp dir %v", err)
            }
            defer os.RemoveAll(terraformModuleDir)

            terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
                TerraformDir: terraformModuleDir,
                Vars:         testCase.terraformVariables,
            })

            defer terraform.Destroy(t, terraformOptions)

            terraform.InitAndApply(t, terraformOptions)

            aws.AssertS3BucketExists(t, awsRegion, testCase.expectedBucketName)
        })
    }
}

func sha256String(str string) string {
    sha256Bytes := sha256.Sum256([]byte(str))
    return hex.EncodeToString(sha256Bytes[:])
}

I have used a pattern called data-driven (or table-driven) tests (read more here https://dave.cheney.net/2019/05/07/prefer-table-driven-tests). Tests are executed in parallel, what is quite nice. To do that you have to copy your module using CopyTerraformFolderToTemp and reassign range variables (two lines after the // capture range variables comment).

Execute this code with:

cd test
go test

There are a few problems here:

These are not unit tests. They need to create your infrastructure, so they are slow and fragile.
Look at this simple algorithm in HCL and think of something a little bit more complicated - it can be really hard to implement, read, and maintain.
You have to use Golang - it is no longer an issue for us, but it used to be.

You can’t use Terratest assertions with LocalStack and there is an open pull request https://github.com/gruntwork-io/terratest/pull/495 solving this issue. Of course, you could write your own assertions and we will do it later in a different language, so let’s skip this now and move on.

Test Terraform with custom “miniframework” based on Spock

We didn’t want to write tests in Golang, so we decided to check if it is possible to use some other language. One of my favorite testing frameworks is Spock - it is JVM based and you write your tests in Groovy. We decided to check it and it turns out that it works very well! You “only” have to write some glue code, what is not that hard to do.

The Terraform code is almost the same, but with LocalStack support, we had to change AWS provider configuration https://github.com/devopsbox-io/example-iac-test/blob/master/terraform/s3/main.tf

provider "aws" {
  region = var.aws_region

  access_key = var.use_localstack ? "fake_access_key" : null
  s3_force_path_style = var.use_localstack
  secret_key = var.use_localstack ? "fake_secret_key" : null
  skip_credentials_validation = var.use_localstack
  skip_metadata_api_check = var.use_localstack
  skip_requesting_account_id = var.use_localstack

  dynamic "endpoints" {
    for_each = var.use_localstack ? [1] : []
    content {
      s3 = "http://localhost:4566"
    }
  }
}

and add a single additional variable:

variable "use_localstack" {
  type = bool
  default = false
}

The test looks like this https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/s3/S3TerraformModuleTest.groovy

class S3TerraformModuleTest extends TerraformIntegrationTest {

    @Shared
    S3 s3

    def setupSpec() {
        s3 = new S3(sdkClients)
    }

    @Unroll
    def "should create s3 bucket #testCase"() {
        given:
        def terraformVariables = new S3TerraformModuleVariables(
                useLocalstack: localstack.enabled,
                awsRegion: awsRegion(),
                companyName: "acme",
                envName: environmentName(),
                appName: "orders",
                bucketPurpose: bucketPurpose,
        )

        when:
        deployTerraformModule("terraform/s3", terraformVariables)

        then:
        s3.checkBucketExists(expectedBucketName)

        cleanup:
        destroyTerraformModule("terraform/s3", terraformVariables)

        where:
        testCase     | bucketPurpose                                                | expectedBucketName
        "short name" | "pictures"                                                   | "acme-" + environmentName() + "-orders-pictures"
        "long name"  | "pictures12345678901234567890123456789012345678901234567890" | DigestUtils.sha256Hex("acme-" + environmentName() + "-orders-pictures12345678901234567890123456789012345678901234567890").substring(0, 63)
    }

    class S3TerraformModuleVariables extends TerraformVariables {
        boolean useLocalstack
        String awsRegion
        String companyName
        String envName
        String appName
        String bucketPurpose
    }
}

The code is quite nice. Notice S3TerraformModuleVariables class - not that bad way to pass input variables to our Terraform stack. One of the problems is the lack of parallelism support (will be supported in Spock 2.0 http://spockframework.org/spock/docs/2.0-M4/parallel_execution.html#parallel-execution). We already have LocalStack support here - long-running tests should be faster because you don’t have to wait for the real infrastructure. However, I think that you should also run your tests with the real cloud - sometimes LocalStack can behave differently, or maybe does not support some cloud resource you need.

To execute this code run:

./gradlew integTest --tests *S3TerraformModuleTest

or with LocalStack:

./gradlew integTest --tests *S3TerraformModuleTest -Dlocalstack.enabled=true

Are there any problems here? Yes:

Still no unit tests
Still not a real programming language

The “miniframework” glue code

Have I mentioned the glue code? There is some, but really not that much. I will only list all the files, describe them but not paste the whole code here:

https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/ProcessRunner.java responsible for executing external processes
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/Localstack.groovy supports execution in LocalStack
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/aws/SdkClientProvider.java creates AWS clients to use in assertions
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/aws/S3.java contains S3 checks - here only checkBucketExists, use this code in your test’s assertions, add more checks when necessary
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/terraform/Terraform.java Terraform wrapper - uses the ProcessRunner class to execute an external Terraform process
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/terraform/TerraformVariables.groovy base class for the Terraform variables
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/terraform/TerraformVariablesMarshaller.groovy creates JSON files from the Terraform variables objects
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/terraform/TerraformIntegrationTest.groovy base class for all your Terraform tests
https://github.com/devopsbox-io/example-iac-test/blob/master/build.gradle the Gradle configuration prepared to run the tests

It is a mix of Groovy and Java. You can copy this code, use it in your project or maybe even rewrite it into another language. If you think we should create a real framework and provide a jar - please let me know, we will do our best.

As you can see - you don’t need any special tool to test your Infrastructure as Code, only your favorite language, unit testing framework, and a few hours to write some glue code.

Test AWS CDK with custom “miniframework” based on Spock

We really wanted to write our code in a general-purpose programming language and had the ability to unit test it. After some research, we found two frameworks:

AWS CDK
Pulumi

Only AWS CDK supported Java, so it was our choice, although I have to write a few words about Pulumi: it is really cool, I used it in one of my projects using TypeScript and I was impressed. Coming back to AWS CDK - when we started to write the code, CDK was in its early stages and AWS changed its API very often, but that is a completely different story…

Here is the AWS CDK code for which we will write tests https://github.com/devopsbox-io/example-iac-test/blob/master/src/main/java/io/devopsbox/infrastructure/test/s3/S3Construct.java

public class S3Construct extends Construct {

    public S3Construct(Construct scope, String id, S3ConstructProps props) {
        super(scope, id);

        String bucketName = props.getBucketName();
        new Bucket(this, bucketName, BucketProps.builder()
                .removalPolicy(RemovalPolicy.DESTROY)
                .bucketName(bucketName)
                .build());
    }
}

we moved our bucket naming logic to another class https://github.com/devopsbox-io/example-iac-test/blob/master/src/main/java/io/devopsbox/infrastructure/test/s3/S3ConstructProps.java

public class S3ConstructProps extends ConstructProps {
    public static final int BUCKET_NAME_MAX_LENGTH = 63;

    private final String bucketPurpose;

    public S3ConstructProps(String companyName, String envName, String appName, String bucketPurpose) {
        super(companyName, envName, appName);
        this.bucketPurpose = bucketPurpose;
    }

    public String getBucketPurpose() {
        return bucketPurpose;
    }

    public String getBucketName() {
        String bucketName = getCompanyName() + "-" + getEnvName() + "-" + getAppName() + "-" + getBucketPurpose();
        if (bucketName.length() > BUCKET_NAME_MAX_LENGTH) {
            bucketName = DigestUtils.sha256Hex(bucketName).substring(0, BUCKET_NAME_MAX_LENGTH);
        }
        return bucketName;
    }
}

There is also a base class for all construct property classes with a set of common properties https://github.com/devopsbox-io/example-iac-test/blob/master/src/main/java/io/devopsbox/infrastructure/test/ConstructProps.java

public class ConstructProps implements Serializable {
    private final String companyName;
    private final String envName;
    private final String appName;

    public ConstructProps(String companyName, String envName, String appName) {
        this.companyName = companyName;
        this.envName = envName;
        this.appName = appName;
    }

    public String getCompanyName() {
        return companyName;
    }

    public String getEnvName() {
        return envName;
    }

    public String getAppName() {
        return appName;
    }
}

And two standard CDK classes:

https://github.com/devopsbox-io/example-iac-test/blob/master/src/main/java/io/devopsbox/infrastructure/test/IacTestApp.java this is the main class, our entry point
https://github.com/devopsbox-io/example-iac-test/blob/master/src/main/java/io/devopsbox/infrastructure/test/IacTestStack.java CDK stack, we create our construct here

class S3CdkConstructTest extends CdkIntegrationTest {

    @Shared
    S3 s3

    def setupSpec() {
        s3 = new S3(sdkClients)
    }

    def "should create s3 bucket"() {
        given:
        def stackId = "S3BucketConstructTest" + environmentName()
        def constructProps = new S3ConstructProps(
                "acme",
                environmentName(),
                "orders",
                "pictures"
        )

        when:
        deployCdkConstruct(stackId, S3Construct, constructProps)

        then:
        s3.checkBucketExists("acme-" + environmentName() + "-orders-pictures")

        cleanup:
        destroyCdkConstruct(stackId, S3Construct, constructProps)
    }
}

To execute this code run:

./gradlew integTest --tests *S3CdkConstructTest

or with LocalStack:

./gradlew integTest --tests *S3CdkConstructTest -Dlocalstack.enabled=true

We are not testing all the cases here, just a single integration test, because we can finally write unit tests! The code looks like this https://github.com/devopsbox-io/example-iac-test/blob/master/src/test/groovy/io/devopsbox/infrastructure/test/s3/S3ConstructPropsTest.groovy

class S3ConstructPropsTest extends Specification {

    @Unroll
    def "should return s3 bucket #testCase"() {
        given:
        def props = new S3ConstructProps(
                "acme",
                "dev",
                "orders",
                bucketPurpose,
        )

        when:
        def bucketName = props.bucketName

        then:
        bucketName == expectedBucketName

        where:
        testCase     | bucketPurpose                                                | expectedBucketName
        "short name" | "pictures"                                                   | "acme-dev-orders-pictures"
        "long name"  | "pictures12345678901234567890123456789012345678901234567890" | DigestUtils.sha256Hex("acme-dev-orders-pictures12345678901234567890123456789012345678901234567890").substring(0, 63)
    }
}

Just run it with ./gradlew test and it completes in milliseconds!

Finally, we can write the code in a real programming language of our choice and create unit tests. Is it perfect? Certainly not. Our integration tests run in a different process, so there are some drawbacks. There is support for running in the same process in Pulumi (https://github.com/pulumi/pulumi/issues/3901) but not in AWS CDK yet (https://github.com/aws/aws-cdk/issues/601). We can also improve our “miniframework” and run tests using a chosen IAM role - we do that in DevOpsBox already, but it is not included here for the sake of simplicity.

More “miniframework” glue code…

We have to add a few files to our “miniframework” to support AWS CDK:

https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/cdk/Cdk.java uses the ProcessRunner class to execute an external AWS CDK process; note the debugging support - when you set the CDK_DEBUG_ENABLED environment variable to true, it will start gradle with “ --debug-jvm” flag and wait for the remote debugger to connect
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/cdk/PropsFileSerializer.groovy responsible for properties file serialization and deserialization
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/cdk/CdkIntegrationTestStack.groovy CDK stack class used only in tests
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/cdk/CdkIntegrationTestMain.groovy main CDK class (entry point) used only in tests
https://github.com/devopsbox-io/example-iac-test/blob/master/src/integTest/groovy/io/devopsbox/infrastructure/test/common/cdk/CdkIntegrationTest.groovy base class for all your AWS CDK tests
https://github.com/devopsbox-io/example-iac-test/blob/master/build.gradle modified Gradle configuration, startCdkTest task added

A little bit less than for Terraform, because we can reuse some classes written before.

Conclusion

The ability to write unit tests was one of the key factors behind choosing AWS CDK as our Infrastructure as Code tool, and after choosing it we found it very convenient to write IaC code in a programming language of our choice, have good code structure, do refactorings, use design patterns, have a great IDE support, use external libraries and much more. It’s great that nowadays we can write IaC in a general-purpose programming language and it is still declarative.

I hope that the terraform-cdk project (https://github.com/hashicorp/terraform-cdk) will be usable soon and maintained in the future. Then, we will be able to have a cake and eat it too!

For more details about the DevOpsBox platform please visit https://www.devopsbox.io/

Golang through the eyes of a Java developer - pros and cons

Maciej Raszplewicz — Wed, 16 Dec 2020 07:00:41 +0000

Recently I had to learn the Go programming language and now I want to share my thoughts from the perspective of a Java developer.

We decided to create Kubernetes operators for the DevOpsBox (https://www.devopsbox.io/) platform (read more about our reasons: https://dev.to/mraszplewicz/my-perfect-aws-and-kubernetes-role-based-access-control-and-the-reality-48fb). It turns out that it is easiest to create them in Golang - there are Kubebuilder and Operator SDK frameworks. The only thing is that we didn't have Golang skills, so I had to learn a new language...

I will start with things I like and move to those I don't. I will try to focus on the language itself.

Things I like

Easy to learn

It is amazing how easy it is to learn the Golang. A Tour of Go (https://tour.golang.org/) covers almost every aspect of the language and the language specification (https://golang.org/ref/spec) is reasonably short and readable. There are some not-so-easy-to-understand features like channels, but they are powerful and there is a reason why they exist.

Before version 5, Java didn't have so many features either, but it has never been as simple as Golang is now.

A fast developer feedback loop

By using the term "developer feedback loop", I mean the time from starting the program to seeing its results.

Sometimes you expect that you have to wait a long time to start a program written in a compiled language. Go main or unit tests start almost instantly when run from the IDE, so the feedback loop can be very short. Results are similar to those in other modern programming languages and even though Go is statically compiled, you don't have to wait long for the compilation process.

It is funny that nowadays we have to build programs written in languages that are by design not statically compiled (e.g. JavaScript) and sometimes wait for the build process to complete.

Static type checking

Go has a very good static type checking system. You don't even have to declare a type of every variable, you simply write:

str := "Hello"

and it knows that variable str is of type string. The same is true when the variable is a result of a function call:

result := someFunction()

Implicit interface implementation

"If it walks like a duck and it quacks like a duck, then it must be a duck."
It means that you don't have to explicitly declare that you are implementing an interface. In Go you can write:

type Duck interface {
    Quack()
}

type Mallard struct {
}

func(mallard *Mallard) Quack() {
}

Mallard is a duck because it can quack! You can write:

func main() {
    var duck Duck
    var mallard *Mallard

    mallard = &Mallard{}
    duck = mallard

    duck.Quack()
}

In Go, you can create your own interfaces even for the existing external code.

Multiple return values

This one is simple - you can just return multiple values from a function:

func Pair() (x, y int) {
    return 1, 2
}

In most languages I know, you will have to create a class or struct to achieve something similar.

But this feature is also used to return errors from functions - more about it in "Things I don't like".

Function values

Functions can be passed as parameters, assigned to variables, etc.:

func doJob(convert func(int) string)  {
    i := 1
    fmt.Print(convert(i))
}

func main() {
    convert := func(intVal int) string {
        return strconv.Itoa(intVal)
    }
    doJob(convert)
}

It is somehow similar to method reference or lambda expressions in Java, but more "built-in" into the language.

Modules

Go has a really good built-in dependency management system. You don't need Gradle or Maven to download dependencies, you just use Go modules.

Unit tests

Unit test support is a part of the language itself. You just create a file with the "_test.go" suffix and write your tests. For example, for hello.go:

package hello

func sayHello() string {
    return "Hello world!"
}

you create hello_test.go file:

package hello

import "testing"

func TestSayHello(t *testing.T) {
    greetings := sayHello()

    if greetings != "Hello world!" {
        t.Errorf("Greeting is different than expected!")
    }
}

and you can just run it from your IDE or in CI/CD pipeline.

If you want to write good tests in Golang, you should probably write "table-driven tests". A good article on this topic: https://dave.cheney.net/2019/05/07/prefer-table-driven-tests

Defer

Go provides something similar to Java's finally keyword but, in my opinion, more powerful and simpler. If you want to run some code when your function returns, for example, to clean up some resources, you use the defer keyword:

func writeHello() {
    file, err := ioutil.TempFile(os.TempDir(), "hello-file")
    if err != nil {
        panic(err)
    }
    defer os.Remove(file.Name())

    file.WriteString("Hello world!")
}

Here we create a temporary file that will be removed at the end of the function execution, just after writing "Hello world!".

Single binary

Go is famous for producing a single binary. When you build your program, you will get a single executable file containing all the dependencies. Of course, you have to prepare a separate binary for every target platform, but the distribution of your program is simpler compared to other languages. This is one of the reasons why Go is often used for creating command-line utilities.

Cobra library

https://github.com/spf13/cobra is the second reason... It is an extremely useful library, which helps to write command-line tools. Having created our operators in DevOpsBox, we also wanted to have our own CLI and it was a pleasure to write them using Cobra.

Things I don't like

Nothing is perfect and Go is no exception, it has some features that I don't like that much...

Error handling

I really don't like Go idiomatic error handling. There are a few main reasons why:

Error handling makes the code less readable

Often in Go programs, you see something like this:

func doJob() ([]string, error) {
    result1, err := doFirstTask()
    if err != nil {
        log.Error(err, "Error while doing the first task")
        return nil, err
    }

    result2, err := doSecondTask()
    if err != nil {
        log.Error(err, "Error while doing the second task")
        return nil, err
    }

    result3, err := doThirdTask()
    if err != nil {
        log.Error(err, "Error while doing the third task")
        return nil, err
    }

    return []string{
        result1,
        result2,
        result3,
    }, nil
}

I think that error handling adds a lot of noise here. Without it, this function would look like this:

func doJob() []string {
    result1 := doFirstTask()
    result2 := doSecondTask()
    result3 := doThirdTask()

    return []string {
        result1,
        result2,
        result3,
    }
}

Java, also, has some issues - namely checked exceptions, which add a lot of unnecessary noise. Thankfully, there are frameworks like Spring Framework, which wraps checked exceptions into runtime exceptions, so you can catch only those that you expect.

You can forget about handling an error

Consider this code:

func doJob() ([]string, error) {
    result1, err := doFirstTask()
    if err != nil {
        log.Error(err, "Error while doing the first task")
        return nil, err
    }

    result2, err := doSecondTask()

    result3, err := doThirdTask()
    if err != nil {
        log.Error(err, "Error while doing the third task")
        return nil, err
    }

    return []string {
        result1,
        result2,
        result3,
    }, nil
}

What's wrong with it? I forgot to handle an error! In a little bit more complicated cases, it is possible to miss it while doing a code review.

In other programming languages, you would have centralized exception handling and stack traces available for debugging purposes.

You don't have your stack trace

I have read some articles about error handling in Golang and there are opinions that stack traces are "unreadable, cryptic", but I got used to them and for me, it is easy to find a problem with help of a stack trace.

Problems with refactoring

IDEs have some issues with refactoring Golang code, for example, when extracting a method or a variable. I think that these problems are related to idiomatic error handling.

Sometimes too brief

Sometimes I feel that Golang is too brief. Why do we have keywords like func, not function? Why are we not forced to use surrounding parentheses in if or for? Why don't we have any keyword like public and we have to declare upper case instead? But it is probably only my personal opinion...

Sometimes inconsistent

Go does not support function/method overloading (https://golang.org/doc/faq#overloading). I am ok with that because there are many programming languages without it, but why does the built-in make function have many variants? Looking at the documentation:

Call             Type T     Result

make(T, n)       slice      slice of type T with length n and capacity n
make(T, n, m)    slice      slice of type T with length n and capacity m

make(T)          map        map of type T
make(T, n)       map        map of type T with initial space for approximately n elements

make(T)          channel    unbuffered channel of type T
make(T, n)       channel    buffered channel of type T, buffer size n

Features hard to remember

It is probably the case of any programming language, but Go is so simple that I thought I would easily remember how to use all the keywords and all the built-in functions. Unfortunately, the reality is quite different. After a year of using the language on a daily basis, I still need to look into the documentation or check examples to find how to use channels, or how to use make. Maybe I have too many different programming languages in my mind...

No streaming API equivalent

There are some features in other languages that let you interact with collections using declarative syntax, like streaming API in Java. On the other hand, in Golang you won’t find anything similar (or maybe there is some library?) and it is idiomatic to use for loops. Loops aren't that bad, but I like the streaming API and got used to it.

Not that good IDE support

I love JetBrains products and I use GoLand to write Go code. I have already mentioned issues with refactoring and here is an example. If you want to extract a method from this code:

func doJob() ([]string, error) {
    result1, err := doFirstTask()
    if err != nil {
        log.Error(err, "Error while doing the first task")
        return nil, err
    }

    result2, err := doSecondTask()
    if err != nil {
        log.Error(err, "Error while doing the second task")
        return nil, err
    }

    result3, err := doThirdTask()
    if err != nil {
        log.Error(err, "Error while doing the third task")
        return nil, err
    }

    return []string {
        result1,
        result2,
        result3,
    }, nil
}

GoLand will do it like this:

func doJob() ([]string, error) {
    result1, err := doFirstTask()
    if err != nil {
        log.Error(err, "Error while doing the first task")
        return nil, err
    }

    result2, result3, strings, err2 := doThirdAndSecond(err)
    if err2 != nil {
        return strings, err2
    }

    return []string{
        result1,
        result2,
        result3,
    }, nil
}

func doThirdAndSecond(err error) (string, string, []string, error) {
    result2, err := doSecondTask()
    if err != nil {
        log.Error(err, "Error while doing the second task")
        return "", "", nil, err
    }

    result3, err := doThirdTask()
    if err != nil {
        log.Error(err, "Error while doing the third task")
        return "", "", nil, err
    }

    return result2, result3, nil, nil
}

which is not that bad, but requires additional manual fixes. IntelliJ does a better job for Java!

Conclusion

Although there are a lot of pros of Golang, cons are significant and therefore I have mixed feelings about the language. I can say that I like it, but it will probably never be my favorite. So which one is? C#, but that is a completely different story… I think it is a personal matter who likes which programming language, and it is all for good!

My perfect AWS and Kubernetes role-based access control and the reality

Maciej Raszplewicz — Mon, 30 Nov 2020 06:13:05 +0000

...or our problems with IAM, Kubernetes RBAC, and how we tried to solve them.

When I wanted to create the role-based access control for DevOpsBox I had a clear vision of what I want to achieve and... I had a clash with the reality of AWS IAM, Kubernetes RBAC, and to a lesser extent with Keycloak. In this article, I want to describe some of my problems and how we solved (or worked around) them.

For those of you who don't have enough time to read the whole story I will list all the problems we encountered:

Kubernetes/EKS RBAC
- It is not possible to provide a resource name for the create (or deletecollection) operation.
- You cannot use any regexp when using a resource name, you can only use the exact name.
- When writing a Kubernetes operator, you do not have any information in your controller code about who created the resource.
- You must add read/write permissions for all the secrets in a namespace if you want to use Helm 3 default release information store (HELM_DRIVER=secret).
- In EKS, if you grant permission to modify any configmap (cluster role without any resource name), it means you also allow to modify the aws-auth configmap in the kube-system namespace, so the user can add himself to the "system:masters" group.
AWS IAM
- With IAM you grant permissions to the AWS API, so it is too fragmented and the policies are getting very big and hard to write and maintain.
- Not all the resources support ABAC (permissions based on tags).
- There are a lot of operations that do not allow restricting permissions to resource names.
- There is a limit for the policy length and the number of policies assigned to an IAM Role.
Keycloak
- You cannot filter roles in the UI when you are assigning them to a user or a group.

Our dreams

These were my requirements:

I wanted our users to have the possibility to grant permissions per combination of an application (or microservice) and its environment.
Some permissions are non-environment-specific (e.g. build) - it should be possible to grant them separately.
It should be possible to have different permissions for deployment with resources (cloud or containers) creation and one that only changes the application's version. This separation allows us to have "power users", who know what they do when they change cloud resources and are also responsible for controlling cloud costs.
The deployment through CI/CD should use the user's cloud and Kubernetes permissions (this is the hardest one...).
A user should not have permissions to anything more than what he has been granted. For example, if he can create the deployment Kubernetes resource for one application, it does not mean that he can do it for another one or in a different environment.
We wanted to have namespace per environment in our Kubernetes clusters. This was our assumption before we started to implement access control.
We should have separate permissions for viewing application data (including logs). This one should help to be GDPR compliant.
Of course, everything should be as clear and simple as possible.

After discussions, we ended up with these permissions’ list (the list is for a hypothetical application named "Orders", which has dev and prod environments):

Application	Environment	Permission	Description
Orders	-	build	non-environment-specific
Orders	dev	fulldeployment	allows creating all cloud resources for the Orders application on the dev environment
Orders	dev	appdeployment	allows only deploying a new version of the Orders application to the dev environment
Orders	dev	stop	allows stopping the Orders application on the dev environment
Orders	dev	logs	allows viewing and querying logs of the Orders application on the dev environment
Orders	dev	monitoring	allows viewing various cloud and Kubernetes resources of the Orders application on the dev environment
Orders	prod	fulldeployment	allows creating all cloud resources for the Orders application on the prod environment
Orders	prod	appdeployment	allows only deploying a new version of the Orders application to the prod environment
Orders	prod	stop	allows stopping the Orders application on the prod environment
Orders	prod	logs	allows viewing and querying logs of the Orders application on the prod environment
Orders	prod	monitoring	allows viewing various cloud and Kubernetes resources of the Orders application on the prod environment

It can be extended in the future

The full deployment process through CI/CD should look like this:

User "clicks" deploy on an environment of some application and the process starts.
The deployment pod starts and the following steps are run in this pod (does not run on CI/CD server and allows us to have better scalability).
Cloud resources are created using the user's permissions (IAM policy).
The pre-deployment pod is executed and runs to the end. It can be used for example to run the database migration. It should also be run on the user's permissions (Kubernetes RBAC).
Kubernetes resources should be created. Here we create lots of resources, a few of them are deployment, service, horizontal pod autoscaler, Istio virtual service. Of course, this step should use user's permissions (Kubernetes RBAC)
The post-deployment pod is executed and runs to the end. You can use it to do some configuration after the application deployment. Again, it should run on the user's permissions.

We thought that our ideas were great, and then we woke up...

The reality

The problem is that almost all of my requirements were impossible to implement. I will go through our "perfect process" and show you how we implemented it or changed our requirements. Then, I will write about AWS and Kubernetes users' permissions. There is also one problem with Keycloak, which I will also describe.

Run the deployment in CI/CD

Here everything is almost as it should be.

The "only" problem is that it is not easy to run the pipeline on the user's credentials. It runs on CI/CD server credentials (Kubernetes service account), but we know who started the process, and we can allow CI/CD to impersonate the user. Everything is autogenerated, and we do not allow users to modify the pipelines, so this should be secure enough.

Starting the deployment pod

So we have impersonated our user, and we want to start the deployment pod on his/her permissions. Here we are, actually, allowing our user to create any pod in some namespace. It is impossible to restrict the resource name for the create verb! And even if it was possible you can't use any kind of regexp, or something in the resource name field (https://github.com/kubernetes/kubernetes/issues/56582). Permission to start any pod will allow you to read any secret in this namespace, assign any service account, and execute lots of not so secure things...

After some research, we decided to create our own Kubernetes operators (one for the build and one for the deployment). It turns out that it is not that hard, but it is easiest to do it in Golang, and we had no skills in this language. Even so, we decided to go for it. We tried both operator-sdk and kubebuilder, and we chose kubebuilder, but we are using our Helm chart instead of kustomize (which is the default).

We have defined the following custom resources:

BoxFullDeployment,
BoxAppDeployment,
BoxFullUndeployment,
BoxStart,
BoxStop,
BoxBuild

We will probably extend the list in the future.

Is having our own operator enough to allow us to restrict permissions using resource names? No, it is not! You can't restrict it for the create verb! But at least, we do not allow the creation of any pod...

If we cannot use resource names, then what can we do? Split Kubernetes namespaces per application's environment. For our hypothetical Orders application we will have the following namespaces:
dev-orders
prod-orders
It was quite a hard decision to have so fragmented namespaces, but it turns out that it is not that bad. The only drawback is the less convenient use of kubectl.

One more problem - who and when creates these namespaces? We run the namespace creation in a CronJob. We also create roles and role bindings in Kubernetes based on Keycloak groups and their permissions in the same CrobJob.

Cloud resources are created from the deployment pod using users’ credentials

We had lots of problems here. First of all, we have no bloody idea who has created a resource in Kubernetes (https://github.com/kubernetes/kubernetes/issues/13306), so we gave up the idea of running everything on the user's permissions. Luckily, after consideration, it should be enough to have operators and grant permissions to our custom resources.

Ok, but if we are creating cloud resources from the deployment pod, should the Kubernetes node (and any application running on it) be allowed to create those resources? Here we can use IAM roles for service accounts, so our deployment pod has a specific IAM role assigned (I will write a separate article about EKS and IAM integration and paste a link here when it is ready).

The pre-deployment pod starts on the user's permissions

Same problems as when creating cloud resources, but related to Kubernetes. The solution was almost the same. The only difference is that we are granting Kubernetes RBAC permissions to the deployment's pod service account.

Kubernetes resources should be created.

I noticed one more problem here. We use Helm 3 to manage resource creation/deletion. We do not use the templating feature as we generate ready to use yamls. The problem with Helm is that it stores its metadata in secrets, and their names change on every deployment/upgrade. It means that if you want to grant the user to do only "helm ls" you have to grant him/her to read all the secrets in the namespace! I have reported an issue on Helm's GitHub (https://github.com/helm/helm/issues/8122), but maybe my description was not clear enough and they closed it. Luckily, you can use HELM_DRIVER=configmap, it is much more secure to let users read all the configmaps than the secrets... but what if you use a third-party Helm chart and you don't know if it contains any sensitive data? We use configmap for our charts (those generated) and secrets for the third party charts.

The post-deployment

The same as pre-deployment

AWS user permissions

We gave up using user's permissions for the deployment process, but we still wanted to grant AWS permissions for the Console, CLI, or API usage. Permissions should still be granted per the combination of an application and its environment. We have written a lambda function that generates an IAM role for every Keycloak group using AWS CDK (My article about running CDK from Lambda: https://dev.to/mraszplewicz/running-aws-cdk-from-a-lambda-function-3502).

First, we looked at ABAC - it is really promising and could solve our problems in quite an elegant way. However, there are a few issues with ABAC:

Not all AWS Services supports ABAC. S3 does not have the full support to name one (look at "Authorization based on tags" in https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html). So we decided to use tags for services that support them, and resource names for services that do not support tags.
Even if a Service supports ABAC, there are still some operations that cannot be restricted by tag (or even a resource name). For example, if you do not have rds:DescribeDBInstances for all RDS instances, you cannot view any instance on the list in the AWS Console. We decided to carefully grant permissions for all resources when it is not too dangerous.
The condition language is quite limited. If you want to grant permissions for the application named Orders on dev and staging environments and the application Customers only on the dev environment, you will have to repeat the whole policy statement e.g.:

{
    "Condition": {
        "StringEquals": {
            "aws:ResourceTag/BoxAppName": [
                "Orders",
                "Customers"
            ],
            "aws:ResourceTag/BoxEnvironment": "dev"
        }
    },
    "Action": [
        "rds:Describe*",
        "rds:List*"
    ],
    "Resource": "*",
    "Effect": "Allow"
},
{
    "Condition": {
        "StringEquals": {
            "aws:ResourceTag/BoxAppName": [
                "Orders"
            ],
            "aws:ResourceTag/BoxEnvironment": "staging"
        }
    },
    "Action": [
        "rds:Describe*",
        "rds:List*"
    ],
    "Resource": "*",
    "Effect": "Allow"
}

There are more issues related to IaM:

The fragmentation of IAM permissions is very high (per every API operation), so it is quite hard to write policies, and they are getting really big. When writing the policies, we used two techniques: checking in Cloud trail (but you have to wait a few minutes for the data) and checking HTTP requests in chrome or firefox dev console. It is often quite easy to read what the problem was from the HTTP path, request, or response body. TIP: not all access denied responses have the status 403, some of them have even 200 and you have to analyze them more deeply.
You can hit the limit when policies are even bigger. You can read here https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html that: "Role policy size cannot exceed 10,240 characters", "The size of each managed policy cannot exceed 6,144 characters.", "You can add up to 10 managed policies to an IAM user, role, or group." and "IAM does not count white space when calculating the size of a policy against these quotas.". We haven't solved the problem yet, but when we hit the limit, we will write some code to split our policies into smaller chunks. Luckily, our policy generator is written in Java, so it shouldn't be that hard to write that code and unit test it.
You will need some naming convention and a mechanism to avoid collisions when using resource names. For example, our policy for s3 can look like this:

{
    "Action": [
        "s3:Get*",
        "s3:List*"
    ],
    "Resource": [
        "arn:aws:s3:::dev-orders-*",
        "arn:aws:s3:::dev-customers-*",
        "arn:aws:s3:::staging-orders-*"
    ],
    "Effect": "Allow"
}

And we are aware that somebody can name his application "orders-whatever" and all the users having permissions for orders will see his buckets. ABAC is much better... We will write some code to notify our users that such a collision exists and it is not allowed.

Kubernetes permissions

After switching to namespace per application and environment approach, it wasn't that hard to create the solution for Kubernetes. We have a CronJob in each cluster, which reads Keycloak groups, adds mappings to aws-auth config map in kube-system namespace, and then, creates appropriate roles and role bindings based on roles granted to Keycloak groups. There is one thing worth mentioning here:
In EKS, if you grant permission to modify any configmap (cluster role without any resource name), it means you also allow to modify the aws-auth configmap in the kube-system namespace, so the user can add himself to the "system:masters" group.

Keycloak

We use Keycloak for our platform tools SSO (but it is not required for applications deployed on our platform). We very like the way it is designed, there are roles, groups, clients, composite roles, etc. But when we started to create our solution, we thought that we will have realm roles for each application/environment. The list will grow quite fast and it is not possible to filter roles when assigning to a group or user. We decided to create a Keycloak client per application to group its roles, so the list is manageable, and it is also possible to filter client names.

Conclusion

Here is our final deployment process:

As you can see, we use a combination of multiple techniques to achieve almost what we wanted. If you think that I am wrong somewhere in this article all know how to do it simpler, please leave a comment or contact me on LinkedIn (https://www.linkedin.com/in/maciejraszplewicz/).

For more details about the DevOpsBox platform please visit https://www.devopsbox.io/

Running AWS CDK from a Lambda function

Maciej Raszplewicz — Thu, 12 Nov 2020 09:57:18 +0000

This is a step by step guide to running AWS CDK inside an AWS Lambda Function using Lambda layers.

We will create an example CDK code, which will create an S3 bucket (can be any AWS resource, this is just an example) and then create another CDK code to deploy a Lambda function which will run the first one. Do you remember the Russian toy called "matryoshka"?

Why do we need this? There are two main reasons:

For fun, to play with Lambda layers and AWS CDK.
It can be sometimes useful and we actually use it in DevOpsBox.

All the source code is available here: https://github.com/devopsbox-io/example-cdk-from-lambda

Prerequisites

You will need several tools:

AWS CDK (tested with 1.71.0)
Java JDK (tested with OpenJDK 11.0.9)
NodeJS (required by AWS CDK, tested with v12.18.3)
Docker (tested with 18.09.5)
Maven (tested with 3.6.3)
AWS CLI (only for testing, tested with 1.18.57)

AWS account with proper credentials is also required. The code will probably work with other versions too.

The solution

First of all, we will create two CDK projects, one inside another (matryoshka):

mkdir example-cdk-from-lambda
cd example-cdk-from-lambda
cdk init app --language=java

mkdir run-cdk-lambda
cd run-cdk-lambda
cdk init app --language=java

Yes, we are using Java here... You can achieve the same results in any other language supported by AWS CDK.

I will not change any package name just to allow you to easily do git diff HEAD~1 and to see all my changes, however, I have removed all the tests and test dependencies - they are not important here.

The smallest matryoshka

Here we just create an S3 bucket but it could be any CDK code.

We need the CDK S3 dependency run-cdk-lambda/pom.xml

<dependency>
    <groupId>software.amazon.awscdk</groupId>
    <artifactId>s3</artifactId>
    <version>${cdk.version}</version>
</dependency>

Next, we have to write some CDK code run-cdk-lambda/src/main/java/com/myorg/RunCdkLambdaStack.java:

new Bucket(this, "created-by-cdk-from-lambda", BucketProps.builder()
    .removalPolicy(RemovalPolicy.DESTROY)
    .build());

The medium matryoshka

If you want to run the AWS CDK, you will have to execute the cdk binary because this is how it works. That is why we need to create a Lambda handler, which will be just a wrapper and will run the cdk.

We have to add the aws-lambda-java-core dependency to our run-cdk-lambda/pom.xml:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-lambda-java-core</artifactId>
    <version>1.2.1</version>
</dependency>

Then, create the CdkWrapper class run-cdk-lambda/src/main/java/com/myorg/CdkWrapper.java:

public class CdkWrapper implements RequestHandler<Map<String, String>, String> {

    public static final String CDK_OUT_DIR = "/tmp/cdk.out";
    private static final String CDK_COMMAND = "java -cp . ";

    @Override
    public String handleRequest(Map<String, String> input, Context context) {

        runCdk(RunCdkLambdaApp.class);

        return "";
    }

    public static void runCdk(Class<?> cdkClass) {
        ProcessBuilder processBuilder = new ProcessBuilder(
                "cdk",
                "deploy",
                "--verbose",
                "--output", CDK_OUT_DIR,
                "--app", CDK_COMMAND + cdkClass.getName(),
                "--require-approval", "\"never\""
        );
        processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
        processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT);

        Process process;
        try {
            process = processBuilder.start();
        } catch (IOException e) {
            throw new RuntimeException("Cannot start cdk process!", e);
        }

        try {
            process.waitFor();
        } catch (InterruptedException e) {
            throw new RuntimeException("Cdk process interrupted!", e);
        }

        int exitValue = process.exitValue();

        if (exitValue != 0) {
            throw new RuntimeException("Exception while executing CDK!");
        }
    }
}

We are doing this in Java using the ProcessBuilder class because our CDK code is also written in Java and we will use the Java 11 runtime in Lambda.

We also have to pack our Java code into an "uber" jar (jar with all the dependencies). We use maven-shade-plugin to do that run-cdk-lambda/pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.4</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
</plugin>

The largest matryoshka

The Java 11 Lambda runtime does not have NodeJS and AWS CDK binaries. We will use Lambda layers to put them inside. We will copy binaries from a docker image created from docker/cdk/Dockerfile:

FROM node:lts

RUN apt-get update && \
    apt-get install -y zip && \
    rm -rf /var/lib/apt/lists/*

ENV AWS_CDK_VERSION=1.71.0
RUN mkdir -p /nodejs && \
    npm config set prefix /nodejs/bin && \
    npm install -g aws-cdk@${AWS_CDK_VERSION}

RUN cd /nodejs/bin && \
    zip -r --symlinks /opt/aws-cdk.zip *
RUN cd /usr/local && \
    zip -r /opt/node.zip bin/node

We need also the code to create the "uber" jar, the docker image with NodeJS and CDK binaries, and copy files from it (actually creating a temporary docker container). This will be a bash script scripts/prepare-bin:

#!/bin/bash
set -e

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
PROJECT_DIR=$(realpath "${SCRIPT_DIR}/..")
DOCKER_IMAGE_TAG=aws-cdk-bin:latest
TMP_BIN_DIR="${PROJECT_DIR}/tmp"

mkdir -p ${TMP_BIN_DIR}

mvn package -f "${PROJECT_DIR}/run-cdk-lambda/pom.xml"
cp ${PROJECT_DIR}/run-cdk-lambda/target/run-cdk-lambda-*.jar ${TMP_BIN_DIR}/run-cdk-lambda.jar

docker build -t ${DOCKER_IMAGE_TAG} ${PROJECT_DIR}/docker/cdk

rm -f ${TMP_BIN_DIR}/docker-cid
docker create --cidfile ${TMP_BIN_DIR}/docker-cid ${DOCKER_IMAGE_TAG}
CID=$(cat ${TMP_BIN_DIR}/docker-cid)

trap "docker rm ${CID}" EXIT

docker cp ${CID}:/opt/node.zip ${TMP_BIN_DIR}/node.zip
docker cp ${CID}:/opt/aws-cdk.zip ${TMP_BIN_DIR}/aws-cdk.zip

Now we have to create the outer CDK code to create the lambda with layers. We need the CDK Lambda dependency pom.xml:

<dependency>
    <groupId>software.amazon.awscdk</groupId>
    <artifactId>lambda</artifactId>
    <version>${cdk.version}</version>
</dependency>

and the CDK code src/main/java/com/myorg/ExampleCdkFromLambdaStack.java:

public class ExampleCdkFromLambdaStack extends Stack {
    public ExampleCdkFromLambdaStack(final Construct scope, final String id) {
        this(scope, id, null);
    }

    public ExampleCdkFromLambdaStack(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);

        createRunCdkLambda();
    }

    private void createRunCdkLambda() {
        String tmpBinDir = getTmpBinDir();

        PolicyStatement cloudformationPolicy = new PolicyStatement(PolicyStatementProps.builder()
                .resources(Arrays.asList(
                        "*"
                ))
                .actions(Arrays.asList(
                        "cloudformation:DescribeStacks",
                        "cloudformation:CreateChangeSet",
                        "cloudformation:DescribeChangeSet",
                        "cloudformation:GetTemplate",
                        "cloudformation:GetTemplateSummary",
                        "cloudformation:DescribeStackEvents",
                        "cloudformation:ExecuteChangeSet",
                        "cloudformation:DeleteChangeSet"
                ))
                .build());

        // we can restrict this policy to certain buckets only
        PolicyStatement s3Policy = new PolicyStatement(PolicyStatementProps.builder()
                .resources(Arrays.asList(
                        "*"
                ))
                .actions(Arrays.asList(
                        "s3:*"
                ))
                .build());

        LayerVersion nodeLayer = LayerVersion.Builder.create(this, "node-layer")
                .description("Layer containing node binary")
                .code(
                        Code.fromAsset(tmpBinDir + "/node.zip")
                )
                .build();

        LayerVersion cdkLayer = LayerVersion.Builder.create(this, "aws-cdk-layer")
                .description("Layer containing AWS CDK")
                .code(
                        Code.fromAsset(tmpBinDir + "/aws-cdk.zip")
                )
                .build();

        Function lambda = new Function(this, "RunCdk", FunctionProps.builder()
                .runtime(Runtime.JAVA_11)
                .handler("com.myorg.CdkWrapper")
                .code(Code.fromAsset(tmpBinDir + "/run-cdk-lambda.jar"))
                .layers(Arrays.asList(
                        nodeLayer,
                        cdkLayer
                ))
                .timeout(Duration.seconds(300))
                .memorySize(512)
                .initialPolicy(Arrays.asList(
                        cloudformationPolicy,
                        s3Policy
                ))
                .functionName("run-cdk")
                .build());
    }

    private String getTmpBinDir() {
        Path tmpPath = Paths.get("tmp");
        return tmpPath.toAbsolutePath().toString();
    }
}

This is our third, outermost "matryoshka" layer. Here we are creating the Lambda policies (CloudFormation execution and S3 full access), layers (NodeJS, AWS CDK binaries), and the actual Lambda resource with the com.myorg.CdkWrapper handler and the code from run-cdk-lambda.jar.

We will also change the cdk.json file to run the prepare-bin script before the CDK execution cdk.json:

{
  "app": "./scripts/prepare-bin && mvn -e -q compile exec:java",
  "context": {
    "@aws-cdk/core:enableStackNameDuplicates": "true",
    "aws-cdk:enableDiffNoFail": "true",
    "@aws-cdk/core:stackRelativeExports": "true"
  }
}

Deploying

You have to set your AWS credentials first. You can do this in the ~/.aws/credentials file or using environment variables. Then, you have to bootstrap the CDK:

cdk bootstrap

Now you can create the Lambda function:

cdk deploy --require-approval never

Testing

Check your S3 buckets using:

aws s3 ls

This should not return any bucket with the "runcdklambdastack-createdbycdkfromlambda" prefix.

Run the Lambda function:

aws lambda invoke --function-name run-cdk tmp/lambda-out

It will run the AWS CDK and create the bucket. The first run can take about 1 minute to complete. Check again your S3 buckets then:

aws s3 ls

It should return a bucket created by CDK run inside a Lambda function (with the "runcdklambdastack-createdbycdkfromlambda" prefix).

Conclusion

Running AWS CDK inside a Lambda function is not an easy task. However, sometimes it is useful and I hope that this article will help you to do it, or maybe will only show you how to use lambda Layers to execute almost any executable binary inside an AWS Lamda. We do use it in DevOpsBox to create appropriate IAM roles and policies based on the roles read from Keycloak.

For more details about the DevOpsBox platform please visit https://www.devopsbox.io/

Horizontal Pod autoscaling based on HTTP requests metric from Istio

Maciej Raszplewicz — Fri, 23 Oct 2020 19:44:59 +0000

After reading this article, you will learn how to configure autoscaling based on the average number of HTTP requests per second.

I haven’t found any article describing the configuration of horizontal pod autoscaling based on the HTTP requests metric from Isio. There are a few sources, but all of them are outdated or much more complicated than my solution.

Some time ago, we have created a similar autoscaling solution based on metrics from AWS Load Balancer, which was scaling containers deployed on ECS. We first tried CPU based autoscaling, but we had a lot of problems because of high CPU usage on an application startup. Scaling based on the number of HTTP requests worked much better. However, in the Kubernetes world, things are completely different…

All source code is available here: https://github.com/devopsbox-io/example-istio-hpa

Prerequisites

I have used several tools in this article, you could probably replace some of them, though. Remember to use a fairly new version of Istio (tested with 1.7.2). Probably older versions do not have istio_requests_total metric available per Pod.
List of tools:

Minikube (tested with v1.10.1, Kubernetes v1.17.12)
KVM (required by Minikube)
Kubectl (tested with v1.17.12)
Helm (tested with v3.2.1)
Siege (tested with 4.0.4)
Istioctl (tested with 1.7.2)

Preparation

First of all, we have to start Minikube:

minikube start --driver=kvm2 --kubernetes-version v1.17.12 --memory=8192 --cpus=4 && minikube tunnel

Few things to mention here: I use kvm2 and Kubernetes version 1.17, but the solution will probably work on different Kubernetes versions and different Minikube drivers (or even other Kubernetes distributions). We need quite a lot of RAM and CPU because we want to test the autoscaling. The last thing - we have to run the Minikube tunnel to access the Istio ingress gateway, so it will ask you for the sudo password and it will lock your terminal, therefore you will have to open a new one.

Next, we need to install Istio:

istioctl install -y

We are using the default Istio configuration. Nothing special here.

Then, we will create namespaces and enable Istio automatic sidecar injection on one of them:

kubectl create namespace monitoring
kubectl create namespace dev
kubectl label namespace dev istio-injection=enabled

I will not explain what a sidecar is and how Istio works, but if you want to know more - just read the Istio documentation.

Then, we will deploy the sample application (code available here: https://github.com/devopsbox-io/example-istio-hpa/blob/main/sample-app-with-istio.yaml):

kubectl -n dev apply -f sample-app-with-istio.yaml

and wait for the deployment to work (probably a few minutes):

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
until curl -f http://$INGRESS_HOST; do echo "Waiting for the application to start..."; sleep 1; done

This is an almost unmodified httpbin application from Istio documentation.

The solution

For our solution we will need Prometheus to scrape metrics from Istio:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm repo update
helm -n monitoring install prometheus prometheus-community/prometheus

It is the default helm chart for the Prometheus installation. We use this one because Istio has a default configuration to expose metrics for it, i.e. pod has the following annotations:

prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true

Having Prometheus installed doesn't mean that we can use its metrics for horizontal Pod autoscaling. We will need one more thing - Prometheus Adapter installed with customized configuration file prometheus-adapter-values.yaml:

prometheus:
  url: http://prometheus-server.monitoring.svc.cluster.local
  port: 80
rules:
  custom:
  - seriesQuery: 'istio_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
    resources:
      overrides:
        kubernetes_namespace: {resource: "namespace"}
        kubernetes_pod_name: {resource: "pod"}
    name:
      matches: "^(.*)_total"
      as: "${1}_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Here, we can see our Prometheus instance URL, port, and one custom rule. Let's focus on this rule:

seriesQuery is needed for metric discovery resources/overrides are mapping fields from the metric (kubernetes_namespace, kubernetes_pod_name) to the names required by Kubernetes (namespace, pod).
name/matches, name/as are needed to change the metric name. We are transforming this metric, so it is good to change the name istio_requests_total to istio_requests_per_second.
metricsQuery here is the actual query (which is actually a query template) and it will be run by the adapter while scraping the metric from Prometheus. rate and [2m] "calculates the per-second average rate of increase of the time series in the range vector" (from Prometheus documentation), here it is the per-second rate of HTTP requests as measured over the last 2 minutes, per time series in the range vector (also, almost from the Prometheus documentation).

Now, as we have the adapter configuration, we can deploy it using:

helm -n monitoring install prometheus-adapter prometheus-community/prometheus-adapter -f prometheus-adapter-values.yaml

Ok, so the last thing is to create the Horizontal Pod Autoscaler using the following configuration:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: httpbin
spec:
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: istio_requests_per_second
      target:
        type: AverageValue
        averageValue: "10"
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: httpbin

Most of the configuration is self-explanatory. scaleTargetRef references our application’s Deployment object and min and max replicas are our boundaries. The most interesting part is metrics — here we tell the autoscaler to use our custom istio_requests_per_second metric (which is calculated per Pod) and that it should scale out after more than 10 average requests per second.

One of the most important things — probably when other articles about this topic were written, istio_requests_total metric wasn’t calculated per pod. Things got much easier because now it is!

Now let’s create the Horizontal Pod Autoscaler:

kubectl -n dev apply -f hpa.yaml

and wait for the metric availability (probably a few minutes):

until kubectl -n dev describe hpa | grep "\"istio_requests_per_second\" on pods:" | grep -v "<unknown> / 10"; do echo "Waiting for the metric availability..."; sleep 1; done

We have our autoscaling up and running. Let’s test it!

Testing

First of all, we can open two new terminal windows and watch what is happening (every line in a separate window):

watch kubectl -n dev get pod
watch kubectl -n dev describe hpa httpbin

Then, let’s start testing:

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
siege -c 2 -t 5m http://$INGRESS_HOST

You can use other tools than siege (e.g. hey). It is important that it needs to support HTTP/1.1 so ab (apache benchmark) is not the right solution.

After a few minutes, you should see more pods running, and that “describe hpa” shows the current number of requests per second.

Conclusion

It is not that hard to create an autoscaling solution based on the HTTP requests per second metric if you know what you are doing. It should be also quite simple, to change it to some other Prometheus metric. But should you really do it yourself? Our DevOpsBox platform has already built-in full autoscaling with reasonable defaults!

For more details about the DevOpsBox platform please visit https://www.devopsbox.io/