Introducing cloudform – tame your AWS CloudFormation templates

Whatever we do here in Bright Inventions, we deeply care about automation, traceability and repeatability. This is why we embraced the DevOps-related practices like Continuous Delivery or containerization, we are careful about properly set up logging and monitoring, we know our stuff when it comes to reliability and resiliency. This is also why, whenever we do anything at the backend, we define our infrastructure as code with the great help of AWS CloudFormation.

Why should I care?

In AWS CloudFormation all the cloud infrastructure required to run the application – from the hardware setup through the networking layer to the actual application environment – is defined within a JSON (or YAML) template file. This is convenient as we can now keep the file in GIT and use it as an input to the deployment scripts.

The problem is, though, that unless our project is really simple, our template file grows quickly. Our typical environment setup based on Elastic Container Service with VPC, failovers, firewalls, load balancing, auto-scaling, logging and monitoring requires ~2.5k lines in our JSON template file. And some of the constructs there – like passing RDS database connection string to the container instance that includes concatenation of multiple parameters, some of which are secret, some other are generated by other resources – are really verbose and complex.

Enter cloudform

Our typical template file was too large. It was too verbose and repetitive. It was error-prone and hard to grasp. It was messy. But all was not lost – the template is just a plain old JSON, anyway. This is how cloudform was born. It is a TypeScript-based imperative way to define AWS CloudFormation templates. With cloudform you can use all the powers of TypeScript to define the elements of your infrastructure and let the library take care of generating that monstrous JSON file out of it.

So, for example, to define a VPC, instead of writing this lengthy and hairy piece of JSON:

"VPC": {
  "Type": "AWS::EC2::VPC",
  "Properties": {
    "CidrBlock": {
      "Fn::FindInMap": [
        "NetworkingConfig",
        "VPC",
        "CIDR"
      ]
    },
    "EnableDnsHostnames": true,
    "Tags": [
      {
        "Key": "Application",
        "Value": {
          "Ref": "AWS::StackName"
        }
      },
      {
        "Key": "Network",
        "Value": "Public"
      },
      {
        "Key": "Name",
        "Value": {
          "Fn::Join": [
            "-",
            [
              {
                "Ref": "AWS::StackId"
              },
              "VPC"
            ]
          ]
        }
      }
    ]
  }
}

you can just have a EC2.VPC object definition that is totally equivalent:

VPC: new EC2.VPC({
    CidrBlock: NetworkingConfig.VPC.CIDR,
    EnableDnsHostnames: true,
    Tags: [
        new ResourceTag('Application', Refs.StackName),
        new ResourceTag('Network', 'Public'),
        new ResourceTag('Name', Fn.Join('-', [Refs.StackId, 'VPC']))
    ]
})

Note what I have done here:

1. No more explicit Type, no Properties wrapper boilerplate. TypeScript's type is enough to figure it out. What's more, it gives us a compile-time check for the validity of the structure.
2. No more magic strings for AWS::StackName or AWS::StackId references. Compile-time constants, again.
3. Instead of complex Fn::FindInMap object traversing, I use TypeScript-level objects directly. Yes, it results in a different entry in the template, but functionally it's an equivalent.
4. I also simplified the construction of key-value pairs used in Tags. Strongly-typed one-liner instead of verbose 4 lines.

But this is not the end. With cloudform (or actually with TypeScript) we can go further:

1. Nothing stops us from using TypeScript-level variables, functions or conditions to build up our objects from reusable, configurable parts.
2. We can split the template definition into multiple files.
3. We can apply all the available refactoring techniques to keep our definitions tidy and easy to use.

With cloudform's syntax our ~2.5k lines of JSON was reduced to ~1k lines of TypeScript object definitions.

Getting started & usage

You're probably already eager to test it out, so let's jump to the crux of the matter. The package is obviously available for you on npm. Type the following:

npm install --save-dev cloudform

Now we need to define our template. Let's have a separate directory for it, so that it doesn't mix with our production or test code. It would also suggest we may break the file into smaller chunks and use standard imports to link it together. Maybe cloudformation/index.ts?

This file needs to end with cloudform function call, so that the tool picks our template up correctly. It takes an object that consists of everything that CloudFormation template might consist of.

import cloudform, {Fn, Refs, EC2, StringParameter, ResourceTag} from "cloudform"

cloudform({
    Description: 'My template',
    Parameters: {
        DeployEnv: new StringParameter({
            Description: 'Deploy environment name',
            AllowedValues: ['dev', 'stage', 'production']
        })
    },

    Resources: {
        VPC: new EC2.VPC({
            CidrBlock: Fn.FindInMap('SubnetConfig', 'VPC', 'CIDR'),
            EnableDnsHostnames: true,
            Tags: [
                new ResourceTag('Application', Refs.StackName),
                new ResourceTag('Network', 'Public'),
                new ResourceTag('Name', Fn.Join('-', [Refs.StackId, 'VPC']))
            ]
        })
    }
})

The simple convention is used here – all the AWS types' namespaces are available directly as exports from the cloudform package. All the resources within this package are available inside. This way EC2.VPC object from our example translates into AWS::EC2::VPC type we can find in CloudFormation documentation. All the properties also match one-to-one, including casing. This API-level compliance is guaranteed to hold true because the cloudform's types are generated from the AWS-provided schema definition file.

See also the example included in the repository.

Now, run the binary that was installed for you specifying the path to your template sources root. The library will print the JSON to the stdout, so you might want to pipe it to the file somewhere:

cloudform cloudformation/index.ts > template.out

And if no compilation errors happen, template.out is now a JSON file ready to be used within CloudFormation.

What cloudform is not

I can feel your excitement already 😆, but let's be clear what cloudform does and what is out of its scope. Basically it translates the TypeScript code into the JSON object as-is. Whatever you put in your TypeScript, as long as it compiles, will be included in the generated file. There is no validation or any domain-level checks implemented – if you reference a resource, parameter or condition that does not exists or use a nonsensical value for a parameter, cloudform will not warn you today (CloudFormation will do most probably).

The goal of this package is not to create a full-blown DSL for CloudFormation. It is rather to get the most value out of TypeScript features, so if any more complex checks are to be implemented, it would most probably be about things that TypeScript can verify compile-time. If you need a full validation, there are some specialized tools out there already (see for example cfn-lint or cfn-check), so it might have sense to chain it after cloudform.

Note also that cloudform does not attempt to interact with your AWS stack in any way, nor reading neither writing – no credentials are needed. It makes it 100% safe to experiment with. All the stack interactions (i.e. deployment) are outside of the scope of this tool.

What's next?

The package source code is MIT-licensed, available on GitHub. If you find out that something is missing or invalid in cloudform, feel free to open an issue on GitHub or – even better – contribute yourself!

In the follow-up post I'm sharing a few more examples and recipes how to build template generation into your Continuous Delivery workflow.

Originally published at brightinventions.pl

By Adam Bar, The Web Guy @ Bright Inventions
Personal blog Twitter Email Github