DEV Community: mklabs

yet another post about type classes in Scala

diogoaurelio — Mon, 02 Jan 2023 16:15:43 +0000

While recently having a chat with a colleague about type classes in scala, I realized I failed at properly explaining the concept. So here I am, giving it (yet) another stab.

In this post we will:

understand the intuition behind type classes (TCs), to better reason when to use them;
provide two concrete examples with code in scala;
conclude with a very brief look into cats, a widely used scala library that heavily relies on TCs;

The examples provided in this post were written in Scala 2, and are available in this github repository.

Intuition

Type classes provide a way to model for different behaviors according to the type of the object being handled. So, at it's core, a type class is (unlike the name might at first sight suggest) actually just an interface - a trait in scala - that defines the behavior with a given method signature that accepts or returns generic type.
We then define concrete implementations for that interface for a given specific type. These we call the type class instances.
Usually, as a last step, one exposes this functionality with a given api, say, for example a static method.

Example 1

Let's say we work in a company that does ratings, and, to simplify matters, issues 5 star ratings. Currently the company started with doing evaluations of restaurants. However, we want to be able to extend those in the future to other aspects in life, such as hotels, cities, concerts, cars, gadgets, etc, you name it.
Modelling our business domain, we'd quickly understand that a car is a completely distinct model than a hotel, so using classic OOP polymorphism by finding a common denominator interface that all of the other classes extend/inherit is a weak solution.
Instead, what we really want is just to define a general interface for rating any of these types - think generics. In more rigorous terms this is called parametric polymorphism, which refers to the ability to define a single code template that can work with any type T parameter.

  // this is our Type Class
  trait Reviewer[T] {
    def rate(obj: T): Int
  }

Our TC has a single method that accepts a generic type, and returns 1-5 star rating which we oversimplified with an Integer (Int) return type.

Let's add two (rather silly but hopefully illustrative) models:

  case class Restaurant(name: String, foodQuality: Int, environment: Int, location: Int)

  case class Dish(name: String, sweetness: Int, saltiness: Int, bitterness: Int, sourness: Int, umami: Int)

Now comes the second part, where we deviate from parametric polymorphism - where the concrete implementation is completely agnostic to the Type T being handled - and add specific TC instances that define different behaviors/strategies depending on the object being handled. For rating a restaurant one could implement a simple algorithm as follows:


  object RestaurantReviewer extends Reviewer[Restaurant] {
    override def rate(obj: Restaurant): Int =
      scala.math
        .round(0.6F * obj.foodQuality + 0.3F * obj.environment + 0.1F * obj.location)
  }

and, on the other hand, for reviewing a dish the following strategy:


    object DishReviewer extends Reviewer[Dish] {
      override def rate(obj: Dish): Int = scala.math.round(
        0.1F * obj.sweetness
          + 0.2F * obj.saltiness
          + 0.1F * obj.bitterness
          + 0.1F * obj.sourness
          + 0.5F * obj.umami
      )
    }

TC are also sometimes referred to as ad hoc polymorphism, which opposes parametric polymorphism because one purposely defines a concrete distinct implementation for every given type. I found Jason McClellan does a great job explaining the different types of polymorphism and in concrete for type classes, so a highly recommend you to have a look at it.

The usual third and last step is to expose the overall rating functionality, for example, as follows:


  object Evaluator {
    def rate[T](obj: T)(evaluator: Reviewer[T]): Int = evaluator.rate(obj)
  }

Alright, this is already enough for us to dry-run in a demo app:


object DemoTC extends App {

  trait Reviewer[T] {
    def rate(obj: T): Int
  }

  case class Restaurant(name: String, foodQuality: Int, environment: Int, location: Int)
  case class Dish(name: String, sweetness: Int, saltiness: Int, bitterness: Int, sourness: Int, umami: Int)

  object ReviewStrategies {

    object RestaurantReviewer extends Reviewer[Restaurant] {
      override def rate(obj: Restaurant): Int =
        scala.math
          .round(0.6F * obj.foodQuality + 0.3F * obj.environment + 0.1F * obj.location)
    }

    object DishReviewer extends Reviewer[Dish] {
      override def rate(obj: Dish): Int = scala.math.round(
        0.1F * obj.sweetness
          + 0.2F * obj.saltiness
          + 0.1F * obj.bitterness
          + 0.1F * obj.sourness
          + 0.5F * obj.umami
      )
    }

  }

  object Evaluator {
    def rate[T](obj: T)(evaluator: Reviewer[T]): Int = evaluator.rate(obj)
  }

  val restaurant1 = Restaurant(name = "Cheesegaddon", foodQuality = 5, environment = 3, location = 2)

  // rate using `RestaurantReviewer` strategy:  
  val restaurant1Rating = Evaluator.rate(restaurant1)(ReviewStrategies.RestaurantReviewer)

  // will print: Restaurant Cheesegaddon final rate is: 4 stars
  println(s"Restaurant ${restaurant1.name} final rate is: ${restaurant1Rating} stars")


}

Note that so far we're not leveraging any of scala's powerful features. As a matter a fact, we can implement the exact same demo code in java (yes java nerds, bear with me, as mentioned in the inline code comments, the above code is not the finest, but it illustrates my point):

public final class DemoTCJava {

    // this is our Type Class
    interface Reviewer<T> {
        int rate(T obj);
    }

    // Note 1: Avoiding using newer Java Records to make this code easily runable on any jdk version 
    // Note 2: made usually private internal vars public as to avoid extra verbosity of implementing getters;
    private static final class Restaurant {
        public final String name;
        public final int foodQuality;
        public final int environment;
        public final int location;

        public Restaurant(final String name, final int foodQuality, final int environment, final int location) {
            this.name = name;
            this.foodQuality = foodQuality;
            this.environment = environment;
            this.location = location;
        }
    }

    private static final class Dish {
        public final String name;
        public final int sweetness;
        public final int saltiness;
        public final int bitterness;
        public final int sourness;
        public final int umami;

        public Dish(final String name, final int sweetness, final int saltiness, final int bitterness, final int sourness, final int umami) {
            this.name = name;
            this.sweetness = sweetness;
            this.saltiness = saltiness;
            this.bitterness = bitterness;
            this.sourness = sourness;
            this.umami = umami;
        }
    }

    private static final class RestaurantReviewer implements Reviewer<Restaurant> {
        @Override
        public int rate(Restaurant obj) {
            return Math.round(0.6F * obj.foodQuality + 0.3F * obj.environment + 0.1F * obj.location);
        }
    }

    private static final class DishReviewer implements Reviewer<Dish> {
        @Override
        public int rate(Dish obj) {
            return Math.round(
                    0.1F * obj.sweetness
                            + 0.2F * obj.saltiness
                            + 0.1F * obj.bitterness
                            + 0.1F * obj.sourness
                            + 0.5F * obj.umami
            );
        }
    }

    private static final class Evaluator<T> {
        public int rate(T obj, Reviewer<T> evaluator) {
            return evaluator.rate(obj);
        }
    }

    public static void main(String[] args) {
        final Restaurant restaurant1 = new Restaurant("Cheesegaddon", 5, 3, 2);
        final RestaurantReviewer reviewer = new RestaurantReviewer();
        final Evaluator<Restaurant> evaluator = new Evaluator<>();
        final int restaurant1Rating = evaluator.rate(restaurant1, reviewer);

        System.out.println(
                String.format("Restaurant %s final rate is: %d stars",
                        restaurant1.name,
                        restaurant1Rating
                )
        );
    }

}

This is an example where scala's implicits really shine. If we made our API - the Evaluator static class - accept the evaluator as an implicit argument:


  object Evaluator {
    def rate[T](obj: T)(implicit evaluator: Reviewer[T]): Int = evaluator.rate(obj)
  }

.. and changed our ReviewStrategies to implicitly define TC instances:


  object ReviewStrategies {

    implicit object RestaurantReviewer extends Reviewer[Restaurant] {
      override def rate(obj: Restaurant): Int =
        scala.math
          .round(0.6F * obj.foodQuality + 0.3F * obj.environment + 0.1F * obj.location)
    }

    implicit object DishReviewer extends Reviewer[Dish] {
      override def rate(obj: Dish): Int = scala.math.round(
        0.1F * obj.sweetness
          + 0.2F * obj.saltiness
          + 0.1F * obj.bitterness
          + 0.1F * obj.sourness
          + 0.5F * obj.umami
      )
    }

  }

.. we could then simply import the intended review strategies based on scope:


  import ReviewStrategies._
  val restaurant1Rating = Evaluator.rate(restaurant1)

And this is one of my favorite things about TCs, namely the flexibility for future extension that they provide. If perhaps tomorrow a new food critic would join the team, one that uses a completely different algorithm for computing the rate, absolutely no change would be required
in our API - we would simply need to define and import the new different set of strategies.

Example 2

Our second example is meant to illustrate an additional nice thing about TCs: the compile time safety.

Let's say we want to prevent the following to happen:


  val aComparison = 2 == "a String"
  // will print: Comparison is false
  println(s"Comparison is $aComparison")

Your IDE will hopefully highlight the fact that we are comparing an int with a string, but compilation will succeed nonetheless, and the expression will evaluate to false.

To solve this, we can use the same strategy as before. Let's define an Eq TC with two concrete implementations, respectively for Int and String:


  trait Eq[T] {
    def equals(val1: T, val2: T): Boolean
  }

  implicit object IntEq extends Eq[Int] {
    override def equals(val1: Int, val2: Int): Boolean = val1 == val2
  }

  implicit object StringEq extends Eq[String] {
    override def equals(val1: String, val2: String): Boolean = val1.equals(val2)
  }

  object Comparison {
    def compare[T](val1: T, val2: T)(implicit comparator: Eq[T]): Boolean = comparator.equals(val1, val2)
  }

  val aNumber: Int = 4
  val anotherNumber: Int = 5

  val result1 = Comparison.compare(aNumber, anotherNumber)

  // will print: 4 == 5 is false
  println(s"$aNumber == $anotherNumber is $result1")

With this approach we are certain that if we try:

Comparison.compare(1, "one")

... compilation will fail, as intended.

We could improve our Comparison implementation just a little further leveraging scala extension methods:

  object MyComparisons {
    implicit class MyIntComparison[T](val1: T)(implicit eq: Eq[T]) {
      def `===`(val2: T): Boolean = eq.equals(val1, val2)
    }
  }

If we import MyComparisons object, we will immediately start having access to our custom === method for all Int and String type objects, allowing the following:

  import MyComparisons._
  // will print: 4 === 5 is false
  println(s"$aNumber === $anotherNumber is ${aNumber === anotherNumber}")

In case you're wondering how extension methods work, essentially the compiler is smart enough to underneath the hood to rap our Int instance stored in variable aNumber into a new instance of:
new MyIntComparison(aNumber)
.. and call the method === towards the other Int instance stored in variable anotherNumber:
new MyIntComparison(aNumber).=== (anotherNumber)

Pretty cool, right?

If you're thinking to implement this by yourself, think again. Scala's Swiss-army-knife library cats provides exactly this same functionality.

Start by importing it in sbt:

val catsVersion = "2.1.1"

libraryDependencies ++= Seq(
  "org.typelevel" %% "cats-core" % catsVersion
)

And use it:


  import cats.instances.int._
  import  cats.syntax.eq._

  // will print: 2 === 3 is false
  println(s"2 === 3 is ${2 === 3}")

Conclusion

I'd like to wish that, if anything, this post manages to convey the intuition behind type classes.

Type classes are useful when we want to model different behaviors according to the specific type of an object. With our previous example, we tried to provide a case where it makes little sense to use polymorphism to model distinct concepts, namely restaurants, hotels, cars, cities, etc. In other words, instead of having a common interface that all of the previous classes inherit, we prefer to rather use generics.
Our type class is nothing but an interface that defines the signature for a given behavior/method leveraging generics.
This leaves the concrete implementation open for accommodate different strategies according to the concrete types, or even different strategies for the same time. This idea is probably what makes type classes so powerful: the flexibility to extend them.

Our second type class example attempted to illustrate one last perk: type safety at compile time. It did so with a simplified example of the cats core library for type safety equality comparison between objects. If you're not familiar with cats, go ahead and give it go.

References

parametric polymorphism
ad hoc polymorphism
Jason McClellan on the different types of polymorphism and where type classes fit
cats core library
repo code for this blog post

Bootstrapping a Startup on AWS with AWS Serverless and Go

Roko Romic — Thu, 17 Jun 2021 11:44:12 +0000

In this article we wanted to share an overview of a recent project to build a logistics platform from scratch using AWS cloud, Go and serverless. Yes, there is already a great abundance of blog posts focusing on all of these individual technologies. What we find often lacks is how to glue all of those together, which is exactly what we describe in this post.
Here is our full code supporting this blog post.

Pre-Requisites

If you want to try to play around with our examples, you'll need to set up those things in case you don't have already set it up:

AWS account
AWS CLI
AWS SAM CLI
Terraform
Docker
Make
AWS VPC
AWS S3 buckets

To be able to completely follow up an article, some basic understanding of those tools and frameworks would be nice, but it's not required.
The concept of this post and examples is to give you an idea how you can set up your own architecture and use those techniques.

Overview - architecture

So basically our architecture consists of:

AWS Lambdas - Go runtime running our business logic
AWS S3 buckets - for hosting frontend and Terraform backend configuration
AWS RDS (Postgres version 12) - instance for storing business related data
AWS DynamoDB - used for Terraform locking mechanism, preventing multiple users to apply simultaneously
AWS SSM - used to connect to our RDS instance within private VPC to be able to monitor and to maintain data
AWS Cognito - used for identity management and authentication
Terraform and AWS SAM (Serverless Application Model) as glue to keep all states synced and up-to-date

From an outside networking point of view, we decided to go with cloudflare as a DNS service. The main benefit of using cloudflare is it’s DDoS protection. The experiences that we had previously with cloudflare were very good, so it was logical to stay with cloudflare. Most of all, adding and maintaining DNS records, we found to be user friendly even for one’s without a networking background.
With this feature on, one thing less to think about while designing a solution. Last awesomeness is that Cloudflare solutions/features (such as automatically setting up SSL certificates for our (sub)domains) are completely free.

The whole project is handled by the GitLab CI/CD pipeline.
You can get free minutes on Gitlab to run your own CI/CD pipelines, when it comes to small projects, this is a fair deal.

You can see in the next picture how we designed a solution for the project.
This illustration represents an overview of the technical solution.

Programming Language

Regarding the programming language choice, Go seemed like an easy choice: low memory footprint (allowing thin run times), fast startup times, static typing, and ease of usage and readability. A lot of work in this platform involved integration with different partners, so the simplicity in Go’s concurrency model was also a big appeal.
Regarding ORM side of things, in the mkops project this was handled by a java 11 backend based on spring boot and hibernate, and go was a pure stateless supporting microservice. This time we had a chance to dive into GORM to model our entities persisted on a Postgres 12.

Serverless model

Due to the nature of most of our projects (related to big data), our default infrastructure goto platform is kubernetes. Here, however, a serverless model for the API provided a quite nice fit.
After reviewing all the workflows that needed to happen, it was clear to us: the logistics platform would be heavy on the background processes that integrate between different platforms that could be started in a cron fashion. The database provided all state locking required, so the processes could run independently and without communicating to each other.
Another argument was the relative low amount of user traffic expected. The cost and administration effort of running a standalone and permanent fargate instance (ECS/EKS) did not add up.
In summary, the motives for our choice are much of the usual suspects: low administration effort, higher security due to ephemeral nature, and lower operating costs.

In order to provide a seamless local development environment AWS provides a cool framework named AWS SAM. It allows you to run AWS Lambdas locally, and, most importantly, with an option to easily create the relevant Service events like API Gateway. So our solution for running things locally and for testing involved Docker with SAM template and of course Localstack for simulating other AWS resources that we were using, such as AWS SES, AWS S3 and AWS SecretManager.

AWS SAM

In the spirit of keeping low a maintenance effort while having a speedy implementation, we decided to start by considering serverless frameworks. The two main options boiled down to AWS SAM or the Serverless Framework.
The main difference between the two is that Serverless is used to deploy FaaS (Function as a Service) functions with support for different providers. SAM, on the other hand, is used specifically for AWS as a cloud provider, deploying not only cloud functions but also the (minimum) underlying infrastructure to expose some business logic with an HTTP Endpoint.

AWS SAM allows us to spin up (most) of the resources it manages locally, including Lambdas and the API Gateway. With this in mind, it made it easier for us to develop and test Lambda functions before deploying them to AWS.

We were fairly set on using AWS as a cloud provider, which helped our decision to go for AWS SAM.

AWS SAM follows a declarative approach for defining the stack and makes use of a command line utility to deploy: resources are declared in one key file called template.yaml. More details on the anatomy of the AWS SAM template here.

Under the hood, AWS SAM is an oversimplified version of CloudFormation albeit very similar. Resources declared in the template.yaml get automatically converted, by the sam cli to CloudFormation so it’s very easy to inspect what exactly is getting created on our behalf.

After installing sam cli for your operating system, these are the commands that you will need to create AWS resources are:

sam package --template-file template.yaml --s3-bucket your_s3_bucket --output-template-file package.yaml
sam deploy --template-file package.yaml --stack-name your_stack_name --capabilities CAPABILITY_IAM

sam package is used for converting the template.yaml into deployable artifacts as well as uploading said artifacts to an AWS S3 bucket. It’s important to note that this bucket needs to be created beforehand.
An important caveat to these deployments to note is that there is a hard limit of how much your total package size can be: 250MB. Meaning, if you have multiple lambdas defined, and their total size exceeds the limit, SAM will throw an error. If you are tempted to create an individual function for each endpoint, then it’s likely you will reach this limit quite fast. One way to solve this, is to use your lambas as docker images (a recent feature of Lambdas, where the hard limit is 10GB),upload it to AWS ECR and reference it in the SAM template or mount EFS volumes within Lambda. More details you can find here. Another way is to write a single lambda with its own router and point all traffic there (read more for an example).
The output generated from command sam package is the input template file for next command, sam deploy.
To delete your resources created by AWS SAM run following command:
aws cloudformation delete-stack --stack-name *your_stack_name*
Here is an example of the AWS SAM template.yaml file. The file is used to create AWS resources like AWS Lambda, and API Gateway with Custom domain through which lambda will be exposed on the Internet.
We’re setting up a Custom domain for API Gateway to be exposed to the Internet over a user-friendly domain name, instead of a generic one created by AWS.

---
AWSTemplateFormatVersion: 2010-09-09

Transform: AWS::Serverless-2016-10-31

Parameters:
  TargetStage:
    Description: "dev/prd"
    Type: String
  DomainName:
    Type: String
  AcmCertificateArn:
    Type: String
  VPCSecurityGroupIDs:
    Description: "An comma-delimited list of strings - the security groups that your Lambda function should be in"
    Type: CommaDelimitedList
  VPCSubnetIDs:
    Description: "An comma-delimited list of strings - the subnet IDs that your Lambda function should be assigned to"
    Type: CommaDelimitedList

Globals:
  Api:
    Cors:
      AllowMethods: "'GET, POST, PUT, OPTIONS, DELETE'"
      AllowHeaders: "'*'"
      AllowOrigin: "'*'"

  Function:
    Runtime: go1.x
    Tracing: Active # https://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html
    Timeout: 30
    VpcConfig:
      SecurityGroupIds:
        Ref: VPCSecurityGroupIDs
      SubnetIds:
        Ref: VPCSubnetIDs

Resources:
  ApiDetails:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Ref TargetStage
      Auth:
        UsagePlan:
          CreateUsagePlan: PER_API
          Description: Usage plan for this API
          Quota:
            Limit: 3000
            Period: MONTH
          Throttle:
            BurstLimit: 50
            RateLimit: 20

  ApiDomain:
    Type: AWS::ApiGateway::DomainName
    Properties:
      RegionalCertificateArn: !Ref AcmCertificateArn
      DomainName: !Ref DomainName
      EndpointConfiguration:
        Types:
          - REGIONAL
      SecurityPolicy: TLS_1_2

  ApiDomainMappings:
    Type: AWS::ApiGateway::BasePathMapping
    Properties:
      DomainName: !Ref DomainName
      RestApiId: !Ref ApiDetails
      Stage: !Ref ApiDetails.Stage

  EchoFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: bin/echo-sample
      MemorySize: 128
      Events:
        AllEvents:
          Type: Api
          Properties:
            Path: /{proxy+}
            Method: any
            RestApiId: !Ref ApiDetails

Outputs:
  ApiCustomDomainRegionalDomainName:
    Description: 'Regional domain name for the API'
    Value: !GetAtt ApiDomain.RegionalDomainName

As you can see in example, there are some sections defined like Parameters, Globals, Resources, Outputs. More details about each section you can read here.
To set up some parameters or override them, you can use option --parameter-overrides for aws sam deploy command. Like an example: sam deploy --stack-name test--template-file packaged.yaml --capabilities CAPABILITY_IAM --parameter-overrides TargetStage=dev, where we’re setting to parameter TargetStage a value dev.
The section of interest is the Resource: this is where we describe how our API Gateway, Custom Domain used for API Gateway and our simple Lambda should look like.

It’s also worthwhile mentioning that, for each API, you can define an Invocation Quota which can be both (1) a way to protect your deployment and (2) avoid cost escalation.
In this example we’re setting API Gateway to Custom Domain. To do so, we’re defining two resources: ApiDomain and ApiDomainMappings. To completely set up a custom domain on API Gateway you have to be an owner of that domain and it doesn’t need to be registered on AWS Route53 - it can be any DNS Registrar. We’re using Cloudflare as our DNS Registrar, so in this example we will use the same DNS.
Next step is to create an ACM certificate. AWS Certificate Manager (ACM) is a service that provisions and manages SSL certificates on AWS. This is a prerequisite if we want to do anything custom domain related in API Gateway. Note: you have to create an ACM certificate in the same AWS region as your API Gateway will be, if you are using an API endpoint type as regional as we do in our example.
In our case, we’re using Cloudflare, and our domain is registered on Cloudflare service. In that case, we're going to need to create an SSL certificate on Cloudflare and import that one into AWS ACM.
Here is a blog post from Cloudflare which can help you to create a certificate, it’s pretty straightforward. After successfully creating an origin certificate, you need to store the origin certificate and private key generated for you in PEM format. You will need this data for importing the certificate into AWS ACM. Note if you don’t save a private key and exit from the page with the generated one, you won’t be able to retrieve it again. In that case you will have to first revoke the certificate and then create a new one again.
To import an existing origin certificate published by Cloudflare, follow these instructions.
Save the newly generated ARN key of the imported certificate, you will assign it in the SAM template in resource ApiDomain.

Note that your custom DomainName property in resource ApiDomain must match the one that was added in the ACM certificate, otherwise it won't be created. In this example we’re using an SSL certificate for api-test.your.domain.com, so we have to set the parameter DomainName exactly like that subdomain.

The last thing to set up for API Gateway and custom domain is to define resource ApiDomainMappings. Basically it will allow us to point the custom domain at an existing API Gateway. It’s a simple mapping that lets us point API stages at specific paths. This is a useful feature if you have multiple APIs making up a suite that you want behind a shared domain.

We ran into an issue while setting up Stage property. At first we tried to simply add value as reference to Parameter Stage: !Ref TargetStage, but we got a strange error saying >“Invalid stage identifier specified”. After some digging we found a github topic with the same problem.
The problem lies in that creating a Stage resource is outside of the SAM template and with just referencing it to name will fail, because the resource ApiDomainMappings it’s being created before resource Stage. Thanks to fellows on github, this suggestion worked.
So we changed Stage property to: Stage: !Ref ApiDetails.Stage.

The very last thing to finish setting up API Gateway is to update over DNS with new records pointing to the domain name of API Gateway generated by AWS.
Generic domain name looks like https://api-id.execute-api.region.amazonaws.com/stage, where api-id is generated by API Gateway, region (AWS Region) is specified by you when creating the API, and stage is specified by you when deploying the API.

All you have to do is to login to your Cloudflare account and add a new CNAME record with a subdomain that matches the one you decided to add into the SAM template in DomainName parameter and pointing to the generated domain name of your API Gateway.

With this set up, you just provide to your developers newly created user-friendly REST API endpoint (exp. https://app-test.your-domain.com/stage).
If you want to avoid manually adding new DNS records to your Cloudflare provider, this is an example of how we managed to automate adding new records to Cloudflare.

resource "null_resource" "get_api_gateway_endpoint" {
  triggers = {
    template   = sha1(file("../../../backend/template.yaml"))
    stack_name = local.stack_name
  }
  provisioner "local-exec" {
    interpreter = ["bash", "-c"]
    command     = "aws cloudformation describe-stacks --stack-name ${self.triggers.stack_name} | jq '.Stacks[0].Outputs[0].OutputValue'| sed 's/\"//g' > ${path.module}/gateway_endpoint.txt"
  }
  depends_on = [null_resource.build_deploy_sam_resource]
}

In this part, we used null_resource to call aws command aws cloudformation describe-stacks to get API Gateway domain name (we defined Output section in the AWS SAM template as what it should look like) and store it in file. With sed command we’re stripping away https:// from domain name, and only leaving domain name that we going to add to new DNS record.

data "local_file" "api_gateway_endpoint" {
  filename   = "${path.module}/gateway_endpoint.txt"
  depends_on = [null_resource.get_api_gateway_endpoint]
}

resource "cloudflare_record" "api_gateway_endpoint" {
 depends_on = [data.local_file.api_gateway_endpoint]
 name       = var.subdomain_name_backend
 value      = data.local_file.api_gateway_endpoint.content
 type       = "CNAME"
 proxied    = true
 zone_id    = lookup(data.cloudflare_zones.default.zones[0], "id")
}

Using Terraform resource cloudflare_record, we set the value as content of the file that we generated above (it’s value where our CNAME record will point to), and the name value represents the subdomain that we want to use for our REST API endpoint.

The last resource defined in the SAM template example is our lambda EchoFunction.
Key things to keep in mind is to set up a lambda handler (pointing to Go executable file located in our project structure) and events (defining what kind of type will trigger our lambda).
In this example we are using the API as a trigger for our lambda.

   Type: Api
          Properties:
            Path: /{proxy+}
            Method: any
            RestApiId: !Ref ApiDetails

As you can see, we defined that any path and method can trigger our lambda over API Gateway that we had previously created. With this approach and with using this solution with the Go Echo framework, we reduced the number of lambdas (also with this approach we solved the problem with a hard limit of total size that all lambdas can have). Using the aws-lambda-go-api-proxy with echo framework we defined a REST controller that is managing real paths and methods that our lambda consumes.

Note, if your lambda needs more permissions to access some other AWS resources, you can set up Role property. One way you can create a role with Terraform and then create a property in the SAM template to pass it ARN of the created role and reference it to Role property.
Another way is to create a role inside of a SAM template and reference it to the Role property.

To expose your lambda over API Gateway the last thing to do is to enable, you already guess, CORS. To do so, all you have to is to define CORS section in the SAM template. In our case we set up as globals, so our API Gateway will automatically have CORS enabled.
Even with enabled CORS on API Gateway, you still have to handle in your lambda CORS headers, otherwise invocation of your lambda will not work.

With the Echo framework we created CORS middlerware to make our lambda functional.
Here is an example how to do it:

e := echo.New()
e.Use(middleware.CORSWithConfig(middleware.CORSConfig{
AllowOrigins: []string{"*"},
    AllowMethods: []string{echo.GET, echo.PUT, echo.POST, echo.DELETE, echo.OPTIONS},
    AllowHeaders: []string{"Accept", "Content-Type", "Content-Length", "Accept-Encoding", "X-CSRF-Token", "Authorization"},
    }))

AWS Lambda

Lambda is the Serverless computing platform service provided on AWS. Using a Lambda function you can run or execute your application code without actually provisioning any application servers.

Since December of 2020, AWS has increased memory that can be assigned to lambda function to max 10GB and increased the number of vCPU to allocate to max 6 vCPU.
Also a new cool feature is that you can now package and deploy Lambda functions as container images of up to 10 GB in size. If you have some big lambda function with huge dependencies, you can then easily use this approach to solve the situation where your lambda handler can have a max file size of 250MB.

The way we approach this hard limit, we used the aws-lambda-go-api-proxy solution to packt multiple lambas into one. Basically this solution allows us to use one lambda to handle multiple methods or paths for one API Gateway, so we don’t need to have multiple lambdas for each path or method. It makes it easier to maintain a code base.

Very cool thing is, if you have already developed your REST API with Echo framework or some other like Gin, Iris, Negroni, Mux, Fiber or Chi and you’re planning to move to serverless (AWS Lambda, GCP Cloud Function, ...) you can easily achieve it by using this solution.
If usage goes up and it becomes price prohibitive, it's a one line change to swap it to a container & run via AWS services like Fargate, ECS.

In the next example we’ll show you how to use the Echo framework with aws-lambda-go-api-proxy to achieve the solution of having one lambda function for multiple paths and methods.
We have defined two paths, /user and /organization, with two supported methods each.

package main

import (
    "context"
    "net/http"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"

    echoadapter "github.com/awslabs/aws-lambda-go-api-proxy/echo"
    "github.com/labstack/echo/v4"
    "github.com/labstack/echo/v4/middleware"
)

type UserData struct {
    UserId      string `json:"userId"`
    DisplayName string `json:"displayname"`
    Status      string `json:"status"`
}

type User struct {
    User UserData `json:"user"`
}

type OrganizationData struct {
    OrganizationId string `json:"organizationId"`
    DisplayName    string `json:"displayname"`
    Status         string `json:"status"`
}

type Organization struct {
    Organization OrganizationData `json:"organization"`
}

var echoLambda *echoadapter.EchoLambda

func init() {
    e := echo.New()
    //define routes
    registerUserRoutes(e)
    registerOrganizationRoutes(e)

    //define CORS
    e.Use(middleware.CORSWithConfig(middleware.CORSConfig{
        AllowOrigins: []string{"*"},
        AllowMethods: []string{echo.GET, echo.PUT, echo.POST, echo.DELETE, echo.OPTIONS},
        AllowHeaders: []string{"Accept", "Content-Type", "Content-Length", "Accept-Encoding", "X-CSRF-Token", "Authorization"},
    }))
    e.Use(middleware.Logger())

    echoLambda = echoadapter.New(e)
}

func handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    return echoLambda.ProxyWithContext(ctx, req)
}

func main() {
    lambda.Start(handler)
}

func registerUserRoutes(e *echo.Echo) {
    user := e.Group("/user")
    user.GET("", getUser)
    user.POST("", createUser)
}

func registerOrganizationRoutes(e *echo.Echo) {
    organization := e.Group("/organization")
    organization.GET("", getOrganization)
    organization.PUT("", updateOrganization)
}

func getUser(c echo.Context) error {
    return c.JSON(http.StatusOK, User{
        User: UserData{UserId: "1", DisplayName: "Test", Status: "active"},
    })
}

func createUser(c echo.Context) error {
    return c.JSON(http.StatusOK, User{
        User: UserData{UserId: "2", DisplayName: "Test2", Status: "created"},
    })
}

func getOrganization(c echo.Context) error {
    return c.JSON(http.StatusOK, Organization{
        Organization: OrganizationData{
            OrganizationId: "1",
            DisplayName:    "Test organization",
            Status:         "active",
        },
    })
}

func updateOrganization(c echo.Context) error {
    return c.JSON(http.StatusOK, Organization{
        Organization: OrganizationData{
            OrganizationId: "1",
            DisplayName:    "Test organization",
            Status:         "updated",
        },
    })
}

This simple lambda function is returning a JSON response in case you have sent a request to the correct path using the correct method.

One cool way of using AWS Lambda is for running migration codes. AS mentioned earlier, we are using GORM as an ORM library for modeling and interacting with RDS-based Postgres.
GORM has the ability to define data models which represent database tables with appropriate constraints, which allows us to have a dedicated Lambda that runs migration/creations of resources (such as tables, indexes, constraints, pre-populated data, etc) in the database without the need to maintain incremental SQL scripts. If you are curious about GORM checkout their documentation here (bonus: major version 2 was recently released with some very nice improvements).

Local testing

To test our lambda functions locally we’re using Docker and AWS SAM.
As we’re using Go runtime environment for our lambda functions, we have defined Docker file with Go base image and also installed awscli and asw-sam-cli tools.

FROM golang:1.15

RUN apt-get update
RUN apt-get install python3 python3-pip -y

RUN pip3 install --upgrade pip
RUN pip3 install awscli
RUN pip3 install aws-sam-cli==1.12

WORKDIR /var/opt

EXPOSE 3003

Next step is to create a docker-compose service that will use our docker base image defined in Dockerfile, and serve our Lambda function with an API Gateway-like local deployment.

In docker-compose service we’re mounting our backend folder which includes SAM template.yaml and the lambda executable. To run a lambda function with API Gateway defined in template.yaml, all you have to do is to define an entrypoint to use SAM command sam local start-api.

version: "3.5"

services:
  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    image: backend:local
    hostname: backend
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./backend:/var/opt
    working_dir: /var/opt
    environment:
      # don’t share statistics usage with AWS
      SAM_CLI_TELEMETRY: 0
      LOG_LEVEL: DEBUG
    ports:
      - "3003:3003"
    networks:
      - sample
    entrypoint:
      - /bin/bash
      - -c
      - |
        sam local start-api \
          --host 0.0.0.0 -p 3003 \
          --docker-volume-basedir "$PWD"/backend \
          --docker-network sample

networks:
  sample:
    name: sample

To start local development api, just use command docker-compose up, or our make command make up-be.

The API is now ready to be tested.

If you try to run a simple curl command curl http://localhost:3003/user you will get a response from the api.

Cool thing about using SAM local command in combo with docker-compose, is that you can easily define new docker-compose services like postgres, localstack (simulate other AWS services) or frontend and connect your lambda function to those services.
This approach allows us to have a full local development cycle without the need to deploy to AWS to test.

Terraform

Terraform is a general purpose infrastructure as code tool that can create infrastructure on different cloud providers like AWS, GCP, AZURE and Oracle Cloud Infrastructure.
Besides support for cloud providers, terraform has support for plenty of different kinds of providers, such as Cloudflare, Helm, etc.
If interested, you can find a full list of supported providers here.
Having the very useful modules system, it's relatively straightforward to deploy large infrastructures to any kind of cloud provider.

We start with Terraform, as it is the glue for all these services and frameworks; not just because we’re big fans of it. In practice we are stitching AWS and Cloudflare together; and, in a clunky way, AWS SAM. Yes, by far not our favorite solution, using null_resource.

At the moment of writing this post there is no better option to run AWS SAM via Terraform, or at least we couldn’t find one. Hoping in future Terraform will support creating SAM resources directly within a native provider.

To glue things up, we go with the previous release of Terraform, at the moment of writing it was version 0.12. To find out some cool features that were released with this version or newer versions, you can read about in our blog post here.

If you take a closer look at our architecture, you’ll notice that Terraform is responsible directly or indirectly for managing all resources that run on AWS.
It’s directly responsible for creating AWS VPC, AWS RDS, AWS Cognito, AWS S3 buckets, AWS DynamoDB, AWS CloudFront distribution, AWS SSM and Cloudflare DNS records.
Indirectly, it’s responsible for creating AWS Lambdas, AWS API Gateway. Meaning that the creation of those resources is managed by AWS SAM cli, defined by SAM template via a null_resouce.

To manage different stages, we’re using the Terraform workspaces. We choose a strategy of naming Terraform workspaces by names of our branches, allowing us to have simple flow, mostly because of managing resources by Gitlab CI/CD.

Below follows an example of how you could deploy SAM resources using a null_resource within your Terraform scripts. Essentially, you need to run both package and deploy commands and ensure all parameters (e.g. VPC, SGs, etc.) are injected into the deployment. To ensure the resource is called when the dependent files change, one could use the sha1 of template as a trigger (as shown).

resource "null_resource" "build_deploy_sam_resource" {
  triggers = {
    s3_bucket     = var.s3_bucket_artifacts
    template      = sha1(file("../../../backend/template.yaml"))
    stack_name    = local.stack_name
    sec_group_ids = join(",", aws_security_group.lambda_sg.*.id)
    subnet_ids    = join(",", var.private_subnet_ids)
    domain_name   = "${var.subdomain_name_backend}.${var.domain_name}"
    target_stage  = "dev"
  }
  provisioner "local-exec" {
    interpreter = ["bash", "-c"]
    command     = <<-EOT
      sam package --template-file ${local.template} --s3-bucket ${self.triggers.s3_bucket} --output-template-file ${local.packaged}
      sam deploy --stack-name ${self.triggers.stack_name} --template-file ${local.packaged} --capabilities CAPABILITY_IAM --parameter-overrides TargetStage=${self.triggers.target_stage} DomainName=${self.triggers.domain_name} VPCSecurityGroupIDs=${self.triggers.sec_group_ids} VPCSubnetIDs=${self.triggers.subnet_ids} AcmCertificateArn=${aws_acm_certificate_validation.backend.certificate_arn}
    EOT
  }
  depends_on = [aws_security_group.lambda_sg, aws_acm_certificate_validation.backend]
}

Cron jobs

Another use case we needed to implement required some business logic to run on a schedule, which you can also create easily with SAM - which will create a CloudWatch alarm to trigger the appropriate Lambda. AWS supports two ways of defining scheduled time to trigger some AWS resources. One is to use cron expression - in format cron(Minutes Hours Day-of-month Month Day-of-week Year). Second one is to use rate expression - in format rate(Value Unit), Where Value is a positive integer and Unit can be minute(s), hour(s), or day(s).
More details on how to define schedule expressions can be found on the AWS site.

Keep in mind that rate expression is being executed when a rule is created. For example, if you set up a scheduler to run every 1 hour using keyword rate, rate(1 hour), the next run time will be the time when your rule was created plus 1 hour. Let's say you created a rule at 10:20 AM, so the next run will occur at 11:20 AM.

An example of how to define scheduled resources within the SAM template.

  ScheduledLambda:
    Type: AWS::Serverless::Function
    Properties:
      Role: !Ref Role
      Timeout: 30
      Handler: bin/scheduled_lambda
      Events:
        ScheduleExample:
          Type: Schedule
          Properties:
            Schedule: cron(0/30 * * * ? *)
            Description: "Example of cron jobs that runs lambda every 30 min"

The most crucial part to define scheduler is assigning type as Scheduler and setting valid cron or rate values for Schedule properties.

 Events:
        ScheduleExample: #custom name of event
          Type: Schedule #mandatory so AWS SAM can know what kind of event this gonna be
          Properties:
            Schedule: cron(0/30 * * * ? *)

Inside the Schedule property you can define your own expressions.
Here is the list of all properties that you can assign to the cloudwatch event resource in the SAM template.

AWS SSM Sessions

After setting up our environment on AWS, we were looking at how to access a database located in a private network. We found System Manager remote sessions service is a very nice solution, because it allows us to have access to private networks without exposing any service publicly. This is great for troubleshooting purposes, accessing our database.
Also allowing us easily create a cron job for backuping databases or any kind of extra work that is required for maintenance.

To be able to create SSM sessions/tunnels, you need to create an EC2 instance running the SSM Agent. Note that SSM Agent is preinstalled, by default, on Amazon's AMIs. In our case,we used Amazon Linux 2 for convenience Of course, in our case, we are using Terraform to spawn an instance.

In the following example, we’ll show you how to create this subset of our stack, including the installation of a Postgres client in the EC2 instance.

locals {
  // internal user used for admin maintenance tasks
  ssm_instance_user  = "example"
  ssm_instance_group = "example"
}

/**
  * SSM instance security group
*/
resource "aws_security_group" "ssm_instance" {
  vpc_id      = var.vpc_id
  name        = "ssm-sg"
  description = "Allow egress from SSM Agent to Internet."
  tags = {
    "Name" = "ssm-sg"
  }
}

resource "aws_security_group_rule" "ssm_instance_allow_egress_https" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "TCP"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = aws_security_group.ssm_instance.id
}

resource "aws_security_group_rule" "ssm_instance_allow_egress_http" {
  type              = "egress"
  from_port         = 80
  to_port           = 80
  protocol          = "TCP"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = aws_security_group.ssm_instance.id
}

data "aws_iam_policy_document" "ssm_instance_default" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ssm_instance" {
  name                      = "ssm-iam-role"
  assume_role_policy = data.aws_iam_policy_document.ssm_instance_default.json
}

resource "aws_iam_role_policy_attachment" "ssm_instance_policy" {
  role           = aws_iam_role.ssm_instance.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

resource "aws_iam_instance_profile" "ssm_instance" {
  name  = "ssm-iam-instance-profile"
  role    = aws_iam_role.ssm_instance.name
}

data "template_file" "install_ssm_instance" {
  template = file("install-ssm-instance.yaml")
  vars = {
    user  = local.ssm_instance_user
    group = local.ssm_instance_group
  }
}

/*
 * Ref: https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/cloudinit_config
 */
data "template_cloudinit_config" "ssm_instance_config" {
  part {
    content_type = "text/cloud-config"
    content      = data.template_file.install_ssm_instance.rendered
  }
  // cloud-init has apparently a 8 year unresolved issue (face-palm) [1], and is unable
  // to create users before write_files directive . Thus, we use this hack.
  // [1] - https://bugs.launchpad.net/cloud-init/+bug/1486113
  part {
    content_type = "text/x-shellscript"
    content      = "/usr/bin/install-pg.sh"
  }
}

resource "aws_instance" "ec2" {
  ami                    = "ami-0d712b3e6e1f798ef"
  instance_type          = "t2.micro"
  subnet_id              = var.private_subnet_ids[0]
  vpc_security_group_ids = [aws_security_group.ssm_instance.id]
  iam_instance_profile   = aws_iam_instance_profile.ssm_instance.name

  root_block_device {
    delete_on_termination = true
    volume_type           = "gp2"
    volume_size           = 20
  }
  user_data = data.template_file.install_ssm_instance.rendered
  tags = {
    "Name" = "ssm-ec2"
  }
}

Example of template install-ssm-instance.yaml

#cloud-config

# See docs for more details: https://cloudinit.readthedocs.io/en/latest/topics/examples.html

# Upgrade database on first boot (run 'apt-get upgrade').
package_upgrade: true

users:
  - default
  - name: ${user}
    gecos: ${user}
    shell: /bin/bash
    primary_group: ${group}
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    lock_passwd: false

# download & install following packages
packages:
  - curl

write_files:
  - permissions: '0750'
    owner: root:root
    content: |
      #!/bin/bash
      set -euo pipefail
      tee /etc/yum.repos.d/pgdg.repo<<EOF
      [pgdg12]
      name=PostgreSQL 12 for RHEL/CentOS 7 - x86_64
      baseurl=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-7-x86_64
      enabled=1
      gpgcheck=0
      EOF
      yum makecache
      yum install -y postgresql12 postgresql12-server
      chown ${user}:${group} -R /home/${user}/
    path: /usr/bin/install-pg.sh
final_message: "The system is finally up, after $UPTIME seconds"

After setting up an EC2 instance with the installed postgres client, the next step is to connect to the EC2 instance, using the following command.

aws ssm start-session --target “instance name” - instance name is you EC2 instance name

In case you are having problems remembering the EC2 instance name, you can easily use make command to make it more straightforward.

Basically everything that you need is EC2 tag value, so using aws cli command to describe EC2 instance filtering by tag will get us the desired instance name.
After everything is ready, to get connected to your EC2 instance, all you have to do is run make command.
In our example we are calling that command ssm-instance, so running make ssm-instance, you will get to the place you wanted to be :).

Here is an example how can you achieve it using make:

SSM_TAG=ssm-ec2
SSM_INSTANCE := $(shell aws ec2 describe-instances --filter "Name=tag:Name,Values=$(SSM_TAG)" --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --output text)

.PHONY: ssm-instance
ssm-instance:
    @echo Connecting to SSM INSTANCE
    @aws ssm start-session --target $(SSM_INSTANCE)

The last thing is to verify if the install-pg.sh script has been created on your EC2 instance by the Terraform example above.

Just type sudo -s in the terminal and run command ./install-pg.sh. Now your postgres client should be downloaded and installed. Voila!
To verify that you have successfully installed postgres client, just run command psql.

In this example you saw how you can use AWS SSM and the EC2 running instance to get deployed to a private VPC.

Cost

If you are a startup at an early stage, then you should definitely consider applying to the AWS startup program as early as possible. While an AWS VPC itself doesn’t cost anything, it does cost to have the bare minimum resources for it to be production ready (e.g. Private/Public subnets with NAT Gateway). We have set up a VPC with a single NAT and internet gateway as a shared resource for our multiple stages (development, integration, production). We agree it’s not ideal, but for projects at early stages, we were avoiding escalation on expenses - we can all agree on this :).

Besides the NAT Gateway, our biggest contributors to cost are of course the encrypted RDS instances db.t2.small.

To sum up, this basic (but very functional) setup costs us around 100´euros per month, which is very fair considering we have multiple stages and the encryption used for all possible resources (RDS, S3, secrets).

Cloudflare

There is a bit of a challenge using Cloudflare strict mode with frontend deployed on AWS S3 bucket. Cloudflare's strict mode requires that the origin server needs to have an SSL certificate and in case of using AWS S3 as hosting there is a bit of a problem.
As you might know, AWS S3 bucket is publicly exposed only on HTTP protocol.

So to overcome this issue, we needed to create an SSL certificate on AWS Route53 associated with our publicly exposed AWS S3 bucket, alongside with AWS Cloudfront distribution applying that SSL certificate. Basically AWS Cloudfront distribution allows us to expose AWS S3 buckets over HTTPS protocol.

In this example we are creating an ACM certificate for a domain that will be used for our frontend exposed via Cloudfront distribution.

provider "aws" {
  alias = "virginia"
  region = "us-east-1"
}

This provider in region us-east-1 needs to be defined so an ACM certificate can be created.

resource "aws_acm_certificate" "default" {
  provider          = aws.virginia
  domain_name       = "${var.subdomain_name}.${var.domain_name}"
  validation_method = "DNS"

    lifecycle {
    create_before_destroy = true
  }
}

data "cloudflare_zones" "default" {
  filter {
    name = var.domain_name
  }
}

resource "cloudflare_record" "validation_domain" {
  name    = aws_acm_certificate.default.domain_validation_options[0]["resource_record_name"]
  value   = trimsuffix(aws_acm_certificate.default.domain_validation_options[0]["resource_record_value"], ".")
  type    = aws_acm_certificate.default.domain_validation_options[0]["resource_record_type"]
  zone_id = lookup(data.cloudflare_zones.default.zones[0], "id")
  depends_on = [aws_acm_certificate.default]
}

This resource will create CNAME records in cloudflare DNS to validate our ACM certificate.

resource "aws_acm_certificate_validation" "default" {
  provider                = aws.virginia
  certificate_arn         = aws_acm_certificate.default.arn
  validation_record_fqdns = cloudflare_record.validation_domain.*.hostname
}

Note that certificate validation can take up to 20min, so have some patience.

The next part is an example of creating a Cloudfront distribution and exposing it’s generic domain name as a user-friendly one that we will assign in Cloudflare.

resource "aws_cloudfront_distribution" "frontend" {
  enabled         = true
  aliases         = ["${var.subdomain_name}.${var.domain_name}"]
  is_ipv6_enabled = true
  // cheapest: https://github.com/laurilehmijoki/s3_website/issues/150
  price_class = "PriceClass_100"

  default_cache_behavior {
    allowed_methods        = ["GET", "HEAD", "OPTIONS"]
    cached_methods         = ["GET", "HEAD"]
    target_origin_id       = var.frontend_s3_origin_id
    viewer_protocol_policy = "redirect-to-https"
    default_ttl            = 0
    max_ttl                = 0

    forwarded_values {
      query_string = false
      cookies {
        forward = "none"
      }
    }
  }

  origin {
    domain_name = var.frontennd_s3_origin_domain_name
    origin_id   = var.frontend_s3_origin_id

    custom_origin_config {
      http_port                = 80
      https_port               = 443
      origin_keepalive_timeout = 5
      origin_protocol_policy   = "http-only" // setting defined after terraform import. can try with https-only
      origin_read_timeout      = 30
      origin_ssl_protocols     = ["TLSv1", "TLSv1.1", "TLSv1.2"]
    }
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn            = aws_acm_certificate_validation.default.certificate_arn
    cloudfront_default_certificate = false
    minimum_protocol_version       = "TLSv1.2_2019"
    ssl_support_method             = "sni-only"
  }
}

resource "cloudflare_record" "frontend_service" {
  name    = "${var.subdomain_name}.${var.domain_name}"
  value   = aws_cloudfront_distribution.frontend.domain_name
  type    = "CNAME"
  proxied = true
  zone_id = lookup(data.cloudflare_zones.default.zones[0], "id")
}

After successfully creating those resources, your frontend will be now available over the internet on domain app-test.your-domain.com. Ta-da! We are now happy, having everything set up on HTTPS. Hurray!

Also worth noticing it that at some point AWS marked cloudflare as non-trusted authority (there are references to Mozilla’s authorities list, more can be found here), so resources that we found online to solve our issue with exposing AWS S3 bucket over SSL weren’t helpful.
Those solutions were suggesting to use Cloudflare SSL certificates and import them into AWS certificate manager and use it within AWS Cloudfront distribution.

Gitlab runners

We used Gitlab CI/CD to power up our Terraform scripts to automatically handle deploying on different environments like test, staging and production. To set up the Gitlab runners to run successful workflows, we simply ensure that AWS cli, SAM cli, Terraform and Make are available/installed during the execution. We are aware this step is highly improvable - we could have a prebuilt Docker image containing all of these or use a community maintained one, but we wanted to first focus on product use cases and later focus on improvements.

Here is an example of how to set up a Gitlab project with gitlab-ci.yaml. The most important part of this example is in before_script section of gitlab-ci.yaml file.

workflow:

variables:
  AWS_DEFAULT_REGION: "eu-west-1"
  WORKSPACE: "dev"
  TF_ENVIRONMENT: "terraform"

stages:
  - terraform-apply

terraform-apply:
  image: google/cloud-sdk:slim
  stage: terraform-apply
  before_script:
    - apt-get install -y unzip make jq
    - curl https://releases.hashicorp.com/terraform/0.12.29/terraform_0.12.29_linux_amd64.zip --output /tmp/terraform.zip
    - unzip /tmp/terraform.zip -d /tmp
    - chmod +x /tmp/terraform
    - mv /tmp/terraform /usr/local/bin/
    - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.0.30.zip" -o "awscliv2.zip"
    - unzip -q awscliv2.zip
    - ./aws/install
    - aws sts get-caller-identity
    - curl --location "https://github.com/aws/aws-sam-cli/releases/download/v1.18.1/aws-sam-cli-linux-x86_64.zip" -o "awssamcli.zip"
    - unzip -q awssamcli.zip
    - ./install
    - sam --version
    - curl --location https://github.com/terraform-linters/tflint/releases/download/v0.21.0/tflint_linux_amd64.zip -o /tmp/tflint.zip
    - unzip /tmp/tflint.zip -d /tmp
    - chmod +x /tmp/tflint
    - mv /tmp/tflint /usr/local/bin/
  script:
    - echo yes | make tf-apply

Final thoughts

Overall we are pretty satisfied with the usability of AWS framework for serverless environments.
The major drawback with this solution that we want to make sure we highlight is the usage of null_resource to integrate with AWS SAM. Terraform null_resources is something everyone should try to stay away from, so we sincerely hope to see future integration of SAM in the AWS provider.
And that is it. Thank you for reading, we hope this has been useful. Feel free to reach out to us if you have questions or suggestions.

Once again, you can find all the code supporting this post here.

(Slightly) Quicker PySpark Tests

João Ferrão — Fri, 11 Jun 2021 21:27:11 +0000

First of all, let me start by stating this isn't a post about the usage of fancy new technology but rather about sharing how to slightly improve the timing of automated pipelines that require testing PySpark.

Recently I needed to implement an automated workflow, used in multiple git repositories which required pyspark tests to run at every commit made on an open pull/merge request as well as after merge (because, you know... we love tests).

As usual, the team was on a rush to put together a lot of different elements simultaneously but, in the spirit of Agile development, we wanted to make sure that (1) everybody understood how to run the tests and (2) that experience was (could be) the same between running them on a machine or in our CICD pipeline.

If anyone has experience with (Py)Spark, there's a lot of common mishaps that you'll know how to handle, but if you have newcomers to the technology, it's sometimes hard to identify some of the aforementioned issues, such has: wrong java version installed/selected, wrong python version, input test file doesn't exist, etc. If there's a small change to introduce to the business logic, it should be easy to know if (and where) the intended use of the logic was broken.

Originally, we ended up this very practical Dockerfile and docker-compose.yaml, which installed whatever requirements.txt we placed in the root of the repo and ran all the tests.

FROM openjdk:8
ARG PYTHON_VERSION=3.7
ENV PATH="/root/miniconda3/bin:${PATH}"

# provision python with miniconda
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh
    && conda install python=$PYTHON_VERSION -y \
    && conda init bash

RUN . /root/.bashrc

We placed it in tests/Dockerfile and then the following docker-compose.yaml in the root of the repo.

version: "3.0"

services:
 test:
    build:
      context: .
      dockerfile: ./tests/Dockerfile
      args:
        PYTHON_VERSION: 3.7
    working_dir: /app
    volumes:
      - .:/app/
    command:
      - /bin/bash
      - -c
      - |
        pip install pytest
        pip install -r ./requirements.txt # including pyspark==2.4.2
        pytest ./tests

Then, you can just run docker-compose up test either on the designated CICD pipeline step and see the results. At least, there's no excuse for anyone to say they can't run the tests.

But waiting for this on every commit...?

If you try to run all of this, you'll notice that just provisioning Java, Python and then PySpark itself takes something between 2-4 minutes, depending on your CICD engine muscle and network bandwidth. Doesn't seem very significant, but if you run such logic every commit and that is required to pass before a Pull Request can be merged, might become relevant over time.

For this reason, we created a public image on Docker Hub, which you can use in your project and which is by no means rocket science but will hopefully be as convenient for your as it was for us.

Yeah, but there's jupyter pyspark notebook images out there with the similar pre-installed stack.

Right, right... But we wanted (1) something stripped down of unneeded packages (2) the ability to choose specific combinations of Python and PySpark versions per docker image and a (3) smaller download footprint - the images have almost half the size when decompressed (1.7 Gb vs 3.3 Gb).

Slightly quicker

The quick improvement would then be to simply reference directly the mklabsio/pyspark:py37-spark242 image in your docker-compose.yaml, get rid of the Dockerfile and leave everything as before.

version: "3.0"

services:
 test:
    **image: mklabsio/pyspark:py37-spark242**
    working_dir: /app
    volumes:
      - .:/app/
    command:
      - /bin/bash
      - -c
      - |
        pip install pytest
        pip install -r ./requirements.txt # including pyspark==2.4.2
        pytest ./tests

We also took the opportunity to create a simple GitHub Actions workflow to build all the required combinations of Python, Java and PySpark.

Hope this could help you in any way and if you see a need for improvement, let us know in the comments below or feel free to open an issue in the dedicated GitHub repository:

Terraforming in 2021 – new features, testing and compliance

diogoaurelio — Sun, 02 May 2021 17:07:29 +0000

Both João and I have been using regularly terraform for some time now, and are big fans of it. Big enough to even have worked for quite a while on a side project of our own to simplify managing terraform environments (which unfortunately did not succeed), at least. And yes, now that we are preparing the next one with mklabs, which naturally also heavily relies on terraform scripts, it made only sense to start by sharing some of our favorite tools that gravitate in the terraform ecosystem.

We kick off with the latest updates in terraform itself, from 0.12 until the latest 0.15, and then go straight to surrounding tooling, where we mainly focus on testing and compliance checking for infrastructure.

You can find all the code supporting this this post here.

Overview of latest Terraform versions

Version 0.15 has just recently been released. Yet, a lot of companies out there are still running environments up to 0.11.X, and for a good reason. Though version 0.12 was launched already a while ago (2019), it brought great disruption. If this is your case, chances are that you gave up on keep up to date with its progress. Thus, we thought it would make sense to first review some of terraform features you might be missing out, before we try to convince you also considering further improvements.

0.12

First-class expression syntax: probably the most noticeable change, no more string interpolation sintax (confettis blowing in the background)!
Generalized type system: (yay!) able to specify types of variables;
Iteration constructs: introduction a the for operator allowing closer proximity to programming languages and thus more expressiveness to the DSL;

More details, see the general availability announcement page.

0.13

Module improvements - brings ability to use several meta tags that until so far were only available for resources in modules. In this case, we can now use depends_on for coupling dependencies with modules (finally!), along with ability instantiate multiple module instances with count or for_each; more details here;

Example specifying a module dependency (a personal favorite), source hashicorp blog

Custom variable validation - when specifying variables one can from this version on wards specify custom rules for the input that is accepted and respective error message to allow fast failure; more details here;

Example variable validation, source hashicorp blog

Support 3rd party providers - terraform now allows one to reference your own provider in the terraform block required_providers. The required_providers keyword in the terraform block already existed in terraform 12, though with was restricted to hashicorp's own providers; now you can reference your own DNS source to host your own registry, and upon terraform init it will be installed same as it used to all other providers; more details here; note that if you are already using the required_providers keyword in terraform block, to migrate from version 12 to 13 you should adapt it, as shown in the following example:

Example required_providers, source terraform upgrade guides

More details, see the general availability announcement page.

0.14

Sensitive variables & outputs - allowing one to flag a variable as "sensitive" will redact its output on the CLI;
Improved diff - this is probably one of the most common complaints I hear, namely the difficulty in reading a terraform plan (and along with it the same for apply and show) output; with this new change, it hides unchanged fields (irrelevant for the diff) while displaying a count of the hidden elements for better clarity; more details here;

Example excerpt diff, source hashicorp blog

dependency lockfile - as soon as you run terraform init with version 14 you will notice that a .terraform.lock.hcl file will be created in that directory and outside the .terraform directory. This file is intended to be added to version control (git committed), to guarantee that the exact same provider versions are run everywhere;

More details, see the general availability announcement page.

0.15

Finally, here are some of the main highlights the latest version just very recently announced:

Remote state data source compatibility: in order to make it easier to upgrade to newer versions, you can use reference remote state data objects that are using older versions; note that this feature has been back ported into previous releases as well, namely 0.14.0, 0.13.6, and 0.12.30;
Improvements to conceal sensitive values (passwords, for example): following version 0.14 work now provider developers can specify the properties that are by default sensitive and should be anyway hidden in outputs; moreover, terraform also added a function (sensitive) for users to be able to explicitly hide values;
Improvements in logging behavior: ability to control provider and terraform log level separately TF_LOG_CORE=level and TF_LOG_PROVIDER=level.

More details, see the general availability announcement page.

Handling multiple versions

Assuming that these new features convinced you to upgrade your existing terraform environments, being realistic here, this will not happen from one day to another. You will have a transition period (if not permanently) where you have environments with different terraform versions. That is OK, you can still keep your sanity while hopping between all of them thanks to tools like the following:

TFEnv - terraform environment switcher inspired (from the ruby world) by rbenv written with shell scripts;
Terraform Switcher - yet another project essentially doing the same written in go;

Both of these projects overlap almost entirely, so we will simply exemplify with one of them, namely tfenv:

Here we are showing how you can switch between installed versions tfenv use <local-version>, how you can check which versions are already locally installed with tfenv list, all versions currently available tfenv list-remote (minor detail: the current version of the library I'm using to record my terminal, terminalizer, does not capture me scrolling up and selecting version terraform 0.14.5)
Last but not least, we also show a cool feature from tfenv, namely the ability to automatically recognize the minimum required version in a given environment. Same goes for the latest version, in case you are wondering. And yes, this is also available in terraform switcher project.

Testing

Testing is probably the most confusing topic in the Infrastructure-as-Code (IaC) land, and terraform not being an exception, as a lot of different tools and procedures get thrown in this same bag, when, well, they probably should not. Usually when talking about testing, people usually mean three different things:

static checks - validation of mainly the structure code without actually running the code; a fast away for performing sanity checks, either local and/or in your CI/CD pipelines;
integration tests - provided you are properly using inversion of control with variables, they give you the power to test your modules or environments for different generic cases;
compliance checks - tests done in the aftermath, after all resources have already been deployed; these can serve distinct purposes that we will explore better later, but essentially the goal is to confirm that what is deployed is what was initially intended and expressed in terraform;

Yes, you got that right: if you were looking forward to some classic unit testing, you can forget that for now. Let us dive straight into more details.

Static checks

Again, these are not tests strictly speaking, but rather just simple validation checks one can run to catch common errors without actually deploying anything. There is an array of tools out there, but let us start with the one provided out of the box in terraform binary.

It is not uncommon for the tool creators to provide their own validators. Kubernetes provides validate kubectl apply -f <file> --dry-run --validate=true, helm provides lint helm lint <path>; terraform provides validate.

In the following example running terraform validate would catch two of the three issues:

Validate will require to have previously run terraform init, so that it can leverage providers. In our example it will detect the typo in the aws bucket resource reference, along with the invalid CIDR provided to create the VPC. What it will not detect, however, is the non existent EC2 instance type.

TFLint comes to the rescue. Being yet another open source tool written in go, it comes as a binary much like terraform and does not even require terraform to be installed.

Running with the deep option requires one to provide credentials, providing a more thorough inspection. In our case, it detected the incorrect instance type, as well as the wrong AMI.

You can also customize tflint to inject variables, define modules to ignore, etc. You can check the user guide for more details.

Now besides linting for configuration issues in your code, another recommendation is to check for security issues, such as too permissive security group rules, unencrypted resources, etc.

Here again more than one tool exists to assist. We will highlight two of the most popular ones here: tfsec and checkov. Both provide a predefined set of checks that they use to inspect your code, allowing to explicitly open exceptions (if you really want to) by annotating your code with comments, and adjust the configuration to ignore some modules, for example.

TFSec is written in Go, and is probably the fastest to get started, and currently provides up to 10 checks for the current main cloud providers (AWS, GCP, and Azure). The potential downside is that works exclusively for Terraform, so you will need to use additional tools to inspect kubernetes/helm/cloudformation etc.

Checkov, on the other hand, despite being a more recent tool, has seen stellar development speed (being developed by a startup with good founding rounds, and PaloAlto Networks acquisition can't hurt). Not only do they have a really comprehensive number of checks across all the main cloud providers, but they also span across multiple technologies, such as Kubernetes, Cloudformation, serverless and ARM templates. And the list keeps growing. Checkov provides you the option to run either a pure static check by just pointing to the terraform directory or terraform file, or by actually running it against a terraform plan file. The nice thing about running it directly is naturally the simplicity of not requiring to have the target environment accessible to test the code.

These tools are grabbing a lot of attention lately, as the double checking for security issues was usually locked in the hands of devops/devsecops teams, which in practice constituted a development bottleneck. By injecting these checks early in CI/CD pipelines, a great deal of development speed is freed without compromising security.

Integration tests

Terratest is probably the closest one can get now a days to testing the specific peace of terraform code. It is a Go library, and requires one to write tests in Go. This is obviously a potential limitation as not all teams have knowledge in Go. On the upside, I would argue that the learning curve of learning Go to get the basics - read enough for writing terraform tests - is not steep if you know already at least one programming language.

Having worked already in several Go projects in the context of mklabs, we're naturally favorably biased towards Terratest. So, even if you have never tried Go, we would still recommend having a look on our sample repository, where we provide go mod setup to make it easy getting started. And if this did not convince you yet, here's what might: you can also use Terratest to test Dockerfiles, Kubernetes and Packer setups.

This might seem intimidating, but we would argue that the benefits are worth it. Let us look at an example skeleton of a test setup for AWS:

Essentially the skeleton is always the same:

start by defining the variables you want to inject as input for your terratest run; you might want to rather inject random variables even, to test in greater depth;
make sure you tell the setup to destroy all created resources after the terraform apply has been completed using the defer statement; this is equivalent to the trap keywork in shell, and it will execute even if something fails in the meanwhile; the only prerequisite is that you declare it before you call terraform.InitAndApply() method;
Let the test be deployed by passing the terraform config setup you have previously declared in terraformOptions to terraform.InitAndApply method;
Finally, declare the things you want to assert; that is, declare what you expect should have been deployed - the expected - and see if they match what was actually deployed - the result. The simplest way is to check in terraform apply output, which is accessible via `terraform.Output()` method. Alternatively, terragrunt also provides you some methods out of the box for frequently checked things on a per provide basis.

Sounds like fun, right? Here is an example of some of the tests we could write to our basic terraform example:

Here is the output of running these tests for our demo setup:

Final thoughts regarding terratest: we find terratest a great tool to implement changes in your infrastructure with confidence, no matter if they are just simple day to day changes, or bigger and complexer upgrades or migrations.

We just scratched the surface here on the tests one can develop with terratest - for example combining SSH access into instances and confirming access to resources, etc. However, there are no free lunches, and this can come at a price: tests can take a long time (depending obviously on what you are testing), and require a non trivial time investment - learning how to use, writing them, and setting up the environment, as you probably will want to run them in an isolated environment. For example, in AWS case, this would ideally be a dedicated account. We recommend reading best practices on how to perform testing from terragrunt, the company behind terratest.

Compliance checks

The last mile is asserting that what you wanted to be deployed, was indeed deployed exactly as you wanted. Abusing terraform null_resource is a classic one leading to unintended surprises. One way of achieving this would be to run these tests right after the terraform apply stage of your CI/CD pipeline.

But the next question you might have is how do I know that these configurations stay that way, that no one changed things inadvertently ? We've seen this situation arise in different forms: changes done by users manually via GUI or CLI; via different terraform environments mutating overlapping resources properties; or as a by-product of using different IaC tools, for example configuring some bits with Cloudformation, some K8s or Helm, etc.

Arguably more interesting would be scheduling to run these checks continuously and repeatedly, to make sure things stay as you expected. Let's see some options out there to achieve this.

The rogue approach: in practice you can do this type of compliance checks with any programming language you favor. As we shown before, one can export terraform' state into json format, and then use, for example, python with pytest and boto3 libraries to compare what is deployed with the desired output. You could even go further, and use boto3 to scan your accounts for different aspects that you considered go against best practices, such as lack of encryption (in-flight and at rest).

While this might be tempting at first sight, you might end up writing way more testing code, than the actual terraform code. Not only it seems kind of silly, but it can get hairy.

The second reason not to follow this approach, is that there are already several solutions out there fairly tested and intuitive to use that can help you with this task, which we will cover now:

Driftctl - open source go library from cloudskiff for terraform;
Sentinel - Hashicorp's own solution;
terraform-compliance - open source BDD based solution dedicated for terraform;
Conftest - test suite for multiple frameworks besides terraform, such as kubernetes and dockerfiles;
Inspec - Chef compliance testing tool, written in ruby;
Built-in cloud provider - each cloud provider has it's own inspections mechanisms in place;

You may be wondering why the first place in our list is a library still in beta. That is a good point, and we would argue that its progress seems to be promising, and that we like to support startups. Moreover we think its simplicity of usage deserves a highlight as it delivers one thing and one thing only: check if what is in your terraform state file is what is actually deployed. Simply point to your terraform statefile when you run driftctl scan command and you will get a detailed report if you have drifted. Due to the low effort required to implement this library and value provided, we think it deserves taking it for a spin.

Next on our list is Hashicorp's (the company behind terraform) own enterprise solution for this, Sentinel. This is could make sense if you are already using other Hashicorp's enterprise functionality, benefiting from Terraform Enterprise.

A direct open source comparable alternative would be using terraform-compliance. It follows BDD directives so that you can specify in an easy human readable way your expectations, using:

given: a given resource type;
when: an optional condition you might want to add;
then: what you expectation is;

Here is an example of a test file for AWS S3 buckets:

Although we do get the appeal of easily understandable tests that do not require knowing how code, we find terraform-compliance lacks flexibility to test various aspects, mainly due to the BDD nature. Moreover, there are more people disenchanted by it with some valid points.

If you like terraform-compliance, Conftest might also be worth having a look. It has its own DSL to write policies, and allows you to test multiple frameworks. We found this blog post from Lennard Eijsackers very informative, and would thus rather recommend you to check it out.

Before we dive into own cloud provider compliance checking services, we want to highlight yet another open source tool, namely InSpec. It allows you to write tests in ruby, and was built on top of RSpec. If you know already awsspec, then this should feel very similar, with the advantage that InSpec also supports GCP and Azure.

Even though none of us is actively a ruby developer, we find InSpec very easy to get started with and, most importantly, very powerful. It allows you to combine IaC checks to target resources deployed and mapped in terraform state files, with other general policies for cloud account configuration. Moreover, it also allows you to combine additional security checks, such as on OS level configurations and services running. Let us illustrate how you could write these checks. The following check illustrates how to define global policies that an account should obey:

The next one illustrates for generic networking:

The previous examples only illustrate generic control checks that validate overall best practices in your acocunt. However, you might want to also perform assertions regarding what is in your terraform state versus what is deployed - what the cloudskiff engineers rightly name drift. Christoph Hartmann - one of InSpecs creators - has a nice blog post explaining how to use InSpec and integrate with terraform. The approach is essentially as described previously in the rogue approach - import terraform state json file and use it as the expected assertion.

Built-in cloud provider policy tools

Each cloud provider has a native tool to address company-wide governance policies. Some examples of services are:

AWS: AWS Config and to manage cloud configuration deviations within one account, Systems Manager to enforce instance OS security, and more recently Control Tower for enforcing rules across multiple accounts;
Azure: Azure Policy for checking cloud configuration deviations and Azure Security Center
GCP: Forseti Config Validator for GCP

These are just some of the services that we know that can be used for such enforcement. Most of these services go beyond just checking cloud config, and also provide security inspections at instance level, for example. The important point to keep in mind is that these security checks are post deployment compliance assertion, not for preventing configuration issues.

Final thoughts

We find that there is not a single silver bullet that solves all problems, and the best strategy is actually a combination of multiple tactics employed at different stages. The two main phases that require different approaches are pre and post deployment.

For pre-deployment, a combination of static checks and actual tests can be used. Static checks are a great starter, as they are easy to setup, and allow you to enforce generally good practices across the whole organization. In other words, static checks provide you an powerful easy win. With terratest, on the other hand, you mainly gain on confidence that the IaC code will actually work, and that you will catch faux pas. However, terratest does not come for free, and does require a learning curve with go, along with time investment to develop the actual tests.

After your code gets deployed comes the next challenge: making sure things stay well architected. Regular scheduled checks that assess if your infrastructure is running as intended should also be part of your devOps strategy, and for this two main routes exist: open source or using dedicated cloud services.

And that is it. Thank you for reading, we hope this has been useful. Feel free to reach out to us if you have questions or suggestions.

Once again, you can find all the code supporting this this post here.

Sources

As usual, here is the summary of sources used for this post:

Terraform version 12 general availability notes
Terraform version 13 general availability notes
Terraform version 14 general availability notes
Terraform Switcher project
Best practices testing with terratest
TFSec
Bridgecrew checkov
Terraform compliance
Driftctl tool
Chef InSpec
Christoph Hartmann blog post how to use Inspec with terraform
AWS Spec
Conftest
Blog post from Lennard Eijsackers on Conftest
The recordings of my terminal were done using terminalizer