DEV Community: diogoaurelio

Dependency Injection in Scala - cake pattern

diogoaurelio — Sat, 18 Feb 2023 13:55:15 +0000

In this post we'll cover Dependency Injection (DI) basics in scala, and focus on how to perform it manually with using the cake pattern.
All code examples are available on github.

Introduction to DI

Dependency injection encourages loose coupling. It tells you that this is wrong:


class MyRepository

class BlueService {
  val a = new MyRepository() 
}

and rather encourages you to instead do:

class MyRepository
class BlueService(a: MyRepository) {}

With this change we're defining explicitly which dependencies the MyService constructor needs, and that those should be injected from outside. The rule of thumb: if you see the new keyword inside a service, that should immediately yield a red flag in your head.

Might not seem much, but this is already a big improvement, as it greatly simplifies the way we can build tests for the MyService class.

Let's do that. So let's start by defining our dependencies (no pun intended) in build.sbt:

libraryDependencies ++= Seq(
  "org.scalamock" %% "scalamock" % "5.2.0" % Test,
  "org.scalatest" %% "scalatest" % "3.2.14" % Test
)

And here a sample layout for our mocking:

import eu.m6r.scalalightningtalks.di.Background.DI2.{BlueService, MyRepository}
import org.scalamock.scalatest.MockFactory
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.must.Matchers

class BackgroundTest extends AnyFlatSpec with Matchers with MockFactory {

  it should "test something BlueService when repository behaves a certain way" in {
    val mockRepository = mock[MyRepository]
    // service under test
    val sut = new BlueService(mockRepository)
    // ...
  }

}

And this way we can easily manipulate the response from MyRepository. If our original implementation was:

    class MyRepository {
      def findOne(): String = "one"
    }
    class BlueService(a: MyRepository) {
      def get(): String = a.findOne()
    }

.. then we could mock the response, for example, as such:

  it should "test something BlueService when repository behaves a certain way" in {
    val mockRepository = mock[MyRepository]
    // service under test
    val sut = new BlueService(mockRepository)
    // mock
    (mockRepository.findOne _)
      .expects()
      .returning("two")
      .once()
    sut.get() shouldBe "two"

  }

However, we're not done yet; consider the following:

trait Repository
class MyRepository extends Repository

class BlueService(a: Repository) {}

By defining a general contract for the Repository (via our trait), we are nicely adding a separation of concerns - in the future we can easily swap to a different concrete Repository implementation (for example, instead using a given DB, using another), without having to change anything in BlueService implementation.

If this is your first time hearing about it, you may be asking yourself: But where then do I instantiate and pass the concrete Repository instance? In other words: Where do I instantiate and inject BlueService dependency?

DI relies on Inversion of Control (IoC) principle, which encourages one to create the concrete service implementations outside of the actual services that use them. This is usually done in a central container, or spread out across multiple centralization containers which determine which dependency is instantiated and gets injected to which class.

Options for DI in Scala

We have several options for DI in Scala:

using libraries from java world, such as Guice;
using pure scala: manually injecting via a central container, with the cake pattern, or with the reader monad;
using scala libraries: MacWire, SubCut, Scaldi

In this post we'll just show case the most basic form of manual DI, and then exemplify the second alternative with the cake pattern.

Manual DI - central container

Let's start with the simplest, and arguably the most frowned upon. If we're being honest here, quite a reasonable one if you're working on a small project.

Just create a main container where your configuration(s) is/are loaded, and passed to create dependencies, and inject them in several services.


trait Repository
class MyRepository extends Repository
class BlueService(repository: Repository)
class GreenService(repository: Repository)

object {
   val myRepo = new MyRepository()
   val blueService = new BlueService(myRepo)
   val greenService = new GreenService(myRepo)
}

The beauty about this solution is two-folded: one it is very simple - it is straight forward, anyone can understand it, there is no magic happening underneath the hood. The second is that we have compile type safety giving us compile time seat-belt for any mistakes.

This has one core drawback, which is that it is not very scalable. It requires tediously declaring all new dependencies, increasing linearly as the project grows, while making sure that the exact instantiation and injection are guaranteed at the right time and place. Eventually we might bump into one of these mistakes:

object {
   val blueService = new BlueService(myRepo)
   val myRepo = new MyRepository()
}

.. and be sure that this will compile, but myRepo will be null when referenced, since it hasn't been initialized (a case of forward reference).

Arguably not the worst thing in the world, and easily solvable by letting the compiler solve it for us with the keyword lazy:

object {
   lazy val blueService = new BlueService(myRepo)
   lazy val myRepo = new MyRepository()
}

However not all is bad; in fact, manual DI does not get nearly as much love as it should. In comparison with most DI frameworks, which use reflection at run-time to dynamically infer the dependency graph, it gives us type-safety at compile time, a slight startup time improvement, and, most of all, removes the magic done behind the scenes. I can't state this enough: no framework magic to fight against (looking at you spring).

Manual DI - Cake pattern

The key building block to implement the cake pattern is to relie on trait mix-in instead of inheritance.
The name cake comes from the idea of multiple layers - as we mix-in multiple traits with each other.
We will dive right next in how the mix-in mumbo jumbo is executed - but just keep in mind that this is how one hooks dependencies, which forces the implementer of the traits to then inject concrete definitions of them.

Let's review some fundamentals about scala traits to make the previous paragraph make any sense.

cake pattern background: traits & diamond problem

The closest thing to a trait in the java world would be an interfaces - more concretely a java 8+ interface. Since java 8, interfaces can even have concrete method implementation using the default keyword, and so can traits. One interface can also, like scala traits, extend more than one interface:


interface Demo1 {}

interface Demo2 {}

// this is fine
interface Demo3 extends Demo1, Demo2 {}

This might ring some alarm bells and cold sweats in you, namely regarding the diamond problem.

The problem comes when the original interfaces being extended share one or more methods with the same signature:

    private interface Demo1 {
        default String greet() {
            return "hello";
        }
    }

    private interface Demo2 {
        default String greet() {
            return "hallo";
        }
    }

    // interface can extend multiple interfaces, 
    // however on conflict, one will need to override

    private interface Demo3 extends Demo1, Demo2 {
        @Override
        default String greet() {
            return Demo1.super.greet();
        }
    }

The way java handles this situation is by forcing the developer to explicitly implement the concrete method in conflict - the greet method in our silly example.

Like java interfaces, one can extend a trait with another trait:

trait FileReader
trait OsFileLocker
// this is also valid syntax:
trait FileWriter extends FileReader with OsFileLocker

However, in presence of conflicting method signatures, scala will behave differently. Here are more complete illustration of the diamond problem:


    trait FileOps {
      def readFile: String
    }

    trait FileReader extends FileOps {
      override def readFile: String = "loaded"
    }

    trait OsFileLocker extends FileOps {
      override def readFile: String = "lock created & file loaded"
    }

    class FileSystemConsumer extends FileReader with OsFileLocker

Note: extending in Scala 3 is, in my opinion, clearer: class FileSystemOperator extends FileWriter, OsFileLocker

The scala compiler will allow such wiring and apply a simple disambiguation rule: on conflicting method signature, use the implementation of the last trait being extended, e.g. the one one furthest to the right (OsFileLocker).

println(new FileSystemConsumer().readFile)
// will print: lock created & file loaded

To make it perfectly clear, if one swapped the order, the print message would simply be "loaded":

    class FileSystemConsumer extends OsFileLocker with FileReader
    println(new FileSystemConsumer().readFile)
    // will print: loaded

cake pattern background: self type annotation

The second essential trick we'll be using is scala's self type - which provides a way to mix-in another trait:

  trait Greeting {
    def intro: String
  }


  trait Greeter {
    this: Greeting =>
    def greet: String = s"$intro, I am Foo"
  }

By specifying this: Greeting => we are defining that whoever extends Greeter will need to also extend the Greeter trait, or anything that extends it. The anything that extends it is the relevant part, and the main reason why we don't simply declare trait Greeter extends Greeting.

If one tried to instantiate a Greeter trait, the compiler would immediately complain:

// this won't compile
val greeter = new Greeter

But if one would define a concrete implementation, for example:


  trait PortugueseGreeting extends Greeting {
    override def intro: String = "Olá"
  }

Then the following would compile:

    val greeter = new Greeter with PortugueseGreeting
    println(greeter.greet)
    // will print: Olá, I am Foo

The key part here to understand is the with PortugueseGreeting - the concrete part where we're injecting the intended concrete implementation of the Greeting contract. One would not be able to force the developer to inject a concrete implementation if one would have used inheritance to define the Greeter trait:

    // bad: using inheritance
    trait Greeter with Greeting {
      def greet: String = s"$intro, I am Foo"
    }

Note that anything available in the Greeting trait will also be available for the Greeter trait - in our silly example the intro method - when using the preferred mix-in implementation.

the cake pattern example (finally)

We are going to use an example often seen in API's code, namely where we want to specify that a service (where business logic resides) needs to be injected with a generic implementation of a repository (where we define CRUD operations with a given data store).

Let's keep this simple; our model:

  case class User(id: Long, name: String)

And our generic UserRepository contract:

    trait UserRepository {
      def findOne(): User
    }

The second step is where specific cake pattern design would kick in. One would define a component, say UserRepositoryComponent:

  trait UserRepositoryComponent {
    // expose the generic contract
    val userRepo: UserRepository
  }

Components have a single purpose: to expose a given contract. Given the property that we talked earlier that the exposed value - userRepo in our example - will be available for use in the trait where any other trait where it is mixed-in.
So if we had a generic service contract:

    trait UserService {
      def findUser: User
    }

We could wire all of these together in a UserServiceComponent:

  trait UserServiceComponent {
    this: UserRepositoryComponent =>

    val userService: UserService = new MyUserService

    class MyUserService extends UserService {
      def findUser: User = userRepo.findOne()
    }

  }

I hope by now the reason for the name cake starts becoming clearer. Like a cake with several layers, our component declares a class inside of it, which, in turn, makes use of the userRepo value thanks to the this: UserRepositoryComponent => mix-in.

At this point we're exactly in the same situation as we were when we couldn't instantiate a new Greeter instance, since we still can't instantiate a UserServiceComponent. No worries, let's implement a concrete mock Repository:

  trait CassandraUserRepositoryComponent extends UserRepositoryComponent {
    val userRepo = new CassandraUserRepository

    class CassandraUserRepository extends UserRepository {
      def findOne(): User = User(1L, "john")
    }
  }

Now we can glue it all together:


    val userServiceComponent = new UserServiceComponent with CassandraUserRepositoryComponent
  println(userServiceComponent.userService.findUser)
  // will print: User(1,john)

Just like before, we're leveraging scala self-type to inject the concrete implementation of our repository only on instantiation.

Conclusion

In this post we went through the basics of dependency injection with examples in scala, and dived into a concrete way to wire your code to perform DI manually, with the cake pattern. As mentioned in the beginning, feel free to grab the code snippets available on github to play with them.

We sincerely hope that we were able to distill all building blocks of the cake pattern in simple terms that didn't require you to take a PhD in category theory.

References

yet another post about type classes in Scala

diogoaurelio — Mon, 02 Jan 2023 16:15:43 +0000

While recently having a chat with a colleague about type classes in scala, I realized I failed at properly explaining the concept. So here I am, giving it (yet) another stab.

In this post we will:

understand the intuition behind type classes (TCs), to better reason when to use them;
provide two concrete examples with code in scala;
conclude with a very brief look into cats, a widely used scala library that heavily relies on TCs;

The examples provided in this post were written in Scala 2, and are available in this github repository.

Intuition

Type classes provide a way to model for different behaviors according to the type of the object being handled. So, at it's core, a type class is (unlike the name might at first sight suggest) actually just an interface - a trait in scala - that defines the behavior with a given method signature that accepts or returns generic type.
We then define concrete implementations for that interface for a given specific type. These we call the type class instances.
Usually, as a last step, one exposes this functionality with a given api, say, for example a static method.

Example 1

Let's say we work in a company that does ratings, and, to simplify matters, issues 5 star ratings. Currently the company started with doing evaluations of restaurants. However, we want to be able to extend those in the future to other aspects in life, such as hotels, cities, concerts, cars, gadgets, etc, you name it.
Modelling our business domain, we'd quickly understand that a car is a completely distinct model than a hotel, so using classic OOP polymorphism by finding a common denominator interface that all of the other classes extend/inherit is a weak solution.
Instead, what we really want is just to define a general interface for rating any of these types - think generics. In more rigorous terms this is called parametric polymorphism, which refers to the ability to define a single code template that can work with any type T parameter.

  // this is our Type Class
  trait Reviewer[T] {
    def rate(obj: T): Int
  }

Our TC has a single method that accepts a generic type, and returns 1-5 star rating which we oversimplified with an Integer (Int) return type.

Let's add two (rather silly but hopefully illustrative) models:

  case class Restaurant(name: String, foodQuality: Int, environment: Int, location: Int)

  case class Dish(name: String, sweetness: Int, saltiness: Int, bitterness: Int, sourness: Int, umami: Int)

Now comes the second part, where we deviate from parametric polymorphism - where the concrete implementation is completely agnostic to the Type T being handled - and add specific TC instances that define different behaviors/strategies depending on the object being handled. For rating a restaurant one could implement a simple algorithm as follows:


  object RestaurantReviewer extends Reviewer[Restaurant] {
    override def rate(obj: Restaurant): Int =
      scala.math
        .round(0.6F * obj.foodQuality + 0.3F * obj.environment + 0.1F * obj.location)
  }

and, on the other hand, for reviewing a dish the following strategy:


    object DishReviewer extends Reviewer[Dish] {
      override def rate(obj: Dish): Int = scala.math.round(
        0.1F * obj.sweetness
          + 0.2F * obj.saltiness
          + 0.1F * obj.bitterness
          + 0.1F * obj.sourness
          + 0.5F * obj.umami
      )
    }

TC are also sometimes referred to as ad hoc polymorphism, which opposes parametric polymorphism because one purposely defines a concrete distinct implementation for every given type. I found Jason McClellan does a great job explaining the different types of polymorphism and in concrete for type classes, so a highly recommend you to have a look at it.

The usual third and last step is to expose the overall rating functionality, for example, as follows:


  object Evaluator {
    def rate[T](obj: T)(evaluator: Reviewer[T]): Int = evaluator.rate(obj)
  }

Alright, this is already enough for us to dry-run in a demo app:


object DemoTC extends App {

  trait Reviewer[T] {
    def rate(obj: T): Int
  }

  case class Restaurant(name: String, foodQuality: Int, environment: Int, location: Int)
  case class Dish(name: String, sweetness: Int, saltiness: Int, bitterness: Int, sourness: Int, umami: Int)

  object ReviewStrategies {

    object RestaurantReviewer extends Reviewer[Restaurant] {
      override def rate(obj: Restaurant): Int =
        scala.math
          .round(0.6F * obj.foodQuality + 0.3F * obj.environment + 0.1F * obj.location)
    }

    object DishReviewer extends Reviewer[Dish] {
      override def rate(obj: Dish): Int = scala.math.round(
        0.1F * obj.sweetness
          + 0.2F * obj.saltiness
          + 0.1F * obj.bitterness
          + 0.1F * obj.sourness
          + 0.5F * obj.umami
      )
    }

  }

  object Evaluator {
    def rate[T](obj: T)(evaluator: Reviewer[T]): Int = evaluator.rate(obj)
  }

  val restaurant1 = Restaurant(name = "Cheesegaddon", foodQuality = 5, environment = 3, location = 2)

  // rate using `RestaurantReviewer` strategy:  
  val restaurant1Rating = Evaluator.rate(restaurant1)(ReviewStrategies.RestaurantReviewer)

  // will print: Restaurant Cheesegaddon final rate is: 4 stars
  println(s"Restaurant ${restaurant1.name} final rate is: ${restaurant1Rating} stars")


}

Note that so far we're not leveraging any of scala's powerful features. As a matter a fact, we can implement the exact same demo code in java (yes java nerds, bear with me, as mentioned in the inline code comments, the above code is not the finest, but it illustrates my point):

public final class DemoTCJava {

    // this is our Type Class
    interface Reviewer<T> {
        int rate(T obj);
    }

    // Note 1: Avoiding using newer Java Records to make this code easily runable on any jdk version 
    // Note 2: made usually private internal vars public as to avoid extra verbosity of implementing getters;
    private static final class Restaurant {
        public final String name;
        public final int foodQuality;
        public final int environment;
        public final int location;

        public Restaurant(final String name, final int foodQuality, final int environment, final int location) {
            this.name = name;
            this.foodQuality = foodQuality;
            this.environment = environment;
            this.location = location;
        }
    }

    private static final class Dish {
        public final String name;
        public final int sweetness;
        public final int saltiness;
        public final int bitterness;
        public final int sourness;
        public final int umami;

        public Dish(final String name, final int sweetness, final int saltiness, final int bitterness, final int sourness, final int umami) {
            this.name = name;
            this.sweetness = sweetness;
            this.saltiness = saltiness;
            this.bitterness = bitterness;
            this.sourness = sourness;
            this.umami = umami;
        }
    }

    private static final class RestaurantReviewer implements Reviewer<Restaurant> {
        @Override
        public int rate(Restaurant obj) {
            return Math.round(0.6F * obj.foodQuality + 0.3F * obj.environment + 0.1F * obj.location);
        }
    }

    private static final class DishReviewer implements Reviewer<Dish> {
        @Override
        public int rate(Dish obj) {
            return Math.round(
                    0.1F * obj.sweetness
                            + 0.2F * obj.saltiness
                            + 0.1F * obj.bitterness
                            + 0.1F * obj.sourness
                            + 0.5F * obj.umami
            );
        }
    }

    private static final class Evaluator<T> {
        public int rate(T obj, Reviewer<T> evaluator) {
            return evaluator.rate(obj);
        }
    }

    public static void main(String[] args) {
        final Restaurant restaurant1 = new Restaurant("Cheesegaddon", 5, 3, 2);
        final RestaurantReviewer reviewer = new RestaurantReviewer();
        final Evaluator<Restaurant> evaluator = new Evaluator<>();
        final int restaurant1Rating = evaluator.rate(restaurant1, reviewer);

        System.out.println(
                String.format("Restaurant %s final rate is: %d stars",
                        restaurant1.name,
                        restaurant1Rating
                )
        );
    }

}

This is an example where scala's implicits really shine. If we made our API - the Evaluator static class - accept the evaluator as an implicit argument:


  object Evaluator {
    def rate[T](obj: T)(implicit evaluator: Reviewer[T]): Int = evaluator.rate(obj)
  }

.. and changed our ReviewStrategies to implicitly define TC instances:


  object ReviewStrategies {

    implicit object RestaurantReviewer extends Reviewer[Restaurant] {
      override def rate(obj: Restaurant): Int =
        scala.math
          .round(0.6F * obj.foodQuality + 0.3F * obj.environment + 0.1F * obj.location)
    }

    implicit object DishReviewer extends Reviewer[Dish] {
      override def rate(obj: Dish): Int = scala.math.round(
        0.1F * obj.sweetness
          + 0.2F * obj.saltiness
          + 0.1F * obj.bitterness
          + 0.1F * obj.sourness
          + 0.5F * obj.umami
      )
    }

  }

.. we could then simply import the intended review strategies based on scope:


  import ReviewStrategies._
  val restaurant1Rating = Evaluator.rate(restaurant1)

And this is one of my favorite things about TCs, namely the flexibility for future extension that they provide. If perhaps tomorrow a new food critic would join the team, one that uses a completely different algorithm for computing the rate, absolutely no change would be required
in our API - we would simply need to define and import the new different set of strategies.

Example 2

Our second example is meant to illustrate an additional nice thing about TCs: the compile time safety.

Let's say we want to prevent the following to happen:


  val aComparison = 2 == "a String"
  // will print: Comparison is false
  println(s"Comparison is $aComparison")

Your IDE will hopefully highlight the fact that we are comparing an int with a string, but compilation will succeed nonetheless, and the expression will evaluate to false.

To solve this, we can use the same strategy as before. Let's define an Eq TC with two concrete implementations, respectively for Int and String:


  trait Eq[T] {
    def equals(val1: T, val2: T): Boolean
  }

  implicit object IntEq extends Eq[Int] {
    override def equals(val1: Int, val2: Int): Boolean = val1 == val2
  }

  implicit object StringEq extends Eq[String] {
    override def equals(val1: String, val2: String): Boolean = val1.equals(val2)
  }

  object Comparison {
    def compare[T](val1: T, val2: T)(implicit comparator: Eq[T]): Boolean = comparator.equals(val1, val2)
  }

  val aNumber: Int = 4
  val anotherNumber: Int = 5

  val result1 = Comparison.compare(aNumber, anotherNumber)

  // will print: 4 == 5 is false
  println(s"$aNumber == $anotherNumber is $result1")

With this approach we are certain that if we try:

Comparison.compare(1, "one")

... compilation will fail, as intended.

We could improve our Comparison implementation just a little further leveraging scala extension methods:

  object MyComparisons {
    implicit class MyIntComparison[T](val1: T)(implicit eq: Eq[T]) {
      def `===`(val2: T): Boolean = eq.equals(val1, val2)
    }
  }

If we import MyComparisons object, we will immediately start having access to our custom === method for all Int and String type objects, allowing the following:

  import MyComparisons._
  // will print: 4 === 5 is false
  println(s"$aNumber === $anotherNumber is ${aNumber === anotherNumber}")

In case you're wondering how extension methods work, essentially the compiler is smart enough to underneath the hood to rap our Int instance stored in variable aNumber into a new instance of:
new MyIntComparison(aNumber)
.. and call the method === towards the other Int instance stored in variable anotherNumber:
new MyIntComparison(aNumber).=== (anotherNumber)

Pretty cool, right?

If you're thinking to implement this by yourself, think again. Scala's Swiss-army-knife library cats provides exactly this same functionality.

Start by importing it in sbt:

val catsVersion = "2.1.1"

libraryDependencies ++= Seq(
  "org.typelevel" %% "cats-core" % catsVersion
)

And use it:


  import cats.instances.int._
  import  cats.syntax.eq._

  // will print: 2 === 3 is false
  println(s"2 === 3 is ${2 === 3}")

Conclusion

I'd like to wish that, if anything, this post manages to convey the intuition behind type classes.

Type classes are useful when we want to model different behaviors according to the specific type of an object. With our previous example, we tried to provide a case where it makes little sense to use polymorphism to model distinct concepts, namely restaurants, hotels, cars, cities, etc. In other words, instead of having a common interface that all of the previous classes inherit, we prefer to rather use generics.
Our type class is nothing but an interface that defines the signature for a given behavior/method leveraging generics.
This leaves the concrete implementation open for accommodate different strategies according to the concrete types, or even different strategies for the same time. This idea is probably what makes type classes so powerful: the flexibility to extend them.

Our second type class example attempted to illustrate one last perk: type safety at compile time. It did so with a simplified example of the cats core library for type safety equality comparison between objects. If you're not familiar with cats, go ahead and give it go.

References

parametric polymorphism
ad hoc polymorphism
Jason McClellan on the different types of polymorphism and where type classes fit
cats core library
repo code for this blog post

Terraforming in 2021 – new features, testing and compliance

diogoaurelio — Sun, 02 May 2021 17:07:29 +0000

Both João and I have been using regularly terraform for some time now, and are big fans of it. Big enough to even have worked for quite a while on a side project of our own to simplify managing terraform environments (which unfortunately did not succeed), at least. And yes, now that we are preparing the next one with mklabs, which naturally also heavily relies on terraform scripts, it made only sense to start by sharing some of our favorite tools that gravitate in the terraform ecosystem.

We kick off with the latest updates in terraform itself, from 0.12 until the latest 0.15, and then go straight to surrounding tooling, where we mainly focus on testing and compliance checking for infrastructure.

You can find all the code supporting this this post here.

Overview of latest Terraform versions

Version 0.15 has just recently been released. Yet, a lot of companies out there are still running environments up to 0.11.X, and for a good reason. Though version 0.12 was launched already a while ago (2019), it brought great disruption. If this is your case, chances are that you gave up on keep up to date with its progress. Thus, we thought it would make sense to first review some of terraform features you might be missing out, before we try to convince you also considering further improvements.

0.12

First-class expression syntax: probably the most noticeable change, no more string interpolation sintax (confettis blowing in the background)!
Generalized type system: (yay!) able to specify types of variables;
Iteration constructs: introduction a the for operator allowing closer proximity to programming languages and thus more expressiveness to the DSL;

More details, see the general availability announcement page.

0.13

Module improvements - brings ability to use several meta tags that until so far were only available for resources in modules. In this case, we can now use depends_on for coupling dependencies with modules (finally!), along with ability instantiate multiple module instances with count or for_each; more details here;

Example specifying a module dependency (a personal favorite), source hashicorp blog

Custom variable validation - when specifying variables one can from this version on wards specify custom rules for the input that is accepted and respective error message to allow fast failure; more details here;

Example variable validation, source hashicorp blog

Support 3rd party providers - terraform now allows one to reference your own provider in the terraform block required_providers. The required_providers keyword in the terraform block already existed in terraform 12, though with was restricted to hashicorp's own providers; now you can reference your own DNS source to host your own registry, and upon terraform init it will be installed same as it used to all other providers; more details here; note that if you are already using the required_providers keyword in terraform block, to migrate from version 12 to 13 you should adapt it, as shown in the following example:

Example required_providers, source terraform upgrade guides

More details, see the general availability announcement page.

0.14

Sensitive variables & outputs - allowing one to flag a variable as "sensitive" will redact its output on the CLI;
Improved diff - this is probably one of the most common complaints I hear, namely the difficulty in reading a terraform plan (and along with it the same for apply and show) output; with this new change, it hides unchanged fields (irrelevant for the diff) while displaying a count of the hidden elements for better clarity; more details here;

Example excerpt diff, source hashicorp blog

dependency lockfile - as soon as you run terraform init with version 14 you will notice that a .terraform.lock.hcl file will be created in that directory and outside the .terraform directory. This file is intended to be added to version control (git committed), to guarantee that the exact same provider versions are run everywhere;

More details, see the general availability announcement page.

0.15

Finally, here are some of the main highlights the latest version just very recently announced:

Remote state data source compatibility: in order to make it easier to upgrade to newer versions, you can use reference remote state data objects that are using older versions; note that this feature has been back ported into previous releases as well, namely 0.14.0, 0.13.6, and 0.12.30;
Improvements to conceal sensitive values (passwords, for example): following version 0.14 work now provider developers can specify the properties that are by default sensitive and should be anyway hidden in outputs; moreover, terraform also added a function (sensitive) for users to be able to explicitly hide values;
Improvements in logging behavior: ability to control provider and terraform log level separately TF_LOG_CORE=level and TF_LOG_PROVIDER=level.

More details, see the general availability announcement page.

Handling multiple versions

Assuming that these new features convinced you to upgrade your existing terraform environments, being realistic here, this will not happen from one day to another. You will have a transition period (if not permanently) where you have environments with different terraform versions. That is OK, you can still keep your sanity while hopping between all of them thanks to tools like the following:

TFEnv - terraform environment switcher inspired (from the ruby world) by rbenv written with shell scripts;
Terraform Switcher - yet another project essentially doing the same written in go;

Both of these projects overlap almost entirely, so we will simply exemplify with one of them, namely tfenv:

Here we are showing how you can switch between installed versions tfenv use <local-version>, how you can check which versions are already locally installed with tfenv list, all versions currently available tfenv list-remote (minor detail: the current version of the library I'm using to record my terminal, terminalizer, does not capture me scrolling up and selecting version terraform 0.14.5)
Last but not least, we also show a cool feature from tfenv, namely the ability to automatically recognize the minimum required version in a given environment. Same goes for the latest version, in case you are wondering. And yes, this is also available in terraform switcher project.

Testing

Testing is probably the most confusing topic in the Infrastructure-as-Code (IaC) land, and terraform not being an exception, as a lot of different tools and procedures get thrown in this same bag, when, well, they probably should not. Usually when talking about testing, people usually mean three different things:

static checks - validation of mainly the structure code without actually running the code; a fast away for performing sanity checks, either local and/or in your CI/CD pipelines;
integration tests - provided you are properly using inversion of control with variables, they give you the power to test your modules or environments for different generic cases;
compliance checks - tests done in the aftermath, after all resources have already been deployed; these can serve distinct purposes that we will explore better later, but essentially the goal is to confirm that what is deployed is what was initially intended and expressed in terraform;

Yes, you got that right: if you were looking forward to some classic unit testing, you can forget that for now. Let us dive straight into more details.

Static checks

Again, these are not tests strictly speaking, but rather just simple validation checks one can run to catch common errors without actually deploying anything. There is an array of tools out there, but let us start with the one provided out of the box in terraform binary.

It is not uncommon for the tool creators to provide their own validators. Kubernetes provides validate kubectl apply -f <file> --dry-run --validate=true, helm provides lint helm lint <path>; terraform provides validate.

In the following example running terraform validate would catch two of the three issues:

Validate will require to have previously run terraform init, so that it can leverage providers. In our example it will detect the typo in the aws bucket resource reference, along with the invalid CIDR provided to create the VPC. What it will not detect, however, is the non existent EC2 instance type.

TFLint comes to the rescue. Being yet another open source tool written in go, it comes as a binary much like terraform and does not even require terraform to be installed.

Running with the deep option requires one to provide credentials, providing a more thorough inspection. In our case, it detected the incorrect instance type, as well as the wrong AMI.

You can also customize tflint to inject variables, define modules to ignore, etc. You can check the user guide for more details.

Now besides linting for configuration issues in your code, another recommendation is to check for security issues, such as too permissive security group rules, unencrypted resources, etc.

Here again more than one tool exists to assist. We will highlight two of the most popular ones here: tfsec and checkov. Both provide a predefined set of checks that they use to inspect your code, allowing to explicitly open exceptions (if you really want to) by annotating your code with comments, and adjust the configuration to ignore some modules, for example.

TFSec is written in Go, and is probably the fastest to get started, and currently provides up to 10 checks for the current main cloud providers (AWS, GCP, and Azure). The potential downside is that works exclusively for Terraform, so you will need to use additional tools to inspect kubernetes/helm/cloudformation etc.

Checkov, on the other hand, despite being a more recent tool, has seen stellar development speed (being developed by a startup with good founding rounds, and PaloAlto Networks acquisition can't hurt). Not only do they have a really comprehensive number of checks across all the main cloud providers, but they also span across multiple technologies, such as Kubernetes, Cloudformation, serverless and ARM templates. And the list keeps growing. Checkov provides you the option to run either a pure static check by just pointing to the terraform directory or terraform file, or by actually running it against a terraform plan file. The nice thing about running it directly is naturally the simplicity of not requiring to have the target environment accessible to test the code.

These tools are grabbing a lot of attention lately, as the double checking for security issues was usually locked in the hands of devops/devsecops teams, which in practice constituted a development bottleneck. By injecting these checks early in CI/CD pipelines, a great deal of development speed is freed without compromising security.

Integration tests

Terratest is probably the closest one can get now a days to testing the specific peace of terraform code. It is a Go library, and requires one to write tests in Go. This is obviously a potential limitation as not all teams have knowledge in Go. On the upside, I would argue that the learning curve of learning Go to get the basics - read enough for writing terraform tests - is not steep if you know already at least one programming language.

Having worked already in several Go projects in the context of mklabs, we're naturally favorably biased towards Terratest. So, even if you have never tried Go, we would still recommend having a look on our sample repository, where we provide go mod setup to make it easy getting started. And if this did not convince you yet, here's what might: you can also use Terratest to test Dockerfiles, Kubernetes and Packer setups.

This might seem intimidating, but we would argue that the benefits are worth it. Let us look at an example skeleton of a test setup for AWS:

Essentially the skeleton is always the same:

start by defining the variables you want to inject as input for your terratest run; you might want to rather inject random variables even, to test in greater depth;
make sure you tell the setup to destroy all created resources after the terraform apply has been completed using the defer statement; this is equivalent to the trap keywork in shell, and it will execute even if something fails in the meanwhile; the only prerequisite is that you declare it before you call terraform.InitAndApply() method;
Let the test be deployed by passing the terraform config setup you have previously declared in terraformOptions to terraform.InitAndApply method;
Finally, declare the things you want to assert; that is, declare what you expect should have been deployed - the expected - and see if they match what was actually deployed - the result. The simplest way is to check in terraform apply output, which is accessible via `terraform.Output()` method. Alternatively, terragrunt also provides you some methods out of the box for frequently checked things on a per provide basis.

Sounds like fun, right? Here is an example of some of the tests we could write to our basic terraform example:

Here is the output of running these tests for our demo setup:

Final thoughts regarding terratest: we find terratest a great tool to implement changes in your infrastructure with confidence, no matter if they are just simple day to day changes, or bigger and complexer upgrades or migrations.

We just scratched the surface here on the tests one can develop with terratest - for example combining SSH access into instances and confirming access to resources, etc. However, there are no free lunches, and this can come at a price: tests can take a long time (depending obviously on what you are testing), and require a non trivial time investment - learning how to use, writing them, and setting up the environment, as you probably will want to run them in an isolated environment. For example, in AWS case, this would ideally be a dedicated account. We recommend reading best practices on how to perform testing from terragrunt, the company behind terratest.

Compliance checks

The last mile is asserting that what you wanted to be deployed, was indeed deployed exactly as you wanted. Abusing terraform null_resource is a classic one leading to unintended surprises. One way of achieving this would be to run these tests right after the terraform apply stage of your CI/CD pipeline.

But the next question you might have is how do I know that these configurations stay that way, that no one changed things inadvertently ? We've seen this situation arise in different forms: changes done by users manually via GUI or CLI; via different terraform environments mutating overlapping resources properties; or as a by-product of using different IaC tools, for example configuring some bits with Cloudformation, some K8s or Helm, etc.

Arguably more interesting would be scheduling to run these checks continuously and repeatedly, to make sure things stay as you expected. Let's see some options out there to achieve this.

The rogue approach: in practice you can do this type of compliance checks with any programming language you favor. As we shown before, one can export terraform' state into json format, and then use, for example, python with pytest and boto3 libraries to compare what is deployed with the desired output. You could even go further, and use boto3 to scan your accounts for different aspects that you considered go against best practices, such as lack of encryption (in-flight and at rest).

While this might be tempting at first sight, you might end up writing way more testing code, than the actual terraform code. Not only it seems kind of silly, but it can get hairy.

The second reason not to follow this approach, is that there are already several solutions out there fairly tested and intuitive to use that can help you with this task, which we will cover now:

Driftctl - open source go library from cloudskiff for terraform;
Sentinel - Hashicorp's own solution;
terraform-compliance - open source BDD based solution dedicated for terraform;
Conftest - test suite for multiple frameworks besides terraform, such as kubernetes and dockerfiles;
Inspec - Chef compliance testing tool, written in ruby;
Built-in cloud provider - each cloud provider has it's own inspections mechanisms in place;

You may be wondering why the first place in our list is a library still in beta. That is a good point, and we would argue that its progress seems to be promising, and that we like to support startups. Moreover we think its simplicity of usage deserves a highlight as it delivers one thing and one thing only: check if what is in your terraform state file is what is actually deployed. Simply point to your terraform statefile when you run driftctl scan command and you will get a detailed report if you have drifted. Due to the low effort required to implement this library and value provided, we think it deserves taking it for a spin.

Next on our list is Hashicorp's (the company behind terraform) own enterprise solution for this, Sentinel. This is could make sense if you are already using other Hashicorp's enterprise functionality, benefiting from Terraform Enterprise.

A direct open source comparable alternative would be using terraform-compliance. It follows BDD directives so that you can specify in an easy human readable way your expectations, using:

given: a given resource type;
when: an optional condition you might want to add;
then: what you expectation is;

Here is an example of a test file for AWS S3 buckets:

Although we do get the appeal of easily understandable tests that do not require knowing how code, we find terraform-compliance lacks flexibility to test various aspects, mainly due to the BDD nature. Moreover, there are more people disenchanted by it with some valid points.

If you like terraform-compliance, Conftest might also be worth having a look. It has its own DSL to write policies, and allows you to test multiple frameworks. We found this blog post from Lennard Eijsackers very informative, and would thus rather recommend you to check it out.

Before we dive into own cloud provider compliance checking services, we want to highlight yet another open source tool, namely InSpec. It allows you to write tests in ruby, and was built on top of RSpec. If you know already awsspec, then this should feel very similar, with the advantage that InSpec also supports GCP and Azure.

Even though none of us is actively a ruby developer, we find InSpec very easy to get started with and, most importantly, very powerful. It allows you to combine IaC checks to target resources deployed and mapped in terraform state files, with other general policies for cloud account configuration. Moreover, it also allows you to combine additional security checks, such as on OS level configurations and services running. Let us illustrate how you could write these checks. The following check illustrates how to define global policies that an account should obey:

The next one illustrates for generic networking:

The previous examples only illustrate generic control checks that validate overall best practices in your acocunt. However, you might want to also perform assertions regarding what is in your terraform state versus what is deployed - what the cloudskiff engineers rightly name drift. Christoph Hartmann - one of InSpecs creators - has a nice blog post explaining how to use InSpec and integrate with terraform. The approach is essentially as described previously in the rogue approach - import terraform state json file and use it as the expected assertion.

Built-in cloud provider policy tools

Each cloud provider has a native tool to address company-wide governance policies. Some examples of services are:

AWS: AWS Config and to manage cloud configuration deviations within one account, Systems Manager to enforce instance OS security, and more recently Control Tower for enforcing rules across multiple accounts;
Azure: Azure Policy for checking cloud configuration deviations and Azure Security Center
GCP: Forseti Config Validator for GCP

These are just some of the services that we know that can be used for such enforcement. Most of these services go beyond just checking cloud config, and also provide security inspections at instance level, for example. The important point to keep in mind is that these security checks are post deployment compliance assertion, not for preventing configuration issues.

Final thoughts

We find that there is not a single silver bullet that solves all problems, and the best strategy is actually a combination of multiple tactics employed at different stages. The two main phases that require different approaches are pre and post deployment.

For pre-deployment, a combination of static checks and actual tests can be used. Static checks are a great starter, as they are easy to setup, and allow you to enforce generally good practices across the whole organization. In other words, static checks provide you an powerful easy win. With terratest, on the other hand, you mainly gain on confidence that the IaC code will actually work, and that you will catch faux pas. However, terratest does not come for free, and does require a learning curve with go, along with time investment to develop the actual tests.

After your code gets deployed comes the next challenge: making sure things stay well architected. Regular scheduled checks that assess if your infrastructure is running as intended should also be part of your devOps strategy, and for this two main routes exist: open source or using dedicated cloud services.

And that is it. Thank you for reading, we hope this has been useful. Feel free to reach out to us if you have questions or suggestions.

Once again, you can find all the code supporting this this post here.

Sources

As usual, here is the summary of sources used for this post:

Terraform version 12 general availability notes
Terraform version 13 general availability notes
Terraform version 14 general availability notes
Terraform Switcher project
Best practices testing with terratest
TFSec
Bridgecrew checkov
Terraform compliance
Driftctl tool
Chef InSpec
Christoph Hartmann blog post how to use Inspec with terraform
AWS Spec
Conftest
Blog post from Lennard Eijsackers on Conftest
The recordings of my terminal were done using terminalizer

Container orchestration in AWS: comparing ECS, Fargate and EKS

diogoaurelio — Sun, 21 Oct 2018 15:31:12 +0000

Before rushing into the new cool kid, namely AWS EKS, AWS hosted offering of Kubernetes, you might want to understand how it works underneath and compares to the already existing offerings. In this post we focus on distinguishing between the different AWS container orchestration solutions out there, namely AWS ECS, Fargate, and EKS, as well as comparing their pros and cons.

Introduction

Before we dive into comparisons, let us summarize what each product is about.

ECS was the first container orchestration tool offering by AWS. It essentially consists of EC2 instances which have docker already installed, and run a Docker container which talks with AWS backend. Via ECS service you can either launch tasks - unmonitored containers suited usually for short lived operations - and services, containers which AWS monitors and guarantees to restart if they go down by any reason. Compared to Kubernetes, it is quite simpler, which has advantageous and disadvantages.

Fargate was the second service offering to come, and is intended to abstract all everything bellow the container (EC2 instances where they run) from you. In other words, a pure Container-as-a-Service, where you do not care where that container runs. Fargate followed two core technical advancements made in ECS: possibility to assign ENI directly and dedicated to a Container and integration of IAM on a container level. We will get into more detail on this later.

The following image sourced from AWS blog here illustrates the difference between ECS and Fargate services.

EKS is the latest offering, and still only available on some Regions. With EKS you can abstract some of the complexities of launching a Kubernetes Cluster, since AWS will now manage the Master Nodes - the control plane. Kubernetes is a much richer container orchestrator, providing features such as network overlay, allowing you to isolate container communication, and storage provisioning. Needless to say, it is also much more more complex to manage, with a bigger investment in terms of DevOps effort.

Like Kubernetes, you can also use kubectl to communicate with EKS cluster. You will need to configure the AWS IAM authenticator locally to communicate with your EKS cluster.

Cost

Let us start with costs, the number one usual favorite suspect that Software Developers like to neglect.

ECS, is a simple service, where the control plane is abstracted away by AWS. You simply launch Worker nodes, which consist of EC2 instance with Docker installed, and a Container daemon running which contains the minimum metadata to know to which cluster it belongs to, and to receive instructions from AWS ECS control plane. Thus, with ECS you only pay per Worker node, read EC2 instance.

Fargate abstracts from you the hosting platform of containers - the EC2 instances. This obviously means that you pay a premium per container, compared to ECS. So yes, the cost per Container in Fargate will naturally be higher. On the other hand, and to perform a fair comparison with pure ECS, you should also take into consideration maintenance costs associated with ECS. The hours invested troubleshooting, upgrading ECS instance agents, updating EC2 instance packages, etc., is translated in the form of salaries. Thus, should not be forgotten.

EKS, being Kubernetes under the hood, requires Master nodes always running to monitor the cluster, the EKS control plane. This does not come for free, and will cost you minimum pro cluster 0.20 US Dollars/ hour, which amounts to 144$ pro month. This just to start playing around, without any Worker node. From here on, you only pay the normal price for EC2 instances you launch for Worker nodes, just like the ECS offering.

Auto-scaling

ECS allows you to scale at the task level. Remember: a task maybe one or more containers. Much like EC2, you can configure rules to scale the number of tasks running. However, these are containers running on EC2 instances, which may not have available CPU or RAM resources. So in practice, the trick here is to define another Auto-scaling rule at the ECS cluster level, where you define scale up or down based on the CPU or RAM reservation on the cluster.

Fargate, as expected, makes the auto-scaling much easier. Here you simply need to define it at the task level, and you are good to go.

More on ECS/Fargate auto-scaling can be found here.

Just recently AWS introduced integration with Kubernetes Horizontal Pod Autoscaler (HPA) for EKS. Much like ECS, this will allow you to scale on the pod level, which means again that you will also need to scale on the EC2 level to support pod growth. More on getting started with HPA here.

Also, if you already have an EKS cluster running, make sure your EKS version supports HPA. All new EKS clusters will be launched with platform version eks.2+, which means they already support HPA at the time of writing this blog post.

Let us now dive deeper technically in the three main Networking aspects that distinguish the three services.

Networking: Traffic Load Balancing

Until AWS integrates EKS with their more sophisticated load balancers, namely Application LB and Network LB, traffic flow from a networking perspective will remain very inefficient. Since EKS only supports classic AWS ELB, traffic is randomly and blindly distributed among EC2 instances, and be prone to multi-hop traveling.

Both ECS and Fargate cleanly integrate with AWS ALB and Network LB, allowing you directly point at the intended containers. Cloudnout provide very good illustrations on their blog post on this topic.

Networking: ENI

Being able to assign a network card directly to a container might not seem like that big of a deal at first sight. However, effectively this means a higher level of security. This way you are able to assign a Security Group dedicated for that individual container, rather that simply opening all ports for the whole cluster.

With ECS, you have the option to associate an ENI directly with a container, by choosing to launch a task in "awsvpc" mode. However - there is always a but, isn't there ? - the maximum number of ENIs (read Virtual NICs) that can be assigned to each EC2 instance varies according to EC2 type; and one could argue that it is rather limited from 2 to 15 at the current time being. So if you are running a lot of small containers, this scenario effectively would mean creating a very big EC2 cluster of small EC2 instances. The following list exemplifies this statement.

With EKS, on the other hand, you have the option to assign a dedicated network interface a a container group, what Kubernetes calls a pod. So that means all containers inside that pod will share the same internal network. However, here come the good news, you can place up to 750 pods per instance. This is a much bigger advantage for EKS.

Networking: Service Discovery

AWS has recently launched service discovery for ECS/Fargate. This was a key differentiator that the ECS was lagging behind compared to Kubernetes, which uses network overlay to allow essentially container level DNS.

However it needs to be noted that only Containers launched in "awsvpc" network mode support this feature.

Security Policy Control (AWS IAM integration)

Returning to the security topic, another pro in favor of ECS/Fargate is the possibility to assign IAM instance profiles dedicated to the Container. What do these policies specify? Well basically access to all other AWS resources, such as S3 buckets, databases/datawarehouses such as DynamoDB, Aurora, Redshift, queing systems such as SQS and Kinesis, etc. You get the picture.

Again, the argument here is similar to the ENI: assigning dedicated IAM roles on a container level is order of magnitudes safer than on the EC2 instance level.

ECS and Fargate allow you to specify permissions at the container level. Currently, EKS does not allow you to so. This means you are limited to constrain permissions at the Workers/EC2 level.

Container Registry - AWS ECR

Nothing special here: as expected, all services integrate with AWS container registry, allowing you to pull from your internal AWS registry.

Summary

To conclusion, ECS and Fargate have a much tighter integration with AWS ecosystem. Kubernetes, on the other hand, has a much tighter integration with generic open source ecosystem, allowing you also to be more Cloud vendor agnostic.

As usual, here is a compilation of some of the useful sources used to write this post which might serve you as well:

Decrypting correctly parameters from AWS SSM

diogoaurelio — Sun, 21 Oct 2018 15:29:54 +0000

Today is yet short one, but ideally will already save a whole lot of headaches for some people.

Scenario: You have stored the contents of a string using AWS SSM parameter store (side note: if you are not using it yet, you should definitely have a look), but when retrieving it decrypted via CLI, you notice that the string has new lines ('\n') substituted by spaces (' ').

In my case, I was storing a private SSH key encrypted to integrate with some Ansible scripts triggered via AWS CodePipeline + CodeBuild. CodeBuild makes it realy easy to access secrets stored in SSM store, however it was retrieving my key incorrectly, which in term domino-crashed my ansible scripts.

Here you can also confirm more people are facing this issue. After following the suggestion of using AWS SDK - in my case with python boto3 - it finally worked. So here is a gist to overwrite an AWS SSM parameter, and then retrieving it back:

Hope this helps!

Getting Started with Spark (part 4) - Unit Testing

diogoaurelio — Sun, 21 Oct 2018 15:27:52 +0000

Alright quite a while ago (already counting years), I published a tutorial series focused on helping people getting started with Spark. Here is an outline of the previous posts:

In the meanwhile Spark has not decreased popularity, so I thought I continued updating the same series. In this post we cover an essential part of any ETL project, namely Unit testing.

For that I created a sample repository, which is meant to serve as boiler plate code for any new Python Spark project.

Let us browse through the main job script. Please note that the repository might contain updated version, which might defer in details with the next gist.

The previous gist recovers the same example used in the previous post on UDFs and Window Functions.

Here is an example how we could test our "amount_spent_udf" function:

Now note the first line on the unit tests script, which is the secret sauce to load a spark context for your unit tests. Bellow is the code that creates the "spark_session" object passed as an argument to the "test_amount_spent_udf" function.

And that is it. We strongly encourage you to have a look on the correspondent git repository, where we specify detailed instructions how to run it locally.

And that is it for today, hope it helped!

Versioning in data projects

diogoaurelio — Tue, 25 Sep 2018 05:35:11 +0000

Reproducibility is a pillar in science, and version control via git has been a blessing to it. For pure Software Engineering it works perfectly. However, machine learning projects are not just only about code, but rather also about the data. The same model trained with two distinct data sets can produce completely different results.

So it comes with no surprise when I stumble with csv files on git repos of data teams, as they struggle to keep track of code and metadata. However this cannot be done for today's enormous datasets. I have seen several hacks to solve this problem, none of them bullet proof. This post is not about those hacks, rather about an open source solution for it: DVC.

Let us exemplify by using a kaggle challenge: predicting house prices with Advanced Regression Techniques

Now clone the repo:

git clone git@github.com:diogoaurelio/kaggle-house-prices.git
cd kaggle-house-prices

In this post we be needing several python packages - such as dvc, boto3, and machine learning related packages - so we are going to exemplify how to configure a local anaconda and install all dependencies required for this tutorial:

conda install -n kaggle python=3.6 -y
source activate kaggle
pip install -r pip_requirements.txt
conda install -c conda-forge --file conda_requirements.txt

Alternatively, if you just want to install DVC, you can do, for example, also via pip:

pip install dvc

Coming back to the git repository, you will notice that there is a directory named ".dvc".

.dvc/
├── cache
└── config

1 directory, 1 file

This was created when I locally ran the command "dvc init", which is meant to be abstracted away from the user (as the dot in the directory name would suggest).

Now comes the interesting part. DVC has the concept of a remote data store, which is used to share and version data. Currently in practice can be of six types: local directory, AWS S3, Google Cloud Storage, Azure Blob Storage, SSH, or HDFS. To configure a remote data store simply map a local directory to the remote data store. In this case I mapped the "data" directory - which is git ignored - to the remote store as follows:

dvc remote add -d data s3://kaggle-datasets-store/house-prices

Next let us download data from kaggle house prices competition into the "data" directory, such as the "train.csv". You can also download from the terminal using the following command:

kaggle competitions download -c house-prices-advanced-regression-techniques

Now let us add it to our data versioned store:

dvc add data/train.csv 

# you should see an output similar to the following:
Adding 'data/train.csv' to 'data/.gitignore'.
Saving 'data/train.csv' to cache '.dvc/cache'.
Created 'hardlink': .dvc/cache/80/ccab65fb115cbad143dbbd2bcd5577 -&gt; data/train.csv
Saving information to 'data/train.csv.dvc'.

As you can see in the standard output, this command will generate a "train.csv.dvc" file, which contains a metadata about our "train.csv" file. Essentially a md5 hash of a pointer to the file.

To push this data into S3 is simple:

dvc push -r data

Note that I am specifying the "data" repository. If you want to check out dvc configuration, just have a look at .dvc/config file. You will notice that I have a AWS profile configured, which matches my local aws CLI credentials file. Here is how you can configure your local aws credentials to access S3 bucket:

aws configure

This will prompt you for the credentials plus default region and output format, and store them in "~/.aws/config" and "~/.aws/credentials". If you would like to have credentials for multiple AWS accounts, then aws cli allows you to configure "profiles" to store these. Simply do as follows:

aws configure --profile

If you look at my ".dvc/config" you will notice that I have configured my AWS profile with alias "diogo" for my own AWS account.

I did not find documentation how to explicitly do this configuration, so in case you are wondering which other configurations you can change, you can do as I did and just have a look at the DVC source code.

After pushing the data to S3 you will be able to share it with other colleagues. They simply need to run:

dvc pull

Note that in our case you will not be able to download from my S3 bucket, as it is not configured for public access, which will also be the same case for your company data.

Let us now tie everything together, namely our models with the data. For this I used the code from a python notebook using XGBoost from DanB. DVC has the run command, in which you are able to specify which dependencies as well as required outputs. In our case:

dvc run -d data/train.csv --file xgboost_base_model.dvc -O data/xgboost_base.pkl python src/xgboost_base.py

This will create a file named "xgboost_base_model.dvc" containing something similar to:

cmd: python src/xgboost_base.py
deps:
- md5: 80ccab65fb115cbad143dbbd2bcd5577
path: data/train.csv
md5: e10b323ae9bb3887626a7dc97d6a64d6
outs:
- cache: false
md5: d53d8021c4c362b88da61c347af52196
path: data/xgboost_base.pkl

Note that this dvc file contains a hash of both our "xgboost_base.py" script, along with a hash of the data. Now the beauty of this is that we can now simply "dvc repro xgboost_base_model.dvc" and it will detect if either the data or the script have changed. If no change has occured, then it will simply output: "Pipeline is up to date. Nothing to reproduce."

After having completed this, we would now be ready to add all our code to .git, including "dvc repro xgboost_base_model.dvc". This allows other people to also be able to reproduce the same results.

This example was fairly simple. However more interesting is chaining dvc run command with several steps in order to build pipelines - a multi-stage operations that define your full Machine Learning process. We strongly encourage you to have a look at DVC's own tutorial about pipelines.

AWS Server-less data pipelines with Terraform - Part 2

diogoaurelio — Tue, 25 Sep 2018 05:31:23 +0000

Alright, it's time for the second post of our sequence focusing on AWS options to setup pipelines in a server-less fashion. The topics that we are covering throughout this series are:

Part 1: Python Lambda to load data into AWS Redshift datawarehouse
Part 2: Terraform setup of Lambda function for automatic trigger
Part 3: Example AWS Step function to schedule a cron pipeline with AWS Lambda

In this post we complement the previous one, by providing infrastructure-as-code with Terraform for deployment purposes. We are strong believers of a DevOps approach also to Data Engineering, also known as "DataOps". Thus we thought it would make perfect sense to share a sample Terraform module along with Python code.

To recap, so far we have Python code that, if triggered by a AWS event on a new S3 object, will connect to Redshift, and issue SQL Copy command statement to load that data into a given table. Next we are going to show how to configure this with Terraform code.

As usual, all the code for this post is available publicly in this github repository. In case you haven't yet, you will need to install terraform in order follow along this post.

Terraform setup

First lets start with our proposed structure. We have found it useful to distinguish and isolate dedicated Terraform code according to the environment it needs to be deployed in. An environment can mean different things to different companies. The most common environments are dev/staging/production, which allow one to safely test code in a controlled setup. However other use cases exist, such as different VPCs or AWS accounts across different tech teams.

For this reason, we actually require one to export ENVIRONMENT variable in the beginning, which is used as a key for the Makefile to find the correct directory to deploy.

export ENVIRONMENT=dev
# create a file for terraform secret variables hosting: "$(ENVIRONMENT).tfvars", which our Makefile expects
touch terraform/environments/dev/dev.tfvars
# Verify what the terraform code plans to deploy
make plan

The make plan command will allow you to get started. Note that Terraform will use your default credentials that you have configured in "~/.aws/credentials". If you want to use different, one simple way (among several) is to export aws access key and secret values before running the command.

Whenever one runs "make plan", this will run three terraform commands: "terraform init", "terraform update" and "terraform plan". The terraform "init" and "update" are specially useful for the first time you initialize terraform, downloading all specific provider packages you require, and updating whenever you add new requirements.

At the time of writing this blog post, here is what I have got printed out when running make plan for the first time:

Initializing modules...
- module.redshift_loader_lambda
  Getting source "github.com/diogoaurelio/terraform-aws-lambda-module"

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.36.0)...
- Downloading plugin for provider "archive" (1.1.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.archive: version = "~> 1.1"
* provider.aws: version = "~> 1.36"

Terraform has been successfully initialized!

Now, since we have not yet filled our "dev.tfvars", you will notice short after the following errors:

##### 1) VPC details: VPC ID and private subnets to use (where the lambda will run)
Error: Required variable not set: private_subnet_ids
Error: Required variable not set: vpc_id

##### 2) Redshift DB details
# The IAM Redshift Role used to issue COPY commands and load data into Redshift
Error: Required variable not set: redshift_data_loader_lambda_iam_role
# The intended Redshift dabase name
Error: Required variable not set: redshift_data_loader_lambda_db_name
# Redshift endpoint or Route53 record to use
Error: Required variable not set: redshift_data_loader_lambda_db_host
# Redshift DB user to use
Error: Required variable not set: redshift_data_loader_lambda_db_user
# Redshift User password to use
Error: Required variable not set: redshift_data_loader_lambda_db_password

# The ARN and name of the bucket where data is being stored (which should be loaded into Redshift)
Error: Required variable not set: s3_bucket_arn
Error: Required variable not set: s3_bucket_name

The "*.tfvars" are gitignored. You should now add the details of your own setup into your "dev.tfvars", in case you want to test drive this code. After you do this, run again "make plan" to see what terraform plans to deploy.

Core AWS Lambda function module

OK, before we deploy, let us have a look on the terraform code, starting with the Lambda function. Here we are using a specific version of a generic module that we had previously created for Lambda functions. The beauty of this module is its flexibility. For example, you can launch the Lambda inside or outside a VPC, you can attach how many additional IAM policies you desire, and specify a SNS or SQS ARN for Dead Letter Queue (DLQ) for failure handling.

The first thing you can notice is the usage of a different dedicated git repository via the "source" keyword for the Lambda module. Go ahead and have a look, this is also public, and we have been using it successfully in several projects. Also note that one can pin point different versions of a given repository, which maps to repository tags. This reassures retro-compatibility and non breaking progress on each module.

The second thing you might notice is that we are passing paths of the Lambda function. The reason behind it is that our terraform lambda module will proactively zip our lambda's "src" module; in our repository located in: "etl/lambda/redshift/src".

The second is that we are instantiating the lambda inside a VPC. As mentioned before, this is optional - the module is flexible enough for you to chose if you want to have the lambda being executed inside a specific VPC or not. In our case, we are assuming that your Redshift cluster is indeed inside a private subnet (as advisable), and thus we need to execute the Lambda function inside that VPC to be able to connect to Redshift.

Please remember to either open Redshift's Security Group either to the specified subnet CIDR, or to our Lambda's security group id.

Lambda environment variables

Next point up the list are environmental variables. The module accepts those via the variable "lambda_env_vars". We have defined those via terraform "locals", with "${local.redshift_loader_lambda_env_vars}". Let us have a look at those briefly:

We are passing mainly all parameters which the lambda function uses to connect to Redshift DB. Note also that we are not passing the explicit Redshift DB password, but rather the name of the parameter stored in SSM store.

You may ask yourself why we are doing this, considering that this lambda module is already encrypting environmental variables by default. Yes, the lambda module creates automatically a KMS key for this purpose. We are doing it in the end more for illustration purposes, although one could argue that concealing these variables in the AWS GUI is also relevant so that not all users that have access to it can discover the password.

Lambda permissions

You will verify that we simplified this code in several aspects. A good example are the broad permissions passed to the lambda function for all buckets, among others. A (much) better approach would be to clearly restrict to a small white list of resources. But, for the sake of the tutorial, we will relax on some of these aspects that you should consider when bringing this inside your company.

Note that we are passing permissions to use KMS keys, which is specially relevant if your buckets are encrypted.

Moreover, we are also passing permission to access the SSM parameter store specific DB password.

The final statement provides the Lambda function permissions to publish in a specific SNS topic, which leads us to the final point: failure handling.

Lambda Failure handling

Currently Lambda functions executed asynchronously - as is the case of S3 triggered Lambdas - will always retry twice on failure. the only thing you have to do is to configure the Dead Letter Queue (DLQ) configured, by passing a SNS topic to publish in case the maximum number of retries is reached. Note: Please make sure you always double check in AWS documentation for updates to find the current limits and default behaviors.

In our case, we have configured a SNS topic, so that subscribers can subscribe and be alerted of failure. For example, one can add an email subscriber, and be notified on Lambda failure. Unfortunately, Terraform does not support the creation of SNS Email subscriptions as explained here. Thus either subscribe manually, or use CloudFormation/custom script for this task.

Deploying the code

Cool, we are ready to rock&roll. Just open your terminal and type:

export ENVIRONMENT=dev
make apply

This will prompt for confirmation, before deploying.

Go ahead and create a table in Redshift, such as for example "kpi". Next upload a new CSV file into your S3 bucket with a prefix "kpi" (so that a key "kpi/your_file.csv" is saved in the bucket). You should be able to see data in your Redshift table in less than a minute. Moreover, you can also consult your Cloudwatch logs, and verify that the Lambda function has executed successfully.

Congratulations, you have deployed a asynchronous pipeline using infrastructure-as-code.

As usual, let us summarize the sources used in this post:

We hope you found this tutorial useful. Also please do not hesitate to contact us at mklabs, or me directly - diogo [at] mklabs [dot] io - if you need any help on your project, be it cloud/devOps/Big Data/ML.

AWS Server-less data pipelines with Terraform - Part 1

diogoaurelio — Mon, 03 Sep 2018 07:47:41 +0000

This post is the first of sequence of posts focusing on AWS options to setup pipelines in a serverless fashion. The topics that we all cover throughout the whole series are:

Part 1: Python Lambda to load data into AWS Redshift datawarehouse
Part 2: Terraform setup of Lambda function for automatic trigger
Part 3: Example AWS Step function to schedule a cron pipeline with AWS Lambda

In this post we lean towards another strategy to setup data pipelines, namely event triggered. That is, rather than being scheduled to execute with a given frequency, our traditional pipeline code is executed immediately triggered by a given event. Our example consists of a demo scenario for immediately and automatically loading data that is stored in S3 into Redshift tutorial.

Note that triggers are not something new exclusive from a Cloud paradigm. Most databases, for example, have triggers that one can setup to execute User Defined Functions. However, the multitude of integrations that AWS Lambda functions have with its own services is quite impressive and useful. In this case, it is AWS S3 that can trigger the execution of the Lambda function.

The typical first use-case that people think for integrating S3 with Lambda functions via triggers is for image processing, such as thumbnails. In this post we are using it for a second use case, namely near real time data imports to AWS Redshift. From the developers perspective, the effort is quite minimal, since AWS does all the lifting in the background.

Without further ado, let us dive into the code.

The Lambda Function

The Python code is a very simple Redshift loading code. We will later show in the terraform code, but important to note is that an S3 bucket will be configured to trigger/invoke our Lambda function whenever a new object/key is saved on that S3 bucket. This trigger is active for the whole Bucket, which means any new file copied into the bucket will trigger our Lambda function.

AWS will pass to the lambda function's parameters the Bucket which triggered the function, as well as the new S3 key. Now there is a very important assumption that you need to know, namely that we assume that this key is created with a given prefix, which is the same as the target Redshift table you want it to be imported. For example, if you copy a new file into "kpis/new_file.csv", the lambda will split on "/", and use "kpis" as the table name in Redshift. And there you go, you are ready to see the code.

The lambda function code is quite simple. It simply builds a string with the Redshift Copy command, implicitly assuming a CSV file as the underlying format. Why use "Copy command"? This is one the Redshift best practices main recommendations, and provides two main benefits: speed (data is loaded in parallel) and data storage optimizations.

As a side comment, please do not assume we are recommending using CSV file format. We just chose it due to the easiness that one can take the code in this tutorial and test it live, rather then using more optimized binary formats, such as Avro or Parquet.

Secrets

Note that the lambda function is receiving all relevant parameters and secrets via environment variables. So, besides the simple bureaucracies for the copy command, there is just one thing more we would just like to highlight. For Redshift's user password, we are using an additional AWS service, namely AWS SSM parameter store. This is a relatively recent service (from 2016 if I am not mistaken) that allows you to store secrets encrypted using KMS key. Now the cool thing about it is that now you can control granularly exactly which instances and roles have access to what.

This means that we need to make sure we provide via IAM policy permissions for this Lambda function to use the specified SSM paramter. We will show how to do this later in the terraform code.

Last thoughts

Before we end our first post, we would like to add that there are a lot of things to be improved in this lambda. One suggestion would be to guarantee deduplication, namely that the same data does not get loaded twice into Redshift. One simple way to do this is by using a database, to guarantee synchronization. This would also allow better error handling.