DEV Community: Adin Hodovic

Creating a Low Cost Managed Kubernetes Cluster for Personal Development using Terraform

Adin Hodovic — Sat, 15 Aug 2020 22:13:05 +0000

Kubernetes is an open-source system that's popular amongst developers. It automatically deploys, scales, and manages containerized applications. Yet for those working outside of the traditional startup or corporate setting, Kubernetes can be CPU and memory-intensive, disincentivizing developers from using a strategically beneficial tool. We'll highlight some of the constraints posed by using Kubernetes and offer a series of low-cost, practically-beneficial solution: Google Kubernetes Engine deployed and managed with Terraform.

Want to run Kubernetes locally? Kind is a tool that’s remarkably easy to set up and experiment with. It even supports multi-node clusters! However, the abundance of pods and nodes can be CPU/memory intensive. Another potential roadblock is the popularity of managed Kubernetes solutions: since many companies go this route, you'd want to have some experience with a managed solution. Gaining this experience in a non-corporate setting means personally using Managed Kubernetes –– a costly investment for a solo developer. The baseline cost for an EKS cluster is $72 per month, with similar pricing for regional AKS and GKE clusters. Add in additional node costs for running your workloads and before you know it, you’re running a hefty monthly bill. Here are some potential ways to address computer and cost constraints with Kubernetes.

Lowering costs

There are two ways to lower the primary managed Kubernetes cluster costs:

A free cluster(no monthly cost for the control plane).
Spot instances/Preemptible VMs - instances that can be terminated at any time within 2 minutes.

AWS does not offer a free cluster in any way, you have a $72 monthly cost from the get go. On top of that their managed node groups do not offer spot instances currently, you'd have to use custom node groups to use spot instances which can be achieved with eksctl or EKS Terraform modules. Regarding AKS, I've never used AKS or Azure Cloud in any form so I'll not explore that option.

Google Cloud offers a free zonal cluster per account. A zonal cluster has a Kubernetes control plane only available in a single zone –– Which means that if there is an outage in that zone, the unavailable control plane prevents you from configuring your workloads. However, GKE still has an SLA with a guaranteed availability of 99.5% uptime for a zonal cluster and you'd most likely not be heavily affected by downtime since you use the cluster for personal learning and experimentation.

Another feature of Google Cloud is the ability to create a managed node pool. Additional benefits include preemptible VMs for your GKE cluster, along with a Terraform resource that supports creating node pools with preemptible VMs. Google Cloud's preemptible VMs have no guaranteed uptime and are removed after 24h. Usually, you’ll have a couple of minutes of downtime daily for all workloads scheduled on preemptible VMs. However, these instances are up to 80% cheaper!

Creating a Zonal Cluster with a Preemptible Node Group using Terraform

To create a zonal cluster you need to specify only a single zone as the cluster location, but the nodes can be located in multiple zones.

resource "google_container_cluster" "default" {
  name               = "my-cluster"
  location           = "europe-west1-b" # MUST BE A SINGLE ZONE, OTHERWISE IT COUNTS AS A REGIONAL CLUSTER
  min_master_version = "version"

  node_locations = ["europe-west1-b", "europe-west1-c"] # CAN BE MULTI ZONE

  # We can't create a cluster with no node pool defined, but we want to only use
  # separately managed node pools. So we create the smallest possible default
  # node pool and immediately delete it.
  remove_default_node_pool = true
  initial_node_count       = 1
}

We will reference the above cluster when creating a node pool.

The node pool will consist of memory-optimized n2d-highmem-4 instances and they will have preemptible set to true. The n2d-highmem-4 instances cost $133.16 monthly but running them in a preemptible node group brings down the costs to $40.27.
These instances have 4 vCPUs and 32GB of memory making it possible to run many various workflows, you can deploy prometheus-operator, elasticsearch and anything else you'd like to learn!
You can scale down and run smaller instance types as the standard n2d-standard-2 machine type that has 2 vCPUs and 8GB of memory which is priced at $14.93 monthly when running in a preemptible node pool($49.36 monthly when not running in a preemptible node pool).
There are even smaller instances as the shared-core f1-micro(monthly $4.53 - default / $1.36 - preemptible) and g1-small(monthly $11.76 - default / $3.54 - preemptible) instance types, but they have low computing power.

There are some additional costs for networking and storage but they should be minimal compared to instance and control plane costs for a small cluster.

resource "google_container_node_pool" "memory_optimized" {
  name     = "Memory optimized node pool"
  cluster  = google_container_cluster.default.name

  version = "my-version"

  initial_node_count = 1
  autoscaling {
    min_node_count = 1
    max_node_count = 3
  }

  management {
    auto_repair = true
  }

  node_config {
    machine_type = "n2d-highmem-4" # Memory-optimized
    preemptible  = true # Preemptible needs to be true
  }
}

Tainting nodes

If you prefer higher uptime for some applications and want to avoid scheduling some pods to the preemptible node pool you can taint the node pool. Tainting means that pods don't get scheduled to a particular node pool unless they tolerate the set taints. The below example sets taints using the key role with the value ops with the effect NO_SCHEDULE, meaning that a pod needs to tolerate these taints to be scheduled onto the nodes in that node pool.

resource "google_container_node_pool" "memory_optimized" {
  name     = "Memory optimized node pool"
  cluster  = google_container_cluster.default.name

  version = "my-version"

  initial_node_count = 1
  autoscaling {
    min_node_count = 1
    max_node_count = 3
  }

  management {
    auto_repair = true
  }

  node_config {
    machine_type = "n2d-standard-4"
    preemptible  = true # Preemptible needs to be true

    # Set the taints
    taint {
      key    = "role"
      value  = "ops"
      effect = "NO_SCHEDULE"
    }
  }
}

To tolerate the node pool taints add the tolerations section to a Kubernetes spec:

---
apiVersion: v1
kind: Deployment
metadata:
  name: '{{ app_name }}'
  labels:
    app: '{{ app_name }}'
  namespace: '{{ namespace }}'
spec:
  selector:
    matchLabels:
      app: '{{ app_name }}'
  template:
    metadata:
      name: '{{ app_name }}'
      labels:
        app: '{{ app_name }}'
    spec:
      ############### This section is needed ###############
      tolerations:
        - effect: NoSchedule
          key: role
          operator: Equal
          value: ops
      containers:
        <your containers>
      ######################################################

Now you can create cheap preemptible node pools that are dedicated to specific workflows. For example you might want to learn how to create a Elasticsearch cluster with multiple replicas, then you might want a preemptible node pool for the cluster since it requires a lot of computing power. Or you can create and use only preemptible node pools for your whole cluster. Combine that with GKE's free zonal cluster and you can bring down costs to ~$20 monthly with a decently sized instance type n2-standard-2. Here's a link to the pricing calculation using GCP's calculator. Take advantage of it, and get experience using a managed Kubernetes solution.

Github's Awesome Repository Traffic Analytics

Adin Hodovic — Sun, 02 Aug 2020 20:59:28 +0000

As someone that loves all types of metrics I was really glad to have discovered Github's traffic analytics. They've been around for a while, but I never knew about them and I wanted to share an introduction to the analytics.

You can access the analytics by navigating to insights -> traffic. Github displays various graphs/tables:

Git clones
Amount of visitors
Referring sites
Popular content

I use Git clones to understand both usage of the repository and OSS contributions by other developers. Repository usage is particularly insightful when creating projects/modules that are downloaded and used via Git. An example of this is my recent Terraform project terraform-cloudflare-maintenance, every time someone uses the module it gets cloned. Here's the analytics graph on Github clones:

The amount of visitors is straightforward, just a simple graph of visitors. However, you can pair this graph with dates you shared your project on Hackernews or Reddit and see if there are any traffic spikes! I shared the Terraform project on Reddit and got around 50 stars and a traffic spike from Reddit as you can see below:

The most interesting data presented by Github are referring sites. Just as any business analytics tool, you can see key referral sites. I measure the success of my previously shared projects on various platforms. Then, I know where I should share all my future projects. So far, I've had great success with SRE/DevOps specifics subreddits (/r/terraform, /r/sre, /r/devops). The same project as in the previous images can be seen below:

Lastly Github tracks the most popular content in your repository, however in my case this has not been particularly useful. I'll anyways share a screenshot of the table from the same repository:

I loved the insights and analytics, hope you do too!

Quick, Pretty and Easy Maintenance Page using Cloudflare Workers & Terraform

Adin Hodovic — Sat, 01 Aug 2020 22:26:07 +0000

Maintenance pages are a neat way to inform your users that there are operational changes that require downtime. Cloudflare Workers allows you to execute Javascript and serve HTML close to your users and when Cloudflare manages your DNS it makes it easy to create a Worker and let it manage all traffic during a maintenance period. This makes a great solution for smaller teams/applications that do not want to invest time into creating and displaying maintenance pages. Terraform is a tool that enables provisioning infrastructure via code and it integrates great with Cloudflare APIs.

A preview of the final implementation can be found here, and I've created a Terraform module for creating the maintenence page.

Cloudflare Worker

The Worker's javascript is simple and it is found and explained in depth in this blog post. The snippet forwards the request to the Web Server if the connecting IP is whitelisted, otherwise it returns a html response that you've defined which in our case is the maintenance page.

async function fetchAndReplace(request) {

  let modifiedHeaders = new Headers()

  modifiedHeaders.set('Content-Type', 'text/html')
  modifiedHeaders.append('Pragma', 'no-cache')


  //Allow users from trusted into site
  if (white_list.indexOf(request.headers.get("cf-connecting-ip")) > -1)
  {
    //Fire all other requests directly to our WebServers
    return fetch(request)
  }
  else //Return maint page if you're not calling from a trusted IP
  {
    // Return modified response.
    return new Response(maintenancePage, {
      status: 503,
      headers: modifiedHeaders
    })
  }
}

let maintenancePage = `
<!doctype html>
<title>Site Maintenance</title>
<div class="content">
    <h1>We&rsquo;ll be back soon!</h1>
    <p>We&rsquo;re very sorry for the inconvenience but we&rsquo;re performing maintenance. Please check back soon...</p>
    <p>&mdash; Awesome Team</p>
</div>
`

Terraform

We'll now use Terraform to create the script and to create the worker route. Creating a worker script with Terraform can be done with the cloud_worker_script resource:

resource "cloudflare_worker_script" "this" {
  name    = "maintenance"
  content = file("/maintenance.js")
}

This will create a Cloudflare worker script which we can reference with the cloudflare_worker_route. The route resource will deploy the worker in a zone with a specified url pattern:

data "cloudflare_zones" "this" {
  filter {
    name = "hodovic.cc/*"
  }
}

resource "cloudflare_worker_route" "this" {
  zone_id     = lookup(data.cloudflare_zones.this.zones[0], "id")
  pattern     = "hodovi.cc/maintenance/*"
  script_name = cloudflare_worker_script.this.name
}

All users that visit the /maintenance/ path will be shown the maintenance page except for the whitelisted IPs.

Terraform Module

The solution presented above is neat, although we'd like a resuable module. Therefore, I've created a Terraform module that does the creation and deployment of a Cloudflare Worker. However, for the module to be reusable we need dynamic html content as:

Logo
Font
Company/Team name
Favicon
Email address for support

Cloudflare supports key/value storage which we can define in the Terraform module. We'll adjust the previous cloudflare_worker_script to add the key/value storages:

resource "cloudflare_worker_script" "this" {
  name    = "maintenance"
  content = file(format("%s/maintenance.js", path.module))

  plain_text_binding {
    name = "COMPANY_NAME"
    text = var.company_name
  }

  plain_text_binding {
    name = "WHITELIST_IPS"
    text = var.whitelist_ips
  }

  plain_text_binding {
    name = "LOGO_URL"
    text = var.logo_url
  }

  plain_text_binding {
    name = "FAVICON_URL"
    text = var.favicon_url
  }

  plain_text_binding {
    name = "FONT"
    text = var.font
  }

  plain_text_binding {
    name = "EMAIL"
    text = var.email
  }
}

We can create a HTML snippet and pass the environment variables:

let maintPage = (company_name, logo_url) => `
<body>
    <div class="content">
        <img class="logo" src="${logo_url}" alt="${company_name}">
        <div class="info">
            <h1>Our site is currently down for maintanence</h1>
            <p>We apologize for any inconveniences caused and we will be online as soon as possible. Please check again in a little while. Thank you!</p>
            <p>&mdash; ${company_name}</p>
        </div>
        <img class="image-main" src="https://i.imgur.com/0uJkCM8.png" alt="Maintenance image">
        <hr />
        <a href="mailto:${email}?subject=Maintenance">You can reach us at: ${email}</a>
    </div>
</body>
`;

We'll also create dynamic CSS for fonts:

body {
    text-align: center;
    font-family: "${font}", sans-serif;
    color: #0C1231;
}

The html/css snippets are more complex and dynamic and the full source code can be found on Github.

The module supports several variables:

Logo
Font (Google fonts)
Email
Team name
Whitelisting IPs

You can create it using Terraform's module:

module "hodovi_cc_maintenance" {
  source          = "git::git@github.com:adinhodovic/terraform-cloudflare-maintenance.git?ref=v0.1.3"
  enabled         = false
  cloudflare_zone = "hodovi.cc"
  pattern         = "hodovi.cc/maintenance/*"
  company_name    = "HoneyLogic"
  email           = "support@honeylogic.io"
  font            = "Poppins"
  logo_url        = "https://s3.eu-west-1.amazonaws.com/honeylogic.io/media/images/Honeylogic-blue.original.png"
  favicon_url     = "https://s3.eu-west-1.amazonaws.com/honeylogic.io/media/images/Honeylogic_-_icon.original.height-80.png"
}

A preview of the maintenance page can be found here, fully adjustable to your team needs!

Django A/B testing with Google Optimize

Adin Hodovic — Sun, 14 Jun 2020 22:13:37 +0000

Struggling to determine your product’s future path? A/B testing helps you decide if you should take the road less traveled by.

To understand your users and their needs, start by creating two product variations and collecting data points. Google Optimize simplifies this process by managing variant weights, targeting rules, and providing analytics. It also seamlessly integrates with other G-Suite products, such as Google Adwords and Google Analytics.

There are two ways to run A/B tests: client-side and server-side. Google Optimize’s client-side A/B tests will execute Javascript in-browser based on your preferred changes, like alterations to style or positioning of elements. While this is useful, it’s also restrictively limited.

On the other hand, server-side A/B tests have few restrictions. You can write any code, displaying different application versions based on the session's active experiment variant.

The Google Optimize basics

When a user visits your site and an experiment is active, he’ll be given an experiment variant which is stored in a cookie called _gaexp. The cookie can have the following content GAX1.2.MD68BlrXQfKh-W8zrkEVPA.18361.1 and it contains multiple values where each value is divided by dots. There are two key values for server-side A/B testing: First, the experiment ID which in the example is MD68BlrXQfKh-W8zrkEVPA. Second, the experiment variant for the session which is the last integer, which in this case is 1.

When multiple experiments are active the experiments will be separated by an exclamation marks as in the following _gaexp cookie GAX1.2.3x8_BbSCREyqtWm1H1OUrQ.18166.1!7IXTpXmLRzKwfU-Eilh_0Q.18166.0. You can extract two experiments from the cookie, 3x8_BbSCREyqtWm1H1OUrQ with the variant 1 and 7IXTpXmLRzKwfU-Eilh_0Q with the variant 0.

Usage in Django

We can parse a ga_exp cookie with this code snippet:

ga_exp = self.request.COOKIES.get("_gaexp")

parts = ga_exp.split(".")
experiments_part = ".".join(parts[2:])
experiments = experiments_part.split("!")
for experiment_str in experiments:
    experiment_parts = experiment_str.split(".")
    experiment_id = experiment_parts[0]
    variation_id = int(experiment_parts[2])
    experiment_variations[experiment_id] = variation_id

The snippet separates the cookie by Google Optimize's delimiters which are dots(.) for different values and exclamation marks(!) to separate experiments. We will then store the experiment variant for each experiment in a dict. Then, we can render various templates based on which experiment variants are active for the current session:

def get_template_names(self):
    experiments.get("my_experiment_cookie_id", None)
        if variant == "1":
            return ["jobs/xyz_new.html"]
        return ["jobs/xyz_old.html"]

I've created a Django package that simplifies this and adds both middleware and experiment management in the Django-admin. The package makes it possible to start, stop and pause experiments through the Django admin. It also simplifies local testing of experiment variants. I use the package at Findwork to A/B test UI/UX changes.

Next up, I'll walk you through how to setup an experiment in Google Optimize and how to render various application versions based on the session's experiment variant using the package.

Using Django-google-optimize to run A/B tests

To get started with django-google-optimize install the package with pip:

pip install django-google-optimize

Add the application to installed Django applications:

DJANGO_APPS = [
    ...
    "django_google_optimize",
    ...
]

I've added a middleware which makes the experiment variant easily accessible both in templates and views so you'll need to add the middleware in you settings.py:

MIDDLEWARE = [
    ...
    "django_google_optimize.middleware.google_optimize",
    ...
]

Now run the migrations ./manage.py migrate and the setup is complete.

Creating a Google Optimize experiment

Head over to Google Optimize and create your first experiment. By default, you'll have one variant –– this is the original variant. Feel free to add as many variants as you’d like for the A/B test. Each variant will be given an index based on the order you added the variants, but the original variant will always have the index 0:

Next, we'll set goals for the A/B test. Be sure to link your Google Analytics property with Google Optimize to set goals, track metrics, and measure the success of your experiment. If you’ve yet to add Google Analytics to your site, django-analytical provides easy integration.

After Google Analytics is linked, you’ll be able to add goals to the experiment. These are categorized in two ways: system default (ex. Bounce rate, page views) or user-defined goals.

There is a predetermined limit of 3 goals. One will be a primary goal, determining the success of the experiment. The other two are visually displayed later in Google Optimize’s experiment report. All other user-defined and system goals will be visible in Google Analytics, tied individually to each experiment. Don’t worry –– this is only visually displayed and does not determine the success of the experiment.

Below is an experiment with two system goals and a goal I've defined for Findwork which measures how many users have subscribed to job emailing list:

Variants and goals are required to start the experiment, but Google Optimize provides a host of other options. Audience targeting (with Google Ads and Google Analytics integration), variant weights, traffic allocation, and more can also enhance your research. If you’re already using Google's products as Google Analytics and Adwords, then Google Optimize is another effective tool in the toolbox.

Adding an experiment in the Django-admin

Let’s head over to the Django-admin to add our experiment. When adding an experiment, only the experiment ID is required. After it’s, added a middleware will add the experiments and their variants to a context variable called google_optimize. The experiments will be accessible in both views and templates (the request context processor needs to be added for the context to be accessible in templates) in the request.google_optimize.<experiment_id> object. However, referring to the experiment by ID in your Django templates and views will make it difficult to grasp which experiment the specific ID refers to. Therefore, you can (and should) add aliases for the experiments and variants. This will make your code legible and parsimonious. Here’s is an example of the differences::

Without an Google Experiment alias you will have to reference your experiment by ID:

{% if request.google_optimize.3x8_BbSCREyqtWm1H1OUrQ == 0 %}
{% include "jobs/xyz_new.html" %}
{% endif %}

Instead of by experiment alias:

{% if request.google_optimize.redesign_landing_page == 0 %}
{% include "jobs/xyz_new.html" %}
{% endif %}

Without a variant alias you will have to reference your variant by index as:

{% if request.google_optimize.redesign_landing_page == 0 %}
{% include "jobs/xyz_new.html" %}
{% endif %}

Instead of by variant alias:

{% if request.google_optimize.redesign_landing_page == "New Background Color" %}
{% include "jobs/xyz_new.html" %}
{% endif %}

For the examples above the following Google Experiment object has been added:

Local Development

To test the variants, you can either add the _gaexp cookie or manage the active variant in the Django-admin. For each browser, using either plugins or developer tools, adjust the _gaexp cookie and test each variant by changing the variant index in the session's cookie. For example, in the cookie GAX1.2.MD68BlrXQfKh-W8zrkEVPA.18361.1 you can adjust the last integer 1 to the variant index you want to test. Alternatively, django-google-optimize provides each experiment with an experiment cookie object, which then sets the active variant. In an experiment's Django admin panel, you can add a cookie like in the image below:

Make sure to set it to active to override or add the variant chosen, otherwise the session's cookie will be used! The experiment cookie only works in DEBUG mode. Google sets the cookie in production and we have no desire to override it.

Usage

As you’ve learned, you can use django-google-optimize in templates as below:

{% if request.google_optimize.redesign_landing_page == "New Background Color" %}
{% include "jobs/xyz_new.html" %}
{% endif %}

To use the package in views and display two different templates based on the experiment variant

def get_template_names(self):
    variant = request.google_optimize.get("redesign_landing_page", None)
    if variant == "New Background Color":
        return ["jobs/xyz_new.html"]
    return ["jobs/xyz_old.html"]

Adjusting the queryset in a view based on the experiment variant:

def get_queryset(self):
    variant = self.request.google_optimize.get("redesign_landing_page", None)

    if variant == "New Background Color":
        qs = qs.exclude(design__contains="old")

Results

Google Optimize continuously reports results on the experiment, displaying both the probability for a variant to be better and the modelled improvement percentage. Let this experiment run for at least two weeks –– after that, you’ll have a clear leader from your A/B testing.

You’ll still receive daily updates on the experiment’s progress, where you'll have all the insights you'll need to end the experiment. At the top of the reporting page, there will be a link to a Google Analytics dashboard for the experiment as seen below in the image:

The final result of the experiment will be displayed like the image below. This displays an overview on how well each experiment variant performed in regards to the experiment goals set:

The above examples ended a few sessions early, making it difficult to pinpoint an exact modelled improvement of a variant. If you’d like concrete results before concluding an experiment, it must run longer than 14 days.

Summary

Google Optimize’s smooth integration with Google Analytics makes it remarkably easy to analyze each A/B variant in depth. The Django-Google-Optimize package simplifies this process even more. Use this step-by-step guide when developing applications to gain a deeper understanding of your customers, their needs, and how you can develop a product to fulfill them.

The project can be found on Github.

The package is published on PyPI.

Documentation is available at read the docs.

Monitoring Kubernetes InitContainers with Prometheus

Adin Hodovic — Fri, 29 Nov 2019 20:37:30 +0000

Kubernetes InitContainers are a neat way to run arbitrary code before your container starts. It ensures that certain pre-conditions are met before your app is up and running. For example it allows you to:

run database migrations with Django or Rails before your app starts
ensure a microservice or API you depend on to is running

Unfortunately InitContainers can fail and when that happens you probably want to be notified because your app will never start. Kube-state-metrics exposes plenty of Kubernetes cluster metrics for Prometheus. Combining the two we can monitor and alert whenever we discover container problems. Recently, a pull-request was merged that provides InitContainer data.

The metric kube_pod_init_container_status_last_terminated_reason tells us why a specific InitContainer failed to run; whether it's because it timed out or ran into errors.

To use the InitContainer metrics deploy Prometheus and kube-state-metrics. Then target the metrics server in your Prometheus scrape_configs to ensure we're pulling all the cluster metrics into Prometheus:

- job_name: 'kube-state-metrics'
  static_configs:
    - targets: ['kube-state-metrics:8080']

kube_pod_init_container_status_last_terminated_reason contains the metric label reason that can be in five different states:

Completed
OOMKilled
Error
ContainerCannotRun
DeadlineExceeded

We want to be alerted whenever a metric that is not 'Completed' is scraped because that means an InitContainer has failed to run. Here is an example alerting rule.

groups:
  - name: Init container failure
    rules:
      - alert: InitContainersFailed
        expr: kube_pod_init_container_status_last_terminated_reason{reason!="Completed"} == 1
        annotations:
          summary: '{{ $labels.container }} init failed'
          description: '{{ $labels.container }} has not completed init containers with the reason {{ $labels.reason }}'

Happy monitoring!

6 ways to speed up your CI

Adin Hodovic — Fri, 29 Nov 2019 17:53:28 +0000

Waiting for CI to finish slows down development and can be extremely annoying, especially when CI fails and you have to run it again. Let's take a look into approaches on how to speed up your CI and minimize the inefficient time spent by developers when waiting on CI to finish.

We'll go through 6 different methods to speed up CI:

Makisu as your Docker build tool
Caching between Stages
Concurrent Jobs for each Stage
Running Tests across all CPUs
Parallel Tasks
Autoscaling CI Runners

Building Docker Images with Makisu + Redis KV storage

We've tried three approaches; default image building with Docker using --cache-from, Google's Kaninko with --cache=true and Uber's Makisu with Redis KV storage.

Kaniko is Google's image build tool which enables building images without a docker daemon, making it great for building images within a Kubernetes cluster as it is not possible to run a docker daemon in a standard Kubernetes cluster.

Makisu is also an image build tool that does not rely on the docker daemon making it work great within a Kubernetes cluster and it also adds a couple of new things as e.g distributed cache support and customizable layer caching, possibility to choose what layers to cache.

Docker provides it's default image building with options as --cache-from to chose images to cache from. Building images with Docker can be excluded as an option if you are running CI within a standard Kubernetes cluster.

We have had the fastest CI with Makisu even though the setup is more complex due to deploying Redis.

With Makisu we got the following perks:

Distributed cache support, builds that share Dockerfile directives can share cache. Cache hit rate is better across branches than Docker's --cache-from which is great for single branch cache.
Great image compression for large images
Customizable layer generation and caching, possibility to choose what layers to cache and generate with the function #!COMMIT.
Overall we experienced a faster build time than with the other build tools.

You can read more on why Makisu is great here. Now let's get going by deploying a Redis node for Makisu.

Deploying Redis(AWS Elasticache) with Terraform

Redis is an in-memory key-value storage and we deploy it alongside Makisu as Makisu uses it as a key-value storage to map the Dockerfile directives with the hash of the Dockerfile directive layers. Elasticache is a fully managed Redis solution by AWS.

Below is a example of how to deploy an Elasticache Redis service with Terraform. If you intend or have already deployed Redis in a different way skip til the next section.

We'll create a Elasticache subnet group in your chosen VPC for the Redis node with Terraform:

resource "aws_elasticache_subnet_group" "gitlab_runner" {
  name       = "gitlab-runner"
  subnet_ids = module.[ci_runners_vpc].private_subnets
}

And then we'll create the Elasticache Redis cluster within the subnet group created.

resource "aws_elasticache_cluster" "gitlab_runner" {
  cluster_id           = "gitlab-runner"
  engine               = "redis"
  node_type            = "cache.t3.micro"
  num_cache_nodes      = 1
  subnet_group_name    = aws_elasticache_subnet_group.gitlab_runner.name
  parameter_group_name = "default.redis5.0"
  engine_version       = "5.0.5"
  port                 = 6379
}

Note that the Elasticache node needs to share the same VPC(Virtual Private Cloud, used to share network amongst AWS resources) as the CI runners deployed, otherwise the runners won't be able to access the Redis service.

Creating an VPC to be sharable between the Elasticache node and your CI runners can be simplified with the Terraform VPC module:

module "gitlab_runner_vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "gitlab_runner_vpc"
  cidr = "10.1.0.0/16"

  azs             = ["eu-west-1a"]
  private_subnets = ["10.1.1.0/24"]
  public_subnets  = ["10.1.101.0/24"]

  map_public_ip_on_launch = "false"

  tags = {
    Environment = var.environment
    Terraform   = true
  }
  ...
}

Now you can specify the VPC created for both the CI VMs and the Elasticache subnet group. For example we use the awesome terraform module to deploy Gitlab runners on cheap spot instances and as shown below you use vpc_id = module.gitlab_runner_vpc.vpc_id to place CI VMs into the VPC created above.

module "gitlab_runner" {
  source = "npalm/gitlab-runner/aws"

  aws_region  = var.aws_region
  environment = var.environment

  vpc_id                   = module.gitlab_runner_vpc.vpc_id
  subnet_ids_gitlab_runner = module.gitlab_runner_vpc.private_subnets
  runners_name             = "gitlab_runner_honeylogic"
  runners_gitlab_url       = "https://gitlab.com"
  runners_concurrent       = "5"
  runners_limit            = "5"
  runners_idle_time        = "1800"
  runners_idle_count       = "0"

  instance_type = "t3.large"
  ...
}

Then when you create the Elasticache subnet group, specify the VPC's subnets you created previously:

resource "aws_elasticache_subnet_group" "gitlab_runner" {
  name       = "gitlab-runner"
  subnet_ids = module.gitlab_runner_vpc.private_subnets
}

Now you have Gitlab runners that have access to an Redis(Elasticache) node which Makisu uses as a key value storage for caching.

Using Makisu

Now when Redis is setup we'll setup the Gitlab-CI stages(we use Gitlab-CI but tailor the setup to your own CI). We create templates for common CI builds so they become reusable across projects. The below template creates a docker image and tags it with latest and the git commit short sha.

gitlab-ci-makisu-build.yml

.build:
  extends:
    - .default_vars
  stage: build
  image:
    name: gcr.io/makisu-project/makisu-alpine:v0.1.12
    entrypoint: [""]
  before_script:
    - echo "{\"$CI_REGISTRY\":{\".*\":{\"security\":{\"basic\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}}}" > $REGISTRY_CONFIG
  variables:
    REDIS_CACHE_ADDRESS: my-redis-address:6379
    REGISTRY_CONFIG: /registry-config.yaml

build:default:
  extends:
    - .default_vars
    - .build
  script:
    - /makisu-internal/makisu --log-fmt=console build --push=$CI_REGISTRY --modifyfs -t=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --replica $CI_REGISTRY_IMAGE --redis-cache-addr=$REDIS_CACHE_ADDRESS --registry-config=$REGISTRY_CONFIG $CI_PROJECT_DIR

To better grasp the YML code piece and all predefined environment variables below I'll explain how the above job operates:

Clones the repository and Gitlab-CI populates a bunch of predefined environment variables.
We echo the credentials by using predefined variables as $CI_REGISTRY(CI registry for the project) and $CI_REGISTRY_USER/PASSWORD into a registry config file which Makisu uses for authorization.
The .build definition is just used as a template for all other Makisu build stages so common tasks are stored in the template. Templating stages is a Gitlab-CI feature.
We create two variables the REGISTRY_CONFIG location and the REDIS_CACHE_ADDRESS which are accessible by any stage that extends the .build stage.
Now we can create the build:default job which extends the build stage and then add the script where we use Makisu to build the image.
We use a couple of variables as -t(required) the tag for the image but also --replica which indicates the additional tags you want to push the image as. As mentioned previously we use image:latest and image:git_commit_short_sha as tags as you can see in the command.
Remember to specify the --redis-cache-addr so it uses Redis as a KV store. You also always need to specify the build context which by default is the $CI_PROJECT_DIR in our case.

Then we include the Makisu build stage in any repository where we need to build docker images:

include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-base.yml'
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-makisu-build.yml'

I've written a blog on Gitlab-CI templates which will help you grasp how the pieces work together and how you extend and reuse stages from templates.

Uber's Makisu increases the speed of Docker builds by a great amount of time(average 30-40% for us, for Uber itself it is from 40% on average up to 90% faster) and it has many more extensible options. You do not have to use Redis, you can use a HTTP cache. There is a possibility as well to not cache each Dockerfile directive as some stages can be excessive to cache and layer. An example is shown below which is used with the --commit=explicit flag. Specify #!COMMIT and it will only cache the layers with the #!COMMIT comment.

Dockerfile

RUN pip install -r requirements.txt #!COMMIT

Caching between Stages

This section is not correlated with the previous Docker image build caching.

When running CIs you'll most likely have several stages, and sometimes you might the run similar commands across stages. An example of this are dependencies e.g React's NPM modules that you install. For each stage you do not want to install all NPM modules, instead you want to cache them and make them reusable to not slow down the CI with repetitive jobs. With Gitlab caching between stages is achievable by defining the cache variable in your CI yaml file.

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
  - node_modules/

Now the node modules created do not need to be recreated for each stage. Many other CIs support this as Travis-CI and Circle-CI.

Running Tests across all CPUs

Run your tests spread across all CPUs which will speed CI up, make sure you run VMs that are compute based. As I am familiar with Python the most I'll show an Python example.

Python with Pytest-xdist

Achieving this when running python tests with pytest and the plugin pytest-xdist is simple. Pytest-xdist is a plugin which allows parallelization for tests amongst other things.

Install pytest-xdist with pip:

pip install pytest-xdist

Add an argument indicating number of cores to be used or auto to automatically detect the number of cores the machine has.

pytest -n [auto/number of cores]

The speed up won't be highly noticeable for small test suites, but for anything larger it is great. Remember to use compute-based VMs with several vCPUs to maximize the effect of parallelization.

Concurrent Jobs for each Stage

Spread your jobs for each stages across several VMs and run all jobs concurrently. Do not stack jobs sequentially, e.g run linting and testing seperately as they do not depend on each other. Gitlab supports this by default, all jobs in a stage run concurrently on different Gitlab-Runner VMs. As for Gitlab-CI you only need to set a limit of runners and a limit of concurrent runner higher than 1. With the Terraform module mentioned previously you can do this by:

module "gitlab_runner" {
  source = "npalm/gitlab-runner/aws"
  runners_concurrent = "5"
  runners_limit = "5"
}

You can read more in the Gitlab's docs to grasp how to fully customize it and for other CIs as e.g Circle-CI, they provide the same functionality.

Parallel Tasks

We use Task as a task runner for CI. Task is a task runner / build tool which I prefer due to it's syntax language. In Task you can run all dependencies for a task in parallel and this makes the tasks complete faster.

With Task you can specify dependencies with deps:

test:
  deps:
    - setup xyz
    - setup abc
    - setup efg
  desc: Run tests
  cmds:
    - pytest -n auto --cov

All dependencies run in parallel for better and faster performance.

Autoscaling your CI Runners

Create and manage your own runners, use AWS/GCP spot instances and autoscale your runners. Do not have jobs waiting around. This is easily achievable with the Terraform module terraform-aws-gitlab-runner. Just increase the limit and amount of concurrent runners, if you are worried about costs, lower the timeout so instances scale down when not used. Make sure to use spot instances as they are around 70% cheaper.

Summary

Several approaches to speed up your CI have been displayed, some trivial and some more advanced. Even though there was a focus on Gitlab-CI and Terraform, same functionality should be achievable without Terraform and a different CI platform. Long CIs can be blocking and are annoying for developers therefore we try to minimize CI completion time as much as possible.

Recipes when building a headless CMS with Wagtail's API

Adin Hodovic — Mon, 11 Nov 2019 15:52:21 +0000

Recently I built a headless CMS using Wagtail's API as a backend with NextJS/React/Redux as a frontend. Building the API I ran into some small issues with Image URL data, the API representation of snippets and creating a fully customized page representation. I'll show some simple recipes which will hopefully simplify it for anyone encountering these issues.

Images in the API

I create a ImageChooserBlock with a custom API representation. You can use the function get_rendition() to get all attributes as e.g the full URL to the image. It's possible to pass arguments suiting your needs as you can when using Wagtail images with Django templates, meaning you can pass max,min,original,fill etc.

from wagtail.images.blocks import ImageChooserBlock as DefaultImageChooserBlock

class ImageChooserBlock(DefaultImageChooserBlock):
    def get_api_representation(self, value, context=None):
        if value:
            return {
                "id": value.id,
                "title": value.title,
                "original": value.get_rendition("original").attrs_dict,
                "thumbnail": value.get_rendition("fill-120x120").attrs_dict,
            }

Then in your StreamBlock just reference the new ImageChooserBlock.

from .common_blocks import ImageChooserBlock

class MyBlock(StructBlock):
    image = ImageChooserBlock()

Custom representations of a page

I use a PageChooserPanel in my header snippet to make it easy for the user to add links to other pages in the header.

@register_snippet
class Header(models.Model):

    links = StreamField(
        [("link", PageChooserBlock(page_type="api.PortfolioPage"))],
        null=True,
        blank=True,
    )

Since the snippet extends the django.db model I won't be able to customize the get_api_representation() method so I use a custom SnippetChooserBlock with a custom api representation.

from wagtail.snippets.blocks import SnippetChooserBlock as DefaultSnippetChooserBlock

class HeaderChooserBlock(DefaultSnippetChooserBlock):
    def get_api_representation(self, value, context=None):
        if value:
            links = []
            for page in value.links:
                links.append({"url": page.value.url, "title": page.value.title})

            return {
                "logo": value.logo.get_rendition("original").attrs_dict,
                "links": links,
                "header_links_color": value.header_links_color,
            }

I loop through all the links in the StreamField and then I can access all the Page fields and add any fields to the API that I would like. Then you can reference your own HeaderChooserBlock when you want a page to have an specific header and you can customize all the fields you'd like it to return from the chosen page from the PageChooserBlock.

Snippets /w SnippetChooserPanel

If we continue from the above example of a HeaderChooserBlock, we will add a similar FooterChooserBlock. We use different blocks due to the varying API representation (reach out to me if you have a less repetitive solution).

from wagtail.snippets.blocks import SnippetChooserBlock as DefaultSnippetChooserBlock

class FooterChooserBlock(DefaultSnippetChooserBlock):
    def get_api_representation(self, value, context=None):
        if value:
            return {
                "copyright": value.copyright,
                "social_links": value.social_links.get_prep_value(),
            }

Remember you can use get_prep_value() to fetch all the fields for nested blocks within Streamfields. Remember that you will not receive any URLs but an ID of the image or page(when using e.g PageChooserBlock or ImageChooserBlock) which you can then use to make a second API call. I do not prefer to make several API calls therefore I customize the exact values returned to include page/image URLs and not IDs.

class LayoutBlock(StreamBlock):
    header = HeaderChooserBlock("api.Header")
    footer = FooterChooserBlock("api.Footer")

    class Meta:
        icon = "snippet"

I wrap then both layout fields (header, footer) in a custom LayoutBlock and then you can use it in any page you'd like.

snippets = StreamField(
    [
        (
            "layout",
            LayoutBlock(
                block_counts={"header": {"max_num": 1}, "footer": {"max_num": 1}}
            ),
        )
    ]
)

content_panels = Page.content_panels + [
    StreamFieldPanel("snippets"),
    ...
]

Summary

I played around first with Wagtail GraphQL as a headless CMS with Gatsby as a frontend, but I did not enjoy GraphQL therefore I opted for NextJS+Wagtail's API. I found it great, I can customize all API calls however I want and more so it has great defaults for almost all blocks/fields so you do not have to tinker with the data returned.

If you have any questions or better solutions, feel free to get in touch. I had some issues as I did not find a large community behind the Wagtail API, thus answers to beginner questions were hard to find, hope this helps.

Creating Group Webhooks with Templates for Gitlab CI

Adin Hodovic — Thu, 24 Oct 2019 20:33:03 +0000

Gitlab stores a vast majority of their functionality into their paid packages. Group webhooks is one of them and if your using a group runner, the group webhook becomes sought after. This is easily achievable without using a paid plan by creating reusable templates. Gitlab offers great functionality for configuring your CI with include and exclude functions. These can be reused to create a base template for all your CI jobs.

Setting up a default CI template

We'll create first a default template which we'll extend when creating a deploy stage.

gitlab-ci-defaults.yml

image: docker:stable

services:
  - docker:dind

# Default Stages
stages:
  - deploy

# Default runner tag
.default_vars:
  tags:
    - shared_aws_spot_runner

You can store your webhook in the default template but I'll create a separate yaml file.

First, create a slack bot and add an incoming webhook. We can use the webhook URL to send a curl request to the Slack API. We'll use Gitlab CI's after_script which runs both if the CI fails or succeeds, which will send the curl request.

gitlab-ci-webhooks.yml

.default_webhooks:
  after_script:
    - >
      curl -X POST -H 'Content-type: application/json' --data '{"text":"*'"$GITLAB_USER_NAME $(cat .job_status) $CI_COMMIT_SHORT_SHA to $CI_COMMIT_REF_NAME"'*\n```
{% endraw %}
'"Docker Image deployed: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA\nPrevious Docker Image: $CI_REGISTRY_IMAGE:${CI_COMMIT_BEFORE_SHA:0:8}\nCommit Title: $CI_COMMIT_TITLE\nCommit Description: $CI_COMMIT_DESCRIPTION\nMerge Request: ${CI_MERGE_REQUEST_PROJECT_URL:-No Merge Request}\nPipeline Details:$CI_PIPELINE_URL\nJob Detail:$CI_JOB_URL\n"'
{% raw %}
```\n>'"$CI_PROJECT_URL"'"}' https://hooks.slack.com/services/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

Include the webhooks in the default template.

gitlab-ci-defaults.yml

image: docker:stable

services:
  - docker:dind

# Add webhooks
include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-webhooks.yml'

# Default Stages
stages:
  - deploy

# Default runner tag
.default_vars:
  tags:
    - shared_aws_spot_runner

Now you can include the default template and extend the default variables. I have a separate project for gitlab-ci templates but Gitlab supports local templates as well. We use before_script to add by default a failed message which is used in the Slack message. In the script we echo deployed and replace the failed message as the last step, which indicates that CI passed since we reached the last step(Gitlab has currently no cleaner way of doing this, where CI status is accessible). By extending the default_webhooks variable we'll add all the default webhooks that exist as after_script(s).

.gitlab-ci.yml

include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-defaults.yml'

deploy:
  extends:
    - .default_vars
    - .default_webhooks
  before_script:
    - echo "failed to deploy" > .job_status
  script:
    - ansible-playbook ansible/deploy_xyz_app.yml -i -i environments/prod -e app_image=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - echo "deployed" > .job_status

Summary

The Slack template will output a message that fully utilizes Gitlab's predefined environment variables:

The solution is still not as clean as having Gitlab's UI where you can add a group webhook through the UI. But I find this method great for smaller teams that do not pay for Gitlab's enterprise features. Plus, a big upside is that the code is infrastructure/CI as code which I prefer way more than any UI. Using base templates can cover extremely much and you can generalize language specific deployment pipelines.

I've written a blog post that goes into details on creating templates for your Gitlab CI Jobs.

Creating templates for Gitlab CI Jobs

Adin Hodovic — Thu, 10 Oct 2019 14:15:34 +0000

Writing Gitlab CI templates becomes repetitive when you have similar applications running the same jobs. If a change to a job is needed it will be most likely needed to do the same change in every repository. On top of this there is a vast amount of excess YAML code which is a headache to maintain. Gitlab CI has many built-in templating features that helps bypass these issues and in addition helps `automating` the process of setting up CI for various applications.

Setting up a base CI template

As in most other coding languages you start projects with a base template. We'll create one which is very basic, and you can tailor it to your CI. The base template will be having the following variables:

image
.default_vars
.default_delay
stages
services

We use .default_vars to store any default variables that ALL jobs will share. You can store .default_delay within these variables but most likely you'd prefer that all jobs don't have a delay. I use a delay to not set up a new AWS auto scaling instance for sequential jobs (as one jobs finish and the next start, it usually creates two instances, I delay the second job so it can reuse the same instance). The .default_vars contain a default tag for CI jobs.

We set up a default image, default stages and the default docker-in-docker service as most jobs use those variables.

I include a webhook CI template for default webhooks which runs with the CI function after_script. You can find the template in my blog post which goes into detail about creating group webhooks for your CI. I reference all my includes by project as I find it the most clean solution but you have other options as local, template, remote.

gitlab-ci-base.yml

image: docker:stable

services:
  - docker:dind

include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-webhooks.yml'

stages:
  - build
  - test
  - deploy

# Default runner tag
.default_vars:
  tags:
    - shared_aws_spot_runner

# Used to not start a new instance
.default_delay:
  when: delayed
  start_in: 20s

Now you can include the default template and extend the default variables. Let's take a look at a default build stage. It'll extend the .default_vars but not the .default_delay as it is the first stage of the build. I use Google's Kaniko as it can use cache, build and push images with tags just by one command instead of docker build, docker push etc... We add all template commands in the before_script section as currently you cannot merge arrays and if you would like to append commands to the script you will overwrite the extended script completely. If your certain that you do not need the option to append commands you can add all commands to a script section.

.gitlab-ci-kaniko-build.yml

build:
  extends:
    - .default_vars
  stage: build
  tags:
    - shared_aws_spot_runner
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  before_script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --cache=true

We don't include the base templates for each stage e.g now in the build stage as you cannot have multiple of the same include variables, the YAML syntax will fail. The .gitlab-ci YAML file will include the base templates and the variables will be available for every stage. We have to add a empty script stage as the YAML syntax lint will fail otherwise and your jobs won't run(if anyone has a idea how to avoid this LMK).

include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-base.yml'
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-kaniko-build.yml'

build:
  script:
    - ""

Using include, extend and customizing the stage

Here is another example where I use the base template and extend it. My base deploy stage contains two stages deploy and deploy_manual. The deploy_manual stage uses YAML syntax to extend the deploy stage. As the deploy stage extends the .default_delay I have to overwrite it when deploying manually as a manual when can't have a start_in variable.

deploy: &deploy
  extends:
    - .default_vars
    - .default_delay
    - .default_webhooks
  stage: deploy
  only:
    refs:
      - master
  image: adinhodovic/ansible-k8s-terraform
  variables:
    ANSIBLE_CONFIG: ./ansible.cfg  # https://github.com/ansible/ansible/issues/42388
  before_script:
    - echo "failed to deploy" > .job_status
    ........ many commands
    - ansible-playbook requirements.yml # Ansible requirements

deploy_manual:
  <<: *deploy
  only:
  when: manual # make the stage manual
  start_in: # overwrite the start_in
  stage: test # part of the test section

Now you can include the deploy template in your .gitlab-ci YAML file. We'll add the script section that makes this application's deployment unique which is a ansible playbook for this repository. We'll create a template called .deployment_script_template which defines the YAML variable &deployment_script_definition. Here we will simply add the repository specific deployment stages and reuse them in the deploy and deploy_manual stages. Now your deploy stage will have all the base commands in the before_script, the repository relevant commands in the script and the webhooks in the after_script. You can now use this repetition of code for any project you create.

include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-base.yml'
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-kaniko-build.yml'
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-k8s-ansible-deploy.yml'

build:
  script:
    - ""

test:
  script:
    - ""

.deployment_script_template: &deployment_script_definition
  script:
    - ansible-playbook hodovi_cc.yml -i environments/base -i environments/prod -e app_image=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - echo "deployed" > .job_status

deploy:
  <<: *deployment_script_definition

deploy_manual:
  <<: *deployment_script_definition

Conclusion

As CI jobs get more complex and an organization has more micro-services to run CI for, there becomes a need to set up reusable templates for all CI jobs. This minimizes maintenance and makes it easier to set up new pipelines for new projects. Gitlab CI offers great functionality that makes this easy to set up and keep your CI modular just as do with any other code you write.

Kickstarting Infrastructure for Django Applications with Terraform

Adin Hodovic — Fri, 04 Oct 2019 14:55:39 +0000

When creating Django applications or using cookiecutters as Django Cookiecutter you will have by default a number of dependencies that will be needed to be created as a S3 bucket, a Postgres Database and a Mailgun domain. On top of this you'd most likely want to add basic monitoring, a DNS service, Google Search Console, Google Analytics etc.

We'll go through all of these services and setup all the resources declaratively(infrastructure as code) with Terraform. This will be then easily repeatable for any new Django application that needs to be deployed. Every service will be using a free plan, this whole setup costs 0$ to run monthly.

DNS with Cloudflare

Cloudflare provides CDN, DNS and various other services for free. Cloudflare is my to-go service when needing DNS services. Adding a Cloudflare record with Terrafrom by code is quite simple. Add the neccesary API keys for the provider as environment variables:

export CLOUDFLARE_EMAIL="example@gmail.com"
export CLOUDFLARE_TOKEN="my-api-token"

Or you can define them as variables in the provider, but I'd avoid it to not push sensitive secrets to git.

Now you can add the provider:

provider "cloudflare" {}

And lastly add the cloudflare record resource so we can access your application by the domain name:

resource "cloudflare_record" "hodovi_cc" {
  domain  = "hodovi.cc"
  name    = "hodovi.cc"
  value   = "your-ip"
  type    = "A"
  ttl     = 1
  proxied = true
}

Cloudflare will now serve as a DNS and a CDN, caching your static content. I'd recommend by default to add two default rules, Flexible SSL (if you have not set up full SSL with Letsencrypt) and automatic redirection from HTTP to HTTPS ensuring that any connection always uses HTTPS. Both of these can be easily set up with terraform.

Flexible SSL

The required terraform resource:

resource "cloudflare_page_rule" "hodovi_cc_flexible_ssl" {
  zone   = "hodovi_cc"
  target = "https://hodovi_cc/*"
  status = "active"

  actions {
    ssl = flexible
  }
}

Always use HTTPS

The required terraform resource:

resource "cloudflare_page_rule" "hodovi_cc_always_use_https" {
  zone   = hodovi_cc
  target = "http://hodovi_cc/*"
  status = "active"

  actions {
    always_use_https = true
  }
}

Cloudflare only allows 3 rules when using the free plan. You can by default set that your whole site has flexible SSL(only feasible through the UI not through terraform) and then you do not need the flexible SSL rule, which gives you space for 2 more rules.

Postgres DB

For the Postgres Database we'll use AWS database service RDS. AWS offers up to 750 hours of free t2.micro instances with 20 GB of storages per month. If you run one instance then 750 hours is enough for a whole month of uptime. A t2.micro instance should be fine for Django applications that are not data heavy. This setup includes daily backups as well. Note that this setup is a bit more complex requiring a VPC for the RDS instance. We use a Terraform module for the RDS instance. To grasp the parameters you'll have to look into the module as the topic is too big to cover in this post.

Note: This setup create the database in a public subnet and makes the database publicly accessible. I run my Django Applications within a GKE cluster therefore they do not share VPCs and the database needs to be publicly accessible. This setup has no failover as well, it is a single instance in a single az. Although if you'd like to add instances in multiple availability zones that should be achievable with this setup too.

module "rds_vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "rds"
  cidr = "10.0.0.0/16"

  azs             = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_dns_hostnames = true

  tags = {
    Terraform   = "true"
    Environment = "${var.environment}"
  }
}

resource "aws_security_group" "rds" {
  name        = "allow_rds"
  description = "Allow RDS inbound traffic"
  vpc_id      = "${module.rds_vpc.vpc_id}"

  ingress {
    from_port = 5432
    to_port   = 5432
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name      = "Allow access to the RDS instance"
    Terraform = true
  }
}

module "hodovi_rds" {
  source  = "terraform-aws-modules/rds/aws"
  version = "~> 2.0"

  identifier                = "hodovidb"
  parameter_group_name      = "default.postgres11"
  create_db_parameter_group = false
  create_db_option_group    = false

  subnet_ids = "${module.rds_vpc.public_subnets}"

  engine            = "postgres"
  engine_version    = "11.4"
  instance_class    = "db.t2.micro"
  allocated_storage = 20

  name     = "hodovidb"
  username = "postgres"
  password = "${var.hodovi_rds_password}" # Set in terraform secret vars
  port     = "5432"

  vpc_security_group_ids = ["${aws_security_group.rds.id}"]

  maintenance_window = "Mon:00:00-Mon:03:00"
  backup_window      = "03:00-06:00"

  # Publicly accessible
  publicly_accessible = true

  tags = {
    Owner       = "hodovi"
    Environment = "${var.environment}"
    Terraform   = true
  }

  # Database Deletion Protection
  deletion_protection = true
}

Django storages S3 bucket

A common way to handle static and media files is to store them in a S3 bucket. Cookiecutter Django offers both S3 buckets and Google Cloud solution. As I'm not that familiar with GCP's solution I'll go through setting up a S3 bucket.

Let's first just add the AWS terraform provider.

provider "aws" {
  region  = "your-aws-region"
}

Add your API credentials.

export AWS_ACCESS_KEY_ID="my-access-key"
export AWS_SECRET_ACCESS_KEY="my-secret-key"

IAM user

First we'll create a IAM user that will will be used to access the static files. We'll call it deployer as I use it's credentials to access the AWS API and store the static files into the S3 bucket.

resource "aws_iam_user" "deployer" {
  name = "Deployer"

  tags = {
    description = "Deploy and access AWS resources"
  }
}

Aws IAM policy Document

Setup an IAM policy document which gives full access to the user defined above. We'll also give read access to the static files to anyone.

data "aws_iam_policy_document" "hodovi_cc" {
  statement {
    sid = "PublicReadForGetBucketObjects"

    actions = [
      "s3:GetObject",
    ]

    resources = [
      "arn:aws:s3:::hodovi.cc/*",
    ]

    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
  }

  statement {
    actions = [
      "s3:*",
    ]

    resources = [
      "arn:aws:s3:::hodovi.cc",
      "arn:aws:s3:::hodovi.cc/*",
    ]

    principals {
      type        = "AWS"
      identifiers = [aws_iam_user.deployer.arn]
    }
  }
}

S3 bucket

Create the S3 bucket and refer to the policy created before. Attach any tags you'd like.

resource "aws_s3_bucket" "hodovi_cc" {
  bucket = "hodovi.cc"
  policy = "${data.aws_iam_policy_document.hodovi_cc.json}"

  tags = {
    Name        = "hodovi.cc"
    Environment = "${var.environment}"
  }
}

Cors

If you use e.g Wagtail and need cross origin access for your S3 bucket just add the cors_rule section.

resource "aws_s3_bucket" "hodovi_cc" {
  bucket = "hodovi.cc"
  policy = "${data.aws_iam_policy_document.hodovi_cc.json}"

  tags = {
    Name        = "hodovi.cc"
    Environment = "${var.environment}"
  }
  cors_rule {
    allowed_headers = ["Authorization"]
    allowed_methods = ["GET"]
    allowed_origins = ["https://hodovi.cc"]
    max_age_seconds = 3000
  }
}

Mailgun

Cookiecutter django uses Anymail with Mailgun's email service and it is an easy way to add "contact me" feature in to your application. It is free up to 10000 emails a month. I've created a terraform module that easily sets this up for you.

First we'll have to add a Mailgun provider. Download a Terraform provider mailgunv3 binary.

Then update your ~/.terraformrc to refer to the binary:

providers {
  mailgunv3 = "/home/example/downloads/terraform-provider-mailgunv3"
}

Add the provider to your provider.tf. You can add your API key here but I use env variables to avoid adding sensitive keys into Git. Export your api key:

export MAILGUN_API_KEY='my-api-key'

If your from europe you can use mailguns EU api, if not just don't add the field base_url.

provider "mailgunv3" {
  base_url = "https://api.eu.mailgun.net/v3"
}

Now you can use the module I've created.

module "hodovi_cc_mailgun_cloudflare" {
  source = "github.com/adinhodovic/terraform-mailgun-cloudflare"

  smtp_password   = var.hodovi_mailgun_password # Set in tf secret vars
  cloudflare_zone = "hodovi.cc"
  mailgun_domain  = "mg.hodovi.cc"
  spam_action     = "tag"
}

Monitoring Server Uptime with Statuscake

You'll most likely want to monitor application uptime and there are
several service providers for monitoring uptime. For this setup we'll use
Statuscake due to their free plan which
should work fine for your django application.

The free plan provides uptime checks every 5 minutes, for a more
frequent check rate you'll have to look into paid features.

As usual add the provider to your providers.tf

provider "statuscake" {
  username = "my-user-name"
}

Export your API key:

export STATUSCAKE_APIKEY="my-api-key"

Add your status cake test:

# Contact group 158719 = DevOps Team

resource "statuscake_test" "https_monitoring_hodovi_cc" {
  check_rate    = 300
  contact_group = ["my-contact-group-id"]
  test_type     = "HTTP"
  user_agent    = "Status Cake"
  trigger_rate  = "0"
  confirmations = "2"
  website_name  = "hodovi.cc"
  website_url   = "https://hodovi.cc"
}

You'll have to add a contact group manually
as terraform does not provide the resource. Then you'll have to replace the contact
group id above with the one you created.

Slack Alerting Integration

Create a Slack app, add an incoming webhook to the channel of your choice.
Now you can go to Statuscake and add an integration of the type Slack.
Give it a name and add the webhook url. Now you can tie your contact group
with the Slack integration. You'll now receive downtime alerts in Slack.

Google Search Console

Next we'll add your domain to google search console which will help you optimize your websites visibility and help monitor search result data for your website. Go to Google Search Engine and choose to add your domain. You'll get a site verification code that needs to be added as a TXT record.

I've created a terraform module which simply adds site verification as a TXT record to your DNS provider. The below example uses Cloudflare as the DNS service. The module supports AWS Route53 as well. If you don't use terraform you can set this manually up through your DNS Service.

The module uses the Cloudflare provider to add a TXT record

module "hodovi_cc_search_console" {
  source = "github.com/adinhodovic/terraform-txt-record"

  cloudflare_zones = [
    {
      "zone" : "hodovi.cc",
      "value" : "google-site-verification=my-site-verification"
    }
  ]
}

Google Analytics

We'll use django-analytical to add Google Analytics tracking. Install and add django-analytical your requirements and add "analytical" to your installed applications.

Note: this is just plain django code.

Add the variable

GOOGLE_ANALYTICS_PROPERTY_ID = env("GOOGLE_ANALYTICS_PROPERTY_ID")

to your config. Now when you deploy add the property ID as an environment variable. Lastly add the required tags to your base template:

{% load analytical %}
<!DOCTYPE html>
<html lang="en">

<head>
  {% analytical_head_top %}
  ... your content
  {% analytical_head_bottom %}
</head>

<body>
  {% analytical_body_top %}
  ... your content
  {% analytical_body_bottom %}
</body>

</html>

Secrets

The mailgun secret and RDS secret is not stored in code, you can store these in a secret.tfvars file.

hodovi_rds_password     = "my-password"
hodovi_mailgun_password = "my-password"

Now when using

terraform apply

you'll specify the secret vars file:

tfaa -var-file=secret.tfvars

tfaa is an alias for terraform apply auto-approve taken from terraform-alias.

Outputs

Lastly we'll create a terraform outputs file to gather all resource endpoints we've created.

output "hodovi_rds_endpoint" {
  value = module.hodovi_rds.this_db_instance_endpoint
}

output "hodovi_s3_website_endpoint" {
  value = aws_s3_bucket.hodovi_cc.id
}

Summary

By now you should have infrastructre as code for the following:

Cloudflare
- CDN
- DNS
- Flexible SSL
- Always HTTPS
AWS S3 Bucket for Static Storage
AWS RDS Database
Uptime Healthcheck with Statuscake
- Slack Alerts
- Email Alerts
Mailgun Domain
Google Analytics
Google Search Console

The monthly cost of all these services should be 0$ unless you exceed the threshholds for each service. This is a basic setup for all services, obviously if your application has high demands you'll have to scale the services.

We could setup an EC2 instance which is free as well (lowest tier up to 750hours a month) but I run a kubernetes cluster where all of my applications are deployed. And I'd assume most have this part figured out already with various cloud providers and various setups.