DEV Community: Taufek Johar

Recursive Query with PostgreSQL

Taufek Johar — Sun, 26 Jul 2020 13:30:00 +0000

If in your database design, you have a resource with self reference
association, you will end up having a tree like structure when you joined them
up. In school or workplace, you might encountered tree-like chart which
is known as Organization Chart or Org Chart for short.

If I were to design the database table for this resource most probably it will
look like below:

Table Name: Employees

column	data type	nullable
id	bigint/primary key	false
full_name	varchar/string	false
role	varchar/string	false
manager_id	bigint/foreign_key	true

In Rails, my database migration will be as following:

class CreateEmployee < ActiveRecord::Migration[6.0]
  def change
    create_table :employees do |t|
      t.string :full_name
      t.string :role
      t.references :manager, foreign_key: { to_table: :employees }
    end
  end
end

And my Employee model looks like below:

class Employee < ApplicationRecord
  belongs_to :manager, optional: true, class_name: 'Employee'
  has_many :subordinates, foreign_key: 'manager_id', class_name: 'Employee'
end

Let's say I want to build a page that will be able to render a tree structure with a given person id. I have a Tree UI component which requires following data structure:

id	full_name	role	manager_id
1	Jack Dorsey	CEO	null
2	Michael Montano	Engineering Lead	1
3	Kayvon Beykpour	Product Lead	1
4	Joy Su	VP Engineering	2
5	Nick Turnow	Platform Lead	2
6	Keith Coleman	VP Product	3
7	Lakshmi Shankar	Sr Director, Strategy & Operations	3

Source: https://theorg.com/org/twitter

Disclaimer: Above is just a partial of Twitter Org Chart which I pulled from above source. It's not complete and most probably not up-to-date.

To build above data, I will use below builder class

class OrganizationTreeBuilder
  def initialize(employee)
    @employee = employee
  end

  def build
    [@employee.attributes] + fetch_subordinates(@employee)
  end

  private

  def fetch_subordinates(employee)
    employee.subordinates.map do |subordinate|
      [subordinate.attributes] + fetch_subordinates(subordinate)
    end.flatten
  end
end

Above builder class accepts an Employee instance via the constructor and when build method is called, it will recursively fetch the subordinates of the
employee until it reaches to the level where an employee does not have any subordinates. Each time it calls employee.subordinates, it will execute a sql
query to fetch list of subordinates. Since we execute the query recursively in app level, I'm calling this as "app recursive query".

Obviously this works but not really efficient if your have very large organization. Before we start to improve this, we will write unit tests for this builder class. Below is the spec file for above builder:

RSpec.describe OrganizationTreeBuilder do
  subject { OrganizationTreeBuilder.new(ceo).build }

  context 'simple organization tree' do
    let!(:ceo) { create(:employee) }
    let!(:dev_manager1) { create(:employee, manager: cto) }
    let!(:dev_manager2) { create(:employee, manager: cto) }
    let!(:developer1_1) { create(:employee, manager: dev_manager1) }
    let!(:developer1_2) { create(:employee, manager: dev_manager1) }
    let!(:developer2_1) { create(:employee, manager: dev_manager2) }
    let!(:developer2_2) { create(:employee, manager: dev_manager2) }
    let!(:not_relevant) { create(:employee) }

    it 'builds organization tree', :aggregate_failure do
      expect(subject).to match_array(
        [
          ceo.attributes,
          dev_manager1.attributes,
          dev_manager2.attributes,
          developer1_1.attributes,
          developer1_2.attributes,
          developer2_1.attributes,
          developer2_2.attributes
        ]
      )
      expect(subject.count).to eq 7
    end
  end
end

Above unit test will pass when run.

To gauge the performance I'm adding below test cases to above spec.

  ...
  context 'horizontal organization tree' do
    let!(:ceo) { create(:employee) }

    before { create_employee(ceo, 10, 2) }

    it 'builds organization tree' do
      time = Benchmark.measure { expect(subject.count).to eq 111 }
      puts time.real
    end
  end

  context 'vertical organization tree' do
    let!(:ceo) { create(:employee) }

    before { create_employee(ceo, 2, 10) }

    it 'builds organization tree' do
      time = Benchmark.measure { expect(subject.count).to eq 2047 }
      puts time.real
    end
  end

  context 'large organization tree' do
    let!(:ceo) { create(:employee) }

    before { create_employee(ceo, 5, 5) }

    it 'builds organization tree' do
      time = Benchmark.measure { expect(subject.count).to eq 3906 }
      puts time.real
    end
  end

  private

  def create_employee(manager, num, level)
    return if (level -= 1) < 0

    create_list(:employee, num, manager: manager).each do |employee|
      create_employee(employee, num, level)
    end
  end
  ...

There are 3 different performance-related test cases:

Horizontal Organization Tree. Organization tree is broad and shallow. Each employee has 10 subordinates with 2 levels.
Vertical Organization Tree. Organization tree is narrow and deep. Each employee has 2 subordinates with 10 levels.
Large Organization Tree. Organization tree is broad and deep. Each employee has 5 subordinates with 5 levels.

Below is how long it took to build the data for each scenario:

Horizontal Organization Tree. 0.3sec
Vertical Organization Tree. 6.2sec
Large Organization Tree. 11.3sec

Time for Optimization

PostgreSQL has build-in recursive query which is made for this. Below is updated OrganizationTreeBuilder class
with PostgreSQL recursive query.

class OrganizationTreeBuilder
  def initialize(employee)
    @employee = employee
  end

  def build
    ActiveRecord::Base.connection.execute(<<~SQL)
      WITH RECURSIVE subordinates AS (
        SELECT
         employees.id,
         employees.full_name,
         employees.role,
         employees.manager_id
        FROM
         employees
        WHERE
         employees.id = #{@employee.id}

        UNION

        SELECT
         employees.id,
         employees.full_name,
         employees.role,
         employees.manager_id
        FROM
         employees
         JOIN subordinates ON subordinates.manager_id = employees.managers.id
      ) SELECT * FROM subordinates ORDER BY id ASC;
    SQL
  end
end

The same unit tests above will still pass since the returned data is still the same.

Below are the comparison stats between previous (app recursive query) and current (PostgreSQL recursive query).

	App Recursive Query	PostgreSQL Recursive Query
Horizontal Organization Tree	0.3sec	0.000005sec
Vertical Organization Tree	6.2sec	0.000005sec
Large Organization Tree	11.3sec	0.000005sec

Obviously, PostgreSQL query is the winner here. Although my test dataset is small, but it does prove the point that PostgreSQL recursive query is more performant than app recursive query.

But what does the query actually do? It does not make sense to me when I read it for the first time. Turns out there are 2 parts within the CTE (Common Table Expression) to make this recursive query to work. There is a non-recursive query being UNION with recursive query.

Below is the non-recursive query which will be use as the base:

  SELECT
   employees.id,
   employees.full_name,
   employees.role,
   employees.manager_id
  FROM
   employees managers
  WHERE
   employees.id = #{@employee.id}

From this query you will get following output:

id	full_name	role	manager_id
1	Jack Dorsey	CEO	null

Then comes the recursive query below:

  SELECT
   employees.id,
   employees.full_name,
   employees.role,
   employees.manager_id
  FROM
   employees
   JOIN subordinates ON subordinates.manager_id = employees.managers.id

The recursive query joins with the CTE named subordinates. It will start off with base data from the non-recursive query and it will get following output:

id	full_name	role	manager_id
2	Michael Montano	Engineering Lead	1
3	Kayvon Beykpour	Product Lead	1

Above are all the subordinates for employee id 1. Then it will do the same query for each of these subordinates and it will keep going until the recursive query didn't get any output. At the end it will UNION all the records you will see the end results as below:

id	full_name	role	manager_id
1	Jack Dorsey	CEO	null
2	Michael Montano	Engineering Lead	1
3	Kayvon Beykpour	Product Lead	1
4	Joy Su	VP Engineering	2
5	Nick Turnow	Platform Lead	2
6	Keith Coleman	VP Product	3
7	Lakshmi Shankar	Sr Director, Strategy & Operations	3

This is a simplified usage of recursive query in PostgreSQL. There are few more things you could do which you could find out from the official documentation.

Bonus

I'm using output from above builder class to build below tree structure using d3.js library.

Source: https://jsfiddle.net/taufek/1anLkfyg/32

Reference: https://bl.ocks.org/d3noob/8329404

ElasticSearch Geo Shape

Taufek Johar — Tue, 16 Jun 2020 16:58:00 +0000

In previous post, we looked at quick start guide on how to use ElasticSearch with Ruby.

In this post, we will look at how to search based on geo location. Lets say we have food delivery app and we would like to find out the delivery surcharge amount for certain area.

Create Index with Geo Shape Mapping

client = Elasticsearch::Client.new

mapping = {
  surcharge: {
    properties: {
      delivery_area: {
        type: 'geo_shape',
      }
    }
  }
}

client.create(index: :surcharges, type: :surcharge, body: {})

client.indices.put_mapping(index: :surcharges, type: :surcharge, body: mapping)

client.indices.get(index: :surcharges)

# Output
{
  "surcharges"=> {
    "aliases"=>{},
    "mappings"=>{
      "surcharge"=>{
        "properties"=>{
          "delivery_area"=>{"type"=>"geo_shape"}
        }
      }
    },
    "settings"=> {...}
  }
}

Above code, will create an index named surcharges and a document named surcharge. It has property of delivery_area with geo_shape data type.

Index Surcharge Document with Polygon Coordinates

Below shows polygon area that I’ve created with Google Maps.

Below is the surcharge document which saying within that polygon area, the surcharge amount is $10.

surcharge = {
  id: 1,
  delivery_area: {
    type: 'polygon',
    coordinates: [[
      # longitude, latitude
      [101.710997, 3.157035],
      [101.710997, 3.152954],
      [101.717316, 3.152954],
      [101.717316, 3.157035],
      [101.710997, 3.157035],
    ]]
  },
  amount: 10.0
}

client.index(
  id: surcharge[:id],
  index: :surcharges,
  type: :surcharge,
  body: surcharge
)

# Output
{
  "_index"=>"surcharges",
  "_type"=>"surcharge",
  "_id"=>"1",
  "_version"=>1,
  "result"=>"created",
  "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0},
  "_seq_no"=>1,
  "_primary_term"=>1
}

Search with Geo Point

Once I have my surcharge data indexed, I can start querying based on a coordinate. If my drop off point in somewhere inside the polygon, I’m expecting the query will return the surcharge document.

query = {
  query: {
    bool: {
      filter: {
        geo_shape: {
          'delivery_area': {
            shape: {
              type: :point,
                coordinates: [
                  101.714997, #longitude
                  3.155647 #latitude
                ]
            }
          }
        }
      }
    }
  }
}

client.search(index: 'surcharges', body: query)

# Output
{
  "took"=>9,
  "timed_out"=>false,
  "_shards"=>{
    "total"=>5,
    "successful"=>5,
    "skipped"=>0,
    "failed"=>0
  },
 "hits"=>{
   "total"=>1,
   "max_score"=>1.0,
   "hits"=>[
     {
       "_index"=>"surcharges",
       "_type"=>"surcharge",
       "_id"=>"1",
       "_score"=>1.0,
       "_source"=>{
          "id"=>1,
          "delivery_area"=>{
            "type"=>"polygon",
             "coordinates"=>[[[101.710997, 3.157035], [101.710997, 3.152954], [101.717316, 3.152954], [101.717316, 3.157035], [101.710997, 3.157035]]]
          },
         "amount"=>10.0
        }
      }
    ]
  }
}

Then if my drop off point in somewhere outside the polygon, I’m expecting the query will not return any surcharge document.

query = {
  query: {
    bool: {
      filter: {
        geo_shape: {
          'delivery_area': {
            shape: {
              type: :point,
              coordinates: [
                101.7077343, #longitude
                3.157032 #latitude
              ]
            }
          }
        }
      }
    }
  }
}

client.search(index: 'surcharges', body: query)

# Output
{
  "took"=>0,
  "timed_out"=>false,
  "_shards"=>{
    "total"=>5,
    "successful"=>5,
    "skipped"=>0,
    "failed"=>0
  },
  "hits"=>{
    "total"=>0,
    "max_score"=>nil,
    "hits"=>[]
  }
}

Conclusions

Geo shape is useful when you have data associated with coordinates of land area. Once you indexed those as a polygon in ES index, you could use Geo Shape query to fetch documents that intersect with a particular geo point. There are many other useful features too such as you could set a polygon with a hole in the middle or you could search based on radius instead of point. For more info you can head to ES official documents (geo-shape-datatype and geo-shape-query).

Learn ElasticSearch with Ruby

Taufek Johar — Mon, 15 Jun 2020 16:00:00 +0000

ElasticSearch (ES) is a popular search and analytics engine. I’ve been using it in most of my Ruby on Rails projects. Usually in a project we will started of by building an ActiveRecord model and establish the ES index mapping before we could start running the search.

In this post, I’ll start with basic and minimal code as possible to get us started with ES.

Pre-requisite

Install elasticsearch gem.

Initialize ES Client

client = Elasticsearch::Client.new

We’ll be using above client instance throughout our example.

Create ES Index Mapping

setting = {
  properties: {
    name: {
      type: :text
    },
    birth_date: {
      type: :date
    }
  }
}

client.create(index: :foo, type: :doc, body: {})

client.indices.put_mapping(index: :foo, type: :doc, body: mapping)
client.create(index: :foo, type: :doc, body: setting)

This creates new index named foo with following properties:

name field as text data type.
birth_date field as date data type.

Index Document(s)

adam = {
  name: 'Adam Hakeem',
  birth_date: '2006-12-08'
}

alif = {
  name: 'Alif Hussain',
  birth_date: '2009-04-28'
}

ammar = {
  name: 'Ammar Hamzah',
  birth_date: '2016-07-03'
}

client.index(id: 1, index: :foo, type: :doc, body: adam)
client.index(id: 2, index: :foo, type: :doc, body: alif)
client.index(id: 3, index: :foo, type: :doc, body: ammar)

Search away

Now you could start searching your data in ES index.

Below search by name with partial keyword am*.

body = {
  query: {
    query_string: {
      query: 'am*'
    }
  }
}

client.search(index: :foo, body: body)

#Output:
{
  "took"=>13,
  "timed_out"=>false,
  "_shards"=>{"total"=>5, "successful"=>5, "skipped"=>0, "failed"=>0},
  "hits"=> {
    "total"=>1,
    "max_score"=>0.2876821,
    "hits"=>[
      {"_index"=>"foo", "_type"=>"doc", "_id"=>"3", "_score"=>0.2876821, "_source"=>{"name"=>"Ammar Hamzah", "birth_date"=>"2016-07-03"}}
    ]
  }
}

Below search by date range.

body = {
  query: {
    range: {
      birth_date: {
        gte: '2009-01-01'
      }
    }
  }
}

client.search(index: :foo, body: body)

#Output:
{
  "took"=>2,
  "timed_out"=>false,
  "_shards"=>{"total"=>5, "successful"=>5, "skipped"=>0, "failed"=>0},
  "hits"=> {
    "total"=>2,
    "max_score"=>1.0,
    "hits"=> [
      {"_index"=>"foo", "_type"=>"doc", "_id"=>"2", "_score"=>1.0, "_source"=>{"name"=>"Alif Hussain", "birth_date"=>"2009-04-28"}},
      {"_index"=>"foo", "_type"=>"doc", "_id"=>"3", "_score"=>1.0, "_source"=>{"name"=>"Ammar Hamzah", "birth_date"=>"2016-07-03"}}
    ]
  }
}

Cleanup After Yourself

If you are done playing around with your index, you might want to remove your dummy data. You can run below to delete our particular dummy index.

client.indices.delete(index: :foo)

Conclusions

Hopefully this will give you a good start on how to learn ElasticSearch with Ruby.

Host Jekyll Site in AWS S3

Taufek Johar — Mon, 15 Jun 2020 05:40:00 +0000

In previous post, I’ve wrote about on how to push a Jekyll site to Heroku. In this post it will be about how I host this blog in AWS S3 bucket.

Turns out it is much simpler to do it in S3.

Create a S3 bucket

Create a S3 bucket and enable Web Hosting in the properties.

Configure Github Action

You will need to configure following secrets in your Github setting.

AWS_S3_BUCKET with your bucket name.
AWS_ACCESS_KEY_ID with your AWS access key id.
AWS_SECRET_ACCESS_KEY with your AWS secret access id.

name: Sync to S3

on:
push:
branches:
    - master

        jobs:
        deploy:
        runs-on: ubuntu-latest
        steps:
    - uses: actions/checkout@v1

    - name: Retrieve Gem Bundles Cache
        uses: actions/cache@v2
        with:
        path: _bundle
        key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
        restore-keys: |
        ${{ runner.os }}-gems-

    - name: Generate site
        run: |
        mkdir -p _bundle/cache
        chmod a+rwx -R .
        docker-compose run -e JEKYLL_ENV=production jekyll bundle install
        docker-compose run -e JEKYLL_ENV=production jekyll bundle exec jekyll build --trace

    - uses: jakejarvis/s3-sync-action@master
        with:
        args: --acl public-read --follow-symlinks --delete
        env:
        AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_REGION: 'ap-southeast-1'
        SOURCE_DIR: '_site'

With a push to master branch, it will trigger a build which will

Checkout code from master branch.
Generate Jekyll static files.
Push generated files inside _site folder to your S3 bucket.

Conclusion

This seems like a cheaper option for hosting a static site and have SSL configured in CloudFront compared to paying for a dyno in Heroku to host a site with SSL. For Heroku, I will need to pay atleast $7/month for a dyno and for CloudFront it will based on usage. Since my blog does not have much visitors, my AWS bill is very small which is around $1/month. This includes S3, Route 53 and CloudFront services.

Deploy Jekyll Site to Heroku

Taufek Johar — Sun, 14 Jun 2020 07:30:00 +0000

Generally there are 2 ways to deploy Jekyll static site to Heroku.

Deploy as rails app using web server gem such as puma, passenger, thin, ‘etc’.
Deploy as Docker image running nginx service.

In this post, I chose 2nd approach since I like to use Docker image.

Prepare Dockerfile

We will serve this Jekyll site via nginx service. In order to map Heroku portnumber to our nginx service, we will need to use below default.conf template.

server {
  # Placeholder for Heroku port
  listen $PORT;

  location / {
    root /usr/share/nginx/html;
    index index.html;
  }
}

Dockerfile.

FROM nginx:alpine

# copy nginx `default.conf`
COPY nginx/default.conf /etc/nginx/conf.d/
# copy Jekyll generated files
COPY _site /usr/share/nginx/html/

# replace $PORT placeholder with HEROKU given port in default.conf and run nginx service
CMD sed -i -e 's/$PORT/'"$PORT"'/g' /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'

You can named this file as Dockerfile or have a suffix to indiciate theHeroku process. For instance, if you want to run this application as webprocess in Heroku, you could name it as Dockerfile.web. This works well ifyou have multiple Dockerfiles with different process such as web, worker, orclock. Later on, when you run heroku container:push --recursive, it willbuild all the Dockerfiles with the correct suffix and push them to Heroku dockerregistry.

Configure Github Action

Even without Github Action, all I need is Heroku CLI to deploy this Dockerimage using below commands.

> bundle exec jekyll build

> heroku container:login

> heroku container:push -a <HEROKU_APP_NAME> --resursive

> heroku container:release -a <HEROKU_APP_NAME> web

Basically, in Github Actions, we are running above commands. Below is the fullGithub Action workflow yaml.

name: Push Container to Heroku

on:
  push:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1

    - name: Retrieve Gem Bundles Cache
      uses: actions/cache@v2
      with:
        path: _bundle
        key: $-gems-$
        restore-keys: |
          $-gems-

    - name: Generate site
      run: |
        mkdir -p _bundle/cache
        chmod a+rwx -R .
        docker-compose run -e JEKYLL_ENV=production jekyll bundle install
        docker-compose run -e JEKYLL_ENV=production jekyll bundle exec jekyll build --trace

    - name: Login to Heroku Container registry
      env:
        HEROKU_API_KEY: $
      run: heroku container:login

    - name: Build and push
      env:
        HEROKU_API_KEY: $
      run: heroku container:push -a $ --recursive

    - name: Release
      env:
        HEROKU_API_KEY: $
      run: heroku container:release -a $ web

With a push to master branch, it will trigger a build which will

Checkout code from master branch.
Generate Jekyll static files.
Build and push the Docker image to heroku registry.
Deploy the Docker image to Heroku with web process.

Conclusion

This approach not only work for static site, but also for other typesof application such as Rails application. Having a Dockerfile will give youfull control on what other linux packages you wish to install within theapplication server instance such as ImageMagick. It will also give youconsistent environment between running your application on your local machineand on Heroku.

Let Cert-Manager Manage LetsEncrypt Certificates

Taufek Johar — Mon, 08 Jun 2020 16:20:00 +0000

Since this blog is using *.dev domain, I will need to configure SSL certificate in order to serve this content in HTTPS. I’m using LetsEncrypt to issue my certificate and the certificate only valid for 3 months. I will need to renew them periodically. I can either do that manually when the certificate is almost expire or I can automate the process.

In this post, we will look into how to automate the certificate renewal with cert-manager.

I’ve been using aws-alb-ingress-controller as Ingress implementation in my cluster and but for this, I will need to swap to another Ingress implementation which is nginx-ingress. This is because currently, there is no way for the cert-manager to upload the certificate to the ALB.

Pre-requisite

Add below IAM policy to your “node” role.

"Version": "2012-10-17",
"Statement": [
  {
    "Effect": "Allow",
    "Action": "route53:GetChange",
    "Resource": "arn:aws:route53:::change/*"
  },
  {
    "Effect": "Allow",
    "Action": [
      "route53:ChangeResourceRecordSets",
      "route53:ListResourceRecordSets"
    ],
    "Resource": "arn:aws:route53:::hostedzone/*"
  },
  {
    "Effect": "Allow",
    "Action": "route53:ListHostedZonesByName",
    "Resource": "*"
  }
]

Install Cert Manager

Run below to install cert-manager and its dependencies.

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.yaml

Apply below Issuer:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: default
spec:
  acme:
  server: https://acme-v02.api.letsencrypt.org/directory
  email: taufek@gmail.com

  privateKeySecretRef:
    name: letsencrypt-prod

  solvers:
      - selector:
      dnsNames:
          - blog.taufek.dev
          http01:
            ingress:
              class: nginx

In this Issuer, we configure to use:

LetsEncrypt production api.
http01 challenge to verify that we own the domain. The other method is dns01.
nginx ingress implementation.

And lastly, we apply below Ingress rule:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    cert-manager.io/issuer: 'letsencrypt-prod'
  spec:
    tls:
      - hosts:
        - blog.taufek.dev
        secretName: blog-taufek-dev-tls
    rules:
      - host: blog.taufek.dev
          http:
          paths:
           - path: /
               backend:
                 serviceName: blog-nodeport-service
                 servicePort: 80

Wait for a minute for cert-manager to do its thing. It will:

Send a CertificationRequest to LetsEncrypt API.
Setup and API to fulfill LetsEncrypt challenge.
Generate the certificate files.
Apply the new/renewed certificate to nginx-ingress.

You can run below to check the certificate status

> kubectl describe certificate blog-taufek-dev

Name: blog-taufek-dev-tls
...
API Version: cert-manager.io/v1alpha3
Kind: Certificate
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Requested 5m20s cert-manager Created new CertificateRequest resource "blog-taufek-dev-tls-918422041"
Normal Issued 4m44s cert-manager Certificate issued successfully

And that’s it. I have successfully setup LetsEncrypt certificate for my blog it it will be auto renew by cert-manager when the time comes.

Bonus

cert-manager also comes with kubectl-plugin. When installed, there are few additional commands I can use via kubectl related to certificate management.

For example, I can run below to renew all my certificates

kubectl cert-manager renew --all

Conclusions

Certificate renewal is a must for production grade application. You wouldn’t want to miss renewing your applications certificate when it almost expires. The only thing that made me not to use this implementation is because it requires Classic ELB and I’m still evaluating if the cost is still within my budget. If I were to setup for a profitable application, this is definitely my choice of implementation.

Fortify Kubernetes Cluster with Bastion

Taufek Johar — Fri, 05 Jun 2020 19:00:00 +0000

In previous post we looked into Ingress as the front facing entry point for your applications users. In this post, we will look into Bastion as the backdoor entry point for infrastructure administration users.

Currently, all the EC2 instances in my cluster are on public subnet which all of them has public IP address. In order to secure the cluster further, we can move our cluster to a private subnet and setup a Bastion instance as a ‘jump’ server. Bastion will be the only instance on your public subnet and it has access to the cluster. We can easily achieve this setup with kops.

Create New Cluster with Bastion

For creating new cluster, you can run below command and your will get a cluster on private network with a Bastion instance ready:

kops create cluster \
  --name=k8s.blog.taufek.dev \
  --state=s3://k8s.blog.taufek.dev.state \
  --zones=ap-southeast-1a,ap-southeast-1b,ap-southeast-1c \
  --dns-zone=k8s.blog.taufek.dev \
  --topology=private \
  --bastion
  --yes

Then run below commands:

kops update cluster \
  --name kubernetes.taufek.dev \
  --yes \
  --state=s3://k8s.blog.taufek.dev.state

In a minute or so you will have your cluster ready. You could run below command to know the status:

kops validate cluster --state=s3://k8s.blog.taufek.dev.state

Add Bastion to Existing Cluster

For existing cluster, you will need to to edit your cluster object. Run below command to open up your cluster yaml file

kops edit cluster --state=s3://k8s.blog.taufek.dev.state

Edit your cluster yml with following settings. You could get the subnets by running kops create command with dry-run flag and output to a file.

  ...
  subnets:
  - cidr: ...
    name: ap-southeast-1a
    type: Private # change this to Private
    zone: ap-southeast-1a
  - cidr: ...
    name: ap-southeast-1b
    type: Private # change this to Private
    zone: ap-southeast-1b
  - cidr: ...
    name: ap-southeast-1c
    type: Private # change this to Private
    zone: ap-southeast-1c
  - cidr: ... # add these new Utility subnets
    name: utility-ap-southeast-1a
    type: Utility
    zone: ap-southeast-1a
  - cidr: ...
    name: utility-ap-southeast-1b
    type: Utility
    zone: ap-southeast-1b
  - cidr: ...
    name: utility-ap-southeast-1c
    type: Utility
    zone: ap-southeast-1c
  topology:
    bastion: # add bastion entry here
      bastionPublicName: bastion.blog.k8s.taufek.dev
    dns:
      type: Public
    masters: private # change to private
    nodes: private # change to private

Then run below to create Bastion instance.

kops create instancegroup bastions \
  --role Bastion \
  --subnet ap-southeast-1a,ap-southeast-1b,ap-southeast-1c \
  --state=s3://k8s.blog.taufek.dev.state

Run below command to push update to your cluster

kops update cluster --yes --state=s3://k8s.blog.taufek.dev.state
kops rolling-update cluster --state=s3://k8s.blog.taufek.dev.state --yes

That’s all too it, and just wait for few minute for your cluster to be reconfigured with private subnet. And you should also be able to ssh into your Bastion instance with following command:

ssh -A admin@bastion.blog.taufek.dev

Below is how the cluster looks like before, without Bastion:

And below is how it looks like now, with Bastion:

One thing that made me hesitated to stick with this setup, is the additional ELBs it created for Bastion and Master instances. This means it will be a significant cost increase in my AWS bill. So I decided to undo this change.

In order to remove Bastion you will need to edit back your cluster yaml file and remove Bastion instance.

Edit your cluster yml with following settings

  ...
  subnets:
  - cidr: ...
    name: ap-southeast-1a
    type: Public # change this back to Public
    zone: ap-southeast-1a
  - cidr: ...
    name: ap-southeast-1b
    type: Public # change this back to Public
    zone: ap-southeast-1b
  - cidr: ...
    name: ap-southeast-1c
    type: Public # change this back to Public
    zone: ap-southeast-1c
  ... # remove the Utility subnets
  topology:
  ... # remove bastion
    dns:
      type: Public
    masters: public # change this back to public
    nodes: public # change this back to public

Run below command to push update to your cluster

kops update cluster --yes --state=s3://k8s.blog.taufek.dev.state
kops rolling-update cluster --state=s3://k8s.blog.taufek.dev.state --yes

Run below command to remove Bastion instance

kops delete instancegroup bastions --yes --state=s3://k8s.blog.taufek.dev.state

Conclusions

Bastion is not a concept solely for Kubernetes but it’s a common practice in fortifying your software infrastructure even without Kubernetes. I found that by looking at all the AWS services created by kops, you could learn so much about best practices in software infrastructure. I don’t have much experience in setting up production environment but I learn a lot in past couple of weeks by looking at how kops wires everything together.

You can’t appreciate enough the complexity of services kops utility created under the hood with just one command, kops create .... I can’t imagine the hours you have to put in to setup everything manually versus doing it with kops. And it is just not about creating but also managing the infrastructure with this utility. Not only you save your time but rest assured that your infrastructure is configured in a correct manner.

Manage Applications with Ingress

Taufek Johar — Wed, 03 Jun 2020 15:00:00 +0000

Ingress is the gate keeper to your cluster. All external access will go through your Ingress Controller and depending on the rules that you set, it will be redirected to a particular service.

There are multiple benefits of having Ingress in my cluster.

Replaced Classic ELB with Application ELB.
Rules and External DNS.
Redirect to HTTPS.

There are few implementations of Kubernetes Ingress we can choose from and the one I’m using is aws-alb-ingress-controller.

Replaced Classic ELB with Application ELB

Below is my previous setup before having Ingress.

When you visited this blog, the request goes through Kubernetes ‘LoadBalancer’ Service and it will be redirected to one of the Pod which will serve this lovely content 😛. For the purpose of this simple site, this is more than sufficient.

But I’m not going to stop here since the purpose of the whole thing is to learn and we will move forward with Ingress.

Below is my current setup with Ingress.

Now the previous Classic ELB has been replaced by Application ELB. ELB acts as an Ingress Proxy. It will be the single point of entry for all external access to my cluster. The Kubernetes ‘LoadBalancer’ Service is now replaced by Kubernetes ‘NodePort’ Services which are only accessible internally.

One benefit of having Application ELB over Classic ELB, the pricing is much cheaper 💰. Yeay!. And the goodness does not stop here.

Rules and External DNS

Ingress Rules contains the instructions that will help Ingress Controller to redirect the request to the correct services.

Below is an excerpt of my Ingress Rules. It says all request that matches the host and path,should go to a particular Kubernetes ‘NodePort’ Service.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  ...
spec:
  rules:
    - host: blog.taufek.dev
      http:
        paths:
         - path: /*
           backend:
             serviceName: blog-nodeport-service
             servicePort: 80

Another cool thing about this is the External DNS which will automatically sync with Hosted Zone in AWS Route 56. If I need to add a new rule with new ‘host’, like below, this will trigger a task to update my Hosted Zone with new Record Set. Or if I remove a particular ‘host’ from the rules, it will also remove the Record Set.

spec:
  rules:
    - host: blog.taufek.dev
      http:
        paths:
         - path: /*
           backend:
             serviceName: blog-nodeport-service
             servicePort: 80
    - host: api.taufek.dev
      http:
        paths:
         - path: /*
           backend:
             serviceName: api-nodeport-service
             servicePort: 8080

Redirect to HTTPS

Not all browsers like Chrome will auto redirect my *.dev domain to https. I would like to have consistent experience for all browsers so I will force all the request to HTTPS. This is easily achieved with Ingress Rules. Below is the configuration to implement HTTPS redirect.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/certificate-arn: <ADD_YOUR_CERT_ARN_HERE>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
spec:
  rules:
    - host: blog.taufek.dev
      http:
        paths:
         - path: /*
           backend:
             serviceName: ssl-redirect
             servicePort: use-annotation
         - path: /*
           backend:
             serviceName: blog-nodeport-service
             servicePort: 80

Conclusions

With this change, not only it helps me to cut down my AWS bill, it also made the cluster to be easily managed. If I need to serve new application, I no longer need to spawn new LoadBalancer Service, configure the SSL cert and etc. I only need to do it once for the Application ELB that serves as Ingress Proxy. All new applications will be managed via Ingress Rules. Although it looks more complex on the diagram but it helps if you plan to have multiple applications in your cluster.

Reduce Kubernetes Cluster Costs on AWS

Taufek Johar — Mon, 01 Jun 2020 10:31:00 +0000

When you spun up a Kubernetes cluster for your own personal blog, you are on a tight budget. Kubernetes cluster can blow a hole in your pocket if you don’t pay attention to the AWS services it uses under the hood.

For this blog, the Kubernetes cluster was provisioned using kops. There are settings you could change only during creation and there are also things you could change post creation. For the former point, you better plan it early before you create the cluster.

Mainly there are 2 things that you need to pay attention for costs reduction.

EC2 instances.
EBS volumes.

EC2 Instances

The minimum number of instances you need to run a proper cluster is 3 instances,1 master and 2 nodes. By default, the EC2 instance will be an on-demand instance so to reduce the cost, you need to change to AWS Spot instances. For this, you have to explicitly set the maximum price for your master and nodes instances. This settings can be done either during creation or post creation.

New Cluster

If you want to set it during cluster creation, you will need to run few commands.

First you will need to generate the cluster yaml file.

kops create cluster \
  --name=k8s.blog.taufek.dev \
  --state=s3://k8s.blog.taufek.dev.state \
  --zones=ap-southeast-1a,ap-southeast-1b,ap-southeast-1c \
  --dns-zone=k8s.blog.taufek.dev \
  --dry-run \
  --output yaml | tee k8s.blog.taufek.dev.yml

Then edit the k8s.blog.taufek.dev.yml file by adding maxPrice: "#.##" for both master and nodes settings.

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  ...
  name: master-ap-southeast-1a
spec:
  ...
  maxPrice: "0.10" # Add this line
  maxSize: 1
  ...
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  ...
  name: nodes
spec:
  ...
  maxPrice: "0.10" # Add this line
  maxSize: 2
  ...

Then run below command to create the cluster

kops create -f k8s.blog.taufek.dev.yml --state=s3://k8s.blog.taufek.dev.state
kops create secret --name k8s.blog.taufek.dev ssh publickey admin -i ~/.ssh/id_rsa.pub
kops update cluster k8s.taufek.dev --yes --state=s3://k8s.blog.taufek.dev.state

Note: Before running above to create the cluster, you might want to read EBS Volumes section below because part of the setting is not easily change after the cluster is created.

Existing Cluster

If you already have a running cluster, you can edit the cluster settings. First, run below command to open up the Cluster object.

kops edit cluster --state=s3://k8s.blog.taufek.dev.state

Then it will open up similar cluster yaml file in creation step above. Add the maxPrice fields and save the file.

Then run below to push the change to your cluster.

kops update cluster --yes --state=s3://k8s.blog.taufek.dev.state
kops rolling-update cluster --yes --state=s3://k8s.blog.taufek.dev.state

Run below to know when your cluster is ready

kops validate cluster --state=s3://k8s.blog.taufek.dev.state

EBS Volumes

kops provisions 1 volume for each instances in your cluster and 2 volumes for etcd cluster. In my 1 master and 2 nodes cluster, I will be given 5 volumes.By default, the volumes size are crazily big, at least too big for running my blog site. By default, master volume size is 64G, nodes is 128G and etcd is 2x20GB volumes.

For master and nodes volume size, they can be change during creation and post creation. Unfortunately, kops, does not allow to change etcd volume size during creation.

New Cluster

First you will need to generate the cluster yaml file.

kops create cluster \
  --name=k8s.blog.taufek.dev \
  --state=s3://k8s.blog.taufek.dev.state \
  --zones=ap-southeast-1a,ap-southeast-1b,ap-southeast-1c \
  --master-volume-size=8 \
  --node-volume-size=8 \
  --dns-zone=k8s.blog.taufek.dev \
  --dry-run \
  --output yaml | tee k8s.blog.taufek.dev.yml

For master and nodes volume, you could specify via the command line above but for etcd cluster, you will need to modify the cluster yaml file manually.

Then edit the k8s.blog.taufek.dev.yml file.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
...
spec:
  ...
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-ap-southeast-1a
      name: a
      volumeSize: 8 # Add this volume in GB
    ...
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-ap-southeast-1a
      name: a
      volumeSize: 8 # Add this volume in GB
    ...

Then run below command to create the cluster

kops create -f k8s.blog.taufek.dev.yml --state=s3://k8s.blog.taufek.dev.state
kops create secret --name k8s.blog.taufek.dev ssh publickey admin -i ~/.ssh/id_rsa.pub
kops update cluster k8s.taufek.dev --yes --state=s3://k8s.blog.taufek.dev.state

Existing Cluster

Run below command to open up the Cluster object.

kops edit cluster --state=s3://k8s.blog.taufek.dev.state

Within the Cluster object, set below to configure master and nodes volumes size:

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  ...
  name: master-ap-southeast-1a
spec:
  ...
  role: Master
  rootVolumeSize: 8 # Add this value in GB

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  ...
  name: nodes
spec:
  ...
  role: Node
  rootVolumeSize: 8 # Add this value in GB

Then run below to push the change to your cluster.

kops update cluster --yes --state=s3://k8s.blog.taufek.dev.state
kops rolling-update cluster --yes --state=s3://k8s.blog.taufek.dev.state

For etcd volumes, you can’t use kops to modify the volume size. But there are workaround mentioned in this article. I personally used this workaround to reduce my etcd cluster volumes.

Conclusions

Before I carried out above cost optimization steps, below were the Cluster setup:

1 master instance with on-demand instance
2 nodes instances with on-demand instance
1 master EBS volume with 64G volume size
2 nodes EBS volume with 128G volume size each
2 etcd EBS volume with 20G volume size each

Now the Cluster setup looks like below:

1 master instance with spot instance (up to 90% price reduction)
2 nodes instances with spot instance (up to 90% price reduction)
1 master EBS volume with 8G volume size
2 nodes EBS volume with 8G volume size each
2 etcd EBS volume with 8G volume size each

At the point of writing this post, my Cluster is just 2 days old. I will update this space about the monthly costs once I have to pay the AWS bill at the end of this month.

Continuous Integration for Static Site with Kubernetes

Taufek Johar — Wed, 27 May 2020 16:00:16 +0000

Problem Statement

I always wanted to work with Kubernetes but the subject comes with steep learning curve.Especially for someone who does not work directly with software infrastructure on day-to-day basis. The best way to learn is to actually use it in actual setting. This is my first attempt to use Kubernetes to serve a content on the Internet.

What: Non-Fictitious CI/CD with Kubernetes

My plan is to setup a working and simple Continuous Integration and Continuous Deployment (CI/CD) workflow that integrates with Kubernetes cluster.What can be simpler than deploying a static site. I might as well I start a new blog site.This blog site is the result of CI workflow that I’ll be describing below.

How: Services that Build and Deploy This Blog to Kubernetes Cluster

Services involved in this CI:

GitHub Action. Workflow orchestration tool that executes steps in CI.
Github Package Registry. Docker image registry provided by Github.
Kubernetes cluster. At the point of writing this post I’m running my cluster on AWS with spot instances provisioned by kops. Still evaluating if this is the cheapest method for running Kubernetes cluster. (This is for future blog post)

Below are the high level steps:

Make changes to my blog repository.
Push to origin with tag (i.e v1.0.0). Pushing to master without tag will not trigger the build which allow me to push my changes to master multiple times without triggering the deployment step.
GitHub Action will do its magic to checkout, build and push newly created image to Kubernetes cluster.

Below is the GitHub Action yml file which wires everthing together.

name: Docker

on:
  push:
    tags:
      - v*
env:
  IMAGE_NAME: blog

jobs:
  push:
    runs-on: ubuntu-latest
    if: github.event_name == 'push'

    steps:
      - uses: actions/checkout@v2

      - name: Retrieve Gem Bundles Cache
        uses: actions/cache@v1
        with:
          path: _bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-

      - name: Generate site
        run: |
          mkdir -p _bundle/cache
          chmod a+rwx -R .
          docker-compose run jekyll bundle install
          docker-compose run jekyll bundle exec jekyll build --trace

      - name: Build image
        run: docker build . --file Dockerfile --tag $IMAGE_NAME

      - name: Check bundle
        run: |
          ls _bundle/gems

      - name: Log into registry
        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin

      - name: Push image
        run: |
          IMAGE_ID=docker.pkg.github.com/${{ github.repository }}/$IMAGE_NAME

          # Change all uppercase to lowercase
          IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')

          # Strip git ref prefix from version
          VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')

          # Strip "v" prefix from tag name
          [["${{ github.ref }}" == "refs/tags/"*]] && VERSION=$(echo $VERSION | sed -e 's/^v//')

          # Use Docker `latest` tag convention
          ["$VERSION" == "master"] && VERSION=latest

          echo IMAGE_ID=$IMAGE_ID
          echo VERSION=$VERSION

          docker tag $IMAGE_NAME $IMAGE_ID:$VERSION
          docker push $IMAGE_ID:$VERSION

          echo "::set-env name=VERSION::$VERSION"

      - name: Check Version
        id: check_version
        run: |
          echo ::set-output name=version::$VERSION

      - name: Update k8s deployment
        uses: Consensys/kubernetes-action@master
        env:
          KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }}
        with:
          args: set image deployment/blog-deployment blog=docker.pkg.github.com/taufek/blog/blog:${{ steps.check_version.outputs.version }}

Github Action Yaml Breakdown

Trigger this Github Action workflow if the push to origin includes tag starting with v. For example v1.0.0.

on:
  push:
    tags:
      - v*

Standard steps to checkout and retrieve cache (if there is any).

  - uses: actions/checkout@v2

  - name: Retrieve Gem Bundles Cache
    uses: actions/cache@v1
    with:
      path: _bundle
      key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
      restore-keys: |
        ${{ runner.os }}-gems-

Generate the static site via Jekyll command. I use Docker image to skip Jekyll setup steps.

  - name: Generate site
    run: |
      mkdir -p _bundle/cache
      chmod a+rwx -R .
      docker-compose run jekyll bundle install
      docker-compose run jekyll bundle exec jekyll build --trace

Build new Docker image for this blog. $IMAGE_NAME value is blog.

  - name: Build image
    run: docker build . --file Dockerfile --tag $IMAGE_NAME

  - name: Log into registry
    run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin

Push the Docker image registry as docker.pkg.github.com/taufek/blog/blog:1.0.0.

  - name: Push image
    run: |
      IMAGE_ID=docker.pkg.github.com/${{ github.repository }}/$IMAGE_NAME

      # Change all uppercase to lowercase
      IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')

      # Strip git ref prefix from version
      VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')

      # Strip "v" prefix from tag name
      [["${{ github.ref }}" == "refs/tags/"*]] && VERSION=$(echo $VERSION | sed -e 's/^v//')

      # Use Docker `latest` tag convention
      ["$VERSION" == "master"] && VERSION=latest

      echo IMAGE_ID=$IMAGE_ID
      echo VERSION=$VERSION

      docker tag $IMAGE_NAME $IMAGE_ID:$VERSION
      docker push $IMAGE_ID:$VERSION

      echo "::set-env name=VERSION::$VERSION"

  - name: Check Version
    id: check_version
    run: |
      echo ::set-output name=version::$VERSION

Update Kubernetes Deployment with new image tag.

  - name: Update k8s deployment
    uses: Consensys/kubernetes-action@master
    env:
      KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }}
    with:
      args: set image deployment/blog-deployment blog=docker.pkg.github.com/taufek/blog/blog:${{ steps.check_version.outputs.version }}

You can view the blog repository here taufek/blog.

I’m planning to post more Kubernetes content in the future as I learn more about Kubernetes.

Keep watching this space.