DEV Community: Robus Gauli

Essence of Terraform

Robus Gauli — Thu, 19 Nov 2020 15:51:01 +0000

This blog post is more on the fundamental principle of Terraform and hopefully should provide you a framework and a perspective to understand the Essence of Terraform.

If anyone of you who is going through this does not have a basic understanding of Terraform, I highly encourage you to go through the basics and get back to this blog post.

Having worked with Terraform since its early inception and professionally migrated and provisioned around seven infrastructures, I do feel like I am somewhat qualified to comment on a few aspects of Terraform.

The Essence

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. -- From Official Terraform Documentation

The above definition of Terraform simply means it is a tool that allows us to write syntactically structured configuration describing infrastructure resources. It is also responsible for managing the lifecycle of resources throughout different stages such as provisioning, changes, versioning, and destruction.

Note: Resources could be your S3 Bucket, AKS cluster, EKS cluster, ECR, EC2 instance, Azure blob, etc. You get the idea.

But if we think about it, Terraform is the best possible documentation you could have of your infrastructure that actively mirrors your live infrastructure.

The advantage here is that Terraform can understand your documentation and augment it with runtime. This runtime will then "ideally" try to reflect your updated documentation with actual infrastructure in your Cloud. For instance, if your documentation mentions 3 EC2 instances and your current infrastructure has 2 instances, it will spawn one extra instance to match your documentation. Similarly, it will remove 2 instances if your documentation mentions 1 instance and your actual infrastructure has 3 instances running.

The only job of Terraform is to match whatever you have documented in the form of configuration against your live infrastructure. And during this process of trying to converge to the desired state from the actual state, it does all sorts of trickery such as deletion of resources, an in-place update of resources, and creation of resources.

It will "magically" figure out the dependency graph of all your resources and comes up with the action plans in order to reach that desired state. This action plan that it comes up with in order to reach the desired state from the actual state is captured by the command called terraform plan. This command will provide you with a series of actions such as create, delete, or update that it will execute to converge to the desired state.

Finally, you will run terraform apply if you think the proposed action plans generated by terraform plan are what you actually intended for. After successful completion of terraform apply, you will have your live infrastructure that exactly matches your documentation.

I believe this is everything you need to know in order to understand the very Essence of Terraform and the rest is implementation details.

P.S. The word "documentation" for writing those configurations is called HCL which stands for Hashicorp Configuration Language. This is JSON compatible and allows us to declare those infrastructure resources which is then interpreted by Terraform. They like to call it "Human Readable", but I am not sure if I agree 100% with that claim.

Do follow me on github, twitter and medium. Thanks a lot.

5 things I wish I knew before adding a variable in .env

Robus Gauli — Wed, 18 Nov 2020 15:31:44 +0000

TLDR: Configuration management is hard. Fix it before digging a deeper hole.

Below is the sample .env file that looks pretty standard and innocent. This does not necessarily map to a real-world configuration where you could expect this file to contain way more environment variables.This is pretty standard in projects and we used to roll with it until I started to deal with configurations over the years in regards to managing, provisioning, rotations, policy enforcement, and secrets management.

1. Explicit over Implicit

Never allow optional/default value for a configurable variable. How many times you have seen localhost in your staging/QA environments? How many times you are not able to connect to the database because it used a default value such as Postgres or Testuser because you missed configuring the database in the foo environment?

This gets even difficult when you have to hand off your project you are working on to the Dev ops engineer and they have to deal with your default values for the configurable variable. The next day, your deployed application redirects to localhost:5000/callback?… when the user is authenticated.

REMEDY

Crash your application and scream with giant and ugly stack trace until configuration variables are explicitly jotted or provisioned.

2. Separation of Build and Run time Configuration

I often see running applications that are deployed in the cloud have environment variables such as ECR_REPO_URL, AWS_ACCESS_KEY, etc. I immediately smell the situation where developers are polluting their run time environment variables with build-time environment variables.

ECR_REPO_URL was used to push the application's OCI image to a remote repository and AWS_ACCESS_KEY was used to authenticate against AWS to upload artifacts to s3. These variables were clearly used to package the application and not used during the application's runtime. And yet we still put those build-time environment variables in our classic .env file all the time which are supposed to be used for runtime.

REMEDY

Separate your environment variables explicitly for different stages in your development and deployment pipelines.

For instance, you could use .env.build for build and .env.run for runtime configuration. Over time, you have to manage configurations for integration tests, documentation, etc but hope you got the idea.

3. Mode is better than Prefixed Variable name

In the above .env file, you could see that it is TEST_APP_HOST for the test environment and APP_HOST for the production environment. TEST is prefixed to signify that the TEST_APP_HOST variable is used for the app host in the test environment. This is just a horrible idea in the first place.

This should remind your code that looks something like this

It becomes exponentially difficult to maintain environment variables based on the environment as a number of environments such as staging, QA, pre-production, etc are introduced. And the worst part is, your unit and integration tests in your development won't ensure the full coverage as you won't be testing the code that branches off when you configure it to production.

REMEDY

Use mode to signify the environment and keep the rest of the environment variables without any prefixes.The below code example should give you the idea where we branch off using mode and keep the rest of the statements the same across environments.

The above code example uses Golang as a programming language but should be fairly simple. We enabled tracer and disabled debugging in production and kept the same variable name APP_HOST and APP_PORT irrespective of different environments.

4. Synchronize .env and .env.example files

Hey, could you pass me your .env file?

Let's admit it, your .env.example file always looks "short and sweet" over time while your .env file includes variables that were added during development which does not exist in the .env.example file. You have to copy-paste your environment files to your team member while they are bootstrapping their project because he/she could not run the project as few environment variables were missing which was not included in the .env.example file.

REMEDY

Make .env.example part of your program and crash your app with the ugly exception (if your language has one) if there is a deviation between your .env and .env.example. This will ensure that your .env and .env.example are in sync and no one in the team has to guess what is missing. You could use tools such as sync-dotenv to enforce this.

5. Handle Secrets secretly

We tend to manage our secrets similar to how we manage other environment variables such as DB_USER, DB_HOST. DB_PASSWORD is a secret and requires way more policies around it such as rotations, leasing, revoking auditing, and much more. These secrets are usually dynamic i.e changes over time. Since these secrets require more bookkeeping than the rest of the environment variables, it's best to use tools that are best fit for managing such complexities.

REMEDY

It really depends on the deployment platform. If you are using Kubernetes, you could use a Config map object to store your configuration and a Secrets object to store your sensitive information. You could also opt for Vault by Hashicorp which manages complexities around secrets. If you don't want to manage the Vault Cluster, you could go for services such as Secret Hub.

Types of programmers using Fibonacci series

Robus Gauli — Tue, 27 Oct 2020 06:25:26 +0000

Warning: Not recommended for fainted ❤️ Enjoy 🤘

Functional prodigy
Meta functional expert
One liner wizard
The Librarian
Classy theorist
Closure fanatic
Lazy idealist
OO based lazy conservative
Generator magician
Iterator maniac
Rob Pike

Functional prodigy

def fib(n):
    if n <= 1:
        return n
    return fib(n-1) + fib(n-2)

Meta Functional Expert

def fib(a, b, nth=0, current=0):
    if nth == current:
        return a
    return fib(b, a + b, nth, current+1)

def main():
    series = [fib(0,1,n) for n in range(50)]
    print(series)

One Liner Wizard

f = lambda n: n if n <= 1 else f(n-1) + f(n-2)  # f and not fib because functional laws

The Librarian

pip install fibonacci

from fibonacci import fibo

series = [fibo(n) for n in range(10)]
print(series)

Classy Theorist

class Fibonacci:

    def __init__(self, nth):
        self.nth = nth
        self.a = 0
        self.b = 1
        self.current = 0

    def get(self):
        while True:
            if self.current == self.nth:
                    return self.a
              self.a, self.b = self.b, self.a + self.b
              self.current += 1

def main():
    f = Fibonacci(100)
    # God bless getters
    print(f.get())

Closure Fanatic

from functools import wraps

def closo_fibo_stream():
    # precious closures
    a, b = 0, 1
    @wraps
    def wrapper(*args, **kwargs):
        nonlocal a, b
        current = a
        a, b = b, a + b
        return current
    return wrapper

def main():
    streamer = closo_fibo_stream()
    series = [streamer() for i in range(10)]
    print(series)

Lazy Idealist

def lazy_fibo():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

def main():
    from itertools import islice
    for n in islice(lazy_fibo(), 100):
        print(n) # i feel good

OO based Lazy conservative

# refactored into class coz why not??
class LazyFibo:
    def __init__(self):
        self.a = 0
        self.b = 1

    def __iter__(self):
        while True:
            yield self.a
            self.a , self.b = self.b, self.b + self.a

def main():
    from itertools import islice
    for n in islice(LazyFibo(),10):
        print(n)

Generator Magician

def fib_magico(a, b):
    yield a
    yield from fib_magico(b, a + b)

def main():
    from itertools import islice
    for n in islice(fib_magico(0, 1), 10):
            print(n)

Iterator Maniac

class fib_iter:

    def __init__(self, n):
        self.n = n
        self.current = 0
        self.a = 0
        self.b = 1

    def __next__(self):
        if self.current > self.n:
            raise StopIteration("don't care anymore")
        current = self.a
        self.a, self.b = self.b, self.a + self.b
        self.current += 1
        return current

class fib:

    def __init__(self, n):
        self.n = n

    def __iter__(self):
        return fib_iter(self.n)

def main():
    for i in fib(10):
        print(i)

Rob Pike

def fib(n):
    res = []
    a, b = 0, 1
    for i in range(n):
        res.append(a)
        a, b = b, a + b
    return res

Thanks!! Checkout the repo below

RobusGauli / n-ways-to-fibonacci-series

But why?

11 ways to Fibonacci series

because I'm bored and software are broken. Don't believe me

Warning: Not recommended for fainted ❤️ Enjoy 🤘

By professionals

Functional prodigy

def fib(n)
    if n <= 1:
        return n
    return fib(n-1) + fib(n-2)

Meta Functional Expert

def fib(a, b, nth=0, current=0):
    if nth == current:
        return a
    return fib(b, a + b, nth, current+1)

def main():
    series = [fib(0,1,n) for n in range(50)]
    print(series)

One Liner Wizard

f = lambda n

…

View on GitHub

How do I work on multiple projects simultaneously without losing my mind

Robus Gauli — Tue, 27 Oct 2020 02:33:43 +0000

When you work for a company that has multiple projects running at the same time and your job is to constantly context switch between projects, it becomes nearly impossible to wrap your head around given the project and problem at hand to work on.

Sometimes I am writing K8s ingress configuration for Azure managed Ingress controller where I am required to open up the terminal with Kubectl client configured with the appropriate context, setup VPN connection, open up K8s dashboard to tail logs and sometimes I am writing Dagster pipeline in Python where I am required to set up Redis instance, Celery worker, Rabbitmq, Python’s Virtualenv required, etc etc etc.

I have to sometimes work on 4 to 5 projects in a day and switching between these projects was becoming a pain in my brain. And the worst thing was, it sometimes took me half an hour to switch to project and set up bells and whistles and finally 3 minutes to fix a bug/issue at hand.

I knew it was getting out of my hand and I had to do something about it. I knew one open-source tool called TMUX that could potentially solve my issues, but I had not given serious thoughts. I started playing with Tmux once again, it gave me a way to create multiple terminal windows, split a single window into multiple child terminals (also known as panes), switch between multiple terminal windows, and a lot more. It also gave me a way to manage multiple projects simultaneously by allowing me to create multiple tmux sessions. I was super pleased with what it had to offer.

The above image shows my setup where I have three windows setup for a single project. They are called function, api, and database. I typically name a window based on what they are intended for. If I need to write/debug SQL queries, I would quickly switch to the database window and I would switch to api if I need to work on the API side of the project. If you look closely, the API window consists of 3 child terminals called panes where I keep Neovim(code editor), Virtualenv activated terminal and Postman process running.

Tmux was a game-changer for me. It gets even better if your primary code editor is Vim. Vim + Tmux simply deserves its Own blog. If I were to put it in one sentence, its nothing but Match made in heaven ❤

Okay, now I was able to manage multiple projects using sessions and create multiple terminal windows using Tmux’s windows and panes, you might be wondering if there is anything left to polish. And the answer is YES and here are the reasons below:

The way to create sessions and windows was using its command line program called tmux . For example, to create a new window, you had to run tmux new-window -c /home/mojo/jojo -n database -t project:1 to split a window into panes, you had to run tmux split-window -c /home/mojo/jojo... . You get the idea. It was pretty verbose and imperative.
I had to remember all those commands every time I restart my laptop or kill the Tmux server.
When your project is relatively complex, you had to run 30/40 of these commands and hope you didn’t mess up. Imagine doing this for 4 to 5 projects. It quickly gets unmanageable.

Therefore, I wanted something that was more declarative. I wanted something where I could just declare What I wanted instead of How I wanted. I present to you mx.sh a tool that I wrote for managing Tmux sessions using the declarative configuration file.

RobusGauli / mx.sh

Lean and opinionated tmux session manager.

Manage tmux sessions with simple configuration. Lean and opinionated

Why another tmux session manager?

It has to work out of box without any bloats.
Because, I didn't want runtime dependency such as Ruby interpreter just to manage my tmux sessions.
Shell scripting is free and native, lets do more of that.

Requirements

bash >= 4.x

Installation

Copy paste this in your terminal

curl https://raw.githubusercontent.com/RobusGauli/mx.sh/v0.6.1-alpha/install.sh | bash

This will install mx script in your path. Run mx to verify the installation.

Getting Started

Below steps assumes that you have a working knowledge of tmux and you understand the concepts of windows, panes and sessions in tmux. Also, you are able to attach and detach to tmux session. If you think you are rusty around these topics, you could reach out to man page or this awesome quick tour blog.

1. Generate configuration template

mx template --json --session euler

2. Start

…

View on GitHub

Now you are only required to describe your intent such as the number of windows, the number of panes in a window, command(s) to run on each pane, name of a session, etc and mx.sh will take care of creating a layout that you have described in your configuration file. It will parse your configuration and creates a session with windows, panes, and commands that you have described in your configuration file. Let’s see a sample configuration file below:

The above configuration file manifests the following resources:

A new session named mojo. This could be the name of the project you are working on.
Two windows named w1 and w2. In real world, this could be API, database, frontend, etc.
Window w1 with two panes whose current working directory is set to /home/robus and each pane execute echo command. As you can imagine, your command could source venv/bin/activate , yarn start , kubectl config set-context --current --namespace=admin , etc.
Window w2 with three panes.

Hopefully, this should give an idea of how you could extend this configuration file to add windows/panes/command to your likes. If you are wondering, how would the above configuration manifest in an actual Tmux session, below is the screenshot.

Finally, the way you spin up the new session that is configured to my likes with all the required dependencies is simply

mx up

And if I want to delete the session named mojo

mx down --session mojo

Yup, I am serious. Below is the demo where I am spinning up a project that requires the following components:

Neovim editor for Lamda functions repository.
Terminal where I run Serverless commands.
Neovim editor with python’s flask source code.
Virtual environment activated terminal session.
Postman instance.
Beekeeper instance as GUI SQL client.
VPN connection.
SSH into Bastion host.
Running ‘alias’ command inside in bastion host.
Access to auth0 dashboard, Jira and Aws console.
blah blah …

All these setup now requires single command and about 7 seconds to roll. Pretty happy now that I can switch between projects in seconds and let mx.sh automate everything from starting my code editor to ssh into a remote machine.

I hope this article is useful for those who are suffering similar symptoms of having to spend most of the time getting that initial setup right. If you find my project useful or think you could make improvements, let's do that too.

Cheers!!