DEV Community: s1ntaxe770r

Deploying a Database Cluster on DigitalOcean using Pulumi

s1ntaxe770r — Sat, 19 Aug 2023 00:24:15 +0000

This article was originally posted on EverythingDevOps

Pulumi is an open source infrastructure as code (IaC) tool that allows you to define and manage cloud resources using popular languages such as Golang, Python, Typescript, and a few others.

Pulumi is often compared to Terraform, which is another infrastructure as code tool that allows users to declaratively manage infrastructure using HashiCorp Configuration Language(HCL). The key difference here Is Pulumi allows you to manage your infrastructure using one of their supported SDKs in your language of choice.

In this guide, you would be using Typescript to deploy a PostgreSQL database cluster on DigitalOcean, as such, this guide assumes some familiarity with Typescript.

Why Pulumi?

Unlike most infrastructure as code tools, Pulumi allows you to define your infrastructure using a general-purpose programming language, this allows your code to be tested much more easily. If you are familiar with Terraform you’d agree that not many testing frameworks exist for it. In comparison to a language like Python, where numerous testing frameworks exist.

Another advantage lies in the use of a general-purpose programming language, most developers would find using their favorite language more intuitive than a DSL (Domain-Specific Language*)* such as HCL.

Prerequisites

To follow along in this tutorial you would need the following:

A DigitalOcean account
DigitalOcean API token
Pulumi CLI
Node.js. ## Initializing a Pulumi project

The Pulumi CLI provides a command for scaffolding new projects. To do this, run the following commands:

mkdir postgres-db && cd postgres-db

Note: It’s important the directory is empty else, the next command will return an error.

pulumi new typescript -y

The above command will show an output showing Pulumi initialization, as shown in the image below.

This generates a new Pulumi project along with a stack, a stack is an Independent instance of a Pulumi program, and each stack is usually used to represent a separate environment (production, development, or staging). This is similar to workspaces if you are familiar with Terraform.

Installing Dependencies

To interact with DigitalOcean resources you would need to install the DigitalOcean Pulumi package:

$ npm install @pulumi/digitalocean

Next, set your DigitalOcean API key:

$ pulumi config set digitalocean:token YOUR_DO_API_KEY --secret

This would store your API key so Pulumi can authenticate against the DigitalOcean API, the --secretflag ensures the value passed in is encrypted.

You could also set your token using an environment variable:

$ export DIGITALOCEAN_TOKEN=YOUR-DO-API-KEY

Provisioning a Database Cluster

To get started open index.ts and follow along with the code below:

import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";

const pg_cluster = new digitalocean.DatabaseCluster("pulumi-experiments", {
    engine: "pg",
    nodeCount: 2,
    region: "nyc1",
    size: "db-s-2vcpu-4gb",
    version: "12",
});

export const db_uri = pg_cluster.uri;

The above code creates a PostgreSQL database cluster with 2 nodes:

engine allows you to specify the database cluster you want to create. In this case, it’s PostgreSQL. However, DigitalOcean supports a few other database types, see this section of the Pulumi documentation for more information.
Another important field to note is size, this allows you to configure the size of each node, see this section of the Pulumi documentation for all valid database slugs.
version allows you to specify what version of PostgreSQL you would like to run. Here is a list of all supported PostgreSQL versions on DigitalOcean.
Finally, you export the database connection URI so you can connect to it using your client of choice.

To deploy the cluster, run the following:

$ pulumi up

In a few minutes, you should have a cluster up and running.

Notice how db_uri is marked as “secret”, this is because it is a sensitive value, to output your database URI run the following command:

$ pulumi stack output db_uri --show-secrets > pass.db

The above would write the database URI to a file called pass.db.

Updating your Infrastructure

Now that you have a database cluster, chances are you are not going to stop here, and you would want to update your infrastructure. You can do this by adding a user to the newly created cluster.

It's good practice to create a separate user to avoid using the admin user, update index.ts with the following code:

import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";

const pg_cluster = new digitalocean.DatabaseCluster("pulumi-experiments", {
    engine: "pg",
    nodeCount: 2,
    region: "nyc1",
    size: "db-s-2vcpu-4gb",
    version: "12",
});

const pg_user = new digitalocean.DatabaseUser("non-admin",{clusterId:pg_cluster.id})

export const db_uri = pg_cluster.uri;
export const pg_user_pass = pg_user.password

Here you create a new instance of digitalocean.DatabaseUser, pass in the clusterId of pg_cluster and export the new user’s password as we did with the database URI.

To see what changes would be applied, run the following command:

$ pulumi preview

The output should look something like this:

Once you are satisfied with the changes, go ahead and apply the changes.

$ pulumi up

Once this completes, you should have a new database user called non-admin, you can output the password by running.

$ pulumi stack output pg_user_pass --show-secrets

Clean up (Optional)

To tear down the resources you just created, run the following command:

$ pulumi destroy

Conclusion

In this post, you deployed a PostgreSQL database cluster using Pulumi and updated it with a non-admin user. However, this is one of several services that you could potentially deploy using Pulumi. Hopefully, this was enough to get you started with Pulumi.

Next Steps

Check out this guide on how to deploy a Kubernetes cluster using Pulumi
Take a look at this section of the Pulumi registry for a full list of supported services

How To Deploy Meshery In Kind

s1ntaxe770r — Sun, 03 Oct 2021 13:23:54 +0000

In this post, I would be showing you how to deploy Meshery on Kubernetes using Kind but first…

What the heck is Meshery?

If you are reading this chances are you are already familiar with Meshery or you are looking to find out what it is. Well, you are in the right place.

Meshery is an open-source Service Mesh management plane. In simpler terms Meshery allows you to orchestrate the installation and management of different Service Meshes, Meshery also allows you to evaluate the performance of Service Meshes using the SMP specification These are just two of the features Meshery provides out of the box. If I have gotten you a tiny bit interested in Meshery head over to https://docs.meshery.io/functionality for a list of additional features Meshery provides.

Now that you’re familiar with what Meshery is let's get it installed.

Setup

Before we get started be sure you have docker and go installed as both are requirements for installing Kind, we'll also be needing helm to deploy Meshery

Installing Kind

to install Kind run the following command

GO111MODULE="on" go get sigs.k8s.io/kind@v0.11.1

if you run into an error along the lines of :

zsh: command not found: kind

Try adding the following alias to your shell configuration

alias kind="$GOBIN/kind"

Creating a cluster

Next, we'll create a kind cluster with an Ingress enabled, this ingress will come in handy when we want to expose Meshery later on.

Create a file called cluster.yaml and populate the file with the following code:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP

Now we the following command to create a cluster using the cluster configuration

kind create cluster --name meshery --config cluster.yaml

After a few minutes, you should have a Kubernetes cluster up and running.

Installing Meshery

hop into your terminal and run the following command to get Meshery installed

 $ git clone https://github.com/layer5io/meshery.git; cd meshery
 $ kubectl create namespace meshery
 $ helm install meshery --namespace meshery install/kubernetes/helm/meshery

Exposing meshery

As mentioned earlier on we would access Meshery by using Ingress, create a file called meshery-ingress.yaml, and add the following configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: meshery-ingress
  labels:
    name: meshery-ingress
spec:
  rules:
  - host: meshery.local
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: meshery
            port: 
              number: 9081

Apply the configuration using kubectl apply -n meshery -f meshery-ingress.yaml .

Now create the following entry in /etc/hosts

127.0.0.1 meshery.local

At this point if you head over to http://meshery.local in your browser you should be able to access Meshery's UI which looks something like this.

Configuring Meshery's command-line client

While you could interact with Meshery from the UI only, at some point you are going to want to use the command line client which is what mesheryctl is. So let's get that installed.

Head over to https://github.com/meshery/meshery/releases/ and download the binary for your operating system. Next unzip the file and move it to your path

unzip mesheryctl_0.5.52_Darwin_x86_64.zip
mv mesheryctl /usr/local/bin/mesheryctl

The version of the binary might differ depending on when you are reading this.

Now that you have mesheryctl installed you should be able to run mesheryctl version .On your first try you should see something like this

~
❯ mesheryctl version     
Missing Meshery config file.
Create default config now [y/n]?

enter y and mesheryctl would generate a config file which we would also be needing later on.

if all went well you should be presented with this error message

Default config file created at /Users/someguy/.meshery/config.yaml
        VERSION     GITSHA      
Client  v0.5.62     35e8d943    
Server  unavailable unavailable 

  Unable to communicate with Meshery: Get "http://localhost:9081/api/system/version": dial tcp [::1]:9081: connect: connection refused
  See https://docs.meshery.io for help getting started with Meshery.

Checking for latest version of mesheryctl...

  v0.5.62 is the latest release.

This happens because mesheryctl is trying to communicate with Meshery on the default address, in our case it's http://meshery.local. Luckily we can change this using the config file Meshery generated earlier.

Open up the config file located at ~/.meshery/config.yaml

contexts:
  local:
    endpoint: http://localhost:9081
    token: Default
    platform: docker
    adapters:
    - meshery-istio
    - meshery-linkerd
    - meshery-consul
    - meshery-nsm
    - meshery-kuma
    - meshery-cpx
    - meshery-osm
    - meshery-traefik-mesh
    - meshery-nginx-sm
    channel: stable
    version: latest
current-context: local
tokens:
- name: Default
  location: auth.json

Taking a closer look we see that the endpoint is set to [localhost:9081](http://localhost:9081), modify the file so it looks like this

contexts:
  local:
    endpoint: http://meshery.local
    token: Default
    platform: kubernetes
    adapters:
    - meshery-istio
    - meshery-linkerd
    - meshery-consul
    - meshery-nsm
    - meshery-kuma
    - meshery-cpx
    - meshery-osm
    - meshery-traefik-mesh
    - meshery-nginx-sm
    channel: stable
    version: latest
current-context: local
tokens:
- name: Default
  location: auth.json

Here i changed the endpoint and platform to match our current configuration. Now run mesheryctl version again and you should see the following:

❯ mesheryctl version      
        VERSION GITSHA   
Client  v0.5.62 35e8d943    
Server  v0.5.62 35e8d943    

Checking for latest version of mesheryctl...

And there you go, we successfully deployed Meshery in kind and configured the CLI to interact with Meshery. If you have any questions or want to contribute to the Meshery project feel free to join the slack workspace using the link here. Now go forth and make a mesh of things

Deploying a Go-based app to AKS using Kubestack

s1ntaxe770r — Mon, 30 Aug 2021 14:16:56 +0000

In this post, I would be showing you how to deploy a golang application to Kubernetes using Kubestack. Kubestack is a Gitops automation built on Terraform to reduce the risk of deploying and increasing development speed. That being said I would not be covering how to set up kubestack but rather how to deploy Kubernetes manifests using Kubestack, for more info on how to up a working Kubestack pipeline check out the getting started guide over here.

Now let's dive in.

Setup

Once you have your kubestack pipeline set up your folder structure should look something like this

    .
    ├── Dockerfile
    ├── Dockerfile.loc
    ├── README.md
    ├── aks_zero_cluster.tf
    ├── aks_zero_ingress.tf
    ├── aks_zero_providers.tf
    ├── manifests
    └── versions.tf

to deploy our Kubernetes manifest we would be making use of a cluster service module. Cluster service modules in Kubestack allows Terraform to interact directly with Kubernetes through Terraform, this covers things like deploying and creating namespaces, creating deployments..etc.

Open up ask-zero_cluster.tf and add the following lines

    ...

    module "custom_manifests" {
      providers = {
        kustomization = kustomization.aks_zero
      }
      source  = "kbst.xyz/catalog/custom-manifests/kustomization"
      version = "0.1.0"
      configuration = {
        apps = {
          namespace = "apps-${terraform.workspace}"
          resources = [
                "${path.root}/manifests/apps/namespace.yaml",
            "${path.root}/manifests/apps/deployment.yaml",
            "${path.root}/manifests/apps/service.yaml",
            "${path.root}/manifests/apps/ingress.yaml"
          ]
          common_labels = {
            "env" = terraform.workspace
          }
        }
        ops = {}
        loc = {}
      }
    }

Here we initialize the custom manifest cluster service module and tell kustomize to use the existing aks_zero module using.

     providers = {
        kustomization = kustomization.aks_zero
      }

The configuration block is where the magic happens, first, we specify we declare what workspace we want our resources to be deployed to using


     apps = {
          namespace = "apps-${terraform.workspace}"
          resources = [
                "${path.root}/manifests/apps/namespace.yaml",
            "${path.root}/manifests/apps/deployment.yaml",
            "${path.root}/manifests/apps/service.yaml",
            "${path.root}/manifests/apps/ingress.yaml"
          ]
          common_labels = {
            "env" = terraform.workspace
          }
        }

Next, we declare what namespace the manifests should be deployed to using namespace = "apps${terraform.workspace}" which would translate to apps-ops or apps-apps depending on your current workspace, next the resources block tells terraform to look in manifests folder in the current directory for the manifests we want to deploy. Now that we have the cluster module set up let's create the manifests.
Run the following commands:

mkdir manifests/apps && cd manifests/apps && touch deployment.yaml service.yaml ingress.yaml

Now populate the files with the following code.

# manifests/apps/namespace.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: apps-apps

---
apiVersion: v1
kind: Namespace
metadata:
  name: apps-ops

Note: since this is a fresh Kubernetes cluster I have included a manifest for a namespace as kubestack would deploy the resources in the order specified in the module so the namespace apps-apps would need to exist first.

#manifests/apps/deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ping-api
spec:
  selector:
    matchLabels:
      app: ping-api
  template:
    metadata:
      labels:
        app: ping-api
    spec:
      containers:
      - name: ping-api
        image: ghcr.io/s1ntaxe770r/evil-ekow:latest
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8080
        env:
          - name: REDIS_CACHE_HOST
            value: "redis-svc"
          - name: REDIS_PORT
            value: "6379"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:alpine
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 6379

# manifests/apps/service.yaml
--
apiVersion: v1
kind: Service
metadata:
  name: redis-svc
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379

---
apiVersion: v1
kind: Service
metadata:
  name: ping-svc
spec:
  selector:
    app: ping-api
  ports:
  - port: 80
    targetPort: 8080
# manifests/apps/ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myingress
  labels:
    name: myingress
spec:
  rules:
  - host: prod.evil.corp
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: ping-svc
            port:
              number: 80

The above manifests will deploy a version of an API that uses Redis to store key-value pairs. You can check the source code out here.

Next, you can test out these changes locally by running kbst apply local, if all looks good you should be able to tag and deploy like this:

$ git tag apps-deploy-1
$ git push origin apps-deploy-1

Accessing your deployment

Because I don't have a valid domain at the time of writing I would be using my ingress controller external IP to access the deployment, however, if you have your domain properly configured pls refer to this portion of the
Kubestack documentation.

First, grab your ingress controller\'s external IP using:

$  kubectl get ingresses -A

Next, create the following entry in /etc/hosts

10.10.20.5   prod.evil.corp

be sure to change the IP to your ingress controllers external IP ,now you should be able to access the swagger docs at
http://prod.evil.corp

Conclusion

In this post, I covered how to deploy your Kubernetes manifests using Kubestack on Azure do note that this is not limited to azure as Kubetack also supports GKE and AWS at the time of writing this post, so be sure to check out the
documentation for any provider-specific steps. Now go forth and git deploying 😄

5 reasons why frameworks make sense for infrastructure as code

s1ntaxe770r — Tue, 13 Jul 2021 13:41:18 +0000

In this post I hope to provide a few reasons why frameworks make sense for infrastructure as code, Having tried out Kubestack this got me thinking, Should more of these frameworks exist? And do they provide the same value as a traditional web framework like Flask, Express.js, or Ruby on Rails. Now you might be thinking don't tools like Terraform, Ansible, Chef already exists?

Well yes, but these technologies themselves are just tools and not necessarily Frameworks.

Back to basics

So what exactly is a framework in the context of software development? A quick google search would give you an array of answers, here are two answers/analogies I really like.

From djangostars.com

A web framework is a software tool that provides a way to build and run web applications. As a result, you don’t need to write code on your own and waste time looking for possible miscalculations and bugs.

From Hashnode

When selecting a web framework for a project there a few things one has to consider.

Speed of development
Maintainability
Eliminate repetitive code
Community support / documentation
Reduce risk

How is any of this related to infrastructure as code?

To understand how this relates to infrastructure it's worth taking a look at a typical scenario where you choose a framework. Say you only want to accept post requests on /login in your app if you were to implement this from scratch you'd find that it is far from ideal in most cases and could potentially have its own flaws, Whereas virtually every modern web framework ships with it's own routing solution, saving you overhead that comes with implementing this from scratch.

As I mentioned earlier the premise for this article was my experience with kubestack, for the remainder of this article I would be making several references to it. But first a little bit on Kubestack.

Kubestack is an infrastructure automation framework built on top of terraform. Kubestack's primary aim is not to reinvent the wheel but foster rapid development using when using terraform.

From kubestack.com

Kubestack is the open-source Terraform framework for teams that want to automate infrastructure, not reinvent automation.

Now the same could be said about building infrastructure, if you were to spin up a Kubernetes cluster using terraform, there is a generic process of obtaining some boilerplate and customizing it to suit your needs. Now this isn't exactly a problem, however in a scenario where you have to create more than one cluster you may find yourself repeating this process once again and this is where my point about eliminating repetitive code comes in. Move over to something like Kubestack and this boilerplate is generated for you, this would be the equivalent of running Django-admin startproject xyz if you are familiar with Django. Obviously, this isn't convincing enough so let's do a side-by-side comparison of some reasons to use a regular web development framework and see if they could apply to infrastructure.

Side-by-Side comparison

For this section I would be looking at the following points:

Speed of development
Maintainability
Reducing repetitive code
Community support
Reduce risk

Here I'll discuss how each of these points also applies to infrastructure as code.

Speed of development

In general speed of development is the time it takes to get an idea from your head into a working application, the tooling/frameworks you use can increase or hinder your development speed.

When creating a Kubernetes cluster a few things are relatively constant across each cloud e.g( picking node size/count, region ), at this point you already start to see how a framework could speed up development by abstracting these components whilst leaving room for you to customize as you please.

This is something Kubestack provides out of the box but I won't go into detail in this article. See here for more info.

Maintainability

Being able to write code quickly isn't always worth the effort if your app becomes a nightmare to maintain. Maintainability in software development refers to the process of writing code that is easy to understand, fix and extend.

"Good programmers write code that humans can understand." - Martin Fowler

I've seen a fair amount of projects become a nightmare from a lack of structure. Frameworks like Django or Ruby on Rails provide a nice scaffold to build your projects on top of, they also import relevant modules you are bound to use at some point. This is not to say that frameworks magically make all your problems go away but in general I think it provides a good stepping stone to making your app maintainable.

This directly transfers over to infrastructure as code to a certain extent, when starting a new project using something like Terraform for instance, it's hard to predict how large it would grow and this could potentially how you structure your modules, having a structure with relevant information pre-filled would significantly increase development. Going back to the inheritance model Kubestack adopts the chances that you misconfigure a module or copy-paste wrongly are significantly reduced.

Ansible actually does better in this regard with ansible-galaxy , helping you generate a rather sensible folder structure for your playbooks. Learn more about ansible-galaxy here.

Reducing repetitive code

reducing repetitive processes is at the core of both infrastructure as code and using software development frameworks, but in trying to automate I often find the process a bit repetitive and error-prone, having to use the same piece of code here every so often, Once again frameworks quickly reduce the amount of effort spent doing small things like this. Many frameworks ensure developers follow DRY principles. Terraform uses HCL for configuration , which by itself can become repetitive very quckily, classic example would be when provision more than one of the same resource

You might find your self copy-pasiting code or doing some weird stuff to get around this, whereas when using Kubestack your configuration follows an inheritance model which helps keep your configuration DRY this means you define once and simply inherit in any child module you wish

to make use of it in.

Community support /Documentation

This is perhaps one of the biggest advantages of using a framework, for every error you run into while using a framework there is a good chance a fellow user has experienced such and if for some reason they haven't this could be an opportunity to get involved with the project if you so choose. I can't imagine learning how to use Flask without the thousands of StackOverflow posts and issues on GitHub. No doubt a healthy community fosters the growth of any language or framework and I think the same holds true for infrastructure as code if the were more people using IAC tools with a framework approach at a lot more questions would be asked and ultimately leading to growth.

Reducing risk

When using IAC tools it's vital that your code is tested properly as the slightest of misconfigurations tends to have a significant impact on your infrastructure, when using a framework things are less likely to break as most features undergo a certain level of testing before it can be accepted, move over to IAC world and most of the testing is left to you, a typical example would be an incident that happened at Spotify, more details on that here. One common problem terraform users often encounter is what I am calling "state drift"(because I don't know what it's called), where your terraform state and code do not match leading to some undesirable results which was the case with Spotify. Going back to Kubestack, when using it, the remote state is automatically set up for you, which means you can focus on developing your infrastructure rather than worrying about your state files. This can be thought of like how Django provides a simple interface for setting up connections for different types of databases. Of all the points so far this is the most critical part, bad infrastructure directly affects end-users.

So what's the point of this article?

This article is not to advertise the use of frameworks in every project in fact avoid using them for as long as it makes sense, this is rather a subtle call to action and to hopefully convince about the benefits it could provide, i think it's time the DevOps tooling evolved in the framework direction.

If you'd like to learn more about Kubestack you can check that out over here.

Provisioning an azure kubernetes cluster with Terraform

s1ntaxe770r — Sat, 27 Mar 2021 07:56:16 +0000

In this post you will learn how to set up an Azure Kubernetes cluster using Terraform.

NOTE: This article assumes some basic knowledge of cloud concepts and the Microsoft Azure platform

Terraform ??

Terraform is an Infrastructure as code tool that allows developers and operations teams to automate how they provision their infrastructure.

Why write more code for my infrastructure ?

If you are new to Infrastructure as code it could seem like an extra step , when you could just click a few buttons on you cloud provider of choice's dashboard and be on your way. But IaC(Infrastructure as code) offers quite a few advantages.

Because your infrastructure is now represented as code it is testable
Your environments are now very much reproducible
You can now track changes to your infrastructure over time with a version control system like Git
Deployments are faster,because you interact with the cloud provider less.

Before diving into Terraform you need a brief understanding of

Hashicorp configuration language(HCL).

HCL ?

Yes Terraform uses a it's own configuration language, this may seem daunting at first but it's quite easy. Here's a quick peek at what it looks like.

resource "azurerm_resource_group" "resource-group" {
  name     = "staging-resource-group"
  location = "West Europe"
}

Your infrastructure in Terraform are represented as "resources", everything from networking to databases or virtual machines are all resources.

This is exactly what the resource block represents. Here we are creating an azurerm_resource_group as the name implies , it's a resource group. Resource groups are how you organize resources together, typical use case would be putting all your servers for single project under the same resource group.

Next we give the resource block a name , think of this as a variable name we can use throughout the Terraform file. Within the resource block we give our resource group a name, this is the name that would be given to our resource group in Azure. Finally we give a location where we want the resource group to be deployed.

If you are coming from something like Ansible you might notice how different Terraform's approach to configuration is, this is because Terraform uses whats known as an imperative style of configuration, simply put. In an imperative style of configuration you declare the state you want your infrastructure in and not how you want to achieve that state. You can learn more about declarative and imperative configuration here

Now that you have an idea of what Terraform configuration looks like lets dive in.

Project setup

Prerequisites:

An Azure account
Git
Azure CLI
Terraform CLI

Once you have all that setup, login to your Azure account through the command line using the following command

$ az login

Next clone the sample project.

$ git clone https://github.com/s1ntaxe770r/aks-terraform-demo.git

Before we begin we need to run terraform init. This would download any plugins that the Azure provider depends on.

$ terraform init

Taking a quick look at our folder structure you should have something like this.

.
├── main.tf
├── modules
│   └── cluster
│       ├── cluster.tf
│       └── variables.tf
├── README.md
└── variables.tf

2 directories, 6 files

Starting from the top lets look at main.tf

#main.tf

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "2.39.0"
    }
  }
}
provider "azurerm" {
  features {}
}

module "cluster" {
  source                = "./modules/cluster"
  ssh_key               = var.ssh_key
  location              = var.location
  kubernetes_version    = var.kubernetes_version

}

First we declare what Provider we are using, which is how Terraform knows what cloud platform we intend on using , this could be Google cloud , AWS or any other provider they support. You can learn more about Terraform providers here. Its also important to note that each provider block is usually in the documentation so you don't need to write this out each time.

Next we define a module block and pass it the folder where our module is located and a few variables.

Modules??

Modules in Terraform are a way to separate your configuration so each module can handle a specific task. Sure we could just dump all of our configuration in main.tf but that makes things clunky and less portable.

Now lets take a look at the cluster folder in modules directory.

modules/cluster
├── cluster.tf
└── variables.tf

0 directories, 2 files

Lets take a look at cluster.tf

#modules/cluster/cluster.tf

resource "azurerm_resource_group" "aks-resource" {
    name = "kubernetes-resource-group"
    location = var.location
}

resource "azurerm_kubernetes_cluster" "aks-cluster" {
    **name = "terraform-cluster"
    location = azurerm_resource_group.aks-resource.location
    resource_group_name = azurerm_resource_group.aks-resource.name
    dns_prefix = "terrafo**rm-cluster"
    kubernetes_version = var.kubernetes_version

    default_node_pool {
      name = "default"
      node_count = 2
      vm_size = "Standard_A2_v2"
      type = "VirtualMachineScaleSets"
    }


  identity {
    type = "SystemAssigned"
  }

    linux_profile {
        admin_username = var.admin_user
        ssh_key {
            key_data = var.ssh_key
        }
    }

    network_profile {
      network_plugin = "kubenet"
      load_balancer_sku = "Standard"
    }

}

in the first part of the part of the configuration we define a resource group for our cluster and cleverly name it "kubernetes-resource-group", and give it a location which would come from a variable which is defined in variables.tf. The next part are the actual specs of our kubernetes cluster. First we tell Terraform we want an azure kubernetes cluster using resource "azurerm_kubernetes_cluster" , then we give our cluster a name , location and a resource group. We can use the location of the resource group we defined earlier by using the reference name aks-resource plus the value we want. In this case it's the location so we use aks-resource.location.

There are two more blocks that we need to pay attention too. The first being default_node_pool block and the second linux_profile.

default_node_pool block lets us define how many nodes we want to run and what type of virtual machines we want to run on our nodes. Its important you pick the right size for your nodes as this can affect cost and performance. You can take a look at what VM sizes azure offers and their use cases over here. node_count tells terraform how many nodes we want our cluster to have. Next we define the VM. Here I'm using and A series VM with 4 gigs of ram and two CPU cores. Lastly we give it a type of "virtual machine scale sets" which basically lets you create a group of auto scaling VM's

The last block we need to look at is linux_profile . This creates a user we can use to ssh into one of our nodes in case something goes wrong. Here we simply pass the block variables.

I intentionally didn't go over all the blocks because most times you don't need to change them and if you do the documentation is quite easy to go through.

Finally lets take a look at variables.tf as you might have guessed this is were we define all the variables we referenced earlier.

#variables.tf

variable "location" {
    type = string
    description = "resource location"
    default = "East US"
}

variable "kubernetes_version" {
  type = string
  description = "k8's version"
  default = "1.19.6"
}

variable "admin_user"{
  type = string
  description = "username for linux_profile"
  default = "enderdragon"
}

variable "ssh_key" {
   description = "ssh_key for admin_user"
}

to define a variable we use the variable keyword , give it a name and within the curly braces we define what type it is, in this case it's a string, an optional description and a default value, which is also optional.

Now we are almost ready to create our cluster but first we need to generate an ssh key , if you remember we created a variable for it earlier. if you have an ssh key pair you can skip this step

$ ssh-keygen -t rsa -b 4096

you can leave everything as default by pressing enter. Next we export the key into an environment variable.

$ export TF_VAR_ssh_key=$( cat ~/.ssh/id_rsa.pub)

Notice the TF_VAR prefix before the name of the actual variable name. This so Terraform is aware of the environment variable and can make use of it. You should also note that the variable name should correspond to the one in variables.tf.

Before we actually create our infrastructure its always a good idea to see what exactly Terraform would be creating luckily Terraform has a command for that

$ terraform plan

The output should look something like this

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.cluster.azurerm_kubernetes_cluster.aks-cluster will be created
  + resource "azurerm_kubernetes_cluster" "aks-cluster" {
      + dns_prefix              = "terraform-cluster"
      + fqdn                    = (known after apply)
      + id                      = (known after apply)
      + kube_admin_config       = (known after apply)
      + kube_admin_config_raw   = (sensitive value)
      + kube_config             = (known after apply)
      + kube_config_raw         = (sensitive value)
      + kubelet_identity        = (known after apply)
      + kubernetes_version      = "1.19.1"
      + location                = "eastus"
      + name                    = "terraform-cluster"
      + node_resource_group     = (known after apply)
      + private_cluster_enabled = (known after apply)
      + private_fqdn            = (known after apply)
      + private_link_enabled    = (known after apply)
      + resource_group_name     = "kubernetes-resource-group"
      + sku_tier                = "Free"

      + addon_profile {
          + aci_connector_linux {
              + enabled     = (known after apply)
              + subnet_name = (known after apply)
            }

          + azure_policy {
              + enabled = (known after apply)
            }

          + http_application_routing {
              + enabled                            = (known after apply)
              + http_application_routing_zone_name = (known after apply)
            }

          + kube_dashboard {
              + enabled = (known after apply)
            }

          + oms_agent {
              + enabled                    = (known after apply)
              + log_analytics_workspace_id = (known after apply)
              + oms_agent_identity         = (known after apply)
            }
        }

      + auto_scaler_profile {
          + balance_similar_node_groups      = (known after apply)
          + max_graceful_termination_sec     = (known after apply)
          + scale_down_delay_after_add       = (known after apply)
          + scale_down_delay_after_delete    = (known after apply)
          + scale_down_delay_after_failure   = (known after apply)
          + scale_down_unneeded              = (known after apply)
          + scale_down_unready               = (known after apply)
          + scale_down_utilization_threshold = (known after apply)
          + scan_interval                    = (known after apply)
        }

      + default_node_pool {
          + max_pods             = (known after apply)
          + name                 = "default"
          + node_count           = 2
          + orchestrator_version = (known after apply)
          + os_disk_size_gb      = (known after apply)
          + os_disk_type         = "Managed"
          + type                 = "VirtualMachineScaleSets"
          + vm_size              = "Standard_A2_v2"
        }

      + identity {
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = "SystemAssigned"
        }

      + linux_profile {
          + admin_username = "enderdragon"

          + ssh_key {
              + key_data = "jsdksdnjcdkcdomocadcadpadmoOSNSINCDOICECDCWCdacwdcwcwccdscdfvevtbrbrtbevF
CDSCSASACDCDACDCDCdsdsacdq$q@#qfesad== you@probablyyourdesktop"
            }
        }

      + network_profile {
          + dns_service_ip     = (known after apply)
          + docker_bridge_cidr = (known after apply)
          + load_balancer_sku  = "Standard"
          + network_plugin     = "kubenet"
          + network_policy     = (known after apply)
          + outbound_type      = "loadBalancer"
          + pod_cidr           = (known after apply)
          + service_cidr       = (known after apply)

          + load_balancer_profile {
              + effective_outbound_ips    = (known after apply)
              + idle_timeout_in_minutes   = (known after apply)
              + managed_outbound_ip_count = (known after apply)
              + outbound_ip_address_ids   = (known after apply)
              + outbound_ip_prefix_ids    = (known after apply)
              + outbound_ports_allocated  = (known after apply)
            }
        }

      + role_based_access_control {
          + enabled = (known after apply)

          + azure_active_directory {
              + admin_group_object_ids = (known after apply)
              + client_app_id          = (known after apply)
              + managed                = (known after apply)
              + server_app_id          = (known after apply)
              + server_app_secret      = (sensitive value)
              + tenant_id              = (known after apply)
            }
        }

      + windows_profile {
          + admin_password = (sensitive value)
          + admin_username = (known after apply)
        }
    }

  # module.cluster.azurerm_resource_group.aks-resource will be created
  + resource "azurerm_resource_group" "aks-resource" {
      + id       = (known after apply)
      + location = "eastus"
      + name     = "kubernetes-resource-group"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

If every thing looks good we can apply our configuration using:

$ terraform apply

Terraform will prompt you one last time to make sure you want to proceed enter yes and watch the magic happen. Once the resources have been provisioned head over to your azure dashboard a look. You should see something like this:

As you can see, Terraform configured everything we needed to spin up a cluster, and we didn't have to specify everything. Click on terraform-cluster and lets make sure everything looks good.

And there you have it, we deployed a kubernetes cluster with our desired specifications and Terraform all did the heavy lifting.

Once you are done it's as easy as running terraform destroy to tear down all the resources you have just provisioned.

Quick recap

You learnt :

Why Infrastructure as code is important is important
The basics of HCL(Hashicorp configuration language)
How to provision a kubernetes cluster with terraform

If you are wondering where to go from here. Here are somethings you can try.

Here we authenticated through the azure CLI but that's not completely ideal. Instead you might want to use a service principal with more streamlined permissions. Check that out over here
You should never store your state file in version control as that might contain sensitive information. Instead you can put it in an azure blob store .
There are better ways to pass variables to terraform which i did not cover here, but this post on the terraform website should walk you through it nicely.
Finally. This article couldn't possibly cover all there is to terraform, so i highly suggest taking a look at the Terraform documentation, it has some boilerplate configuration to get yo u started provisioning resources.

All the code samples used in this tutorial can be found here

Automating python deployment environments setup with ansible

s1ntaxe770r — Wed, 26 Aug 2020 13:18:00 +0000

Prerequsites

to follow along with this post you would need the folowing installed on your machine

ansible
python3 (on your local machine and remote server).

In this post i will show you how you can automate creating python deployment environment's using Ansible.

But setting this up manually isn't a problem ?

say you have a web application that uses the flask framework, deploying this to an individual server manually wouldn't be a problem, scale this up to more than three applications with only slight variations in dependencies and things start to get pretty repetitive.This is where ansible comes in.

What's this ansible thing anyway ?

Ansible is an open source automation tool that can be used for configuration management, application deployment and infrastructure provisioning,to mention a few.

Ok lets do it then!

Start by creating the following folder structure.

Directory_Name
├── ansible.cfg
├── inventory
├── requirements.txt
└── playbook.yml

Lets start with ansible.cfg this is where we would set configuration values that we want ansible to use when it runs.

ansible.cfg

[defaults]
inventory = ./inventory

The above simply tells ansible to use the inventory file in the current directory.

Whats an inventory file?

ansible uses this to determine what what machine you would like to run your playbooks against.

inventory

[deployment server]
127.0.0.1

within the square brackets i specified the the name i would use to refer to a group of servers here i have just one but in practice 127.0.0.1 would be the ip address of the machines you want to run the playbooks.If you need to create a new group simply add another pair of square brackets and add the ip address of the targets machines below like above.If you would like to run this against your machine replace 127.0.0.1 with 127.0.0.1 ansible_connection=local,which is what i did.

Playbooks

playbooks in ansible simply refer to where you define the tasks you want to execute on the remote machine, playbooks are written in Yaml which is a markup language.

playbook.yml

---
- hosts: all
  gather_facts: true
  tasks:
    - name: install dependencies from requirements.txt
      pip:
        requirements: requirements.txt
        chdir: .
        executable: pip3


    - name: additional dependency
        pip:
          name: gunicorn
          executable: pip3

we use hosts to indicate what group we want to run the playbook against here i used all to run against all the ip addresses i have in my inventory, to run against a single group replace all with the name you placed within square brackets in the inventory ,ansible uses gather_facts to collect information about the target.

tasks is where we define what we actually want to do each task has a name which is used to make playbooks more readable and is also used in the output to indicate what task ansible is currently executing.pip3 is the ansible module we are using , when installing from a requirements.txt is i used chdir to tell ansible what path on my local machine the requirements.txt is located , in the second instance i pass the name of package i wish to install. you might want to install dependencies from a requirements.txt if you have specific version of your dependencies check out the ansible pip docs for more info.Now use ansible-playbook playbook.yml. Your output should look something like this

ok: [127.0.0.1]

TASK [install dependencies requirements.txt] ***********************************
changed: [127.0.0.1]

TASK [additional dependency] ***************************************************
ok: [127.0.0.1]

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

PLAY RECAP is a summary of the tasks that where executed, ok tells you the number of tasks that ran successfully , changed shows you how many tasks changed the state of the target machine in this case 1 because i have some of the dependencies installed. The rest are pretty self explanatory so i would not dive into them.

If you would like to take a look at the source files for this post you can find them in this repository

If you're wondering where to go from here check out this post, that goes into more detail with Ansible tips and tricks

In a future post i would show you how to deploy a simple web app using ansible, if you have any questions feel free to reach me on twitter or create an issue in the repo.

How to setup an ssh server within a docker container

s1ntaxe770r — Tue, 26 May 2020 05:09:13 +0000

In this post I will walk you through my process of setting up ssh access to your docker container.

Why run an ssh server within a container in the first place?

The major reason why you might want to do this is for testing purposes, perhaps you are testing infrastructure automation or provisioning with something like ansible which requires ssh access to the target machine, you'd want to test this in a safe environment before going live.

This article assumes you have docker installed on your machine if not you can refer to this page to get it installed here

The Dockerfile!


FROM ubuntu:latest

RUN apt update && apt install  openssh-server sudo -y

RUN useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1000 test 

RUN  echo 'test:test' | chpasswd

RUN service ssh start

EXPOSE 22

CMD ["/usr/sbin/sshd","-D"]

Here I am using ubuntu as the base image for the container, then on line 2 i install open-ssh server and sudo.

Sudo?

By default docker does not have sudo installed , hence the need to install it along with the open ssh server

On line 3 i create a user called test and add it to the sudo group

echo 'test:test' | chpasswd sets the password for the user test to test

Line 5 starts the ssh service and line 6 tells docker the container listens on port 22 ( which is the default for ssh) and finally i start the ssh daemon.

Building the image

To build the image run docker build -t IMAGE_NAME . , once that's done you can run the image using docker run IMAGE_NAME -p 22:22. finally you can connect to the container using the user you created , in this case it will be test so ssh test@ip_address enter your password in the prompt and your all setup

The original Dockerfile can be found on my github here