DEV Community: mkdelta

Deploying to DigitalOcean Kubernetes using Terraform Cloud and GitHub Actions

mkdelta — Tue, 14 Dec 2021 15:30:43 +0000

This is a follow-up to my previous post about my submission to the DigitalOcean Kubernetes Challenge! I recommend that you at least skim through it for context on Kubegres, the Kubernetes operator we'll be using to deploy Postgres.

Disclaimer: This isn't an intro to any of the technologies mentioned! If you haven't used Terraform with GitHub Actions before, I highly suggest going through this tutorial from HashiCorp itself. I'll mostly be riffing off of it, pointing out important departures throughout.

You will need:

A GitHub account and a working Git installation
A Terraform Cloud account and a working Terraform installation
A DigitalOcean account (the process for other providers is very similar, however)

Brief overview

I had recently deployed a scalable Postgres cluster to DigitalOcean Kubernetes, but I did it manually. The process is straightforward but quite tedious, which makes it a prime candidate for automation.

How it works

Infrastructure configuration is pushed to the GitHub repo, triggering a GitHub Actions workflow
GitHub Actions checks out code to a runner
Runner connects to Terraform Cloud to plan and apply the configuration
Terraform Cloud connects to the provider (DigitalOcean in this case) to provision the needed resources

The steps

Set up Terraform Cloud
Set up the GitHub repository
Set up the Terraform file
Push to the repository
Cleanup!

1. Set up Terraform Cloud

1.1. From your DigitalOcean account, create a personal access token.

1.2. From your Terraform Cloud account, create a new workspace, selecting API-driven workflow as its type.

1.3. In your newly created workspace, go to the variables tab and make a new workspace variable called DIGITALOCEAN_TOKEN. Select the env variable type and check the Sensitive box.

1.4. From your Terraform Cloud account, go to the User settings page, select Tokens from the sidebar, and generate a new token. We'll need this for GitHub Actions.

2. Set up a GitHub repository

2.1. Create a new repository. Go to the Settings tab and select Secrets from the sidebar.

2.2. Create a new secret called TF_API_TOKEN and paste the Terraform Cloud token you just generated.

2.3. Navigate to the Actions tab in your repository and find the Terraform template. Click Set up this workflow.

Refer to the Review Actions workflow section in this tutorial for a breakdown of the workflow steps. The template we're using is slightly different in that it doesn't have the update pull request steps.

2.4. Commit the file. The workflow is going to be triggered but it'll quickly error out because we don't have a Terraform file yet!

3. Set up the Terraform file

Click here to see the Terraform file I used. This section of the tutorial is gonna be a breakdown of the file instead of a sequence of steps. For the experts in the audience: I'm new to Terraform so go easy on me! I tried ordering it in a way conducive to explanation.

terraform {
  backend "remote" {
    organization = "your-org-here"

    workspaces {
      name = "your-workspace-name-here"
    }
  }

This part tells Terraform to use Terraform Cloud to plan, apply, etc. instead of doing it locally. This also means the state of your deployment will be stored remotely and securely.

required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "~> 2.16.0"
    }

    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.6.0"
    }

    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.7.0"
    }
  }

Pretty straightforward. The kubectl provider is super useful for elegantly doing kubectl apply to our cluster (we did a lot of that manually last time). We'll see it in action later.

resource "digitalocean_project" "k8s_challenge" {
  name        = "k8s-challenge"
  description = "Entry for the DigitalOcean Kubernetes Challenge"
  purpose     = "Just trying out DigitalOcean"
  environment = "Development"

  resources = [
    digitalocean_kubernetes_cluster.postgres.urn
  ]
}

resource "digitalocean_vpc" "k8s" {
  name   = "k8s-vpc"
  region = "sgp1"

  timeouts {
    delete = "4m"
  }
}

DigitalOcean uses projects to organize resources. We'll put our cluster in a new one and create a new VPC for it. The delete timeout section in the VPC resource makes sure everything else has been deleted before deleting the VPC (it'll throw an error in the destroy process otherwise; I've found that deletions take a few minutes to register).

data "digitalocean_kubernetes_versions" "prefix" {
  version_prefix = "1.21."
}

resource "digitalocean_kubernetes_cluster" "postgres" {
  name         = "postgres"
  region       = "sgp1"
  auto_upgrade = true
  version      = data.digitalocean_kubernetes_versions.prefix.latest_version

  vpc_uuid = digitalocean_vpc.k8s.id

  maintenance_policy {
    start_time = "04:00"
    day        = "sunday"
  }

  node_pool {
    name       = "worker-pool"
    size       = "s-2vcpu-2gb"
    node_count = 3
  }
}

Here we're finally configuring the cluster itself. We're more or less creating a default one. Notice that we're using the id of the VPC we created. The maintenance policy determines when DigitalOcean will install updates and patches.

provider "kubernetes" {
  host  = digitalocean_kubernetes_cluster.postgres.endpoint
  token = digitalocean_kubernetes_cluster.postgres.kube_config[0].token
  cluster_ca_certificate = base64decode(
    digitalocean_kubernetes_cluster.postgres.kube_config[0].cluster_ca_certificate
  )
}

provider "kubectl" {
  host  = digitalocean_kubernetes_cluster.postgres.endpoint
  token = digitalocean_kubernetes_cluster.postgres.kube_config[0].token
  cluster_ca_certificate = base64decode(
    digitalocean_kubernetes_cluster.postgres.kube_config[0].cluster_ca_certificate
  )
  load_config_file = false
}

Here we're configuring our providers to get credentials from the cluster for adding Kubegres resources.

variable "superUserPassword" {}
variable "replicationUserPassword" {}


resource "kubernetes_secret" "postgres_secret" {
  metadata {
    name      = "mypostgres-secret"
    namespace = "default"
  }

  data = {
    superUserPassword       = var.superUserPassword
    replicationUserPassword = var.replicationUserPassword
  }

  type = "Opaque"
}

This is basically the equivalent of the my-postgres-secret.yaml in the Kubegres tutorial.

Short detour: put these secrets in your Terraform Cloud workspace variables!

data "kubectl_path_documents" "docs" {
  pattern = "./manifests/*.yaml"
}

resource "kubectl_manifest" "kubegres" {
  for_each  = toset(data.kubectl_path_documents.docs.documents)
  yaml_body = each.value
}

Here we're telling the kubectl provider to apply all the manifests in our ./manifests/* directory. We're using kubectl_path_documents instead of kubectl_filename_list because the kubegres.yaml file actually consists of multiple documents defining different resources. I got stuck on this the first time around :^)

See the kubectl provider docs for more details.

Short detour: create a manifests directory in your repo and put the required manifests in it! Also check the previous post for context.

4. Push to the repository

4.1. You should be pretty much done! Push everything to the repository. At the minimum, you should have a main.tf file, a manifests directory, and a .github/workflows directory.

4.2. Look at your Actions tab to see the triggered workflow. You should see something like the following.

A lot of configuration is hidden in that Kubegres manifest. Don't panic if the console throws thousands of lines of output at you.

You can also check the ongoing run in your Terraform Cloud account. The terraform apply part takes a few minutes. Grab a cup of your favorite beverage and sit tight!

After a few minutes:

You can also view the cluster in your DigitalOcean control panel.

Clusters also come with a dashboard by default.

5. Cleanup!

5.1 Since we used Terraform Cloud, we can simply queue up a destroy plan! Just go to your workspace Settings and select Destruction and Deletion. Click the red Queue destroy plan and confirm by entering the name of your cluster.

5.2. You should be taken to a new run. Click Confirm & Apply below, add a comment, and click Confirm Plan.

5.3. Wait a few minutes and your cluster should be destroyed! The created DigitalOcean project should also disappear from your control panel shortly.

As you can see, the VPC took some time to get destroyed.

Thank you!

And that's it! I know this tutorial was a bit gisty so feel free to ask questions and ask for debugging help. Thanks to DigitalOcean for organizing the challenge! The repo can be found here.

Deploying a scalable PostgreSQL cluster on DigitalOcean Kubernetes using Kubegres

mkdelta — Sat, 11 Dec 2021 09:38:09 +0000

This was done for the DigitalOcean Kubernetes Challenge!

Disclaimer: This isn't an intro to Kubernetes! Some working knowledge is required, but honestly not a lot. It's also assumed that you have kubectl installed already.

Brief overview

When deploying a database on Kubernetes, you have to make it redundant and scalable. You can rely on database management operators like KubeDB or database-specific solutions like Kubegres for PostgreSQL or the MySQL Operator for MySQL.

The challenge prompt says it all. I had barely done anything stateful with Kubernetes before, so I was a bit intimidated. Thankfully, Kubegres was very easy to set up. All you have to do is apply some manifests and it'll handle replication, failover, and backups for you. We really only scratch the surface here, so check out their site for more details. Massive props to the contributors!

The steps

Set up a cluster using the GUI
Connect to your cluster using doctl
Apply Kubegres manifests
Done! Delete some pods ~~for fun~~ to test replica promotion
Cleanup

Simple, right?

1. Set up a cluster using the GUI

I had never used DigitalOcean before, so I opted to use the GUI the first time around.

1.1. In your control panel, click the Create button at the top of the screen and select Kubernetes.

1.2. Customize your cluster then click Create Cluster at the bottom of the page. As for me, I just left everything on default :D
That also means the cluster will be associated with my default project (every user has one) and made in the default VPC of the default region.

1.3. You should be taken to your cluster's page. Provisioning should be done in a few minutes.

2. Connect to your cluster using doctl

Once the cluster is up, the Getting Started section on your cluster's page should, well, get you started!

2.1. Install and configure doctl, DigitalOcean's CLI, if you haven't yet. Here's a guide. You'll have to create a personal access token as well.

2.2. Once you've verified that doctl is working, run the command given on your cluster's Getting Started section to connect to your cluster.

You should get something like:

Notice: Adding cluster credentials to kubeconfig file found in "/your/kube/config/path"
Notice: Setting current-context to <your-cluster-id>

2.3. Confirm that you have access to your cluster.

$ kubectl get nodes

NAME                   STATUS   ROLES    AGE   VERSION
my-node-pool-asdfjkl   Ready    <none>   10m   v1.21.5   
my-node-pool-asdfjkm   Ready    <none>   10m   v1.21.5
my-node-pool-asdfjkn   Ready    <none>   10m   v1.21.5

3. Apply Kubegres manifests

Here, the Kubegres getting started guide takes over. Still very straightforward. I'll give the condensed version.

3.1. Install the Kubegres operator and check its components in the created namespace.

$ kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.14/kubegres.yaml

$ kubectl get all -n kubegres-system

3.2. Check the storage class. If you did everything correctly, DigitalOcean Block Storage should be the default for your cluster.

$ kubectl get sc

NAME                         PROVISIONER                 ...
do-block-storage (default)   dobs.csi.digitalocean.com   ...

3.3. Create a file for your Postgres superuser and replication user credentials.

$ vi my-postgres-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: mypostgres-secret
  namespace: default
type: Opaque
stringData:
  superUserPassword: postgresSuperUserPsw
  replicationUserPassword: postgresReplicaPsw

The keys and values under stringData are arbitrary, but we'll use these for now.

3.4. Create the secret in your cluster

$ kubectl apply -f my-postgres-secret.yaml

3.5. Create a file for your Kubegres resource:

$ vi my-postgres.yaml

apiVersion: kubegres.reactive-tech.io/v1
kind: Kubegres
metadata:
  name: mypostgres
  namespace: default
spec:
   replicas: 3
   image: postgres:14.1
   database:
      size: 1Gi
   env:
      - name: POSTGRES_PASSWORD
        valueFrom:
           secretKeyRef:
              name: mypostgres-secret
              key: superUserPassword
      - name: POSTGRES_REPLICATION_PASSWORD
        valueFrom:
           secretKeyRef:
              name: mypostgres-secret
              key: replicationUserPassword

The Kubegres tutorial uses 200Mi for the database size but the minimum block storage size provisioned by DigitalOcean is 1Gi. This bit me the first time around :^)

3.6. Apply using

$ kubectl apply -f my-postgres.yaml

and watch K8s spin up the pods with

$ kubectl get pods -o wide -w

3.7. If it seems like something went wrong, run

$ kubectl get events

to get more information.

4. Done!

4.1. Check all the created resources with

$ kubectl get pod,statefulset,svc,configmap,pv,pvc -o wide

You should get something like:

NAME                 READY   STATUS    NODE
pod/mypostgres-1-0   1/1     Running   worker1
pod/mypostgres-2-0   1/1     Running   worker2
pod/mypostgres-3-0   1/1     Running   worker3

NAME                            READY
statefulset.apps/mypostgres-1   1/1
statefulset.apps/mypostgres-2   1/1
statefulset.apps/mypostgres-3   1/1

NAME                         TYPE
service/mypostgres           ClusterIP
service/mypostgres-replica   ClusterIP

NAME
configmap/base-kubegres-config

NAME                          CAPACITY
persistentvolume/pvc-838...   1Gi
persistentvolume/pvc-da6...   1Gi
persistentvolume/pvc-e25...   1Gi

NAME                                               CAPACITY
persistentvolumeclaim/postgres-db-mypostgres-1-0   1Gi
persistentvolumeclaim/postgres-db-mypostgres-2-0   1Gi
persistentvolumeclaim/postgres-db-mypostgres-3-0   1Gi

4.2. Check the dashboard! Go to your cluster's page and click Kubernetes Dashboard to the right of the cluster's name. It's pretty comprehensive!

Look at your pods by clicking Pods in the sidebar, under Workloads. Find the primary Postgres pod using the labels on each pod's page.

4.3. Back to the terminal! Check which pod is primary with

$ kubectl get pods --selector replicationRole=primary

NAME             READY   STATUS    RESTARTS   AGE
mypostgres-1-0   1/1     Running   0          3m21s

Check which ones are replicas with

$ kubectl get pods --selector replicationRole=replica

NAME             READY   STATUS    RESTARTS   AGE
mypostgres-2-0   1/1     Running   0          2m59s
mypostgres-3-0   1/1     Running   0          2m19s

4.4. Delete the primary pod with

$ kubectl delete pod <pod-name>

and watch as a replica gets promoted with

$ kubectl get pods -w --selector replicationRole=primary

Blink and you'll miss it!

4.5. Recheck the primary and replica pods with commands from previous steps.

$ kubectl get pods --selector replicationRole=primary

NAME             READY   STATUS    RESTARTS   AGE
mypostgres-2-0   1/1     Running   0          51s

$ kubectl get pods --selector replicationRole=replica

NAME             READY   STATUS    RESTARTS   AGE
mypostgres-3-0   1/1     Running   0          4m21s
mypostgres-4-0   1/1     Running   0          34s

The king is dead, long live the king!

5. Cleanup

We wouldn't want to rack up a huge bill, would we?

5.1. Go to your cluster's page and select the Settings tab.

5.2. Scroll down to the very bottom and click the red Destroy button.

5.3. You'll be asked if you want to destroy the persistent volumes provisioned along with your cluster. Select Destroy All, enter the cluster's name, and click the red Destroy button.

Thank you!

And that's it! Thank you for reading my first post on dev.to :) And thank you to the folks at DigitalOcean for organizing the challenge and inspiring me to get off my butt!

In my next post I'll demonstrate how to do the same deployment using Terraform and GitHub Actions! Check out the repo in the meantime.