DEV Community: Philipp Strube

Goodbye Cloud, Hello CLI: Sunsetting Kubestack Cloud

Philipp Strube — Tue, 09 May 2023 19:53:11 +0000

I've recently released a major update for Kubestack, the Terraform framework for Kubernetes platform engineering teams. This update moves all functionality previously provided by Kubestack Cloud into the kbst CLI.

I decided to make this change, because Kubestack Cloud was only able to provide a better developer experience on day one. But, once exported to Terraform, the UI was not helpful any more on day two and all following days.

But my goal is to improve the developer experience and day-to-day lives of platform engineering teams at all times. This latest kbst release is a major step towards achieving this goal.

If this is the first time you hear about Kubestack Cloud: Kubestack Cloud was a browser based UI and allowed users to design a Kubernetes platform by following a step-by-step wizard and then exporting and downloading the designed platform's Terraform code.

However, the disconnect between the UI and the code in the repository on a developer's local machine, diminished the value of Kubestack Cloud on day two and beyond. To address this issue, I moved all this functionality into the kbst CLI, where access to the local code is easier.

The kbst CLI, which previously also only scaffolded new repositories, now has CRUD (create, read, update, delete) functionality for clusters, node-pools, and services. This means users can use the CLI to scaffold Terraform code to add or remove clusters, node-pools, or services inside their existing Kubestack repositories.

If you want to see the new CLI in action, give the updated tutorial a try or read the documentation on adding and removing cluster modules, node pool modules or service modules.

But if you'd like to learn more about how this works under the hood, keep reading.

How this works

If you're already familiar with Kubestack, you know that Kubestack repositories follow a convention-over-configuration approach to define the clusters, node pools, and services that make up a Kubernetes platform in a single Terraform codebase. At the root of each repository, there are several .tf files that follow a specific naming convention. These files contain module calls that define each platform component.

To add or remove components, or update the versions of existing component modules, the kbst CLI parses the necessary subset of Terraform code to understand the components of the platform. You can list the Kubestack component modules it discovered using the kbst list command. By appending --all to the list command, you can also see any non-Kubestack modules.

You can add node pools or services to existing clusters or add more clusters from the same or even a different cloud provider. The CLI will scaffold the additional required .tf files and update the Dockerfile's FROM line to specify the correct image, in case of changing from a single to a multi-cloud environment or vice versa. Likewise, it will also remove module calls and the respective .tf files if you remove a service, a node pool or even a cluster from your platform.

But don't worry, the kbst CLI only changes local files and does never change any cloud or Kubernetes resources.

You can use it to avoid writing repetitive boilerplate code or manually deleting module calls and Terraform files, while still owning your codebase and retaining the ability to extend or modify the code to meet specific needs.

Once you're happy with the code, you can follow the Kubestack GitOps workflow to peer-review, validate, and promote changes to your platform's environments as usual.

In conclusion, the shift from Kubestack Cloud to the kbst CLI provides a better developer experience not only on day one, but also on day two and makes it easier for platform engineering teams to manage their Kubernetes based platforms.

What happened to the platforms I designed with Kubestack Cloud?

If you have previously designed a platform with Kubestack Cloud, you can sign in with your existing user and will see instructions how to scaffold your existing platforms using the new CLI.

Here's an example screenshot of how that will look like.

Getting rigorous about investing in the Kubestack project

Philipp Strube — Sun, 27 Nov 2022 14:52:30 +0000

Sometimes when you spend a long time solving a problem, it makes it harder to see your solution clearly.

In 12 years helping companies adopt modern cloud computing, I saw so many of the same snags repeating across multiple organizations. From these lessons, I built Kubestack as guardrails to make it easier to avoid the pain in the first place. Kubestack has been used in companies large and small for years, but I haven’t always known where it has been most helpful to its users. Without knowing this, it’s hard to bring more people in as both users and contributors. So earlier this year, I contracted with Ana Hevesi to support Kubestack's open source efforts.

Ana operates a developer experience consultancy. After working in technical community building for companies like Stack Overflow and Nodejitsu, Ana now works with devtools founders to create evidence-based approaches for growing their ecosystems.

We started by doing some research into how Kubestack serves your goals.

Research methods

Our objective was to learn about users’ career trajectories and aspirations, and get a clear picture of what role Kubestack plays in your success.

Ana recommended we aim for 5 interviews, citing it as a good “goldilocks zone” for initial quantity of data to work with. I then reached out to a spectrum of new and long-tenured Kubestack users to ask for their time in a 60 minute user interview.

Ana wrote a standard interview script which included bandwidth for conversational “side quests.” After, Ana analyzed recordings, picked out repeat themes, and came to me with conclusions and recommendations.

Areas of positive impact

Kubestack helps careers

Participants attributed their use of Kubestack to positive career outcomes, such as developing a reputation for reliably delivering for users, or scaling on a tight timeframe with limited prior experience. Others reported it was a key learning tool when they were just starting as platform engineers.

Works so well it disappears

The most consistent feedback we received was that users can assume Kubestack is just going to work. Multiple participants had been relying on the framework for many months without needing to give it a second thought.

Areas to improve

Documentation for advanced features needs improvement

Kubestack works great for months on end for most orgs setting up their first K8s cluster, but those who wished to modify Kubestack outside of existing use cases told us error messages and upgrade processes were opaque.

Backwards compatibility and multi-cloud support presents friction to open source contributions

Adding new features requires working knowledge of both Terraform and cloud provider functionality across historical versions, and at times, their interactions with one another. Furthermore, while Kubestack is committed to supporting EKS, AKS, and GKE, a contributor may wish to implement functionality for only one of these cloud providers. Inviting more PRs from a wider array of contributors necessitates a plan for tiered support of legacy versions or defining contributor scope to accommodate this complexity.

How we’re applying these findings

Connecting with the people who need us most

Kubestack makes a huge impact on early-stage teams and emerging professionals. We’re exploring ways to better tailor our communication and outreach to make sure they know about the opportunities this framework provides, improving both adoption and contributions to the project. Kubestack only succeeds because you succeed.

Benefits before features

The current iteration of Kubestack’s landing page assumes a fairly high level of existing knowledge of the platform engineering space. As such, an upcoming iteration of the Kubestack site will aim to engage folks who aren’t already deep in the jargon and progressively bring them into the fold, while still being legible to seasoned professionals.

Open source participation onramps

Since enabling users to learn from each other and communicating where the project is going is an important part of growing an open source community, we’ll be experimenting with office hours and public communication about recent releases. We’ll have scheduling details coming soon.

Leveling up, together

I created Kubestack so that folks coming to Kubernetes for the first time could take immediate advantage of the separation of concerns that containers provide. User research says that this works as intended!

Now comes the iterative task of communicating my own knowledge and experience in ways that make it easier to build together, while learning from your use of the project to fill in its gaps. Ultimately, the intent is a healthy community where we’re all working together to make the project better serve your needs.

Finally, a big thank you to Tomas, AJ, Brendan, Christoph, and Mark for your time and candor. Kubestack is better for it.

A Better Way to Provision Kubernetes Resources Using Terraform

Philipp Strube — Wed, 04 May 2022 18:01:47 +0000

Terraform is immensely powerful when it comes to defining and maintaining infrastructure as code. In combination with a declarative API, like a cloud provider API, it can determine, preview, and apply changes to the codified infrastructure.

Consequently, it is common for teams to use Terraform to define the infrastructure of their Kubernetes clusters. And as a platform to build platforms, Kubernetes commonly requires a number of additional services before workloads can be deployed. Think of ingress controllers or logging and monitoring agents and so on. But despite Kubernetes' own declarative API, and the obvious benefits of maintaining a cluster's infrastructure and services from the same infrastructure as code repository, Terraform is far from the first choice to provision Kubernetes resources.

With Kubestack, the open-source Terraform framework I maintain, I'm on a mission to provide the best developer experience for teams working with Terraform and Kubernetes. And unified provisioning of all platform components, from cluster infrastructure to cluster services, is something I consider crucial in my relentless pursuit of said developer experience.

Because of that, the two common approaches to provision Kubernetes resources using Terraform never really appealed to me.

On the one hand, there's the Kubernetes provider. And while it integrates Kubernetes resources into Terraform, maintaining the Kubernetes resources in HCL is a lot of effort. Especially for Kubernetes YAML you consume from upstream. On the other hand, there are the Helm provider and the Kubectl provider. These two use native YAML instead of HCL, but do not integrate the Kubernetes resources into the Terraform state and, as a consequence, lifecycle.

I believe my Kustomization provider based modules are a better alternative because of three distinct benefits:

Like Kustomize, the upstream YAML is left untouched, meaning upstream updates require minimal maintenance effort.
By defining the Kustomize overlay in HCL, all Kubernetes resources are fully customizable using values from Terraform.
Each Kubernetes resource is tracked individually in Terraform state, so diffs and plans show the changes to the actual Kubernetes resources.

To make these benefits less abstract, let's compare my Nginx ingress module with one using the Helm provider to provision Nginx ingress.

The Terraform configuration for both examples is available in this repository. Let's take a look at the Helm module first.

The Helm-based module

Usage of the module is straightforward. First, configure the Kubernetes and Helm providers.

provider "kubernetes" {
  config_path    = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

Then define a kubernetes_namespace and call the release/helm module.

resource "kubernetes_namespace" "nginx_ingress" {
  metadata {
    name = "ingress-nginx"
  }
}

module "nginx_ingress" {
  source  = "terraform-module/release/helm"
  version = "2.7.0"

  namespace  = kubernetes_namespace.nginx_ingress.metadata.0.name
  repository = "https://kubernetes.github.io/ingress-nginx"

  app = {
    name          = "ingress-nginx"
    version       = "4.1.0"
    chart         = "ingress-nginx"
    force_update  = true
    wait          = false
    recreate_pods = false
    deploy        = 1
  }

  set = [
    {
      name  = "replicaCount"
      value = 2
    }
  ]
}

If you now run a terraform plan for this configuration, you see the resources to be created.

Terraform will perform the following actions:

  # kubernetes_namespace.nginx_ingress will be created
  + resource "kubernetes_namespace" "nginx_ingress" {
      + id = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + name             = "ingress-nginx"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.nginx_ingress.helm_release.this[0] will be created
  + resource "helm_release" "this" {
      + atomic                     = false
      + chart                      = "ingress-nginx"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = true
      + id                         = (known after apply)
      + lint                       = true
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "ingress-nginx"
      + namespace                  = "ingress-nginx"
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://kubernetes.github.io/ingress-nginx"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = []
      + verify                     = false
      + version                    = "4.1.0"
      + wait                       = false
      + wait_for_jobs              = false

      + set {
          + name  = "replicaCount"
          + value = "2"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

And this is the key issue with how Helm is integrated into the Terraform workflow. The plan does not tell you what Kubernetes resources will be created for the Nginx ingress controller. And neither are the Kubernetes resources tracked in Terraform state, as shown by the apply output.

kubernetes_namespace.nginx_ingress: Creating...
kubernetes_namespace.nginx_ingress: Creation complete after 0s [id=ingress-nginx]
module.nginx_ingress.helm_release.this[0]: Creating...
module.nginx_ingress.helm_release.this[0]: Creation complete after 3s [id=ingress-nginx]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Similarly, if planning a change, there's again no way to tell what the changes to the Kubernetes resources will be.

So if you increase the replicaCount value of the Helm chart, the terraform plan will merely show the change to the helm_release resource.

set = [
  {
    name = "replicaCount"
    value = 3
  }
]

What will the changes to the Kubernetes resources be? And more importantly, is it a simple in-place update, or does it require a destroy-and-recreate? Looking at the plan, you have no way of knowing.

Terraform will perform the following actions:

  # module.nginx_ingress.helm_release.this[0] will be updated in-place
  ~ resource "helm_release" "this" {
        id                         = "ingress-nginx"
        name                       = "ingress-nginx"
        # (27 unchanged attributes hidden)

      - set {
          - name  = "replicaCount" -> null
          - value = "2" -> null
        }
      + set {
          + name  = "replicaCount"
          + value = "3"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

The Kustomize-based module

Now, let's take a look at the same steps for the Kustomize-based module. Usage is similar. First require the kbst/kustomization provider and configure it.

terraform {
  required_providers {
    kustomization = {
      source = "kbst/kustomization"
    }
  }
}

provider "kustomization" {
  kubeconfig_path = "~/.kube/config"
}

Then call the nginx/kustomization module.

module "nginx_ingress" {
  source  = "kbst.xyz/catalog/nginx/kustomization"
  version = "1.1.3-kbst.1"

  configuration_base_key = "default"
  configuration = {
    default = {
      replicas = [{
        name  = "ingress-nginx-controller"
        count = 2
      }]
    }
  }
}

Unlike for the Helm-based module though, when you run terraform plan now you will see each Kubernetes resource and its actual configuration individually. To keep this blog post palatable, I show the details for the namespace only.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"] will be created
  + resource "kustomization_resource" "p0" {
      + id       = (known after apply)
      + manifest = jsonencode(
            {
              + apiVersion = "v1"
              + kind       = "Namespace"
              + metadata   = {
                  + annotations = {
                      + "app.kubernetes.io/version"      = "v0.46.0"
                      + "catalog.kubestack.com/heritage" = "kubestack.com/catalog/nginx"
                      + "catalog.kubestack.com/variant"  = "base"
                    }
                  + labels      = {
                      + "app.kubernetes.io/component"  = "ingress-controller"
                      + "app.kubernetes.io/instance"   = "ingress-nginx"
                      + "app.kubernetes.io/managed-by" = "kubestack"
                      + "app.kubernetes.io/name"       = "nginx"
                    }
                  + name        = "ingress-nginx"
                }
            }
        )
    }

  # module.nginx_ingress.kustomization_resource.p1["_/ConfigMap/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/Service/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/Service/ingress-nginx/ingress-nginx-controller-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-create"] will be created
  # module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-patch"] will be created
  # module.nginx_ingress.kustomization_resource.p1["networking.k8s.io/IngressClass/_/nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRole/_/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRole/_/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRoleBinding/_/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRoleBinding/_/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/Role/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/Role/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"] will be created

Plan: 19 to add, 0 to change, 0 to destroy.

Applying, again, has all the individual Kubernetes resources. And because the modules use explicit depends_on to handle namespaces and CRDs first and webhooks last, resources are reliably applied in the correct order.

module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"]: Creating...
module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"]: Creation complete after 0s [id=369e8643-ad33-4eb4-95dc-f506cef4a198]
module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx"]: Creating...
module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-create"]: Creating...

...

module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-patch"]: Creation complete after 1s [id=58346878-70bd-42f2-af61-2730e3435ca7]
module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx"]: Creation complete after 0s [id=f009bbb7-7d2e-4f28-a826-ce133c91cc15]
module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"]: Creating...
module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"]: Creation complete after 1s [id=3185b09f-1f67-4079-b44f-de01bff44bd2]

Apply complete! Resources: 19 added, 0 changed, 0 destroyed.

Naturally, it also means that if you increase the replica count like this...

replicas = [{
  name = "ingress-nginx-controller"
  count = 3
}]

...the terraform plan shows which Kubernetes resources will change and what the diff is.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] will be updated in-place
  ~ resource "kustomization_resource" "p1" {
        id       = "81e8ff18-6c6c-440d-bd8b-bf5f0d016953"
      ~ manifest = jsonencode(
          ~ {
              ~ spec       = {
                  ~ replicas             = 2 -> 3
                    # (4 unchanged elements hidden)
                }
                # (3 unchanged elements hidden)
            }
        )
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Maybe more importantly even, the Kustomization provider will also correctly show if a resource can be changed using an in-place update. Or if a destroy-and-recreate is required because there is a change to an immutable field, for example.

This is the result of two things:

That, as you've just seen, every Kubernetes resource is handled individually in Terraform state, and
that the Kustomization provider uses Kubernetes' server-side dry-runs to determine the diff of each resource.

Based on the result of that dry-run, the provider instructs Terraform to create an in-place or a destroy-and-recreate plan.

So, as an example of such a change, imagine you need to change spec.selector.matchLabels. Since matchLabels is an immutable field, you will see a plan that states that the Deployment resource must be replaced. And you will see 1 to add and 1 to destroy in the plan's summary.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] must be replaced
-/+ resource "kustomization_resource" "p1" {
      ~ id       = "81e8ff18-6c6c-440d-bd8b-bf5f0d016953" -> (known after apply)
      ~ manifest = jsonencode(
          ~ {
              ~ metadata   = {
                  ~ labels      = {
                      + example-selector               = "example"
                        # (6 unchanged elements hidden)
                    }
                    name        = "ingress-nginx-controller"
                    # (2 unchanged elements hidden)
                }
              ~ spec       = {
                  ~ replicas             = 2 -> 3
                  ~ selector             = {
                      ~ matchLabels = {
                          + example-selector               = "example"
                            # (4 unchanged elements hidden)
                        }
                    }
                  ~ template             = {
                      ~ metadata = {
                          ~ labels      = {
                              + example-selector               = "example"
                                # (4 unchanged elements hidden)
                            }
                            # (1 unchanged element hidden)
                        }
                        # (1 unchanged element hidden)
                    }
                    # (2 unchanged elements hidden)
                }
                # (2 unchanged elements hidden)
            } # forces replacement
        )
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Try it yourself

You can find the source code for the comparison on GitHub if you want to experiment with the differences yourself.

If you want to try the Kustomize modules yourself, you can either use one of the modules from the catalog that bundle upstream YAML, like the Prometheus operator, Cert-Manager, Sealed secrets, or Tekton, for example.

But this doesn't only work for upstream services. There is also a module that can be used to provision any Kubernetes YAML in the exact same way as the catalog modules - called the custom manifest module.

Get involved

Currently, the number of services available from the catalog is still limited.

If you want to get involved, you can also find the catalog source on GitHub.

Photo by Vladislav Babienko on Unsplash

Google Anthos with Terraform and Kubestack

Philipp Strube — Fri, 02 Jul 2021 12:49:15 +0000

For a project, I'm currently evaluating Google Anthos. Since the client is a multi-national company, the idea is to build a multi-region and multi-cloud Kubernetes platform. But the sensible kind, no need to bring the pitchforks. So different applications each in one region and cloud. Not the mythical one application in multiple regions and clouds where complexity and necessity are often entirely disproportionate to each other. But that's not really the point here.

Google sales is pushing Anthos hard. And the client's team is open to the argument that an opinionated stack might help them move faster compared with having to first evaluate various alternatives for each part of the stack and then building the know-how to productize this custom stack. It's a fair argument to make.

Long story short, we're now evaluating Anthos with GKE and EKS clusters connected to it, because some application teams are drawn to AWS and some are drawn to Google Cloud with their respective workloads. Individual reasons for this are pretty diverse, ranging from data stickiness like terabytes of data already in S3 to quota/capacity limits of specific types of GPUs or preferring certain managed services from one provider over the other cloud provider's alternative.

I tend to agree with this kind of multi-cloud strategy making a lot of sense. Yes, individual apps may still end up locked-in to one vendor. But at least it's not all eggs in one basket which has real benefits both on blast radius and pricing negotiations, if you're big enough.

I've been working on this evaluation for a couple of days now and thought I'd share my experience because I couldn't find a lot of hands-on reports about Anthos with Terraform. Most content seemed primarily hypothetical and it seemed like most writers hadn't actually gotten their hands properly dirty before writing about it. I already washed mine to calm down and make sure this doesn't end up in some obnoxious rant.

The first thing, that really surprised me about Anthos though, is that Anthos does not provision the Kubernetes clusters for you. I totally expected it would. Instead, you have to provision the clusters and then connect them to the Anthos hub. Which basically requires some IAM setup, an Anthos hub membership and running an agent inside each cluster.

Anthos leaves it up to you to provision clusters any way you like. But since I'm part of the project, it may not come as a surprise that in our case, infrastructure as code and Terraform are the attack plan.

Now, Google does even provide its own Anthos Terraform modules. But these are only for GKE, meaning for EKS we'd need to use modules from another source. Leaving us to deal with different module usage and update schedules.

But more importantly, Google's Terraform modules constantly shell out to kubectl or gcloud CLIs. Which I consider a last resort that should be avoided at any cost for Terraform modules, because long term maintainability. Frankly, calling CLI commands like this has no place in declarative infrastructure as far as I'm concerned.

Unsurprisingly, my biased proposal is to use Kubestack to provision the GKE and EKS clusters leveraging the Kubestack framework's unified GKE and EKS modules, and to write a custom module to connect the resulting clusters to Anthos. The bespoke module would integrate the IAM, Anthos and Kubernetes resources required fully into the Terraform state and lifecycle instead of calling kubectl and gcloud like the official Google modules do.

Below is the current work in progress state of the experimental module and some of the challenges I hit so far.

The first requirement is an IAM identity and role for the agent inside the cluster. For GKE clusters, workload identities can be used but for non GKE clusters, EKS in our case, it seems shared credentials in the form of a service account key are the only option. Creating google_service_account, google_project_iam_member and google_service_account_key resources is easy enough. I'm sure this is overly simplified and I may have to add more roles as my evaluation continues.

resource "google_service_account" "current" {
  project = local.project_id
  account_id   = local.cluster_name
  display_name = "${local.cluster_name} gke-connect agent"
}

resource "google_project_iam_member" "current" {
  project = local.project_id
  role    = "roles/gkehub.connect"
  member  = "serviceAccount:${google_service_account.current.email}"
}

resource "google_service_account_key" "current" {
  service_account_id = google_service_account.current.name
}

The next step is to register the cluster as a member of the Anthos hub. Which means adding a google_gke_hub_membership resource.

resource "google_gke_hub_membership" "current" {
  provider = google-beta

  project = local.project_id
  membership_id = local.cluster_name
  description = "${local.cluster_name} hub membership"
}

Finally, the agent needs to be provisioned inside the cluster and set up to use the service account as its identity.

By default, joining the cluster to the hub and provisioning the Kubernetes resources of the agent on the cluster is done via the gcloud beta container hub memberships register CLI command. But the command has a --manifest-output-file parameter, that allows writing the Kubernetes resources to a file instead of applying it to the cluster directly.

To not also have to fall back to calling the register gcloud command from Terraform, I opted to write the manifests to a YAML file and use them as the base that I patch in a kustomization_overlay using my Kustomization provider.

This way, I will have each individual Kubernetes resource of the Anthos agent to be provisioned and tracked using Terraform. While at the same time being able to use the attributes from my service account and service account key resources to configure the agent.

data "kustomization_overlay" "current" {
  namespace = "gke-connect"

  resources = [
      "${path.module}/upstream_manifest/anthos.yaml"
  ]

  secret_generator {
    name = "creds-gcp"
    type = "replace"
    literals = [
      "creds-gcp.json=${base64decode(google_service_account_key.current.private_key)}"
    ]
  }

  patches {
    # this breaks if the order of env vars in the upstream YAML changes
    patch = <<-EOF
      - op: replace
        path: /spec/template/spec/containers/0/env/6/value
        value: "//gkehub.googleapis.com/projects/xxxxxxxxxxxx/locations/global/memberships/${local.cluster_name}"
    EOF

    target = {
      group = "apps"
      version = "v1"
      kind = "Deployment"
      name = "gke-connect-agent-20210514-00-00"
      namespace = "gke-connect"
    }
  }
}

The manifests gcloud writes to disk can't be committed to version control because they include a Kubernetes secret with the plaintext service account key embedded. The key file is unfortunately a required parameter of the hub memberships register command. So I had to delete this secret from the YAML file. And have to remember doing this whenever I rerun the command to update my base with the latest upstream manifests.

In the kustomization_overlay data source, I then use a secret_generator to create a Kubernetes secret using the private key from the google_service_account_key resource.

Additionally, the agent has a number of environment variables set. The URL to the hub memberships resource is one of them and needs to be patched with the respective cluster name. Unfortunately, the environment variables are set directly in the pod template. So the patch will break if the number or order of environment variables changes. It would be better to change this to envFrom and set the environment variables dynamically in the overlay using a config_map_generator. But the downside of this is, again, that there's one more modification to the upstream YAML which has to be repeated every time it is updated.

While we're on the topic of updates. One thing that makes me suspicious is that the generated YAML has a date as part of its resource names. E.g. gke-connect-agent-20210514-00-00. Call me a pessimist, but I totally expect this to become a problem with updates in the future.

Ignoring that for now. Next on my evaluation was to apply my Terraform configuration and hopefully have my clusters connected to Anthos.

Unfortunately, on the first try, that wasn't quite the case. The clusters did show up in the Anthos UI. But had a big red unreachable warning. As it turned out, this was due to the agent pod crash looping with a permission denied error.

kubectl -n gke-connect logs gke-connect-agent-20210514-00-00-66d94cff9d-tzw5t
2021/07/02 11:40:33.277997 connect_agent.go:17: GKE Connect Agent. Log timestamps in UTC.
2021/07/02 11:40:33.298969 connect_agent.go:21: error creating tunnel: unable to retrieve namespace "kube-system" to be used as externalID: namespaces "kube-system" is forbidden: User "system:serviceaccount:gke-connect:connect-agent-sa" cannot get resource "namespaces" in API group "" in the namespace "kube-system"

Which is weird, because from reading the gcloud generated YAML I remember there were plenty of RBAC related resources included. Digging in, it turned out the generated YAML has a Role and RoleBinding. And if you followed carefully, you probably guessed the issue already. Here's the respective part of the generated resources:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    hub.gke.io/project: "ie-gcp-poc"
    version: 20210514-00-00
  name: gke-connect-namespace-getter
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    hub.gke.io/project: "ie-gcp-poc"
    version: 20210514-00-00
  name: gke-connect-namespace-getter
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: gke-connect-namespace-getter
subjects:
- kind: ServiceAccount
  name: connect-agent-sa
  namespace: gke-connect

Unless I'm terribly wrong here, this obviously can't work. Creating a namespaced role and role binding inside the kube-system namespace can not grant permissions to get the kube-system namespace, because namespaces are not namespaced resources.

So I changed the Role to a ClusterRole and the RoleBinding to a ClusterRoleBinding and reapplied my Terraform configuration. And I now have a running agent, that established a tunnel to the Anthos control plane and prints lots of log messages. I have yet to dig into what it actually does there.

With the RBAC fix the generated YAML now already requires three changes to maintain over time. I can't say I'm particularly excited about that. I also wonder if the generated YAML is only broken if the --manifest-output-file parameter is used or if the RBAC configuration is also broken when directly applying the Kubernetes resources to the cluster using the gcloud CLI.

That's it for my evaluation with Google Anthos, Terraform and Kubestack so far. Maybe by sharing my findings, I may safe somebody out there a bit of time in their own evaluation when they hit the same issues.

Next step for me is to look into provisioning the Anthos Service Mesh. It's not quite clear to me yet, if that should be done via Terraform, the fact that Google has this kubectl based Terraform module for that may suggest so. But why wouldn't I do everything after the cluster is connected to Anthos using Anthos Config Management?

What Terraform can learn from PHP

Philipp Strube — Mon, 08 Feb 2021 10:30:47 +0000

TL;DR:

Writing infrastructure as code shows many of the same challenges as writing code for application development, because many of these challenges are not language or use-case specific.
Terraform and its surrounding ecosystem are still evolving and share many similarities with early PHP and the web. Just like PHP evolved by learning from other language ecosystems, Terraform can as well.
Use-case specific frameworks are a major driver of innovation, improved developer experience and productivity on the application development side. But are not yet established parts of the infrastructure as code ecosystem.
The paradigm shift to containers and Kubernetes made use-case specific frameworks possible for infrastructure as code by providing a powerful abstraction between application and infrastructure layer. And the cloud native community is evolving rapidly, extending this abstraction to additional use-cases.
Organizations that adopted application development frameworks for their improved developer experience and productivity, can leverage the same benefits for automating Kubernetes by using an infrastructure as code framework and avoid leaving the cluster the weakest link in their GitOps automation.

Learning from other language ecosystems

PHP’s ease of getting started is widely quoted as the boon and bane of the language. It seems as if making fun of the spaghetti code bases of the early PHP days never gets old. Even in 2021. But there is no doubt that PHP is an extremely successful programming language.

You may ask, what does this have to do with Terraform? Well, hear me out. Terraform and PHP have more in common than you may think. PHP was created when the web was in its infancy and quickly became extremely popular. Don’t forget, PHP is the P in LAMP stack. Similarly, infrastructure as code is still an emerging ecosystem today, and Terraform is by far the most popular language in this ecosystem.

But the modern PHP of today is vastly different from the early PHP we all like to make fun of. And since Terraform today is so similar to where PHP was when it started, there’s a good chance that the Terraform community can learn a lot from how PHP evolved.

Rasmus Lerdorf, the creator of PHP, is famously quoted as never having intended to write a programming language. But PHP got popular and they had to keep going. In addition, the web and its request-response model were new, even to experienced developers. But the endless possibilities of the web got people excited, and the unintentional programming language PHP was easy to get started with. This combination led to the stereotypical poor quality code bases that ended up powering major parts of the early web.

Similarly, infrastructure as code offers huge benefits and gets people excited as well. But it also requires both operations and coding experience, and people coming from either one background have to learn a lot about the respective other, before they can be fully productive.

Languages like Python released a few years before PHP, or Ruby and Java, which were released in the same year as PHP, were intentionally designed programming languages for professional use. While not specific to the web, it is of course possible to build web applications in either one of them. So the self-evident thing was to use these more mature and consistent languages to build web applications, and have more easily maintainable code bases as a result.

And not only were the languages more mature, but so were their ecosystems. The majority of challenges, developers face when writing code, are not language specific. And many are not even use-case specific. You may need different dependencies for building a web application instead of a desktop application for example. But in both cases having dependency management is greatly useful. A feature Python, Ruby and Java all already had.

This led to the creation of frameworks like Django, Ruby on Rails or Spring that made it easy to build web applications in Python, Ruby or Java respectively, leveraging their existing language ecosystems.

A great idea that works in one ecosystem, however is quick to inspire similar development in other languages. And PHP’s wide adoption easily justified major investments to improve the PHP core as well as the surrounding ecosystem. All those teams looking for the best way to maintain their growing PHP code bases were smart to look at other languages and how these same challenges were solved there.

The result are frameworks like Symfony or CakePHP, heavily inspired by Spring and Rails respectively. This is also how Composer brought modern dependency management to PHP. And last but not least, this was when the PHP community adopted Git for version control and slowly moved away from just editing production files directly via FTP.

It’s all about the code

Let's get back to infrastructure as code. Yes, in a lot of ways automating infrastructure is different from application development. But many of the challenges of writing code, that applied across languages and use-cases on the software development side, also apply to infrastructure as code. Code is kind of the keyword here.

So just like PHP learned from other languages, their frameworks and their tooling, Terraform can only benefit from doing so as well.

One area where Hashicorp, the makers of Terraform, recently made major improvements is dependency management. Terraform had the ability to download required providers for quite some time. But it was limited to only Hashicorp’s own providers. Community maintained providers required involving, manual installation. A recent Terraform release introduced support for registry namespaces, which means community providers can now also be installed from the official registry. In addition, required providers and versions can now be specified more explicitly. Even including the ability to vendor providers, and thereby hardening automation runs against failing when the registry is unavailable.

The missing piece

All the language ecosystems we discussed share one key piece that heavily improves the developer experience, but which isn’t a thing yet in the infrastructure as code world. I’m referring to frameworks of course. And concretely use-case specific frameworks. By being use-case specific, the aforementioned software development frameworks drastically reduce upfront and maintenance effort, and provide the best developer experience and workflow possible.

If I’m building a cloud native application in Java, using Spring Boot will make my life much easier. Likewise, if my goal is to build a Jamstack website, a framework like Gatsby will get me there much faster.

But the reason why frameworks are not a thing in the infrastructure as code world yet is not merely that the ecosystem is still evolving. For frameworks to be useful, we also required a strong abstraction layer that kept the infrastructure layer clear from application specific requirements. Containers and Kubernetes are extremely popular because they provide this very abstraction. And this means two things: First, that with using Terraform to manage Kubernetes there is a popular and very specific use-case for an infrastructure as code framework. And second, that because of the powerful abstraction, such a framework makes sense for the first time.

Kubestack is this use-case specific, Terraform GitOps framework. If you’re building GitOps automation for Kubernetes cluster infrastructure and cluster services using Terraform, Kubestack may be the framework for you. Think of Kubestack as the Ruby on Rails of infrastructure automation, the Gatsby of GitOps, or the Spring Boot of Terraform and Kubernetes.

And just like application frameworks copied ideas that worked well from one language to another, Kubestack does the same from application development to infrastructure as code.

Talent borrows, genius steals

One example is Kubestack’s convention over configuration based repository layout. Another one is its inheritance based configuration to prevent drift between environments. A third one is the ability to easily vendor dependencies in the repository, like the Nginx ingress controller or Prometheus monitoring operator. Or, as the last but not the least example, local development environments that automatically update as you make changes to the code.

Slow feedback loops are poison for developer productivity. And infrastructure as code is notoriously known for mandatory, slow pipeline runs. This makes the local development environment the perfect example how Kubestack drastically improves the developer experience, because it’s a use-case specific framework.

The strong abstraction between the application and infrastructure layers is a key mantra of what we know as cloud native. And if you take a look at recent developments from the cloud native community the direction is clear. As more and more organizations shift their workloads and use-cases to cloud native, we continue to see new innovation and iterative improvements that extend this powerful abstraction.

This is both positive for the future of infrastructure as code and Terraform as well as for use-case specific infrastructure as code frameworks.

Terraform loves cloud native

Systems that provide a separation between declaring desired state and current state are the current state-of-the-art. This is a core principle of Kubernetes and high-level managed cloud services, but also of VM auto-scaling groups, as a lower level example of this principle. On the surface there’s an API to declare the desired state. And behind the API are control loops that keep the current state in sync with the desired state.

Terraform shines when being combined with such a system, because it is great at planning and applying changes triggered by a commit in a repository. And it can also be run periodically, to detect drift and either alert or overwrite. But when operating distributed systems, there are various failure scenarios where continuously running controllers, that can take immediate action based on more events than just code changes, are clearly superior. The important thing to understand here is, Terraform is great to provide a way for teams to reason about proposed changes and keeping the committed state and desired state in sync. But keeping desired and current state in sync is, in most cases, better left to a continuously running control loop.

It’s common for teams to hit this limitation when using infrastructure as code to automate legacy systems that don’t provide this separation of concerns. And this frequently leads to automation that only manages the lifecycle partially and causes complex issues for teams to coordinate automation and manual operations. Facing this significantly limits the value of infrastructure as code, and many teams justifiably may hold back on adopting Terraform for this very reason.

But Kubernetes or managed cloud services are not the only systems that rely on declared desired state and reconciliation loops to keep current state in sync. An example doing this for infrastructure automation outside the cloud provider’s walled gardens is ClusterAPI. This cloud native community initiative aims to provide the same separation across on-premise and cloud. And through integration into vSphere, ClusterAPI is readily available to VMware’s vast installed base.

The future of infrastructure is code

As an industry, we’re clearly heading into one direction. And as we continue to adopt this paradigm, the limitations that held infrastructure as code back, when working with legacy systems, do not apply any more. As infrastructure as code becomes more viable for more organizations, more teams can benefit from use-case specific frameworks to get the best possible developer experience and productivity.

Already now, many teams are using Terraform successfully. Yes, there are edge cases to consider and there is a steep learning curve, no matter if your background is in operations or software development. But as the cloud native ecosystem continues to evolve, the benefits of infrastructure as code will be applicable to more teams and more use-cases and just like PHP grew by learning from other language ecosystems, Terraform will too.

As far as Kubernetes is concerned, if you’re already adopting GitOps, the Kubestack framework is an opportunity to implement full-stack GitOps that covers both the cluster infrastructure and cluster services and not just the application workloads on the cluster. This way, you can avoid having the foundation of your system, the cluster, be the weakest link by not managing it manually via UI.

Localhost EKS development environments with EKS-D and Kubestack

Philipp Strube — Tue, 01 Dec 2020 19:04:32 +0000

Today Amazon announced EKS Distro, or EKS-D for short. A Kubernetes distribution making the same release artifacts used by Amazon EKS available to everyone.

This allows teams to use the exact same bits and pieces that power EKS, to build clusters for anything from integration tests to on-premise use-cases. As a launch partner, I got access to EKS-D in advance to integrate it into Kubestack’s local development environments.

Kubestack is about providing the best GitOps developer experience for Terraform and Kubernetes, from local development, all the way to production.

Because I believe platform engineers automating Kubernetes deserve the same great developer experience that application engineers building applications on top of Kubernetes already have.

To achieve this, the Kubestack framework integrates all the moving pieces from Terraform providers, to resources, and modules into a GitOps workflow ready for day-2 operations.

On top of the reliable automation to propose, validate and promote infrastructure changes, Kubestack is focused on giving platform teams a modern developer experience to iterate quickly using local development environments.

Now, whenever Kubestack simulates an EKS cluster locally, it uses EKS-D to do so. Let’s take a look at how Kubestack’s infrastructure automation from local development to production works.

Local Development

Imagine you’re tasked with provisioning the Prometheus operator to deploy a Prometheus instance and configuring it to scrape the metrics from your team’s application for each environment.

If you’re like me, it may take a few iterations to get the label and namespace selectors in the Prometheus resource just right and configure the RBAC for the Prometheus instance’s service account. Especially RBAC is notorious for taking a bit of trial and error for many folks to get right.

Using Kubestack’s local development environment, you can iterate on the exact same manifests that will later be used in production. The local development environment automatically updates as you make changes, and provides immediate feedback, without waiting minutes for CI/CD pipeline runs every time. It’s just like in the infamous XKCD comic, except it’s applying, not compiling.

All you have to do to get started on this task is change into your checkout of the infrastructure repository and run one kbst CLI command. Then you’re all set to work on the Prometheus manifests.

$ kbst local apply
...
Switched to workspace "loc".
...
Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
2020/11/18 12:55:53 #### Watching for changes

The new EKS-D integration means the local environment is now even closer to the EKS production environment. And fewer differences between environments reduce the risk that promoting a change fails. This is also why Kubestack uses inheritance between environments. Differences are sometimes necessary, but configuration inheritance makes them explicit.

Environment Promotion

Eventually, I’ll have the monitoring setup working locally and it’s time to push my changes and ask for a peer-review. This is the first step where the Kubestack GitOps workflow kicks in.

There are two things to review to decide if you want to apply this change. Your code changes of course, and the terraform plan provided by Kubestack’s pipeline for every branch.

If the reviewers requires changes, you can push additional commits to the branch and the pipeline will run terraform plan again. When your team approved, merge the change into master.

This triggers the pipeline and applies the merged changes to the ops environment. A terraform plan is not enough to ensure that the changes will apply correctly. That’s why Kubestack uses the ops environment, to validate the configuration change against real cloud infrastructure. The ops environment does not run applications, so that teams can feel confident to merge infrastructure changes at any time, without worrying about blocking team members or breaking applications.

Finally, if the change to ops applied successfully, the pipeline will additionally provide a terraform plan to show the required changes for the apps environment. The additional plan helps teams decide if they want to promote this change into the apps environment now.

Having a reliable workflow is crucial for teams to trust their automation. Kubestack, by combining purpose built Terraform modules and its proven triggers helps teams build infrastructure automation that is ready for day-2 operations.

Get Started

If you’ve made it this far and want to learn more, you can get started with the Kubestack framework by following the tutorial.

Keep Application Pipelines Simple by Provisioning Managed Cloud Services using Kubernetes YAML

Philipp Strube — Thu, 09 Jul 2020 10:55:53 +0000

Teams adopt Kubernetes because the fully declarative Kubernetes YAML can drastically reduce the complexity of application deployment automation. This is also a key enabler for GitOps.

However, applications that depend on managed cloud services like e.g. databases, object storage buckets or queues require the deployment automation to also handle these non Kubernetes resources. Since this is a common use-case, the full benefit Kubernetes brings to application deployment is often unfortunately drastically reduced.

Teams with this requirement have three options to move forward:

Handle the infrastructure dependencies in a separate pipeline.
Handle the infrastructure dependencies in the application pipeline itself.
Extend Kubernetes to provide a declarative way to handle these infrastructure dependencies as well.

Options 1 and 2 are well known but have significant downsides. Handling these non Kubernetes infrastructure dependencies in a separate pipeline requires manual orchestration for changes that affect both infrastructure services and the application. But implementing this orchestration in a single pipeline is anything but trivial. For teams choosing options 1 or 2, Terraform is a popular choice because it can handle both infrastructure resources and Kubernetes resources, but the Terraform integration with Kubernetes also has some limitations to be aware of.

With Kubestack, the open-source Terraform GitOps Framework I maintain, I strongly encourage teams to keep infrastructure and application automation strictly separated from each other because interdependencies are a frequent source of blockers between different tasks. So while Kubestack uses Terraform to provision the AKS, EKS or GKE clusters and all cluster services that are required before applications can be deployed, using the same pipeline to provision the managed database service an application needs is discouraged because it would break the important separation.

This leaves us with option 3. In principle, the idea is to use custom resources, to extend the Kubernetes API to support additional resource types, and then have custom controllers, commonly called operators, manage the lifecycle of these new resource types the same way a built-in controller handles deployment resources. This keeps the logic out of the application pipelines giving us back the simplicity of handling just the Kubernetes YAML.

While there are also operators that run the service workloads themselves inside the Kubernetes clusters, I will only be looking at solutions that help provision managed cloud services for this post.

Let's take a look at some available projects:

AWS Service Operator

At first glance the AWS Service Operator does not look like a project anyone should trust their infrastructure automation with.

The first version was a thin wrapper around cloud formation templates and has since been archived. The blog post that announced the original version end of 2018 has been edited in February 2020 to refer to an effort to rewrite the operator that started mid-2019. Looking at the MVP branch in the new repository shows recent activity and gives the impression of active development. But there hasn’t been a release yet.

Still, the possibility to have an operator that allows managing AWS resources from within Kubernetes using custom resources built and maintained by AWS engineers justifies keeping an eye on the AWS Service Operator for teams running on AWS.

Update Aug, 20 as of today, AWS has released a preview of the new version under a new name. Read more about the new AWS Controller for Kubernetes on the AWS blog.

Azure Service Operator

A few days ago, Azure announced the release of the Azure Service Operator. The repository shows active development and the latest release was 10 days ago. This leaves a solid first impression.

The Azure Service Operator states support for a number of Azure services like EventHub, Azure SQL, CosmosDB, Storage Accounts, and more. For teams running applications on Kubernetes on Azure, having access to an operator maintained by Microsoft is a promising option.

Crossplane

Where the AWS and Azure Service Operators are cloud specific, Crossplane is an open-source project that offers a multi-cloud solution. The repository shows active and recent development. Crossplane is divided into a cloud agnostic part and cloud specific providers. These providers exist for AWS, GCP, Azure and Alibaba.

Crossplane is supported by Upbound and not one of the cloud providers directly, but makes up for that by offering multi-cloud support.

Even if multi-cloud support is not a requirement, Crossplane is worth taking a look at because it has been around significantly longer which suggests more maturity. Looking at the custom resources Crossplane makes available inside the cluster it seems there are about 70 different services supported between the four cloud providers.

Terraform Operator

Hashicorp in March announced the alpha release of the official Terraform operator. Terraform already has wide support for cloud resources across many cloud providers including their managed services. But as discussed earlier, having to maintain both Kubernetes YAML and Terraform HCL in one repository and having to run Terraform as part of the application pipeline adds complexity to that pipeline.

The Terraform operator takes the approach that users specify a Workspace custom resource which it syncs with the workspace on the mandatory linked Terraform cloud account. Based on the activity in the repository, the Terraform operator also seems actively developed.

For teams already invested into Terraform, this may be a good option. Given Terraform’s wide support, it may also support significantly more services than the other three. But this approach isn’t as purpose built as the other three options we discussed earlier and may require all team members to get comfortable with HCL and the Terraform concepts of providers and modules.

Conclusion

For teams that want to keep their application pipelines simple and only have to worry about maintaining Kubernetes YAML to specify their application’s requirements, there are options. Handling it this way makes perfect sense, because it helps further strengthening the separation between application and platform layer that Kubernetes enables.

With the AWS and Azure Service Operators there are two examples of implementations directly from the cloud providers. Crossplane is a multi-cloud alternative maintained as an open-source project. And with the Terraform Operator, there is also an option to keep the application pipeline simple but still having access to the Terraform ecosystem.

None of the solutions have announced general availability yet so it’s too early to make any recommendations. But the approach described here is powerful to help teams clearly separating between application and platform automation and this very much justifies keeping an eye on the future development in this space.

Why now is the time for the Spring Boot of infrastructure automation

Philipp Strube — Wed, 24 Jun 2020 11:13:07 +0000

Gartner predicts that through 2022, 75% of DevOps initiatives will fail to meet expectations due to issues around organizational learning and change.

But DevOps initiatives are important, because they enable critical capabilities to compete in the future. Without demolishing the figurative wall legacy IT built between development and operations, delivering the kind of digital experiences customers expect today will more likely fail than not.

So before we can get into how an infrastructure automation framework can help your DevOps initiative succeed, we first need to take a closer look at why so many fail.

Why do so many DevOps initiatives fail?

In the last two years at Container Solutions, I’ve worked with teams from steel to fintech and agriculture to telecommunications on various aspects of their cloud native transformations. Adopting DevOps is always a big part of that transformation. Probably the hardest thing for companies is to adapt their existing organizational structure to cross-functional DevOps teams.

Organizations need this change to happen, because digital experiences are already the new normal for a majority of consumers. This puts enormous pressure on the teams. The changes required are huge, especially from a legacy operations perspective. Almost everything changes. Team setup, workflow, responsibilities and tooling. And don’t forget it’s rarely on a green field. People in those initiatives are often asked to continue supporting the old while at the same time learning the new.

For individuals, change brings lots of uncertainty. People may not openly admit it — or worse, hide it behind outright hostility — but self-doubt is common and most certainly understandable. People wonder if they personally will be able to succeed in this new world and are afraid what this means for their careers.

And on top of the high pressure and the self-doubt, still overshadowed by the “them vs. us” mindset that legacy IT nurtured for years, people with a traditional IT-ops background are asked to work more like “them”.

Application teams and platform teams

First, getting the “them vs. us” out of the way is crucial. The way I like to think about this is, all teams are cross-functional DevOps-teams. Some just work at the application and some work at the platform layer.

But still, while application teams use Spring Boot or similar frameworks to move fast, platform teams have to build infrastructure and automation by integrating low level tooling from scratch.

This lack of tool chain maturity combined with the high pressure and need to learn new skills adds a very strong and counterproductive feeling to the mix. The feeling of being treated unfairly. It’s hard to not feel treated unfairly if you’re expected to move as fast as your application counterparts when at the same time having a dramatically less mature tool chain at your disposal.

It is as if one team is given a 3D printer to print a house and the other team is given a bunch of parts to first assemble the excavator before they can dig the hole for the foundation with it.

Application frameworks and platform frameworks?

In the past, the abstraction between applications and infrastructure was the operating system. But operating systems are a weak abstraction, which leads to application requirements leaking into the platform layer. This meant infrastructure configuration often did not have enough in common, even between application tiers, for reusable framework components to be feasible.

Containers did not improve the abstraction layer qualities of operating systems, but they still fixed this issue by allowing separate application and platform layer operating system instances.

Furthermore, Kubernetes sits right at the border of application and platform layer and provides a robust API for teams to have clear responsibilities and not interfere with each other.

This is huge. Powerful abstraction between the layers marks a paradigm-shift that — for the first time — allows platform teams to benefit from frameworks as well.

I believe that not only is now the time for such a framework but that it will also help more DevOps initiatives to succeed.

How can frameworks help your DevOps initiative succeed?

Frameworks are popular in application development, because they help teams move faster both when creating new applications and also when maintaining these applications.

Bringing the same benefit to the platform layer and removing the reason to feel unfairly treated by leveling the playing field with the application layer takes out the emotion and removes a big source of conflict.

But frameworks, based on reusable components, also allow teams to use these components without needing to fully understand their implementation. Allowing teams to make progress, even when the technology is new for them. The usage and common use-cases can be documented. This is a prime resource for teams to learn the new skills and knowledge they need to drive their company’s DevOps initiatives forward.

Last but not least, frameworks foster a user community of people working on the same challenges and available to help each other. Be it through sharing experience through blog posts and conference talks, writing guides and tutorials or being available as consultants.

Frameworks have the potential to help your DevOps initiative succeed, because they help solve two of the biggest reasons for failure.

First, the feeling of unfairness from needing to perform like the application counterpart while having a dramatically less mature tool chain at your disposal.

And second, the uncertainty whether you will be able to succeed personally. Having access to a framework gives people the trust and confidence to move forward and learn along the way.

Luckily, this is more than just theory

If you now think, this all sounds great if only such a framework would exist, I have good news for you.

I’ve been working on a framework like this for the last one and a half years. Kubestack is an open-source Terraform GitOps framework for infrastructure automation. It’s designed for teams that want to automate Kubernetes based infrastructure and not reinvent automation. Think of it this way, Kubestack is to Terraform and infrastructure automation, what Spring Boot is to Java and cloud native applications.

The framework supports all three major cloud providers and has been used as the foundation for a number of real world customer projects as part of my colleagues’ and my consulting work. It is fully documented, has a step-by-step tutorial to help users get started and even includes a local GitOps development lab. So you can test-drive Kubestack and learn more about GitOps for infrastructure automation in the comfort of your own localhost.

For questions there is a #kubestack channel on the Kubernetes community Slack and the repository on Github.

Speed up multi-stage Docker builds in CI/CD with Buildkit’s registry cache

Philipp Strube — Wed, 29 Apr 2020 09:49:13 +0000

Working on a GitOps framework around Kubernetes, I naturally run everything in containers. The two Dockerfiles that matter most for me, unfortunately, both have to download a lot of dependencies not included in the repository at build time. Which means the layer cache is crucial. Unfortunately, ephemeral CI/CD runners like GitHub Actions, start each run with an empty cache.

The first of the two Dockerfiles builds the image for the framework itself. This image is used for bootstrapping, automation runs and also disaster recovery. As such, it’s not your run-of-the-mill Dockerfile. It includes a number of dependencies installed from Debian packages, various Go binaries and last but not least the Python based CLIs of AWS, Azure and Google Cloud. It makes heavy use of multi-stage-builds and has different build stages for common dependencies and each Cloud provider’s specific dependencies. The layers of the final image also mirror the build stage logic.

Dockerfile number two is for the Kubestack website itself. The site is built using Gatsby and has to download a lot of node modules during the build. The Dockerfile is optimized for cache-ability and uses multi-stage builds to have a build environment based on NodeJS and a final image based on Nginx to serve the static build.

Build time for both, the framework image and the website image, heavily benefits from having a layer cache.

Docker has had the ability to use an image as the build cache using the --cache-from parameter for some time. This was my preferred option because I need the ability to build and push images anyway. Storing the cache alongside the image is a no-brainer in my opinion.

For the website image the first step of my CI/CD pipeline is to pull the cache image. Note the || true at the end to ensure a missing cache doesn’t prevent my build from running.

docker pull gcr.io/$PROJECT_ID/$REPO_NAME:latest-build-cache || true

Step two runs a build targeting the dev stage of my multi-stage Dockerfile and tags the result as the new build-cache.

docker build \
  --cache-from gcr.io/$PROJECT_ID/$REPO_NAME:latest-build-cache \
  --target dev \
  -t gcr.io/$PROJECT_ID/$REPO_NAME:latest-build-cache \
  .

The next step runs the actual build that produces the final image and tags it as well.

docker build \
  --cache-from gcr.io/$PROJECT_ID/$REPO_NAME:latest-build-cache \
  -t gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA \
  .

Finally, the pipeline pushes both images.

docker push gcr.io/$PROJECT_ID/$REPO_NAME:latest-build-cache
docker push gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA

For a simple multi-stage build with only two stages, like my Gatsby website’s Dockerfile, this works pretty well.

But when I tried this for a project with multiple build stages, one for Python and one for JS, specifying two images under --cache-from never seemed to work reliably. Which is double unfortunate, because having a layer cache here would save time not downloading Python and JS dependencies on every run.

Having cache pull and cache build steps for every stage also makes for a growingly verbose pipeline file the more stages you have.

So for the framework Dockerfile, I need something better.

Enter buildkit. Buildkit brings a number of improvements to container image building. The one’s that won me over are:

Running build stages concurrently.
Increasing cache-efficiency.
Handling secrets during builds.

Apart from generally increasing cache efficiency, it also allows more control over caches when building with buildctl. This is what I needed. Buildkit has three options for exporting the cache. Called inline, registry and local. Local is not particularly interesting in my case, but would allow writing the cache to a directory. Inline includes the cache in the final image and pushes cache and image to the registry layers together. But this only includes the cache for the final stage in multi-stage builds. Finally, the registry option does allow pushing all cached layers of all stages into a separate image. This is what I needed for my framework Dockerfile.

Let’s take a look at how I’m using this in my pipeline. Having the cache export and import included in buildkit means I can reduce the three steps into one. And it also stays one step, no matter how many stages my Dockerfile has.

docker run \
  --rm \
  --privileged \
  -v `pwd`/oci:/tmp/work \
  -v $HOME/.docker:/root/.docker \
  --entrypoint buildctl-daemonless.sh \
  moby/buildkit:master \
    build \
    --frontend dockerfile.v0 \
    --local context=/tmp/work \
    --local dockerfile=/tmp/work \
    --output type=image,name=kubestack/framework-dev:test-${{ github.sha }},push=true \
    --export-cache type=registry,ref=kubestack/framework-dev:buildcache,push=true \
    --import-cache type=registry,ref=kubestack/framework-dev:buildcache

This one command handles pulling and importing the cache, building the image, exporting the cache and pushing the image and the cache. By running the build inside a container, I also don’t have to worry about installing the buildkit daemon and cli. The only thing I needed to do was providing the .docker/config to the build inside the container to be able to push the image and the cache to the registry.

For a working example, take a look at the Kubestack release automation pipeline on Github.

Using the cache, the framework image builds in less than one minute. Down from about three minutes before using buildkit without the cache export and import.

Don’t mistake Kubestack for yet another way to build Kubernetes clusters

Philipp Strube — Fri, 24 Apr 2020 14:59:08 +0000

I’m writing this post for the ones with stuff to get done in the seemingly never sleeping cloud native community. Talk about two things that don’t mix well.

When only looking at Kubestack briefly, you may mentally file it as yet another tool to build Kubernetes clusters and move on. But doing so, you’d be missing out, in my humble, and arguably biased opinion.

Kubestack aspires to be for GitOps and Terraform, what Spring Boot or Rails are for application development and Java or Ruby respectively. Yes, that's a big goal.

Kubestack does not turn a bunch of machines into a Kubernetes cluster. Instead, the open source framework maintains the desired state of clusters and cluster services and applies changes following a GitOps approach to an API. This API may be a cloud provider API or the Kubernetes API, depending on what part of the desired state changed. Currently, Kubestack supports managed Kubernetes from Amazon (EKS), Azure (AKS) and Google (GKE). For on-premise and other cloud providers I consider Cluster API the most promising development.

Teams basing their infrastructure automation on Kubestack can

reason about proposed changes using pull requests,
and test proposed changes on the non-critical ops-environment
before applying the changes to the critical apps-environment.

Kubestack enables this for both the infrastructure of the cluster itself, and the services that run on top of the cluster. Think services that need to be on the cluster before application workloads can run there. Using Kubestack you can maintain both in one repository and have changes applied reliably through the same GitOps automation.

Following this approach, teams can jointly make changes to both infrastructure and applications without one blocking the other.

And that is Kubestack in a nutshell for you. If you want to learn more, head over to the Kubestack GitOps framework documentation for all the details.