DEV Community: Jakob Gillich

Why Kubernetes Is So Complex

Jakob Gillich — Mon, 17 Oct 2022 19:20:17 +0000

The container wars are over, and Kubernetes won. Docker the company has given up on Swarm, and is refocusing on developer tools, and Mesos development has slowed down to a crawl.

Kubernetes is embraced by large companies with dedicated infrastructure teams, but the hobbyists and small businesses are being left behind. Unnecessary complexity is probably the commonly mentioned reason for not adopting Kubernetes.

But why is Kubernetes so complex? And is the complexity actually unnecessary? Let's look at how Kubernetes works behind the scenes, and why the complexity may be a tradeoff worth making.

Kubernetes is modular

Beginners are often overwhelmed by Kubernetes. CRD? CNI? CSI? etcd? GroupVersionKind? Why is deploying containers so complicated?

At the core of Kubernetes is the API server, which is a CRUD API, meaning we can create, read, update and delete resources. The key to understanding the API server is the CustomResourceDefinition. They tell the API server which resources exist and what fields they have. Core resources are technically not CRDs, but their behavior is otherwise identical.

When we send a resource manifest to the API server, the following happens:

It validates all fields against a stored CustomResourceDefinition.
It calls registered webhooks ("admission controllers"). They perform additional validations that are specific to the resource.
It stores the resource in its storage backend (typically etcd).

At this stage, we haven't actually done anything yet. No images are being pulled, no containers are being deployed. How does that happen? We left out one feature of the API server: The ability to watch for changes.

Nearly everything in Kubernetes happens through programs that watch the API server for changes, we call them controllers. When a resource changes, a controller watching it will run and perform an action. We call that reconciliation, turning the desired state into the actual state. When we create a Deployment, a controller uses the information provided in the deployment manifest to find one or more nodes, and creates Pod resources for the nodes. The node agent or kubelet watches the pod resource and deploys the container(s). When the pods are deployed (or failed to deploy), the kubelet returns its result by writing to the pod's status field.

This mode of operation is at the very core of Kubernetes, resources are detached from their implementation. The CSI (Container Storage Interface) is an excellent example of the strengths of this approach. Where Docker volumes are stored locally, Kubernetes' flexibility allows it to support any kind of volume: local, Ceph, NFS or provider-specific volume implementations.

It is important that we don't do the same thing twice however - if we try to provision a local and a NFS volume to the same mount point, we'll run into problems. Most resources are reconciled by a single controller, but for storage, you sometimes have more. The storage class is a string field on volume claims that identified the responsible controller. Notably, this is not a feature of the API server, we depend on the controllers to respect it.

Kubernetes is eventually consistent

Anyone using Kubernetes will have come across one thing: CrashLoopBackOff. But why does it occur?

Kubernetes is a big system with many components, and some things must happen in a certain order. However, there is no attempt to synchronize or order operations, instead, failed operations are retried.

When a pod requires a volume, it will have to wait for the CSI driver to create and mount it. How long does that take? Usually a couple of seconds, but if there's an error with the CSI driver, it may not happen for hours or days. So when a pod is created, the kubelet will check if the volume is available. If it is not, it will wait a few seconds and try again. But, as to not waste system resources, it will wait a little longer every time. This is called error back-off, and it's built into all controllers.

All the above has one major implication: It's difficult to figure out where things went wrong. When docker run fails, you will be told why. When you create a deployment, your initial feedback will be nothing at all. Only the status of the deployment and pod will tell you more. And even they may not point directly at the problem.

With some experience and a user interface like Lens, debugging becomes easier. And there are great monitoring solutions for production use. But this is still a big hurdle for beginners taking their first steps with Kubernetes.

Kubernetes is declarative

Much has been said about the advantages of declarative configuration, infrastructure as code or gitops. If you believe in them, Kubernetes is for you. Kubernetes is the natural progression of Terraform into an always running API. Only Kubernetes does not require fixed ordering and scales infinitely better.

Flux lets us use a git repository as the single source of truth for our cluster, but for dynamic resources we will want to use the API server instead. We use both of these approaches; Flux sets up our cluster, and applications are created using our own Application resource.

Let's write a controller

What does a controller actually look like? For Cloudplane, we have a resource to request SMTP credentials for our applications.

apiVersion: cloudplane.org/v1alpha1
kind: SMTPCredentials
metadata:
  name: example-app-smtp

When we submit this resources to the API server, a controller will be watching this resource type and use it to generate a secret. By convention, we emit the secret with the same name as the SMTPCredentials resource.

For development, we have deployed a instance of MailHog to our cluster, a simple mail server that catches all mails without forwarding them. It allows any user/password combination. Here is what our mailhog controller looks like:

func (r *SMTPCredentialsReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  secret := corev1.Secret{
    TypeMeta: metav1.TypeMeta{APIVersion: corev1.SchemeGroupVersion.String(), Kind: "Secret"},
    ObjectMeta: metav1.ObjectMeta{
      Name:      req.Name,
      Namespace: req.Namespace,
    },
    Data: map[string][]byte{
      "host":     []byte("mailhog.dev"),
      "port":     []byte("1025"),
      "username": []byte("foo@example.com"),
      "password": []byte("smtp-password"),
    },
  }

  if err := ctrl.SetControllerReference(&cred, object, r.Scheme); err != nil {
    return ctrl.Result{}, err
  }

  if err := r.Patch(ctx, object, client.Apply, client.ForceOwnership, client.FieldOwner("smtpcredentials-controller")); err != nil {
    return ctrl.Result{}, err
  }

  return ctrl.Result{}, nil
}

Since we already know the hostname of our mailhog instance, and we don't need to generate any username/password, all values are static. Setting an owner reference on the secret means it will be deleted by the API server when the SMTPCredentials resource is deleted, and finally Patch creates or updates the secret. If this function returned an error, it would be automatically retried and we would see the error back-off behavior described earlier.

In production, we replace this controller with our implementation for SES.

Kubernetes works

Kubernetes was created by engineers at Google who had been running a similar system for years. They knew exactly what they were doing. The design of Kubernetes is very intentional.

There are aspects of Kubernetes that are unfinished (StatefulSets) or need to be reworked (Ingress->Gateway API), it is not a perfect system. But it probably is the best system we currently have.

The good news is that there are many efforts to make Kubernetes easier to use. Acorn aims to simplify application packaging and deployment. And services like Render and our very own Cloudplane use Kubernetes under the hood to deliver a user-friendly solution that requires zero Kubernetes knowledge.

Originally published at https://cloudplane.org.

Apko: A Better Way To Build Containers?

Jakob Gillich — Thu, 13 Oct 2022 18:15:46 +0000

Chainguard recently announced Wolfi, a set of container images that are built directly from Alpine packages and not Dockerfiles.

Ever since Docker released in 2013, we have been using Dockerfiles to build containers. They work well, but there are some problems:

They make it hard to share components between images, a single base image is the only option. Code reuse is difficult.
Some components like package managers are not required to run a container, but they are included for the convenience of the builder. There is no separation between build and runtime dependencies (although multi-stage builds can mostly achieve this).
Docker does not support reproducible builds (but kaniko does).
Layers serve no purpose for production builds, most images use one giant and messy RUN statement that does everything. That however slows down rebuilds during development.

Let's look at the solutions provided by Chainguard that try to address these (and more) problems.

melange

Melange is a builder for Alpine packages. It uses pipelines similar to common CI/CD services, and it builds for multiple architectures by default. Here is a simplified example of a package build for the forum software NodeBB:

package:
  name: nodebb
  version: 2.5.3
  dependencies:
    runtime:
      - nodejs

environment:
  contents:
    packages:
      - alpine-baselayout
      - ca-certificates-bundle
      - nodejs
      - npm
      - git

pipeline:
  - uses: fetch
    with:
      uri: https://github.com/NodeBB/NodeBB/archive/refs/tags/v${{package.version}}.tar.gz
      expected-sha256: 92e390d7cda190e7f098833cbbbf03fbe1c50f25653656ad589ae97dc18a7684
      strip-components: 0
  - runs: |
      mkdir -p "${{targets.destdir}}/usr/share/nodebb"
      cd NodeBB-${{package.version}}
      cp install/package.json .
      npm install --omit=dev
      cp -a ./. "${{targets.destdir}}/usr/share/nodebb"

Under the environment section, we set up a build environment. Then we call the built-in fetch pipeline to fetch the source code bundle, verify the hash and extract it. There are a few more built-in pipelines, but we can also write our own.

Next, we run a short script to install dependencies, run a build and copy the resulting files into a destination directory. Everything under targets.destdir is copied into the package, which will only depend on nodejs, as we do not need npm or git at runtime.

Now, we can use melange to build an apk package. This isn't a container yet, but for that, we use the next tool.

apko

apko takes apk packages and builds them into OCI images (aka Docker images). Sounds quite simple, because it is:

contents:
  repositories:
    - https://dl-cdn.alpinelinux.org/alpine/edge/main
  packages:
    - alpine-baselayout
    - nodebb

entrypoint:
  command: /usr/share/nodebb/nodebb start

Note that there is no base image being used, instead, we get exactly what's in the packages we list. alpine-baselayout adds busybox and the standard directory structure.

We can include other apko definitions as a base or create users and groups and set up file system permissions, but that's about it. apko lacks most capabilities of traditional OCI image builders as it's meant to be combined with melange, adding a files and folders directly is not supported. But what it does have is native support for s6 in case we want to run multiple services.

apko is quite fast, all it does is extract apk packages into OCI images and it doesn't run scripts. This is what enables fast rebuilds of images after an individual package is updated, and also makes them reproducible.

Using both tools, did we solve the problems we laid out before? Let's see.

melange's pipelines enable modularity at the build level, and apko lets us combine multiple packages into one image. Code reuse is easy.
Images built with apko contain exactly the packages we want, and nothing else. Images are smaller than comparable Docker images using the Alpine base image.
Both melange and apko provide fully reproducible builds.
Instead of layers, we use packages to keep build times fast and image sizes small.

Should I use this?

We already use apko to create images for our own use (planned public release at a later stage). The technology works well, but we have questions. The state of glibc support isn't clear to us, and documentation is bare-bones. And while melange and apko are technically faster than Dockerfiles because they can be more selective about what needs to be rebuilt, there isn't tooling that takes advantage of that yet.

There are always the Wolfi images to use as base images in standard Dockerfiles. For the story behind the creation of Wolfi, this insightful Twitter thread by Ariadne Conill is worth a read.

This post was originally released on the Cloudplane blog.

Why We Use CUE (and Not Helm)

Jakob Gillich — Sun, 09 Oct 2022 10:56:46 +0000

Cloudplane is a managed hosting platform built on Kubernetes using our own resource templates. We evaluated several templating tools and eventually settled on CUE. Here's how we arrived at that conclusion.

Go

At the core of Cloudplane is a Kubernetes operator that extends the Kubernetes API with various types, such as Application:

kind: Application
spec:
  instance: small
  replicas: 1
  type: discourse

Our templating layer uses application resources to generate Kubernetes resources such as deployments and services, similar to what Helm charts do with values. Since we wrote our operator in Go, it seemed logical to use Go for templating as well. But it quickly became apparent that this approach would not scale well.

For one, Go is not an elegant language. Constructing Kubernetes resources in Go is significantly more verbose than e.g. YAML, and string interpolation via placeholders is difficult to read. On top of that, Go does not have built-in support for merging objects (but there is mergo).

Go works when you have few resources, but for heavy templating use-cases, you want a language dedicated to that purpose.

Jsonnet

Jsonnet promises to be JSON plus templating, and that's exactly what it delivers. It allows you to include other files and has many useful features such as variables and functions. Jsonnet overall was a pretty decent
experience, and we could've stopped there.

But Jsonnet is a dynamic language with barely any editor integration, and we strongly prefer static types for editor hints and autocomplete. So we kept looking.

Dhall

Dhall is a functional configuration and programming language. Static typing and its purely functional nature are things we liked about Dhall, but for a configuration language, it can be comically verbose and difficult to read. The following YAML:

matchLabels:
  app.kubernetes.io/instance: {{.App.Metadata.Name}}

Turns into this Dhall:

matchLabels = Some
  ( toMap
      { `app.kubernetes.io/instance` =
          merge
            { Some = λ(_ : Text) → _, None = "" }
            app.metadata.name
      }
  )

But the biggest issue with Dhall is performance. Even small templates can take several minutes and gigabytes of RAM to build, and Dhall's VS Code plugin was nearly unusable for us. This is a known issue, and until it is resolved, we cannot recommend Dhall for Kubernetes users.

Helm

Helm is a package manager for Kubernetes. We like using it for third-party dependencies, but templating in Helm is not very good. It uses YAML, a whitespace-sensitive data language, and layers a generic text templating engine on top of it. Charts are typically full of boilerplate and indentation hacks.

We originally started with third-party Helm charts, but ran into frequent issues around secrets. We provide all kinds of credentials through secrets, and pods have a simple mechanism for injecting environment variables from them:

- name: FOO
  valueFrom:
    secretKeyRef:
      name: secret-name
      key: FOO

Unfortunately, most charts require passing secrets as plain text in the values object, and if they do support secrets, the keys are often not configurable. Helm's biggest strength and also its biggest weakness is the abstraction via values, you don't have to think about the underlying K8s resources, but when they matter to you, your only way forward is forking the chart.

We continue to use Helm for third-party packages, but for our own needs, we decided to use...

CUE

CUE is a configuration language that is statically typed and also supports validations. Like Jsonnet, it's a superset of JSON, but with many useful extensions such as comments, variables, modules, and of course
types. It's not quite as powerful at templating as Jsonnet, it does not have functions, but so far we haven't run into serious roadblocks.

Its design is inspired by Go, all files in a single directory (called package) are combined into the same output. Since configuration can be spread out over many files (and modules), CUE has strict rules about what you can and cannot do.

// types are just values, * denotes a default value
baz: string | *"baz"

// overriding default value, this is fine
baz: "buz"

// conflicting value, this is an error
baz: "bux"

As you might have guessed, CUE makes quotes and curly brackets mostly optional. It's not quite as slick as YAML, but it's much improved from raw JSON.

The killer feature for Kubernetes users is that CUE can import Go types:

cue get go k8s.io/api/core/v1

This command will fetch the Go source code and convert it to CUE so we get full type validation:

import corev1 "k8s.io/api/core/v1"

foo: corev1.#ConfigMap & {
  apiVersion: "v1"
  kind:       "ConfigMap"
  data: {
    bar: "bar"
  }
  invalid: "abc" // this will error
}

There are some issues we have with CUE. For one, its syntax is very unusual, _|_ is assigned when there is an error and "\(foo)-bar" is what string interpolation looks like. We would gladly give up JSON compatibility for a more elegant language.

Programmability is also limited. There are built-in functions one can call, but defining custom functions is not possible. There is if but not else. A lot can be achieved with some trickery, but once again, we wish these were built-in concepts using elegant syntax.

CUE is a young project with some rough edges, but we think CUE is the future of Kubernetes templating. Its adoption is growing quickly, with Dagger and Acorn as some of its recent adopters.