Why We Use CUE (and Not Helm)

Cloudplane is a managed hosting platform built on Kubernetes using our own resource templates. We evaluated several templating tools and eventually settled on CUE. Here's how we arrived at that conclusion.

Go

At the core of Cloudplane is a Kubernetes operator that extends the Kubernetes API with various types, such as Application:

kind: Application
spec:
  instance: small
  replicas: 1
  type: discourse

Our templating layer uses application resources to generate Kubernetes resources such as deployments and services, similar to what Helm charts do with values. Since we wrote our operator in Go, it seemed logical to use Go for templating as well. But it quickly became apparent that this approach would not scale well.

For one, Go is not an elegant language. Constructing Kubernetes resources in Go is significantly more verbose than e.g. YAML, and string interpolation via placeholders is difficult to read. On top of that, Go does not have built-in support for merging objects (but there is mergo).

Go works when you have few resources, but for heavy templating use-cases, you want a language dedicated to that purpose.

Jsonnet

Jsonnet promises to be JSON plus templating, and that's exactly what it delivers. It allows you to include other files and has many useful features such as variables and functions. Jsonnet overall was a pretty decent
experience, and we could've stopped there.

But Jsonnet is a dynamic language with barely any editor integration, and we strongly prefer static types for editor hints and autocomplete. So we kept looking.

Dhall

Dhall is a functional configuration and programming language. Static typing and its purely functional nature are things we liked about Dhall, but for a configuration language, it can be comically verbose and difficult to read. The following YAML:

matchLabels:
  app.kubernetes.io/instance: {{.App.Metadata.Name}}

Turns into this Dhall:

matchLabels = Some
  ( toMap
      { `app.kubernetes.io/instance` =
          merge
            { Some = λ(_ : Text) → _, None = "" }
            app.metadata.name
      }
  )

But the biggest issue with Dhall is performance. Even small templates can take several minutes and gigabytes of RAM to build, and Dhall's VS Code plugin was nearly unusable for us. This is a known issue, and until it is resolved, we cannot recommend Dhall for Kubernetes users.

Helm

Helm is a package manager for Kubernetes. We like using it for third-party dependencies, but templating in Helm is not very good. It uses YAML, a whitespace-sensitive data language, and layers a generic text templating engine on top of it. Charts are typically full of boilerplate and indentation hacks.

We originally started with third-party Helm charts, but ran into frequent issues around secrets. We provide all kinds of credentials through secrets, and pods have a simple mechanism for injecting environment variables from them:

- name: FOO
  valueFrom:
    secretKeyRef:
      name: secret-name
      key: FOO

Unfortunately, most charts require passing secrets as plain text in the values object, and if they do support secrets, the keys are often not configurable. Helm's biggest strength and also its biggest weakness is the abstraction via values, you don't have to think about the underlying K8s resources, but when they matter to you, your only way forward is forking the chart.

We continue to use Helm for third-party packages, but for our own needs, we decided to use...

CUE

CUE is a configuration language that is statically typed and also supports validations. Like Jsonnet, it's a superset of JSON, but with many useful extensions such as comments, variables, modules, and of course
types. It's not quite as powerful at templating as Jsonnet, it does not have functions, but so far we haven't run into serious roadblocks.

Its design is inspired by Go, all files in a single directory (called package) are combined into the same output. Since configuration can be spread out over many files (and modules), CUE has strict rules about what you can and cannot do.

// types are just values, * denotes a default value
baz: string | *"baz"

// overriding default value, this is fine
baz: "buz"

// conflicting value, this is an error
baz: "bux"

As you might have guessed, CUE makes quotes and curly brackets mostly optional. It's not quite as slick as YAML, but it's much improved from raw JSON.

The killer feature for Kubernetes users is that CUE can import Go types:

cue get go k8s.io/api/core/v1

This command will fetch the Go source code and convert it to CUE so we get full type validation:

import corev1 "k8s.io/api/core/v1"

foo: corev1.#ConfigMap & {
  apiVersion: "v1"
  kind:       "ConfigMap"
  data: {
    bar: "bar"
  }
  invalid: "abc" // this will error
}

There are some issues we have with CUE. For one, its syntax is very unusual, _|_ is assigned when there is an error and "\(foo)-bar" is what string interpolation looks like. We would gladly give up JSON compatibility for a more elegant language.

Programmability is also limited. There are built-in functions one can call, but defining custom functions is not possible. There is if but not else. A lot can be achieved with some trickery, but once again, we wish these were built-in concepts using elegant syntax.

CUE is a young project with some rough edges, but we think CUE is the future of Kubernetes templating. Its adoption is growing quickly, with Dagger and Acorn as some of its recent adopters.