DEV Community: Nick Santos

Go Rant: Buffered Channels Should be Tossed in a Fire

Nick Santos — Fri, 21 May 2021 17:40:04 +0000

Channels are Go’s fundamental tool for concurrent programming.

Buffered channels are a small, innocent-looking feature on top of channels. I’m going to try to convince you that they’re a monstrous abomination.

Quick Recap on What I’m Ranting About

First, a quick recap of how channels work if you haven’t used Go in a while.

A channel lets you send data on one goroutine, and receive it on another, concurrent goroutine.

In a normal unbuffered channel, sending data will block until the data is received.

This blocks forever:

// https://play.golang.org/p/eJGdsxiHOIg
ch := make(chan int, 0)
ch <- 1
fmt.Println("Impossible")

This prints “Received 1\nPossible”

// https://play.golang.org/p/bBIkIiFXeF2
ch := make(chan int, 0)
go func() {
  fmt.Printf("Received: %d\n", <-ch)
}()
ch <- 1
fmt.Println("Possible")

But you can make buffered channels of arbitrary size N. A buffer of size N will block if you send more than N times before receiving.

This prints “Possible” then deadlocks.

// https://play.golang.org/p/GCL8SY_8AUk
ch := make(chan int, 1)
ch <- 1
fmt.Println("Possible")
ch <- 1
fmt.Println("Impossible")

My Assertion

There are only 3 reasonable values of N:

0 — the channel always blocks.
1 — the channel lets you push one element at a time.
Infinite — the channel lets you keep pushing elements until you run out of memory.

All other values of N will inevitably lead to bugs.

Why

There are two ways to make this case: the empirical argument (why real projects blow up when they use buffered channels) and the theoretical one (why even hypothetical imagined projects will blow up when they use buffered channels).

From the Empirical Argument

Every opensource project I’ve ever seen that used an arbitrary-N buffered channel was using that channel incorrectly. And ended up having bugs.

Someone would inevitably send a list of elements to that channel. But the problem is that you can only do this safely if:

You can guarantee that the list is less than N.
You can guarantee that no one else is also sending on the channel.
OR you can guarantee that another goroutine is available to receive on the channel.

These are often hard guarantees to make! They’re often about other actors that you don’t control (particularly if you got your list of elements somewhere else.) And they’re guarantees that the programming language doesn’t help you enforce at all.

That hints at:

From the Theoretical Argument

“I can only call this function N times synchronously” is an abnormal API constraint. And it’s not a constraint that programming languages do anything to help you enforce.
Programming languages have lots of constraints they do help you to enforce!

They help you ensure that a function isn’t called (think: private functions).
They help you ensure that a function is called exactly once (think: constructors, sync.Once).
They help you ensure that a function is called only once at a time (think: locks/mutexes and all the tooling around them).

Now, I don’t want to harsh your buzz for programming language constraint checks. We live in 2021! Rust’s borrow checker can do compile-time checks that I never would have imagined!

But let’s be honest: the Go-team does not have the appetite for the kind of programming language features that would help you use buffered channels well.

Hope

I liked this proposal for Go2 for an unlimited-capacity buffered channel.

proposal: spec: add support for unlimited capacity channels

But I also liked Ian Lance Taylor’s response!

If we had generic types, unlimited channels could be implemented in a library with full type safety. A library would also make it possible to improve the implementation easily over time as we learn more.

So I’m hoping for a one-two punch:

Go gets generics!
We can add a channel wrapper that makes more sense!!
They can remove buffered channels from the core language!!!
Profit!!!!

Originally published at https://nicksantos.medium.com/go-rant-buffered-channels-should-be-tossed-in-a-fire-d36dcc9dbf86

Three Ways to Run Kubernetes on CI and Which One is Right for You!

Nick Santos — Fri, 02 Apr 2021 22:24:21 +0000

When we first started developing Tilt, we broke ALL THE TIME.

Either Kubernetes changed. Or we had a subtle misunderstanding in how the API works. Our changes would pass unit tests, but fail with a real Kubernetes cluster.

I built out an integration test suite that used the latest version of Tilt to deploy real sample projects against a real cluster.

At the start, it was slow and flakey. But the tooling around running Kubernetes in CI has come a long way, especially in the last 1-2 years. Now it's less flakey than our normal unit tests 😬. Every new example repo we set up uses a one-time Kubernetes cluster to run tests against.

A few of our friends have been asking us how we set it up and how to run their own clusters in CI. I've now explained it enough times that I should probably write down what we learned.

Here are three ways to set it up, with the pros and cons of each!

Strategy #1: Local Cluster, Remote Registry

Here's how I set up our first integration test framework.

I created a dedicated gcr.io bucket for us to store images, and a GCP service account with permission to write to it.
I added the GCP service account credentials as a secret in our CI build.
I forked kubeadm-dind-cluster, a set of Bash scripts to set up Kubernetes with Docker-in-Docker techniques.

All our test projects had Tilt build images, push them to the gcr.io bucket, then deploy servers that used these images.

I barely got this working. A huge breakthrough! It caught so many subtle bugs and race conditions.

I wouldn't call the Bash scripts readable. But they are hackable, cut-and-pasteable. There were examples of how to run it on CircleCI and TravisCI. kubeadm-dind-cluster has been deprecated in favor of more modern approaches like kind. But I learned a lot from its Bash scripts. We still use a lot of the techniques in this project today.

There were other downsides though:

When drive-by contributors sent us PRs, the integration tests failed. They didn't have access to to write to the gcr.io bucket. This made me so sad. Contributors felt unwelcome. I never figured out a way to make this secure.
We didn't reset the gcr.io bucket between test runs. So it was hard to guarantee that images weren't leaking between tests. For example, if image pushing failed, we wanted to be sure we weren't picking up a cached image from a previous test.

Strategy #2: Local Registry On a VM

When I revisited this, I wanted to make sure:

Anyone could write to the image registry.
The image registry would reset between runs.

By this time, kind was taking off as the default choice for testing Kubernetes itself. kind also comes with the ability to run a local registry, so you can push images to the registry on localhost:5000 and pull them from inside kind.

I set up a new CI pipeline that:

Creates a VM.
Installs all our dependencies, including Docker.
Creates a kind cluster with a local registry, using their script.

This worked well! And because the registry was local, it was faster than pushing to a remote registry. We still use this approach to test ctlptl with both minikube and kind. Here's the CI config.

But I wasn't totally happy! Most of our team is more comfortable managing containers than managing VMs. VMs are slower. Upgrading dependencies is more heavyweight. We wondered: can we make this work in containers?

Strategy #3: Local Registry On Remote Docker

The last approach (and the one we use in most of our projects) uses some of the tricks that kubeadm-dind-cluster uses.

The CI pipeline:

Creates a container with our code.
Sets up a remote Docker environment outside the container. (This avoids the pitfalls of running Docker inside Docker.)
Starts a kind cluster with a local registry inside the remote Docker environment.
Uses socat networking jujitsu to expose the remote registry and Kubernetes cluster inside the local container.

The socat element makes this a bit tricky. But if you want to fork and hack it, check out this Bash script.

But once it's set up: it's fast, robust, and easy to upgrade dependencies.

Putting it Together

Hacking together this with Bash was the hard part.

Tilt-team maintains ctlptl, a CLI
for declaratively setting up local Kubernetes clusters.

I eventually folded all the logic in the Bash script into ctlptl. As of ctlptl 0.5.0, it will try to detect when you have a remote docker environment and set up the socat forwarding.

The Go code in ctlptl is far more verbose than the Bash script, comparing number of lines. But it includes error handling, cleanup logic, and idempotency, which makes it more suitable for local dev. (CI environments don't need any of this because we tear them down at the end anyway.)

We use image-management tools that auto-detect the registry location from the cluster, which helps with the configuration burden. I like the general trend of Kubernetes as a general-purpose config-sharing system so that tools can interoperate, rather than having to configure each tool individually.

We currently use ctlptl to set up clusters and test the services on real Kube clusters in all of our example projects.

It's been a long journey! But I hope the examples here will make that journey a lot shorter for the next person 🙈.

Shout-outs

CircleCI's remote Docker environment is good!
Thanks to the kind team for working with us on the local registry wiring!
kubeadm-dind-cluster, we salute you as an early adventurer in this problem space!

Kubernetes is so Simple You Can Explore it with Curl

Nick Santos — Thu, 18 Mar 2021 21:51:15 +0000

A common take on Kubernetes is that it's very complicated.

... and because it's complicated, the configuration is very verbose.

... and because there's so much config YAML, we need big toolchains just to handle that config.

I want to convince you that the arrow of blame points in the opposite direction!

Kubernetes has a simple, genius idea about how to manage configuration.

Because it's straightforward and consistent, we can manage more config than we ever could before! And now that we can manage oodles more config, we can build overcomplicated systems. Hooray!

The configs themselves may be complicated. So in this post, I'm going to skip the configs. I'll focus purely on the API machinery and how to explore that API.

Building APIs this way could benefit a lot of tools.

What is the Idea?

To explain the simple, genius idea, let's start with the simple, genius idea of Unix:

Everything is a file.

Or to be more precise, everything is a text stream. Unix programs read and write text streams. The filesystem is an API for finding text streams to read. Not all of these text streams are files!

~/hello-world.txt is a text file
/dev/null is an empty text stream
/proc is a set of text streams for reading about processes

Let's take a closer look at /proc. Here's a Julia Evans comic about it.

You can learn about what's running on your system by looking at /proc, like:

How many processes are running (ls /proc - List the processes)
What command line started process PID (cat /proc/PID/cmdline - Get the process specification)
How much memory process PID is using (cat /proc/PID/status - Get the process status)

What is the Kubernetes API?

The Kubernetes API is /proc for distributed systems.

Everything is a resource over HTTP. We can explore every Kubernetes resource with a few HTTP GET commands.

To follow along, you'll need:

kind - or any small, throwaway Kubernetes cluster
curl - or any CLI tool for sending HTTP requests
jq - or any CLI tool for exploring JSON
kubectl - to help curl authenticate

Let's start by creating a cluster:

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.19.1) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

$ kubectl proxy &
Starting to serve on 127.0.0.1:8001

kubectl proxy is a server that handles certificates for us, so that we don't need to worry about auth tokens with curl.

The Kubernetes API has more hierarchy than /proc. It's split into folders by version and namespace and resource type. The API path format looks like:

/api/[version]/namespaces/[namespace]/[resource]/[name]

On a fresh kind cluster, there should be some pods already running in the kube-system namespace we can look at. Let's list all the system processes in our cluster:

$ curl -s http://localhost:8001/api/v1/namespaces/kube-system/pods | head -n 20
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/namespaces/kube-system/pods",
    "resourceVersion": "1233"
  },
  "items": [
    {
      "metadata": {
        "name": "coredns-f9fd979d6-5zxtx",
        "generateName": "coredns-f9fd979d6-",
        "namespace": "kube-system",
        "selfLink": "/api/v1/namespaces/kube-system/pods/coredns-f9fd979d6-5zxtx",
        "uid": "a30e70cc-2b53-4511-a5de-57c80e5b68ad",
        "resourceVersion": "549",
        "creationTimestamp": "2021-03-04T15:51:21Z",
        "labels": {
          "k8s-app": "kube-dns",
          "pod-template-hash": "f9fd979d6"

That's a lot of text! We can use jq to pull out the names of objects.

$ curl -s http://localhost:8001/api/v1/namespaces/kube-system/pods | jq '.items[].metadata.name'
"coredns-f9fd979d6-5zxtx"
"coredns-f9fd979d6-bn6jz"
"etcd-kind-control-plane"
"kindnet-fcjkd"
"kube-apiserver-kind-control-plane"
"kube-controller-manager-kind-control-plane"
"kube-proxy-sn64n"
"kube-scheduler-kind-control-plane"

The /pods endpoint lists out all the processes, like ls /proc. If we want to look at a particular process, we can query /pods/POD_NAME.

$ curl -s http://localhost:8001/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane | head -n 10
{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "kube-apiserver-kind-control-plane",
    "namespace": "kube-system",
    "selfLink": "/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane",
    "uid": "a8f893b7-1cdb-48fd-9505-87d71c81adcb",
    "resourceVersion": "458",
    "creationTimestamp": "2021-03-04T15:51:17Z",

Or, again, we can use jq to fetch a particular field.

$ curl -s http://localhost:8001/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane | jq '.status.phase'
"Running"

How to unpack what `kubectl` is doing

All of the things above can be done with kubectl. kubectl provides a more friendly interface. But if you're ever wondering what APIs kubectl is calling, you can run it with -v 6:

$ kubectl get -v 6 -n kube-system pods kube-apiserver-kind-control-plane
I0304 12:47:59.687088 3573879 loader.go:375] Config loaded from file:  /home/nick/.kube/config
I0304 12:47:59.697325 3573879 round_trippers.go:443] GET https://127.0.0.1:44291/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane 200 OK in 5 milliseconds
NAME                                READY   STATUS    RESTARTS   AGE
kube-apiserver-kind-control-plane   1/1     Running   0          116m

For more advanced debugging, use -v 8 to see the complete response body.

The point isn't that you should throw away kubectl in favor of curl to interact with Kubernetes. Just like you shouldn't throw away ps in favor of ls /proc.

But I've found disecting Kubernetes like this is helpful to think of it as a process-management system built on a couple straightforward principles:

Everything is a resource over HTTP.
Every object is read and written the same way.
All object state is readable.

These are powerful ideas¹. They help us build tools that fit together well.

In the same way that we can pipe Unix tools together (like jq), we can define new Kubernetes objects and combine them with existing ones.

Sometimes they're silly! Like in this Ellen Körbes talk on how to build a Useless Machine.

In future posts, I want to talk about how to write code that uses these APIs effectively. And how we're leaning into these ideas in Tilt. Stay tuned!

Originally published on the Tilt Dev Blog.

Image Credit: "Curling;--a Scottish Game, at Central Park" by John George Brown. Via Wikipedia.

REST is an old idea (measured on the scale of Internet Time). What I like about Kubernetes' take on REST is that it's less focused on "how do we make sure new API endpoints we define obey the REST gospel", and more focused on "how do we autogenerate APIs from data types?" ↩

The First Modern Mention of Kubernetes

Nick Santos — Thu, 23 Jul 2020 21:14:56 +0000

I found the first mention of Kubernetes in computer science!!

It comes from a book. "Cybernetics: or Control and Communication in the Animal and Machine" by Norbert Wiener. Originally published in 1948. (Yes, even in 1948, non-fiction book titles abused the colon.)

The book has its own Wikipedia page. So many people read it that he published a sequel!

It's surprisingly hard to find. The New York Public Library has two copies offsite, only available on advanced request. The Brooklyn Public Library has zero copies.

I like to imagine I have the only physical copy in pandemic-lockdown New York City! Because right before the lockdown, I borrowed the 1961 edition from a science historian. And she hasn't asked for it back yet 😬.

In the book, Wiener tries to come up with a name for his new field. He writes: "All the existing teminology has too heavy a bias [...] we have been forced to coin at least one artificial neo-Greek expression to fill the gap."

He suggests "Cybernetics," from the Greek word χυβερνήτης, also pronounced "Kubernetes":

Why Cybernetics?

You may think that this is just a pointy-headed in-joke to make the rest of us feel dumb.

But it's actually a very relevant pointy-headed in-joke that only incidentally makes the rest of us feel dumb!

When Wiener published "Cybernetics," the dominant model of computing was finite automata and Turing machines. These are systems that take inputs at the start and produce outputs at the end.

Wiener points out that there are two problems with this model:

1) In any system with lots of closely coupled inputs, we need statistical models to handle the complexity. In chapter one, he compares astronomy versus meteorology. We can count how many stars there are, and can capture the interaction between stars with simple formulas. But in meteorology, there are simply too many particles and the interactions between them are too complex.

2) Information is distributed over time. We can use the time component to learn which inputs cause which outputs, and build feedback loops.

Wiener argues that if computer science does a better job embracing complex statistical systems and time-based feedback loops, we'll be able to better understand lots of non-mechanical systems, like biology and sociology.

The book gets very galaxy-brained to be honest, from "How do we build better thermostats?" to "Could this help us find a cure for Parkinson's disease?"

What Does That Mean For How We Build Systems?

Once you understand this parallel, you see it everywhere!

Consider Kubernetes. Before Kubernetes, you may have had a deploy script. That deploy script worked like a finite automata: look at some inputs, then deploy a server.

One of the key insights of Kubernetes is that when you're working with multiple servers with close coupling, this isn't enough. You need a system with runtime feedback loops to handle the runtime dependencies between servers.

I think this insight applies to developer tools as well. From Make in the 70s to Bazel today, build systems are still stuck in a world of finite automata, mapping inputs to outputs.

But the servers we're building have closely coupled runtime dependencies! Multi-service developer tools need runtime feedback loops as well.

Originally published on the Tilt Blog.