DEV Community: Tim Downey

Easily Produce a Software Bill of Materials with Cloud Native Buildpacks

Tim Downey — Sun, 05 Jun 2022 19:49:23 +0000

Cover Photo by Vadim Sherbakovon Unsplash

It seems like every week we hear that there is a new vulnerability in some core dependency. Or maybe yet another package we all depend on has been hacked (maintainers, please enable two-factor auth!!). The only thing we can do as developers in these cases is patch our software back to a safe version of the dependency as soon as possible. That's easier said than done, however. For many of us even just knowing what dependencies our deployed code is using is a struggle.

That's where having a Software Bill of Materials comes in.

What is an SBOM?

A Software Bill of Materials, aka an SBOM, is a key component in software supply chain security. SBOMs are really just a fancy term for a nested list of all of the dependencies that make up a piece of software. There are a variety of different formats such as Syft JSON, SPDX, and Cyclone DX. Others do a much better job at explaining their content and differences, but at their core all SBOMs contain a list of dependencies and information about each dependency such as its name, version, supplier, cryptographic hashes (checksums), and other unique identifiers. And even if you don't personally care about SBOMs (even though you should!!), the US Federal Government sure does. There is an executive order mandating that all software sold to the US Federal Government include them. So how do we go about generating a Software Bill of Materials for our code?

Producing SBOMs with Buildpacks

There are a number of different utilities that can generate SBOMs for container images after the fact (such as Syft), but I like to get them for free without doing any additional work. 🙃 Plus, it's really nice to generate them at build time since if you're using a compiled language and producing minimal, distroless images it can be difficult to even determine what went into that binary.

That's where Cloud Native Buildpacks come in.

I first became a fan of buildpacks after I pushed my first Rails app to Heroku nearly a decade ago and have continued to appreciate them throughout my time as a Cloud Foundry contributor. Cloud Native Buildpacks (CNBs) take all that is nice about buildpacks from those ecosystems and bring them to the world of containers and Kubernetes. Using the pack CLI I can quickly build a container image capable of running my source code without having to maintain a Dockerfile and have it come with a whole load of best practices already baked in. Among those is the fact that many CNBs have first-class support for SBOMs baked in. By simply building your app with one of these buildpacks it will include a layer in the resulting image that contains Syft, SPDX, and Cyclone DX formatted SBOMs which you can easily access with the pack sbom download command. By baking the SBOM into the image, you can keep both of them together and even sign the image to know that it hasn't been tampered with and exactly what version of the code it belongs to.

Sound cool? While the buildpack docs cover how to do it, let's do a quick demonstration with a real codebase.

Demo

I'll be using the pack CLI to build my images and the dive CLI to inspect them in more depth. Both utilities were readily available for my M1 Mac via brew.

The codebase I'm using is the Cloud Foundry Korifi project -- a set of Golang Kubernetes controllers.

So first thing's first, we've got to clone the repo and cd into its directory. Next (assuming you haven't already configured pack), set the default builder:

pack config default-builder paketobuildpacks/builder:base

This basically is telling pack that you want to use the core set of Paketo Cloud Native Buildpacks with a versatile base OS image. Since this is a Golang project I could have chosen the tiny builder (similar to distroless), but the base builder is more flexible and supports other languages that I use frequently so I like to keep it as my default builder.

Next, build the project! Since Korifi contains the code for other components of this project, we need to set the BP_GO_TARGETS to build the "controllers" binary specifically.

pack build korifi-controllers --env BP_GO_TARGETS="./controllers"

It will take a bit of time to build, but once it's done that's it! We've now got a runnable container image for our Kubernetes controllers with some SBOMs baked in. The pack CLI let's us extract those SBOMs with the following command:

pack sbom download korifi-controllers -o /tmp/korifi-controllers-sbom

This outputted them to the temp directory I specified with -o:

ls /tmp/korifi-controllers-sbom/layers/sbom/launch/paketo-buildpacks_go-build/targets
sbom.cdx.json  sbom.spdx.json sbom.syft.json

As you can see, it output SBOMs in the three main formats: Cyclone DX, SPDX, and Syft.

{
 "artifacts": [
  {
   "id": "f81915beaeb286e0",
   "name": "cloud.google.com/go/compute",
   "version": "v1.6.1",
   "type": "go-module",
   "foundBy": "go-module-binary-cataloger",
   "locations": [
    {
     "path": "controllers"
    }
   ],
   "licenses": [],
   "language": "go",
   "cpes": [
    "cpe:2.3:a:go:compute:v1.6.1:*:*:*:*:*:*:*"
   ],
   "purl": "pkg:golang/cloud.google.com/go/compute@v1.6.1",
   "metadataType": "GolangBinMetadata",
   "metadata": {
    "goCompiledVersion": "go1.18.1",
    "architecture": "amd64"
   }
  },
  {
   "id": "24aa113dfb7228c8",
   "name": "code.cloudfoundry.org/eirini-controller",
   "version": "v0.3.0",
   "type": "go-module",
   "foundBy": "go-module-binary-cataloger",
   "locations": [
    {
     "path": "controllers"
    }
   ],
   "licenses": [],
   "language": "go",
   "cpes": [],
   "purl": "pkg:golang/code.cloudfoundry.org/eirini-controller@v0.3.0",
   "metadataType": "GolangBinMetadata",
   "metadata": {
    "goCompiledVersion": "go1.18.1",
    "architecture": "amd64"
   }
  },
...

As I mentioned earlier, the SBOMs are a part of the image itself so there is no doubt what they belong to and no need to store them separately.

You can see them using the dive utility I mentioned earlier:

dive korifi-controllers

This is a really cool utility in general because it lets you view all of the files that make up each layer in a container image. Here we can use it to navigate to one of the smaller layers at the end that contains our SBOMs.

So that's cool. It also means that if you sign your images (always a good idea) that the SBOMs is conveniently signed as well and this will make it tamper resistant.

Vulnerability Scanning our SBOMs

So what else can we do with those SBOM files? We can scan them for vulnerabilities! For this we can use the grype CLI (like Syft, it is also built by Anchore) and point it at the SBOM files we downloaded earlier.

grype sbom:/tmp/korifi-controllers-sbom/layers/sbom/launch/paketo-buildpacks_go-build/targets/sbom.syft.json --only-fixed

 ✔ Vulnerability DB        [no update available]
 ✔ Scanned image           [0 vulnerabilities]

[0000]  WARN some package(s) are missing CPEs. This may result in missing vulnerabilities. You may autogenerate these using: --add-cpes-if-none
No vulnerabilities found

Grype is really cool and something I'd recommend running in CI periodically against your SBOMs/images. There's always going to be CVEs out there so you can use the --fail-on flag to specify what level you'd like to fail on and the --only-fixed flag to only report CVEs that you can take action on.

SBOMs are Cool

Well there you have it. Software Bill of Materials are currently all the rage when it comes to securing your software supply chain and as we've seen, they're not too difficult to produce! I hope you learned something new about buildpacks and SBOMs from this post, and even if you haven't, thanks for sticking around this long! 😊

Installing Folding at Home on a Raspberry Pi

Tim Downey — Sun, 31 Jan 2021 21:30:32 +0000

tl;dr? Feel free to skip ahead to the installation guide below.

Back when I was a kid I used to run Folding@Home on our old family computer -- a silver Gateway tower with a 1.8 GHz Pentium 4. I'm not sure how I found out about it, but I remember being inordinately excited by the prospect of contributing CPU cycles to simulate protein folding and help discover cures for cancer and other diseases. Yeah, I was a nerd. 🤓

Anyways, I kind of forgot about the project until this past year -- thanks to Covid-19. 😒 In addition to the typical work units, Folding@Home was now distributing work to help researchers better understand the SARS-CoV-2 virus (you can read more on that here!). So once again I did my civic duty and fired up my computers to contribute!

My desktop with its GTX 1070 graphics card chewed through simulations like crazy -- leagues faster than the old Gateway ever could. It got me wondering, though, "How would my Raspberry Pis fare?" I've got a ton of these suckers sitting dormant these days, both from my Pi Kubernetes cluster and the Raspberry Pi fan performance tests that I conducted. Rather than collect dust I thought it would be fun to put them to work (even though it's probably not an efficient use of power). After all, they're probably just as powerful as that old Gateway PC.

Unfortunately, when I first looked, no folding client existed for ARM processors. After all, why would the Folding@Home developers waste time building the software for low power ARM devices when desktop x86 processors and GPUs can run circles around them? I decided to search anyway and forum posts like these started to give me hope. And sure enough, now it is!

Installing Folding at Home on Raspberry Pi

Here's what it takes to start folding on a Raspberry Pi.

64 bit Raspberry Pi

You'll need a Raspberry Pi with a 64 bit processor to be able to run Folding@Home. A 3B or 4 will do nicely. Folding uses around 500MB of RAM so you'll be good with either a 1GB or 2GB model. These days, I recommend the Raspberry Pi 4 in either its 2GB or 4GB configuration for the best bang for your buck.

Install the OS

In addition to your 64 bit Pi, you'll need a 64 bit OS to run the arm64 Folding@Home client. The typical Raspbian/Raspberry Pi OS is 32 bit (since until recently Raspberry Pis did not have more than 4 gigs of RAM), so you'll need to download it specially.

You can find the latest ones here:

I chose to go with the "lite" option since I run my Pis headless. Next install it as you normally would (follow these steps if you need help!).

I usually do a few more things after flashing the OS image to the SD card. Namely, I...

Create a file named ssh in the root directory of the card via touch ssh. This enables ssh on the card for the default pi user (password is raspberry remember to change this!).
Configure WiFi for the Pi by creating a wpa_supplicant.conf file in the root directory. This looks like:

country=US
update_config=1
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev

network={
  scan_ssid=1
  ssid="your-network-SSID-here"
  psk="your-network-password-here"
}

This way I can quickly log on to my router, see what IP it gave the Pi, and connect to it over ssh.

ssh pi@192.168.1.10

Installing the Folding@Home Client

Open up a terminal or ssh on to the Pi and make a new directory in your home directory called fah where we'll download the client (you can name this directory whatever you want).

mkdir ~/fah && cd ~/fah

Now, download the arm64 Folding@Home Debian package. At the time of writing this, the latest version was 7.6.21, but you can find the most recent ones here.

wget https://download.foldingathome.org/releases/public/release/fahclient/debian-stable-arm64/v7.6/fahclient_7.6.21_arm64.deb -O fahclient.deb

Now install it! This will pop up a fancy interface for you to input your username, team number, and passkey (if you've got one).

sudo dpkg -i --force-depends fahclient.deb

It will also ask if you want to run it as a service automatically on start up. If you choose this you'll be able to see an entry for FAHClient.service under your systemd services. You can always change your config later by editing /etc/fahclient/config.xml.

That's it! If you ever want to uninstall Folding@Home from your Pi, just run sudo dpkg -P fahclient.

Folding Performance Comparison

These things get pretty hot under heavy load, so I decided to test out three different configurations to see what kind of cooling was necessary to fold at max power without thermal throttling. I recorded temperature and clock speed stats while folding using the same script that I used in my cooling experiment.

The amount and difficulty of work distributed by Folding@Home is variable so I let the Pis run for two weeks to see what they could accomplish.

Raspberry Pi 3B in Aluminum Flirc Enclosure

First I tried out an old Raspberry Pi 3B that I had lying around. It's in a passively cooled aluminum Flirc Case that does a pretty decent job of keeping the CPU cool under normal usage. Even at 100% CPU usage during folding the Pi only reached 67C. Not that cool, but not hot enough to throttle either.

Specs: Broadcom BCM2837 ARM Cortex-A53 Quad Core CPU at 1.2 GHZ
Results: 9 work units (WUs) completed and 4,179 points (Credit) earned

Raspberry Pi 4B in Aluminum Argon NEO Enclosure

Next I tried one of my Raspberry Pi 4Bs, this one in an aluminum Argon NEO case that's pretty similar to the Flirc case on the 3B. Raspberry Pi 4Bs run both faster and hotter than the 3Bs did so I was curious to see if passive cooling was enough. In the cooling experiments I had done, under heavy synthetic load the 4B would have to throttle its CPU in this case. Fortunately, this was not the case for folding. It was able to keep running at a full 1.5 GHz, albeit at a toasty 72C.

Specs: Broadcom BCM2711B0 ARM Cortex-A72 Quad Core CPU at 1.5 GHZ
Results: 24 work units (WUs) completed and 19,648 points (Credit) earned

Raspberry Pi 4B with Tower Cooler and 40mm Fan

My last experiment used another Raspberry Pi 4B with a ridiculous cooling tower. This one did great during the cooling experiments so I expected it to perform decently for folding as well. Sure enough, it did. It was able to run at a steady 1.5 GHz and keep at a balmy 42C. This one could probably be overclocked to 1.6 or 1.7 GHz in order to eke out a tad more performance, but given the variability of work units it would be difficult to compare. It was able to complete more work than the other 4B, but given that it wasn't being throttled I'll just chalk that up to luck.

Specs: Broadcom BCM2711B0 ARM Cortex-A72 Quad Core CPU at 1.5 GHZ
Results: 27 work units (WUs) completed and 22,642 points (Credit) earned

Summary

So, to wrap it all up, a modern desktop CPU and a GPU could smoke these Pis... and that's ok. I had fun waking up these long dormant little computers and that was worth it to me. If you've got an unused Pi sitting around consider putting it to use for a little bit as well. Maybe it will help cure something! 😊

Capturing Network Traffic from a Kubernetes Pod with Ephemeral Debug Containers

Tim Downey — Sun, 27 Sep 2020 17:24:21 +0000

Running tcpdump against a Running Pod

The other day I had a situation where I needed to debug network traffic between an app and its Envoy sidecar proxy. Fortunately, since the app image was Ubuntu-based and it was an unimportant dev cluster, I was able to just kubectl exec into a shell on the container and apt install tcpdump.

Now that I had tcpdump installed, I could run it and pipe the output to Wireshark on my local machine.

kubectl exec my-app-pod -c nginx -- tcpdump -i eth0 -w - | wireshark -k -i -

It was pretty slick, if I do say so myself, and made me feel like a Hackers character. 😎

There's some issues with this, though. 😳

I had to kubectl exec and install arbitrary software from the internet on a running Pod. This is fine for internet-connected dev environments, but probably not something you'd want to do (or be able to do) in production.
If this app had been using a minimal distroless base image or was built with a buildpack I wouldn't have been able to apt install.
If I rebuilt the app container image to include tcpdump that would have required the Pods to be recreated. Not ideal if the bug is tricky to reproduce.

So installing tcpdump as needed isn't always an option. Why not just include it when building the initial container image for the app so that it's always available? That path leads to image bloat and the more unecessary packages we include in our image the more potential attack vectors there are.

So what else can we do?

Fortunately for us, newer versions of Kubernetes come with some alpha features for debugging running pods.

Ephemeral Debug Containers

Kubernetes 1.16 has a new Ephemeral Containers feature that is perfect for our use case. With Ephemeral Containers, we can ask for a new temporary container with the image of our choosing to run inside an existing Pod. This means we can keep the main images for our applications lightweight and then bolt on a heavy image with all of our favorite debug tools when necessary.

For the following examples I'll be using my mando app which is running as a Pod named mando-655449598d-fqrvb. It's built with a Go buildpack (you can read more on that here), so it's the perfect example of an app with a minimal image.

To demonstrate how this can be hard to work with, let's first try to open a shell in it the traditional way.

kubectl exec -it mando-655449598d-fqrvb -- /bin/sh

error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "3ca55f9b6457995be6c6254a8d274706e42d89f431956b5b02ad9eade5e5f788": OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown

No /bin/sh? That's rough. Let's provide a shell with an Ephemeral Container using the busybox image:

kubectl alpha debug -it mando-655449598d-fqrvb --image=busybox --target=mando -- /bin/sh

If you don't see a command prompt, try pressing enter.
/ # echo "hello there"
hello there
/ # ls
bin   dev   etc   home  proc  root  sys   tmp   usr   var

Now we can do all sorts of shell-like activities!

For our use case, though, we want to capture network packets. So let's use an image that's optimized for network troubleshooting: nicolaka/netshoot.

We can use kubectl alpha debug with run tcpdump and pipe the output to our local Wireshark just as we'd done before!

Here's a concrete example of me using tcpdump to capture packets on the eth0 interface:

kubectl alpha debug -i mando-655449598d-fqrvb --image=nicolaka/netshoot --target=mando -- tcpdump -i eth0 -w - | wireshark -k -i -

Since I'm using Istio as my service mesh, capturing packets from eth0 primarily shows traffic to and from the Envoy sidecar proxy. If we want to debug traffic between the proxy and the mando app itself we can do the same thing against the lo (loopback) interface:

kubectl alpha debug -i mando-655449598d-fqrvb --image=nicolaka/netshoot --target=mando -- tcpdump -i lo -w - | wireshark -k -i -

I've found both of these commands to be invaluable when debugging the service mesh interactions on my clusters.

Caveats

As of Kubernetes 1.19, Ephemeral Containers are still an alpha feature and are not recommended for production clusters. So chances are you won't have access to them yet in your environment. As an alpha feature, the interface and functionality of the feature is liable to change, so don't get too attached to the current implementation! It's such a useful feature, however, I'd doubt they'd cut it entirely.

There are still ways to get early access today, though.

If you're using a managed Kubernetes service like GKE you can create an (unsupported) alpha cluster that will have all sorts of experimental features enabled. I'm less familiar with other managed offerings, but chances are they'll have some form of alpha release channel was well.

Or if you're running a local kind cluster, BenTheElder shows how you can enable ephemeral containers here with the following kind config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  EphemeralContainers: true

If you're using a custom kubeadm deployed cluster, you can configure the Kubernetes control plane components to run with the --feature-gates=EphemeralContainers=true flag.

Alternatives

If for some reason you can't enable ephemeral containers and you really want to capture some packets, don't despair! 😌

Check out ksniff as an alternative. It can create a privileged Pod that will create a new container that shares the same network namespace as your target Pod and let you capture packets that way.

If you can't run privileged pods and can't add tcpdump to the container image yourself, well...

I'm sure you'll figure something out.

Best of luck! 🙂

Cover photo credit: Taylor Vick on Unsplash

Public Routing to a Private Kubernetes Cluster

Tim Downey — Tue, 25 Aug 2020 03:56:42 +0000

For personal projects and hacking around on Kubernetes, I'm a fan of inexpensive managed clusters (like the ones offered by Digital Ocean). For about ten bucks a month, I can spin up a small single-node cluster and hit the ground running. Unfortunately, with these small clusters you only get about a gig of usable RAM and a single vCPU at best. If you want more, prices quickly climb.

At home, I've got a pretty decent Linux box with 12 vCPU (thanks Ryzen!) and 48G of RAM that I've lovingly named tomorrowlan. A similar configuration on Digital Ocean would run about $240 a month! When I'm messing around on personal projects, I really don't care about resiliency or production worthiness. It would be awesome to use my existing hardware while retaining the ability to share what I've built with others on the wider internet.

Thanks to a handful of open-source tools, I can make this dream a reality. With Inlets, Contour, KinD, and external-dns, I can easily spin up new clusters and get them externally routable within seconds.

Components

The diagram above provides an abridged view of how all of these components work together. It honestly looks more complicated than it actually is.

In this architecture, we have a Kubernetes cluster running on hardware running at home. It is behind a home router's NAT and whatever networking nonsense the home ISP is doing. Typically, this cluster would be utterly unreachable without special configuration of the router. Plus, you're most likely are issued a dynamic IP address from your ISP that will change over time. Besides, even if you did set this all up, you probably don't want randoms from the internet connecting directly to your home network.

Instead, we can use an inexpensive cloud VM with a public IP to serve as our entry point. Thanks to all of the components mentioned earlier, we can tunnel traffic from this VM to our private cluster. Now, I'm going to dig into what each of these components is doing. Additionally, I'll demonstrate how they can be used to expose a simple app to the outside world. What I'm not going to do is go in-depth on how to install or configure them. They're all well maintained and continuously updated, so that aspect of it is best left to the official docs.

Inlets

First, we'll start with Inlets since it's arguably the most important piece of all of this. We'll use it to tunnel traffic from a publically reachable cloud instance (I use a $5-a-month Digital Ocean VM) to our internal cluster. Inlets is open-core software, so if you need Layer 4 routing or TLS termination, you'll have to pay for the "Pro" version. For hobbyist use cases like my own, you can make do with what the free version provides. Specifically, we'll be using inlets-operator.

The inlets-operator is a Kubernetes operator that you'll install to the cluster. It continuously watches for new LoadBalancer services and uses inlets to provision what it calls an "exit node" VM on the cloud of your choice and a public IP to expose the LoadBalancer service. In this case, it will be exposing the LoadBalancer service of our Ingress Controller.

Installation

Follow this tutorial for getting started with the inlets-operator.

external-dns

This one is definitely a bit more of a "nice to have." Inlets will give you a public IP address that will work for reaching your apps. Raw IP addresses are pretty unwieldy, however. So instead, if you've got a domain handy, you can configure DNS to map that IP to your domain. For my cluster, I tend to use *.k8s.downey.cloud. This is typically a manual process, but it doesn't have to be!

You can use a project called external-dns to configure DNS for your LoadBalancer services automatically! It's pretty slick. All you've got to do is annotate the service with external-dns.alpha.kubernetes.io/hostname and your desired hostname. Then the external-dns controller will do the rest and configure your DNS for you! Since I'm using a Digital Ocean VM as my Inlets node, I have external-dns set up to configure Digital Ocean's Cloud DNS.

Installation

Installation of external-dns is pretty dependent on what DNS provider you're using. Follow the steps in the README for the cloud you're using. For Digital Ocean I followed these.

Contour Ingress Controller

Using an Ingress Controller with Inlets isn't strictly necessary. You could create a LoadBalancer service for every single app you want to expose, and Inlets will happily create VMs to route to those services. Though with that approach, costs will quickly add up.

With an Ingress Controller we can only expose a single proxy and have that proxy route to all of our apps internally. Any Ingress Controller will work for this purpose, but I like using Contour for several reasons:

It's a Cloud Native Computing Foundation (CNCF) project just like Kubernetes
It uses Envoy proxy, another CNCF project which has been battle-tested at scale and is pretty light on resource consumption
Envoy proxy can hot-reload config changes, so there is no downtime as you add and remove backend apps
It has excellent support for TLS termination out of the box (this is a moot point with the free version of Inlets, however -- you'd need the TCP routing support of inlets-pro -- but in general this is good)

Installation

Follow the Getting Started docs for quickly getting Contour up and running. Contour provides the LoadBalancer service YAML that Inlets will expose. To get it working well with external-dns, I use the following ytt overlay¹ to annotate it with my domain:

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind": "Service", "metadata":{"name": "envoy", "namespace": "projectcontour"}, "spec":{"type":"LoadBalancer"}}),expects=1
---
metadata:
  #@overlay/match missing_ok=True
  annotations:
    #@overlay/match missing_ok=True
    #@overlay/merge
    external-dns.alpha.kubernetes.io/hostname: "*.k8s.downey.cloud"

As an example, you can tweak the installation command slightly to do the ytt templating inline:

kubectl apply -f <(ytt -f https://projectcontour.io/quickstart/contour.yaml -f /tmp/external-dns-overlay.yaml --ignore-unknown-comments=true)

¹ - If you haven't heard of ytt before, I recommend checking it out. It's part of the Carvel suite of Kubernetes utilities and is my go to for transforming YAML that I don't control. Like this Contour installation.

KinD

I use KinD, or Kubernetes in Docker, to run my clusters because I find it super convenient. All it requires is the kind CLI and having a Docker daemon running. It's very quick to spin up and destroy clusters when they're just running as Docker containers.

That said, there's plenty of other valid options here, such as microk8s, minikube, or directly on the OS with kubeadm.

Installation

To install KinD follow their quick start guide.

Demonstration

So, how does this all work together? Let's view it from the perspective of deploying a single app: httpbin.

We can deploy httpbin to the cluster with the following Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

Next we can make it easily reachable from within the cluster by creating a ClusterIP service for it.

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: httpbin

That's cool. We can now reach it at httpbin.default.svc.cluster.local from within the cluster on port 80. This next step isn't necessary, but for fun we can also reach it locally from the by using the kubectl port-forward command.

tim@tomorrowlan:~/workspace/k8s-yaml$ kubectl port-forward service/httpbin 8000:80
Forwarding from 127.0.0.1:8000 -> 80
Forwarding from [::1]:8000 -> 80

With that running you can just curl localhost:8000 and hit the httpbin pods. So far so good!
Go ahead and kill the kubectl port-forward. It's time to make it reachable for everyone else.

We can do this by creating an Ingress resource to let Contour know about our httpbin service and what traffic should be routed to it.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: httpbin
spec:
  rules:
  - host: httpbin.k8s.downey.cloud
    http:
      paths:
      - backend:
          serviceName: httpbin
          servicePort: 80

That's it! We can now access the httpbin app on the internet at http://httpbin.k8s.downey.cloud! 😌

Recap

So what all just happened? Well let's look back at the diagram from before...

Contour came with a LoadBalancer service that points to its Envoy proxy which we annotated with external-dns.alpha.kubernetes.io/hostname: "*.k8s.downey.cloud"
The inlets-operator saw we have a LoadBalancer service that has no IP so it provisioned a cloud VM with a public IP
The inlets-operator oepened a tunnel between an Envoy running on the cloud VM and Contour's Envoy
Since Contour's LoadBalancer service has the external-dns.alpha.kubernetes.io/hostname annotation, external-dns configured DNS to point to the service's public IP
Contour saw the Ingress resource we created for httpbin and configured its Envoy to direct requests going to httpbin.k8s.downey.cloud to the httpbin ClusterIP service
The httpbin ClusterIP service directed the traffic to the httpbin pods
This Rube Goldberg machine resulted in our pods being publicly reachable

What's awesome is that the steps for the httpbin app itself are not actually any different than using a Deployment, Service, and Ingress on a managed Kubernetes cluster! There was a lot that happened behind the scenes, but once it was all set up it basically just works.

We've now had the chance to see one of my favorite aspects of Kubernetes -- how extensible it is! By combining a few building blocks, we're able to quickly replicate the LoadBalancer service experience of a managed Kubernetes cluster on a personal dev machine. I've got 12 vCPU and 48 gigs of RAM at my disposal so I have enough resources to develop non-trivial workloads and I can still demo the end result online. All for the five bucks a month it costs to run the Inlets exit node.

I, at least, think that's pretty cool.

Cheers! 🐢

How to Make Kubectl Exec Run a Command Against Multiple Pods

Tim Downey — Tue, 21 Jul 2020 18:05:21 +0000

I was really surprised to discover the other day that kubectl does not support running the same command against multiple Pods out of the box.
I get why that wouldn't be supported for interactive terminals, but seems like non-interactive commands should be fine.

Oh well. We can still do what we want thanks to UNIX tools like xargs.

kubectl get pods -o name | xargs -I{} kubectl exec {} -- <command goes here>

Just replace the <command goes here> bit with what you want to do.

Example: Setting Log Level to Debug for All Istio IngressGateway Envoys

Here's a real world example of when and how you might want to do this. The other day I was troubleshooting our Istio installation on a dev cluster and needed to set the log level of all of our ingress Envoy proxies to debug. One way to do this is to configure it through a POST request to each Envoy's admin /logging endpoint (on Istio this is on port 15000 by default).

We had five Envoys and I was feeling lazy, so I cooked up the following:

kubectl -n istio-system get pods -l app=istio-ingressgateway -o name | xargs -I{} kubectl -n istio-system exec {} -- curl -s localhost:15000/logging?level=debug -X POST

I'm sure there are other UNIX incantations that could do the same, but this got the job done for me and I'm proud of it. 😊👍

How to Make Kubectl Jsonpath Output On Separate Lines

Tim Downey — Mon, 20 Jul 2020 01:41:12 +0000

Getting kubectl to output jsonpath results on separate lines is something that I have to Google every time. 😖

For example, the following command extracts the podIP for every running Pod across all namespaces.

kubectl get pods -A -o jsonpath='{.items[*].status.podIP}'

It returns something like the following:

10.244.0.11 10.244.0.8 10.244.0.14 10.244.0.10 10.244.0.6 10.244.0.12 10.244.0.13 10.244.0.15 10.244.0.7 10.244.0.9 10.244.0.3 10.244.0.2 10.244.0.5 172.18.0.2 172.18.0.2 172.18.0.2 172.18.0.2 172.18.0.2 172.18.0.2 10.244.0.4

That's not the friendliest output to work with, that's for sure. 🙅‍♀️

Adding New Lines

You can use the jsonpath range function to iterate over the list and tack on a new line after each element with {\n}.

kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}'

This outputs:

10.244.0.11
10.244.0.8
10.244.0.14
10.244.0.10
10.244.0.6
10.244.0.12
10.244.0.13
10.244.0.15
10.244.0.7
10.244.0.9
10.244.0.3
10.244.0.2
10.244.0.5
172.18.0.2
172.18.0.2
172.18.0.2
172.18.0.2
172.18.0.2
172.18.0.2
10.244.0.4

Awesome! Now we can work with the output using all sorts of standard UNIX utilities that operate on new line (e.g. sort, xargs, uniq, etc.).

Bonus

You can use other whitespace characters too. So imagine we wanted to print the Pod namespaces/names along with their IPs and separate them by a comma.

$ kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{","}{.status.podIP}{"\n"}{end}'

default/fah-cpu-7c66fc7948-582sr,10.244.0.11
default/fah-cpu-7c66fc7948-c9xb5,10.244.0.8
default/fah-cpu-7c66fc7948-dlm5z,10.244.0.14
default/fah-cpu-7c66fc7948-g25cb,10.244.0.10
default/fah-cpu-7c66fc7948-g2svf,10.244.0.6
default/fah-cpu-7c66fc7948-hxmfn,10.244.0.12
default/fah-cpu-7c66fc7948-jxkp8,10.244.0.13
default/fah-cpu-7c66fc7948-n7rvt,10.244.0.15
default/fah-cpu-7c66fc7948-txvpg,10.244.0.7
default/fah-cpu-7c66fc7948-vzpbz,10.244.0.9
default/mando-57fff9d5f5-rdxrx,10.244.0.3
kube-system/coredns-66bff467f8-r9g25,10.244.0.2
kube-system/coredns-66bff467f8-xfd5k,10.244.0.5
kube-system/etcd-kind-control-plane,172.18.0.2
kube-system/kindnet-g6jvd,172.18.0.2
kube-system/kube-apiserver-kind-control-plane,172.18.0.2
kube-system/kube-controller-manager-kind-control-plane,172.18.0.2
kube-system/kube-proxy-9t7tt,172.18.0.2
kube-system/kube-scheduler-kind-control-plane,172.18.0.2
local-path-storage/local-path-provisioner-bd4bb6b75-zdv22,10.244.0.4

Outputting in jsonpath can be pretty handy!

Though I'll still have to look up how to do it everytime. 😌

The Great Raspberry Pi Cooling Bake-Off

Tim Downey — Sat, 04 Jul 2020 19:28:19 +0000

Two years ago, I found myself alone over the Fourth of July and a bit bored. So, of course, I did the reasonable thing and conducted a performance comparison of a few of the cooling options available for the Raspberry Pi 3B. This comparison resulted in my post "Raspberry Pi Heat Sink Science".

2020 and its Pandemic has gifted me with even more isolation (ahem social distancing 😷) this Fourth, so why not make the best of it and create Raspberry Pi Heat Sink Science Episode II: The 4th Awakens! This time with Raspberry Pi 4Bs! Raspberry Pi 4s have way more RAM (up to 8GB) and also run faster than their predecessor -- their quadcore ARM A72 can be overclocked to upward of 2GHz. Unfortunately, they also need more power and run much hotter.

I'm super excited about having more RAM, so I'm planning on modernizing my Raspberry Pi Kubernetes Cluster for 2020, but want to dig into just how much hotter these things get since that might affect my build design. I also just wanted an excuse to buy a ridiculous cooling tower for a Raspberry Pi. 😅

Configurations and Components

I will be testing a Raspberry Pi 4B 4GB under the following configurations.

Full disclosure: these are Amazon affiliate links, so if you buy anything from them I'll receive a small percentage as compensation. It might help me recoup some of the costs of these experiments. 🙏

No Heatsink
Heatsink (with Thermal Paste)
Heatsink + 30mm Fan
Power over Ethernet (PoE) Adapter with 25mm Fan
Argon NEO Case
GeeekPi Cooler Tower Heatsink + 40mm RGB Fan

Because having a stable power source matters when overclocking the Raspberry Pi 4, I chose to use an 18 Watt/3.5 Amp power supply by Argon that had pretty good reviews. For the Power over Ethernet experiment, I used a tp-link 4-port PoE switch.

Methodology

I started testing with a stock Raspberry Pi 4 using the latest firmware. At 1.5 GHz it was incredibly stable, even with no heatsink. This was boring. So the first thing I did was overclock my Pi to 2.0 GHz. This was surprisingly easy and stable with a decent power supply.

Benchmarking the Raspberry Pi 4

In my previous post, I used the stress tool to create synthetic load on the Raspberry Pi's CPU. This time I wanted to do something slightly more real and opted to use sysbench and its prime number verification test to create CPU load.

sysbench --test=cpu --cpu-max-prime=50000 --num-threads=4 run

Measuring Temperature and CPU Clock Frequency

While running this benchmark, I ran this script in the background to output a CSV containing the temperature and CPU frequency for every second of the experiment. Feel free to check out the script itself, but the important bits are the vcgencmd measure_temp command for getting the Raspberry Pi's temperature and the vcgencmd measure_clock arm command for getting the current clock frequency of the Pi's ARM processor.

The vcgencmd measure_clock arm command was a new one for me. Last time I just used whatever was in the /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq file, but discovered that this is more like a "desired frequency." The vcgencmd measure_clock arm command was giving me the actual frequency, which is useful for detecting thermal throttling¹.

$ vcgencmd measure_temp
temp=34.0'C

$ vcgencmd measure_clock arm
frequency(48)=2000478464

To chart the results I used Python and friends: Pandas, Matplotlib, and Seaborn. If you're curious, you can check out my (pretty rough) plotting code here.

The benchmarks typically took around three minutes to complete (completion time is indicated by a vertical dashed line on the graphs), and measurements were collected for five minutes to see how quickly the Pi cooled back down. So without further ado, let's look at some results!

¹ - The Raspberry Pi 4 will thermal throttle, or slow down its CPU, when its temperature is between 80-85°C, so this clock frequency measurement can tell us just how much!

Experiment 1: No Heatsink

The first experiment I ran was with the Pi in its default configuration: no heatsink whatsoever. Just the processor's heat spreader. Unsurprisingly this configuration fared the worst.

The sysbench CPU benchmark took around 216 seconds to complete -- almost 40 seconds longer than the other configurations -- and the Pi exhibited significant thermal throttling. Notice how the CPU frequency oscillates between 2.0 and 1.3 GHz.

Temperatures started out fairly high at around 50°C and quickly reached the throttle zone. Without any cooling the Pi never reached its base temperature during the five minute measurement window.

Noise levels: Silent! It's not cooled at all!

Experiment 2: Heatsink (with Thermal Paste)

For the second experiment I used a cheap aluminum heatsink and some inexpensive Cooler Master thermal compound to affix it to the CPU.

The benchmark took around 181 seconds to complete -- significantly faster than the previous experiment.

Temperatures started out much lower at around 40°C. They still reached the thermal throttle zone, but it took much longer to get there. The throttling didn't begin until 150 seconds into the experiment, whereas with no heatsink throttling started just 60 seconds in. This is a significant improvement and shows that if your Pi is typically used with short, bursty workloads you can likely get by with just a simple heatsink. If your Pi is running sustained CPU-intensive workloads, however, the heatsink alone won't suffice.

🙉 Noise Levels: Silent! It's passively cooled.

Experiment 3: Heatsink + 30mm Fan

For the third experiment I kept the same heatsink on and added a cheap 30mm Raspberry Pi fan that I ordered. Unfortunately, the fan came with no mounting hardware so I had to improvise with some rubber bands. It's not pretty, but it got the job done.

The benchmark took a little over 178.6 seconds to complete. A few seconds faster than with the heatsink alone and basically the same result as the following tests.

Temperatures were much improved with just this simple fan. We started at 37°C and reached 64°C at our highest point. There was no thermal throttling and the Pi was able to rapidly cool itself back down. If I actually had a case to mount this fan on, I could probably stop here. But I don't, and we've got more experiments to run!

🙉 Noise Levels: Medium. The 30mm fan was pretty quiet. There was a slight hum, but nothing too bad.

Experiment 4: Power over Ethernet Adapter + 25mm Fan

The fourth experiment was probably the most practical one for me. As I mentioned earlier, for my next Kubernetes cluster I'm planning on powering the Pis with Power over Ethernet (PoE) so that I don't have to mess around with a handful of expensive power supplies. The PoE adapter (aka PoE Hat) comes with a built in 25mm fan. Since my cluster will have multiple Pis in close proximity it's important that this adapter can keep it cool.

Like before, the benchmark took a little over 178.6 seconds to complete. It was a few milliseconds faster than the 30mm fan, but since I only ran the benchmark once this isn't enough data to claim it was absolutely faster.

Temperatures were great -- we maxed out at around 62°C with no thermal throttling. I'm pretty confident now that this will do fine in a clustered-configuration.

🙉 Noise Levels: Medium-Loud. The 25mm fan was fairly loud and pretty whiny. In a cluster I could see this definitely getting annoying. So if you're using Raspberry Pis because they're quiet... you might want to reconsider if you plan on using the PoE adapter. I mean, it's not that bad, but if you are running it in a small dorm or apartment I could see it being a nuisance.

Experiment 5: Argon NEO Case

For the fifth experiment I tested a case that I purchased for its aesthetics. The Argon NEO case is made of aluminum and pretty slim. For $15 it feels really high quality and makes the Pi look like a "real" computer!

Internally the Argon NEO is affixed to the Pi's CPU with a piece of included thermal tape. This connects it to an aluminum column that lets it radiate heat throughout the entire case. Effectively making it into a massive heatsink! The results speak for themselves. We start at 34°C and make a very gradual climb as the benchmark runs. It tops out at 58°C -- our best performance yet!

I thought for sure that the small size of the case would inhibit air flow and cause it to get pretty hot, but I was pleasantly surprised with how well it performed². For my Raspberry Pi 3 I've been using a Flirc Case that I've been pretty happy with. This one, however, definitely gives it a run for its money.

🙉 Noise Levels: Silent! It's passively cooled.

² - I ran a longer experiment later and the Argon NEO was not able to keep a Pi cool when under load for hours at time. See the update at the bottom of the post for more details.

Experiment 6: Cooling Tower + 40mm RGB Fan

For the sixth and final experiment I got to test out my fun new toy: a cooler tower with a 40mm RGB fan that Amazon had recommended to me (they know me too well). It was a little irritating to assemble and mount, but overall not too bad.

As you can clearly see, this configuration performed the best. We started out at ~32°C and reached a peak of 50°C. The large heatsink tower with its (supposedly) copper heat pipes did an excellent job at drawing heat away from the CPU and the large, flashy RGB fan was able to blow all that heat away. Not only that, but it was able to cool the Pi back down extremely quickly once the benchmark had finished.

The only downsides are that it's bulky (about triples the height of the Pi) and cost the most at $22.

🙉 Noise Levels: Quiet. The quietest of the three fan configurations. The fan can also be powered off of the 3.3v port for even quieter operation (at the cost of higher temperatures). Pretty good!

Summary

If all you care about is absolute cooling performance, go with the cooler tower. It looks a bit ridiculous, it's expensive, and it's bulky, but it can sure keep a Pi cool.

However, if you can spare a few degrees and would prefer a more protective and practical case, I strongly recommend the Argon NEO. This case is good if you have short (under 10 minute) bursty load, but not if you expect to run the Pi under sustained CPU load. Otherwise seek something with active cooling.

If you want to get by on a budget, just buy an aluminum heatsink. They only cost a few cents each when purchased in bulk. If you think you're likely to have sustained CPU intensive workloads, spend a few dollars more and get a case with a small fan.

If you're interested in the PoE adapter because you want it for its Power over Ethernet capabilities, know that it will do a fine job at cooling your Pi. However, definitely don't buy it just for the fan! I'm satisfied enough, though, that the PoE adapters will be good enough for my future cluster.

Important Update

2020-07-06

I was impressed with the Argon NEO so chose to use it to run some long-running CPU-intensive tasks. I checked in on it later and just about burned myself on it! 🔥🚒

It was so hot I decided to put it on a silicone coaster to protect the furniture and run some longer tests. Here are the results of it versus the Cooler Tower when run for the better part of a day.

After around 10 minutes of continuous CPU load the Raspberry Pi in the Argon NEO case eventually reached 80°C and reached a max temperature of 86°C. During this time it experience significant thermal throttling and was unable to cool itself back below 80. Once the CPU load was removed it took the case a while to passively cool the Pi back down to reasonable levels.

The Cooling Tower + 40mm fan, on the other hand, only reached a max of 32°C and was able to rapidly cool itself back to the baseline once the experiment stopped.

How are you keeping in touch with your team while working from home?

Tim Downey — Thu, 02 Jul 2020 02:04:06 +0000

Photo by Daniel Cheung on Unsplash

Over these past few months, COVID has turned many of us into remote employees for the first time. This is quite a change for those used to working colocated with their teams from a busy office.

No more group lunches with coworkers, ping pong breaks, or serendipitous chit-chat. Without these things it can be tougher to bond or build camaraderie.

What are some techniques or activities that you or your team has started doing to make up for this?
If you were already working as part of a distributed team before all of this, what has been working for you?

Creating a Simple Kubernetes Debug Pod that Runs Ubuntu

Tim Downey — Sat, 27 Jun 2020 02:15:00 +0000

Originally posted as a note on my blog. Sharing this technique here in case someone may find it helpful.

Sometimes it can be helpful to deploy a simple Ubuntu container to a cluster when debugging. Say you just applied some new NetworkPolicy and want to test network connectivity between namespaces. Or maybe you added a new mutating admission webhook to inject sidecar containers and you need something to test it out with. Or maybe you just want a sandbox container to deploy and play around in.

One thing I like to do is deploy a Pod running Ubuntu that will let me install whatever tools I want. No need to worry about thin, distroless images that are so secure I can't do anything! With the Ubuntu image everything is just an apt install away. 😌

However, it's not as simple as running the ubuntu image on its own. You need to make it actually do something or the container will just exit immediately. Fortunately this is easy enough... just make the container sleep for a long time!

I do this fairly often and hate having to write the YAML from scratch everytime. So this post will serve as a breadcrumb for my future self to find and copy and paste from in the future. 🤞

The YAML

The following YAML will deploy a Pod with a container running the ubuntu Docker image that sleeps for a week. Plenty of time to do what you need!

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  labels:
    app: ubuntu
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always

Applying the YAML

You can apply this via the following by piping stdin to kubectl:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  labels:
    app: ubuntu
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always
EOF

Or you can apply the raw contents of this Gist directly:

kubectl apply -f https://gist.githubusercontent.com/tcdowney/b8a0297241b74f94ef1fc6627f7ea69a/raw/eaae035f5adca37ca00d4a49f1c1958fe3db89e3/ubuntu-sleep.yaml

Using The Pod

Start up an interactive shell in the container:

$ kubectl exec -it ubuntu -- /bin/bash

root@ubuntu:/#

Now you can install whatever you want! For example, I often install curl via the following:

$ apt update && apt install curl -y

What About Ephemeral Debug Containers?

If you've been following along with the latest Kubernetes releases, you may be aware of a new alpha feature in Kubernetes 1.18 known as ephemeral debug containers. This features lets you take a running Pod and attach an arbitrary "debug" container that has all of the tools you might need to debug it. This is really powerful for several reasons:

If a Pod is misbehaving you can attach the container to it and see what's going on directly.
You can continue to follow best practices and publish small container images. No need to include debug utilities "just in case."
No need to look up this page to copy paste some YAML for a hacky Ubuntu sleep pod!

I'm really looking forward to them. However, Kubernetes 1.18 is still pretty bleeding age (at least at the time of writing this post) and the feature is still in alpha. There's also some use cases for the Ubuntu pod that it doesn't cover so this method still has some life in it yet!

Save Money and Skip the Kubernetes Load Balancer

Tim Downey — Sun, 14 Jun 2020 19:29:13 +0000

LoadBalancer Services are super convenient. Just change the type of a NodePort service to LoadBalancer and your cloud provider will provision a new cloud load balancer, external IP address, and firewall rules to make your workload reachable to the world. It's a fantastic user experience! It's no wonder that example installations and tutorials love to include them.

Load balancers, however, come with a cost. On Digital Ocean, for example, each load balancer will run you ten dollars a month -- and for small development clusters this cost can quickly dwarf the cost of the cluster itself.

In this post I'll show how you can save money by skipping the load balancer for your development clusters. We'll see how with a few minor tweaks we can route traffic with an Ingress Controller and plain old fashioned DNS.

The Plan

In place of our cloud load balancer we are going to configure DNS for our domain to round-robin between each of our cluster's worker nodes. This will result in requests to our domain going directly to a worker node. We will install an Ingress Controller on our cluster and have it listen on ports 80 and 443 to handle incoming HTTP(S) traffic and forward it to workloads running on our cluster.

So, in short, we'll do the following:

Install an Ingress Controller as a DaemonSet
Use a NodePort Service instead of a LoadBalancer service
Configure the ingress Pods to listen on hostPort 80 and 443
Configure DNS for our domain to have A records pointing to each of our node external IP addresses
Double check any firewall rules that apply to the worker VMs and ensure ingress traffic is allowed for ports 80 and 443
Create an Ingress resource to expose our app

Installing the Ingress Controller

I have yet to find an Ingress Controller that this doesn't work with. I've done this with Istio, Contour, and NGINX Ingress and they've all been fine. So choose an Ingress Controller and read its documentation on how to install it. To accomplish the steps outlined above, we'll need to tweak the installation YAML somewhat.

Contour can be installed with minimal changes. By default it installs itself as a DaemonSet and already configures the hostPorts to do what we want. So if you're using Contour, feel free to skip ahead to the "Converting a LoadBalancer Service into a NodePort Service" section.

NGINX Ingress Controller and Istio require some extra work. We'll need to convert their ingress proxy Deployments into DaemonSets and configure the hostPort properties for ports 80 and 443. The steps below apply directly to NGINX Ingress, but can be adapted for Istio.

Converting a Deployment into a DaemonSet

First we'll convert the ingress-nginx-controller Deployment into a DaemonSet. Why use a DaemonSet?
Well, for dev environments with a small number of nodes I prefer using a DaemonSet over a Deployment for managing the ingress Pods. This ensures that there will be an instance of our ingress proxy running on all nodes in the cluster and greatly simplifies configuring the DNS. We can just create an A record for every node! Otherwise we'd have to get fancier with placement and keep track of which nodes are configured to be ingress nodes. It's the "cattle not pets" philosophy.

Of course, if you have a large number of nodes it's probably a waste of resources to have them all running ingress proxies -- but if you're operating at that scale you'll probably be using a real load balancer anyway.

So to do this programatically we can use a tool called ytt. We can use ytt to apply the overlay below on top of what the NGINX Ingress Controller provides by default to transform a Deployment named ingress-nginx-controller into a valid DaemonSet.

Configuring Host Ports

We can also use this same overlay to update the containerPorts to include hostPorts so that the container can listen on port 80 and 443. This is necessary because we can't do any fancy port-forwarding with plain DNS, so the ingress proxy needs to be able to listen on the standard HTTP and HTTPS ports.

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind": "Deployment", "metadata":{"name":"ingress-nginx-controller"}}),expects=1
---
#@overlay/replace
kind: DaemonSet
spec:
  #@overlay/match missing_ok=True
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
  template:
    spec:
      containers:
      #@overlay/match by=overlay.subset({"name":"controller"}),expects=1
      - ports:
        #@overlay/match by=overlay.subset({"containerPort":80}),expects=1
        - containerPort: 80
          #@overlay/match missing_ok=True
          hostPort: 80
        #@overlay/match by=overlay.subset({"containerPort":443}),expects=1
        - containerPort: 443
          #@overlay/match missing_ok=True
          hostPort: 443

So if we save this overlay to a file -- let's call it /tmp/nginx-deployment-to-daemonset.yaml -- we can apply it as so:

ytt -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml \
-f /tmp/nginx-deployment-to-daemonset.yaml \
--ignore-unknown-comments=true

Like I said earlier, though. If you're using Contour it's done this already for you so you can skip this part.

Converting a LoadBalancer Service into a NodePort Service

This next step you'll likely need to do regardless of what Ingress Controller you chose. Any pesky LoadBalancer services will need to be transformed into NodePort services. Everything we've done so far will all still work with a LoadBalancer Service, but since the whole point is not to waste money on one, we've got to convert it.

Note: You might be wondering why we needed to configure hostPorts earlier if a NodePort Service can fulfill a similar role. That's an excellent question. The main reason is that by default, most clusters will only allow a small, high-range of ports for the NodePort to select from... typically ports 30000-32767. A cluster admin would explicitly need to allow low port numbers like 80 and 443 and that's pretty uncommon. And probably not a very secure choice. We're converting the LoadBalancer Service into a NodePort mostly for any cluster internal networking that the Ingress Controller may be using it for. All external traffic will bypass the Service and go directly to the nodes.

Anyways, the following overlay will do this for you -- let's save it in /tmp/convert-lb-to-nodeport.yaml.

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind": "Service", "spec":{"type":"LoadBalancer"}}),expects=1
---
spec:
  #@overlay/replace
  type: NodePort

It can be applied the same way as before using ytt.

Note: I go into more detail on how to convert a LoadBalancer Service into a NodePort Service with ytt here.

Validating our Changes

If you'd like to validate that you transformed the Deployment correctly, you can use kubeval.

ytt -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml \
-f /tmp/nginx-deployment-to-daemonset.yaml \
-f /tmp/convert-lb-to-nodeport.yaml \
--ignore-unknown-comments=true | kubeval - --strict

Installing to the Cluster

Now let's apply both overlays together and install to the cluster:

ytt -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml \
-f /tmp/nginx-deployment-to-daemonset.yaml \
-f /tmp/convert-lb-to-nodeport.yaml \
--ignore-unknown-comments=true | kubectl apply -f -

Configuring DNS

You'll want to configure a wildcard A record for the domain you want to be using. How you actually do this depends a lot on your DNS provider, but it's a pretty standard thing so hopefully it's not too rough.

For me, I'm using Digital Ocean to manage DNS for my domain, downey.cloud. I have two nodes in my cluster, each with an external IP (157.245.191.103 and 138.68.53.184). I want to be able to host workloads under the *.k8s.downey.cloud wildcard subdomain so I will need to configure two A records: one for each node.

Once you've done that, you can verify it all works by using a tool like dig or nslookup after waiting for the DNS changes to propagate. You should see something like this:

$ dig example.k8s.downey.cloud

; <<>> DiG 9.10.6 <<>> example.k8s.downey.cloud
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40148
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;example.k8s.downey.cloud.  IN  A

;; ANSWER SECTION:
example.k8s.downey.cloud. 300   IN  A   157.245.191.103
example.k8s.downey.cloud. 300   IN  A   138.68.53.184

;; Query time: 109 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sun Jun 14 10:22:19 PDT 2020
;; MSG SIZE  rcvd: 85

Configuring Firewall Rules

Like DNS, how to do this step is also dependent on your cloud provider. Wherever you configure firewall rules that apply to the worker node VMs in your cluster, you will want to allow incoming TCP traffic for ports 80 and 443. In Digital Ocean, for example, this is done on the "Firewall Rules" page under the "Networking" tab.

Configuring Ingress

Now that we have our Ingress Controller installed and DNS configured, all the pieces are in place and it's time to deploy an app. For example's sake, I'll be deploying a simple Go app called mando (as in Mandalorian).

It has some example Kubernetes deployment configuration prebaked for us:

We can install it with a kubectl apply for each of those files. It's all pretty run-of-the-mill, but let's take a closer look at that Ingress resource.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: mando
spec:
  rules:
  - host: mando.k8s.downey.cloud
    http:
      paths:
      - backend:
          serviceName: mando
          servicePort: 8080

This configures the Ingress Controller to direct traffic to mando.k8s.downey.cloud to the Service named mando on port 8080. So our DNS gets everything going to *.k8s.downey.cloud routed to our ingress proxies and this bit of config gets it to go to the correct backend application.

With this in place, we can now reach our application! Let's try curling it real quick.

$ curl mando.k8s.downey.cloud/this -v

*   Trying 157.245.191.103...
* TCP_NODELAY set
* Connected to mando.k8s.downey.cloud (157.245.191.103) port 80 (#0)
> GET /this HTTP/1.1
> Host: mando.k8s.downey.cloud
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< date: Sun, 14 Jun 2020 17:34:44 GMT
< content-length: 18
< content-type: text/plain; charset=utf-8
< x-envoy-upstream-service-time: 1
< server: envoy
<
this
is the way

Yep, the request went directly to one of our node IPs (157.245.191.103) on port 80 and we can see it was proxied through our Ingress Controller's (Contour in this case) Envoy!

We can create additional Ingress resources for other apps we have deployed, or do fancy things like weighted routing, TLS termination, or path-based routing. That stuff is all dependent on what your Ingress Controller supports, though, and out of scope for this post. 🙃

That's It!

What an achievement! We can now save tens of dollars a month! 🤑

That's not much if your employer is paying for it, but for the small personal clusters that I like to experiment with it basically halves the price I pay. Again, this probably isn't something you'll want to do for running production workloads, but it's definitely a handy technique. Cheers!

How to use kbld to Rapidly Iterate on Kubernetes Deployed Apps

Tim Downey — Thu, 28 May 2020 01:27:55 +0000

Obligatory shipping container cover image by Boba Jovanovic on Unsplash

What is kbld?

When creating applications that extend or interact with Kubernetes, there are times when it's necessary to deploy and develop against a real K8s cluster. While Kubernetes makes it trivial to apply and roll out new changes, the building and pushing new dev Docker images for your application can be a rigamarole. On top of that, you also have to remember to configure the imagePullPolicy for your containers to Always. Otherwise, who knows if the node your app ends up running on has the old image cached!

Fortunately, there is a tool that can help solve all of these problems. The kbld CLI (pronounced "k build") assists with all things around image building and pushing for Kubernetes. It's under active development so refer to the kbld site for the most up to date feature set, but I'm a fan of its ability to do the following:

Know where my source code to build lives
Build an OCI image (using Docker or Cloud Native Buildpacks)
Tag the image and push it to the registry of my choice (local registry for KIND, DockerHub, GCR, etc.)
Find references to the image in Kubernetes deployment YAMLs and replace vague references with image digests for deterministic deployments

I'm less of a fan, however, of its terse documentation. So in this post, I'm going to show how I use kbld to build Docker images for my projects.

Understanding kbld Configuration

Like most things in the Kubernetes ecosystem, the kbld CLI is configured by YAML files. There are several options here, but the two main YAML objects I use are Sources and ImageDestinations.

Sources

apiVersion: kbld.k14s.io/v1alpha1
kind: Sources
sources:
- image: image-repository/image-name
  path: /path/to/source/code
  pack:
    build:
      builder: heroku/buildpacks:18

A Sources object declares the images that kbld should be responsible for building. It includes information about the path for the source code of an image as well as configuration for the image builder (docker or pack).

ImageDestinations

apiVersion: kbld.k14s.io/v1alpha1
kind: ImageDestinations
destinations:
- image: image-repository/image-name
  newImage: docker.io/image-repository/image-name

ImageDestinations tell kbld how it should tag and push the images that it has built. It's a pretty simple resource, and I was surprised at first that there was nothing about authentication here for private registries. That config, however, comes in either through your Docker config or as environment variables. See these kbld authentication docs for more information on that.

Are these Kubernetes Resources?

An astute developer might recognize that these kbld resources look suspiciously similar to Kubernetes resource objects and wonder if there are any CRDs involved here. That's not the case, though. The similarities are purely superficial, and these resources are used client-side directly by the kbld CLI.

As always, refer to the kbld config documentation for the latest on what is possible.

How to Use kbld

tcdowney / mando

this is the way

mando

this is the way

just a little app for testing path-based routing

docker run -p 8080:8080 downey/mando

build and deploy

can be built and deployed to Kubernetes with kbld

kbld -f build -f deploy | kubectl apply -f -

View on GitHub

The following examples refer to a simple Go app called mando that will be built and deployed to Kubernetes using kbld.

Building an app with kbld using a Dockerfile

If you wish to follow along, you'll need the following:

Install docker
Sign up for a free DockerHub account or have access to a different image registry
Install kbld
Have access to a Kubernetes cluster and kubectl if you want to deploy

To start, since we'll be publishing to an OCI image registry, we'll first need to authenticate. Since I'm pushing my images to DockerHub that means I just need to docker login.

For the following, I'll be working off of the kbld-dockerfile-example branch of my test app repo.

In this repo, I have an example Deployment for Kubernetes in the deploy directory and the kbld files within the build directory.

---
apiVersion: kbld.k14s.io/v1alpha1
kind: Sources
sources:
- image: downey/mando
  path: .

Here I've configured kbld to build my image, downey/mando, using the code and Dockerfile at the root of my repository.

---
apiVersion: kbld.k14s.io/v1alpha1
kind: ImageDestinations
destinations:
- image: downey/mando
  newImage: docker.io/downey/mando

This ImageDestinations configuration tells kbld to tag and push my image to DockerHub at docker.io/downey/mando.

Now to use this configuration, in the root of the app directory we can run:

kbld -f build -f deploy

We will then see kbld work its magic. It will:

Build the mando app using its Dockerfile
Push it to DockerHub
Update the references to the image in our Kubernetes Deployment to use the digest for the image we just built
Output the Kubernetes YAML with all changes

We can then either write this output to a file or deploy it directly to Kubernetes:

kbld -f build -f deploy | kubectl apply -f -

It might not seem like much at first. But after dozens of cycles of docker build, docker push, updating Kubernetes config to point to a new tag, and deploying, kbld can end up saving a bunch of time!

Where I really find kbld useful though, is with Cloud Native Buildpacks.

Building an app with kbld using Buildpacks

For this section I'll be working off of the kbld-pack-example branch of my test app repo. If you're unfamiliar with the concept of buildpacks, I'd encourage you to learn more about them or check out my blog post on deploying apps to Kubernetes with Buildpacks.

Using Buildpacks instead of a Dockerfile to build is simple with kbld. The flow is pretty much the same -- instead of a Dockerfile, we will use the pack CLI (install it if you haven't already) and make some minor tweaks to our Sources YAML.

---
apiVersion: kbld.k14s.io/v1alpha1
kind: Sources
sources:
- image: downey/mando
  path: .
  pack:
    build:
      builder: cloudfoundry/cnb:tiny

Here we tell kbld which "builder" to use. I'm using the cnb:tiny builder since it's optimized for creating "distroless" lightweight images for Go binaries. Perfect for this use case. If you're unsure which builder to use, you can always run pack suggest-builders to get an up-to-date list of builders from Cloud Foundry and Heroku.

Anyways, as I said earlier, the flow is the same as before. To build and deploy, just run:

kbld -f build -f deploy | kubectl apply -f -

The kbld CLI will now build using pack instead of docker! Since I use the pack CLI pretty infrequently, I'm more than happy to hand over the reigns to kbld and let it orchestrate the build and push.

Summary

Well that's about it. If you've gotten this far, hopefully this post has helped demystify some of the basic use cases for kbld and how it can help streamline the Docker image push-build-deploy flow. If you're looking to learn more, check out the kbld docs or join the #k14s channel on Kubernetes Slack. Good luck! 🌝

Web Monetization Sidecar Proxy [Grant For The Web Hackathon Submission]

Tim Downey — Mon, 25 May 2020 00:22:36 +0000

What I built

A simple proof-of-concept sidecar proxy¹ that injects Web Monetization meta tags into HTML responses from the backend application. Offloading this responsibility to a sidecar is valuable for several reasons:

It allows applications to benefit from Web Monetization without having to change the code of the application itself. Useful for cases where there are many applications being deployed or when the application is not modifiable (e.g. commercial off the shelf software).
Platform operators could automatically deploy these sidecars across their entire cluster without application developer intervention.

While the proxy I built for this Hackathon is far from production ready, I hope that this proof-of-concept can inspire further work. One idea might be writing a WASM filter for Envoy proxies that does this work.

Included with the code is an example Kubernetes Deployment that showcases how the proxy can easily be added to an existing Pod as a sidecar container.

¹ - What is a sidecar proxy? It is a sidecar container that runs a proxy sitting in front of your "main" application container. Sidecars are frequently used by service meshes to enable features like intelligent load balancing, rate-limiting, certificate / TLS management, and more. Envoy is a commonly used sidecar proxy.

Submission Category:

Foundational Technology

Demo

Demo: https://monetization-proxy.k8s.downey.dev/

I've deployed a sample Ghost blog running with the monetization proxy sidecar to a small development Kubernetes cluster running on Digital Ocean². The site can be accessed at here. Additionally, you can view the Kubernetes deployment configuration here.

² - This cluster costs money so I will tear it down in a few weeks.

Link to Code

tcdowney / web-monetization-proxy

Simple Go proxy for injecting Web Monetization meta tags. Done as part of the Dev "Grant For The Web" Hackathon

web-monetization-proxy

Docker repo: https://hub.docker.com/repository/docker/downey/web-monetization-proxy

Simple proxy for injecting Web Monetization meta tags. Intended to be deployed as a sidecar process alongside Web Monetization unaware webapps.

disclaimer

This proof-of-concept project was created for fun as part of the DEV "Grant For The Web" Hackathon. It should not be relied on for production use cases and merely exists to demonstrate what is possible. In the future it might be worth exploring doing something similar using Envoy proxies and a WASM plugin that implements this functionality.

configuration

The proxy relies on the following environment variables:

PROXY_PORT -- the port the proxy listens on
BACKEND_PORT -- the port of the backend application that requests are forwarded to
PAYMENT_POINTER -- an Interledger Payment Pointer string

Reference the example Deployment to see how you might configure these in Kubernetes.

development

This project uses Go modules which work best with Golang 1.13+. To run the project's…

View on GitHub

The proxy is available as a Docker image here.

How I built it

The proxy is built using Go and uses an httputil.NewSingleHostReverseProxy to do a lot of the heavy lifting around proxying. A response modifying function is applied to each response from the backend and does the following:

Checks to see if the response Content-Type is HTML. If it is not, we leave the response alone since we don't want to inject meta tags into Javascript or CSS responses!
If it is HTML, we parse the DOM and inject the Web Monetization meta tag into the document head.

For unit testing I used https://github.com/sclevine/spec which provided some lightweight BDD goodies to Go's built-in testing library. It was my first time using it (I traditionally have used Ginkgo and Gomega) and I found it to be easy to work with.

For building the OCI images I used the Go Cloud Native Buildpack and kbld to streamline the build/push/deploy to Kubernetes flow. More details on that are in the project README.

To deploy the demo I set up a Kubernetes cluster on Digital Ocean. I used NGINX as my Ingress controller and set up automatic provisioning of TLS certificates using Cert-Manager and LetsEncrypt.

Additional Resources/Info

This was a fun little project for me. I got to write some Go, set up a a new K8s cluster, and play around with some new libraries / tools.

One thought I had while working on this though was "I sure wish there was just a Web Monetization HTTP header" instead of having to rely on meta tags. This would have let me just use an off-the-shelf sidecar proxy like Envoy which supports header manipulation and would have opened it up to support non-HTML applications.