DEV Community: Michael Hausenblas

10 things you didn't know about Kubernetes

Michael Hausenblas — Fri, 25 Sep 2020 12:26:44 +0000

1. Initially written in Java

Yeah, right? According to Kris' FOSDEM 2019 talk.

2. Code named "7"

As per the all-knowing Wikipedia entry:

The original codename for Kubernetes within Google was Project 7, a reference to the Star Trek ex-Borg character Seven of Nine.

I suppose one can't hide the Borg origins ;)

3. One of the fastest growing OSS projects

Kubernetes is one of the fastest growing OSS projects which has changed the industry quite a bit: beyond the bits, a number of concepts and terms that used to be mainly ops folks talk (rolling update deployment, load balancer, network policies, etc.) are now kinda standardized vocabulary that also developers throw around.

4. Lots of distributions to choose from

Although only some 6 years old, almost 70 certified distributions exist. It's kinda early days so still some half of the users (?) roll their own Kubernetes distro. But then, again, we used to roll our own Linux distros well into the 2000s, so there's that.

5. Throwing a Fedora into the ring

Red Hat bet very early on Kubernetes, did a total rewrite of OpenShift on top of Kubernetes. This investment paid off and in the course RH not only established OpenShift as an enterprise-grade Kubernetes distro but also upstreamed a number of (typically security-related) features such as RBAC or the pod security context.

6. From Raspberry Pi to Mainframe

My former Mesosphere colleague Elizabeth K. Joseph, now with IBM, talked about Wait, People Run Kubernetes on Mainframes? in the last pre-Covid KubeCon and Seth Kenlon of Red Hat gave us 5 reasons to run Kubernetes on your Raspberry Pi homelab. So, what will it be for you?

7. Pronunciation is hard …

… and yes, kubectl is pronounced kubecuddle.

8. Inclusivity rulz!

While Kubernetes is up there with Linux in terms of OSS projects in terms of number of contributors or LOC, there's one thing that is pretty unique to Kubernetes: an extremely diverse and inclusive community. The only other community that comes to mind that is equally set up is the Rust community.

9. Lots and lots of upstream

You would not believe what you find when trawling the Kubernetes source code (and what else would you want to do, since we're not travelling anymore, LOL). Some examples:

Websockets to exec into and log from pods.
The venerable secure copy protocol scp as well as the good old tar to copy stuff from and to containers (immutable infra, much?).

10. First commit? First commit!

Thanks to the magic of time travel (erm, Git history) we can see what the first commit was:

First commit 

jbeda committed on 7 Jun 2014
0 parents commit 2c4b3a562ce34cddc3f8218a2c4d11c7310e6d56
Showing  with 47,501 additions and 0 deletions.

Joe was sure busy on that day :)

With that, thanks for stopping by and I'm wondering: what's your favorite thing about Kubernetes that few people know? Care to share?

The cover image uses a photo by Marcin Wichary - originally posted to Flickr, CC BY 2.0.

10 things you didn't know about the terminal

Michael Hausenblas — Thu, 24 Sep 2020 08:49:30 +0000

1. Why still 80 characters default

We regularly use monitors that have a gazillion of pixels available and yet, the default terminal width is 80 characters. Why, oh why?

TL;DR: we owe this one to IBM and their standardized punchcards, introduced in the 1920s, sporting 80 columns:

2. Terminal: not the same as the shell or CLI

Terminal. Shell. Command Line Interface (CLI). Three distinct and different things. Very much related, but they are and do different things: terminal is that window that may host a shell as well as other programs using a textual interface. CLIs are line-oriented, textual interfaces and come in all forms and shapes, for example the aws CLI or when you are looking for something (aka: Google search). Learn more about the differences from this Q&A piece.

3. It's literally the end

The word terminal comes from Latin "terminus", meaning "an end, a limit, boundary line". Not too surprising: from where the mainframe sits, the people working on computer terminals are indeed the end of the communication line.

4. Not the real thing? Emulator much!

Unless you're in a museum, you're not really using a terminal these days. What you're using is a program called a terminal emulator, rather directly emulating a good old video or computer terminal from the 70s. Nice!

5. Tame the beast

Since the terminal emulator is a piece of software, one would expect to be able to configure it to their liking. And indeed there are a number of things you can learn from your terminal's configuration and even set some. Here's what I see when I look at mine (Alacritty on macOS; edited down to only show the beginning):

$ infocmp -L -1                                                                                                                                                                                                                                     
# Reconstructed via infocmp from file: /usr/share/terminfo/73/screen-256color
screen-256color|GNU Screen with 256 colors,
        auto_right_margin,
        backspaces_with_bs,
        eat_newline_glitch,
        has_hardware_tabs,
        has_meta_key,
        move_insert_mode,
        move_standout_mode,
        columns#80,
...

6. Why `vi` and YouTube use `j` and `l`?

Ever used vi and wondered why pressing the j key sends you a line down? Or, likewise, when you're watching a YouTube video and you want to fast-forward some 10 sec by hitting the l key?

Peter has the answer:

When Bill Joy created the vi text editor he used the ADM-3A terminal, which had the arrows on hjkl keys. Naturally he reused the same keys and the rest is history.

So, we can thank Bill Joy's terminal or more specifically its builtin keyboard layout.

7. One terminal is not enough!

As the title suggests and your monitor confirms: one terminal is not enough. Sure, you can launch multiple terminal windows or use tabs or whatever is kewl these days. We elderly people use terminal multiplexer such as tmux (preferred) or screen (if you're really, really old).

8. GPU here I come

You have a GPU I'm sure and you don't know how to keep its cache lines warm? Here's a suggestion: ditch your old terminal and give one of the GPU-powered terminals a try. I've personally migrated from iTerm2 to alacritty and have only good things to say about it (moar YAML for config, yay) but there are also other options like kitty. Might take you a day or so to move over but totally worth it, if you spend more than 50% of your time in the terminal ;)

9. Terminal vs. `tty`

Remember when we talked about software terminals in 4., above? I didn't really tell you the whole truth back then. Now that you're a terminal expert: meet tty which is short for “teletypewriter.” Learn more about What is a TTY on Linux? (and How to Use the tty Command) and thank me later.

10. Terminal vs `stdin`, `stdout`, and `stderr`

In the context of the shell and from programming languages you've almost certainly come across terms like stdin or stdout. Turns out those are simply per-process fixed files (with well-known file descriptors) that you're using to interact with, in the case of stdout … wait for it … your terminal! Read more in Your terminal is not a terminal: An Introduction to Streams.

With that, thanks for stopping by and I'm wondering: what's your favorite thing about the terminal that few people know? Care to share?

Cover image kudos to AT&T Archives, depicting Brian Kernighan rocking an early terminal.

10 things you didn't know about Docker

Michael Hausenblas — Tue, 22 Sep 2020 15:12:33 +0000

You have heard of Docker, right? Maybe done a docker run …, yes? Here are 10 things about Docker you (likely) did not know …

1. Docker is French

To be precise, dotCloud, the original company creating it, had its origins in France. Three well-known names, connected with the original company, in that context are Solomon Hykes, Sebastien Pahl, and Jérôme Petazzoni should help driving this point home.

2. Docker: software or company?

Docker can be either seen as the software or the company.

3. What is OCI and how does it relate to Docker?

In June 2015 Docker (the company) established the Open Container Initiative and many of its specifications are based on or directly derived from the original Docker (software).

4. What is Moby and how does it relate to Docker?

To make the go-to-market and relationship with the software clearer Docker (the company) created Moby, explaining it with:

The components and tools in the Moby Project are initially the open source components that Docker and the community have built for the Docker Project. [...] Docker is committed to using Moby as the upstream for the Docker Product.

5. Docker and Go

Docker (both the software and the company) has contributed greatly to making Go (the programming language) mainstream.

6. Docker running on fruits?

You can use Docker on many different platforms, from Linux to macOS to Windows. My fav has got to be Docker on Raspberry Pi … since 2016, nonetheless(!)

7. Docker in the cloud

Not only can you run Docker on almost any desktop (yeah, I know, laptop) but also on pretty much any cloud. For example, here at AWS it's super easy launching Docker containers using Copilot.

8. Docker the company (part 2)

As of end of 2019, Docker really exists in two companies: Mirantis acquired Docker’s Enterprise business and team and Docker, Inc. focuses on developer tooling.

9. Docker has many ancestors

Starting in 1979 where the chroot system call was introduced to FreeBSD jails and Solaris zones in the 2000s, Docker continued the ops tradition and made it mainstream for devs. Learn more about the history in this nice K2 Cyber Security article.

10. Docker is a world-wide topic

Hundreds of Docker User Groups exist world-wide and the conferences (pre-Covid) were legend. I remember DockerCon 2015 in San Francisco and the excitement around the technology as if it was yesterday.

With that, thanks for stopping by and I'm wondering: what's your favorite thing about Docker that few people know? Care to share?

Cover image kudos to Rubén Bagüés on Unsplash.

A Lesson Learned In Going Serverless

Michael Hausenblas — Tue, 04 Dec 2018 14:44:42 +0000

When I was working on the serverless variant of imgn, a simple image sharing demo app, I ran into an issue around handling multipart form-data POST requests with the API Gateway and AWS Lambda. This post shows how I solved it.

The imgn app is really very simple: it allows you to upload image files and then you can view them in a public gallery. In the background, it extracts some image metadata (just the dimensions, for now) and displays it along with the images:

During the development of the serverless variant of imgn I ran into an issue in the context of uploading the image to S3 using a Lambda function via the API Gateway. You can read up on the issue description in greater detail on StackOverflow, but the essence is: the payload that the API Gateway hands over to the Lambda function gets somehow butchered, so the images end up being corrupted when they land in the respective S3 bucket.

To narrow down where this SNAFU happens, I first replaced my initial code that would create a http.request and use the ParseForm method to get to the image data with my own implementation of parsing the multipart form-data. Still, the corruption was the same.

Next, in order to exclude the "upload to the S3 bucket" part as the culprit for the data corruption, I returned the parsed image data I got from the API Gateway as a result of the Lambda call. This allowed me to use hexdump to compare the resulting file with the original, but still no bueno. The problem persisted.

The only thing left in the request path was the handover of the (binary) image data from the API Gateway to the Lambda function.

After trawling StackOverflow and reading many seemingly relevant posts in AWS fora, I decided it was time for a different approach. I read up on pre-signed URLs and decided to give this a try. These pre-signed URLs enable you to allow people to manipulate objects in S3 buckets and you can set restrictions in terms of what operation, on which object, and for how long you allow it to happen.

So, the idea would be that when a user issues a multipart form-data POST request via the UI, the UploadImageFunction would not directly upload the image to S3 but create a pre-signed URL for a PUT request into said S3 bucket, valid for the file referenced in the previous form-data POST request and restricted to 5 minutes:

req, _ := s3c.PutObjectRequest(&s3.PutObjectInput{
        Bucket: aws.String(gallerybucket),
        Key:    aws.String(imgname),
})
presurl, err := req.Presign(5 * time.Minute)

… and the JavaScript code in the UI would then issue a second PUT request, using the pre-signed URL to directly upload the image into the S3 bucket. And that works like a charm!

I'm still unsure if I should qualify this strategy as a hack or a good practice and would certainly appreciate to hear from other serverless practitioners what they think.

Revisiting PromCon 2018 Panel On Prometheus Long-Term Storage

Michael Hausenblas — Tue, 20 Nov 2018 11:06:26 +0000

While Prometheus itself does not offer a clustered storage solution to store data across multiple machines out of the box, there are a number of so called Long-Term Storage (LTS) options available. In this article we do a high-level review of some the LTS options that were the topic of a PromCon 2018 panel I had the pleasure to witness in person, back in August of this year:

Note that below I won't provide a recommendation which LTS solution you should pick, since it very much depends on your specific requirements and preferences.

Prometheus stores data on local storage, which limits the data you can query or otherwise process to the most recent days or weeks, depending on how much space you have available, locally. If you, however, have the need to retain data for longer time periods, for example for long-term capacity planning, analyzing usage trends, or for regulatory reasons in a certain vertical—think: financial domain or health care—then you likely benefit from a LTS solution. Let's have a look at the offerings discussed in the PromCon panel (in alphabetical order):

Cortex

Cortex provides horizontally scalable, multi-tenant, long term storage for Prometheus metrics when used as a remote write destination, and a horizontally scalable, Prometheus-compatible query API.

See also:

Cortex: open-source, horizontally-scalable, distributed Prometheus, 06/2017
Project Frankenstein: Multitenant, Scale-Out Prometheus: video | slides, 09/2016

InfluxDB

InfluxDB is a time series database designed to handle high write and query loads and is meant to be used as a backing store for any use case involving large amounts of timestamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics. InfluxDB supports the Prometheus remote read and write API.

See also:

InfluxDB Now Supports Prometheus, 09/2017
InfluxDB + Prometheus presentation, 08/2017

M3

M3 is a metrics platform, originally developed at Uber, that is built on M3DB, a distributed time series database and it provides an integration of M3DB with Prometheus.

See also:

Thanos

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity. It can be added seamlessly on top of existing Prometheus deployments and leverages the Prometheus 2.0 storage format to cost-efficiently store historical metric data in any object storage while retaining fast query latencies. Additionally, it provides a global query view across all Prometheus installations and can merge data from Prometheus HA pairs on the fly.

See also:

The new Prometheus storage engine, 06/2018
Introducing Thanos: Prometheus at scale, 05/2018

So to sum up: while Prometheus itself does not support long-term retention of the time series data of interest, there are a number of solutions you can choose from to keep the metrics around for as long as needed. Hope this quick review gives you an idea of some of the available options and can serve as the basis for your own research, should you find yourself in the a situation to have to select one. I wish you successful monitoring and please do share your findings and/or hands-on experiences with above discussed or other LTS solutions not covered here.

Cover image kudos to jesse orrico via Unsplash.

On defaults in Kubernetes RBAC

Michael Hausenblas — Sun, 11 Nov 2018 17:33:19 +0000

In Kubernetes, the Role-Based Access Control (RBAC) method defines a number of default roles and bindings. In this article, we will have a closer look at some of those and discuss where and how to use them.

Let's start with the user-facing cluster roles:

$ kubectl get clusterroles | grep -v '^system'
NAME                                                                   AGE
admin                                                                  33h
cluster-admin                                                          33h
edit                                                                   33h
view                                                                   33h

So we found four cluster roles, admin, cluster-admin, edit, and view available by default in this setup (a Kubernetes 1.11 install). What can you do with them? Using commands such as kubectl get clusterrole/admin -o yaml reveals what each of these roles is about:

Both cluster-admin and admin give read-write access to all (respectively many) resources. In general, if you want to make someone superuser or cluster root, then use the cluster-admin cluster-role with a cluster role binding. Think hard if you want to do that. If your organization sports the responsibility of a namespace admin, you can use either cluster-admin and admin with a role binding and that enables this person then to do everything or mostly everything with all (respectively almost all) resources in the namespace the role binding exists. The only real difference between cluster-admin and admin is that the latter is a little more restrictive when it comes to certain resources such as the namespaces themselves.
The edit role allows one to create, update, delete many of the normal core resources such as deployments, services, or configmaps, however manipulating RBAC permissions is not allowed.
The view role is for read-only access on most resources (besides secrets).

Oftentimes these default roles are a nice starting point for human users. For apps, in reality you will likely end up using tools like audit2rbac to figure out tailored permissions, based on concrete access patterns.

Next, we have a look at the one and only default cluster binding cluster-admin which in my install is defined as follows (edited in the following for clarity by dropping irrelevant fields):

$ kubectl get clusterrolebindings/cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters

Seems that the cluster role binding cluster-admin gives the system:masters group super powers; that's good to know but not directly super useful.

OK, let's move on with service accounts (the identities for non-human users such as Kubernetes components like nodes, controllers, or your own app):

$ kubectl get sa --all-namespaces
NAMESPACE     NAME                                 SECRETS   AGE
default       default                              1         1d
kube-public   default                              1         1d
kube-system   attachdetach-controller              1         1d
kube-system   bootstrap-signer                     1         1d
kube-system   certificate-controller               1         1d
kube-system   clusterrole-aggregation-controller   1         1d
kube-system   coredns                              1         1d
kube-system   cronjob-controller                   1         1d
kube-system   daemon-set-controller                1         1d
kube-system   default                              1         1d
kube-system   deployment-controller                1         1d
kube-system   disruption-controller                1         1d
kube-system   endpoint-controller                  1         1d
kube-system   expand-controller                    1         1d
kube-system   generic-garbage-collector            1         1d
kube-system   horizontal-pod-autoscaler            1         1d
kube-system   job-controller                       1         1d
kube-system   kube-proxy                           1         1d
kube-system   namespace-controller                 1         1d
kube-system   nginx-ingress                        1         1d
kube-system   node-controller                      1         1d
kube-system   persistent-volume-binder             1         1d
kube-system   pod-garbage-collector                1         1d
kube-system   pv-protection-controller             1         1d
kube-system   pvc-protection-controller            1         1d
kube-system   replicaset-controller                1         1d
kube-system   replication-controller               1         1d
kube-system   resourcequota-controller             1         1d
kube-system   service-account-controller           1         1d
kube-system   service-controller                   1         1d
kube-system   statefulset-controller               1         1d
kube-system   storage-provisioner                  1         1d
kube-system   token-cleaner                        1         1d
kube-system   ttl-controller                       1         1d

As you can see, there are tons of control plane components with their own service accounts each and the infamous default service accounts, which you shouldn't be using, always create a dedicated one.

If you're ever in doubt if a certain entity, that is, a user or a service account is permitted to carry out a certain action (like creating a secret), you can use the kubectl auth can-i command to check that, for example:

# let's create a dedicated namespace called secdemo:
$ kubectl create ns secdemo
kubectl create ns secdemo

# let's create a service account nsadmin:
$ kubectl --namespace=secdemo create sa nsadmin
serviceaccount/nsadmin created

# checking if that service account is allowed to create secrets:
$ kubectl --namespace=secdemo auth can-i create secrets \ 
          --as=system:serviceaccount:secdemo:nsadmin
no

# let's give the service account nsadmin the permissions:
$ kubectl --namespace=secdemo create rolebinding nsadmin \
          --clusterrole=admin \
          --serviceaccount=secdemo:nsadmin 
rolebinding.rbac.authorization.k8s.io/nsadmin created

# … and now the service account nsadmin is allowed to 
# create secrets in the namespace secdemo:
$ kubectl --namespace=secdemo auth can-i create secrets \ 
          --as=system:serviceaccount:secdemo:nsadmin
yes

OK, that was it and if you want to learn more about RABC, I recommend to watch this awesome KubeCon 2017 talk:

And if you want some more info on RBAC usage, check out:

Thanks for checking in here and see you around some time soon.

DEV Community: Michael Hausenblas

10 things you didn't know about Kubernetes

1. Initially written in Java

2. Code named "7"

3. One of the fastest growing OSS projects

4. Lots of distributions to choose from

5. Throwing a Fedora into the ring

6. From Raspberry Pi to Mainframe

7. Pronunciation is hard …

8. Inclusivity rulz!

9. Lots and lots of upstream

10. First commit? First commit!

10 things you didn't know about the terminal

1. Why still 80 characters default

2. Terminal: not the same as the shell or CLI

3. It's literally the end

4. Not the real thing? Emulator much!

5. Tame the beast

6. Why vi and YouTube use j and l?

7. One terminal is not enough!

8. GPU here I come

9. Terminal vs. tty

10. Terminal vs stdin, stdout, and stderr

10 things you didn't know about Docker

1. Docker is French

2. Docker: software or company?

3. What is OCI and how does it relate to Docker?

4. What is Moby and how does it relate to Docker?

5. Docker and Go

6. Docker running on fruits?

7. Docker in the cloud

8. Docker the company (part 2)

9. Docker has many ancestors

10. Docker is a world-wide topic

A Lesson Learned In Going Serverless

Revisiting PromCon 2018 Panel On Prometheus Long-Term Storage

Cortex

InfluxDB

M3

Thanos

On defaults in Kubernetes RBAC

6. Why `vi` and YouTube use `j` and `l`?

9. Terminal vs. `tty`

10. Terminal vs `stdin`, `stdout`, and `stderr`