DEV Community: Chuck Ha

Data types in Kubernetes: PriorityQueue

Chuck Ha — Mon, 22 Apr 2019 14:57:32 +0000

Kubernetes Version: v1.13.2

The Kubernetes scheduler (kube-scheduler) is the component of Kubernetes that is responsible for assigning pods to nodes. This is called scheduling and thus the name kube-scheduler. The kube-scheduler has a feature called pod priority and preemption.

Pod priority and preemption allows pods to be assigned a priority value in the pod spec. During pod scheduling, Kubernetes will take the priority into account. Pods with a higher priority will be scheduled ahead of lower priority pods. Additionally, lower priority pods can be evicted in favor of a higher priority pod in low resource situations.

First, let's look at how the kube-scheduler works. The kube-scheduler has access to a queue of pods that need to be scheduled. Whenever a pod is created or modified, the pod is added to the kube-scheduler's pod queue. The kube-scheduler waits for pods to exist on the pod queue, dequeues a pod and schedules it.

The kube-scheduler's pod queue can either return pods in the order in which the pods joined the queue or based on the pod's priority. In the case of first-in-first-out, if 5 pods are created and need to be scheduled, the kube-scheduler may schedule the first 4 and then run out of resources for the last pod. If that last pod was the most important pod to schedule then you are out of luck. You'll have to delete some pods until you have enough resources to allow the scheduler to schedule it.

If the kube-scheduler's pod queue is based on priority then you have the ability to assign priorities to pods. If the same 5 pods get scheduled but each have a different priority, the highest priority will be scheduled first, or, if it comes in after the other 4 have already been scheduled and there are no more resources, the kube-scheduler will try to move the lower priority pods around until there are enough resources on a node for the highest priority pod to be scheduled.

The kube-scheduler is able to swap out the implementation of the pod queue because it is abstracted behind an interface. If pod priority is disabled then the kube-scheduler will use a plain first-in-first-out queue as the data structure to satisfy the scheduling queue interface. However in the default case where pod priority is enabled, the kube-scheduler will use a priority queue to implement the scheduling queue.

A priority queue has the same interface as a regular queue:

type Queue interface {

    // Put an item on the queue
    Enqueue(interface{})

    // Remove an item from the queue
    Dequeue() interface{}
}

The difference is entirely in the underlying implementation.

In Go it's common to see a first-in-first-out queue implemented with a slice or a channel as the underlying data structure.

If the queue is a priority queue instead of a first-in-first-out queue then a heap can be used as the data structure to implement the queue creating a priority queue.

Go has a built in heap interface defined in the container package. This lets you implement some methods of the heap interface and get the common heap operations implemented efficiently by the Go library authors.

We can see the two queue implementations in the kube-scheduler's code:

To recap from an outside-in-perspective:

The kube-scheduler depends on a SchedulingQueue to get the next pod to schedule.
The SchedulingQueue can be implemented in a variety of ways but by default uses a priority queue (pod priority is enabled).
The priority queue uses a heap to keep a priority-based order of the pods it knows about. This means that when the kube-scheduler gets the next pod to schedule, the pod with the highest priority is returned first.

Please give me feedback if you feel the desire! I'm always looking to improve my technical writing skill.

An Error Wrapping Strategy for Go

Chuck Ha — Tue, 20 Nov 2018 21:07:39 +0000

Does this look output familiar?

error running program: error parsing yaml: error opening file: error no such file "somefile.yaml"

The above is a pseudo-stack trace that you've likely encountered. It's better than just getting error no such file "somefile.yaml" as your error since this at least gives a bit of context as to where the error came from. But it's not great. It's hard to trace the code and not very pretty to read. The code that generated this error might look like this:

func main() {
    if err := run(); err != nil {
        fmt.Printf("error running program: %v\n", err)
    }
}

func run() error {
    if err := parseYAML("somefile.yaml"); err != nil {
        return fmt.Errorf("error parsing yaml: %v\n", err)
    }
    return nil
}

func parseYAML(file string) error {
    _, err := os.Open(file)
    if err != nil {
        return fmt.Errorf("error opening file: %v", err)
    }
    return nil
}

Link to the go playground with the code above.

This method is both prone to programmer error and it can still be a pain to trace the code without line numbers and file information. The improvement that can be made to this code is to record the stack trace!

The code above should look more like this after this improvement:

func main() {
    if err := run(); err != nil {
        fmt.Printf("encountered an error: %v\n", err)
        if stackerr, ok := err.(StackTrace); ok {
            fmt.Println(string(stackerr.StackTrace()))
        }
    }
}

func run() error {
    return parseYAML("somefile.yaml")
}

// A local error type that holds a stack trace
type Error struct {
    Err   error
    Stack []byte
}
func (e *Error) Error() string {
    return e.Err.Error()
}
func (e *Error) StackTrace() []byte {
    return e.Stack
}

type StackTrace interface {
    StackTrace() []byte
}

func parseYAML(file string) error {
    _, err := os.Open(file)
    if err != nil {
        return &Error{
            // keep the original error that was found
            Err: err,
            // Modify it with the stack trace
            Stack: debug.Stack(),
        }
    }
    return nil
}

Link to the go playground with the code above

Now the output specifies exactly what error occurred and where it occurred.

The rules for this refactoring are pretty straightforward:

Wrap errors at the edge of your program.
Print the stack at the top level.

Rule 1

Technically, this rule should be "wrap any error that doesn't already have a stack trace" but since there is no guarantee that libraries will add a stack trace to the errors they return it is easier to assume no library will add a stack trace and to always add one. The standard library is no exception to this rule.

Rule 2

Printing or logging the stack is very important, especially when the system/environment that generated the stack is unaccessible. In the open source world it's very common to get an issue where the submitter copy and pastes the output of the command they ran and the error they encountered. If your error doesn't have a stack trace, good luck trying to get someone to asynchronously rerun your program or a custom binary or with a special flag under the same circumstance. Some people do and that's great, but it would be fantastic if this information were available on every issue.

If you're looking for a library that does this for you, look no further than https://github.com/pkg/errors.

Forcing a kubernetes version during build

Chuck Ha — Mon, 19 Nov 2018 21:06:09 +0000

Date: November, 19th, 2018
Kubernetes: https://github.com/kubernetes/kubernetes/commit/8848740f6d0f84c2c4c5165736e12425551a6207
The current release is just before v1.13.0 so the newest tag is v1.14.0-alpha.

When a Kubernetes binary is built with bazel it will set a default version on the binary for you. That is, when you run kubectl version the version is populated automatically.

However, sometimes you want a custom version, probably for testing purposes. For example, if you are testing kubeadm's upgrade feature it helps to force a version to avoid some annoying behavior.

Bazel calculates this version through a script that is defined by a flag called --workspace_status_command. That argument is specified in the .bazelrc file.

print-workspace-status.sh calls a bash function and prints some version information in a very specific format that bazel uses.

If you want to customize this, you have two options.

Option 1 You can define a custom `--workspace_status_command` script that generates these versions with whatever versions you want

Create this file as workspace-status.sh and make it executable chmod +x workspace-status.sh.

#!/usr/bin/env bash

cat <<EOF
gitCommit $(git rev-parse "HEAD^{commit}")
gitTreeState clean
gitVersion v2.0.0
gitMajor 2
gitMinor 0
buildDate $(date \
  ${SOURCE_DATE_EPOCH:+"--date=@${SOURCE_DATE_EPOCH}"} \
 -u +'%Y-%m-%dT%H:%M:%SZ')
EOF

Now run bazel build --workspace_status_command=./workspace-status.sh //cmd/kubeadm and you can see it works when you run kubeadm version and get this output:

kubeadm version
kubeadm version: &version.Info{Major:"2", Minor:"0", GitVersion:"v2.0.0", GitCommit:"679d4397cfdb386ebd3ae4bcb9972273b3f75ca3", GitTreeState:"clean", BuildDate:"2018-11-19T20:43:30Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}

Option 2

You can define an environment variable, KUBE_GIT_VERSION_FILE, that defines a file in which the versions are already specified.

Create a file called version.txt and put the following contents in it:

KUBE_GIT_COMMIT=abcd
KUBE_GIT_TREE_STATE="clean"
KUBE_GIT_VERSION="v2.0.3"
KUBE_GIT_MAJOR=2
KUBE_GIT_MINOR=0

Now get that file as an environment variable with export KUBE_GIT_VERSION_FILE=version.txt and run bazel build //cmd/kubeadm.

Then when you run kubeadm version you will see:

$ kubeadm version
kubeadm version: &version.Info{Major:"2", Minor:"0", GitVersion:"v2.0.3", GitCommit:"abcd", GitTreeState:"clean", BuildDate:"2018-11-19T20:58:25Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}

Anki 2.1.x Add-ons

Chuck Ha — Sat, 03 Nov 2018 01:38:25 +0000

Installing Anki add-ons is not too bad, but it's slightly different between versions 2.0.x and 2.1.x.

For background, I started here with this Low Key Anki guide and wanted to install one of the provided plugins (ResetEZ.py) but it didn't work out of the box.

Add-ons, in OS X, are found in $HOME/Library/Application Support/Anki2/addons21. Make a directory, I named mine resetez. Inside, add a file named __init__.py. Take the contents of ResetEZ.py found on the link above (maybe a page or two ahead) and put it as the contents of __init__.py. Read the instructions and configure. Restart Anki and you should see a new menu item under Tools!

Customized Kubespray Deployment

Chuck Ha — Wed, 24 Oct 2018 15:23:06 +0000

Versions

kubespray	ansible	python	terraform
this fork	2.7.0	3.6.1	0.11.8

⚠️ Warning! Use this fork: https://github.com/kubernetes-incubator/kubespray/pull/3486.

Motivation

This post serves as documentation for creating a kubernetes cluster from nothing using kubespray on aws with ubuntu images behind a bastion host. I could not find all the documentation that puts all of this together and wanted to write it down for myself the next time I need to do this.

Steps

Clone kubespray and set up some default files

git clone https://github.com/bartlaarhoven/kubespray
cd kubespray
virtualenv ks && . ks/bin/activate
pip install -r requirements.txt
cp -Rp inventory/sample/ inventory/mycluster

Create an IAM user with admin privileges in some account (TODO probably scope this down?)
Create an EC2 key pair
Copy the terraform environment file to credentials.tfvars and modify it with the user's key and secret along with the ssh key pair name and the region you'd like the infrastructure to exist in.
Customize the terraform file with the architecture you'd like, I used 1 master, 1 worker, 1 etcd and left bastions as default. Also modify the inventory file to be ../../../inventory/mycluster/hosts.ini.
Modify the variables.tf to be

    data "aws_ami" "distro" {
      most_recent = true

      filter {
        name = "name"

        values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"]
      }

      filter {
        name = "virtualization-type"

        values = ["hvm"]
      }

      owners = ["099720109477"]
    }

Run terraform

    terraform apply --var-file credentials.tfvars

Modify the ansible.cfg file to use a bastion host by changing the ssh_args value to

    ssh_args = -F ssh-bastion.conf

Modify the hosts.ini file. Use the internal DNS names as the ansible_host for each of the instances in the private subnet and include the ansible_user to be ubuntu, for example:

    kubernetes-devtest-master0 ansible_host=ip-10-250-205-127.us-west-2.compute.internal ansible_user=ubuntu

Modify the bastion lines to be the public DNS names as the ansible_host and include the ansible_user to be ubuntu, for example:

    bastion-0 ansible_host=ec2-22-222-22-22.us-east-2.compute.amazonaws.com ansible_user=ubuntu

Run ansible-playbook

    ansible-playbook -i ./inventory/mycluster/hosts.ini ./cluster.yml -b --become-user=root --flush-cache

If you'd like more logs add -v or -vv up to -vvvvv. I also like to pipe this to tee and write the logs to disk or inspection later in case of failure.

Enjoy your new cluster~!

P.S. if anyone knows how to get the code as part of the item so the bulleted stuff works please comment, I'd love to fix the numbering.

Running dev.to in a container Part 2

Chuck Ha — Wed, 15 Aug 2018 04:12:51 +0000

In my last post I covered building a docker image, but it was so rushed I won't even link it here.

This article aims to improve on that mess of a Dockerfile.

The commit can be found on my fork of the dev.to repo.

In order to get this version of dev.to running locally follow these steps (no cloning necessary):

# Setup a data directory for postgres
mkdir data

# Create the network that is shared between containers
docker network create devto

# Run the database
docker run --rm --network devto --name db -e POSTGRES_PASSWORD=devto -e POSTGRES_USER=devto -e POSTGRES_DB=PracticalDeveloper_development -v $(PWD)/data:/var/lib/postgresql/data postgres:10

and then in another terminal follow these steps for the application server:

# Setup the database and ensure dependencies are met
docker run --rm -it --network devto -e CONNECT_TIMEOUT=30 -e DATABASE_URL=postgresql://devto:devto@db:5432/PracticalDeveloper_development chuckdha/dev.to:latest bin/setup

# Run the webserver
docker run --rm -it --network devto -p 3000:3000 -e CONNECT_TIMEOUT=30 -e DATABASE_URL=postgresql://devto:devto@db:5432/PracticalDeveloper_development chuckdha/dev.to:latest bin/rails s

Then visit http://localhost:3000 in a web browser.

Let me know what you think!

Running dev.to in a container

Chuck Ha — Mon, 13 Aug 2018 04:21:03 +0000

NEW POST, BETTER UX!

Running dev.to in a container Part 2

Chuck Ha

#container#docker#devto#webdev

Original post below here

Dev.to is open source! Awesome! Getting the application running is relatively easy, so props to the dev.to team for good documentation and a clean rails project.

This article walks you through how to get a development version of dev.to running in a container.

The architecture

dev.to requires a web server and a database to run in development mode. It may require more services for production, but that's out of scope for now.

The database

The database needs to exist first as it is a dependency of the app. The database container and the app container will live in the same network so they can communicate over tcp.

The database state will live on disk on our host machine (my laptop in this case). I like to mount a data directory into the postgres container so that I have some persistence when my container restarts. Here is some optional reading that might make this next part a little more clear.

# create the data directory
mkdir data

# create the docker network that will be shared by db and app containers
docker network create devto

# launch postgres
#   `--rm` removes the container when it stops running. This just helps us clean up
#   `--network devto` connects this container to our created network
#   `--name db` names the container and is shown in the output of `docker ps`
#   `-e POSTGRES_PASSWORD=devto` sets an environment variable which will set the password we will use to connect to postgres
#   `-e POSTGRES_USER=devto` sets the env var which will define the user that will connect to postgres
#   `-e POSTGRES_DB=PracticalDeveloper_development` sets the env var that defines the default database to create
#   `-v $(PWD)/data:/var/lib/postgresql/data` mounts the data directory we created above into the postgres container where all the data will live
#   `postgres:10` runs postgres using the latest stable 10 release (example: v10.5)
docker run --rm --network devto --name db -e POSTGRES_PASSWORD=devto -e POSTGRES_USER=devto -e POSTGRES_DB=PracticalDeveloper_development -v $(PWD)/data:/var/lib/postgresql/data postgres:10

The web application

We have to modify the code a little bit to get this working with the following Dockerfile. Here are the changes I made:

Add gem "tzinfo-data" to the Gemfile (I think this is ubuntu related, not 100% sure yet)
Set url: <%= ENV['DATABASE_URL'] %> in the default database configuration in config/database.yaml
Comment out host: localhost in the same file under the test configuration

After I finished this work I found out there is both an issue and a WIP pull request that already exist. The approach presented in this article is a first pass and needs clean up, but I tried to make it clear about what is going on.

FROM ubuntu:18.04

ADD . /root/dev.to
WORKDIR /root/dev.to/

# Set up to install ruby
RUN apt update && apt install -y autoconf bison build-essential libssl-dev libyaml-dev libreadline-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm5 libgdbm-dev

# This uhh helps when you run the container in interactive mode
RUN echo 'export PATH=/root/.rbenv/bin:/root/.rbenv/shims:$PATH' >> ~/.bashrc

# install rbenv-installer
RUN apt install -y curl git && \
    export PATH=/root/.rbenv/bin:/root/.rbenv/shims:$PATH && \
    curl -fsSL https://github.com/rbenv/rbenv-installer/raw/master/bin/rbenv-installer | bash

# install rbenv
RUN export PATH=/root/.rbenv/bin:/root/.rbenv/shims:$PATH && \
    rbenv install && \
    echo 'eval "$(rbenv init -)"' >> ~/.bashrc

# Install gems and yarn
RUN export PATH=/root/.rbenv/bin:/root/.rbenv/shims:$PATH && \
    gem install bundler && \
    gem install foreman && \
    curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
    echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
    apt-get update &&\
    apt install -y yarn libpq-dev && \
    bundle install && \
    bin/yarn

Modify this command below with the correct Algolia keys/app id then build and run the docker image.

docker build . -t dev.to:latest

# setting the various timeouts to large numbers 10000 since the docker version of this app and database tend to be *extremely* slow.
# -p 3000:3000 exposes the port 3000 on the container to the host's port 3000. This lets us access our dev environment on our laptop at http://localhost:3000.
docker run -it --rm --network devto -p 3000:3000 -e RACK_TIMEOUT_WAIT_TIMEOUT=10000 -e RACK_TIMEOUT_SERVICE_TIMEOUT=10000 -e STATEMENT_TIMEOUT=10000 -e ALGOLIASEARCH_API_KEY=yourkey -e ALGOLIASEARCH_APPLICATION_ID=yourid -e ALGOLIASEARCH_SEARCH_ONLY_KEY=yourotherkey -e DATABASE_URL=postgresql://devto:devto@db:5432/PracticalDeveloper_development dev.to:latest /bin/bash

> bin/setup
> bin/rails server
...

Then open up http://localhost:3000 your laptop.

If you have trouble, please leave a comment and I'll update this post!

This is part 1 of a series, getting dev.to running on Kubernetes. Stay tuned for the next article!

Logging approaches in Go

Chuck Ha — Mon, 23 Jul 2018 16:56:21 +0000

During a kubeadm clean up cycle I found a neat feature of the glog library that made me think about various approaches to logging.

This flag is what set off my thought process:

  -log_backtrace_at value
        when logging hits line file:N, emit a stack trace

I really needed this feature for the ticket I was working on, but it's not a feature I've seen in other logging libraries. glog takes the stance that logging should be global. You import the glog library and log whatever lines you need wherever you want. You don't configure glog with code, you configure it at runtime with flags. That's why glog requires you to run flag.Parse() before using it. glog gets the job done quickly as a global logger.

On the opposite end of the spectrum, you have logging as a proper dependency. The logger is a parameter to your function or exists on the struct. Either way, it's configurable via code which makes writing unit tests a lot easier. You don't have to resort to Go's testable examples to make sure your application is logging correctly. This approach is good when you rely on the output to be a certain way or depend on logs for more than just debug output. Check out logrus for a nice library that is easily wrapped to be a proper logging dependency.

As is the answer of almost every tech question, choose the method that works for the project at hand. Likely you'll end up somewhere in the middle and use both types of logging and that's ok. The only important thing is to treat each log type as it should be treated. So go ahead and skip unit testing of global loggers and don't forget to build in extra time for a well tested library by injecting the logging dependency.

Understanding Kubernetes through a concrete example

Chuck Ha — Fri, 13 Jul 2018 03:29:41 +0000

Date: July 12th, 2018
Kubernetes Version: v1.11.0
📝 is a sidebar
⚠️ is a warning

Kubernetes is a container orchestration system. You can think of a container as an application and multiple containers as a pod. The orchestration part means you tell Kubernetes what containers you want running and it will take care of actually running the containers in your cluster, routing traffic to the correct pods, and many other features.

Kubernetes provides a schema for your pod definitions. This means you fill in a well documented template and give it to the Kubernetes cluster. Kubernetes then figures out what will run where, runs your pods and configures the cluster network. If a pod crashes, Kubernetes will notice the system is not in the correct state anymore. Kubernetes will act to return the system to the defined state.

Let's look at an example: A blogging platform called devtoo.com on a Kubernetes cluster.

Components

The first step is to figure out the components of devtoo.com. Let's say these are all the components necessary:

A web server that accepts HTTP traffic from the internet. Examples of web servers include nginx and apache
An application server that loads the rails app into memory and serves requests. This would be the rails application that powers devtoo.com.
A database to store all of our awesome posts. Postgres, mysql and MongoDB are all database examples.
A cache to bypass the application and database and immediately return a result. Examples of caches include redis and memcached.

The end goal

The next step is to figure out what the final system should look like. Kubernetes gives you a lot of choice here. The components could each run in their own pod or they could all be put into one pod. I like to start at the simplest place and then fix the solution if it sucks. To me, that means each component will be run in its own pod. A typical web request will enter the system and hit the web server. The web server will ask the cache if it has a result for that endpoint. If it does, the result is returned immediately. If it does not, the request is passed on to the application server. The application server is configured to talk to the database and generate dynamic content which gets sent back to the web browser.

Defining the system

Kubernetes maps services to pods. There will be one service for each pod. This will allow you to reference other pods with DNS.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: web
spec:
  containers:
  - name: nginx
    image: registry.hub.docker.com/library/nginx:1.15
    ports:
    - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

The service defines a selector, app: web. The service will route traffic to any pod that matches that selector. If you look at the pod definition you will see that there is an app: web label defined on the pod. That means traffic comes into the service on port 80 and gets sent to the nginx pod on the targetPort, also 80 in this case. The targetPort and containerPort must match.

Here, you use your magic wand and produce an nginx config that is embedded in the nginx image that sends traffic to the cache and then if there is no result to the app server.

Here is the cache definition:

apiVersion: v1
kind: Pod
metadata:
  name: redis
  labels:
    app: cache
spec:
  containers:
  - name: redis
    image: registry.hub.docker.com/library/redis:4.0
    ports:
    - containerPort: 6379
---
apiVersion: v1
kind: Service
metadata:
  name: cache
spec:
  selector:
    app: cache
  ports:
  - protocol: TCP
    port: 6379
    targetPort: 6379

And the database definition:

apiVersion: v1
kind: Pod
metadata:
  name: postgres
  labels:
    app: db
spec:
  containers:
  - name: db
    image: registry.hub.docker.com/library/postgres:10.4
    ports:
    - containerPort: 6379
---
apiVersion: v1
kind: Service
metadata:
  name: database
spec:
  selector:
    app: cache
  ports:
  - protocol: TCP
    port: 6379
    targetPort: 6379

Those are all of the dependencies that were considered for this deployment of devtoo.com. Next the application itself must be configured. Rails can use an environment variable to connect to a database. You could define that in the pod YAML like this:

⚠️This is super insecure! Kubernetes has much better ways to do this but I'm omitting them to keep the scope of this post "small".

apiVersion: v1
kind: Pod
metadata:
  name: app
  labels:
    app: app
spec:
  containers:
  - name: devtoo-com
    env:
    - name: DATABASE_URL
      value: postgresql://user1:password1@database/dev_to_db
    image: registry.hub.docker.com/devtoo.com/app:v9001
    ports:
    - containerPort: 3001
---
apiVersion: v1
kind: Service
metadata:
  name: devtoo-com
spec:
  selector:
    app: app
  ports:
  - protocol: TCP
    port: 3001
    targetPort: 3001

The last piece needed is an ingress point, a place where traffic can enter the cluster from the outside world.

📝I'm glossing over IngressControllers because, while required, they are an implementation detail to be ignored at this level of understanding.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dev-to
spec:
  backend:
    serviceName: web
    servicePort: 80

This says that any traffic received at this ingress point will be sent to the service with a name of web on port 80.

Now your cluster is set up, let's trace a packet to get this blogpost. You enter http://devtoo.com/chuck_ha/this-post into your browser. http://devtoo.com resolves in DNS to some IP address which is a load balancer in front of your kubernetes cluster. The load balancer sends the traffic to your ingress point. Since there is only one service on the ingress, the traffic is then sent to the web service which is mapped to the nginx pod. The nginx container inspects the packet and sends it to the cache service which is mapped to the redis pod. The redis pod has never seen this URL before so execution continues from nginx. The request is sent to the application server where this page is generated, cached and returned to your web browser.

Then you click on the 🦄 button!

📝 A list of things I skipped so you could focus on the meat and not get lost in the details:

Can we see analytics for our posts anywhere?

Chuck Ha — Mon, 09 Jul 2018 17:15:28 +0000

I am assuming I can't embed a GA snippet into my post...is it possible to have any insight into the views a post is getting?

Reading Kubernetes logs

Chuck Ha — Sun, 08 Jul 2018 17:32:15 +0000

Date: July 8th, 2018
Kubernetes Version: v1.11.0

Reading logs is part of an essential toolkit that is needed to debug a Kubernetes cluster.

Kubernetes logging

There are generally only two methods needed to find Kubernetes logs, systemd and docker. You will have to identify what is managing the service you're interested in and then know how to extract logs from that manager. For example, you need to know how to read systemd logs if the kubelet is managed by systemd.

systemd logs

Logs for services running on systemd can be viewed with journalctl. Here is an example reading logs of the kubelet, a service generally run through systemd:

journalctl --unit kubelet

If you are using a different service manager, please consult the documentation for how to extract logs for your particular service manager.

Docker logs

Control plane components can be managed by the kubelet using static pods. You can get their logs through kubectl logs <podname> or if you are on the node where the static pod is running you can access the docker logs directly:

docker logs <container_id>

This is good for cases when the kube-apiserver is not working and therefore kubectl commands are not working.

Learning to read logs

Get your bearings

Get your bearings by reading logs from a fully functional Kubernetes cluster. You'll get a glimpse into what the component is doing as well as familiarize yourself with log lines that can be ignored which will help you notice lines that might contain useful debugging information.

Don't be afraid to dig into the source code and see where each log line is coming from. Kubernetes code search is really good for this. Copy and paste parts of the log line until you get some hits and read the surrounding code. You could even use these log lines as the starting point for the strategies I outline in my Learning the Kubernetes codebase post to gain a very deep understanding of the component.

Once you're comfortable reading through logs of a working Kubernetes cluster it's time to break things. Turn off etcd and watch the logs. Turn it back on and see how the component responds. Do that with other components and watch what happens. You will start to understand which components communicate to other components and it will help you identify failures from symptoms faster.

The three most common components you will look at are the kubelet, the kube-apiserver and etcd. I'd suggest focusing on those, but do look at logs from all the components.

Examples

These examples come from a kubeadm cluster. The kubelet is managed by systemd and all control plane components are managed by the kubelet as static pods.

kubelet logs

Jul 08 16:32:51 ip-10-0-25-163 kubelet[20747]: E0708 16:32:51.415697   20747 kubelet_node_status.go:391] Error updating node status, will retry: error getting node "ip-10-0-25-163.us-west-2.compute.internal": Get https://10.0.25.163:6443/api/v1/nodes/ip-10-0-25-163.us-west-2.compute.internal?resourceVersion=0&timeout=10s: context deadline exceeded                                                          
Jul 08 16:33:01 ip-10-0-25-163 kubelet[20747]: W0708 16:33:01.416933   20747 reflector.go:341] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: watch of *v1.Service ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Unexpected watch close - watch lasted less than a second and no items received                                                                                           
Jul 08 16:33:01 ip-10-0-25-163 kubelet[20747]: W0708 16:33:01.417001   20747 reflector.go:341] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Unexpected watch close - watch lasted less than a second and no items received                                                                               
Jul 08 16:33:01 ip-10-0-25-163 kubelet[20747]: W0708 16:33:01.417031   20747 reflector.go:341] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: watch of *v1.Node ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Unexpected watch close - watch lasted less than a second and no items received                                                                                              
Jul 08 16:33:01 ip-10-0-25-163 kubelet[20747]: E0708 16:33:01.417105   20747 mirror_client.go:88] Failed deleting a mirror pod "etcd-ip-10-0-25-163.us-west-2.compute.internal_kube-system": Delete https://10.0.25.163:6443/api/v1/namespaces/kube-system/pods/etcd-ip-10-0-25-163.us-west-2.compute.internal: read tcp 10.0.25.163:36190->10.0.25.163:6443: use of closed network connection; some request body already written

The errors and warnings here are:

deleting a mirror pod failed
watches fail
updating node status fails

These errors together indicate either etcd or the kube-apiserver is not working. If you know how the kubelet and the kube-apiserver interact, you would even be able to directly tell this is a problem with etcd and not a problem with the kube-apiserver.

But in the interest of examples, let's go look at the kube-apiserver logs.

kube-apiserver logs

The kube-apiserver is currently crashing and restarting over and over. The logs look like this before the kube-apiserver is restarted:

Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0708 16:38:26.448081       1 server.go:703] external host was not specified, using 10.0.25.163
I0708 16:38:26.448218       1 server.go:145] Version: v1.11.0
I0708 16:38:26.775022       1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0708 16:38:26.775040       1 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0708 16:38:26.775722       1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0708 16:38:26.775737       1 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
F0708 16:38:36.778177       1 storage_decorator.go:57] Unable to create storage backend: config (&{ /registry [https://127.0.0.1:2379] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt true false 1000 0xc42029ce80 <nil> 5m0s 1m0s}), err (dial tcp 127.0.0.1:2379: connect: connection refused)

The last failure line is the most important. This tells us that the kube-apiserver is not able to connect to etcd because the connection was refused.

If we restart etcd, everything comes back up and all components are happy again.

etcd logs

There are generally no logs for normal behavior, unlike the kubelet which likes to tell you what it's doing constantly even when everything is fine.

Here are some example etcd logs:

2018-07-08 16:42:20.351537 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
2018-07-08 16:42:21.039713 I | raft: 8e9e05c52164694d is starting a new election at term 4
2018-07-08 16:42:21.039745 I | raft: 8e9e05c52164694d became candidate at term 5
2018-07-08 16:42:21.039777 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 5
2018-07-08 16:42:21.039792 I | raft: 8e9e05c52164694d became leader at term 5
2018-07-08 16:42:21.039803 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 5
2018-07-08 16:42:21.040764 I | etcdserver: published {Name:ip-10-0-25-163.us-west-2.compute.internal ClientURLs:[https://127.0.0.1:2379]} to cluster cdf818194e3a8c32
2018-07-08 16:42:21.041029 I | embed: ready to serve client requests
2018-07-08 16:42:21.041449 I | embed: serving client requests on 127.0.0.1:2379
WARNING: 2018/07/08 16:42:21 Failed to dial 127.0.0.1:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

The final warning is a bug in a particular version of etcd but has no impact on the functionality. This is the github issue for that bug.

everything else

Take a look at all the other components while you're there and you'll start to build both an understanding of each component and how to recognize when something is wrong.

Learning the Kubernetes codebase

Chuck Ha — Thu, 05 Jul 2018 22:27:15 +0000

Date: July 5th, 2018
Most recent Kubernetes release: 1.11.0

Kubernetes is a big piece of software clocking in just shy of 2 million lines of code. It can be intimidating to get started, but once you get the hang of it, you'll be amazed at how many tickets start to make sense and how you can quickly identify and fix bugs!

The very beginning

If you are just starting your Kubernetes journey, I would encourage you to read this documentation and get a high level understanding of the major components. To call something a Kubernetes cluster it needs most of the components listed there running on at least one node. Conveniently, all of the code for the components lives in the Kubernetes monorepo.

The first step

Now that you know what each component does, you can pick a component that sounds interesting to you. If you're having trouble picking a component, one thing I like to do is pick the thing that is farthest outside of my comfort zone. For example, if networking is a weak spot of yours, pick the kube-proxy. If you've got the network chops but don't really do web servers, pick the kube-apiserver.

One foot after the other

One tool every coder needs is a good code editor. The feature that helps me the most when reading code is jump to definition. I use VSCode with the typical Go plugins and that fulfills that requirement to my satisfaction. You could use vim, Goland, or emacs, just so long as you are comfortable with the tool and you can jump between functions effectively, you'll be ready to go.

One small step

Kubernetes lives at https://github.com/kubernetes/kubernetes. Usually, the import path for Go code that lives on github is the URL without the scheme (https://). So you might expect go get -d github.com/kubernetes/kubernetes to work, but that's going to confuse your tools. The package name is actually k8s.io/kubernetes. So go ahead and clone the repo using this command go get -d k8s.io/kubernetes. If you don't have go installed, follow the instructions on golang.org.

Another small step

Open up the Kubernetes directory with your trusted code editor. Take a look inside cmd/. This is where you find the entry point to all of the components that live inside Kubernetes. Every component in Kubernetes is run as a command line program and the references are all online. Here is where the kube-proxy's entry point is. This patter of cmd/<component>/<component>.go is going to be pretty much the same across every component.

Repeat forever

Now start exploring the file! Chances are the first file you open under cmd/<component>/<component>.go is going to be very small. Read through each line. Do you know what it does? If you go to the definition of the function call on this line you will realize how deep this code goes. You have a lot to read!

Now do this until you are satisfied with your understanding. It will take a while and that's totally normal. This stuff is not easy to pick up. Heck, you could even get a post or two out about what you found useful, interesting rabbit holes, and things you found surprising about the Kubernetes code base!

If you're trying to break into Kubernetes development, there's no better way to read lots of the code!

DEV Community: Chuck Ha

Data types in Kubernetes: PriorityQueue

An Error Wrapping Strategy for Go

Rule 1

Rule 2

Forcing a kubernetes version during build

Option 1 You can define a custom --workspace_status_command script that generates these versions with whatever versions you want

Option 2

Anki 2.1.x Add-ons

Customized Kubespray Deployment

Versions

Motivation

Steps

Running dev.to in a container Part 2

Running dev.to in a container

NEW POST, BETTER UX!

Running dev.to in a container Part 2

Chuck Ha

Original post below here

The architecture

The database

The web application

Logging approaches in Go

Understanding Kubernetes through a concrete example

Components

The end goal

Defining the system

Can we see analytics for our posts anywhere?

Reading Kubernetes logs

Kubernetes logging

systemd logs

Docker logs

Learning to read logs

Get your bearings

Examples

kubelet logs

kube-apiserver logs

etcd logs

everything else

Learning the Kubernetes codebase

The very beginning

The first step

One foot after the other

One small step

Another small step

Repeat forever

Option 1 You can define a custom `--workspace_status_command` script that generates these versions with whatever versions you want