DEV Community: Fred Richards

Stuff in my Lab, it Evolves

Fred Richards — Fri, 27 May 2022 20:51:10 +0000

I realized recently I haven't written much. Subject matter came to mind about specific situations, or recent experiments, but then I thought why not just talk about the Lab. It's constantly changing and evolving, and besides, it will be a fun thing to drone on about.

One of my main philosophies is keeping things as simple as possible, to the lowest common denominator. Considering I work with Kubernetes, with its massive amounts of abstractions, you'd think this is a paradox. These philosophies, or tenets, or principals, whatever you'd like to call them, will sometimes pop in my head as simple sentences. "Keep it simple". "Follow the convention". You get the idea.

Years ago my lab was a mess. For a living, I build stuff. Sometimes it was building Linux servers, or network services. Sometimes building networks themselves. Sometimes it was building cloud architecture. So I had to decide, what did I want out of my lab?

I wanted a quick way to test VMs. New version of Debian released? Ok, spin up a small vm and test it out. I wrote some scripts to call qemu-kvm on the commandline, made a dedicated routed vm-only network and connected it with a virtual switch, namely OpenVSwitch. These scripts were normally ansible and/or bash. There is not much these scripts do that one cannot get out of libvirt tools these days. My tooling was just born before libvirt was popular and matured.

This worked for a great while, but again, introduce Kubernetes. It is by it's very nature an affront to my minimalist habits!

So the next great step? K3s. I could run the very same guest VMs I have always, pop k3s on them and now, BOOM, they're a single-node cluster. This also works well, because one can create automation that spins up the vms, installs k3s and then applies manifests or other api-like behavior. The whole stack can spin up onsite, and with a bit of tweaking, under your favorite cloud provider.

Recently it's evolved even further. Sometimes I need not only a minimal cluster of 1-2 nodes, but an ephemeral one, something that will not live long. Spin up a cluster, run some experiment, or QA-like operation, capture result data, then destroy everything. This may just be peak-minimalism.

So now I use k3d and skip the guest VM middle man. OR get this, maybe run k3d inside the VM, if for some reason the single-node-lab-in-a-box needs to be fungible.

Some people run clusters of huge boxes, or massive datacenters filled with physical machines, or even clusters with thousands of nodes. Not me, I'll have dozens of optimized clusters and guest VMs in my lab, along with their resulting data, configs, backups, and the automation to create it all over again if needed.

K3S Upgrades with Fleet

Fred Richards — Mon, 09 May 2022 20:42:15 +0000

K3s has a great option to automate upgrades with the system-upgrade-controller. While SUC may seem like an odd name it does work well. Why can't we automate the upgrades of k3s with gitops?

In my lab, I traditionally run a bunch of small Linux VM guests of differing operating system distributions, and recently decided to make them single-node k3s clusters. I have at any time, about six of these either guest kvm machines or raspberry pi-style computers running k3s in the lab.

The system-upgrade-controller for k3s is simple, it takes a custom resource called a Plan and it applies to nodes with a certain label, like k3s-upgrade=true. The three step process looks like this:

install system-upgrade-controller
apply any Plans that are required
label nodes to trigger the upgrade, per Plan

This works if the cluster consists of one node or 100 nodes. My lab is pretty minimalistic, so I want to keep any gitops light and simple.

Fleet can use mainfests, kustomize, helm charts, or any combination of the three. To keep with the minimalist theme, I use manifests. More info on the Fleet architecture is here.

Fleet can use the three-step process above to upgrade each of the k3s clusters. I decided to let Fleet handle the first two steps, and the third is already covered by other automation.

I've carefully planned out the organization of how my Fleet gitops will operate. Here are my definition for common Fleet topics -

Workspace - a simple namespace, on the local Rancher cluster where the fleet-controller lives.
GitRepo - a path from a repo that Clusters subscribe to. Note: using a path makes it flexible, that way one can have a lot of GitRepos under one hosted provider like Github or Gitlab.
Clusters - the downstreams that will subscribe to certain GitRepos under Workspaces.
ClusterGroups - grouping of clusters by common criteria, project, or attribute. I often group my project, or create empty groups for later.
Bundle - a fleet.yaml manifest describing what is to be deployed.
BundleDeployment - gluing together the Bundle and the relevant GitRepos, applied to certain ClusterGroups, after it is deployed to specific Clusters.

Here is a tree-view example of my GitRepo paths ...

├── bitnami
│   ├── openldap
│   │   ├── fleet.yaml
│   │   └── openldap.yaml
│   └── README.md
├── default-dev
│   ├── plan
│   │   ├── 121-12-plan.yaml
│   │   └── fleet.yaml
│   ├── README.md
│   └── upgr
│       ├── fleet.yaml
│       └── system-upgrade-controller.yaml
├── gxize-testing
│   ├── rancher
│   │   └── longhorn
│   │       ├── fleet.yaml
│   │       └── longhorn.yaml
│   └── README.md
├── k3s
│   ├── plans
│   │   ├── 121-11-plan.yaml
│   │   └── 121-12-plan.yaml
│   ├── readme.md
│   └── suc
│       ├── fleet.yaml
│       └── system-upgrade-controller.yaml
├── README.md
└── zerk-final
    └── README.md

You'll notice a few organizational choices, first, the bitnami and k3s paths are not only named for vendors, but also are temporary spaces with nothing pointing to them. I could remove these entirely and no clusters are subscribed to these dirs, so nothing would happen.

The next is my three-tier lifecycle, dev, testing and final (like prod). For the k3s system-upgrade project, we're only concerned with the /default-dev path. Under my UI, this is called general. I also decided to keep everything under the default Fleet namespace, fleet-default.

Bundles and BundleDeployments will derive their names from the <GitRepo_Name>-<Bundle_Path>. So a Bundle name might be from the example above general-default-dev-upgr.

Lastly, I created a bunch of ClusterGroups on different criteria. Sometimes this is the networking on the node, the node's OS, or just by project name.
I would suggest and recommend to make empty groups, it will assist in organization later. For this project in my dev environment of three clusters, I had a pre-existing zone-orange ClusterGroup, which keys on the label zone=orange for each cluster. If I wish to add or remove clusters from the ClusterGroup, I simply add or remove the label.

From my three-step process above, I want Fleet to install the upgrade-controller and the Plan. I create fleet.yaml bundles in my Git Repo, under the proper GitRepo-path:

.../general$ cat default-dev/upgr/fleet.yaml                                           
defaultNamespace: default
targetCustomizations:                                                                              
- name: upgr-orange
  clusterGroup: zone-orange                                                                        

.../general$ cat default-dev/plan/fleet.yaml                                           
defaultNamespace: default
targetCustomizations:                                                                              
- name: upgr-orange
  clusterGroup: zone-orange                                                                        
  dependsOn: general-default-dev-upgr

... note the dependsOn option, this means that the second Bundle for the Plan relies on an Active status of the first Bundle for the system-upgrade-controller.

Once I commit these changes into git, the fleet-controller picks up any changes because the hash has changed, and pushes those changes to the ClusterGroups mentioned in the fleet.yaml Bundle. I can also check the status of the BundleDeployments to ensure all three clusters have both the system-upgrade-controller and plan manifests completed.

My last step, is to run kubectl label node/<node-name> k3s-upgrade=true as per the nodeSelector in the Plan. This allows me to control the cadence of the upgrades on a per-node basis, and there is also a .spec.concurrency to cordon/upgrade nodes in batches if these were more than one node.

After I'm satisfied with the upgrade, I can remove the label ... kubectl label node/<node-name> k3s-upgrade- ... in preparation for the next upgrade.

Now that everything is in place and organized, my next upgrade is updating at least the Plan manifest, committing git changes, and labeling the nodes again.

Some pointers and tips -

consider the GitRepo as a path, use these paths for your own project's organization.
manage ClusterGroups, not Clusters! Creating empty groups can help plan for the future.
you can use one repo for different lifecycles, or use different Workspaces for different lifecycles/projects.
use dependsOn with the Bundle name for when you require pieceA to be ready before pieceB, for example with supporting crds before a specific helm chart.

Log4j Vulnerability -- CVE-2021-44228

Fred Richards — Sat, 11 Dec 2021 21:54:23 +0000

Lots of my technology friends are concerned about the new CVE-2021-44228, the log4j vulnerability. What should you do? First of all, don't panic. Yes, the library is widely used -- in software which leverages java. If your code or app isn't java there's a slim to no chance it will be affected. If you do find the code is java-based, check with your vendor. Even my home lab uses the UniFi controller from Ubiquiti, and they promptly released a patch this weekend.
(https://community.ui.com/releases/UniFi-Network-Application-6-5-54/d717f241-48bb-4979-8b10-99db36ddabe1)
Rest assured your friendly neighborhood software vendor is on it!

Shift the TZ for your K8s CronJobs

Fred Richards — Fri, 19 Feb 2021 18:31:20 +0000

In your Kubernetes clusters, the timezone is typically assumed to be UTC, which is a very cloud-native way to provision things. The resources you have might be across the globe, so having every node or host use the same timezone makes sense, especially if you're correlating information in logs.

But in other environments, there is a need to have the local timezone configured.

What happens if your cronjobs in the cluster are not aligned with the local timezone? This could cause some minor havoc.

The answer is simple. The timezone settings for cronjobs are managed by the kube-controller-manager component. Assuming all of your control-plane nodes are provisioned prior with localized and correct timezone settings, just add an extra mount, adding in /etc/localtime into the kube-controller-manager component.

Inside Rancher RKE it may look something like this in your cluster.yaml file:

services:
  kube-controller:
    extra_binds:
      - '/etc/localtime:/etc/localtime'

Provision your clusters with the localtime setting mounted and your jobs will be in the proper timezone.

NetworkManager, leave Calico Interfaces alone.

Fred Richards — Fri, 19 Feb 2021 18:20:27 +0000

We've all been there. You set up your nifty new Kubernetes cluster, and choose your favorite CNI, Calico, to handle networking.

But then NetworkManager comes along and has to watch and interfere with all of these new virtual interfaces introduced by Calico.

We can tell NetworkManager to ignore these interfaces by making the new file...

/etc/NetworkManager/conf.d/calico.conf

... and including the following content inside of it.

[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico

Now those virtual interfaces used by Calico will get ignored and will not see any interference from NetworkManager. Of course, you can also always decide not to use NetworkManager, but this may not be practical in all environments.

DNS Haiku with TXT Records and Terraform

Fred Richards — Wed, 23 Oct 2019 10:24:35 +0000

Recently, I wanted to test the basic powerdns terraform provider, and whether I could write multiple TXT record entries for one name.

Hey, what do you know, it worked. I just had little test messages in there, and decided, maybe it would be neat to put the "DNS Haiku" in a text record. I used the latest Terraform, 0.12.12.

Here's my code. Notice I changed the haiku just a little.

// don't forget to put vars in separate file
// for pdns_api_key & pdns_server_url

provider "powerdns" {
  api_key    = var.pdns_api_key
  server_url = var.pdns_server_url
}

// example of multiple TXT records for name
resource "powerdns_record" "text" {
  zone    = "geexology.org."
  name    = "text.geexology.org."
  type    = "TXT"
  ttl     = 900
  records = ["\"It's not DNS.\"","\"There's no way it's DNS.\"","\"Oh no, DNS.\""]
}

Example tfvars.tf.

// cat tfvars.tf 

variable "pdns_api_key" {
  description = "current api key"
  default     = "notrealzlolputyourkeyheresilly"
}

variable "pdns_server_url" {
  description = "current api url"
  default     = "http://127.0.0.1:8087"
}

The reply from the server for the dig query.

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> txt text.geexology.org @172.29.100.102
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22695
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;text.geexology.org.            IN      TXT

;; ANSWER SECTION:
text.geexology.org.     900     IN      TXT     "It's not DNS."
text.geexology.org.     900     IN      TXT     "There's no way it's DNS."
text.geexology.org.     900     IN      TXT     "Oh no, DNS."

;; Query time: 76 msec
;; SERVER: 172.29.100.102#53(172.29.100.102)

;; WHEN: Wed Oct 23 06:19:43 EDT 2019
;; MSG SIZE  rcvd: 134

When refreshing state, or performing terraform plan the id looks like this.

powerdns_record.text: Refreshing state... [id=text.geexology.org.:::TXT]

Nested, Muxed, Screens on the CommandLine

Fred Richards — Sun, 13 Oct 2019 13:47:12 +0000

I'm a huge fan of the command line and always have been. If you're like me, you're most comfortable at the keys in a dark terminal session. Most of my day to day work can get accomplished through a terminal, with the rare case of needing a graphical web browser.

As both a network engineer and systems admin, I have a lot going on at any given time. There are two tools I believe we all use, stuff happening in the background, and an organized classification of our work areas, often times separated.

This is a good way to classify projects, environments, testing, coding, you name it. Each workspace has it's own separation. I'm often logged into dozens of servers, iot devices, lab machines, vms, cloud instances. It helps to stay organized.

And I'm not going to log into these resources everytime I swap client machines I'm working behind. I could be using a linux machine, a chromebook, a windows laptop, bastion host, or temporary container. I need stuff to continue to run in the background.

This is where GNU Screen comes in. I've used it for years. How to use screen is a bit out of the scope of this post, but the quick description is that it's a terminal muxer. It takes one ssh session and creates a back-end of many sessions. You can disconnect and leave them running, move to a different client, and re-connect later. This is great for long-running processes like compiling or creating stacks or container environments. Commands are based on a hot-key "ctrl-a". For example, to switch to the next screen session, "ctrl-a n".

Now somewhere on the internet, there's a holy war raging. There are people who prefer tmux, an alternative to GNU screen. Each has their own pros and cons, and for a while I switched from exclusively screen to exclusively tmux. But wait, these are open source tools, can't I use both? The hot-key for tmux is "ctrl-b" and some of the commands overlap. Ie, next-session in tmux is "ctrl-b n".

The diagram shows my new setup. Tmux handles my groups. Screen handles my sessions inside those groups.

Typically I use up to nine sessions in each screen session, so this allows me to have eighteen sessions muxed into one, grouped into two groups. I've found a more comfortable working number is somewhere between 12-15.

This is just the tip of the iceberg when it comes to tools. Don't even get me started on mosh, the mobile shell which uses protocol buffers (from Google) to resume roaming sessions.

[Update]: I've been using the same tmux-start script for a while, and roughly 23 sessions (20 screens + 1 more tmux group with 3 panes). I was comfortable, so automated it a bit.

I have also added it as a gist here, please ignore my joke about it exploding everything, but seriously you should completely understand things you find randomly on the internet, before running.

Show me your Bash Functions / Aliases!

Fred Richards — Sat, 21 Sep 2019 23:19:11 +0000

Do any of you have your favorite bash functions or aliases? I've always been a fan of quick "lightning tips" that you can show maybe on one line, and do something specific.

Here are some of mine!


## timestamp dmesg with the current date, this sudo doesnt require password
alias datemsg='date | sudo tee /dev/kmsg'

## how many ms since the epoch?
alias epocms='date +%s%3N'

## ksm is kernel same-page merging for virtual machines,
## this allows me to keep tabs on performance and metrics
alias ksm-info='~/opt/ksm-info.sh'

( ksm-info is just a one-liner ...
for ki in /sys/kernel/mm/ksm/* ; do echo -n "$ki: " ; cat $ki ; done
...)

## normal mutt is gmail, mm == local mail
alias mm='mutt -f ~/Mail/fredr'

## python pretends to be jq sometimes
alias pj='python -m json.tool'

## run a lot of ansible playbooks ... plans == play ansible
alias plans='ansible-playbook'

### .bashrc functions

## check on kernel entropy for ssh, vpn, encryption
entropy () { cat /proc/sys/kernel/random/entropy_avail; }

## pipe colorfied jq to less while keeping colors
jql () { jq -C . $1 | less -R ; }

Many other minor stuff too, like loading environment vars for Go or BC.
So what are some of yours?

Organization, Classification of your personal data

Fred Richards — Fri, 12 Jul 2019 11:21:26 +0000

I was having a quick chat with my brother recently. See, I've been into IT and technology for as long as I can remember. A lot of what we do is about organization of data. How do you think about data, how is it visualized inside your head, in your thoughts?

One of the things I recommend is to create conventions for classification. And this goes for everywhere, not just in code, or on servers, but your own local workstations as well.

One of my favorite conventions is topic or type, and date. My backups look like directories of the same type of file sorted by year. If I moved in 2016, and scanned everything into PDF files, that's a super-quick way to find it.

Same thing goes for my chrome bookmarks. Chrome allows you to bookmark all your current tabs. I have a special group called Stamped, where the current tabs I was concentrating on at the time are in Year-Month-Day format. I can tell you on 20140521 what I was reading, or focused on because the history is all there. You get a holistic view of what your thoughts were at the time, where you've gone from there, how you've grown, and maybe some topics you've dropped on the floor, for whatever reason.

This one seems silly, but it's worked for me for years. I know the vast majority of people with a coding skill set probably have a good handle on their own conventions. What ones do you use, any unique ones? And why?