DEV Community: Jay R. Wren

Benchmark Testing Kubernetes Controller Runtime With Envtest

Jay R. Wren — Wed, 01 Mar 2023 22:11:45 +0000

The envtest docs push you into Ginko, but I'm a huge proponent of idiomatic (add another t to this word and I still fit) Go and Ginko doesn't fit me.

Thankfully you can use standard Go test benchmark functions without any extra testing packages and profile code to improve controllers.

Here is how I did it...

package something
import (
    "context"
    "path/filepath"
    "testing"

    "k8s.io/apimachinery/pkg/types"
    "k8s.io/client-go/kubernetes/scheme"
    controllerruntime "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"

    "something/mypkg"
)
func BenchmarkMy_Reconcile(b *testing.B) {
    testEnv := &envtest.Environment{
        CRDDirectoryPaths: []string{"path/to/crds"},
    }
    cfg, err := testEnv.Start()
    if err != nil {
        b.Fatal(err)
    }
    defer func() {
        err := testEnv.Stop()
        if err != nil {
            b.Error(err)
        }
    }()
    err = mypkg.AddToScheme(scheme.Scheme)
    if err != nil {
        b.Fatal(err)
    }
    k8sClient, err := client.New(cfg, client.Options{Scheme: scheme.Scheme})
    if err != nil {
        b.Fatal(err)
    }
    r := &MyController{
        Client:      k8sClient,
        Log:         controllerruntime.Log.WithName("controllers").WithName("mypkg"),
        Scheme:      scheme.Scheme,
        InitMetrics: map[string]string{},
    }
    rreq := reconcile.Request{NamespacedName: types.NamespacedName{
        Namespace: "test", Name: "test"}}
    for i := 0; i < b.N; i++ {
        got, err := r.Reconcile(context.Background(), rreq)
        if err != nil {
            b.Fatalf("MyController.Reconcile() error = %v", err)
            return
        }
        b.Logf("%#v", got)
    }
}

On my mac, I use brew to install etcd and kubectl. I clone kubernetes and build kube-apiserver myself and copy it to /usr/local/bin

brew install kubectl etcd
git clone git@github.com:kubernetes/kubernetes.git && \
pushd kubernetes && \
git checkout v1.26.1 && \
go install ./cmd/kube-apiserver && \
cp ~/go/bin/kube-apiserver /usr/local/bin && popd

Then I run this test with KUBEBUILDER_ASSETS=/usr/local/bin go test -benchmem -run=^$ -bench ^BenchmarkMy_Reconcile$ -memprofile mem.prof ./something

The output is the standard Go test benchmark output and the mem.prof which I examine using go tool pprof mem.prof.

I haven't been programming much lately and so I didn't have dot installed on my Mac and I had to lookup where to get it. I had to brew install graphviz.

I use top10 to see the highest memory users and I use web to generate an SVG of the memory creation call graph.

In my case, the code base was over using fmt.Sprint(f) and I was able to optimize these string allocations. I consider this a win.

How to write Go

Jay R. Wren — Fri, 17 Feb 2023 19:11:12 +0000

I find myself on a team where where small tiny little mistakes are being made when writing Go. I'd like to collect a reference of how to write better Go. I am putting it here.

https://go.dev/doc/effective_go
https://go.dev/doc/faq
https://github.com/golang/go/wiki/CodeReviewComments
https://go-proverbs.github.io
https://github.com/golang-standards/project-layout/issues/10 (pkg directory should not be recommended for us)
https://github.com/golang-standards/project-layout/issues/117
If the last 2 don't convince you, then: https://github.com/go-standard/project-layout
Consider: https://github.com/uber-go/guide/blob/master/style.md
More: https://rakyll.org/style-packages/

Things I see newer Go programers do which should be avoided:

Too many packages

Circular packages are not allowed in Go, unlike Java, C#, Javascript, Ruby, or Python. This means that you can structure yourself into a corner which requires great feats of moving things around.
When should you create a new package? Only when you must!
You'll know when you must.
Common musts:

Another executable and you want it smaller so you avoid including some larger packages.
You are building libraries (packages) consumed by 3rd-parties and you want those 3rd parties to have a choice regarding sizing their resulting executables. Let YAGNI and KISS guide you.

Remember, Go is a derivative of C more than any other language, not Python, Ruby, JavaScript, C#, or Java. The compilation artifact of those languages are different than C or Go. If you aren't very familiar with C, then your instinct in Go is probably misguiding.

Defining an unused interface

https://github.com/golang/go/wiki/CodeReviewComments#interfaces says it best.

Go's implicit interfaces allow consumers to create the interface they need far better than you can.

Common or util packages

NEVER.
https://dave.cheney.net/2019/01/08/avoid-package-names-like-base-util-or-common

Don't store context

https://go.dev/blog/context

You'll notice the RFC for "relax recommendation against storing context" https://github.com/golang/go/issues/22602 is still open. Never expect this to be closed.

In fact, if you read that open issue, you'll learn that it is not requesting to relax the recommendation, but to clarify it and the clarification is, "don't store context"

No unused parameters (unless you are implementing an interface)

This really applies to ALL programming languages, but us Go programmers like it because we have so many "no unused..." rules already.

Clear is better than clever

https://go-proverbs.github.io https://www.youtube.com/watch?v=PAAkCSZUG1c&t=14m35s

Beware of reading too much

This is basic defensive programming which is often ignored which Go stdlib helps make trivial.

You've made your net/http Request and have a response:

Do:

body, err := io.ReadAll(io.LimitReader(resp.Body, 32768))

Don't:

body, err := io.ReadAll(resp.Body)

Vary the 32768 to as large as you expect the response to be, but do limit it. Don't do unlimited reads from the network.

Overuse of init()

init() almost always implies globals. See next item.

Overuse of globals

This really applies to all programming languages, but when you have a global, it is probably a sign that you've designed the structure of your application poorly.

Early on in a project is the best time to spend time to avoid this.

haproxy tuning for TLS termination

Jay R. Wren — Wed, 24 Aug 2022 18:45:43 +0000

I'm working on a platform which has been a little neglects from a performance tuning point of view. We had some issues recently which effectively caused denial of service, even though we were doing it to ourselves. As a result, I'm tuning some things and I've learned some things which I didn't know.

You can probably DoS your haproxy instance too, if you've not already tuned it. I'm using the wonderful hey tool to create connections to my haproxy.

My haproxy has multiple certificates with unique private keys associated with each. Some are 4096 bit keys. Most are 2048 bit keys. Which certificate and key is used depends on which hostname is used to connect to the server.

I see huge differences in the performance based on 2k or 4k PK used for signing. I've learned that this is expected and that openssl ships with a speed command so that you know what to expect. In my case, haproxy is running on nodes with 8 cores (c5.2xlarge) and is configured to use them (cpu-map 1- 0-).

I've seen the openssl speed rsa command, but I had to hunt down the rsa2048, rsa4096, and -multi commands and option.

In my case, there isn't a great solution because I'm mixing 4k and 2k PK. Don't do that. I'll be advocating to my team to use 2k as much as possible.

Here is example output from the commands and the resulting sign/s number is pretty much the value I suggest using for the maxsslrate setting in haproxy.

# openssl speed  -multi 8 rsa2048
Forked child 0
Forked child 1
Forked child 2
Forked child 3
Forked child 4
Forked child 5
Forked child 6
Forked child 7
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+DTP:2048:private:rsa:10
+R1:8096:2048:10.00
+DTP:2048:public:rsa:10
+R1:8202:2048:10.00
+DTP:2048:public:rsa:10
+R1:8279:2048:10.00
+DTP:2048:public:rsa:10
+R1:7769:2048:10.00
+DTP:2048:public:rsa:10
+R1:8329:2048:10.00
+R1:7351:2048:10.00
+DTP:2048:public:rsa:10
+DTP:2048:public:rsa:10
+R1:7355:2048:10.00
+DTP:2048:public:rsa:10
+R1:7363:2048:10.00
+DTP:2048:public:rsa:10
+R2:258668:2048:10.00
+R2:280384:2048:10.00
+R2:274594:2048:10.00
Got: +F2:2:2048:809.600000:25866.800000 from 0
+R2:262465:2048:10.00
+R2:279210:2048:10.00
+R2:255086:2048:10.00
Got: +F2:2:2048:832.900000:27921.000000 from 1
Got: +F2:2:2048:735.100000:25508.600000 from 2
Got: +F2:2:2048:827.900000:27459.400000 from 3
Got: +F2:2:2048:776.900000:26246.500000 from 4
+R2:266538:2048:10.00
Got: +F2:2:2048:735.500000:26653.800000 from 5
Got: +F2:2:2048:820.200000:28038.400000 from 6
+R2:297361:2048:10.00
Got: +F2:2:2048:736.300000:29736.100000 from 7
OpenSSL 1.1.1  11 Sep 2018
built on: Mon Jul  4 11:25:51 2022 UTC
options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-wL7Fqk/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.000159s 0.000005s   6274.4 217430.6

# openssl speed  -multi 8 rsa4096
Forked child 0
Forked child 1
Forked child 2
Forked child 3
Forked child 4
Forked child 5
Forked child 6
Forked child 7
+DTP:4096:private:rsa:10
+DTP:4096:private:rsa:10
+DTP:4096:private:rsa:10
+DTP:4096:private:rsa:10
+DTP:4096:private:rsa:10
+DTP:4096:private:rsa:10
+DTP:4096:private:rsa:10
+DTP:4096:private:rsa:10
+R1:1170:4096:10.00
+DTP:4096:public:rsa:10
+R1:1144:4096:10.00
+DTP:4096:public:rsa:10
+R1:1195:4096:10.00
+DTP:4096:public:rsa:10
+R1:1152:4096:10.00
+DTP:4096:public:rsa:10
+R1:1070:4096:10.00
+DTP:4096:public:rsa:10
+R1:1178:4096:10.01
+DTP:4096:public:rsa:10
+R1:1195:4096:10.00
+DTP:4096:public:rsa:10
+R1:1145:4096:10.02
+DTP:4096:public:rsa:10
+R2:75477:4096:10.00
+R2:71013:4096:10.00
+R2:82989:4096:10.00
+R2:84313:4096:10.00
Got: +F2:4:4096:115.200000:8431.300000 from 0
+R2:77447:4096:10.00
+R2:76991:4096:10.00
Got: +F2:4:4096:117.682318:7699.100000 from 1
Got: +F2:4:4096:119.500000:8298.900000 from 2
Got: +F2:4:4096:107.000000:7744.700000 from 3
Got: +F2:4:4096:117.000000:7547.700000 from 4
+R2:75476:4096:10.00
Got: +F2:4:4096:119.500000:7547.600000 from 5
+R2:78033:4096:10.00
Got: +F2:4:4096:114.271457:7803.300000 from 6
Got: +F2:4:4096:114.400000:7101.300000 from 7
OpenSSL 1.1.1  11 Sep 2018
built on: Mon Jul  4 11:25:51 2022 UTC
options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-wL7Fqk/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
                  sign    verify    sign/s verify/s
rsa 4096 bits 0.001082s 0.000016s    924.6  62173.9

6200 is much larger than 920.

2k keys and maxsslrate 6200 it is.

Failing Digital Ocean Kubernetes Challenge

Jay R. Wren — Sat, 04 Dec 2021 17:13:47 +0000

Goal: Vanilla k8s install

The first thing is signing up with Digital Ocean. Apparently, I've never used their product, so I had to signup. When entering my credit card information, I'm a little disappointed that they don't support the autofill features of my browser. I was able to autofill the CC number, but I had to type my name and address.

Apparently, this page attempts to verify identity using the CC and it does it before clicking submit. The "Save Payment Method" button is disabled if it cannot verify that the card details are correct, but it doesn't feedback about what is invalid. I use this CC everywhere on the internet but DOs form will not validate for me. I wonder how much money DO is leaving on the table by not accepting valid cards and identity. I know everything on this form is correct. I use the exact same details to buy things all over the internet. DO doesn't accept it.

It took me a good couple of minutes of staring at the form to realize the CVC is required and not filled in. Terrible UX. DO should be ashamed. There was NO feedback. What should have happened: the field should have been red, there should have been a * next to the field. There should have been text which said, "CVC required". I'm starting to see why there are only 6 requests in the GH repo so far.

Now that I see the welcome page and can access the control panel, I'm going to follow this to get a k8s cluster up. https://docs.digitalocean.com/products/kubernetes/quickstart/

I'm in Ann Arbor, MI and so I am not going to use the default Amsterdam region and instead I'll select NYC3.

There is a warning by default that says Important: You have reached the 1 Droplet limit on your account. Request Increase I click the request button and say I want 3 droplets. Since I'm new to DO and not 100% sure what a droplet is, I'm a little surprised, but I guess I'll assume that a droplet is not a container like I thought, but is a VM. I want 3 VMs because I know that I need at least that many to run a k8s cluster.

I also change the node type since I won't be doing much with this cluster. The $10/mo nodes seem like plenty.

The "Create Cluster" button is disabled. No feedback on the form again. I wonder if DO is using some web framework that doesn't support Safari. I thought the web was beyond this at this point, but either DO web devs aren't eating their own dog food, or they ignore users like me. 😢

I try lowering the node count from 3 to 1. This pops up a "Recommended:
A minimum of 2 nodes is required to prevent downtime during upgrades or maintenance. Request Increase" message. Ok, that makes sense, but... the plus button to increase the node count is disabled. I can't use that, but I can type 2 into the textbox and the message goes away.

At this point, I do find it strange that AWS will let me rack up a giant bill as soon as they have my billing info, but DO is limiting me to 1 droplet OOTB, even though they have my billing info AND my account has a $60 credit on it. 🤔

At this point I give up and leave this form and start exploring my account. Settings has the droplet listed too so I try to raise it there. Same thing, it looks like a human reviews these raise requests. I wonder what AWS limits are like by default and I search for it. I see something about 20 instances per region. Seems more reasonable.

I can't move on, so I'll have to wait.

About an hour later I received this email:

Thank you so much for reaching out to us for getting the Droplet limit increased, Unfortunately, your account does not qualify for the requested limit based on our internal assessment guidelines. 

We would be happy to consider your request for a higher Droplet limit after you have established additional billing history with us with no instances of abuse. 

We hope you would understand we have these guidelines in place to ensure the stability of our platform.

Well, I guess I won't be playing with k8s on DO.

haproxy and certbot on ubuntu

Jay R. Wren — Mon, 29 Nov 2021 23:12:38 +0000

This is largely inspired from https://phansch.net/posts/haproxy-letsencrypt-certbot/

I recently changed my setup such that haproxy listens on 80 and 443 and proxies into various services. I like it MUCH better now. IMO, it is far easier to manage.

One outstanding issue is certificate renewal. The certbot package on ubuntu (and probably debian, but I'm on ubuntu) includes /lib/systemd/system/certbot.service and /lib/systemd/system/certbot.timer files and I would like to work with them. Systemd is awesome and allows me to do so using overrides. I don't even have to think about how to manage the overrides. I use sudo systemctl edit certbot.

I add this to the editor which starts:

[Service]
ExecStart=
ExecStart=-/usr/bin/certbot -q renew --preferred-challenges http --http-01-port 9785
ExecStartPost=/etc/haproxy/certs/certbot-renew

There is an excellent post here which describes why the blank = and the -=. I shall not repeat it here. Go read the reply. It is great: https://askubuntu.com/a/659268/1668

To my haproxy.cfg I added to both frontends:

        acl letsencrypt-req path_beg /.well-known/acme-challenge/
        use_backend letsencrypt if letsencrypt-req

and that letsencrypt backend:

backend letsencrypt
   server letsencrypt 127.0.0.1:9785

Cool, now when letsencrypt tries to verify on my system, that request for /.well-known/acme-challenge/ will be redirected to the certbot process listening on port 9785.

There is also that ExecStartPost= which I must explain.

My haproxy.cfg is setup with crt-base /etc/haproxy/certs in the global section and an HTTPS frontend which starts like this:

frontend https-in
  bind :::443 v4v6 ssl crt one.example.com crt two.example.com crt e.example.net alpn h2,ht
tp/1.1

Yes, I have a few different certificates from letsencrypt. One with a dozen or so SNI names and the other two as wildcard certificates. I use the haproxy features of reading all pem files in a directory to load them. I'd like any certbot renew to automatically configure them for haproxy.

This is done with that ExecStartPost=/etc/haproxy/certs/certbot-renew script:

#!/bin/bash
set -e
cd /etc/haproxy/certs
LELIVE=/etc/letsencrypt/live
RELOAD=false
for d in $(find . -type d -not -name .) ; do
        if [[ $LELIVE/$d/fullchain.pem -nt $d/fullchain.pem ]] ; then
                RELOAD=true
                cp $LELIVE/$d/fullchain.pem $d/fullchain.pem
                cp $LELIVE/$d/privkey.pem $d/fullchain.pem.key
                chown haproxy:haproxy $d/fullchain.pem $d/fullchain.pem.key
        fi
done

if [[ $RELOAD == true ]] ; then
        systemctl reload haproxy
fi

The script looks at each directory in /etc/letsencrypt/live and if the fullchain.pem in that directory is newer than the one in /etc/haproxy/certs then it copies the pem file and keys into the haproxy directory with a filename which haproxy will read. If it did anything then it calls systemctl to reload haproxy.

It is possible that I won't have to deal with certificate renewal every again.

nginx with brotli support on Centos 6

Jay R. Wren — Tue, 19 Oct 2021 16:21:39 +0000

I started a new job 6 weeks ago and I'm still learning the environment. I got a new task yesterday and it was an issue which had been bounced around for a while. The issue seemed simple enough, enable brotli on a virtual host. This is the story of how simple may not be so simple.

The first issue: this is a Centos 6 host.

Centos 6 is end of life as of November 2020. After a bit of discussion, the plan is to continue to investigate if we can do this as a stopgap until the entire solution can be reworked.

So, how would one investigate how to do this? It turns out, it isn't so trivial, and even just starting an investigate can be tricky. My first thought was, start a docker instance so I can poke around and try various RPMs.

docker pull centos:6.7
Great start!
docker run -ti centos:6.7

Cool, I've got a prompt, now what?

After searching the web, I see that nginx.org has rpm packages. I'll try one and see if it has brotli support.

curl -LO https://nginx.org/packages/mainline/centos/6/x86_64/RPMS/nginx-1.19.5-1.el6.ngx.x86_64.rpm curl: (35) SSL connect error

WAT? ok, maybe the certs on this 2yo docker image (I had checked docker hub) are out of date. curl -k the same thing and get the same error. WTF? add the -v to curl and see the libnss message * warning: ignoring value of ssl.verifyhost Well ain't that something? At this point I mumble under my breath about how debian/ubuntu uses openssl linked curl by default and not libnss and I wonder if it would have behaved the same.

As I write this, I realize that maybe I should have used a more recent Centos 6 docker image, 6.10 perhaps. Unfortunately I'm not as experienced with Centos as I should be. Part of this fun is diving in and learning. I used 6.7 because that is what this server under question says it is.

Alright, so new problem...

How to update certificates on a 2yo Centos 6.7 docker image?

yum update says It can't do its thing.

$ yum update
Loaded plugins: fastestmirror
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base

Alright, some searching tells me to use baseurl instead of mirrorlist and that it is now at vault.centos.org. I try that and get a new error: http://vault.centos.org/centos/6/os/x86_64/repodata/repomd.xml: [Errno 14] Peer cert cannot be verified or peer cert invalid Yes, that says http:, but port 80 just redirects to port 443 and then we get the cert error. We have a catch-22. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. cough ouch

(Update: I'm doing this often enough that editing that file takes too long and so there is this: sed -i 's/#baseurl=http:\/\/mirror/baseurl=http:\/\/vault/;s/mirrorlist/#mirrorlist/' /etc/yum.repos.d/CentOS-Base.repo )

For a solution, I got lucky. I knew to try to get the ca-certificates rpm manually and install it without yum. Worse come to worse I could download it on another host and copy it with docker cp or use netcat or python simple server or put it on a non-TLS webserver or any other method of moving data around. I was manually browsing the repo and got this URL curl -LO https://archive.kernel.org/centos-vault/6.10/updates/x86_64/Packages/ca-certificates-2020.2.41-65.1.el6_10.noarch.rpm and it turns out that the cert serving archive.kernel.org was acceptable.

rpm -U ./ca-cert*.rpm

And now I can yum update ; yum upgrade

Now we can look for nginx with brotli

Search search search, search the web and you will find no clear answers. There are posts about compiling it yourself. There are posts about using a subscription repo. (https://www.getpagespeed.com)

Incidentally, that subscription repo also had blog post which helped confirm my findings above: https://www.getpagespeed.com/server-setup/how-to-fix-yum-after-centos-6-went-eol along with https://forums.centos.org/viewtopic.php?f=13&t=78238 which verified what I had guessed about updating certificates.

Now my first thought was, nginx has this by default now, right? Use the newest package from nginx.org. Look at all of 'em here: https://nginx.org/packages/centos/6/x86_64/RPMS/

I curl -LOed (curllo is a verb ya see) that https://nginx.org/packages/mainline/centos/6/x86_64/RPMS/nginx-1.19.5-1.el6.ngx.x86_64.rpm and installed it (rpm -i ./nginx*rpm) and was disappointed that it had no brotli (strings /usr/sbin/nginx | grep brot confirms). (That is after I yum install initscripts as required by that package)

Well bummer.

Keep searching and reading...

The next thing I tried were the packages from https://repo.aerisnetwork.com but, in short they didn't do what I wanted or had some drawback.

Next, I tried poking at the packages from getpagespeed.com, I even browsed in my browser, got this URL: https://extras.getpagespeed.com/redhat/6/mainline/x86_64/RPMS/nginx-1.21.3-1.el6.ngx.x86_64.rpm and tried to download and install it, but it didn't actually download. Registration is required and apparently they whitelist registered users by IP. Jumping through those hoops would not be sustainable for me in my work environment so I discarded getpagespeed.com.

I have no idea why, but next I tried this https://nginx.org/packages/centos/6/x86_64/RPMS/nginx-1.18.0-2.el6.ngx.x86_64.rpm Older than the pervious nginx.org package, but it isn't in the mainline repo and I'm Centos n00b enough that I don't know the difference. Again, strings /usr/sbin/nginx|grep brot showed nothing, so that got a quick rpm -e nginx.

Next, I tried curl -LO https://repo.aerisnetwork.com/archive/nginx-more-1.13.6-1.el6.x86_64.rpm and https://repo.aerisnetwork.com/archive/nginx-more-1.14.2-4.el6.x86_64.rpm. (No idea why I grabbed 1.13 and 1.14.) There were a bunch of various nginx packages on repo.aerisnetwork.com For this to work there were some requirements so I ran yum install gd libxslt but there was still the case of libmaxminddb.so.0

Next, I tried repo.codeit.guru packages. I don't recall exactly where I found this repo. I think it was multiple sources. One of them was https://nixcp.com/brotli-compression-nginx/. I'd probably also seen https://codeit.guru/en_US/2020/04/nginx-1-18-0-stable-with-brotli-support-tls-1-3-final-rfc-8446-built-against-openssl-1-1-1g-for-red-hat-enterprise-linux-and-centos/. I was desperate enough to copy and page the root repo URL and hope for a Centos 6 directory. There was one!

curl -LO https://repo.codeit.guru/packages/mainline/centos/6/x86_64/nginx-1.19.5-1.el6.codeit.x86_64.rpm
rpm -i ./nginx-1.19.5-1.el6.codeit.x86_64.rpm

warning: ./nginx-1.19.5-1.el6.codeit.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 898b43f4: NOKEY
error: Failed dependencies:
    libbrotli = 1:1.0.7 is needed by nginx-1:1.19.5-1.el6.codeit.x86_64
    libmaxminddb.so.0()(64bit) is needed by nginx-1:1.19.5-1.el6.codeit.x86_64

curl -LO https://repo.codeit.guru/packages/mainline/centos/6/x86_64/libbrotli-1.0.7-1.codeit.el6.x86_64.rpm
rpm -i ./libbrotli-1.0.7-1.codeit.el6.x86_64.rpm

There is obviously brotli support but also the same libmaxminddb.so.0 dependency.

libmaxminddb on Centos 6

Now I had to briefly forget nginx and focussing on where to find this dependency. Surely there is a package?

$ yum install libmaxmindddb
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
No package libmaxmindddb available.
Error: Nothing to do
$ yum install libmaxmindddb-dev
...
No package libmaxmindddb-dev available.
$ yum install libmaxmindddb-devel
...
No package libmaxmindddb-devel available.

It required much searching. I wondered if it was in the nginx-module-geoip rpm, but no.

Finally, I found some comments about epel. I didn't have epel repo enabled. It is not part of vault AFAICT, but I was able to web browse epel and find a package.

curl -LO https://archives.fedoraproject.org/pub/archive/epel/6/x86_64/Packages/l/libmaxminddb-1.1.1-5.el6.x86_64.rpm
rpm -i ./libmaxminddb-1.1.1-5.el6.x86_64.rpm

And now nginx package from codeit.guru can be installed.

$ rpm -i ./nginx-1.19.5-1.el6.codeit.x86_64.rpm 
warning: ./nginx-1.19.5-1.el6.codeit.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 898b43f4: NOKEY
----------------------------------------------------------------------

Thanks for using nginx!

Please find the official documentation for nginx here:
* http://nginx.org/en/docs/

Please subscribe to nginx-announce mailing list to get
the most important news about nginx:
* http://nginx.org/en/support.html

Commercial subscriptions for nginx are available on:
* http://nginx.com/products/

----------------------------------------------------------------------
$ rpm -ql nginx
/etc/logrotate.d/nginx
/etc/nginx
/etc/nginx/conf.d
/etc/nginx/conf.d/default-ssl.conf.example
/etc/nginx/conf.d/default.conf
/etc/nginx/fastcgi_params
/etc/nginx/koi-utf
/etc/nginx/koi-win
/etc/nginx/mime.types
/etc/nginx/modules
/etc/nginx/nginx.conf
/etc/nginx/php.inc
/etc/nginx/scgi_params
/etc/nginx/uwsgi_params
/etc/nginx/win-utf
/etc/rc.d/init.d/nginx
/etc/rc.d/init.d/nginx-debug
/etc/sysconfig/nginx
/etc/sysconfig/nginx-debug
/usr/lib64/nginx
/usr/lib64/nginx/modules
/usr/sbin/nginx
/usr/sbin/nginx-debug
/usr/share/doc/nginx-1.19.5
/usr/share/doc/nginx-1.19.5/COPYRIGHT
/usr/share/man/man8/nginx.8.gz
/usr/share/nginx
/usr/share/nginx/html
/usr/share/nginx/html/50x.html
/usr/share/nginx/html/index.html
/var/cache/nginx
/var/log/nginx
$ strings /usr/sbin/nginx | grep brot
ngx_http_brotli_static_module_ctx
ngx_http_brotli_filter_module
ngx_http_brotli_static_module
...

Look at those beautiful symbols!

Next task is to discuss with the team all of the reasons that we should NOT integrate this into our environment and instead migrate off of Centos 6 instead.

Go Standards Project Layout

Jay R. Wren — Tue, 11 May 2021 15:48:21 +0000

There are only two hard things in Computer Science: cache invalidation and naming things.

-- Phil Karlton

For new Go programmers a third hardest thing might be package layout. The truth is, it doesn't have to be hard at all.

Alright, here is the secret to the hard party: forget everything you know from other languages, unless that other language is C.

Strangely, this might be the most controversial issue in the Go community, including advice so bad that project contributors had to file a bug: https://github.com/golang-standards/project-layout/issues/117

Honestly, https://medium.com/@benbjohnson/standard-package-layout-7cdbc8391fc1#.ds38va3pp is probably the best advice, but don't feel that you need to follow all of it immediately. You probably won't have mocks at first. You probably don't need to decouple things at first. Remember, loose coupling is only a good thing if you need it. Otherwise you are paying a cost. Simplicity is far more valuable than future flexibility especially when YAGNI.

There is a list of some great TODO and DO NOT items here:
https://rakyll.org/style-packages/

Along with common and util, I believe api is a package name which should be avoided. It should be obvious why: all packages have an API. api certainly is a short name, but there is nothing clear about it. https://blog.golang.org/package-names Package names should be short and clear. If you want to name a package api, instead, ask yourself, api for what? and name the package the answer to that question.

Strangely, there is a ton of official guidance on that page which is completely ignored by other recommendations. I'll link it again: https://blog.golang.org/package-names

When you read someone else's advice, such as: https://peter.bourgon.org/go-best-practices-2016/#repository-structure consider each point against the official advice of https://blog.golang.org/package-names and if it doesn't fit, discard it. Also consider each point against YAGNI.

If you read enough of these advice posts you'll being to notice some are not prescriptive. That is great. Prescriptive advice is harmful. No one can know your specific needs and circumstances. Following the wrong advice can be very expensive in the long run. Instead look at advice like https://www.ardanlabs.com/blog/2017/02/design-philosophy-on-packaging.html Notice how it is about philosophy and usability. It is not about how the programmer writing the code feels or wants things to be.

At any step, ask yourself, has this practice added value? If it hasn't, then don't do it. Undo it even. Value is also subjective. While I don't find a pkg dir to have any value, another team might find it to be very valuable to put everything there, away from the non-Go code things (docs, scripts, etc) in their repo. That is fine. Recognize that there is no right on wrong here, there is only our different values.

My favorite line of advice is from: https://christine.website/blog/within-go-repo-layout-2020-09-07 which reasons about the advice to say: "...Leaves the development team a lot more agency to decide how to name things" Take your agency. Ignore prescription.

All that said, how would I divide MVC into packages I wouldn't... until I need to.

Limit the Number of Connections to an Endpoint In Go

Jay R. Wren — Fri, 07 May 2021 17:42:06 +0000

I found myself implementing this pattern twice in recent times. As a result, I thought I'd share it.

First, some background. I wrote a web crawler for reasons. The crawler would find links in a webpage and create a go routine to follow the link and download what was at that endpoint, and recurse. Quickly, this caused the system running the crawler to run out of open file handles. Goroutines are so powerful that we had the opposite problem that we have in a serial downloader. Rather than going to slowly because of a one at a time situation, we did so much at once that we ran out of free file handles. I just wrote another crawler type program which downloads links discovered via a JSON API and needed the same solution.

Browsers are examples of good behavior to the point that they don't open very many connections to remote hosts. There is a great stack overflow post https://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser which describes how web browsers limit connections. It is very easy to make my Go programs behave in a similar way.


func main() {
    ...
    for {
        go httpSave(currentURL, currentFileName)
    }
    ...
}

// httpSave saves u to file named name.
func httpSave(u, name string) {
    resp, err := http.Get(u)
    if err != nil {
        // Queue errors?
        log.Print(err)
        return
    }
    f, err := os.OpenFile(name, os.O_RDWR|os.O_CREATE, 0666)
    if err != nil {
        // Queue errors?
        log.Print(err)
    }
    defer f.Close()
    n, err := io.Copy(f, resp.Body)
    log.Printf("wrote %d bytes to %s", n, name)
}

The above httpSave function is real, and the main is partially omitted code for how it might be used.

There are many ways one might limit the number of connections, but a simple to understand way IMO, rather than using locks and a counter is to use a channel of length N where N is the number of concurrent things. I'm still going to use a lock, but that is for synchronizing access to the map which maps the host we are limiting to the channel.

I'm using globals, but for a larger app I would add the map and lock to the struct that holds data around my app state for this operation.

var domainLimit map[string]chan struct{}
var domainListMutex sync.Mutex

func main() {
    domainLimit = make(map[string]chan struct{})
    ...
}


func getDomainToken(u string) func() {
    u2, err := url.Parse(u)
    if err != nil {
        log.Print(err)
        return func() {}
    }

    domainListMutex.Lock()
    defer domainListMutex.Unlock()
    f, ok := domainLimit[u2.Host]
    if !ok {
        f = make(chan struct{}, 6) // Six connections per host.
        domainLimit[u2.Host] = f
    }
    f <- struct{}{}
    return func() {
        <-f
    }
}

To use this, I add 1 line to the top of the httpSave function.

func httpSave(u, name string) {
    defer getDomainToken(u)()
    ...

If I had other functions doing http operations I'd need to add that same one line to the top of them. In my case, I don't.

I like it because it is simple and it works.

How to make a nice high dev salary into not as much, frugally

Jay R. Wren — Fri, 07 May 2021 17:34:35 +0000

Let's say you are senior and have a lot of experience and make well above the average salary and you aren't located coastal where everything is in a bubble. Let's take a nice round number, $150,000 salary.

Max out your 401k. -> 150 - 19 = 131
Max out your IRA AND your spouse's IRA. -> 131 - 12 = 119
Max out family HSA. -> 119 - 7 = 112
Max out ESPP @ 10% of your salary. -> 112 - 15 = 97
Save $3600/yr for child's college. -> 97,000 - 3600 = 93,400

Wow, 37% ($56,600) of income is going to saving for the future and 68% ($38,000) of that is before taxes.

I was reading this post by Troy Hunt https://www.troyhunt.com/10-personal-finance-lessons-for-technology-professionals/ and I realized I wanted to know my own numbers and so I ran these myself (These are not exactly my numbers).

Concurrent Thinking Might Lead to Better Code

Jay R. Wren — Fri, 07 May 2021 17:33:18 +0000

For whatever reason I was listening to Rob Pike talk about Concurrency is not Parallelism yet again. I had also been using Kafka for the first time recently. It triggered an epiphany.

I first heard about Kafka around six or seven years ago. I asked what it was and I was told of its wonders and how it would solve all my problems and make everything better and how any app not written around Kafka was inferior and it is the future and blah blah blah. I immediately disregarded it all and ignored it.

Having now used it, I can say for certain that I don't know what the fuss is about. I'd never choose to use it again. I'd never design an app around it. I would, instead, strive for simplicity and solve the types of problems which Kafka solves a different way, preferably by avoiding them altogether.

Now, while listening to Rob Pike talk about concurrency it occurred to me one possibility why Kafka was so beloved by those who espouse it. It is a JVM project and far more heavily used in that circle than outside of it. JVM folk are very likely used to a certain way of doing things and building around Kafka entirely breaks those ways. It shakes things up, possibly in a good way. Kafka creates a communication boundary that wouldn't have otherwise existed in a monolithic JVM app. Kafka creates the need for multiple processes doing smaller tasks utilizing varying Kafka features, rather than a single monolith. The requirement of these structures could be a huge benefit and explains the sentiment which I heard all of those years ago.

I think I've learned a bit thinking about these things...

... or I'm completely full of shit.

Consider Not Using 3rd Party CDNs

Jay R. Wren — Sun, 11 Oct 2020 19:41:35 +0000

I just read https://shkspr.mobi/blog/2020/10/please-stop-using-cdns-for-external-javascript-libraries/ and I feel the need to expound on the points.

Caching

Great questions from @edent, but no answers. I am disappointed. Thankfully, a bit of searching and I learned some things.

https://zoompf.com/blog/2010/01/should-you-use-javascript-library-cdns/ says that the network effect of caching simply isn't there. That was in 2010. With even more varying javascript libraries in 2020, it is probably even more true. Further, because of latency of DNS lookup and TLS session creation, a connection to a new 3rd party host is slower than doing none of that because it is already done. Further still, Hoffman says that assuming a slowish internet speed -- although not all that slow for a highly variable mobile connection -- the time to connect is about the same as the time to download at ⅓ of a second. Excellent analysis by Hoffman, I highly recommend reading his post.

Speed

Eden gives a strawman argument. I am disappointed. Of course we want our sites to be fast. Attempting to make them faster by misguidedly offloading some large assets to another server isn't a wrong idea. The notion that you shouldn't do A unless you can do B is foolish. But none of it matters given the argument from 1, so we can ignore 2.

Versioning

Indeed the low hit rates cited by Hoffman in #1 are surely contributed to by the varying versions of libraries out there. This is the world of javascript where there is a new version every week and so there are 520 versions of jquery in the past 10 yrs alone. ;)

Reliability

Unless you ARE a CDN, it is highly like that the CDN will be more reliable than your own site. I do not feel this is a believable argument.

Privacy

Excellent questions, but I'd go much further and not form them as a question. By using a CDN you are giving away all of your users usage patterns to a 3rd party. This is unacceptable.

Security

Excellent point, and while IMO it is far less likely that a popular, well-run CDN is going to be compromised than your typical website run by our average IT department or devops group, it is absolutely imperative that we consider the risk.

So What?

So, understand what risk you are taking by using a 3rd party CDN and consider not using them.

SOLID principles in the Go programming language

Jay R. Wren — Tue, 31 Dec 2019 03:19:24 +0000

There was a discussion on SOLID principles in the Go programming language and I threw in my two-cents. Once I was done writing, I was surprised with what I came up with.

Here is why you shouldn't think very much about SOLID in Go:

Consider each principle:

Single Responsibility Principle:

"A class should only have a single responsibility..."

Go doesn't have classes. Already the principle doesn't apply...

I write that tongue in cheek, because of course a struct is a bit like a class, and of course SRP is great for a Go struct type, but also, since Go struct types do not have inheritance, we don't get ~classes~types inheriting behaviors. Thus we don't have to worry about un-intended inherited responsibilities. It is easier to achieve SRP in Go.

Open-close Principle:

"Software entities ... should be open for extension, but closed for modification."

No inheritance, thus no extension. Go gets this for free.

Liskov substitution principle:

"Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program." See also design by contract.

Go doesn't have subtypes. YAY. So this only applies to interfaces implementations. Yes, it is an absolute must for interface implementations.

Interface segregation principle:

"Many client-specific interfaces are better than one general-purpose interface."

This is such idiomatic Go that it should go without saying.

Dependency Inversion Principle:

One should "depend upon abstractions, not concretions"

This is another way of saying accept interfaces. Yes, we value this highly in Go.

In summary:
We think about things slightly differently in Go, because Go isn't OOP, but once translated to Go's type system, SOLID absolutely matches with Go idioms.