DEV Community: Maël Valais

Logging into Synology NAS with Personal Google Accounts

Maël Valais — Wed, 17 Apr 2024 17:50:00 +0000

Introduction

My parents and I share a DS923+. We mainly use it for storing photos. To save energy, the NAS is stopped at night and restarted in the morning. Restarting the NAS means that you need to log in to the UI at least once every day.

And since my parents don't use the Synology often, they would first struggle remembering what the username thing is about. And then, they would forget about their password, which would lead to me having to reset it because Synology hasn't implemented a way to reset the password over email:

Over the past two years, my parents forgot their passwords a couple of times. It led me to look for an alternative way to log into the NAS... why not use their Google accounts using the single sign-on mechanism?

Challenges with Local Accounts and SSO

A couple of years ago, Synology introduced SSO (single sign-on). Since 7.2, DSM supports generic OIDC providers, and supports logging into local users (it used to be only possible for LDAP users).

Since my parents are always signed in into their Google account, I figured it would be possible to use OIDC with Google... Except it won't work with local accounts.

Here is what I tried: I created a My OIDC configuration for Google looked like this:

Then, I configured my Synology to use Google's OIDC endpoint:

The problem arose with the "Username claim". I want to log into my local account mael.valais, but none of the claims in Google's ID tokens contain that username. Here is an example of a Google ID token:

{
  "iss": "https://accounts.google.com",
  "aud": "1234987819200.apps.googleusercontent.com",
  "sub": "10769150350006150715113082367",
  "email": "jsmith@example.com",
  "email_verified": "true",
  "iat": 1353601026,
  "exp": 1353604926
}

Forking Dex to use it as an OIDC middleware for Google OIDC

I figured I could use Dex to act as a middleware between Synology's OIDC client and Google's OIDC server. My goal was to "augment" Google's JWTs with Synology's usernames by looking up the user by email.

Dex isn't as flexible as I would have hoped. To make it work, I had to fork it to change the internals of the Google OIDC connector.

Fork: https://github.com/maelvls/dex/tree/google-to-synology-sso

This fork is a fork of the fork presented in https://github.com/dexidp/dex/pull/2954. It builds on the idea of the ExtendPayload interface, which I slightly adjusted to pass the original claims since I needed access to the email contained in the JWT provided by Google.

With this fork, you will need to set three more environment variables:

SYNO_PASSWD=redacted
SYNO_USER=mael.valais
SYNO_URL=http://127.0.0.1:5000

When the OIDC flow with Google is done and before Dex issues its own JWT, I added some code to add the claim username. With this modified Dex, the JWT looks like this:

{
  "at_hash": "-j6HZYvzDaqkQB2KxIgSyw",
  "aud": "caddy",
  "c_hash": "8SK3tobDYgaI3cnDzkmi5g",
  "email": "mael65@gmail.com",
  "email_verified": true,
  "exp": 1713387587,
  "iat": 1713301187,
  "iss": "https://login.mysynodomain.dev/dex",
  "name": "Maël Valais",
  "nonce": "MFZFSkESL1XqdQmbvr0T43Kn7v0CzLap",
  "sub": "ChUxMDAzNjk3OTQzNjg3MDAwOTk5MTISBmdvb2dsZQ",
  "username": "mael.valais"
}

Here is the updated configuration in DSM:

Using the fork of Dex

Create a file dex.yaml on your NAS:

issuer: https://login.mysynodomain.dev/dex

storage:
  type: sqlite3
  config:
    file: dex.sqlite

web:
  http: 0.0.0.0:5556

logger:
  level: debug

oauth2:
   skipApprovalScreen: true
   alwaysShowLoginScreen: false

staticClients:
- id: synology
  name: 'Synology'
  redirectURIs:
  - 'https://mysynodomain.dev/'
  secret: foo # Use openssl rand -hex 16 to generate this.

connectors:
- type: google
  id: google
  name: Google
  config:
    issuer: https://accounts.google.com
    clientID: $GOOGLE_CLIENT_ID
    clientSecret: $GOOGLE_CLIENT_SECRET
    redirectURI: https://login.mysynodomain.dev/dex/callback

# I have disabled email login.
enablePasswordDB: false

Finally, run Dex:

docker run --name dex -d \
  -v $HOME/dex.yaml:/dex.yaml \
  -v $HOME/dex.sqlite:/dex.sqlite \
  -e SYNO_PASSWD=redacted \
  -e SYNO_USER=mael.valais \
  -e SYNO_URL=http://127.0.0.1:5000 \
  -e GOOGLE_CLIENT_ID=207842732284-l7nhetlsvimmds80fa2knir8fundp3h4.apps.googleusercontent.com \
  -e GOOGLE_CLIENT_SECRET=redacted \
  -p 5556:5556 \
  ghcr.io/maelvls/dex:google-to-synology-sso-v2@sha256:252713d98c8369612994fbbed6f257d79dc35ff84b2cbb6952a11d63c57b64bb serve /dex.yaml

With this command, you will be using Dex images that I built:
ghcr.io/maelvls/dex
I wouldn't recommend using random Docker images from the internet, especially since this is about authentication. I might be a malicious actor trying to steal your Synology credentials! But if you still want to proceed, here is an image! Note that I am not monitoring the image for security vulnerabilities, and do not guarantee that it is secure. Use at your own risk!

The Docker image

The Docker image is available on GitHub Container Registry: https://ghcr.io/maelvls/dex.

Rebuilding the image yourself and pushing it to your Synology NAS

First, install zig and ko. That will allow you to cross-compile Dex to linux/amd64 on macOS without Buildx (cross-compiling is required because Dex's sqlite library needs CGO)

brew install ko zig

Clone the fork:

git clone https://github.com/maelvls/dex --branch google-to-synology-sso

Then, build the image:

CC="zig cc -target x86_64-linux" CXX="zig c++ -target x86_64-linux" CGO_ENABLED=1 \
  KO_DOCKER_REPO=ghcr.io/maelvls/dex \
  KO_DEFAULTBASEIMAGE=alpine \
  ko build ./cmd/dex --bare --tarball /tmp/out.tar --push=false

Then, copy the image to your NAS:

ssh yournas /usr/local/bin/docker load </tmp/out.tar

(Just so that I don't forget) Here is how I pushed `ghcr.io/maelvls/dex` to GitHub Container Registry

export VERSION=google-to-synology-sso-v6
git tag $VERSION -m "Release $VERSION"
git push maelvls $VERSION

Then:

CC="zig cc -target x86_64-linux" CXX="zig c++ -target x86_64-linux" CGO_ENABLED=1 \
  KO_DOCKER_REPO=ghcr.io/maelvls/dex \
  KO_DEFAULTBASEIMAGE=alpine \
  ko build ./cmd/dex --bare --push=true --tags $VERSION \
    --image-annotation "org.opencontainers.image.created=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
    --image-annotation "org.opencontainers.image.url=https://maelvls.dev/synology-sso-with-personal-google-account/" \
    --image-annotation "org.opencontainers.image.source=https://github.com/maelvls/dex" \
    --image-annotation "org.opencontainers.image.version=$VERSION" \
    --image-annotation "org.opencontainers.image.revision=$(git rev-parse HEAD)" \
    --image-annotation "org.opencontainers.image.vendor=Maël Valais" \
    --image-annotation "org.opencontainers.image.title=google-to-synology-sso" \
    --image-annotation "org.opencontainers.image.description=Fork of Dex to use Synology SSO with Google accounts" \
    --image-annotation "org.opencontainers.image.documentation=https://maelvls.dev/synology-sso-with-personal-google-account/" \
    --image-annotation "org.opencontainers.image.authors=Maël Valais <mael.valais@gmail.com>" \
    --image-annotation "org.opencontainers.image.licenses=Apache-2.0" \
    --image-annotation "org.opencontainers.image.ref.name=google-to-synology-sso"

See the history below to know the image hashes.

History

June 21st, 2025: v5

Release: google-to-synology-sso-v5

I somehow didn't realize that I was hardcoding the Synology URL. In this
version, I've added SYNO_URL (I thought I had already added it, but I
hadn't!).

I've also renamed the fork to google-to-synology-sso to help with
discoverability.

The image:

ghcr.io/maelvls/dex:google-to-synology-sso-v5@sha256:e805a95be565268421ccdb2271dfc0d85ae12b6b53cf82c47b294d34891ff3d1

June 14th, 2025: v4

Release: google-to-synology-sso-v4

The dex container kept crashing due to i/o timeouts when Dex was trying to connect to the Synology API. I fixed that by adding a retry mechanism with an exponential backoff and a maximum of 10 retries and maximum of 1 hour between retries.

The image:

ghcr.io/maelvls/dex:google-to-synology-sso-v4@sha256:f8bf15901c2b994337994c4f60c48c154437af656cbe85701cb8d1d7d94127ba

June 4th, 2025: v3

Release: google-to-synology-sso-v3

Reduced the Synology SSO loading time from 10 seconds to 1 second. The reason it was so slow is that I wasn't caching the Synology users and was fetching them every time someone was logging into Synology. The image:

ghcr.io/maelvls/dex:google-to-synology-sso-v3@sha256:d0d889e32400ef70529daef32e7a77bf9da021cbaff9954589db2204a5c49335

June 1st, 2025: v2

Release: google-to-synology-sso-v2

The google-to-synology-sso-v1 tag was buggy, the ExtendPayload func wasn't being called correctly. I've pushed google-to-synology-sso-v2 to fix that. Here is the new image:

ghcr.io/maelvls/dex:google-to-synology-sso-v2@sha256:252713d98c8369612994fbbed6f257d79dc35ff84b2cbb6952a11d63c57b64bb

Apr 12nd, 2024: v1

Release: google-to-synology-sso-v1

Image:

ghcr.io/maelvls/dex:google-to-synology-sso-v1@sha256:345c8fec6b222c308759f21864c6af3b16c373801fd5e0b7ad4b131a743d3b07

Conclusion

With this method, my parents can log into the NAS with their Google account and no longer have to remember their Synology username and password.

Although it works, I wish I didn't have to fork Dex to customize the claims it puts into the JWT payload. I came across a couple of designs that would aim to make Dex more extendable, but none have been implemented yet.

The login flow is much smoother now: click "Login with Google", select the Google account, and you're in! Just two screens:

mitmproxy hangs on TLS renegotiation: a debugging story

Maël Valais — Sat, 25 Sep 2021 08:31:00 +0000

I frequently use mitmproxy to inspect what HTTP requests and responses look like under the hood. Inspecting HTTP flows comes in handy with tools that tend to hide the actual JSON error messages.

One tool I have been using a lot is vcert. vcert allows you to request X.509 certificates from Venafi Trust Protection Platform (TTP) as well as Venafi Cloud.

In the following example, I give an unknown client ID duh. vcert just tells me about the error in the initial HTTP header:

$ vcert getcred --username foo --password bar --client-id=duh --verbose
vCert: 2021/09/19 19:23:56 Getting credentials...
vCert: 2021/09/19 19:23:56 Got 400 Bad Request status for POST https://venafi-tpp.platform-ops.jetstack.net/vedauth/authorize/oauth
vCert: 2021/09/19 19:24:05 unexpected status code on TPP Authorize. Status: 400 Bad Request

I am sure that the HTTP API is returning more than just the HTTP header 400 Bad Request!

But when I try using mitmproxy, I get the following error:

$ HTTPS_PROXY=:9090 vcert getcred --username foo --password bar --client-id=duh --verbose
vCert: 2021/09/19 19:29:20 Getting credentials...
vCert: 2021/09/19 19:29:50 Post "https://venafi-tpp.platform-ops.jetstack.net/vedauth/authorize/oauth": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Looking at the "Detail" tab for the HTTP call recorded in mitmproxy, I can see that the HTTP request was acknowledged by the TPP server (see the "Request complete" time). For some reason, the response never arrives.

Let us dig a bit deeper and see what is the difference between "with" and "without" mitmproxy using WireShark. Without mitmproxy, vcert succeeds:

But with mitmproxy, vcert hangs:

The difference between both flows seems to be the "Encrypted Handshake Message" that occurs right after the HTTP requests has been sent.

Note that you can see the above two PCAP traces by downloading vcert-with-and-without-mitmproxy-through-iap.pcapng.

Now, let us decrypt this TLS flow. WireShark being a passive capture tool (unlike mitmproxy), it won't be able to decrypt traffic. Fortunately, mitmproxy is able to dump the master secret (see the page Wireshark and SSL/TLS Master Secrets. After giving the master secret to WireShark, we can see the decrypted traffic:

The above screenshot validates the fact that the "Application Data" that we could see in the previous two screenshots was in fact the HTTP request.

We also notice that the mysterious "Encrypted Handshake Message" is a Hello Request, signifying a TLS renegotiation as described in RFC 5246.

As a side note, here is what the master secret look like when mitmproxy records it (that's the file I passed to WireShark):

# Content of sslkeylogfile
CLIENT_RANDOM 5bdb7b7d88a325a08ee922a89b7b8acbbcda36f7d800c4e2f763b4689cfd870b 083f22af3099558997f784c47c4145dd9155c20f97fa5701bffe7003d80c7ad31801cfabb9be5838bf8f4f58e6b971f7
SERVER_HANDSHAKE_TRAFFIC_SECRET caa526a0b8e35551a2feee9de685e74547a7b0abc3ae5422a4bebe42d74132b4 7ac126e6a9b651ee9585a902248ac5bc5cdfa3a8989852f491b52ec013a9be170dce24076c305c5b9e7209c325a9f530
CLIENT_HANDSHAKE_TRAFFIC_SECRET caa526a0b8e35551a2feee9de685e74547a7b0abc3ae5422a4bebe42d74132b4 37fa20e94a385b79901a0615ef7c2de091e98757e85958deb9a242a3b0bed74e3be290d816555a3da816dd60d4f0c9f3
EXPORTER_SECRET caa526a0b8e35551a2feee9de685e74547a7b0abc3ae5422a4bebe42d74132b4 d436ff8840245be887e15d11000a98d971df4f1f0d825b7eb470a37fe2e9f7d1759967fe52e055cb9b5e74373e36f276
SERVER_TRAFFIC_SECRET_0 caa526a0b8e35551a2feee9de685e74547a7b0abc3ae5422a4bebe42d74132b4 615847eb91285a78981cf5072a9597e2ef4549c92d695818b9e05c270950b5ec88fcb62969b353b38ea2cd18597ee01d
CLIENT_TRAFFIC_SECRET_0 caa526a0b8e35551a2feee9de685e74547a7b0abc3ae5422a4bebe42d74132b4 4705201529b85b5cbdcbd74dfdf5cf2019ff9e520de551d2aaaa14962668ed3aa0a79416d1795d5eb53be93ddeb077de

The master secret is comprised of 96 hexadecimal characters, which corresponds to 48 bytes.

It seems like mitmproxy does not support server-side initiated TLS renegotiation. Looking at its source code, it seems like everything after the end of the first handshake is considered to be data, and subsequent Handshake Messages make mitmproxy "hang".

Now, why would TLS renegotiation be necessary after the HTTP request is sent? The issue named Venafi Issuer error when configuring cert-manager. "local error: tls: no renegotiation" gives us a clue: Microsoft IIS, which is the web server used by Venafi TPP, is capable of authenticating an incoming connection using a client certificate depending on the HTTP request's URL path. Looking at the IIS configuration, we can see that the /vedauth endpoint has its "client certificate" settings configured to "Accept":

Let us see this behavior in action by creating a TLS tunnel using openssl. With the path /vedsdk, we can see that openssl does the initial TLS handshake and no renegotiation seem to happen after sending the HTTP request:

The HTTP request I made was:

HTTP /vedsdk/ HTTP/1.1
Host: venafi-tpp.platform-ops.jetstack.net
Content-Length: 0

With the endpoint /vedauth, which is the one for which mitmproxy is hanging on, openssl does the initial TLS handshake, sends the HTTP request and then a TLS renegotiation is triggered:

The request was:

HTTP /vedauth/ HTTP/1.1
Host: venafi-tpp.platform-ops.jetstack.net
Content-Length: 0

In the first example (/vedsdk), the HTTP response is sent immediately without any TLS negotiation. In the second example (/vedauth), a TLS renegotiation is performed right after the HTTP request is sent, and the HTTP response is sent over this new TLS session.

The only explanation for this behavior is that this allows IIS to refuse clients that present an invalid client certificates only for paths for which client certificates are enabled, instead of refusing to serve all paths. I would need to learn a bit more about TLS and how IIS is able to give the information about the identity of a client certificate to the HTTP layer to know whether it should look for an Authorization: Bearer foo HTTP header.

The only workaround I have come up with is to turn the client certificate option to "Ignore":

This time, vcert over mitmproxy works!

One thing I would like to do in the future is to add support for TLS renegotiation to mitmproxy!

Understanding the Available condition of a Kubernetes deployment

Maël Valais — Tue, 07 Jul 2020 14:33:00 +0000

The various conditions that may appear in a deployment status are not documented in the API reference. For example, we can see what are the fields for DeploymentConditions, but it lacks the description of what conditions can appear in this field.

In this post, I dig into what the Available condition type is about and how it is computed.

Since the API reference does not contain any information about conditions, the only way to learn more is to dig into the Kubernetes codebase; quite deep into the code, we can read some comments about the three possible conditions for a deployment:

// Available means the deployment is available, ie. at least the minimum available
// replicas required are up and running for at least minReadySeconds.
DeploymentAvailable DeploymentConditionType = "Available"

// Progressing means the deployment is progressing. Progress for a deployment is
// considered when a new replica set is created or adopted, and when new pods scale
// up or old pods scale down. Progress is not estimated for paused deployments or
// when progressDeadlineSeconds is not specified.
DeploymentProgressing DeploymentConditionType = "Progressing"

// ReplicaFailure is added in a deployment when one of its pods fails to be created
// or deleted.
DeploymentReplicaFailure DeploymentConditionType = "ReplicaFailure"

The description given to the Available condition type is quite mysterious:

At least the minimum available replicas required are up.

What does "minimum available replicas" mean? Is this minimum 1? I cannot see any minAvailable field in the deployment spec, so my initial guess was that it would be 1.

Before going further, let's the description attached to the reason MinimumReplicasAvailable. Apparently, this reason is the only reason for the Available condition type.

// MinimumReplicasAvailable is added in a deployment when it has its minimum
// replicas required available.
MinimumReplicasAvailable = "MinimumReplicasAvailable"

The description doesn't help either. Let's see what the deployment sync function does:

if availableReplicas + deploy.MaxUnavailable(deployment) >= deployment.Spec.Replicas {
    minAvailability := deploy.NewCondition("Available", "True", "MinimumReplicasAvailable", "Deployment has minimum availability.")
    deploy.SetCondition(&status, *minAvailability)
}

Note: in the real code, the max unavailable is on the right side of the inequality. I find it easier to reason about this inequality when the single value to the right is the desired replica number.

Ahhh, the actual logic being Available! If maxUnavailable is 0, then it becomes obvious: the "minimum availability" means that number of available replicas is greater or equal to the number of replicas in the spec; the deployment has minimum availability if and only if the following inequality holds:

available	+	acceptable unavailable	≥	desired
1️⃣		2️⃣		3️⃣

Let's take an example:

kind: Deployment
spec:
  replicas: 10 # 3️⃣ desired
  strategy:
    rollingUpdate:
      maxUnavailable: 2 # 2️⃣ acceptable unavailable
status:
  availableReplicas: 8 # 1️⃣ available
  conditions:
    - type: "Available"
      status: "True"

In this example, the inequality holds which means this deployment has "minimum availability" (= Available = True):

`availableReplicas`	+	`maxUnavailable`	≥	`replicas`
8		2		10

Default value for `maxUnavailable` is 25%

Now, what happens when maxUnavailable is not set? The official documentation maxUnavailable says:

maxUnavailable is an optional field that specifies the maximum number of Pods that can be unavailable during the update process. The value can be an absolute number (for example, 5) or a percentage of desired Pods (for example, 10%). The absolute number is calculated from percentage by rounding down. The value cannot be 0 if maxSurge is 0. The default value is 25%.

For example, when this value is set to 30%, the old ReplicaSet can be scaled down to 70% of desired Pods immediately when the rolling update starts. Once new Pods are ready, old ReplicaSet can be scaled down further, followed by scaling up the new ReplicaSet, ensuring that the total number of Pods available at all times during the update is at least 70% of the desired Pods.

Let's take an example with a deployment that has no maxUnavailable field set, and imagine that 4 pods are unavailable due to a resource quota that only allows for 5 pods to start:

kind: Deployment
spec:
  replicas: 10
status:
  availableReplicas: 5
  unavailableReplicas: 5
  conditions:
    - type: "Available"
      status: "False"

This time, the inequality does not hold:

`status.availableReplicas`	+	`spec.strategy.rollingUpdate.maxUnavailable`	≱	`spec.replicas`
5		lower(25% * 10) = 2		10

Let's dig a bit more and see how MaxAvailable is defined:

// MaxUnavailable returns the maximum unavailable pods a rolling deployment
// can take.
func MaxUnavailable(deployment apps.Deployment) int {
    if !IsRollingUpdate(&deployment) || deployment.Spec.Replicas == 0 {
        return 0
    }
    _, maxUnavailable, _ := ResolveFenceposts(deployment.Spec.Strategy.RollingUpdate.MaxSurge, deployment.Spec.Strategy.RollingUpdate.MaxUnavailable, *(deployment.Spec.Replicas))
    if maxUnavailable > *deployment.Spec.Replicas {
        return *deployment.Spec.Replicas
    }
    return maxUnavailable
}

The core of the logic behind maxUnavailable is in ResolveFenceposts (note: I simplified the code a bit to make it more readable):

// ResolveFenceposts resolves both maxSurge and maxUnavailable. This needs to happen in one
// step. For example:
//
// 2 desired, max unavailable 1%, surge 0% - should scale old(-1), then new(+1), then old(-1), then new(+1)
// 1 desired, max unavailable 1%, surge 0% - should scale old(-1), then new(+1)
// 2 desired, max unavailable 25%, surge 1% - should scale new(+1), then old(-1), then new(+1), then old(-1)
// 1 desired, max unavailable 25%, surge 1% - should scale new(+1), then old(-1)
// 2 desired, max unavailable 0%, surge 1% - should scale new(+1), then old(-1), then new(+1), then old(-1)
// 1 desired, max unavailable 0%, surge 1% - should scale new(+1), then old(-1)
func ResolveFenceposts(maxSurge, maxUnavailable *instr.IntOrString, desired int) (int, int, error) {
    surge, _       := instr.GetValueFromIntOrPercent(instr.ValueOrDefault(maxSurge, instr.FromInt(0)), desired, true)
    unavailable, _ := instr.GetValueFromIntOrPercent(instr.ValueOrDefault(maxUnavailable, instr.FromInt(0)), desired, false)
    return surge, unavailable, nil
}

The false boolean turns the integer rounding "up", which means 0.5 will be rounded to 0 instead of 1.

The maxUnavailable and maxSurge (they call them "fenceposts" values) are simply read from the deployment's spec. In the following example, the deployment will become Available = True only if there are at least 5 replicas available:

kind: Deployment
spec:
  replicas: 10
  strategy:
    rollingUpdate:
      maxUnavailable: 0% # 🔰 10 * 0.0 = 0 replicas

Hands-on example with the `Available` condition

Imagine that we have a namespace named restricted that only allows for 200 MiB, and our pod requires 50 MiB. The first 4 pods will be successfully created, but the fifth one will fail.

Let us first apply the following manifest:

apiVersion: v1
kind: Namespace
metadata:
  name: restricted
--------
apiVersion: v1
kind: ResourceQuota
metadata:
  namespace: restricted
  name: mem-cpu-demo
spec:
  hard:
    requests.memory: 200Mi
--------
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: restricted
  name: test
spec:
  replicas: 5
  strategy:
    rollingUpdate:
      maxUnavailable: 0 # 🔰
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - name: test
          image: nginx:alpine
          resources:
            requests:
              memory: "50Mi" # the 5th pod will fail (on purpose)
          ports:
            - containerPort: 80

After a few seconds, the Available condition stabilizes to False:

# kubectl -n restricted get deploy test -oyaml
kind: Deployment
spec:
  replicas: 5
  strategy:
    rollingUpdate:
      maxUnavailable: 0 # 🔰
status:
  conditions:
    - lastTransitionTime: "2020-07-07T14:04:27Z"
      lastUpdateTime: "2020-07-07T14:04:27Z"
      message: Deployment does not have minimum availability.
      reason: MinimumReplicasUnavailable
      status: "False"
      type: Available
    - lastTransitionTime: "2020-07-07T14:04:27Z"
      lastUpdateTime: "2020-07-07T14:04:27Z"
      message: 'pods "test-7df57bd99d-5qw47" is forbidden: exceeded quota: mem-cpu-demo, requested: requests.memory=50Mi, used: requests.memory=200Mi, limited: requests.memory=200Mi'
      reason: FailedCreate
      status: "True"
      type: ReplicaFailure
    - lastTransitionTime: "2020-07-07T14:04:27Z"
      lastUpdateTime: "2020-07-07T14:04:38Z"
      message: ReplicaSet "test-7df57bd99d" is progressing.
      reason: ReplicaSetUpdated
      status: "True"
      type: Progressing
  replicas: 4
  availableReplicas: 4
  readyReplicas: 4
  unavailableReplicas: 1
  updatedReplicas: 4

We are asking for at most 0 unavailable replicas and there is 1 unavailable replica (due to the resource quota). Thus, the "minimum availability" inequality does not hold which means the deployment has the condition Available = False:

`availableReplicas`	+	`rollingUpdate.maxUnavailable`	≱	`replicas`
4		0		5

Update 9 July 2020: added a paragraph on the default value for maxUnavailable, and fixed the yaml example where I had mixed unavailableReplicas with maxUnavailable.

Pull-through Docker registry on Kind clusters

Maël Valais — Fri, 03 Jul 2020 13:13:00 +0000

Note: (August 2023) As mentioned by Wolfgang Schnerring in this comment, the method presented in this document is very limited: many images are still being pulled directly and it only proxies a single "upstream" registry. Nowadays, docker-registry-proxy is a better alternative: https://github.com/rpardini/docker-registry-proxy#kind-cluster.

TL;DR:

to create a local pull-through registry to speed up image pulling in a Kind cluster, run:

  docker run -d --name proxy --restart=always --net=kind \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2
  kind create cluster --config /dev/stdin <<EOF
  kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  containerdConfigPatches:
    - |-
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["http://proxy:5000"]
  EOF

you can't use this pull-through proxy registry to push your own images (e.g. to speed up Tilt builds), but you can create two registries (one for caching, the other for local images). See this section for more context; the lines are:

  docker run -d --name proxy --restart=always --net=kind \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2
  docker run -d --name registry --restart=always --net=kind \
    -p 5000:5000 registry:2
  kind create cluster --config /dev/stdin <<EOF
  kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  containerdConfigPatches:
    - |-
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["http://proxy:5000"]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
        endpoint = ["http://registry:5000"]
  EOF

in case you often create & delete Kind clusters, using a local registry that serves as a proxy avoids redundant downloads
KIND_EXPERIMENTAL_DOCKER_NETWORK is useful but remember that the default network (bridge) doesn't have DNS resolution for container hostnames
the Docker default network (bridge) has limitations as [detailed by Docker][default-bridge].
If you play with ClusterAPI with its Docker provider, you might not be able to use a local registry due to the clusters being created on the default network, which means the "proxy" hostname won't be resolved (but we could work around that).

Kind is an awesome tool that allows you to spin up local Kubernetes clusters locally in seconds. It is perfect for Kubernetes developers or anyone who wants to play with controllers.

One thing I hate about Kind is that images are not cached between two Kind containers. Even worse: when deleting and re-creating a cluster, all the downloaded images disappear.

In this post, I detail my discoveries around local registries and why the default Docker network is a trap.

Contents:

Kind has no image caching mechanism
Creating a caching proxy registry
Creating a Kind cluster that knows about this caching proxy registry
Check that the caching proxy registry works
Docker proxy vs. local registry
Improving the ClusterAPI docker provider to use a given network

Kind has no image caching mechanism

Whenever I re-create a Kind cluster and try to install ClusterAPI, all the (quite heavy) images have to be re-downloaded. Just take a look at all the images that get re-downloaded:

# That's the cluster created using 'kind create cluster'
% docker exec -it kind-control-plane crictl images
IMAGE                                                                      TAG      SIZE
quay.io/jetstack/cert-manager-cainjector                                   v0.11.0  11.1MB
quay.io/jetstack/cert-manager-controller                                   v0.11.0  14MB
quay.io/jetstack/cert-manager-webhook                                      v0.11.0  14.3MB
us.gcr.io/k8s-staging-capi-docker/capd-manager/capd-manager-amd64          dev      53.5MB
us.gcr.io/k8s-artifacts-prod/cluster-api/cluster-api-controller            v0.3.0   20.3MB
us.gcr.io/k8s-artifacts-prod/cluster-api/kubeadm-bootstrap-controller      v0.3.0   19.6MB
us.gcr.io/k8s-artifacts-prod/cluster-api/kubeadm-control-plane-controller  v0.3.0   21.1MB

# I also use a ClusterAPI-created cluster (relying on CAPD):
% docker exec -it capd-capd-control-plane-l4tx7 crictl images ls
docker.io/calico/cni                  v3.12.2             8b42391a46731       77.5MB
docker.io/calico/kube-controllers     v3.12.2             5ca01eb356b9a       23.1MB
docker.io/calico/node                 v3.12.2             4d501404ee9fa       89.7MB
docker.io/calico/pod2daemon-flexvol   v3.12.2             2abcc890ae54f       37.5MB
docker.io/metallb/controller          v0.9.3              4715cbeb69289       17.1MB
docker.io/metallb/speaker             v0.9.3              f241be9dae666       19.2MB

That's a total of 418 MB that get re-downloaded every time I restart both clusters!

Unfortunately, there is no way to re-use the image registry built into your default Docker engine (both on Linux and on macOS). One solution to this problem is to spin up an intermediary Docker registry in a side container; as long as this container exists, all the images that have already been downloaded once can be served from cache.

Creating a caching proxy registry

We want to create a registry with a simple Kind cluster; let's start with the registry:

docker run -d --name proxy --restart=always --net=kind -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2

Details:

--net kind is required because Kind creates its containers in a separate network; it does that the because the "bridge" has [limitations][default-bridge] and [doesn't allow you][dns-services] to use container names as DNS names:

By default, a container inherits the DNS settings of the host, as defined in the /etc/resolv.conf configuration file. Containers that use the default bridge network get a copy of this file, whereas containers that use a custom network use Docker’s embedded DNS server, which forwards external DNS lookups to the DNS servers configured on the host.

which means that the container runtime (containerd) that runs our Kind cluster won't be able to resove the address proxy:5000.

REGISTRY_PROXY_REMOTEURL is required due to the fact that by default, the registry won't forward requests. It simply tries to find the image in /var/lib/registry/docker/registry/v2/repositories and returns 404 if it doesn't find it.

Using the pull-through feature (I call it "caching proxy"), the registry will proxy all requests coming from all mirror prefixes and cache the blobs and manifests locally. To enable this feature, we set REGISTRY_PROXY_REMOTEURL.

Other interesting bit about REGISTRY_PROXY_REMOTEURL: this environement variable name is mapped from the registry YAML config API. The variable
REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io
is equivalent to the following YAML config:
proxy:
  remoteurl: https://registry-1.docker.io

⚠️ The registry can't be both in normal mode ("local proxy") and in caching proxy mode at the same time, see below.

[default-bridge]: https://docs.docker.com/network/bridge/#use-the-default-bridge-network
[dns-services]: https://docs.docker.com/config/containers/container-networking/#dns-services

Creating a Kind cluster that knows about this caching proxy registry

The second step is to create a Kind cluster and tell the container runtime to use a specific registry; here is the command to create it:

kind create cluster --config /dev/stdin <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["http://proxy:5000"]
EOF

Note: containerdConfigPatches is a way to semantically patch /etc/containerd/config.conf. By default, this file looks like:

% docker exec -it kind-control-plane cat /etc/containerd/config.toml
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]

Note 2: the mirror prefix (docker.io) can be omitted for images stored on Docker Hub. For other registries such as gcr.io, this mirror prefix has to be given. Here is a table with some examples of image names that are first prepended with "docker.io" if the mirror prefix is not present, and we get the final address by mapping these mirror prefixes with mirror entries:

image name	"actual" image name	registry address w.r.t. mirrors
alpine	docker.io/alpine	https://registry-1.docker.io/v2/library/alpine/manifests/latest
gcr.io/istio-release/pilot	gcr.io/istio-release/pilot	https://gcr.io/v2/istio-release/pilot/manifests/1.9.1
foo.org/something/someimage	foo.org/something/someimage	https://foo.org/v2/something/someimage/manifests/latest

Check that the caching proxy registry works

Let's see if the proxy registry works by running a pod:

% kubectl run foo -it --rm --image=nicolaka/netshoot
% docker exec -it proxy ls /var/lib/registry/docker/registry/v2/repositories
nicolaka

We can also see through the registry logs that everything is going well:

# docker logs proxy | tail
time="2020-07-26T14:52:44.2624761Z" level=info msg="Challenge established with upstream : {https registry-1.docker.io /v2/}" go.version=go1.11.2 http.request.host="proxy:5000" http.request.id=15e9ac86-7d79-4883-a8ce-861a7484887c http.request.method=HEAD http.request.remoteaddr="172.18.0.2:57588" http.request.uri="/v2/nicolaka/netshoot/manifests/latest" http.request.useragent="containerd/v1.4.0-beta.1-34-g49b0743c" vars.name="nicolaka/netshoot" vars.reference=latest
time="2020-07-26T14:52:45.4195817Z" level=info msg="Adding new scheduler entry for nicolaka/netshoot@sha256:04786602e5a9463f40da65aea06fe5a825425c7df53b307daa21f828cfe40bf8 with ttl=167h59m59.9999793s" go.version=go1.11.2 instance.id=ba959eb9-2fa3-47c0-beb7-91480c8a31ee service=registry version=v2.7.1
172.18.0.2 - - [26/Jul/2020:14:52:43 +0000] "HEAD /v2/nicolaka/netshoot/manifests/latest HTTP/1.1" 200 1999 "" "containerd/v1.4.0-beta.1-34-g49b0743c"
time="2020-07-26T14:52:45.4204299Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="proxy:5000" http.request.id=15e9ac86-7d79-4883-a8ce-861a7484887c http.request.method=HEAD http.request.remoteaddr="172.18.0.2:57588" http.request.uri="/v2/nicolaka/netshoot/manifests/latest" http.request.useragent="containerd/v1.4.0-beta.1-34-g49b0743c" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=1.6697112s http.response.status=200 http.response.written=1999

Docker proxy vs. local registry

A bit later, I discovered that you can't push to a proxy registry. Tilt is a tool I use to ease the process of developping in a containerized environment (and it works best with Kubernetes); it relies on a local registry in order to cache build containers even when restarting the Kind cluster.

Either the registry is used as a "local registry" (where you can push images), or it is used as a pull-through proxy. So instead of configuring one single "proxy" registry, I configure two registries: one for local images, one for caching.

docker run -d --name proxy --restart=always --net=kind -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2
docker run -d --name registry --restart=always -p 5000:5000 --net=kind registry:2
kind create cluster --config /dev/stdin <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["http://proxy:5000"]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
      endpoint = ["http://registry:5000"]
EOF

Note that we do use a port-forwarding proxy (-p 5000:5000) so that we can push images "from the host", e.g.:

% docker tag alpine localhost:5000/alpine
% docker push localhost:5000/alpine
The push refers to repository [localhost:5000/alpine]
50644c29ef5a: Pushed
latest: digest: sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 size: 528

# Let's see if this image is also available from the cluster:
% docker exec -it kind-control-plane crictl pull localhost:5000/alpine
Image is up to date for sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e

If you use Tilt, you might also want to tell Tilt that it can use the local registry. I find it a bit weird to have to set an annotation (hidden Tilt API?) but whatever. If you set this:

kind get nodes | xargs -L1 -I% kubectl annotate node % tilt.dev/registry=localhost:5000 --overwrite

then Tilt will use docker push localhost:5000/you-image (from your host, not from the cluster container) in order to speed up things. Note that there is a proposal (KEP 1755) that aims at standardizing the discovery of local registries using a configmap. Tilt already supports it, so you may use it!

Improving the ClusterAPI docker provider to use a given network

When I play with ClusterAPI, I usually use the CAPD provider (ClusterAPI Provider Docker). This provider is kept in-tree inside the cluster-api projet.

I want to use the caching mechanism presented above. But to do that, I need to make sure the clusters created by CAPD are not created on the default network (current implementation creates CAPD clusters on the default "bridge" network).

I want to be able to customize the network on which the CAPD provider creates the container that make up the cluster. Imagine that we could pass the network name as part of a DockerMachineTemplate (the content of the spec is defined in code here):

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: DockerMachineTemplate
metadata:
  name: capd-control-plane
  namespace: default
spec:
  template:
    spec:
      extraMounts:
        - containerPath: /var/run/docker.sock
          hostPath: /var/run/docker.sock

      network: kind # 🔰 This field does not exist yet.

Update 26 July 2020: added a section about local registry vs. caching proxy. Reworked the whole post (less noise, more useful information).

Using mitmproxy to understand what kubectl does under the hood

Maël Valais — Wed, 01 Jul 2020 17:17:00 +0000

In Oct 2019, Ahmet Alp Balkan wrote this blog post that explains how to use mitmproxy to observe the requests made by kubectl. But I couldn't use the tutorial for two reasons:

I use kind to create local clusters which means I hit the Go net/http limitation (skips proxying for hosts localhost and 127.0.0.1, see this blog post)
I use client certs authentication, which can't work with the method presented by Ahmet; it can only work for header-based authentication (e.g. token) but not for client certs.

In the following, I detail how I managed to make all that work.

# Let's use a separate kubeconfig file to avoid messing with ~/.kube/config.
export KUBECONFIG=~/.kube/kind-kind

# Now, let's create a cluster.
kind create cluster

# Let's see what the content of $KUECONFIG is:
cat $KUBECONFIG

We can see that the kubeconfig uses a client cert as a way to authenticate to the apiserver.

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EWXlPVEE1TURJeU4xb1hEVE13TURZeU56QTVNREl5TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS25qCmhmRzBvenZVb05jMXY1STkvYm13dFBqb2QvK0RyczF4TFZOcWgxQjhFcTY1S3lnQjBSbDdQcTJueUhyRVRnTmsKZ2VPQ2RzRURObGpKOE12SC9GbE9nRkUvS0dqK2hwN2hwc0dHRExReWFUOExUY25JN0FNL2t3KzY5R0hidlN3Nwo2V0Y1VERTMDU1RkRqdnRveGdHcERycmZQZTk5bXN5Zmk3aWtteDk5MmRyMHFQd0xxanJpZHNkWU52MUZqU1Y1CndkRlFISGxBS2hBcmlUWmpQMnhNL3poOENBOFhndjF0UUxVVk5IS1hrSG5UYWlkeFY1MkduaUVaQmd0d2tSK3oKQ3hQempVZFAxQ1JRYzU0YmxDYW9FQVdTc0NYTUVPREhTSnowSi9CWHJCU2JaeGZIakd0Y0k0bEhBUmx2aURNawp3U3lOLy9qdE9tbWhDT29BTzBzQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFIMDBrYnZpaWNNT3IxdFJoTkQweVpGa3ZkT1AKUzFGOEZKK1BFd1o1WExUTVVyVG1yekVlZmgza21xWkxYUnlyK3c0Snk1a0grK1o3enBpdlp6Q3BGOEtwclJaWAp5N2Z6TkJOeWMrOHFKN3dCek0xZ21BdTRha3BlNVBYbkw3akZIcU9aUmNXTmZKcUpTci9BS05sK2c2SjAzV2pnCmJDc0NQNTNrZ1czSDZYMkRoS1JzZFRWQlc1UGVlek1YRVlON2VYbGpnWE4xcElMdEU1VE5oWVZIcTZaUDVCaDkKMmU2Y1U4bEFuNlhYQ0Ira2RsTkhudGp1cTlSL2lPQVJRaWpVSVlWSFdCT1NyWGp6TnhkTklEakRRd3lKUnViMApDVEhrcDg4eUlNaEV1Nk1OV1VCUU82ZnVqbURHbUJJbTlzdzJpSnI1UlArSUVUVnQyWFc0Q2dHT1cwVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
      server: https://127.0.0.1:53744
    name: kind-helix
contexts:
  - context:
      cluster: kind-helix
      user: kind-helix
    name: kind-helix
current-context: kind-helix
kind: Config
preferences: {}
users:
  - name: kind-helix
    user:
      client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJYTZXc25HT0RGZW93RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TURBMk1qa3dPVEF5TWpkYUZ3MHlNVEEyTWprd09UQXlNekJhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXRkbkxsb0V1cnFoSmRrMGgKME5VcXpFSHhUbmVHS040QTNtWDBobmFLcE1TT3hISlBQcTB6V0t4WEVxNTZPTkdhSkhvS0VTRUM2Yjh2MGcwTQo5dXVGOXJyQ1VORkpXMmdHcFArRG96TEpVS3RtN2F0WFZRYXNVQ2tjbFRtQ003aHlqOGZTTlRDd2lxT3RNOUFvCnFVVTJYd1k0a2xhL0RuMkRBc1V1VmNiUTdITDA1N0tFbjZvVzlFVy9mQ2hMVE5OTWhmelpkNzFISGs3MFNOZDMKTG5jTXNjNmp2eS9kN2MwbjM4ek45SjVzWVZOTjNsS1YvOU5MT2FaTEQ0bSsyVU5XV1FVazRuci8rZjlaZk1IMwphVlFibDZSWEZwaW1jYWg4UjRJTmhRNkhYbmVEbUI2dUl3RGdjQnhhQUtoNFVPVlZwODB1Uy9pYUVLV1VSWE1PCkg4YXBZUUlEQVFBQm95Y3dKVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFJaWtmbis5TkJQc2FNSzNVMU5pWmtBaWxsM1pWVXJDdzNsWgpIeGRGbm9MZGxZYmtPeFVEN25EK3lrNXhZWW4yaDc3WUU4NlU5czJ3aitkdFJTR2Ezc3p4WVJQbkt5MDN3L1M4CkhqZzhwSkZOZFhHWlNPUWJDQTBmT1BONXl5MlpYRlVWd0JwZ3VTMC9nTkNUUkRPUDl3QmNjQXhiSGRmSjMxQzQKa2hCemF4QXI5UEliMVBzUlhqS2ZSRnkvcllwYWRBYVhhYmZMOFRvbTJ4cGpLVDYreGxoL3lJb2tZbVhxSnlXRgp1SGFmWG1qUEdyRDFoZUo2UnR5U0xRM0dKUXQxWmVPK3BaYlJlRjcxU0ZRSjRQdFhtTHhGMjZQL3dlV3F0NWFJClhTK0ZnanRYbzZqUG1mWWRweDZ3bWp3UHRaZ1pRdXhSL2tsejVvQnErNnFaYUZDWXg4RT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
      client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdGRuTGxvRXVycWhKZGswaDBOVXF6RUh4VG5lR0tONEEzbVgwaG5hS3BNU094SEpQClBxMHpXS3hYRXE1Nk9OR2FKSG9LRVNFQzZiOHYwZzBNOXV1RjlyckNVTkZKVzJnR3BQK0RvekxKVUt0bTdhdFgKVlFhc1VDa2NsVG1DTTdoeWo4ZlNOVEN3aXFPdE05QW9xVVUyWHdZNGtsYS9EbjJEQXNVdVZjYlE3SEwwNTdLRQpuNm9XOUVXL2ZDaExUTk5NaGZ6WmQ3MUhIazcwU05kM0xuY01zYzZqdnkvZDdjMG4zOHpOOUo1c1lWTk4zbEtWCi85TkxPYVpMRDRtKzJVTldXUVVrNG5yLytmOVpmTUgzYVZRYmw2UlhGcGltY2FoOFI0SU5oUTZIWG5lRG1CNnUKSXdEZ2NCeGFBS2g0VU9WVnA4MHVTL2lhRUtXVVJYTU9IOGFwWVFJREFRQUJBb0lCQVFDVkNJOXZJeVB0QkFKZwpyOG44NmhhUEc2UDFtTU1jandUTFAyZHRJNDF3aDU0eHBUVUl1czJQNkgzYjA1NWJIbnhqVkprWGZLUjBpTGxhClBsUFhzU0l6R00vVGlCSEVsYmFNVnRPOVZndml6dllsNWZ4R3RKZFhncm5vR2g5NDM3c1Qxc0dSMGZ0OVE3TFkKK2NtNUgvMzFWcFhhYUxsZjJNRWI3aG1STnNWV1lXWm9MUHJ2QUJPemVmUnpKU0RONHU0M3M0eVpnNHJoL3IzWQo3YUhYOS9CSHpJRk93WTRNL1BRQzdYaFhDMXBIeWNPY2lSWVhFTkpuMTdQN2NOSkdoMnJ4dnc2OVQ1QW9rdXY5Ckcxd2lUNmVrVmZuaVYrSlVnL1lwcTNSRGxNdVFETDRZdCtGNG1zVGJtN3NFZk5yVVMzWGZFMFdmcjB2Wk1YN2sKMnE3dXkxNEJBb0dCQU5jb1BxVHNBa3N5YzR6UWtmQmNzb0Y1ekdJM3NQTUFGTHRSMGF6S2x1Q1V3VGMzYVk5dgpiTE05OTBVK3VpVzdtRHF0VHpZMVNVSzZGNW0yMzJoSW5hTnBCTUVJcDZHQ0I3dis5WWZIQURVSkljaVdodmJhCmpIY0M5Qjl4SG5uUTBydU5aNUErdWVtRE9EZkRKUllIVUhrMEh3MlJpTE1qRFhVRmhBanJBUGJ4QW9HQkFOaGUKL3BHb2FWOWtUREtNOWZwTTZUQkZwMFVtb2lSb3crVG9TWjhWeVBmMkl1VlNpa0dtaXUydHU3MzZCZkJtVzNtQgpGWmFRc01rNkMydExkK1NBQ0lSbktQTUN4czYzR05WNkJ0aXRIRFlBaGNtRCtvU3VpaFRrd3l1WXlpQml4aFovCms1UDY3UHFKVkQ2YnhkNDV6YlFsS2VNaUVtMUhnQUVGSEF0UFBqbHhBb0dBYzNnaHhwankwakNOV3ZGRW9WN2UKWGlaanpnSmRjTXlHVTlHaFdiNlFJbzh5OHROR1Q3aFkrZ2t6ZjNJZXJNbDA5V2kxcmo0Q3gxRGdBWnJuWXl3MQpqZEY2djY1SmFLQkVUbHlTb1AvbjJJN0NGc2pTUGdFa2lXcUlZYWR2MTZoK3NERS9kMlp5bUNQWU0vVURIa05tCnFPV1VGTkFhTVNtS3UxYnVlV3JGNWNFQ2dZQnJvNTVySWVnQjM2aVVnVkdoU24rN1Z2dG15RmhqV29jUnFvbHQKamUzamhWeEl6eTRlaU5hV2RSWnY1U0R0UGs2RmZMVWJxVEY1ZWRuU2I4SGVOOStFMXJrbFk1MDVteGJNcEo4aApUY1U2RERxQ1RKamxSdHRFbDZXTVc3ODZLMGsyU2hORnk4LzJ0empreUtPLzhPdW5rZEZyd0RpQWl0QmdNWVdKCkRzdjYwUUtCZ1FDR2htYnZrODRaYm1sdERxYW9hOVR6YU5UT3FrN3dtb0tDby9FVEg5NEtNVVZMVmk3bHhpN1kKcXFQYUdiSm9aYkZGd0NvWW12Rk5jU21nYk9SNjdCVVBkdUEzMHo0VythV1pmQVkwZGpUR08yTC8zVVhoWHdHegpVUGpZTXZZQU5sdzFmZTVmZmk3UExJSGJvZXFhTmhhaDliRVJXdFNRbm95NnYzV2hlNTh6VkE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

Let's run mitmproxy using the above client cert and keys:

mitmproxy -p 9000 --ssl-insecure --set client_certs=<(kubectl config view --minify --flatten -o=go-template='{{(index (index .users 0).user "client-key-data")}}' | base64 -d && kubectl config view --minify --flatten -o=go-template='{{(index (index .users 0).user "client-certificate-data")}}' | base64 -d)

Now, let's make sure that we have some DNS alias to 127.0.0.1 to work around the Go proxy limitation which skips proxying for 127.0.0.1 and localhost:

grep "^127.0.0.1.*local$" /etc/hosts || sudo bash -c 'echo "127.0.0.1 local" >> /etc/hosts'

Finally you can run kubectl to see the logs of kube-scheduler:

% HTTPS_PROXY=:9000 kubectl --kubeconfig=<(sed "s|127.0.0.1|local|" $KUBECONFIG) --insecure-skip-tls-verify logs -n kube-system -l component=kube-scheduler,tier=control-plane
I0701 11:11:11.585249       1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0701 11:11:11.585329       1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0701 11:11:12.976581       1 serving.go:313] Generated self-signed cert in-memory
W0701 11:11:17.407371       1 authentication.go:349] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0701 11:11:17.407465       1 authentication.go:297] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0701 11:11:17.407483       1 authentication.go:298] Continuing without authentication configuration. This may treat all requests as anonymous.
W0701 11:11:17.407513       1 authentication.go:299] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0701 11:11:17.478349       1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0701 11:11:17.478483       1 registry.go:150] Registering EvenPodsSpread predicate and priority function
W0701 11:11:17.491616       1 authorization.go:47] Authorization is disabled
W0701 11:11:17.491729       1 authentication.go:40] Authentication is disabled
I0701 11:11:17.491883       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0701 11:11:17.500576       1 secure_serving.go:178] Serving securely on 127.0.0.1:10259
I0701 11:11:17.500678       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0701 11:11:17.500919       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0701 11:11:17.500931       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0701 11:11:17.718085       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0701 11:11:17.801300       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/kube-scheduler...
I0701 11:11:34.659176       1 leaderelection.go:252] successfully acquired lease kube-system/kube-scheduler

It works!! Here is what mitmproxy is showing:

18:17:42 GET  HTTPS   local /api/v1/namespaces/kube-system/pods/kube-scheduler-helix-control-plane      200 …plication/json  5.2k 119ms
18:17:42 GET  HTTPS   local /api/v1/namespaces/kube-system/pods/kube-scheduler-helix-control-plane/log  200      text/plain 2.46k 227ms

Epic journey with statically and dynamically-linked libraries (.a, .so)

Maël Valais — Sat, 30 May 2020 18:45:00 +0000

Between May and June 2016, I worked with ocamlyices2, an OCaml package that binds to the Yices C++ library. Both projects (as well as many Linux projects) are built using the "Autotools" suite.

The Autotools suite includes tools like autoconf, automake and libtool. These tools generate a bunch of shell scripts and Makefiles using shell scripts and the M4 macro language. The user of your projects ends up two simple commands to build your project:

./configure
make

Why did I bother with this? As part of my PhD, I worked on a tool, touist, which uses SMT solvers like Yices. The touist CLI was written in OCaml (a popular language among academics, including my supervisor), which meant I had to go through hoops to interoperate with C/C++ solvers like Yices.

My first challenge with ocamlyices2 was the fact that I needed a statically-linked libyices.a. And since Yices depends on GMP, I had to dive deep into Yices2's configure.ac and find a way to select the static version of GMP.

But building a static library libyices.a was not enough. I was to build in PIC mode (position-independant code, enabled with -fPIC in gcc). The position-independant code is required when you want to embed a static library into a dynamically-linked library. That's due to the fact that OCaml requires both the .so and .a versions of the "stub" library (a stub library is a C library that wraps another C library using OCaml C primitives). And naturally, Yices' build system had not been written to support building a static PIC libyices.a.

I remember these days as an epic struggle against old build systems. This experience taught me everything about autoconf, Makefiles, position-independant code, gcc, ldd and libtool. And in this post, I want to share these discoveries and how I progressed into contributing to the ocamlyices2 project.

Here is a diagram showing the dependencies between libraries. Ocamlyices2 depends on Yices and Yices depends on GMP.

                            build:   dune
             touist         lang:    ocaml
                |           output:  touist binary (statically linked)
                |
                |
                |depends on
                |
                |
                |
                v
           ocamlyices2      build:   autoconf + make
                |           lang:    ocaml + C
                |           output:  libyices2_stubs.a
                |
                |depends on
                |
                |
                |
                v           build:   autoconf + hacky make
              yices         lang:    C++
                |           output:  libyices.a (with PIC enabled)
                |
                |
                |depends on
                |
                |
                |
                v           build:   autoconf + automake + libtool
               gmp          lang:    C, assembly
                            output:  libgmp.a (with PIC enabled)

The most important thing about this diagram is that since we need a PIC-enabled static library libyices.a in order to build the "binding libraries": the shared library yices2.cmxs and the static library libyices2_stubs.a.

And in order to get this PIC-enabled libyices.a, I needed to make sure that the GMP library picked by the Yices ./configure would be static and PIC-enabled.

In early 2016, the Yices build system was limited to building a shared library with no support for cross-compilation (a requirement to build on Windows) and no support for enforcing PIC in libgmp.a and libyices.a.

Yices2 & autoconf: an attempt at fixing a limited build system

I remember the warmth that we had in May 2016. My daughter had turned three and we were still living in a tiny appartment since I was technically still a student.

My first patch to the ocamlyices2 project took over a month of intense work. The stake was immense: I aimed at revamping the whole Yices2 build system. The original build system didn't allow developers to statically build libyices.a. More generally, it was a pain to work with: most configuration was happening at ./configure time, but a ton of things had still to be passed at make time (e.g., make VAR=value).

This change (25f5eb15) brought a ton of features. But the most important ones were:

Better ./configure && make experience. Instead of having some parts of the build configuration being passed as Makefile variables, I moved everything to nice flags that you would pass to ./configure.
Proper PIC support. Sometimes, Yices' ./configure would pick up a libgmp.a that would not be "PIC". So I wrote a new M4 macro, CSL_CHECK_STATIC_GMP_HAS_PIC, that would do the PIC check using libtool. For example, the developer would be able to ask for a static library with PIC support:

   ./configure --with-pic --enable-static --disable-shared --with-pic-gmp=$PWD/with-pic/libgmp.a

Static libyices.a. The original build system did not
Using libtool. Instead of hand-crafted targets for building the static & dynamic libraries, libtool allowed me to build both with very little effort (at the cost of slightly longer build times). I remember trying to fiddle with the Makefile to be able to get PIC/non-PIC as well as dynamic/static .o units. Using libtool made it so easier to build simultanously the static and dynamic versions of libyices (libyices.a and libyices.so).
Cross-compilation. I wanted to be able to release binaries for Windows users, which meant that I needed to cross-compile from a Cygwin environment to a Win32 executable.

I also fixed weird bugs like a failure on Alpine 3.5 due to a race condition in ltmain.sh. Yes, you heard well! A race condition in a build system!

Not sure, but 25f5eb15 might be my biggest commit ever:

Author: Maël Valais <mael.valais@gmail.com>
Date:   Fri May 5 10:31:37 2017 +0200
Commit: 25f5eb15

    libyices: build system: moved all config from Makefile to ./configure

    What I tried to fix:
    * Impossible to produces static archive libyices.a for the mingw32 host.
    * So much configuration done in Makefile instead of ./configure.
    * The 'make build OPTION= MODE=' is kind of unexpected and not really standard.
    * As an end-user builder, it is hard to guess that my system-installed libgmp.a
      is not PIC. The build system should help me with that.
    * CFLAGS and CPPFLAGS cannot be overritten by the end-user builder at 'make'
      time. See 'Preset Output Variables' in the autoconf manual.
    * `smt2_commands.c`, `smt2_parser.c` and `smt2_term_stack.c` are all including
      `smt2_commands.h`, and in `smt2_commands.h` there is a definition of a type,
      so there is a clash of symbols at link time. Solution 1: use 'extern' in .h
      and definition in .c. Solution 2: typedef the enum.
    * The build system should be fully parallel-proof. Note that 'make -j' is a
      thread bomb; prefer using 'make -j4' if you have 4 processors.
    * Why removing libyices.a when using 'dist'? (rm -f $(distdir)/lib/libyices.a)
      libyices.a should be distributed to the end-users along with the shared lib.
    * Why does libgmp.a contain more than libgmp.so? I guess it is because some
      function are needed by the tests and the tests are never dynamically linked
      to the shared libyices.so... For example, the test `test_type_matching` calls
      a function 'delete_pstore' that is not exported. This means we cannot use the
      created libyices.so! We must instead use the object files directly.

    Features of the new build system:
    * It is now possible to select what you want to build using --enable-static
      and --enable-shared. By default, both are built. You can speed up the build
      by disabling one of the modes, for example --disable-shared.
    * Running ./configure will tell you if the libgmp.a found/given with
      --with-static-gmp or --with-pic-gmp is PIC or not.
    * Libtool now handles all shared library naming with version number and symbolic
      links. on linux, .so, on mac os .dylib, on windows .a, .dll.a and .dll...
    * It is now possible to choose if you want PIC-only code in the static
      library libyices.a using --with-pic.
    * The ./configure now configures and the Makefile makes; moved all the
      configuration steps that were done in the Makefile.
    * It is not required anymore to pass OPTION when using make. OPTION is now
      handled by passing for example --host=i686-w64-mingw32 when running
      ./configure, instead of 'make OPTION=mingw32',
    * It is not required anymore to pass MODE in make. MODE is now handled by
      the argument --with-mode=release (for example) when running ./configure.
    * Removed the confusing mess of build/<host>/... and configs/make.<host>.
      Build objects are simply put in build/.
    * Merged the many Makefiles that were sharing a lot of code in common.
    * Standardized the 'make' target and experience:
      - make build for building binaries and library (no need for OPTION or MODE)
      - make dist (non-standard) for showing the results of the build as it would
        be distributed
      - make distclean for removing any files created by ./configure
      - make lib if you only want the library
      - make install and uninstall for installing/uninstalling (DESTDIR supported)
    * On Windows, we can now build a static library libyices.a.
    * On Windows, shared and static libraries can be built at once. The static
      version of libyices.a can be renamed using --with-static-name.
    * DESTDIR works as expected: it will reproduce the hierarchy using the prefix
      when running 'make install DESTDIR=/path'
    * A nice summary of the configuration is now printed when running ./configure.
      It allows to check if libgmp.a is PIC or not, and helps to have a clear
      view of what is going to happen.
    * Moved version number of Yices in configure.ac
    * If the user wants to use --with-pic-gmp but his libgmp.a is not PIC, give him
      an indication of what command to run to build the with-PIC libgmp.a.
    * Moved the gmaketest into configure; warn the user if 'make' is not gnu make.
    * Parallel build is now fully supported (make -j)
    * when using --with-static-gmp (and other similar flags), try to find gmp.h
      in . and ../include automatically, and fall back with the system-wide gmp.h
    * check for gmp.h even when no --with-static-gmp-include-dir is given
    * moved all csl_* functions into autoconf/m4/csl_check_libs.m4 so that they
      can be reused somewhere else
    * compute dependencies only if not in release mode and at compile time
      instead of ahead of time. This saves time during compilation, because deps
      are not necessary if the .c or .h files are not changed (i.e., if the builder
      is the end-user). If the builder is a developer, then he will set
      --with-mode={debug,profile...} and this will trigger the deps to be computed.
    * 'make test' compiles and runs all tests in tests/unit
    * 'make test_api12' will compile and run the test tests/unit/test_api12.c
    * removed version_*.c file as it is rebuilt at make time
    * gperf is now only necessary when changing the tokens.txt or keywords.txt files,
      the end-user builder does not have to have gperf installed.

    Side notes:
    * CPPFLAGS, CFLAGS, LDFLAGS and any other makefile variable can be overwritten
      using 'make LDFLAGS=...' (was already the case with the previous version of the
      build system)

    Known issues:
    * on Mac OS X, linking executables agains non-PIC libgmp.a will throw the
      following warning:
      ld: warning: PIE disabled. Absolute addressing (perhaps -mdynamic-no-pic)
      not allowed in code signed PIE, but used in ___gmpn_mul_1 from libgmp.a(mul_1.o).
      To fix this warning, don't compile with -mdynamic-no-pic or link with -Wl,-no_pie

    Todo:
    * produce libyices.def on Windows
    * make sure the test on libgmp-10.dll is future-proof (remove the '-10')
    * fix the 'echo summary'
    * I did not test the checks made configure.ac on mcsat and libpoly; we should
      do the same tests as done on libgmp.a (for checking that it is PIC) and
      add a helping message at the end of configure.ac.
    * For the tests, I read that two kind of tests were compiled:
      - 'tests' where the tests are linked to the non-PIC libgmp.a and static libgmp.a
      - 'tests-static' where the tests are linked to the PIC libgmp.a and shared
        GMP library.
      I changed the second one: it links to the shared version of libyices and shared
      version of GMP. But building the with-PIC libyices.a would be really easy
      (and it is still possible using the flag --with-pic).
    * 'make dist' is not a staging area (for now) for 'make install'. They both
      install from built objects.
    * it is not possible to compile the tests against the shared library
      because many symbols which are used in the tests are not exported ('export'
      in the C code). They exist if we do 'nm' but they are just not usable.

    * pstore issue:

    The visibility is T (visible) in static mode and t (hidden) in shared mode. This
    is because 'abstract_values.c' has not been 'exported' (=added in `yices_api.c`).
    Here is an example of the difference (functions from )
    ```


    # nm build/lib/libyices.a | grep pstore
    0000000000000090 T _delete_pstore
    0000000000000000 T _init_pstore
    # nm build/lib/.libs/libyices.2.dylib | grep pstore
    0000000000037e10 t _delete_pstore
    0000000000037d80 t _init_pstore


    ```

    I tweaked the ltmain.sh to be able to pass different CPPFLAGS for the compilation
    of static and shared objects by the `%.o: %.c` rule. CPPFLAGS must be different
    because of Windows dlls: `-DNOYICES_DLL -D__GMP_LIBGMP_DLL=0` for example.

    I added two variables LT_STATIC_CFLAGS and LT_SHARED_CFLAGS. They allow me
    to pass CFLAGS to libtool for .c -> .o targets. It allows me to pass things
    like -DYICES_STATIC, -DNOYICES_DLL and -D_LIBGMP_DLL.

    ```

 diff
    diff --git a/ext/yices/autoconf/ltmain.sh b/ext/yices/autoconf/ltmain.sh
    index bf5d83b..7b7dd3a 100644
    --- a/ext/yices/autoconf/ltmain.sh
    +++ b/ext/yices/autoconf/ltmain.sh
    @@ -3500,10 +3500,10 @@ compiler."
           fbsd_hideous_sh_bug=$base_compile

           if test no != "$pic_mode"; then
    -       command="$base_compile $qsrcfile $pic_flag"
    +       command="$base_compile $LT_SHARED_CFLAGS $qsrcfile $pic_flag"
           else
            # Don't build PIC code
    -       command="$base_compile $qsrcfile"
    +       command="$base_compile $LT_SHARED_CFLAGS $qsrcfile"
           fi

           func_mkdir_p "$xdir$objdir"
    @@ -3552,9 +3552,9 @@ compiler."
         if test yes = "$build_old_libs"; then
           if test yes != "$pic_mode"; then
            # Don't build PIC code
    -       command="$base_compile $qsrcfile$pie_flag"
    +       command="$base_compile $LT_STATIC_CFLAGS $qsrcfile$pie_flag"
           else
    -       command="$base_compile $qsrcfile $pic_flag"
    +       command="$base_compile $LT_STATIC_CFLAGS $qsrcfile $pic_flag"
           fi
           if test yes = "$compiler_c_o"; then
            func_append command " -o $obj"


    ```

What a massive commit message, right?!

Oviously, I still needed to use all the new features that I had just addeed to the Yices ./configure. So I proposed a second patch (ccb5a563) with changes to the ocamlyices2's own ./configure; that took the form of new flags so that I would be able to build a static libyices.a by specifying the static version of the GMP library libgmp.a.

I also used the new cross-compilation capability of the Yices ./configure so that it would be possible to build ocamlyices2 on Windows. The compilation would rely on the POSIX-compliant Cygwin suite and cross-compile to a native Win32 executable using MinGW32 (which a port of GCC).

`eml
Author: Maël Valais mael.valais@gmail.com
Date: Fri May 5 16:18:46 2017 +0200
Commit: ccb5a563

ocamlyices2: only build the static stub using static libgmp.a.

The library libgmp.a will be searched in system dirs or you can use the flag
--with-static-gmp= when running ./configure for setting your own libgmp.a. If
no libgmp.a is found, the shared library is used. You can force the use of
shared gmp library with --with-shared-gmp.

If --with-shared-gmp is not given, the libgmp.a that has been found
will be included in the list of installed files. The reason is because
if we want to build a shared-gmp-free binary, zarith will sometimes pick the
shared library (with -lgmp) over the static lbirary libgmp.a.

Including libgmp.a in the distribution of ocamlyices2 is a convenience for
creating gmp-shared-free binaries.

Why do we prefer using a static version of libgmp.a?
===================================================

This is because we build a non-PIC static version of libyices.a. If
we wanted to build both static and shared stubs, we should either
- build a PIC libyices.a but it would conflict with the non-PIC one
- build a shared library libyices.so.

For now, I chose to just skip the shared stubs (dllyices2_stubs.so).

Also:

* turn on -fPIC (in configure.ac) only if non-static gmp
* added a way to link statically to libgmp.a (--with-static-gmp)
* use -package instead of -I/lib for compiling *.c in ocamlc

This option uses the change I made to the build system of libyices.
Why? Because I want the possibility of producing binaries that do
not need any dll alongside.

Guess the host system and pass it to libyices ./configure
=========================================================
It is now possible to use ./configure for mingw32 cross-compilation.

A month past and I soon realized that GMP was not embedded at all in libyices.a. I could see the symbols as undefined (U) when running nm libyices.a. It took me a while to figure this out... back to tweaking the fragile Yices ./configure!

Along the way, I also realized how different Linux distributions are. Arch Linux is notably lacking support for partial linking (ld -r). Which meant I had to add a flag for this specific purpose in commit 38200b0a:

`eml
Author: Maël Valais mael.valais@gmail.com
Date: Fri Jun 16 21:18:37 2017 +0200
Commit: 38200b0a

libyices: added option --without-gmp-embedded

This option will disable the partial linking that allows to embed
GMP into libyices.a (only with --enable-static).

For example, on Arch linux, the partial linking command

    ld -r -lgmp *.o -o libyices.o

would fail even if libgmp.so is correclty installed. It seems that 'ld -r'
would only work with a static libgmp.a (but the arch linux repo only installs
the shared gmp library).

Why this `ld -r`? This command is the only way I found to compile a
static libyices.a from either a shared or a static libgmp and produce
a gmp-depend-free libyices.a.

Two solutions:

1. Drop the necessity for building a libyices.a free of gmp dependency.
   In this case, I could remove the `ld -r`.
   It would then create a libyices.a that depends on libgmp.a/so.
2. Separate the gmp-dependency-free libyices.a from the normal
   gmp-dependent libyices.a. For example, I could use the option
   `--without-gmp-embedded`

So I went with solution (2).

This option disables the embedding of GMP inside libyices.a. This
'embedding' is made using partial linking (ld -r) which seems to
be failing on Arch Linux when using the shared GMP library.

Although I had already added a check (CSL_CHECK_STATIC_GMP_HAS_PIC in 25f5eb15) to make sure that the user-provided libgmp was PIC when using --with-pic, I realized that it was trickier that what I thought in 55c8e92a...

`eml
Author: Maël Valais mael.valais@gmail.com
Date: Sat Jun 17 14:09:12 2017 +0200
Commit: 55c8e92a

libyices: with --enable-static and --with-pic, enforce PIC libgmp.a

One problem I came across was that most of the time, when I was doing

    ./configure --enable-static --with-pic

the produced libyices.a would still contain non-PIC libgmp.a, although
being itself PIC. In this commit, I enforce that if --with-pic is given,
then:

1) either --with-pic-gmp has been given, in this case we use that for
   creating the PIC libyices.a;
2) or --with-pic-gmp has not been given and thus we try to simply use the
   shared gmp through -lgmp.

Reminder: we also check that the system libgmp.a or the libgmp.a given with
--with-static-gmp is PIC. If it is the case, the PIC libgmp.a will be used
for --with-pic.

Now, I also had to fix ocamlyices2's ./configure since ld -r (partial linking) was not supported on Arch Linux (see [PR's CI failure](pull request](https://github.com/ocaml/opam-repository/pull/9086). I remember waiting for hours for the CI to run on all imaginable systems: Debian, Ubuntu, Suse, Arch Linux, Alpine, CentOS, Fedora, macOS and Windows...

Commits df7c89a1 and 70dc5de5) fix the partial linking issue on Arch Linux:

`eml
Author: Maël Valais mael.valais@gmail.com
Date: Sat Jun 17 16:52:59 2017 +0200
Commit: df7c89a1

ocamlyices2: added --with-libyices and --with-libyices-include-dir

These options allow to give your own libyices.a and the include directory
where the libyices headers are.

Author: Maël Valais mael.valais@gmail.com
Date: Sat Jun 17 17:01:58 2017 +0200
Commit: 70dc5de5

ocamlyices2: use --without-gmp-embedded by default

After giving it some thoughts, the need for having a self-contained libyices.a
(which would only need -lyices, no -lgmp needed) in ocamlyices2 is pointless
as 'zarith' will still need '-lgmp' anyway.

The Makefile will still put libgmp.a and libyices.a inside src/ so that
the static version of gmp is used (with -L.) instead of the shared version.

Rationale: disabling the partial linking fixes the build on Arch Linux, which
(I re-tested on a docker image) cannot accept partial linking with -lgmp when
only libgmp.so is available. Here is the failing command:

    ld -r *.o -lgmp -o libyices.o

Final result of the Yices2 build system

The experience with the re-written ./configure (here) is very different from the original one. When the user wants to compile the library with PIC, they get a warning if one of the dependencies is not PIC. There is a much finer control over what the user wants: dynamic vs. static, PIC vs. non-PIC. But also more control over dependencies like GMP, since the user must be able to pass a static or dynamic version of libgmp.

The ./configure that you can see here gained many features like --with-shared-gmp or --with-pic.

sh % ./configure --helpconfigure' configures Yices 2.5.2 to adapt to many kinds of systems.

Usage: ./configure [OPTION]... [VAR=VALUE]...

System types:
--build=BUILD configure for building on BUILD [guessed]
--host=HOST cross-compile to build programs to run on HOST [BUILD]

Optional Features:
--enable-shared[=PKGS] build shared libraries [default=yes]
--enable-static[=PKGS] build static libraries [default=yes]
--disable-libtool-lock avoid locking (might break parallel builds)
--enable-mcsat Enable support for MCSAT. This requires the libpoly
library.

Optional Packages:
--with-pic[=PKGS] try to use only PIC/non-PIC objects [default=use
both]
--with-static-gmp=
Full path to a static GMP library (e.g., libgmp.a)
--with-static-gmp-include-dir=
Directory of include file "gmp.h" compatible with
static GMP library
--with-pic-gmp= Full path to a relocatable GMP library (e.g.,
libgmp.a)
--with-pic-gmp-include-dir=
Directory of include file "gmp.h" compatible with
relocatable GMP library
--with-static-libpoly=
Full path to libpoly.a
--with-static-libpoly-include-dir=
Path to include files compatible with libpoly.a
(e.g., /usr/local/include)
--with-pic-libpoly=
Full path to a relocatable libpoly.a
--with-pic-libpoly-include-dir=
Path to include files compatible with the
relocatable libpoly.a
--with-shared-gmp By default, a static version of the GMP library will
be searched. This option forces the use of the
shared version. This applies for both shared and
static libraries.
--without-gmp-embedded (Only when --enable-static) By default, the static
library libyices.a created will be partially linked
(ld -r) so that the GMP library is not needed
afterwards (i.e., only -lyices is needed). If you
want to disable the partial linking (and thus -lgmp
and -lyices will be needed), you can use this flag.
--with-mode=MODE The mode used during compilation/distribution. It
can be one of release, debug, devel, profile, gcov,
valgrind, purify, quantify or gperftools. (default:
release)
--with-static-name=name (Windows only) when building simultanously shared
and static libraries, allows you to give a different
name for the static version of libyices.a.
`

I also added a ton of diagnostic information that appears at the end of ./configure. That's very useful when you want to make sure that ./configure has picked up the right version of libgmp:

`plain
configure: Summary of the configuration:
EXEEXT:
SED: /usr/bin/sed
LN_S: ln -s
MKDIR_P: /usr/local/opt/coreutils/libexec/gnubin/mkdir -p
CC: gcc
LD: /Library/Developer/CommandLineTools/usr/bin/ld
AR: ar
RANLIB: ranlib
STRIP: strip
GPERF: gperf
NO_STACK_PROTECTOR: -fno-stack-protector
STATIC_GMP: /usr/local/lib/libgmp.a
STATIC_GMP_INCLUDE_DIR:
PIC_GMP: /usr/local/lib/libgmp.a
PIC_GMP_INCLUDE_DIR:
ENABLE_MCSAT: no
STATIC_LIBPOLY:
STATIC_LIBPOLY_INCLUDE_DIR:
PIC_LIBPOLY:
PIC_LIBPOLY_INCLUDE_DIR:

Version: Yices 2.5.2
Host type: x86_64-apple-darwin19.4.0
Install prefix: /Users/mvalais/code/ocamlyices2/ext/yices
Build mode: release

For both static and shared library:
CPPFLAGS: -DMACOSX -DNDEBUG
CFLAGS: -fvisibility=hidden -Wall -Wredundant-decls -O3 -fomit-frame-pointer -fno-stack-protector
LDFLAGS:

For static library libyices.a:
Enable: yes
STATIC_CPPFLAGS: -DYICES_STATIC
STATIC_LIBS:
Libgmp.a found: yes
Libgmp.a path: /usr/local/lib/libgmp.a
Libgmp.a is pic: yes (non-PIC is faster for the static library)
PIC mode for libyices.a: default
Use shared gmp instead of libgmp.a: no
Embed gmp in libyices.a: yes

For shared library:
Enable: yes
SHARED_CPPFLAGS:
SHARED_LIBS:
Libgmp.a with PIC found: yes
Libgmp.a path: /usr/local/lib/libgmp.a
Use shared gmp instead of libgmp.a: no
`

A final word about the Makefile: since all the build configuration is handled by ./configure, you don't have to pass any variables at make time anymore, which really helps when you need to make multiple times in a row and don't want to type the variables every single time.

Inpecting static and dynamic libraries

Here are two tips that I learned along the way. First, I very often need to know what libraries an executable is depending on:

`sh

macOS

% otool -L /bin/ls
/bin/ls:
/usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

Linux (Alpine Linux 3.9)

ldd /bin/ls
/lib/ld-musl-x86_64.so.1 (0x7fc9b2c84000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fc9b2c84000)
`

I also had to dig into the symbols of libraries in order to make sure that static libraries contained all the needed symbols:

sh % nm ext/libyices_pic_no_gmp/lib/libyices.a ext/libyices_pic_no_gmp/lib/libyices.a(libyices.o): 00000000000ef6c0 t _convert_rba_tree 0000000000049a20 t _convert_simple_value 00000000000c2f70 t _convert_term_to_bit 00000000000d97e0 t _convert_term_to_conditional 0000000000049700 t _convert_term_to_val 0000000000049b30 t _convert_val 00000000000028b0 T _yices_or 0000000000002fa0 T _yices_or2 0000000000002ca0 T _yices_or3 0000000000003480 T _yices_pair 000000000002ad30 t _yices_parse 0000000000006420 T _yices_parse_bvbin 00000000000064b0 T _yices_parse_bvhex 0000000000004200 T _yices_parse_float 0000000000004180 T _yices_parse_rational 000000000000b150 T _yices_parse_term 000000000000b090 T _yices_parse_type U _memcpy U _memset U _memset_pattern16 U ___error 0000000000145280 S ___gmp_0 000000000017dcf0 D ___gmp_allocate_func 00000000001099e0 T ___gmp_assert_fail 0000000000109980 T ___gmp_assert_header 0000000000145460 S ___gmp_binvert_limb_table 000000000014527c S ___gmp_bits_per_limb 0000000000109a60 T ___gmp_default_allocate 0000000000109ae0 T ___gmp_default_free 0000000000109a90 T ___gmp_default_reallocate 0000000000145290 S ___gmp_digit_value_tab

The letter before the symbol is the "symbol type" (from man nm):

Each symbol name is preceded by its value (blanks if undefined). This value is followed by one of the following characters, representing the symbol type:

U = undefined,

T (text section symbol),

D (data section symbol),

S (symbol in a section other than those above).

If the symbol is local (non-external), the symbol's type is instead represented by the corresponding lowercase letter. A lower case u in a dynamic shared library indicates a undefined reference to a private external in another module in the same library.

For example, the symbol _yices_parse_float is an external symbol, meaning that this symbol isn't static to libyices.a. On the other side, _convert_simple_value is statically defined (t).

Contributing the new Yices build system to upstream

On 16 June 2016, I sent an email to Bruno Dutertre, one of the developers at SRI (the company behind the Yices SMT solver). I proposed all these changes with links to the various patches on GitHub. Unfortunately, it didn't work out, and the reason might be that the whole patch was enormous and very hard to review.

Hi Maël,

Thanks for the message and for your efforts. We'll look into your updates.

We know that the Yices build system is unconventional because most of the work is done in the Makefiles rather that in the configure script. There are historical reason for this (and it should be able to build PIC libraries without problems).

By the way, Yices is now open-source (GPL) on github: https://github.com/SRI-CSL/yices2. Take a look when you have time,

Thanks again,

Bruno

I wish we had a unit-test framework for autoconf. The autoconf ecosystem generates very fragile scripts and the only way to test them is to run them with all possible flag combinations, which is pretty much impossible.

Update 31 May 2020: added the ascii "dependency" diagram to give a better sense of what the challenge was.

Github Actions with a private Terraform module

Maël Valais — Sat, 09 May 2020 14:02:00 +0000

A common way of sharing terraform modules is to move them in a separate repo. And for companies, that means a private repo. When terraform init is run, the terraform module is fetched and if this module is stored on a Github private repo, you will need to work around the authentication.

Imagine that these shared modules are stored on the private Github repo "github.com/your-org/terraform-modules". Importing this module from a different repo would look something like:

module "some_instance_of_this_module" {
  source = "git@github.com:your-org/terraform-modules.git//path/to/module?ref=master"
}

Using git+ssh as a way of fetching this private module will work great locally since you might probably have a private key that Github knows about. Locally, terraform init will work.

But what about CI, should I create a key pair and store the private key as a secret and have the public key known by Github (or Gitlab)?

Using SSH key pairs is not ideal. The key pair is tied to an individual and can't be linked to a Github App like github-bot. Plus, the key pair mechanism doesn't offer access to a specific repo, which means all your company's repositories are exposed. And finally, some companies do not have port 22 open for security reasons, which means git+https is their only option.

To use https instead of ssh, we start by changing the way we import these modules:

 module "some_instance_of_this_module" {
-   source = "git@github.com:your-org/terraform-modules.git//path/to/module?ref=master"
+   source = "git::https://github.com/your-org/terraform-modules.git//path/to/module?ref=master"
 }

Local development & git over https

Locally, you will have to make sure you can git clone this private repo, for example, the following should work:

git clone https://github.com/your-org/terraform-modules.git

If it doesn't work, Github has a helpful page "Caching your GitHub password in Git".

Continous integration & git over HTTPS

It is a bit trickier to get HTTPS working on the CI. In the following, I'll take the example of Github Actions but that will work for any CI provider. There are two main solutions:

using url.insteadof,
or using credential.helper.

For both options, you will need a PAT (personal access token) linked your own account. I refer to this token as GH_TOKEN. To create a PAT, you can go to your Github settings.

⚠️ NOTE: around April 2020, Github decided to prevent users from using Github Secrets names that begin with GITHUB_. We used to use the name GITHUB_PAT frequently in Github Actions readmes, I guess we will all have to update everything!

Solution 1: `url.insteadof`

actions/checkout@v2 sets the 'Authorization' for any git command issued from the checked out repo. But since this uses the Github Actions GITHUB_TOKEN which is limited to the current repository. And since we want to access another private repo, we have to disable that.
instead, you can use git config --global url.insteadof. The GH_TOKEN Github Secret is a Github personal token that has the 'repo' scope (full control of private repositories).

Note: when using git over https with a token on https://github.com, the username doesn't matter, that's why we put foo here.

- run: |
    git config --local --remove-section http."https://github.com/"
    git config --global url."https://foo:${GH_TOKEN}@github.com/your-org".insteadOf "https://github.com/your-org"
  env:
    GH_TOKEN: ${{ secrets.GH_TOKEN }}

Solution 2: `credential.helper`

When git config url.insteadof does not work, you can try using git credential.helper instead. For example:

- run: |
    cat <<EOF > creds
      #!/bin/sh
      echo protocol=https
      echo host=github.com
      echo username=foo
      echo password=${GH_TOKEN}
    EOF
    sudo install creds /usr/local/bin/creds
    git config --global credential.helper "creds"
  env:
    GH_TOKEN: ${{ secrets.GH_TOKEN }}

Solution 2 bis: `credential.helper` with a Github Action

Same as solution 2 but wrapped in a neat Github Action setup-git-credentials.

This action stores the credentials in the file $XDG_CONFIG_HOME/git/credentials and configures git to use it by calling:

git config --global credential.helper store/

The actions also makes sure all ssh calls are rewritten to https. The 'credentials' field must be of the form https://foo:$GH_TOKEN@github.com/

- uses: fusion-engineering/setup-git-credentials@v2
  with:
    credentials: https://foo:{{secrets.GH_TOKEN}}@github.com

Learning Kubernetes Controllers

Maël Valais — Wed, 22 Apr 2020 09:58:00 +0000

Kubernetes' extensibility is probably its biggest strength. Controllers and CRDs are all over the place. But finding the right information to begin writing a controller isn't easy due to the sheer amount of tribal knowledge scattered everywhere. This post intends to help you start with controllers.

Let us begin with some terminology:

controller: a single loop that watches some objects. We often refer to this loop as "controller loop" or "sync loop" or "reconcile loop".
controller binary is a binary that runs one or multiple sync loops. We often refer to it as "controllers".
CRD (Custom Resource Definition) is a simple YAML manifest that describes a custom object, for example this CRD defines the acme.cert-manager.io/v1alpha3 Order resource. After applying this CRD to a Kubernetes cluster, you can apply manifests that have the kind "Order"

Note: CRDs and controllers are decoupled. You can apply a CRD manifest without having any controller binary running. It works in both ways: you can have a controller binary running that doesn't require any custom objects. Traefik is a controller binary which relies on built-in Service objects.

Note: the "CRD" manifest is just a schema. It doesn't carry any logic (except for the basic validation the apiserver does). The actual logic happens in the controller binary.

operator: the term "operator" is often used to mean a controller binary with its CRDs, for example the elastic operator.

Here are the links that I would give to anyone interested in writing their own controller:

sig-api-machinery/controllers.md gives a good intuition as to what a "controller" is:

A Kubernetes controller is an active reconciliation process. That is, it watches some object for the world's desired state, and it watches the world's actual state, too. Then, it sends instructions to try and make the world's current state be more like the desired state.

Note: the client-go's informers and listers and workqueue are not mandatory for writing a controller: you can just rely on client-go's Watch primitive to reconcile state. The informers and workqueue add important scalability and reliability features but these also come with the cost of heavy abstractions. Use client-go's Watch first to have a sense of what it can offer, and then try out informers and workqueue.

Kubebuilder book is a nice starting point. Kubebuilder uses code generation a lot and that's what most controllers use nowadays (Rancher uses a somehow forked version of controller-runtime and controller-tools, Wrangler, that also generates code but with their own "style" – for example, simple flat interfaces instead of client-go's deeply nested interfaces that don't feel like Go).
The Kubernetes API conventions is an amazing document. It summarizes a lot of the "tribal knowledge" around naming and how sync loops are conceived and what they mean by "level-based behaviour".
Github search "language:yaml language:go kubernetes controllers", tons of nice examples of controllers
cert-manager's codebase is a nice controller to take a look at
ClusterAPI proposals and codebase (capi, capa) and (we took a lot of inspiration from what they do)
The ClusterAPI Meeting notes contains a ton of useful information on Machine, MachinePool... (crazy how much I learned from that).
The Kubernetes status field is tricky. You can take a look at "conditions vs. phases vs. reasons".
The Kubernetes codebase itself is also a very nice read. It might feel overwhelming at first; I invite you to take a look at a few of the following sync loops contained in the kube-controller-manager, kube-scheduler and kubelet. Since each sync loop reads or updates different objects, I also detail which objects are updated or created by each sync loop:

| binary | sync loop = component | reads | creates | updates |
| ----------------------- | ---------------------------------- | ----- | ---------- | ---------- |
| kube-controller-manager | syncDeployment | Pod | ReplicaSet | Deployment |
| kube-controller-manager | syncReplicaSet | | Pod | |
| kubelet | syncPod | | | Pod |
| kube-scheduler | scheduleOne | | | Pod |
| kubelet | syncNodeStatus | | | Node |

The podcast episode "Gotime #105 – Kubernetes and Cloud Native" (Oct. 2019) with Joe Beda (initiator of Kubernetes) and Kris Nova is very interesting and tells us more about the genesis of the project, which things like why is Kubernetes written in Go and why client-go feels like Java. For example: > Kris Nova: I think there’s a fourth role. I think there’s what we called in the book an infrastructure engineer. These are effectively the folks like Joe and myself. These are the folks who are writing software to manage and mutate infrastructure behind the scenes. Folks who are contributing to Kubernetes, folks who are writing the software for the operators, folks who are writing admission controller implementations and so forth… I think it’s this very new engineer role, that we haven’t seen until we’ve started having – effectively, as Joe likes to put it, a platform-platform.
The operator-sdk (RedHat) is a package that aims at helping dealing with the whole scafollding when writing a sync loop. It relies on controller-runtime. I don't use either of them but taking a look at these projects helps getting more understanding about the challenges (read: boilerplate) that comes when writing controllers. I personally write all the controller-related boilerplate myself (creating the queue, setting event handlers, running the loop itself...).

And a final note: CRDs are not necessary for writing a controller! You can write a tiny controller that watches the "standard" Kubernetes objects. That's exactly what ingress controllers do: they watch for Service objects.

Update 23 April 2020: I added a quote from Kris Nova! 😁
Update 2 May 2020: Rephrased the "terminology" bullet points to make them clearer and added a note on CRD vs. controller binary.

The Client-go Transitive Hell

Maël Valais — Wed, 15 Apr 2020 00:00:00 +0000

⚠️ I'm not sure this is a transitive issue. It might just not be due to transitivity at all!

Looks like the Kubernetes people chose to break the client-go API without bumping the major version (with a new import path e.g k8s.io/client-go/v13) when adding support for context.Context that comes with Kubernetes v1.18. Darren Shepherd reported the issue in February 2020. We were warned:

The v0.x.y tags indicate that go APIs may change in incompatible ways in different versions.

At first, I thought that it would not affect us at Ori since we don’t use any extra dependency that rely on client-go — so no transitive dependencies on client-go, we are the sole user of it. We then began working on some tooling depending on the API of our main project. And the transitive hell began.

Here is what the error looks like:

# k8s.io/client-go/rest
../../../go/pkg/mod/k8s.io/client-go@v11.0.0+incompatible/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher
        have (*versioned.Decoder)
        want (watch.Decoder, watch.Reporter)

On top of that, you might also have a version mismatch between k8s.io/api and k8s.io/apimachinery with the following error:

../../../go/pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/clientcmd/api/v1/conversion.go:52:12: s.DefaultConvert undefined (type conversion.Scope has no field or method DefaultConvert)

If you cannot upgrade to v0.18.* (e.g. you have a transitive dependency that still relies on v0.17.4), a workaround is to set client-go to use the latest pre-v1.18 version:

 require (
-    k8s.io/client-go v0.18.1
-    k8s.io/apimachinery v0.18.1
-    k8s.io/api v0.18.1 //indirect
+    k8s.io/client-go v0.17.4
+    k8s.io/apimachinery v0.17.4
+    k8s.io/api v0.17.4 //indirect
 )

The v0.17.4 version is the last version of apimachinery and api that stays compatible with client-go pre-v1.18.

Long-term: you want to upgrade to client-go v0.18.* as soon as possible. If the breaking dependencies that still require client-go with no context.Context argument, ask their maintainer (if it is an open-source project) and hopefully that will work... but then, anyone relying on this project will also have then stuff broken.

The project still has v11, v12... version but they stopped tagging new major versions like they did with v11.0.0 (there is no v18.0.0) because they don't use the major used with Go Modules. That's due to the fact that the client-go project doesn't follow the "semantic import versioning" rule. You can't do:

% go get k8s.io/client-go@v12.0.0
go get k8s.io/client-go@v12.0.0: k8s.io/client-go@v12.0.0: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v12

That's why the Kubernetes team maintains another set of tags that begin with v0.* (e.g., v0.18.0) for that specific reason. Just a clever way of escaping the semantic import versioning, but all these different versions make it very confusing...

Oh, and when you use the kubernetes-1.17.4 tag, it redirects to the v0.17.4 tag (my guess is that it is infered by go get):

% go get k8s.io/client-go@kubernetes-1.17.4
go: k8s.io/client-go kubernetes-1.17.4 => v0.17.4
go: downloading k8s.io/client-go v0.17.4

And to make even more confusing, these tags (kubernetes-v1.17.4, v0.17.4 and v12.0.0) are not set on the master branch; instead, they all live on headless branches.

This issue is a reminder that we (Kubernetes hackers who write controllers for a living) heavily rely on the “good will” of the Kubernetes team. These decisions as they might affect anyone relying on Kubernetes "as a platform"... 🤔

How do packets find their way back?

Maël Valais — Mon, 13 Apr 2020 00:00:00 +0000

In a previous post "The Packet's-Eye View of a Kubernetes Service", I studied how traffic flows in when using Kubernetes Services. In the last diagram of that post, I could not clearly see how traffic could make its way back to the user. In this article, I will try to understand how packets are able to flow back to the user and where stateless rewriting happens.

In the following diagram, we can see a packet coming from a user, then being re-written by Google's VPC firewalls and finally coming into a VM "node 1".

So, how come the packet can come back ~~and does it use conntrack~~ and does Google's firewall have to remember some state?

Update 14th April: I initially thought that the conntrack kernel module would not register anything when using DNAT. @networkop1 showed me that conntrack registers the connection even for stateless DNATing.

conntrack is a part of the netfilter suite in the Linux kernel. It is in charge of remembering connections that are forwarded. The initial packet hits the iptables machinery and conntrack remembers it so that further packets don't need to go through iptables again. You can list the tracked connections using the conntrack(8) tool. I mention it in "Debugging Kubernetes Networking".

Let us dive a bit more and add the "response" packets. For the following diagram, I used the excellent textik ascii drawing tool.

           D-NAT (dest-based NAT, also called port-forwarding)

+--------------------------------------------------------------------------+
|   src: 90.76.45.149:32345                       src: 35.211.248.124:80   |
|   dst: 35.211.248.124:80                        dst: 90.76.45.149:32345  |
+------------------------------90.76.45.149--------------------------------+
               |                 (user)                     |
               |                                            |
+--------------------------------------------------------------------------+
|              |                one-to-one                  |              |
|              v              port forwarding               |              |
|   src: 90.76.45.149:32345          =          - src: 10.142.0.62:80      |
| - dst: 35.211.248.124:80      no need for     + src: 35.211.248.124:80   |
| + dst: 10.142.0.62:80         conntrack to      dst: 90.76.45.149:32345  |
|              |                 remember!                  ^              |
|              |                (stateless)                 |              |
|              |                                            |              |
+--------------|--------------35.211.248.124----------------|--------------+
               |              (Google's VPC)                |
               |                                            |
               |                                            |
+--------------|--------------------------------------------|--------------+
|              v             userland process               |              |
|   src: 90.76.45.149:32345      response         src: 10.142.0.62:80      |
|   dst: 10.142.0.62:80      ----------------&gt;    dst: 90.76.45.149:32345  |
|                                                                          |
+-------------------------------10.142.0.62--------------------------------+
                                (VM in VPC)

By reading through the diagram, we can see that the packet is re-written by Google's firewall using DNAT: the destination is replaced by a fixed IP, the one of the VM.

Why do I say "packets" but what I should really say is "segments"? That's because I don't really know anyone using this strict terminology. Outside of the kernel and TCP/IP stack implementors, who actually cares about the L3 layer "units"? And I enjoy the "packet" word too more than "segment"!

Now, why do I care about Google's firewalls storing state? That's because if some state about a connection has to be remembered, it means it is harder to distribute the firewall horizontally, which makes it harder to scale.

As we can see on the diagram, the firewall does not need to remember anything: it is just a static one-to-one relation between 10.142.0.62 and 35.211.248.124.

Here is what I want to remember from this post:

Outgoing traffic from your broadband modem router has to be SNATed (source-based NAT). The router needs to keep track of outgoing connections using conntrack.
Incoming traffic from the Internet to a Google Cloud VM has to go through the VPC firewall. The packet rewriting is very fast and very scalable since it only uses DNAT, which means no need to remember anything.
Most packet forwarding in Kubernetes relies on stateless DNATing (e.g. hostPort or nodePort). Some parts of Kubernetes rely on stateful SNAT rewriting, for example when you use externalTrafficPolicy: Cluster in a which is the default policy for a Service. The following diagram shows where this rewriting happens (extracted from the last diagram in "The Packet's-Eye View of a Kubernetes Service"):

The evolution of my home office from 2019 to 2022

Maël Valais — Mon, 30 Mar 2020 00:00:00 +0000

In this post, I document the changes to my office setup through time.

2022 Update 2

I decided to move my office out of my home! I am now located in a co-working space in the city of Toulouse. It is called "HarryCow".

2022 Update

I added a second monitor. It is the same as the one I bought in 2019.

2021 Second Update

As you can see in the above picture, I went from a Macbook Pro to a custom PC I built!! I wanted a machine purposed for Linux and with as much CPU as I could afford. I made a gist with all the tips I have learned from my experience of switching from macOS to Linux.

Here is the part list (also available on pcpartpicker):

Part	Name	Price (01 May 2021)
CPU	~~AMD Ryzen 9 5900X 3.4 GHz 12-Core Processor~~ (I sent it back)	€ 708 (street price)
CPU	AMD Ryzen 9 5950X 3.4 GHz 16-Core Processor (bought on AMD.com at MSRP)	€ 795 (MSRP)
CPU Cooler	Noctua NH-D15 CHROMAX.BLACK 82.52 CFM CPU Cooler	€ 99
Motherboard	Gigabyte X570 AORUS MASTER ATX AM4 Motherboard	€ 359
Memory	G.Skill Trident Z RGB 32 GB (2 x 16 GB) DDR4-3600 CL16 Memory	€ 304
Storage	Samsung 980 Pro 1 TB M.2-2280 NVME Solid State Drive	€ 219
Video Card	~~Gigabyte Radeon RX 6700 XT 12 GB EAGLE~~ (I sent it back)	€ 858 (street price)
Video Card	AMD Radeon RX 6800 16 GB (bought on AMD.com at MSRP)	€ 579 (MSRP)
Case	Fractal Design Define 7 Compact ATX Mid Tower Case	€ 133
Power Supply	be quiet! Dark Power Pro 11 750 W 80+ Platinum Semi-modular ATX Power Supply	€ 243
Case Fan × 4	be quiet! Silent Wings 3 59.5 CFM 140 mm Fan PWM	€ 29 × 4
Total		€ 2847

2021 Update

I switched from the Apple Magic AZERTY keyboard to a QWERTY mechanical keyboard: the Durgod Taurus K320 TKL. I chose the Cherry MX Brown switches, and I love them. It feels so much better to be typing on this keyboard. Moving from AZERTY to QWERTY was the hardest part. It took about 3 months to get as confortable as I was using the AZERTY layout. I really wanted to stop working on a "second class" keyboard layout that has poor shortcut support on most apps. The other adjustment I had to do was to go to the macOS settings and swap the Windows key with the Alt key so that the ⌘ and ⌥ keys are at in the right order.
Two UTEBIT articulating arms that hold (1) an old iPhone 7 that I use as a secondary screen using Duet and (2) a Razer Kiyo camera.
A Razer Kiyo camera (USB 3). I wish I had a proper DSLR camera though: the Kiyo needs a ton of light to have a good image quality.
Two UTEBIT tabletop arms that I use to hold the flood lights.
Two NEEWER bi-color 660 flood lights.
A set of UTEBIT 1/4 to 3/8 screw adapters.
24 panels of the t.akustik WAS-7 Absorber. It greatly reduced the amount of echo/reberb in the room, and allows me to put the microphone a bit further away without losing on sound quality.

2020 Update

Yeti Compass boom arm > Note that the boom arm is meant to carry the shock mount + the mic and since I only mounted the Yeti mic, the spring mechanism is a bit too tight which means the arm tends to go up. I unscrewed the screw in the base of the boom arm at its minimum but the tension is still too high and the boom arm tends to go up.
Apple Magic Keyboard. I also use a Sharkoon PureWriter TKL from time to time when I want to bother my collegues with the clank-clank sound of the red switches
USB-C hub Ugreen and AUKEY USB3 hub.

2019 Office

LG 27UL850-W 27 inches 4K monitor
Griffin Elevator stand
The Flexispot standing desk E5B (B = black) with a €25 wood board I mounted on top.
I use the Beats Solo 3 headphones. I only use it for audio output. I never use the mic in Bluetooth mode since it greatly degrades my colleague's listening experience.
Blue Yeti USB microphone > Why a standalone mic just for Zoom? One very important aspect of the remote-only work setup is the importance of being properly understood. I take this very seriously and think that the sound quality of the mic I use every day influences how effective my meetings are. I also make a number of calls to potential candidates, which means I should sound perfect.
Logitech K760 (wireless solar keyboard).
Logitech MX Master Just an excellent mouse
AUKEY mousepad XXL.
Some random chair (I wish I could afford the Herman Miller Aeron)

And finally, here is a picture from the backyard. 🙂

Migrating from GKE to Civo's K3s

Maël Valais — Sun, 22 Mar 2020 00:00:00 +0000

To learn and play with Kubernetes, I keep a "playground" cluster to try things on it (helm files I use are here). Since 2019, I have been using GKE (Google's managed Kubernetes service), which works great with the one-year \$300 credit that you get initially. A few days ago, reality hit me hard with this message:

I had only 2 days to find a plan and migrate everything away from GKE! My current setup was only using a single n1-standard-1 on us-west-1 and with no network load balancer. But that was still around €15 a month, and I just didn't want to pay. I chose to migrate to Civo's managed K3s since they are in beta and I really wanted to try K3s.

Civo's Managed K3s

Civo is a company that offers a public cloud as well as managed Kubernetes clusters. Their managed Kubernetes offer, named "KUBE100", is quite new (launched in mid-2019). Unlike most managed Kubernetes offerings like EKS or GKE, Civo went with Rancher's K3s Kubernetes distribution.

Compared to the standard Kubernetes distribution, K3s is a way lighter: simple embedded database with sqlite, one single binary instead of four; a single VM can both host the control plane and run pods. With the traditional Kubernetes distribution, you have to run the control plane on a separate VM. Note that Civo has a whole blog post on "Kubernetes vs. K3s".

This lightweight feature is what brings me here, and also the fact that K3s comes by default with Traefik & their own tiny Service type=LoadBalancer controller, which means you don't even need an expensive network load balancer like on GKE when you want to expose a service to the internet.

Of course, K3s has drawbacks and is probably meant for IoT-based clusters, but it's also perfect for playing around with Kubernetes! So I went ahead and create a two-nodes cluster:

Note that since this is K3s, the master can also be scheduled with pods. So I could have gone with a single node.

Migrating ExternalDNS, cert-manager and Traefik

That's the easy part. I just had to run ./helm_apply which runs helm upgrade --install for each helm chart I use.

In the process, I decided to go with *.k.maelvls.dev instead of *.kube.maelvls.dev. The shorter, the better! I forgot to add that I still use Google's CloudDNS. I did not find an easy alternative yet. Maybe just Cloudflare for now?

After creating Traefik, I knew that its service would automatically be populated with the status.loadBalancer field thanks to K3s' servicelb. Let's see:

% kubectl -n traefik get services
NAME     TYPE          CLUSTER-IP        EXTERNAL-IP      PORT(S)
traefik  LoadBalancer  192.168.255.134   91.211.152.190   443:32164/TCP,80:32684/TCP

Great! Now, Traefik will propagate this external IP to the ingresses, and ExternalDNS will use the status.loadBalancer from these ingresses in order to set A records.

If you want to know more about how "servicelb" works, you can take a look at The Packet's-Eye View of a Kubernetes Service where I describe how Akrobateo works (K3s' servicelb has the exact same behavior).

Migrating MinIO from the old to the new cluster

I use minio for various uses. It is great if you want a S3-compatible storage solution. In order to migrate, I followed this. It's quite painless:

$ kubectl -n minio run a --generator=run-pod/v1 -it --rm --restart=Never --image=alpine

% wget https://dl.minio.io/client/mc/release/linux-amd64/mc && install mc /usr/bin
% mc config host add old https://minio.kube.maelvls.dev AKIAIOSFODNN7EXAMPLE "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" --api S3v4
% mc config host add new http://minio:9000 AKIAIOSFODNN7EXAMPLE "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" --api S3v4

% mc ls old/         # List buckets since I had to create manually each bucket.
% mc mb new/bucket1  # Then create each bucket one by one.
% mc cp --recursive old/bucket1/ new/bucket1/

I also decided to change the access key & secret key. Again, quite painless. As mentioned in the documentation, I changed the secret stored in Kubernetes:

kubectl -n minio edit secret minio

and then I temporarily added the MINIO_ACCESS_KEY_OLD and MINIO_SECRET_KEY_OLD to the deployment by editing it.

kubectl -n minio edit deployment minio

After that, the pods get recreated, and MinIO picks up the new secret. Note: I also had to edit the deployment again in order to remove the temporary _OLD environment variables.

To recap, the whole migration was painless. The only data I migrated was MinIO. Note that I didn't have any SLA to comply with, but if I had planned a bit better, I could have moved over with almost zero downtime.

In order to get almost-zero-downtime, I would have made sure to keep the old and new MinIO instances replicated until the move was over. The only problem with the whole migration is the DNS change. I cannot precisely know when the propagation will take. After completing the migration and assuming that all the DNS entries are propagated, if for some reason people keep hitting the old IP due to outdated DNS entries, the old and new clusters would have become out-of-sync. To mitigate that issue, I could have chosen to "cordon" the old cluster just to make sure that this case never happens.

The repo for my Kubernetes playground cluster (*.k.maelvls.dev) is available here.

Update 7 May 2020: better introduction explaining why I use GKE, tell what Civo and K3s are all about.