DEV Community

sanskar arora
sanskar arora

Posted on Edited on

🐈 HomeLab Chronicles: Episode 11 - schrodingers cluster

Hey all πŸ‘‹

Last episode was an RCA with seven root causes and a port number. This one is an RCA with one root cause, five wrong theories, and a cluster that reported itself healthy for thirty-eight hours while doing absolutely nothing.

State of the lab going in: two nodes, Flux in charge, Airflow on a public domain through a Cloudflare tunnel, nightly backups, DHCP reservations finally ticked. Episode 10 ended with "the router remains the weakest member of the team." I would like to amend that statement. The router has competition.

Here's what happened, in the order I found out, which is not the order it happened.


🚨 Symptom: The Tunnel Is Down

Sunday night, 10 PM. Cloudflare's Zero Trust dashboard, which has shown two happy connectors since Episode 9, says the tunnel is down. No connectors. Nothing.

First look at the cluster:

$ kubectl get nodes
NAME   STATUS     ROLES    AGE   VERSION
dell   NotReady   <none>   64d   v1.35.6
msi    Ready      <none>   64d   v1.35.6
Enter fullscreen mode Exit fullscreen mode

Dell is down. Has been for two and a half days, it turns out. And the pods tell the failover story exactly the way the textbook says it should go:

$ kubectl -n cloudflared get pods -o wide
NAME                           READY   STATUS        RESTARTS   AGE     NODE
cloudflared-6cf54796d6-42rg9   1/1     Running       0          2d16h   msi
cloudflared-6cf54796d6-7g425   1/1     Terminating   0          5d9h    dell
cloudflared-6cf54796d6-jkt44   1/1     Running       0          2d16h   msi
cloudflared-6cf54796d6-t567p   1/1     Terminating   0          5d9h    dell
Enter fullscreen mode Exit fullscreen mode

Dell died, the old pods went Terminating, two fresh replicas came up on msi 2d16h ago. Running. 1/1. Zero restarts. Kubernetes did its job.

Small aside before we continue. Episode 9 proudly said: two replicas with pod anti-affinity so the tunnel survives a node dying. Look at the ages. Both old replicas were on dell. Anti-affinity on a two-node cluster is a preference, and the scheduler preferred dell twice. The tunnel survived the node dying anyway, because failover worked. It just didn't survive what came next.

So the question was simple. Two healthy connectors on msi, dashboard says zero. Why?


πŸ”¬ Hypothesis 1: The Failover Path Was Never Tested

Both replicas had always lived on dell. This was the first time cloudflared had ever run from msi. Maybe msi's egress was different: UDP 7844 blocked, QUIC failing, DNS for argotunnel.com not resolving from msi's pods.

The test: read the pod's log. The fix if true: TUNNEL_TRANSPORT_PROTOCOL=http2 to force TCP, or fix whatever was eating UDP.

What happened instead:

$ kubectl -n cloudflared logs cloudflared-6cf54796d6-42rg9
Error from server: Get "https://192.168.0.201:10250/containerLogs/...":
tls: failed to verify certificate: x509: certificate is valid for 192.168.68.210, not 192.168.0.201
Enter fullscreen mode Exit fullscreen mode

Episode 10, Sidequest 2, verbatim, except on the other node. When I moved subnets, I fixed dell's kubelet certificate because that's where the pod I wanted to read was. msi's kubelet had the same stale certificate the entire time, and nobody noticed because nobody asked msi for logs. kubectl get doesn't touch the kubelet. kubectl logs does.

Fine. The host has the logs on disk. Over ssh:

sudo tail -n 80 /var/log/pods/cloudflared_cloudflared-6cf54796d6-42rg9_*/cloudflared/0.log
Enter fullscreen mode Exit fullscreen mode

And the log was a complete short story:

2026-10-31T01:11:58Z INF Registered tunnel connection connIndex=1 ... location=blr03 protocol=quic
2026-10-31T01:11:58Z INF Registered tunnel connection connIndex=2 ...
2026-10-31T01:11:59Z INF Registered tunnel connection connIndex=3 ...
2026-10-31T01:12:03Z INF |  SUMMARY: Environment is healthy. cloudflared will use 'quic' as primary protocol.  |
...
2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=0
2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=3
2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=2
2026-11-01T07:24:54Z ERR Serve tunnel error error="... timeout: no recent network activity" connIndex=1
...
2026-11-01T07:25:19Z INF Initiating graceful shutdown due to signal terminated ...
2026-11-01T07:25:22Z ERR no more connections active and exiting
2026-11-01T07:25:22Z INF Tunnel server stopped
Enter fullscreen mode Exit fullscreen mode

Hypothesis 1, dead. Failover worked perfectly. The msi replica registered four connections on Friday morning and every pre-check passed. Then on Saturday at 12:54 PM (the logs are UTC; I'm not), all four connections timed out in the same second, a SIGTERM arrived twenty-five seconds later, and the process exited cleanly.

And then nothing. No new container. The log file ends there. For thirty-eight hours, the API said Running, 0 restarts about a process that had logged its own obituary.

That changed the question. It was no longer "why is the tunnel down." It was "why did the kubelet not restart a dead container, and why is it lying about it?"


πŸ“‘ Interlude: Who Killed It

Four connections dying in the same second is not Cloudflare. That's the floor falling out from under the network. msi's journal for that minute:

Nov 01 07:24:49 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-BEACON-LOSS
Nov 01 07:24:50 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-DISCONNECTED bssid=... reason=4 locally_generated=1
Nov 01 07:25:32 msi microk8s.daemon-kubelite[1122435]: ...   <- kubelite starting
Nov 01 07:25:43 msi microk8s.daemon-kubelite[1122802]: ...   <- and again
Nov 01 07:25:53 msi microk8s.daemon-kubelite[1123187]: ...   <- and again
Nov 01 07:25:54 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-CONNECTED
Nov 01 07:26:55 msi microk8s.daemon-kubelite[1123923]: ...   <- fourth time's the charm
Nov 01 07:28:19 msi kernel: wlo1: deauthenticated from ... (Reason: 7=CLASS3_FRAME_FROM_NONASSOC_STA)
Nov 01 07:28:19 msi kernel: iwlwifi 0000:00:14.3: 0x004D9C74 | branchlink2
Enter fullscreen mode Exit fullscreen mode

Wi-Fi power save. The laptop's radio napped through too many beacons, decided the router was gone, and disconnected itself (reason=4 locally_generated=1, which is the card saying "this was my idea"). When the address vanished, so did the thing dqlite is bound to. Episode 5 explained this exact mechanism: a single-node MicroK8s keeps its datastore on loopback and is immune to this, and the day the Dell joined, dqlite moved to the LAN address and became a hostage of the Wi-Fi card. kubelite crash-looped four times in two minutes. Then the router kicked msi off twice more (reason=7, the router's idea this time), with an Intel firmware crash dump for garnish.

Somewhere in that mess, cloudflared got its SIGTERM. I never pinned down who sent it; my best guess is a liveness probe doing its job during the chaos. It doesn't matter much. What matters is that the kubelet came back after the fourth restart and then did nothing, for a day and a half.

The Episode 5 fix for the symptom, applied on the spot:

sudo iw dev wlo1 set power_save off
Enter fullscreen mode Exit fullscreen mode

The actual fix is a cable. We'll get there.


πŸͺͺ Blocker: The Certificate, Again

Before any more theories I needed kubectl logs and exec working on msi, because a debugging session without them is archaeology. Episode 10's trick was --rotate-server-certificates=true: let the kubelet request a fresh cert, approve the CSR, done. On msi it produced a different error:

Error from server: Get "https://192.168.0.201:10250/containerLogs/...": remote error: tls: internal error
Enter fullscreen mode Exit fullscreen mode
$ kubectl get csr
No resources found
Enter fullscreen mode Exit fullscreen mode

tls: internal error is the kubelet saying "rotation is on, I have nothing to serve yet, and I'm not going to serve the old file either." And it never filed a CSR. No error about why. Just a kubelet politely waiting for a certificate it never asked for. Hold that thought, because in hindsight it's the same bug wearing a different hat.

I stopped debugging rotation and signed the certificate by hand with the cluster CA. Same private key, new SAN list:

cd /var/snap/microk8s/current/certs
sudo cp -p kubelet.crt kubelet.crt.bak

printf '%s\n' \
  'basicConstraints=CA:FALSE' \
  'keyUsage=critical,digitalSignature,keyEncipherment' \
  'extendedKeyUsage=serverAuth' \
  'subjectAltName=DNS:msi,IP:192.168.0.201,IP:127.0.0.1' > /tmp/kubelet-ext.cnf

sudo openssl req -new -key kubelet.key -subj "/CN=msi" -out /tmp/kubelet.csr
sudo openssl x509 -req -in /tmp/kubelet.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -CAserial /tmp/kubelet-ca.srl -days 3650 -sha256 \
  -extfile /tmp/kubelet-ext.cnf -out /tmp/kubelet.crt

sudo install -m 660 -o root -g microk8s /tmp/kubelet.crt kubelet.crt
sudo sed -i '/^--rotate-server-certificates=true$/d' /var/snap/microk8s/current/args/kubelet
sudo snap restart microk8s.daemon-kubelite
Enter fullscreen mode Exit fullscreen mode
$ echo | openssl s_client -connect 192.168.0.201:10250 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    DNS:msi, IP Address:192.168.0.201, IP Address:127.0.0.1
Enter fullscreen mode Exit fullscreen mode

Certificate fixed. kubectl logs again:

$ kubectl -n kube-system logs deploy/coredns --tail=3
Error from server (NotFound): the server could not find the requested resource ( pods/log coredns-78894c95f4-l8n8j)
Enter fullscreen mode Exit fullscreen mode

A 404. From the kubelet. The TLS handshake succeeded, the API server asked msi's kubelet for coredns's logs, and the kubelet said "never heard of it." About a pod the API says has been running on msi for 64 days.

That's not a certificate problem. That's the kubelet not knowing what it's running.


πŸ”¬ Hypothesis 2: The Kubelet Has Lost Its Pods

The test: ask the kubelet directly for its pod list, through the API server's node proxy, and compare with what the API thinks.

sudo microk8s kubectl get pods -A --field-selector spec.nodeName=msi -o name | wc -l
# 34
sudo microk8s kubectl get --raw /api/v1/nodes/msi/proxy/pods
# {"kind":"PodList", ..., "items": null}
Enter fullscreen mode Exit fullscreen mode

Thirty-four pods in the API. Zero in the kubelet. Not "some missing." Null. And the kubelet's journal, every ten seconds, since a very specific moment:

Nov 01 07:26:57 msi kubelite: E1101 07:26:57 kubelet_node_status.go:392]
  "Error getting the current node from lister" err="node \"msi\" not found"
Enter fullscreen mode Exit fullscreen mode

07:26:57. Two seconds after the fourth kubelite restart on Saturday. The kubelet can't find its own node, and it's been saying so 15,000 times while kubectl get nodes says msi Ready.

Confirmed, then: the kubelet's view of the cluster is empty. Everything downstream follows. It won't restart a container belonging to a pod it doesn't know. It won't write status for it, so the API keeps showing the last status it ever wrote (the frozen Running, 0 restarts). It answers log requests with 404. And meanwhile containerd, which doesn't care about any of this, keeps running whatever was already running:

$ sudo microk8s ctr tasks ls | awk 'NR>1{print $3}' | sort | uniq -c
     59 RUNNING
Enter fullscreen mode Exit fullscreen mode

Airflow was fine. Postgres was fine. Fifty-nine containers happily running with no supervisor. The only things missing were the ones that had died, which nobody was around to restart.

But "the kubelet's view is empty" is a symptom. The question is why.


πŸ”¬ Hypothesis 3: The API Server's Watch Cache Is Empty

Quick mechanism, because the rest of the post depends on it. The API server has two ways to answer a read. A plain GET goes to the datastore. But every component that watches the cluster (the kubelet, the controller-manager, the scheduler, Flux) reads through the watch cache: an in-memory copy of the datastore that the API server keeps current by watching the datastore itself. Thousands of watchers, one datastore watch. That's what lets Kubernetes scale. You can also read from it on purpose, by asking for resourceVersion=0.

If that cache were empty, you'd get exactly this: kubectl sees a healthy cluster (datastore), every component sees nothing (cache).

The test: the same question, both ways. The fix if true: restart kubelite, since the cache is rebuilt on every API server start.

$ sudo microk8s kubectl get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi' | ...
34
$ sudo microk8s kubectl get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&resourceVersion=0' | ...
34
Enter fullscreen mode Exit fullscreen mode

Dead. The cache had all 34 pods. (I had also restarted kubelite twice by then, so an empty cache would have refilled anyway. Keep that in mind; it comes back.)


πŸ”¬ Hypothesis 4: The Kubelet's Credentials Are Broken

The API server answers me fine. Maybe it doesn't answer the kubelet. Episode 7 was an entire episode about the control plane being forbidden from reading its own objects, so this had form.

The test: replay the kubelet's exact startup queries using the kubelet's own kubeconfig. The fix if true: whatever the error says, probably certificates or RBAC.

$ K="sudo /snap/microk8s/current/kubectl --kubeconfig /var/snap/microk8s/current/credentials/kubelet.config"
$ $K auth whoami
Username   system:node:msi
Groups     [system:nodes system:authenticated]
$ $K get --raw '/api/v1/nodes?fieldSelector=metadata.name%3Dmsi&resourceVersion=0' | ...
1 ['msi']
$ $K get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&resourceVersion=0' | ...
34
$ $K get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&watch=1&resourceVersion=0' | head -c 200
{"type":"ADDED","object":{"kind":"Pod", ...
Enter fullscreen mode Exit fullscreen mode

Dead. The kubelet's identity gets the node, gets all 34 pods, and can open a watch. Every request the kubelet could make works when I make it.

So the kubelet isn't making them.


πŸ”¬ Hypothesis 5: The Kubelet's Informers Are Failing

Each component keeps its own local cache of the objects it cares about (an informer): one LIST to fill it, then a WATCH to keep it current. The kubelet won't touch pods until its informers have loaded once. If the LIST or WATCH were failing, the kubelet would log it and retry. Loudly.

The test: grep for reflector errors. The fix if true: depends on the error, but at least there'd be one.

$ sudo journalctl -u snap.microk8s.daemon-kubelite --since "90 min ago" \
    | grep -E 'reflector|failed to list|Failed to watch|forbidden|deadline exceeded'
(nothing)
Enter fullscreen mode Exit fullscreen mode

Dead, in a way that was more informative than an error would have been. Nothing was failing. And the controller-manager, which runs in the same kubelite process, told me what "nothing" looked like from the inside:

$ sudo journalctl -u snap.microk8s.daemon-kubelite --since "90 min ago" | grep -E 'Waiting for caches to sync|Caches are synced'
Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync"
Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync"
Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync"
... forty of these, all from the same second ...
Enter fullscreen mode Exit fullscreen mode

Fifty minutes after startup. Forty controllers waiting for caches to sync. Zero Caches are synced. Every informer in the entire control plane was stuck in the waiting room, not erroring, just waiting. Which, incidentally, is why kubectl rollout restart deployment/cloudflared an hour earlier had done precisely nothing: the deployment controller was in the same waiting room.

This also explains the certificate rotation that never filed a CSR. The kubelet's cert manager needs the node informer to have synced before it can build the request. Same wait, different symptom.

Four hypotheses down. Every test says the API server is fine, the data is fine, the credentials are fine, nothing is failing, and nothing is happening.


🧊 Hypothesis 6: The Cache Isn't Empty. It's Frozen.

Here's the hole in Hypothesis 3. When an informer starts, it doesn't ask for "whatever the cache has." It asks for the current state, and the API server, before answering, waits for its watch cache to catch up to the datastore's latest revision. A fresh snapshot, guaranteed. My resourceVersion=0 test never asked for that guarantee. It asked for the stale copy and got it.

A frozen cache, one that took a snapshot and then stopped receiving updates, would pass my test and hang every informer forever. And my test couldn't tell frozen from live because nothing in the cluster was changing, because everything that changes things was stuck waiting on the cache. Circular. Beautiful. Infuriating.

To catch it I needed an object that was still being written. The kubelet renews its node lease every ten seconds, straight to the datastore, informers or no informers.

The test: read the lease from the datastore, then from the cache.

$ date -u
Mon Nov  2 21:19:24 UTC 2026
$ ... leases/msi               -> rv 3815025  renewTime 2026-11-02T21:19:19Z
$ ... leases/msi?resourceVersion=0 -> rv 3810400  renewTime 2026-11-02T20:23:53Z
Enter fullscreen mode Exit fullscreen mode

Confirmed. The datastore's copy was five seconds old. The cache's copy was fifty-five minutes old, from 20:23:53, which is three seconds before I had restarted kubelite for the certificate. The cache took one snapshot at startup and then received nothing. Four thousand six hundred revisions behind and counting.

Every informer in kubelite was waiting for a cache to catch up to a datastore it had stopped listening to.

And why did restarting kubelite not fix it, twice? Because the cache is fed by a watch stream from k8s-dqlite, which is its own snap service:

$ systemctl show snap.microk8s.daemon-k8s-dqlite -p ActiveEnterTimestamp
ActiveEnterTimestamp=Wed 2026-10-28 07:07:22 UTC
Enter fullscreen mode Exit fullscreen mode

Running since the Episode 10 fix. The one component in the whole chain that had never restarted. And its log from Saturday, during the Wi-Fi hole:

Nov 01 07:32:33 k8s-dqlite: [go-dqlite] attempt 10: server 192.168.0.201:19001: dial: connect: network is unreachable
Nov 01 07:32:33 k8s-dqlite: error in txn: update transaction failed for key /registry/longhorn.io/nodes/longhorn-system/msi: context canceled
Nov 01 07:32:42 k8s-dqlite: error while range on /registry/serviceaccounts/: query (try: 0): context canceled
Enter fullscreen mode Exit fullscreen mode

The datastore, unable to reach its own address, cancelling its own queries. Episode 10's dial ... i/o timeout to your own old IP, except this time the IP was correct and simply didn't exist for ninety seconds.

k8s-dqlite is really kine, a shim that makes dqlite speak etcd's API. kine finds new writes by polling its table and pushes them to anyone watching, including the API server's cache. Somewhere in that context canceled storm, the push stopped, while reads and writes carried on working. Every kubelite restart after that got a fresh cache with a perfect snapshot and a dead feed. I never found the exact line of kine that wedged. I found the fix, which was good enough at 3 AM.


πŸ”§ The Fix

Restart the datastore, with the API server stopped so it isn't holding connections mid-stream, then bring the API server back to build a cache on a live feed. Containers keep running throughout; containerd doesn't care about any of this.

sudo microk8s dbctl backup /var/backups/microk8s/pre-restart-$(date +%F)   # paranoia is a feature
sudo snap stop microk8s.daemon-kubelite
sudo snap restart microk8s.daemon-k8s-dqlite
sleep 15
sudo snap start microk8s.daemon-kubelite
Enter fullscreen mode Exit fullscreen mode

Ninety seconds later:

$ ... leases/msi               -> rv 3817920  renewTime 2026-11-02T21:50:45Z
$ ... leases/msi?resourceVersion=0 -> rv 3817920  renewTime 2026-11-02T21:50:45Z
$ journalctl ... | grep -c 'Caches are synced'
58
$ journalctl ... | grep 'Watching apiserver'
Nov 02 21:49:13 msi kubelite: I1102 21:49:13 apiserver.go:52] "Watching apiserver"
$ ... /api/v1/nodes/msi/proxy/pods -> 34
Enter fullscreen mode Exit fullscreen mode

Same revision from both read paths. Fifty-eight controllers synced. The kubelet watching the API server again. Thirty-four pods known. And:

$ kubectl -n cloudflared get pods
NAME                           READY   STATUS    RESTARTS      AGE
cloudflared-6cf54796d6-42rg9   1/1     Running   1 (38h ago)   2d20h
cloudflared-6cf54796d6-jkt44   1/1     Running   1 (38h ago)   2d20h
Enter fullscreen mode Exit fullscreen mode

RESTARTS 1 (38h ago). The kubelet woke up, found two dead containers, restarted them, and for the first time in a day and a half wrote a status that was true. The "38h ago" is when the previous container died: Saturday, 12:55 PM, exactly where the log file ended.

$ kubectl -n cloudflared logs cloudflared-6cf54796d6-42rg9 --tail=5
INF Registered tunnel connection connIndex=0 ...
INF Registered tunnel connection connIndex=1 ...
INF Registered tunnel connection connIndex=2 ...
INF Registered tunnel connection connIndex=3 ...
Enter fullscreen mode Exit fullscreen mode

Dashboard: Healthy. Two connectors. Airflow back on its domain.


🧠 What This Taught Me

  • A cluster has two read paths, and they can disagree for days. kubectl get reads the datastore. Every controller reads the watch cache. When the cache freezes, you get a cluster where every human check passes and every automated action silently stops. Nothing errors, because waiting isn't an error.
  • "Running, 0 restarts" is a claim the kubelet made once. Pod status is written by the kubelet. If the kubelet stops writing, the API keeps serving the last thing it wrote, indefinitely. Status is a snapshot, not a heartbeat.
  • Every canary I had lied. Episode 7's readyz was green. Episode 7's coredns was 1/1 (it was frozen there). The controller-manager lease was even renewing (leader election doesn't need informers). The one thing that told the truth was a renewTime read two ways. New laminated card: kubectl get lease -n kube-node-lease <node> with and without resourceVersion=0. They must match.
  • Restarting the thing that's stuck doesn't help when the stuck thing is downstream. I restarted kubelite twice. The feed it depended on lived in a process I hadn't touched since October. Find the oldest PID in the chain.
  • Hypotheses die by test, not by argument. Five theories, each plausible, each killed by one command. The frozen cache only became visible when I stopped testing content and started testing freshness, with an object that was still moving.
  • Anti-affinity on two nodes is a suggestion. Both tunnel replicas lived on the same node for five days. The fix is a topologySpreadConstraint, which is a rule, not a wish.
  • Joining the second node moved the datastore onto Wi-Fi, and Wi-Fi power save is a slow-motion power cut. Episode 5 said this about the router dying. Turns out the laptop can do it to itself, from the inside, whenever the radio feels sleepy. power_save off is on now. The real fix is a cable into msi, and I've stopped pretending otherwise.
  • The certificate was never the problem, and I'd have been lost without fixing it. The stale kubelet cert blocked logs and exec, and the rotation trick from Episode 10 failed because of the same frozen cache I hadn't found yet. Hand-signing it with the cluster CA took five minutes and unblocked everything after.

πŸš€ What's Next

Dell is still NotReady, with its ghost pods stuck in Terminating and a triggerer StatefulSet that won't schedule a replacement until the old pod is confirmed dead. That's its own evening and its own post. After that: a spread constraint and a /ready liveness probe on the tunnel pods, so a dead connector gets restarted instead of memorialized. And a monitoring question I don't have a good answer to yet: what do you alert on when the cluster's own health signals are the thing that froze?

πŸ’¬ Final Thoughts

For thirty-eight hours my cluster was SchrΓΆdinger's: alive if you asked the datastore, dead if you asked anything that actually does work, and perfectly consistent with itself the whole time. The Wi-Fi card took a nap, the datastore cancelled its own queries, the cache stopped listening, and every controller in the building sat down to wait for an update that was never coming. Nobody errored. Nobody complained. Fifty-nine containers kept running on autopilot with no one at the controls, which is either reassuring or terrifying depending on how recently you've slept.

Popcorn 🍿, coffee β˜•, UPS πŸ”Œ, laminated card πŸͺͺ, chart-source bookmark πŸ”–, a domain 🌐, a DHCP reservation πŸ“Œ, and now a second laminated card that just says compare the renewTimes.

Top comments (0)