Hi guys! In my previous articles, I mostly worked with Terraform and Google Cloud. I focused on provisioning infrastructure, creating reusable modules, and understanding how to manage infrastructure through code.
This time, I wanted to explore something different.
What happens when I need to manage the machines underneath Kubernetes myself?
For this project, I started exploring how to build a non-production Kubernetes platform without relying on a managed Kubernetes control plane. Unlike GKE or other managed services, where the provider handles much of the control-plane management, a self-managed setup means I have to think about more of the underlying Linux, runtime, and networking configuration myself (Google Cloud, no date; Kubernetes, no date b).
That's what made this interesting to me.
The first milestone was getting the cluster up and running.
I ran:
kubectl get nodes
And the nodes showed Ready.
Great. The cluster was running. But then I started thinking about what that status actually meant.
Did it mean my applications could communicate with each other? Did it mean Kubernetes was using the correct network interface on every node? Did it mean my deployments would work consistently, or that I could recover the platform if something broke?
Not necessarily.
A node reporting Ready is useful, but it doesn't prove that the entire platform works as expected. Kubernetes describes Ready as a node condition, not a full application or platform validation (Kubernetes, no date h).
That became one of the main things I wanted to understand while building this platform.
This is the first article in my Building the Platform series. I'll document the technical problems I explored, the decisions I needed to understand, and why I'd consider one approach over another.
I'll start with the foundation: preparing the machines, configuring the container runtime, dealing with multiple network interfaces, and testing whether the cluster actually works.
Note: This article is about general Kubernetes engineering concepts in a non-production context. The diagrams, resource names, and configuration snippets are illustrative examples.
1. What I Wanted to Build
I started by looking at the foundation of a small self-managed Kubernetes cluster: a control plane, worker nodes, and the network that connects them. Kubernetes documents this general approach in its kubeadm installation guide (Kubernetes, no date b).
Instead of treating the underlying infrastructure as something a cloud provider would handle, I needed to think about the hosts themselves. This meant considering things that I normally wouldn't spend as much time thinking about when working with managed cloud services.
For example:
- Host operating-system settings had to be predictable.
- A node might have more than one network interface.
- IP addresses and routing needed to be deliberate rather than accidental.
- Host settings needed to remain consistent after a reboot.
- The operating system, container runtime, and Kubernetes components all needed to work together.
I also wanted this platform to support actual applications eventually, not just run a few test Pods.
That meant I would eventually need application routing, CI/CD, GitOps, persistent storage, monitoring, logging, and recovery procedures.
However, I didn't want to build everything at once without understanding how the individual components worked.
My approach was to start with the infrastructure foundation, validate it, and then build on top of it.
The different layers I wanted to understand looked something like this:
Developers
|
Source Repositories
|
CI/CD
|
GitOps
|
Kubernetes Cluster
|
+-------------+-------------+
| | |
Applications Platform Tools Persistent
Workloads
| | |
Services Monitoring Database
| and Logs |
+-------------+-------------+
|
Shared Gateway
|
Private LAN
This is a conceptual map of the capabilities I wanted to understand, not an exact deployment diagram. And it definitely isn't proof that every component works just because it's on the diagram.
For example, having a backup configuration is not the same as successfully restoring a database. I'll cover those distinctions as I document the different parts of the platform.
2. Preparing the Machines Before Installing Kubernetes
My first priority was making sure the machines were ready to run Kubernetes.
It's tempting to start directly with kubeadm init, install the CNI, and move on to deploying applications. But if the underlying operating system or network configuration is unreliable, those problems can show up later as Kubernetes issues.
I wanted to avoid that as much as possible.
2.1. Making Host Configuration Predictable
One of the things I had to consider was that Kubernetes depends on the hosts underneath it. If the operating system isn't configured predictably, I can end up troubleshooting Kubernetes when the problem is actually somewhere else.
So before focusing on Kubernetes, I needed to check a few things:
- Would the node remain available during normal operation?
- Would important configuration changes survive a reboot?
- If swap was deliberately disabled, would that setting survive a reboot?
- Was the intended network interface configured?
- Could I access the machine remotely when needed?
These might seem like small details, but they matter when you're trying to operate a cluster.
Imagine a worker node suddenly becoming unavailable because of a host-level configuration issue. Kubernetes would report a node condition, but that alone wouldn't tell me what caused it (Kubernetes, no date h).
I wanted to avoid situations like that.
2.2. Making Swap Configuration Persistent
One example was disabling swap. For this walkthrough, I'm using a swap-disabled host as the baseline. Kubernetes can support swap when it is explicitly configured, so this is an example configuration rather than a rule that swap must always be disabled (Kubernetes, no date k).
I could run:
sudo swapoff -a
This disables swap immediately, but it doesn't necessarily prevent swap from being enabled again after a reboot (util-linux, no date).
So I also checked the current swap configuration:
swapon --show
grep -nE '^[^#].*\bswap\b' /etc/fstab
The first command shows active swap devices. The second checks for active swap entries in /etc/fstab. The swapon and swapoff documentation explains how swap entries in this file are used (util-linux, no date).
If an entry exists, I need to review it and make the appropriate change rather than assuming that disabling swap once is enough.
This is a small example, but it illustrates the difference between changing the current state of a machine and configuring it so that the intended state persists.
I applied the same general approach to other host-level requirements, including kernel modules, network configuration, and container runtime settings.
2.3. Checking Host Availability and Power Management
I also wanted to understand what could affect host availability outside Kubernetes.
For Linux systems using systemd, the login manager includes power-related settings. For example, IdleAction= and IdleActionSec= can control idle-triggered behavior when sessions correctly report their idle state (systemd, no date).
But that doesn't mean changing a single systemd setting makes a host reliable. I still need to consider how the system behaves after a restart, how network services come back, and whether the intended configuration is still there.
The point was to make the underlying machines as predictable as possible before moving on to the Kubernetes installation.
3. Containerd, CRI, and CNI: Understanding What Was Actually Broken
After preparing the machines, I moved on to the container runtime.
One problem I came across while learning the setup was trying to inspect the runtime with crictl, only to find that the command wasn't installed.
At first glance, this might look like a container runtime problem. However, a missing diagnostic tool doesn't automatically mean that the underlying service is broken. Kubernetes documents crictl as a separate command-line tool for inspecting CRI-compatible runtimes (Kubernetes, no date d).
I needed to distinguish between three different components:
crictl -> Diagnostic client for the CRI
CRI -> Interface Kubernetes uses to communicate with the runtime
containerd -> Container runtime
CNI -> Configuration and plugins responsible for Pod networking
These components are related, but they don't serve the same purpose. The Kubernetes runtime documentation distinguishes the CRI from the network configuration that a runtime uses to support Pods (Kubernetes, no date a).
If crictl isn't installed, I need to install the diagnostic tool first. If it is installed but cannot communicate with containerd, I need to investigate the runtime endpoint and socket. If the runtime works but Pod networking isn't configured yet, I need to investigate the CNI separately.
This distinction helped me avoid assuming that every problem during Kubernetes installation had the same cause.
I used the following commands to inspect the runtime:
# Check whether containerd is running
systemctl is-active containerd
# Check whether the containerd socket exists
sudo ls -l /run/containerd/containerd.sock
# Check whether crictl is installed
command -v crictl || echo 'crictl not installed'
After installing crictl from a trusted, version-appropriate source and configuring the correct endpoint, I could inspect the CRI (Kubernetes, no date d):
sudo crictl \
--runtime-endpoint unix:///run/containerd/containerd.sock \
info
The last command assumes that crictl has already been installed and configured appropriately.
Another thing I needed to consider was the cgroup driver.
Kubernetes and containerd need compatible cgroup configurations. On a systemd-based Linux distribution, using the systemd cgroup driver is the usual approach (Kubernetes, no date a).
If the configurations don't match, the cluster can encounter problems that aren't necessarily obvious from simply checking whether the containerd service is running.
The official Kubernetes documentation explains this relationship in more detail (Kubernetes, no date a).
For me, the important part was understanding the order in which these components depend on one another.
Linux Configuration
|
v
Containerd and CRI
|
v
Kubelet and kubeadm
|
v
Kubernetes Control Plane
|
v
CNI Installation
|
v
Pod Networking Tests
I didn't want to jump straight to debugging Pod networking if the container runtime wasn't working correctly in the first place.
Getting the order right made the troubleshooting process much easier to reason about.
4. Multi-Interface Networking: Don't Let Address Selection Be Accidental
The networking decision was more interesting.
Systems with more than one network interface can make node address selection ambiguous. A host might have physical and virtual interfaces, and a component may select an address without knowing which network is intended for cluster communication (Tigera, no date a).
There's a difference between a host having several valid IP addresses and Kubernetes consistently using the address intended for cluster communication.
4.1. Making the Node's InternalIP Part of the Design
One approach I wanted to understand was making the intended node address explicit at the kubelet layer, then letting Calico use the Kubernetes node's InternalIP. Kubernetes provides the --node-ip option, and Calico documents its NodeInternalIP autodetection method (Kubernetes, no date f; Tigera, no date a).
The relationship looks like this:
Selected node network address
|
v
kubelet --node-ip
|
v
Kubernetes NodeInternalIP
|
v
Calico node autodetection
|
v
Pod traffic
I can inspect the addresses Kubernetes has registered with:
kubectl get nodes \
-o custom-columns='NAME:.metadata.name,INTERNAL_IP:.status.addresses[?(@.type=="InternalIP")].address'
Calico's operator configuration can express the address-selection policy like this:
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
calicoNetwork:
nodeAddressAutodetectionV4:
kubernetes: NodeInternalIP
This is just a configuration snippet. It needs to be merged with the rest of the installation settings.
Calico also supports other address-selection strategies, including interface names, interface exclusions, and CIDR-based selection. For a system where more than one address could be selected, using the Kubernetes InternalIP can make the node-address configuration easier to reason about (Tigera, no date a).
4.2. Why Full VXLAN Can Make Sense for an Initial Platform
I also wanted to understand why someone might choose full VXLAN encapsulation instead of cross-subnet-only VXLAN for an initial network model.
That doesn't mean VXLANCrossSubnet is wrong. It's a valid mode, and avoiding unnecessary encapsulation can improve efficiency in the right network topology. In fact, Tigera recommends cross-subnet encapsulation where suitable because it reduces unnecessary overhead (Tigera, no date b).
The trade-off looks like this:
| Mode | Behavior | Consideration |
|---|---|---|
| VXLAN | Encapsulates inter-node workload traffic using VXLAN | Consistent overlay behavior, with encapsulation overhead |
| VXLANCrossSubnet | Encapsulates traffic when it crosses node subnet boundaries | Can reduce overhead; behavior depends on subnet topology |
For a small non-production setup, I can see why someone might prefer a consistent overlay model at first. It's easier to reason about, even if it introduces encapsulation overhead. And if there is no need to advertise routes externally, Calico can use VXLAN without BGP for internal cluster routing (Tigera, no date b).
That doesn't make full VXLAN the best choice everywhere. I would still compare it against cross-subnet mode based on how the network actually behaves.
For example, an illustrative Calico configuration fragment could look like this:
# Fragment of the Calico operator Installation resource
spec:
calicoNetwork:
bgp: Disabled
ipPools:
- cidr: <POD_CIDR>
encapsulation: VXLAN
natOutgoing: Enabled
<POD_CIDR> is a placeholder. Choose a Pod CIDR that doesn't overlap with the networks already in use, and check the complete operator API for your installed Calico version.
The two VXLAN modes and their trade-offs are documented in Tigera's public overlay-networking guide (Tigera, no date b). The example above is not copied from a running cluster.
5. Validation: What a Green Node Status Doesn't Prove
Once the control plane and workers show up in Kubernetes, I still need to separate a few different levels of validation.
5.1. Level One: The Node Has Joined the Cluster
I'd start with the usual checks:
kubectl get nodes -o wide
kubectl get pods -A -o wide
These commands show node status and where system Pods are running. They're useful, but Ready doesn't prove that every workload path, DNS query, or application requirement is working (Kubernetes, no date h).
5.2. Level Two: A Workload Can Use Cluster Networking
For a disposable smoke test, I would validate service discovery separately from node readiness. Kubernetes also documents testing DNS resolution from inside a Pod when diagnosing cluster DNS problems (Kubernetes, no date c).
First, create a namespace and a small Deployment with two replicas:
kubectl create namespace network-smoke
kubectl create deployment web-smoke \
-n network-smoke \
--image=nginx:1.27 \
--replicas=2
kubectl expose deployment web-smoke \
-n network-smoke \
--port=80 \
--target-port=80 \
--type=ClusterIP
kubectl rollout status deployment/web-smoke -n network-smoke
Then create a temporary Pod for DNS and HTTP checks:
kubectl run dns-smoke \
-n network-smoke \
--image=busybox:1.36 \
--restart=Never \
--command -- sleep 3600
kubectl wait --for=condition=Ready pod/dns-smoke \
-n network-smoke --timeout=120s
kubectl exec -n network-smoke dns-smoke -- \
nslookup web-smoke.network-smoke.svc.cluster.local
kubectl exec -n network-smoke dns-smoke -- \
wget -qO- http://web-smoke.network-smoke.svc.cluster.local
After checking the results, remove the temporary resources:
kubectl delete namespace network-smoke
These commands are an example smoke-test procedure, not a captured transcript from my cluster. They assume that the CNI and DNS are working and that the cluster can access the specified container images.
A successful test means the temporary Pod can resolve the Service and receive a response from the web application. That still doesn't prove resilience. To test scheduling and failure behavior, I would also need to check where the replicas land and what happens when a worker becomes unavailable.
5.3. How I'd Diagnose a Failed Smoke Test
The useful part of a smoke test isn't just seeing a successful response. If it fails, I need to know which layer to investigate next instead of immediately changing random configuration.
| Symptom | First thing I'd investigate | Evidence to collect |
|---|---|---|
Pod never becomes Ready
|
Image pull, scheduling, CNI, or startup |
kubectl describe pod, events, assigned node |
nslookup can't resolve the Service |
DNS configuration or reachability | CoreDNS Pods, Pod resolver configuration, DNS-related policies |
| DNS resolves but the Service times out | Missing endpoints or a blocked network path | Service, EndpointSlices, Pod readiness, NetworkPolicies |
| Direct Pod connectivity fails across nodes | Underlay reachability, Calico state, or encapsulation | Node InternalIP values, routes, Calico health, firewall rules |
| Service works inside the cluster but not externally | Exposure, routing, or TLS | Gateway status, HTTPRoute status, certificates, edge connectivity |
For example, a Service that doesn't respond isn't automatically a Calico problem. I'd first check the Service and its endpoints:
kubectl get service web-smoke -n network-smoke -o wide
kubectl get endpointslice -n network-smoke \
-l kubernetes.io/service-name=web-smoke
kubectl get pods -n network-smoke -o wide
If there are no ready endpoints, the Service might not have anything healthy to forward traffic to. A successful DNS lookup doesn't change that. EndpointSlices record backend endpoints and their readiness conditions (Kubernetes, no date e).
If the EndpointSlices contain healthy endpoints, I can move on to connectivity and policy instead of changing the Deployment without evidence. NetworkPolicy enforcement also depends on having a network plugin that implements it (Kubernetes, no date g).
This is the approach I want to use throughout the series: establish a baseline, isolate a layer, change one thing, and repeat the same test.
5.4. Level Three: The Platform Can Support Normal Operations
This is where the rest of the platform work comes in.
| Capability | Question I want to answer |
|---|---|
| NetworkPolicy | Can I restrict traffic that shouldn't be allowed? (Kubernetes, no date g) |
| Shared gateway | Can multiple applications use a common entry point without exposing every application separately? (Gateway API, no date) |
| GitOps | Can the cluster reconcile its declared state without manually applying every change? (Argo CD, no date) |
| Monitoring | Can I understand node and workload behavior? |
| Alerting | Will important conditions reach an operator without someone constantly watching dashboards? |
| Persistent storage | Does application data survive the failures it's designed to survive? (Kubernetes, no date i) |
| Recovery | Have I actually restored a backup and verified the result? (Microsoft, no date) |
I didn't need to solve all of these during the initial installation. But I needed to include them in my definition of what a usable platform would eventually look like.
6. Deliberate Omissions Are Architecture Decisions Too
There were also things I didn't want to add just to make the architecture diagram look more enterprise-ready. Hey, this is a non-production ;)
For example:
- I wouldn't say PostgreSQL is highly available just because Kubernetes can restart its Pod on another worker. A StatefulSet helps manage stable identities and storage, but that doesn't automatically give a database high availability (Kubernetes, no date j).
- I wouldn't say storage is highly available just because a PersistentVolume retains data. Persistence and availability are different things, and I'd still need to think about failure domains (Kubernetes, no date i; Microsoft, no date).
- I wouldn't say an RTO has been achieved before completing a timed restore test. Recovery targets need to be measured against actual restore results (Microsoft, no date).
- I wouldn't introduce every observability component before establishing the basics: metrics, logs, and actionable alerts.
This isn't an argument against highly available databases, distributed storage, or distributed tracing. Those are useful tools when the requirements justify their cost and complexity.
For a small non-production environment, I'd rather understand exactly what a single-instance system can recover from than describe it as highly available simply because Kubernetes can restart containers.
7. What I Learned From This First Phase
The biggest change in my approach was to stop treating deployment as proof that something works.
A diagnostic binary being missing doesn't mean the runtime is broken.
A node having an IP address doesn't mean Kubernetes is using the right one.
Installing a CNI doesn't prove that DNS and cross-Pod communication work.
And a cluster reporting Ready doesn't prove that its security boundaries, delivery process, monitoring, or recovery procedures work.
This is what made the project more interesting to me than following a standard installation guide. Each layer required me to decide what I could trust, what I needed to test, and what could wait until later.
In the next article, I'll go deeper into the networking decisions: what changes when Kubernetes nodes have multiple interfaces, how Calico selects addresses, and why the overlay mode matters.
Until then, my working rule is:
Don't move to the next layer until I understand what the current layer has actually proved.
References
Argo CD (no date) Automated sync policy. [Online]. [Accessed 12 October 2026]. Available from: https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/
Gateway API (no date) HTTPRoute. [Online]. [Accessed 12 October 2026]. Available from: https://gateway-api.sigs.k8s.io/reference/api-types/httproute/
Google Cloud (no date) GKE cluster architecture. [Online]. [Accessed 12 October 2026]. Available from: https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture
Kubernetes (no date a) Container runtimes. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/setup/production-environment/container-runtimes/
Kubernetes (no date b) Creating a cluster with kubeadm. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
Kubernetes (no date c) Debugging DNS resolution. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
Kubernetes (no date d) Debugging Kubernetes nodes with crictl. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/
Kubernetes (no date e) EndpointSlices. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
Kubernetes (no date f) kubelet. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
Kubernetes (no date g) Network policies. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Kubernetes (no date h) Node status. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/reference/node/node-status/
Kubernetes (no date i) Persistent volumes. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
Kubernetes (no date j) StatefulSets. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
Kubernetes (no date k) Swap memory management. [Online]. [Accessed 12 October 2026]. Available from: https://kubernetes.io/docs/concepts/cluster-administration/swap-memory-management/
Microsoft (no date) Architecture strategies for reliability testing. [Online]. [Accessed 12 October 2026]. Available from: https://learn.microsoft.com/en-us/azure/well-architected/reliability/reliability-test
systemd (no date) logind.conf(5) — Linux manual page. Available at: https://man7.org/linux/man-pages/man5/logind.conf.5.html (Accessed: 12 October 2026).
Tigera (no date a) Configure IP autodetection. [Online]. [Accessed 12 October 2026]. Available from: https://docs.tigera.io/calico/latest/networking/ipam/ip-autodetection
Tigera (no date b) Overlay networking. [Online]. [Accessed 12 October 2026]. Available from: https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip
util-linux (no date) swapon(8) — Linux manual page. [Online]. [Accessed 12 October 2026]. Available from: https://man7.org/linux/man-pages/man8/swapoff.8.html
Top comments (0)
Some comments may only be visible to logged-in visitors. Sign in to view all comments. Some comments have been hidden by the post's author - find out more