Breaking Things (Slightly) with a Nexus Upgrade: A Retrospective

Xavier Alexander — Mon, 03 Mar 2025 17:20:40 +0000

Pre-Upgrade

A few weeks ago, I upgraded Nexus Repository Manager at my job. Nexus acts as a central hub for storing and distributing binaries, serving as a proxy for remote repositories, and hosting private artifacts. It sits behind Traefik, both of which are defined in a Docker Compose file. Since Traefik also needed an update, I figured I’d kill two birds with one stone 😅. I wasn’t nervous about the Nexus upgrade—it was a minor version jump. But Traefik? That made me nervous because it was a big jump.

The Upgrade

After reviewing the release notes, I felt confident. I bumped the image tags and restarted the containers. Everything seemed to be going well. I could access the Nexus UI and successfully push and pull images. I went to bed feeling extremely accomplished, as this was my first-ever production upgrade.

The Aftermath

The next morning, I logged in and went through my normal tasks—checking emails, pull requests, and double-checking that I could still push and pull images from Nexus. Everything seemed fine… until I got a message from someone in IT:

“Hey man. Sorry to be the bearer of bad news, but Nexus is mega-bricked.”

My heart sank. This issue affected a high-priority team—we’ll call them Team Alpha. Minutes later, a ticket came in with more details: some of their Jenkins jobs were returning a 502 error when attempting to push artifacts to Nexus. Oddly, other jobs were successfully pushing artifacts to Nexus.

After a few hours of digging and troubleshooting, I got another ticket—this time from Team Alpha’s team lead. Again, my heart sank. Now they were requesting an elevated priority for the ticket. I reached out to their team lead, who mentioned that this same issue happened a few years ago, before I joined the company:

“I can’t remember exactly how you guys fixed it, but it was something with the proxy.”

To me, this was good news—if it happened before, that meant it had already been fixed once. Though I didn’t know how, just knowing there was a solution was a step in the right direction.

He also mentioned that the 502 timeout only happened when pushing really large images—the kind that take some time to upload. And it was failing at exactly 60 seconds. That immediately made me think of Traefik. I went back through the release notes, and then it hit me right in the face:

Starting with v2.11.2, the entryPoints.readTimeout option default value changed to 60 seconds.

The pushes of the large files were timing out because they were exceeding 60 seconds, the default value for readTimeout. This also explains why other jobs , pushing smaller files to Nexus, were succeeding. I let Team Alpha’s lead know, and he gave me the go-ahead to update the Docker Compose file and restart Traefik.

After what felt like hours of waiting, he responded:

“Issue fixed!”

🎉🎉🎉

Lessons Learned

Don’t wait until the night of an upgrade to review release notes. If I had started going over the relase notes a week earlier, I would’ve caught the readTimeout change.
Avoid big gaps between upgrades if possible. Jumping from version 2.0.0 → 2.0.5 is much easier than 2.0.0 → 5.1.0. 2.Expand the scope of the post upgrade testing and validating. Ok it works for my team, but what about other teams who use it?
Really, really read every change in the release notes. (Yes, every change.)

Setting Up A 3-Node Kubernetes Cluster

Xavier Alexander — Thu, 06 Feb 2025 22:46:45 +0000

Six months ago, I started a new job as a DevSecOps engineer. During the interview process the team gave me a heads up about what tools they used. I was excited to learn Kubernetes was one of them. I thought to myself, "Cool, I'll be ready. I've done a few tutorials on this." But after the first few days,I quickly realized I was wrong—I was not ready 😂. I wasn’t just dealing with single-container pods like in the tutorials. I was now faced with terms like Ingress, CRDs, and PVCs… none of it made sense. The days of simple walkthroughs were long gone. I needed a homelab to start building for real. I ordered the following beelink, installed proxmox, and hit the ground running.

In this post I'll be walking through how I set up a 3 node kubernetes cluster

Requirements

3 VMs
2 GB or more of RAM per machine
2 CPUs or more for control plane machines.
OS - I'm using Rocky Linux

On Every Node

The following should be completed on every node.

Disable Swap

Explanation:

The default behavior of a kubelet is to fail to start if swap memory is detected on a node. This means that swap should either be disabled or tolerated by kubelet.

In /etc/fstab comment out the line that declares swap.

#/dev/mapper/rl-swap     none                    swap    defaults        0 0

Confirm by running the free command. You should see 0s for the Swap row.

[xavier@lab-cp ~]$ free -m
               total        used        free      shared  buff/cache   available
Mem:            3562        1725         769          17        1316        1836
Swap:              0           0           0

Install Containerd

Explanation:

To run containers in pods, we need a container runtime. I went with containerd.

# This command adds the docker repoistory 
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# This command installs containerd
sudo dnf install containerd

Ensure containerd is healthy:

[xavier@lab-cp ~]$ systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; preset: disabled)
     Active: active (running) since Tue 2025-01-21 18:32:57 EST; 2 weeks 1 day ago
       Docs: https://containerd.io

Install kubelet, kubeadm, and kubectl

Explanation

kubelet: The primary node agent that runs on each node, ensuring containers are running.
kubeadm: A tool that helps bootstrap and manage Kubernetes clusters.
kubectl: The command-line tool used to interact with the Kubernetes API and manage cluster resources.

# This overwrites any existing configuration in /etc/yum.repos.d/kubernetes.repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

#--disableexcludes=kubernetes ensures that no repository exclusions prevent the installation of these packages.
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Configure cgroup Driver

Explanation:

The kubelet and the underlying container runtime need to interface with cgroups(coontrol groups) to enforce resource management for pods and containers which includes cpu/memory requests and limits for containerized workloads. The cgroup (control group) driver is responsible for managing and allocating system resources such as CPU and memory to containers.

# This command moves (renames) the existing config.toml file to config.toml.bak as a backup.
$ sudo mv /etc/containerd/config.toml /etc/containerd/config.toml.bak

# This command generates the default configuration for containerd, a container runtime, and writes it to a new config.toml file.
$ containerd config default > config.toml

In the config.toml file, set SystemdCgroup to true

Network Configuration

Explanation:

When setting up a Kubernetes cluster, we need to adjust certain kernel parameters to ensure that networking functions correctly. These settings control how the Linux kernel handles network traffic, particularly when dealing with bridged network interfaces and packet forwarding.

Edit the /etc/sysctl.d/k8s.conf file.

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

On The Control Plane Only

The following should be completed on the control plane, only.

Open The Required Ports

Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	6443	Kubernetes API server	All
TCP	Inbound	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	10259	kube-scheduler	Self
TCP	Inbound	10257	kube-controller-manager	Self

Initialize The Cluster

Explanation:

kubeadm init sets up the control plane, configuring essential components like the API server, scheduler, and controller manager.

sudo kubeadm init --apiserver-advertise-address {HOST IP} --pod-network-cidr 10.244.0.0/16

If you're successful, you'll get something like :

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join {HOST IP}:6443 --token {TOKEN} \
    --discovery-token-ca-cert-hash sha256:{HASH}

Save this output somewhere. You'll need it shortly.

Set Up KUBECONFIG

Explanation:

A kubeconfig file is a configuration file used by Kubernetes to store information about clusters, users, namespaces, and authentication mechanisms, allowing the kubectl command-line tool to connect and interact with a Kubernetes cluster by providing necessary credentials and access details to the cluster's API server; essentially, it acts as a central hub to manage access to multiple Kubernetes clusters from a single location.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Deploy A CNI

Explanation:

In order for your pods to communicate with each other, you need to install a Container Network Interface(CNI)

There are a ton of network plugins to choose from. I ended up going with Cilium. Installation steps can be found here.

On The Worker Nodes Only

The following should be completed on the worker nodes only.

Open The Required Ports

Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	10256	kube-proxy	Self, Load balancers
TCP	Inbound	30000-32767	NodePort Services†	All

Now you can join worker nodes to the control plane. From the output you saved earlier, as sudo, run the kubeadm join command.

kubeadm join {HOST IP}:6443 --token {TOKEN} --discovery-token-ca-cert-hash sha256:{HASH}

This will take a few minutes, but if it was successful, you'll get a message with the following. This node has joined the cluster

Conclusion

If all went well, you should now have a 3 node cluster! To verify, create a pod

[xavier@lab-cp ~]$ kubectl run mypod --image=nginx:latest
pod/mypod created
[xavier@lab-cp ~]$ k get pods
NAME            READY   STATUS    RESTARTS   AGE
mypod           1/1     Running   0          36s

There are much easier ways to set up a cluster, but using kubeadm gives you a deeper understanding of how Kubernetes works under the hood.

This method also provides more flexibility, allowing you to customize your setup based on your needs. Whether you’re setting up a test environment or preparing for a production deployment, mastering the fundamentals will make troubleshooting and scaling your cluster much easier.

DEV Community: Xavier Alexander

Breaking Things (Slightly) with a Nexus Upgrade: A Retrospective

Pre-Upgrade

The Upgrade

The Aftermath

Lessons Learned

Setting Up A 3-Node Kubernetes Cluster

Requirements

On Every Node

Disable Swap

Explanation:

Install Containerd

Explanation:

Install kubelet, kubeadm, and kubectl

Explanation

Configure cgroup Driver

Explanation:

Network Configuration

Explanation:

On The Control Plane Only

Open The Required Ports

Initialize The Cluster

Explanation:

Set Up KUBECONFIG

Explanation:

Deploy A CNI

Explanation:

On The Worker Nodes Only

Open The Required Ports

Conclusion