DEV Community: Sacha Thommet

Kubernetes homelab - Learning by doing, Part 6: Automation

Sacha Thommet — Sat, 09 Nov 2024 13:06:17 +0000

In this section, I'll dive into how I automated my Kubernetes cluster, using two tools: Ansible for machine configuration and ArgoCD for application deployments.

Why automate ?

1. Reducing Human Error

In IT operations, even small mistakes can lead to service outages or security vulnerabilities. Human errors like typos may happen, that's why automation is important — it ensures that each task is executed consistently.

2. Minimizing Repetitive Tasks

Automation reduces repetitive, time-consuming tasks like updates and patching, this let teams to work on more strategic tasks.

3. Scalability and Reproducibility

Automating infrastructure setup enables large-scale deployments with consistent configurations, regardless of the size of the environment.

4. Version Control

Tools like Ansible use declarative files that can be tracked in Git, allowing quick rollbacks and maintaining an accessible record of changes.

5. Documentation

Declaratives files are easy to understand — they describe WHAT the infrastructure should look like rather than HOW, like in the imperative way. Being versioned in Git, these files are accessible to the team, allowing them to quickly review the current state of the infrastructure.

Infrastructure as Code (IaS) with Ansible

Ansible is an open-source tool that excels in infrastructure configuration. With an agentless architecture (no services need to be installed on the managed machines), it communicates with machines over SSH.

The inventory file is where you list the machines that Ansible will manage. It contains the IP addresses or hostnames of each machine. Here's mine:

inventory.yaml

k8s:
  hosts:
    server1:
    server2:

Then, the tasks performed by Ansible are defined in files called playbooks.

playbook.yaml

- name: Ensure useful packages are present
  hosts: k8s
  gather_facts: false # don't gather information on nodes as I don't use them
  become: true
  tasks:
    - name: Ensure all apt packages are updated to their latest version
      ansible.builtin.apt:
        update_cache: true      # run the equivalent of apt-get update
        cache_valid_time: 86400 # in seconds: one day
        upgrade: yes            # run apt-get upgrade

    - name: TLP - Optimize Linux Laptop Battery Life - https://linrunner.de/tlp/
      ansible.builtin.apt:
        name: tlp
        state: present

- name: Enforce security
  hosts: k8s
  gather_facts: false
  become: true
  tasks:
    - name: Ensure firewall is configured and running
      block:
        - name: Ensure firewall package is installed
          ansible.builtin.apt:
            name: ufw
            state: present

        - name: Allow incoming HTTPS traffic
          community.general.ufw:
            rule: allow
            comment: Allow incoming HTTPS traffic
            protocol: tcp
            port: 443

        - name: Allow everything from LAN
          community.general.ufw:
            rule: allow
            comment: Allow everything from LAN
            protocol: any
            from_ip: 192.168.1.0/24

        ...

        - name: Ensure firewall is up and running
          community.general.ufw:
            state: enabled

- name: Ensure microk8s is up and running
  hosts: k8s
  roles:
    - role: 'istvano.microk8s'
      vars:
        microk8s_version: 1.29/stable
        ...

With that, I only need to run the following command to configure my nodes:

# run only once to install the required role used by the playbook
ansible-galaxy role install istvano.microk8s

ansible-playbook playbook.yaml -i inventory.yaml

Ansible is idempotent

Idempotence is originally a concept from Mathematics. As defined by Wikipedia,

Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

In Ansible, idempotence is offered as a built-in feature of many of the Ansible modules (be careful, some modules like shell or command don't support this feature). This means that re-running a playbook will produce the same final state without any unwanted side effects.

See this blog post for more explanation.

GitOps with ArgoCD

While Ansible manages the infrastructure, ArgoCD focuses on deploying and updating applications in a Kubernetes environment.
ArgoCD automatically synchronizes applications with their configurations defined in a Git repository. When the repository is updated, ArgoCD adjusts the state of the applications in the Kubernetes cluster to match the repository's configuration. This provides a declarative and versioned approach for deploying and managing Kubernetes applications.

Note that this diagram explains the pull-based approach, where ArgoCD regularly pulls the latest changes from the Git repository. You can also use the push-based approach with ArgoCD, where Git notifies ArgoCD via a webhook that there is a change.

Additionally, ArgoCD comes with a web interface that provides a clear view of deployment statuses. The screenshot below illustrates this interface.

With these tools, almost everything stays in my Git repository (which I plan to make public soon, once I’ve cleared the repo of hard-coded secrets!). Git is the single source of truth.

Kubernetes homelab - Learning by doing, Part 5: Monitoring

Sacha Thommet — Fri, 08 Nov 2024 16:03:38 +0000

Keeping a close eye on your Kubernetes cluster is essential to detect issues early. In this section, I'll explain how I monitor my cluster.

A ridiculously over-engineered setup for a homelab

At this point, I know I'm over-engineering the cluster; I'm the only one using it, and my portfolio hosted on it gets minimal traffic, 5 visits per month at most (measured by Google Search Console). So, if it goes down, no one will be impacted.

The reason I am doing all this is to learn things. I LEARNED A LOT (and invested a lot of time too)…

I chose to deploy Prometheus with Kibana using the Kubernetes prometheus operator which greatly simplifies the deployment process. This stack is widely used in the professional world, so I wanted to explore it.

Prometheus gather data from nodes and pods.
Then, Grafana displays the data.
I could also set up AlertManager to configure alerts via email, webhooks, SMS or whatever, but I haven't gone that far yet.

Prometheus operator

The official documentation offers three ways to install the Prometheus Operator. I use a GitOps approach with ArgoCD to deploy everything to the cluster and chose the Helm chart for installation.

Chart.yaml

apiVersion: v2
name: prometheus-subchart
type: application
version: 60.3.0
appVersion: "60.3.0"
dependencies:
  - name: kube-prometheus-stack
    version: 60.3.0
    repository: https://prometheus-community.github.io/helm-charts

values.yaml

kube-prometheus-stack:
  namespaceOverride: prometheus-stack
  defaultRules:
    rules:
      alertmanager: false
      etcd: false    # microk8s does not use etcd if HA is enabled
      windows: false
  alertmanager:
    enabled: false
  prometheus:
    prometheusSpec:
      retention: 30d
  grafana:
    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod
      hosts:
        - grafana.mydomain
      tls:
        - secretName: certificate-prod-grafana
          hosts:
            - grafana.mydomain
    sidecar:
      datasources:
        alertmanager:
          enabled: false
  kubeEtcd:
    enabled: false # microk8s does not use etcd if HA is enabled

And voila!

Grafana is accessible through grafana.mydomain with the default credentials admin:prom-operator (change it!):

Multiple dashboards are also configured by default with the operator.

You can now configure metric scraping for your pods and services using PodMonitor and ServiceMonitor CRDs thanks to the operator.

Kubernetes homelab - Learning by doing, Part 4: Storage

Sacha Thommet — Thu, 31 Oct 2024 15:18:52 +0000

Welcome to the fourth part of my Kubernetes homelab guide. I will explain how I manage to set up the storage using Longhorn.

Distributed storage

Pods and their containers are ephemeral. Various factors can lead to a pod restarting, such as being OOM-killed, application crashes, or scaling down...
The problem is, when a pod restarts, its filesystem is wiped, meaning it does not retain data persistently.

So, how can we persist data inside pods, and how can we share data across multiple pods on different nodes ?

With volume mounting and distributed storage!

Distributed storage systems enable us to store data that can be made available clusterwide. Excellent! But dynamically apportioning storage across a multi-node cluster is a very complex job. So this is another area where Kubernetes typically outsources the job to plugins (e.g. Cloud providers like Azure or AWS, or systems like Rook or Longhorn).

External storage systems like these connect to Kubernetes by way of the Container Storage Interface (CSI). This provides a standard interface between different types of storage systems

Why Longhorn ?

I’m already over-engineering by using a Kubernetes cluster to host a few private services , so the main reason I chose Longhorn was its ease of setup and management.

In addition its backup functionality is extremely useful and easy to use!
Simply set up a backup target — an endpoint for Longhorn to access a backup store. This backup store can be an NFS, SMB/CIFS server, Azure Blob Storage, or any S3-compatible server holding Longhorn volume backups. Then, to restore your entire Kubernetes storage, just point Longhorn to the backup target.

Installation

The soft is very easy to install, as always with containers:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/<version>/deploy/longhorn.yaml

The "hard" part is with requirements. Each node in the cluster where Longhorn is installed must fulfill multiple requirements. Luckily, the official documentation is clear and straightforward.

Usage

Longhorn comes with its StorageClass, which provides dynamic provisioning of persistent volumes:

For instance:

apiVersion: v1
kind: Pod
metadata:
  name: vaultwarden
  ...
spec:
  containers:
    - image: vaultwarden/server:latest
      name: vaultwarden
      ...
      volumeMounts:
        - mountPath: /data
          name: vaultwarden-data
  volumes:
    - name: vaultwarden-data
      persistentVolumeClaim:
        claimName: vaultwarden-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vaultwarden-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: longhorn

Kubernetes homelab - Learning by doing, Part 3: Networking

Sacha Thommet — Thu, 31 Oct 2024 11:17:26 +0000

In this part of the Kubernetes homelab, we’ll dive into the networking setup.

Network configuration

My networking implementation is straightforward. All of the cluster nodes, along with the router with a built-in firewall, are on a single /24 private network. This is a standard home setup.

I set up my router's DHCP to assign static IPs to servers 1 and 2 by mapping them to their respective MAC addresses.

server1: 192.168.1.11
server2: 192.168.1.10
router: 192.168.1.254

Kubernetes internal networking

In order to expose your applications, you'll need an Ingress Controller. This runs on every node in the cluster and listens on ports 80 and 443 (HTTP and HTTPS). I choose the Nginx Ingress Controller, which is easy to install on Microk8s:

microk8s enable ingress

Then, I configured the router to forward incoming requests on ports 80 and 443 to any one of the nodes, in my case server2.
All other ports are blocked by the router’s firewall, ensuring that only necessary traffic reaches the servers.

server2 will handle all ingress traffic, and use the Calico network plugin to route the requests to the pods on the corresponding nodes.

I chose Calico for its support to NetworkPolicies, but Kubernetes allows you to use other Container Network Interfaces (CNIs) that may better suit your setup.

Note:
This means that if server2 is unavailable for some reason, the cluster will not respond to any incoming requests. It is a Single Point Of Failure.
One solution would be to use an IP failover mecanism like keepalived.

Finally, I also installed Cert Manager, to handle SSL certificate requests for my HTTPS routes and automatically manage renewals.

Installing it is a simple as:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/<version>/cert-manager.yaml

With this setup, I simply create an Ingress, then the NGINX Ingress Controller along with Cert Manager takes care of the rest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: portfolio-ingress
  namespace: portfolio
spec:
  ingressClassName: nginx
  rules:
    - host: mydomain
      http:
        paths:
          - backend:
              service:
                name: portfolio
                port:
                  number: 3000
            path: /
            pathType: Prefix
  tls:
    - hosts:
        - mydomain
      secretName: certificate-prod-portfolio

Kubernetes homelab - Learning by doing, Part 2: Installation

Sacha Thommet — Thu, 17 Oct 2024 13:35:26 +0000

In this part, I'll walk you through the (few) installation steps for the cluster.

Operating system

I opted for Ubuntu Server since I'm already familiar with Ubuntu because I'm using it on my desktop computer.

Maybe in the future I will try others systems, like Talos which is designed for Kubernetes - secure, immutable, and minimal.

Kubernetes Distribution

Given that MicroK8s is developed by Canonical, the same team behind Ubuntu, I naturally choose it.

It supports all the features that I need:

It can run on a single node without requiring high availability (HA).
Yet it also supports HA, I can add a third node in the future.
Easy to install & to update.

Installation

Installing MicroK8s was super simple by following the official guide available here.

Prerequisites

As with any installation of this kind, the first step is to ensure your system meets the requirements.

At least Ubuntu 16.04 LTS, or newer to run the commands (or another operating system which supports snapd - see the snapd documentation).
MicroK8s runs in as little as 540MB of memory, but to accommodate workloads, Canonical recommend a system with at least 20G of disk space and 4G of memory.
An internet connection

Install

Run these commands on every node, in my case server1 and server2.

# install microk8s
sudo snap install microk8s --classic --channel=<version>

# Add your current user to the microk8s group
# and gain access to the .kube directory (where some of the k8s configuration goes on) 
sudo usermod -a -G microk8s $USER
mkdir -p ~/.kube
chmod 0700 ~/.kube

Then, create a cluster by adding server2 as a worker to the server1 master node.

# On the master node (server1)
microk8s add-node

# Then from the node you wish to join to this cluster, run the command displayed by the command above, like:

# Join as a worker, not running the Kubernetes control plane
microk8s join <server1_ip>:25000/92b2db237428470dc4fcfc4ebbd9dc81/2c0cb3284b05 --worker

# Finally, from the master node, run to see the nodes in the cluster:
microk8s kubectl get nodes

And voila, it's that simple!

Kubernetes homelab - Learning by doing, Part 1: Hardware

Sacha Thommet — Tue, 12 Mar 2024 14:15:27 +0000

I recently did a 6-month internship, which was extremely interesting and challenging for me!

My role as a Kubernetes consultant intern was to help make sure that the applications we were working on could be deployed reliably.

Kubernetes is a powerful orchestrator that will ease deployment and automatically manage your applications on a set of machines, called a cluster.

With great power comes great complexity. Thus, learning Kubernetes is oftentimes considered to be cumbersome and complex, namely because of the number of new concepts you have to learn.

Hardware

Originally, I wanted to build a three-node cluster (for High Availability).

Here is a summary of the hardware that I have:

	Laptop	Intel NUC NUC10i3FNK	Raspberry PI 4B	Raspberry PI 0
RAM	16GB DDR4 · 2133Mhz	32GB DDR4 · 3200Mhz	1GB DDR4 · 3200Mhz	512Mb DDR2
Storage	512GB SSD SATA	1To SSD NVMe	64GB MicroSD	16GB MicroSD
CPU	i5-7200U · 2C/4T	I3-10110U · 2C/4T	2C/4T	1C/1T
GPU	Integrated	Integrated	Integrated	Integrated

Issues encountered

First issue: The Raspberry PI 0 is incompatible

Unfortunately, the Raspberry PI 0 is incompatible with Kubernetes, as described in this Github issue.

The Raspberry Pi Zero has a processor that adopts the armv6 architecture. Unfortunately, its support has been dropped by Kubernetes since v1.6.

Second issue: The Raspberry PI 4B does not have enough RAM

With its 1GB RAM, the Raspberry PI 4B was not very useful. The Kubernetes control plane would use the majority of the free RAM, which leaves no room for the workload.

The issues of my hardware forced me to settle for only two nodes instead of the intended three in my High Availability cluster build.

Concerning the network, I have a modem provided by my ISP which includes a 500Mbps switch, a router and a firewall:

Power consumption

I measured the power consumption to estimate the monthly cost of running the homelab:

My router accounts for 12W out of the total 54 Watts, while the two nodes consume 42W when idle.

As of March 2024, the cost of one kWh is 0.25€ in France. Therefore, the operational cost of the homelab is 42*24*30/1000=30kWh * 0.25€ = 7€50 per month.

The Learning Adventure: Tales from a Junior Developer

Sacha Thommet — Sun, 02 Jul 2023 20:41:37 +0000

As a passionate junior developer, I want to share my learning journey in the world of software development. In this blog post, I'll discuss the strategies that have worked well for me and contributed to my growth.

1. Studies

My master's degree in computer science provided a theoretical foundation that I found very useful.

While you may not regularly utilize advanced algorithms, the process of developing logical reasoning skills through their study is invaluable.

I have also gained soft skills that are difficult to acquire through self-learning. Skills such as effective communication, teamwork, and time management are really useful.

But the most useful skill I've learned is the ability to learn new things and to do research by using the internet (RTFM) and seeking guidance from experienced individuals.

2. Online tutorials, documentation and whatever.

While the studies gives you the fundamentals, you will need to learn more by yourself by looking for tutorials on the internet.

The countless hours spent searching computer-related content on Google even resulted in an unexpected invitation to partake in the intriguing Google Foobar recruitment easter egg.

Google send the invitation on the basis of your search history and your problem solving related keyword searches, like if you are a developer, it is obvious that you search a lot of problems related to programming on Google or Stack Overflow. And based on Google search algorithms, they show you an invitation for Google Foobar.

3. Books

So far, I have read two popular books, "Clean Code" and "Clean Architecture" by the same author. These books have been immensely helpful, providing valuable guidance and principles for writing clean and maintainable code.

Despite facing criticism for various reasons, these two books present an opportunity to engage in critical thinking and reflection on various aspects of clean code. I recommend exploring books by different authors to gain diverse perspectives and opinions.

4. Side projects

And the most important one that works great for me: learning by doing mistakes. You should find the answers to every “Why”. Good developers are constantly wondering why issues are solved in one way and not another. This allows us to reflect on the processes and improve them with the passage of experience.

I enjoy working on personal projects, like managing my own virtual private server running on Linux. With this server, I learned so much about networking, system administration and Docker.

Currently, I am focused on learning how to build a web application using microservices architecture. As I strive to apply best programming practices, I find myself conducting extensive research (yes, again research!) to ensure I'm following industry standards.
Some things could be bad, over-engineered, not well understood. However, I try to do my best as a junior and it's totally normal to make mistakes!
Also, I explain in the readme.md file all my choices whenever I thing it is interesting.

In the future, I also plan to participate in Open Source projects.

5. Learn with others

I've recently learned (a bit late) the significance of networking, engaging in discussions with colleagues, asking questions, sharing ideas, and attending technical events/conferences. These experiences have proven to be invaluable for learning new things and connecting with passionate individuals in the field.

For instance, I recently started attending to tech meetups in France and it's a great way to meet people and learn many things!