DEV Community: Ștefănescu Liviu

DevOps and SRE: The Dynamic Duo Transforming the Software Development Landscape

Ștefănescu Liviu — Thu, 13 Apr 2023 08:33:40 +0000

An easy-to-understand introduction to DevOps and Site Reliability Engineering for the general audience

Introduction

In the ever-evolving world of software development, two concepts have emerged as vital components for delivering high-quality, reliable software: DevOps and Site Reliability Engineering (SRE). These approaches have revolutionized the way software is built, deployed, and maintained, and their adoption has led to increased efficiency and collaboration across organizations. In this article, we'll explore the main concepts of DevOps and SRE, and explain their significance in a way that's easy for a general audience to understand.

What is DevOps?

DevOps, a combination of the words "development" and "operations," is a set of practices and cultural philosophies that bridge the gap between software development and IT operations teams. The goal of DevOps is to create a seamless, collaborative environment where developers and operations teams can work together to deliver high-quality software rapidly and reliably.

Key Concepts of DevOps

Collaboration: DevOps encourages increased communication and cooperation between development and operations teams, fostering a shared understanding of goals and breaking down silos.
Continuous Integration (CI): This practice involves regularly merging code changes into a central repository, followed by automated building and testing. CI helps detect integration issues early and speeds up the development process.
Continuous Delivery (CD): CD is the process of automatically deploying code changes to production-like environments after they pass testing, making it easier to release new features and bug fixes quickly.
Infrastructure as Code (IaC): IaC is the management of infrastructure (such as networks, servers, and storage) through code, which allows for version control, easy rollbacks, and collaboration between team members.
Monitoring and Feedback: DevOps emphasizes the importance of monitoring applications and infrastructure to gather insights and feedback, enabling teams to continuously improve processes and address issues proactively.

What is Site Reliability Engineering (SRE)?

Site Reliability Engineering (SRE) is a discipline that combines aspects of software engineering and IT operations to ensure the reliability, availability, and performance of software systems. SREs are responsible for defining service level objectives (SLOs), monitoring system performance, and implementing automated solutions to improve the reliability and efficiency of software systems.

Key Concepts of SRE

Service Level Objectives (SLOs): SLOs are measurable goals that represent the desired level of system reliability, such as uptime, latency, and error rates. SREs work closely with development teams to establish and maintain these objectives.
Error Budgets: An error budget is a defined tolerance for system failures or performance issues. By allocating an error budget, SREs can balance the need for system reliability with the desire to innovate and release new features.
Automation: SREs focus on automating tasks that are repetitive, error-prone, or time-consuming, freeing up resources to work on more valuable tasks and improving overall system reliability.
Monitoring and Observability: SREs use monitoring and observability tools to gain insights into the performance and health of software systems, enabling them to identify potential issues and proactively address them.
Blameless Postmortems: When incidents occur, SREs conduct blameless postmortems to review the event, identify the root cause, and implement improvements to prevent future occurrences, fostering a culture of learning and continuous improvement.

The Benefits of DevOps and SRE

The adoption of DevOps and SRE practices offers numerous advantages, including:

Faster Time-to-Market: By streamlining the development and deployment processes, organizations can bring new features and products to market more quickly.
Improved Collaboration: DevOps and SRE foster better communication and collaboration between development and operations teams, breaking down silos and resulting in more efficient problem-solving and decision-making.
Enhanced Reliability: By focusing on system reliability and implementing automated solutions, SREs can ensure that software systems are more stable, secure, and resilient.
Higher Quality Software: DevOps practices such as CI/CD and automated testing help to catch issues early, leading to higher quality software and fewer defects in production.
Cost Efficiency: By automating tasks and optimizing resource usage, DevOps and SRE can help organizations save time and money, while also reducing the risk of human errors.
Continuous Improvement: Both DevOps and SRE promote a culture of learning, feedback, and continuous improvement, enabling teams to learn from mistakes and proactively address potential issues.

Conclusion

DevOps and Site Reliability Engineering have transformed the way software is developed, deployed, and maintained. By fostering collaboration, streamlining processes, and focusing on reliability, these practices have helped organizations deliver high-quality software more quickly and efficiently. With a better understanding of the main concepts of DevOps and SRE, anyone can appreciate the profound impact these approaches have on the software development landscape.

DevOps & SRE Roadmap explained by AI - part 3

Ștefănescu Liviu — Thu, 06 Apr 2023 16:37:06 +0000

This part is for what you need to know about Cloud Native tools and principles.
Cloud native is essential for DevOps and SRE roles ☁️🌎 It allows you to build and run scalable applications in modern, dynamic environments 🚀 Cloud native technologies are highly customizable and can be used across a wide range of systems 🌐

👷‍♂️👷‍♀️ Infrastructure as code (IaC) is a DevOps methodology that uses versioning with a descriptive model to define and deploy infrastructure such as networks, virtual machines, load balancers, and connection topologies.

👩‍💻👨‍💻 Virtual machines are software that emulate hardware and run operating systems. They are useful for DevOps and SRE roles because they allow testing, deploying and scaling applications in different environments.

🌩️🌐👩‍💻 Cloud Computing Private cloud: dedicated resources for one organization, more control and security, higher cost and maintenance. Public cloud: shared resources for multiple organizations, less control and security, lower cost and maintenance. Choose wisely!

🤖🔧👨‍💻 Automation: using software to perform tasks that are repetitive, error-prone, or time-consuming, such as testing, deployment, monitoring, etc. Configuration management: using tools to manage the state and behavior of systems and applications, such as Ansible, Puppet, Chef, etc. Work smarter, not harder!

🐳🚀👩‍💻 Container Runtime is the software layer that enables containers to run on a host machine. It provides an interface between the container engine and the operating system. It also manages the container lifecycle and resources.

🌐🔌👩‍💻 Modern API Technologies are the tools and methods that enable developers to create, test, document, and deploy APIs. They include frameworks, protocols, standards, and platforms that facilitate API development and integration.

🐙🛠👩‍💻 Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It orchestrates clusters of nodes and pods that run containers across different environments.

🕸🛡👩‍💻 Service Proxy and Service Mesh are technologies that enable communication and security between microservices. A service proxy is a software agent that intercepts and handles network requests. A service mesh is a network of service proxies that manage traffic and policies.

🔄🚀👩‍💻 CI/CD is a set of practices that enable faster and reliable delivery of software. CI stands for continuous integration , which means merging code changes frequently and testing them automatically. CD stands for continuous delivery or deployment , which means releasing software to production with minimal manual intervention.

📺📨👩‍💻 Streaming and Messaging are techniques that enable asynchronous and real-time data processing for app development. Streaming is the continuous ingestion and analysis of data from various sources. Messaging is the exchange of data between applications or services via a broker or a queue.

👀📊👩‍💻 Observability is the ability to monitor and understand the internal state and behavior of a system based on the external outputs. It involves collecting and analyzing metrics, logs, and traces from various components and sources. Observability helps to identify and troubleshoot issues, optimize performance, and improve reliability.

🚀🌩️🔥 Serverless is a cloud computing model that allows developers to run code without provisioning or managing servers. It is ideal for DevOps and SRE roles who want to focus on business logic, scalability and cost-efficiency.

That's all for this part - Cloud Native.
Next part is NON SRE/DEVOPS, a small one so it's below.

NON DevOps/SRE

There is no such a thing as NON-SRE/DevOps part in a SRE/DevOps guide, what I meant is some additional knowledge that you need or don't depending on your situation.
I'm a telecommunications engineer so I know that pretty well. You definitely need to learn some networking.
Extra things that I (and maybe you) know or want to understand better are cryptography, project management (mostly Agile), licencing and important Organisation (like CNCF, FSF, etc.). As I'm a huge Bitcoin only fan I know some things around blockchains too.

DevOps & SRE Roadmap explained by AI - part 2

Ștefănescu Liviu — Wed, 05 Apr 2023 16:33:28 +0000

This part is for what you need to know about Operating Systems.

You will need to understand some OS fundamentals . And if you're in a very rare case of a company that is Windows exclusive you will only need to live in a Linux terminal 99% of your time. So learn only Linux and learn it well.

Linux is a powerful open-source operating system that’s essential for DevOps and SRE roles 🐧🔧 It provides a stable, secure, and customizable platform for software development, deployment, and maintenance 💻 Linux is highly scalable and can be used across a wide range of systems 🚀

Linux commands are essential for DevOps and SRE roles 🔧💻 They allow you to manage hardware and software resources, automate tasks, and deploy software with confidence 🚀 Linux commands are highly customizable and can be used across a wide range of systems 🌎

Bash scripting is essential for DevOps and SRE roles 📜💻 It allows you to automate tasks, manage systems, and deploy software with ease 🚀 Bash scripts are highly customizable and can be used across a wide range of systems 🌎
(lol AI really likes to write can be used across a wide range of systems 🌎)

Essentially you need to know Linux very well!

This is all for Operating Systems.
Next part is the biggest one - Cloud Native.

DevOps & SRE Roadmap explained by AI - part 1

Ștefănescu Liviu — Wed, 05 Apr 2023 16:04:54 +0000

Introduction

(you can view the screenshots in a better resolution on the twitter thread here)

Hey there, budding tech enthusiasts! Welcome to the exciting world of DevOps! In this article, we'll walk you through a roadmap to kickstart your DevOps journey, and with a little help from our AI friend, we'll demystify key concepts and techniques that you'll need along the way. So, buckle up, and let's dive into the realm where development and operations join hands to deliver the best software experience possible!

No one knows everything from a full roadmap including the one made by me. I made it a couple of years ago to start my new #DevOps role and it's a WIP. You only need to understand most of the notions and to really know a few of them.
My roadmap has 4 parts. And every part and sub-part has something below it. The full circles are a must and dotted circles are more important.

Higher resolution here

I'll use:

ChatGPT from OpenAI
Claude from AnthropicAI
Bing Chat from bing
Bard from Google to explain some DevOps and SRE concepts in an easy way and in a more complex one too. And what tools to know for each. So let's start with the first part

DEVELOPMENT

While it's not a bad thing to know a language, it's not a must. You will need how to write some scripts, but now with #GPT4 and #Copilot you can make easy scripts in seconds.

VCS
Version control is a practice of tracking and managing changes to software code that helps high performing development and DevOps teams prosper 🚀👨‍💻👩‍💻. It allows developers to move faster and preserve efficiency as the team scales 📈👥

Deployments are a process of deploying applications into production environments in a consistent and reliable way 🚀👨‍💻👩‍💻. It enables faster development of new products and easier maintenance of existing deployments 📈👥

Architectural patterns are a set of best practices that help you design and build reliable, scalable, and secure applications in the cloud 🌥️🚀 They are essential in a DevOps or SRE role as they enable you to automate multistage DevOps pipelines and achieve continuous delivery 🔁

Python scripting is a powerful tool for DevOps teams 🔧🐍 It’s used for automating repetitive tasks, infrastructure provisioning, and API-driven deployments 🚀 Python’s flexibility and accessibility make it a great fit for this job, enabling teams to build web applications and data visualizations 🌐

That's all for the first part - Development.
Next part is Operating systems.

Kubernetes: The Revolution in Managing Digital Applications Made Simple

Ștefănescu Liviu — Wed, 05 Apr 2023 07:14:53 +0000

A beginner's guide to understanding Kubernetes and its impact on the digital world

Introduction

In the fast-paced world of technology, it's essential to understand the driving forces behind some of the most cutting-edge innovations. One such force, Kubernetes, has revolutionized the way digital applications are managed and deployed. But what exactly is Kubernetes, and how does it benefit both tech giants and small businesses alike? In this article, we'll break down the main concepts of Kubernetes in a way that's easy for a general audience to understand.

What is Kubernetes?

Kubernetes (often abbreviated as K8s) is an open-source platform that automates the deployment, scaling, and management of containerized applications. It was originally developed by Google and later donated to the Cloud Native Computing Foundation (CNCF). Containers, which are lightweight, self-contained software packages, enable developers to build, test, and deploy applications more efficiently and reliably.

Key Concepts of Kubernetes

Containers: As mentioned earlier, containers are the fundamental building blocks of Kubernetes. They package an application and its dependencies into a single unit, ensuring that the application runs consistently across different computing environments.

Nodes: In Kubernetes, a node is a physical or virtual machine that hosts one or more containers. Nodes can be easily added or removed, depending on the required computing resources, making it possible to scale applications quickly and efficiently.

Clusters: A cluster is a group of nodes working together to run containerized applications. Kubernetes uses clusters to distribute workloads evenly, ensuring that applications are highly available and can recover quickly from failures.

Pods: A pod is the smallest and most basic unit in the Kubernetes architecture. It represents a single instance of a running application and can contain one or more containers. Pods are designed to be ephemeral, which means they can be easily replaced if they fail or need to be updated.

Services: A service is a stable network endpoint that provides access to one or more pods running an application. It allows users to interact with the application without needing to know the specific details of the underlying pods or nodes.

Controllers: Controllers are responsible for maintaining the desired state of the Kubernetes system. They continuously monitor the system and make necessary adjustments to ensure that the actual state matches the desired state. Examples of controllers include the ReplicaSet controller, which ensures that a specified number of replicas of an application are running at all times, and the Deployment controller, which manages updates and rollbacks of applications.

The Benefits of Kubernetes

Kubernetes offers numerous advantages to businesses and developers, including:

Scalability: Kubernetes allows applications to scale up or down quickly and easily, depending on demand. This enables businesses to save resources and respond swiftly to changes in the market.

Portability: Because containers can run consistently across different environments, Kubernetes applications can be easily moved between on-premises, public cloud, or hybrid environments without requiring significant changes.

High availability: Kubernetes automatically distributes workloads and ensures that applications remain available even if individual components fail, improving overall reliability and uptime.

Streamlined deployment: Kubernetes simplifies the deployment process, allowing developers to focus on building and improving applications rather than managing complex infrastructure.

Cost efficiency: By optimizing resource usage and reducing manual intervention, Kubernetes can help organizations save both time and money.

Conclusion

Kubernetes has quickly become the industry standard for container orchestration and application management. Its ability to simplify deployment, ensure high availability, and enable seamless scaling has made it an invaluable tool for businesses and developers alike. By understanding the main concepts of Kubernetes, anyone can appreciate the remarkable impact this technology has on the digital world.

K8s cluster with OCI free-tier and Raspberry Pi4 (part 4)

Ștefănescu Liviu — Wed, 15 Feb 2023 19:31:21 +0000

This long read is a multiple part tutorial for building a Kubernetes cluster (using k3s) with 4 x OCI free-tier ARM instances and 4 x Raspberry Pi 4. Plus some applications needed for installation (Terraform and Ansible) and a lot of things installed on the cluster.
Part 4 is adding applications to the cluster.
GitHub repository is here
I'll just install the apps and access them. But you need to configure to use all of them.

Lens

First thing installed is a beautiful dashboard - Lens. Install the desktop app on your PC, go to File > Add Cluster. You have to paste here all that you receive when running kubectl config view --minify --raw on server. Edit 127.0.0.1:6443 from that result with your server IP, in my case 10.20.30.1:6443.

MetalLB

Metal LB will work as our load balancer, it will give an external IP to every service type LoadBalancer. Install it with kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml. Create a file config-metallb.yaml and write this in it:

---
apiVersion: metallb.io/v1beta1

kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
  - 192.168.0.150 - 192.168.0.250

---
apiVersion: metallb.io/v1beta1

kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
  - default

Edit IP range with your values from home network. Apply it with kubectl apply -f config-metallb.yaml. Now for every time in the rest of the guide you acces a service with node ip (10.20.30.1 -10.20.30.8:port) you could only use the external IP given by metallb.

Helm & Arkade

On the server it seems git is not installed. So sudo apt install git first. Plus KUBECONFIG env variable wasn't configured until now. So add this line to ~/.bashrc.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Helm is the package manager for Kubernetes. Installing helm is very easy, just run following commands on server:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Arkade is an Open Source marketplace For Kubernetes. Installation is just curl -sLS https://get.arkade.dev | sudo sh.

Bash completion

To run commands faster and easier you can use auto-completion for kubectl. Run apt install bash-completion and then echo 'source <(kubectl completion bash)' >>~/.bashrc. Now after every kubectl you can hit tab and it will autocomplete for you. Try with a running pod, just write kubectl get pod and hit TAB.

Traefik dashboard

First write a yaml file traefik-crd.yaml and apply it. It should have this content:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--api"
      - "--api.dashboard=true"
      - "--api.insecure=true"
      - "--log.level=DEBUG"
    ports:
      traefik:
        expose: true
    providers:
      kubernetesCRD:
        allowCrossNamespace: true

This should be available at http://10.20.30.4:9000/dashboard/#/ in your browser (or any ip from those 8 that are nodes).

Longhorn

Longhorn is a cloud native distributed block storage for Kubernetes. First create a new install-longhorn.yml file for Ansible playbook. Paste in it :

---
- hosts: all
  tasks:
  - name: Install some package for Longhorns
    apt:
      name:
        - nfs-common
        - open-iscsi
        - util-linux
...

Run it with ansible-playbook install-longhorn.yml -K -b to install some extra components on nodes. Now run ansible -a "lsblk -f" all to find what are the names of the drives.
Move to server. Run helm repo add longhorn https://charts.longhorn.io then helm repo update
and then helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace. It will last a while, ~7 minutes. Now create a longhorn-service.yaml file and paste this:

apiVersion: v1
kind: Service
metadata:
  name: longhorn-ingress-lb
  namespace: longhorn-system
spec:
  selector:
    app: longhorn-ui
  type: LoadBalancer
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http

Run it with kubectl apply -f longhorn-service.yaml. Now you can access the Longhorn Dashboard in your browser in 10.20.30.1:port (or any of your nodes IPs). The port you get it from running bectl describe svc longhorn-ingress-lb -n longhorn-system | grep NodePort
Now to make Longhorn default StorageClass. Run kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' and now you should have only one default in kubectl get storageclass.

Portainer

This app is a container management platform. We'll be installed using again helm. Run helm repo add portainer https://portainer.github.io/k8s/ and then helm repo update.
Now just run helm install --create-namespace -n portainer portainer portainer/portainer. You can acces portainer UI from 10.20.30.1:30777 (or any other IP from your Wireguard network 10.20.30.1 - 10.20.30.8 :30777). It uses 10 GB, check with kubectl get pvc -n portainer (you can check in your Longhorn Dashboard the new volume created). There you will create a user and password. This is what I have after clicking on Get Started:

ArgoCD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
Installation very easy with kubectl create namespace argocd then kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml and wait a few minutes, or check progress with kubectl get all -n argocd. Now to access the UI run kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'. Find your port with kubectl describe service/argocd-server -n argocd | grep NodePort and access the UI from 10.20.30.1:port (or another IP form your network). User is _admin _and password is stored in kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo.

Prometheus & Grafana

Prometheus is probably the best monitoring system out there and Grafana will be used as it's dashboard. Will be installed from the ArgoCD UI using a official helm chart kube-prometheus-stack . Open Applications and click New App. Edit:

App Name : kube-prometheus-stack
Project Name : default
Sync Policy : Automatic
check Auto Create Namespace and Server Side Apply
Repository URL : https://prometheus-community.github.io/helm-charts (select Helm)
Chart : kube-prometheus-stack
Cluster URL : https://kubernetes.default.svc
Namespace : kube-prometheus-stack
alertmanager.service.type : LoadBalancer
prometheus.service.type : LoadBalancer
prometheusOperator.service.type : LoadBalancer
grafana.ingress.enabled : true.

Now hit Create. We configured the services as LoadBalancer, because by default they are ClusterIP and if you wanted to access them you have to do port-forwarding every time. You can acces the Prometheus UI from node:port (port you get it with kubectl describe svc kube-prometheus-stack-prometheus -n kube-prometheus-stack | grep NodePort). The same for Prometheus Alert Manager (get port with kubectl describe svc kube-prometheus-stack-alertmanager -n kube-prometheus-stack | grep NodePort). For Grafana Dashboard you need to go to any nodeIP/kube-prometheus-stack-grafana:80. User is admin and password get it with kubectl get secret --namespace kube-prometheus-stack kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d.

K8s cluster with OCI free-tier and Raspberry Pi4 (part 3)

Ștefănescu Liviu — Wed, 15 Feb 2023 19:30:55 +0000

This long read is a multiple part tutorial for building a Kubernetes cluster (using k3s) with 4 x OCI free-tier ARM instances and 4 x Raspberry Pi 4. Plus some applications needed for installation (Terraform and Ansible) and a lot of things installed on the cluster.
Part 3 is linking the RPi4 to the k3s cluster on OCI.
GitHub repository is here

Preparing

At this moment I added the OCI machines to the C:\Windows\System32\drivers\etc\hosts file (WSL reads this file and updates it in it's /etc/hosts file). Now my hosts looks like this:

...
192.168.0.201   rpi4-1
192.168.0.202   rpi4-2
192.168.0.203   rpi4-3
192.168.0.204   rpi4-4
140.111.111.213 oci1
140.112.112.35  oci2
152.113.113.23  oci3
140.114.114.22  oci4

And I added them to the Ansible file too (/etc/ansible/hosts). Now this file looks like this:

[big]
rpi4-1  ansible_connection=ssh
[small]
rpi4-2  ansible_connection=ssh
rpi4-3  ansible_connection=ssh
rpi4-4  ansible_connection=ssh
[home:children]
big
small
[ocis]
oci1    ansible_connection=ssh ansible_user=ubuntu
[ociw]
oci2   ansible_connection=ssh ansible_user=ubuntu
oci3   ansible_connection=ssh ansible_user=ubuntu
oci4   ansible_connection=ssh ansible_user=ubuntu
[oci:children]
ocis
ociw
[workers:children]
big
small
ociw

It is not the best naming convention, but it works. Ansible reservers the naming all so if I want to interact with all the objects I can always use ansible -m command all. Test it using ansible -a "uname -a" all. You should receive 8 responses with every Linux installed. Now you can even re-run the update ansible playbook created last part, to update OCI instances too.

K3s can work in multiple ways (here), but for our tutorial we picked High Availability with Embedded DB architecture. This one runs etcd instead of the default sqlite3 and so it's important to have an odd number of server nodes (from official documentation: "An etcd cluster needs a majority of nodes, a quorum, to agree on updates to the cluster state. For a cluster with n members, quorum is (n/2)+1.").
Initially this cluster was planned with 3 server nodes, 2 from OCI and 1 from RPi4. But after reading issues 1 and 2 on Github, there are problems with etcd being on server nodes on different networks. So this cluster will have 1 server node (this is how k3s names their master nodes): from OCI and 7 agent nodes (this is how k3s names their worker nodes): 3 from OCI and 4 from RPi4.
First we need to free some ports, so the OCI cluster can communicate with the RPi cluster. Go to VCN > Security List. You need to click on Add Ingress Rule. While I could only open the needed ports for k3s networking (listed here), I decided to open all OCI ports toward my public IP only, as there is no risk involved here. So in IP Protocol select All Protocols. Now you can test if everything if it worked by ssh to any RPi4 and try to ping any OCI machine or ssh to it or try another port.

Netmaker

Now to link all of them together.
We will create a VPN between all of them (and if you want to, plus local machine, plus VPS) using Wireguard. While Wireguard is not the hardest app to install and configure, there's an wonderful app that does almost everything by itself - Netmaker.
On your VPS, or your local machine (if it has a static IP) run sudo wget -qO /root/nm-quick-interactive.sh https://raw.githubusercontent.com/gravitl/netmaker/master/scripts/nm-quick-interactive.sh && sudo chmod +x /root/nm-quick-interactive.sh && sudo /root/nm-quick-interactive.sh and follow all the steps. Select Community Edition (for max 50 nodes) and for the rest pick auto.
Now you will have a dashboard at a auto-generated domain. Open that link that you received at the end of the installation in a browser and create a user and password.
It should have created for you a network. Open Network tab and then open the new network created. If you're ok with it, that's great. I changed the CIDR to something more fancier 10.20.30.0/24 and activated UDP Hole Punching for better connectivity over NAT. Now go to Access Key Tab, select your network and there you should have all your keys to connect.
Netclient, the client for every machine, needs wireguard and systemd installed. Create a new ansible playbook wireguard_install.yml and paste this:

---
- hosts: all
  tasks:
  - name: Install wireguard
    apt:
      name:
        - wireguard
...

Now run ansible-playbook wireguard_install.yml -K -b. To check everything is ok until now run ansible -a "wg --version" all and then ansible -a "systemd --version" all.
Create a new file netclient_install.yml and add this:

---
- hosts: server
  tasks:
  - name: Add the Netmaker GPG key
    shell: curl -sL 'https://apt.netmaker.org/gpg.key' | sudo tee /etc/apt/trusted.gpg.d/netclient.asc

  - name: Add the Netmaker repository
    shell: curl -sL 'https://apt.netmaker.org/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/netclient.list

  - name: Update the package list
    shell: apt update

  - name: Install netclient
    shell: apt install netclient
...

Now run it as usual ansible-playbook netclient_install.yml -K -b. This will install netclient on all hosts. To check, run ansible -a "netclient --version" all.
Last step is easy. Just run ansible -a "netclient join -t YOURTOKEN" -b -K. For the part in brackets, copy your Join Command from Netmaker Dashboard > Access Key. Now all hosts will share a network. This is mine, 11 machines (4 RPi4, 4 OCI instances, my VPS, my WSL and my Windows machine; last 3 are not needed).

Ssh to the OCI server and run: first sudo systemctl stop k3s, then sudo rm -rf /var/lib/rancher/k3s/server/db/etcd and then reinstall but this time with curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-iface=nm-netmaker" INSTALL_K3S_CHANNEL=latest sh -.
For agents will make an ansible playbook workers_link.yml with following content:

---
- hosts: workers
  tasks:
  - name: Install k3s on workers and link to server node
    shell: curl -sfL https://get.k3s.io | K3S_URL=https://10.20.30.1:6443 K3S_TOKEN=MYTOKEN INSTALL_K3S_EXEC=--"flannel-iface=nm-netmaker" INSTALL_K3S_CHANNEL=latest sh -v
...

You have to paste the content from file on server sudo cat /var/lib/rancher/k3s/server/node-token as MYTOKEN, and change ip address of server if you have another. Now run it with ansible-playbook ~/ansible/link/workers_link.yml -K -b.
Finally over. Go back to server node, run sudo kubectl get nodes -owide and you should have 8 results there, 1 master node and 7 worker nodes.

References

Netmaker from here and documentation here;

K8s cluster with OCI free-tier and Raspberry Pi4 (part 2)

Ștefănescu Liviu — Wed, 15 Feb 2023 19:30:34 +0000

This long read is a multiple part tutorial for building a Kubernetes cluster (using k3s) with 4 x OCI free-tier ARM instances and 4 x Raspberry Pi 4. Plus some applications needed for installation (Terraform and Ansible) and a lot of things installed on the cluster.
Part 2 is running a Kubernetes cluster on 4 Raspberry Pi 4 using Ansible.
GitHub repository is here

Requirements

At least 2 Raspberry Pi. I've got 4 of them, 3 with 4GB and 1 with 8GB. Every one needs a SD Card, a power adapter plus network cables (plus an optional switch and 4 cases);
And the same from part 1. Windows 11 with WSL2 running Ubuntu 20.04, but this will work on any Linux & Win machine and Terraform installed (tested with v1.3.7)- how to here;

My Pis

Preparing

Installing an OS on Pi is very easy. Just insert the SD card in your PC and use Imager from official website. I choose the same OS as in the OCI cluster, that is Ubuntu Server 22.04.1 64bit. In advanced settings (bottom right of Imager) pick Set Hostname and write your own. I have rpi4-1 rpi4-2 rpi4-3 and rpi4-4. Pick Enable SSH and Set username and password, this way you can connect to Pi immediately, without a monitor and keyboard for it. Then hit Write and repeat this step for every one.

From my home router I found the devices IPs. Still there you can configure Address reservation so every time they keep their IP. Did some Port forwarding too so I can access them from everywhere using DDNS from my ISP, as I don't have static IPv4. All of these settings are configured from you home router so Google how to if you're interested.

I added every PI to local machine C:\Windows\System32\drivers\etc\hosts file to be able to control them easier.

192.168.0.201 rpi4-1
192.168.0.202 rpi4-2
192.168.0.203 rpi4-3
192.168.0.204 rpi4-4

We can check if everything is ok by running some ping rpi4-1 or ssh user@rpi4-1.

Ansible

For configuration management I picked Ansible as it is agentless and not so difficult (spoiler alert, it is though). We can control all RPI4 from local machine. Install Ansible first sudo apt install ansible.
Now from your PC run the following commands (assuming you already generated ssh keys from Part 1, using ssh-keygen):

ssh-copy-id -i ~/.ssh/key.pub user@rpi4-1
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-2
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-3
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-4

This will allow Ansible to connect to every Pi without requesting the password every time.
I had to uncomment with sudo vi /etc/ansible/ansible.cfg the line with private_key_file = ~/.ssh/key. And in /etc/ansible/hosts i added the following:

[big]
rpi4-1  ansible_connection=ssh

[small]
rpi4-2  ansible_connection=ssh
rpi4-3  ansible_connection=ssh
rpi4-4  ansible_connection=ssh

[home:children]
big
small

This will add the big rpi4 in a biggroup and the rest of workers to an small group. Plus a home group having them all. My PC user and RPIs user is the same, but if you have a different one you have to add this to the same file:

[all:vars]
remote_user = user

Now test if everything is ok with ansible home -m ping. Green is ok.
I like to keep all my systems updated to latest version, especially this one used for testing. So we'll need to create a new file update.yml and paste below block in it:

---
- hosts: home
  tasks:
    - name: Update apt repo and cache
      apt: update_cache=yes force_apt_get=yes cache_valid_time=3600
    - name: Upgrade all packages
      apt: upgrade=yes force_apt_get=yes
    - name: Check if a reboot is needed
      register: reboot_required_file
      stat: path=/var/run/reboot-required get_md5=no
    - name: Reboot the box if kernel updated
      reboot:
        msg: "Reboot initiated by Ansible for kernel updates"
        connect_timeout: 5
        reboot_timeout: 90
        pre_reboot_delay: 0
        post_reboot_delay: 30
        test_command: uptime
      when: reboot_required_file.stat.exists

This is an Ansible playbook that updates Ubuntu and then reboots the PIs. Now save it and run ansible-playbook update.yml -K -b which will ask for sudo password and run the playbook. It lasted ~10 minutes for me. On another shell you can ssh to any of the PIs and run htopto see the activity.
Now try ansible home -a "rpi-eeprom-update -a" -b -K to see if there's any firmware update for you Raspberry Pi 4.
Next step is to enable cgroups on every PI. Create a new playbook append-cmd.yml and add:

---
- hosts: home
  tasks:
  - name: Append cgroup to cmdline.txt
    lineinfile:
      path: /boot/firmware/cmdline.txt
      backrefs: yes
      regexp: "^(.*)$"
      line: '\1 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1'
...

This will append to end of file /boot/firmware/cmdline.txt the strings cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1. Run it with ansible-playbook append-cmd.yml -K -b.
I won't add a graphical interface and won't use Wi-Fi and Bluetooth so we can steal some memory from GPU memory, Wi-Fi and BT to be available to the Kubernetes cluster. So we need to add a few lines to /boot/firmware/config.txt using a new Ansible playbook append-cfg.yml with following content:

---
- hosts: home
  tasks:
  - name: Append new lines to config.txt
    blockinfile:
      path: /boot/firmware/config.txt
      block: |
       gpu_mem=16
       dtoverlay=disable-bt
       dtoverlay=disable-wifi
...

Run it again with ansible-playbook append-cfg.yml -K -b.
Next 2 mods must be enabled with command ansible home -a "modprobe overlay" -a "modprobe br_netfilter" -K -b.
Next playbook will create 2 files and add some lines to it. Let's call it iptable.yml and add:

---
- hosts: home
  tasks:
  - name: Create a file and write in it 1.
    blockinfile:
      path: /etc/modules-load.d/k8s.conf
      block: |
        overlay
        br_netfilter
      create: yes
  - name: Create a file and write in it 2.
    blockinfile:
      path: /etc/sysctl.d/k8s.conf
      block: |
        net.bridge.bridge-nf-call-ip6tables = 1
        net.bridge.bridge-nf-call-iptables = 1
        net.ipv4.ip_forward = 1
      create: yes
...

Run it with ansible-playbook iptable.yml -K -b. After that run ansible home -a "sysctl --system" -K -b. My PIs started to be laggy so i rebooted here with ansible home -a "reboot" -K -b.
Now the last step. Installing some apps on PIs. I'm still unsure if this is needed, but I'm pretty sure it won't harm the cluster. Create a new file install.yml and add copy this:

---
- hosts: home
  tasks:
  - name: Install some packages
    apt:
      name:
        - curl
        - gnupg2
        - software-properties-common
        - apt-transport-https
        - ca-certificates
        - linux-modules-extra-raspi
...

Now run it with ansible-playbook install.yml -K -b.

That's all folks. For this part. If you are here only for the Raspberry part now you should install k3s very easy, just follow the steps from official documentation.

References

One of my old k3s install on Raspbbery PI article on LinkedIn - here - inspired from braindose.blog;
Official Ansible documentation - here;
ChatGPT helped me a lot of time, use it here.

K8s cluster with OCI free-tier and Raspberry Pi4 (part 1)

Ștefănescu Liviu — Wed, 15 Feb 2023 19:30:09 +0000

This long read is a multiple part tutorial for building a Kubernetes cluster (using k3s) with 4 x OCI free-tier ARM instances and 4 x Raspberry Pi 4. Plus some applications needed for installation (Terraform and Ansible) and a lot of things installed on the cluster.
Part 1 is running a Kubernetes cluster on OCI free-tier resources using Terraform.
GitHub repository is here

Requirements

Obvious, an OCI account, get it from here - oracle.com/cloud. If you already have an account, be careful not to have any resources provisioned already (even for other users, or compartments), this tutorial will use all free-tier ones. Also be extra careful to pick a region not so popular, as it may have no resources available. Pick a region with enough ARM instances available. If during final steps, terraform is stuck, you can check in OCI > Compute > Instance Pools > select your own > Work requests , if there is Failure _and in that log file there's an error _Out of host capacity, then you must wait, even days until resources are freed. You can run a script from here which will try to create instances until there's something available. When that happens, go fast to your OCI, delete all that was created and then run the terraform scripts;
I used Windows 11 with WSL2 running Ubuntu 20.04, but this will work on any Linux machine;
Terraform installed (tested with v1.3.7 - and OCI provider v4.105)- how to here;

Preparing

(following official guidelines from Oracle)

For safety we should use a separate compartment and user for our OCI configuration, not root ones. Now is a good time to create a new notes file and add some values there, you will need them later. Mostly you will add 3 things for each value (user + user_name_you_created + it's_OCID; group ... etc.).
Go to Identity & Security > Compartments and click on Create Compartment. Open it and copy the OCID to your notes file. Then in Identity & Security > Users click on Create User. Open it and copy the OCID to your notes file.
Then in Identity & Security > Groups click on Create Group. The same as above with the OCID. Here click on Add User to Group and add the newly created user.
In Identity & Security > Policies click on Create Policy, Show manual editor and add the following

allow group group_you_created to read all-resources in <compartment compartment_you_created>
allow group group_you_created to manage virtual-network-family  in compartment <compartment_you_created>
allow group group_you_created to manage instance-family  in compartment <compartment_you_created>
allow group group_you_created to manage compute-management-family  in compartment <compartment_you_created>
allow group group_you_created to manage volume-family  in compartment <compartment_you_created>
allow group group_you_created to manage load-balancers  in compartment <compartment_you_created>
allow group group_you_created to manage network-load-balancers  in compartment <compartment_you_created>
allow group group_you_created to manage dynamic-groups in compartment <compartment_you_created>
allow group group_you_created to manage policies in compartment <compartment_you_created>
allow group group_you_created to manage dynamic-groups in tenancy

Then you need access to OCI from your machine. So, create a new folder in HOME directory
mkdir ~/.oci
Generate a private key there
openssl genrsa -out ~/.oci/key.pem 2048
Change permissions for it
chmod 600 ~/.oci/key.pem
Then generate you're public key
openssl rsa -pubout -in ~/.oci/key.pem -out $HOME/.oci/key_public.pem
And then copy that public key, everything inside that file
cat ~/.oci/key_public.pem
This key has to be added to your OCI new user. Go to OCI > Identity & Security > Users select the new user and open API Keys , click on Add API Key, select Paste Public Key, and there paste all your copied key.
After you've done that you need to copy to notes the fingerprint too. Save the path to the private key too.
*note: Use ~ and not $HOME, that's the only way it worked for me.

To your notes file copy the following too:

Tenancy OCID. Click on your avatar (from top-right), and select Tenancy.
Region. In the top right there is the name of your region too. Now find it here and copy it's identifier (ex. eu-paris-1).
The path to the private key. In our case - ~/.oci/key.pem

Now create a new folder to test if terraform is ok and linked with OCI. In that folder create a file main.tf and add this:

terraform {
  required_providers {
    oci = {
      source  = "oracle/oci"
      version = "4.105.0"
    }
  }
}


# Configure the OCI provider with an API Key

provider "oci" {
  tenancy_ocid     = "ocid1.tenancy.oc1..aaaaaaYOURTENANCY3uzx4a"
  user_ocid        = "ocid1.user.oc1..aaaaaaYOURUSER4s5ga"
  private_key_path = "~/.oci/oci.pem"
  fingerprint      = "2a:d8:YOURFINGERPRINT:a1:cd:06"
  region           = "eu-YOURREGION"
}

#Get a list of Availability Domains
data "oci_identity_availability_domains" "ads" {
  compartment_id = "ocid1.tenancy.oc1..aaaaaaYOURTENANCYuzx4a"
}

#Output the result
output "all-availability-domains-in-your-compartment" {
  value = data.oci_identity_availability_domains.ads.availability_domains
}

Now you should run terraform init to download the OCI provider and then terraform plan to see what will happen and terraform apply to receive final results. This small demo configuration file from above should return the name of the availability domains in that regions. If you receive something like "name" = "pmkj:EU-YOURREGION-1-AD-1" and no errors then everything is ok until now. This file can be deleted now.

Provisioning

You will need to add some new values to the notes file:

From inside OCI, on top right corner, click on Developer Tools > Cloud Shell and there write oci iam availability-domain list. Save the name, not id (if more than 1, pick one). This is your availability_domain variable;
Again from this console type oci compute image list --compartment-id <YOUR-COMPARTMENT> --operating-system "Canonical Ubuntu" --operating-system-version "22.04 Minimal aarch64" --shape "VM.Standard.A1.Flex", to find the OS Image ID. Probably the first result has the latest build, in my case was Canonical-Ubuntu-22.04-Minimal-aarch64-2022.11.05-0. From here save the id. This is your os_image_id variable;
Now just google "my ip" and you will find your Public IP. Save it in CIDR format, ex. 111.222.111.99/32. This is your my_public_ip_cidr variable. I use a cheap VPS just to have a static IP. If you don't have a static IPv4 from your ISP, I don't know a quick solution for you, maybe someone can comment one. You can setup DDNS, but that can't be used in Security List afaik. Only solution every time your IP changes, go to VCN > Security List and modify the Ingress rule with the new IP;
Your public_key_path is your public SSH keys. If you don't have any, quickly generate them with ssh-keygen. You should have one now in ~/.ssh/key.pub (I copied the private key, using scp to the VPS, so I can connect to OCI from local machine and from VPS);
Last is your email address that will be used to install a certification manager. That will be your certmanager_email_address variable. I didn't setup one, as this is just a personal project for testing.

After you've cloned the repo, go to oci/terraform.tfvars and edit all values with the ones from your notes file.
This build uses the great terraform configuration files from this repo of garutilorenzo (using version 2.2; if you have errors running all of this, you should check what changed in this repo since v2.2, or 01.02.23). You can read here if you want to customize your configuration and edit the main.tf file. This is the diagram that garutilorenzo made and how your deployment will look like (this tutorial is without Longhorn and ArgoCD, with 1 server nodes + 3 worker nodes and with ingress controller set as Traefik):

*note - I've got some problems with clock of WSL2 not being synced to Windows clock. And provisioning didn't worked so if you receive clock errors too, verify your time with datecommand, if out of sync just run sudo hwclock -s or sudo ntpdate time.windows.com.
Now just run terraform plan and then terraform apply. If everything was ok you should have your resources created.

When the script finishes save the outputs (or you can find the values in OCI):

Outputs:
k3s_servers_ips = [
  "152.x.x.115",
]
k3s_workers_ips = [
  "140.x.x.158",
  "140.x.x.226",
]
public_lb_ip = tolist([
  {
    "ip_address" = "140.x.x.159"
    "ip_version" = "IPV4"
    "is_public" = true
    "reserved_ip" = tolist([])
  },
  {
    "ip_address" = "10.0.1.96"
    "ip_version" = "IPV4"
    "is_public" = false
    "reserved_ip" = tolist([])

Now you can connect to any worker or server IP using ssh -i ~/.ssh/key ubuntu@152.x.x.115. Connect to server IP and write sudo kubectl get nodes to check all nodes.

That's all for now.

References

Official OCI provider documentation from Terraform - here.
Official OCI Oracle documentation with Tutorials - here and Guides - here.
Great GitHub repo of garutilorenzo - here. There are a few others who can help you with k8s on OCI too 1 with repo, 2, 3.