DEV Community: Arthur

Setting Up OpenZFS on Rocky Linux

Arthur — Thu, 28 Sep 2023 22:45:17 +0000

The RHEL distribution of OpenZFS has two main implementations of ZFS, outlined below:

DKIMS (Dynamic Kernel Module Support)
This implementation is built on the premise that; the system should automatically recompile all DKMS modules if a new kernel version is installed. This allows drivers and devices outside the mainline kernel to continue working after a Linux kernel upgrade.
kABI-tracking kmod packages
The Kernel Application Binary Interface (kABI) is a set of in-kernel symbols used by drivers and other kernel modules. Each major and minor RHEL kernel release has a bunch of in-kernel symbols that are whitelisted. A kABI-tracking kmod package contains a kernel module that is compatible with a given kABI, that is, for a given major and minor release of the EL kernel.

The RHEL OpenZFS packages are provided by the following repository:

For EL7:

yum install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm

and for EL8 and 9:

dnf install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm

After adding that repository, update your repository cache with either;

EL7

yum update -y

EL8 And above

dnf update -y

After that, you have the option to install either the DKMS or kABI-tracking kmod style packages.
I have not had any luck using the DKMS style package (which is the default) and as such, my personal preference is the kABI-tracking.

The commands that follow will only show how to install OpenZFS using EL 8 and above if you need to do this for EL 7 and below; replace dnf with yum and dnf config-manager with yum-config-manager

DKMS

Installing the DKMS style, requires the following three (3) commands:

dnf install -y epel-release
dnf install -y kernel-devel
dnf install -y zfs

kABI Style

Installing the kABI-Tracking kmod, disable the default DKMS style and then install zfs with the following commands.

dnf config-manager --disable zfs
dnf config-manager --enable zfs-kmod
dnf install zfs

By default, the OpenZFS kernel modules are automatically loaded when a ZFS pool is detected. If you would prefer to always load the modules at boot time you can create such configuration in /etc/modules-load.d:

Below is a helper command:

echo zfs >/etc/modules-load.d/zfs.conf

After that, you can confirm if zfs is properly set by using modprobe, as shown below:

/sbin/modprobe zfs

If you get an empty response, then all is well. Otherwise, get a cup of coffee and have fun debugging the issue.

Kubernetes (RKE2)

Arthur — Thu, 17 Aug 2023 13:08:00 +0000

This guide will help you quickly launch a cluster with default options for Rancher Kubernetes Engine

Prerequisites

Make sure your environment fulfills the requirements. If NetworkManager is installed and enabled on your hosts, ensure that it is configured to ignore CNI-managed interfaces.

NOTE: For RKE2 versions 1.21 and higher, if the host kernel supports AppArmor, the AppArmor tools (usually available via the apparmor-parser package) must also be present prior to installing RKE2.

The RKE2 installation process must be run as the root user or through sudo.

Server Node Installation

RKE2 provides an installation script that is a convenient way to install it as a service on systemd based systems. This script is available at https://get.rke2.io. To install RKE2 using this method do the following:

1. Run the installer

curl -sfL https://get.rke2.io | sh -

This will install the rke2-server service and the rke2 binary onto your machine. Due to its nature, It will fail unless it runs as the root user or through sudo.

2. Enable the rke2-server service

systemctl enable rke2-server.service

3. Start the service

systemctl start rke2-server.service

4. Follow the logs, if you like

journalctl -u rke2-server -f

Conclusion:

The rke2-server service will be installed. The rke2-server service will be configured to automatically restart after node reboots or if the process crashes or is killed.
Additional utilities will be installed at /var/lib/rancher/rke2/bin/.

They include: kubectl, crictl, and ctr. Note that these are not on your path by default.

Two cleanup scripts, rke2-killall.sh and rke2-uninstall.sh, will be installed to the path at:

/usr/local/bin for regular file systems

/opt/rke2/bin for read-only and brtfs file systems

INSTALL_RKE2_TAR_PREFIX/bin if INSTALL_RKE2_TAR_PREFIX is set

A kubeconfig file will be written to /etc/rancher/rke2/rke2.yaml.

A token that can be used to register other servers or agent nodes will be created at /var/lib/rancher/rke2/server/node-token

Setting Up Kubernetes Cluster with K3S

Arthur — Tue, 18 Apr 2023 11:51:55 +0000

Deploying a high availability Kubernetes cluster using k3s with 3 masters and ETCD as the storage, the backend is a reliable way to ensure that your applications run seamlessly, even when one or more nodes fail. In this article, we will guide you through the process of setting up such a cluster with detailed examples.

Prerequisites

Before you start deploying a k3s cluster with high availability, you will need to prepare the following:

Three or more servers with 3 running any Linux distribution (these servers will be used as Kubernetes nodes).
A user account with sudo privileges on each server.
A working network connection between the servers.
A firewall is installed on each server, allowing traffic on ports 6443, 2379, 2380, and 8472.

NEW CLUSTER

To run K3s in this mode, you must have an odd number of server nodes. We recommend starting with three nodes. The odd number of nodes is to prevent a split brain. A split-brain is a state of a server cluster where nodes diverge from each other and have conflicts when handling incoming I/O operations. The servers may record the same data inconsistently or compete for resources. In a situation where there is an odd number, there will always be a majority vote, which can automatically be elected as the source of truth for the conflict.
To get started, first launch a server node with the cluster-init flag to enable clustering and a token that will be used as a shared secret to join additional servers to the cluster.

NOTE Replace the token with something secure and keep it safe from third-party access


curl -sfL https://get.k3s.io |K3S_TOKEN=123456789 sh -s - server --cluster-init

After launching the first server, join the second and third servers to the cluster using the shared secret with the server IP address of the first node (the one used in the initial step), in this scenario, we will assume that IP is 10.51.10.10 :


curl -sfL https://get.k3s.io | K3S_TOKEN=123456789 sh -s - server --server https://10.51.10.10:6443

Now you have a highly available control plane. Any successfully clustered servers can be used in the --server argument to join additional server and worker nodes. Joining additional worker nodes to the cluster follows the same procedure as a single server cluster.
There are a few config flags that must be the same in all server nodes (You don't have to worry about these if you use the above method as it automatically configures all these for you):

Network-related flags: --cluster-dns, --cluster-domain, --cluster-cidr, --service-cidr
Flags controlling the deployment of certain components: --disable-helm-controller, --disable-kube-proxy, --disable-network-policy and any component passed to --disable
Feature-related flags: --secrets-encryption

Cluster Access

The kubeconfig file is stored at /etc/rancher/k3s/k3s.yaml is used to configure access to the Kubernetes cluster. If you have installed upstream Kubernetes command line tools such as kubectl or helm you will need to configure them with the correct kubeconfig path. This can be done by either exporting the KUBECONFIG environment variable or invoking the --kubeconfig command line flag. Refer to the examples below for details.
The configuration file /etc/rancher/k3s/k3s.yaml can be found on any of the master servers in your cluster. Copy this file to your local machine and install kubectl on your local dev machine:
Check the official Kubernetes website for instructions on how to install kubectl on your operating system.
Leverage the KUBECONFIG environment variable:


export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

kubectl get pods --all-namespaces

If you get a list of namespaces on your cluster, then you are good to go.
While this setup is sufficient to get you started, we also need to have a highly available storage service. K3S comes with hostPath storage, which is not ideal for HA workloads.

Longhorn

Longhorn delivers simplified, easy-to-deploy and upgrade, 100% open source, cloud-native persistent block storage without the cost overhead of open core or proprietary alternatives.
The Longhorn block storage can easily be added to your cluster with the following command


kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

Conclusion

You have now finally deployed an enterprise-grade Kubernetes cluster with k3s. You can now deploy some work on this cluster. Some components to take note of are for ingress, you already have Traefik installed, longhorn will handle storage and Containerd as the container runtime engine.

Kubernetes Ingress With Traefik & SSL Certificate

Arthur — Tue, 18 Apr 2023 11:34:35 +0000

Create Secret

SSL certificates in kubernetes are stored as secrets.
To do this, create a TLS secret with the following command.


kubectl create secret tls example-co-zm-ssl --key private.key --cert cert.crt -n longhorn-system

The above command creates a TLS secret in the longhorn-system namespace.

We need to create another secret to store the basic authentication users, this can be done as:


htpasswd -nb arthur P@55w0rd | openssl base64

The add the output in the secret.yaml file, to look like:


apiVersion: v1
kind: Secret
metadata:
  name: basic-auth-secret
  namespace: longhorn-system
data:
  users: |
    YXJ0aHVyOiRhcHIxJER1dXdPUmtMJGlsOFJiZnpiNjgzdGpPU0dLSGxNczAKCg==

With all the certififcates out of the way, we can now create the two middlewares, one for redirecting to https if traffic originates from http
and the other to request a basic authentication prompt for users to provide username and password to access the resource on the url.

Basic Authentication

Create a basic authentication middleware with the following contents:


apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: basic-auth
  namespace: longhorn-system
spec:
  basicAuth:
    secret: basic-auth-secret

Note: This middleware contains the basic authentication secret that stores the username password combination for allowed users.

HTTP to HTTPS Middleware

Creata middleware to route all HTTP to HTTPS with below content.


apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: enforce-https
  namespace: longhorn-system
spec:
  redirectScheme:
    scheme: https
    permanent: true

Note: The above middleware contains the scheme to redirect to,which is https and the permanent attribute set to true to make this a permanent 301 redirect.

With all that done, you can create an ingress that contains the following contents.


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-ingress
  namespace: longhorn-system
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: longhorn-system-enforce-https@kubernetescrd,longhorn-system-basic-auth@kubernetescrd
spec:
  tls:
    - secretName: example-co-zm-ssl
  rules:
  - host: storage.example.co.zm
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: longhorn-frontend
            port: 
              number: 80

The above ingress contains annotations that add two middlewares to this ingress.

NOTE: The middleware has a syntax that requires that the middleware semantic follow a pattern of:


< namespace >-< middleware >@kubernetescrd

Which for the basic-auth middleware which is in the longhorn-system namespace translates to longhorn-system-basic-auth@kubernetescrd
The middlewares can be chained together by comma separating them.

THe tls part contains a list of TLS secrets, in our case, we only have one.

The rules dictate how route matching is processed, what service to connect to and the port on the service being connected to.

To apply these configurations, use kubectl apply -f


kubectl apply -f secret.yaml -f basic-auth-middleware.yaml -f https-middleware.yaml  -f longhorn-ingress.yaml

Conclusion

This is with the assumption that your kubernetes cluster has traefik as the ingress controller and longhorn as the block storage controller already setup.
Otherwise,you might need tomodify this toi suit the ingress controller being used and the service being connected to.

Kubernetes kubeconfig scoped to a namespace

Arthur — Sun, 09 Apr 2023 12:10:41 +0000

This article is meant to be a guide in setting up a multi-user namespace scoped kubernetes cluster.

Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.

While kubernetes does not have a notion of users, it has what are called service accounts. These are accounts which define the scope of the role(s) or operations which can be performed on different kubernetes resources. A service account provides an identity for processes that run in a Pod.

Before you can access the kubernetes API Service, a service account with the necessary roles is required.
This article assumes that you already have a roles and namespaces already set. You can ignore the namespace if you don't want to scope the service account to a namespace.

To create a service account,

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: devspace
  name: arthur

Aside from the above, you also need to create a secret before getting the token to use with your service accounts as follows:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  namespace: devspace
  name: auth-secret
  annotations:
    kubernetes.io/service-account.name: arthur
type: kubernetes.io/service-account-token
EOF

With the service and tokens created, we can proceed to creating a kubeconfig file, (used to authenticate operations sent to the API service).

The kubeconfig file is a yaml file that can be created by replacing the bash file below with your own values.
Create a bash script file and give a name, e.g kubeconfig.sh, make it executable

chmod +x ./kubeconfig.sh

and finally add the content below to the file. Make any changes to suit your needs.

#!/usr/bin/env sh

# The script returns a kubeconfig for the ServiceAccount given
# you need to have kubectl on PATH with the context set to the cluster you want to create the config for

# Cosmetics for the created config
clusterName='SwiftCloudCluster'
# your server address goes here get it via `kubectl cluster-info`
server='https://kube-master:6443'
# the Namespace and ServiceAccount name that is used for the config
namespace='devspace'
serviceAccount='arthur'

# The following automation does not work from Kubernetes 1.24 and up.
# You need to
# define a Secret, reference the ServiceAccount there and set the secretName as described in the [article](dev.to/arthurkay)!
# See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-a-long-lived-api-token-for-a-serviceaccount for details
#secretName=$(kubectl --namespace="$namespace" get serviceAccount "$serviceAccount" -o=jsonpath='{.secrets[0].name}')

# For kubernetes v1.24 and above, use:
secretName="arthur-secret"

######################
# actual script starts
set -o errexit


ca=$(kubectl --namespace="$namespace" get secret/"$secretName" -o=jsonpath='{.data.ca\.crt}')
token=$(kubectl --namespace="$namespace" get secret/"$secretName" -o=jsonpath='{.data.token}' | base64 --decode)

echo "
---
apiVersion: v1
kind: Config
clusters:
  - name: ${clusterName}
    cluster:
      certificate-authority-data: ${ca}
      server: ${server}
contexts:
  - name: ${serviceAccount}@${clusterName}
    context:
      cluster: ${clusterName}
      namespace: ${namespace}
      user: ${serviceAccount}
users:
  - name: ${serviceAccount}
    user:
      token: ${token}
current-context: ${serviceAccount}@${clusterName}
"

To create the actual kubeconfig file, you need to execute the created bash script and pipe the result to a yaml file.

./kubeconfig.sh >> kubeconfig

This creates a file kubeconfig that can be used for authenticating with your kubernetes cluster.

K3S Node Status (NotReady)

Arthur — Sun, 09 Apr 2023 06:56:24 +0000

K3S is a lightweight production grade kubernetes platform.
Its light weight because a lot of unnecessary components have been removed; the unnecessary components include cloud providers, beta features, defaults and the deployment runs in a single binary.

Having a single binary makes the deployment a whole lot easier. And now back to the topic of discussion.

NotReady status can be caused by a lot of things, it might be that the node was initially connected to the cluster but is no longer able to communicate with the cluster or the underlying machine is not active. This can be fixed by either starting up the machine if it's off or making sure that the node is able to establish two communication with other nodes in the cluster. This in most cases will turn out to be the underlying host machines networking.

If upon checking on the machine, and it turns out that the machine is running and network connectivity is just fine. Check on the status of the node status using kubectl.

kubectl describe node <node-hostname>

The above command gives an very verbose out put of the state of the node. Among the things to look out for in this output are, memory pressure, disk pressure, PID pressure.

A pressure in any of the above can cause the kubelet to fail to run and hence, the node not being usable.

If you notice that the kubelet is not able to start due to memory pressure or CPU & memory limits going going beyond 100%. This is a sign of over-committing resources on your node. This happens when the sum of all Kubernetes resource limits is bigger than the capacity of that resource. When you are over-committing resources in your cluster, everything might run perfectly in normal conditions, but in high load scenarios, the containers could start consuming CPU and memory up to their limit. And in some instances make your node unavailable.

To make your node available with over-committed resources, you'll need to allow this in your kernel, by creating a file in /etc/sysctl.d, lets call this file /etc/sysctl.d/50-kubelet.conf. In that file, add the following:

kernel.panic=10
kernel.panic_on_oops=1
vm.overcommit_memory=1

To enble the changes just added, execute:

sysctl --system -p

After this, restart the k3s daemon if need be (K3S is set to auto-restart by default).

Another issue that might bring up such an issue is the the invalidDiskCapacity 0. This is just a warning that normally clears up after sometime, but during the time it shows up, the node wont be available. For further debugging check the github issue #1857.

While the above will allow the node to be available with over-committed resources, this is not good practice. To solve this issue you might need to add compute resource limits in your service deployment files that are well within your nodes range.

An even better approach is to set the limits with role(s)/clusterRole(s) so pods cannot use more than allocated.

GCP App Engine Primer

Arthur — Mon, 13 Mar 2023 15:40:20 +0000

Google App Engine is a cloud computing platform as a service for developing and hosting web applications in Google-managed data centres.

This comes with the added advantage of running apps load balanced across multiple servers without the worry of managing underlying infrastructure.

The other cool thing is you get a free SSL certificate and URL endpoint provided by google. While this is fine, a lot of the times, we want our own domain. Thankfully, this is also provided out of the box.

But before you can use a custom domain, you need to have one. If you don't have, you can purchase from any domain registrar, even google itself.

With that out of the way; head over to your google cloud console.
This is with the assumption that you already have a GCP project with app engine API enabled.

If you have not yet done that, you can search for app engine in cloud console.

Then proceed to creating a new project. After you have that taken care of, you can go ahead and get the gcloud CLI (SDK as google incorrectly calls it).

This can be installed from here.

Allow the gcloud CLI to login into your GCP with the following command:


gcloud auth login

The above command opens your default browser asking you to login into you GCP account. Once authorised, the CLI can be used to deploy your app engine service(s).

App Engine Environments

App engine provides two deployment environments, i.e standard and flexible.

Standard

According to google, in the standard environment,the "applications run in a secure, sandboxed environment, allowing the standard environment to distribute requests across multiple servers and scale servers to meet traffic demands. Your application runs within its own secure, reliable environment that is independent of the hardware, operating system, or physical location of the server". Aside from this, the only supported languages under standard environment are PHP, Nodejs, GO, Java, Ruby and Python as of the time of this writing.

The standard package also provides a free tier, if resource consumption is low.

For the actual free tier resources, follow this link.

Flexible

The flexible environment has everything understand, with more "flexibility" as the name suggest.
This means that you are not limited by the choice of the programming language.

Using a programming language that is not part of the standard app engine programming languages is as easy as adding your own Dockerfile, and setting the runtime value to custom in the app.yaml file.

Aside from this, you can also specify the compute resources (i.e CPU, RAM,etc)to allocate to the app engine instances

Conclusion

This is going to be a series of articles meant to help one get started with GCP app engine, and the next article will dive into adding the app.yaml file to your project to make it deployable to app engine.

Be on the look out.

Self Hosted Supabase with External Postgresql

Arthur — Tue, 28 Feb 2023 18:24:35 +0000

Supabase is a self-hosted open source application back-end with a cloud offering (hosted by the developers of the platform). The platform is considered an open source firebase alternative by many. Whether or not that statement is true, is a discussion for another day.

What really stands out mostly for me, is the on the fly API & API documentation and most importantly simplified Row Level Security and realtime notifications are by far the biggest selling points.

Requirements

Linux (Debian)
Docker
Docker-compose
postgres
make
git

Dependencies

Install postgres and configure it to allow tcp connections from networks other than localhost.

You'll need to edit the postgres.conf and pg_hda.conf files.

Enter your psql terminal:

Get the postgres.conf file

SHOW config_file;

You'll get the following output:

               config_file               
-----------------------------------------
 /etc/postgresql/12/main/postgresql.conf
(1 row)

Open the file, then change the line:

 listen_addresses = 'localhost'

 listen_addresses = '*'

Get the pg_hba.conf file

SHOW hba_file;

You'll get the following output:

              hba_file               
-------------------------------------
 /etc/postgresql/12/main/pg_hba.conf
(1 row)

Open this file and add the following lines:

host    all             all              0.0.0.0/0                       md5
host    all             all              ::/0                            md5

Then restart the postgres daemon, for the above changes to take, effect.
The default postgres user postgres does not have a password, now would be the right time to create a user with password, that will be used by supabase, and grant that user superuser with login, create db, create role, bypass rls roles

To install custom postgres extensions, the dev postgresl packages are required with make. To get these:


sudo apt install sudo apt install postgresql-server-dev-XX cmake make

Where xx is the version of your postgresql db. In my case, the above command is:


sudo apt install sudo apt install postgresql-server-dev-12 cmake make

Install docker and docker-compose. You'll also need to install Git.

Get the pg_jwt source code from github by cloning the project to your local machine:


git clone https://github.com/michelp/pgjwt

Then change directory into the pgjwt directory, then:


sudo make install

The above creates the pgjwt extension which will be installed with postgres create extension pgjwt

At this point we now need to get supabase from github; clone the repository t your local machine and cd into the project directory


git clone https://github.com/supabase/supabase && cd supabase

Setting up secrets

Whilst you can use the defaults provided, it is advisable to set your own secrets.

cp .env.example .env

I recommend you change the default anon key and service keys in the .env file

Use your JWT_SECRET to generate a anon and service API keys using the JWT Geneartor.

Replace the values in these files:

 .env:
    ANON_KEY - replace with an anon key
    SERVICE_ROLE_KEY - replace with a service key
 volumes/api/kong.yml
    anon - replace with an anon key
    service_role - replace with a service key

Having made the above changes; in the .env file add credentials to your external postgresql database

POSTGRES_PASSWORD=your-super-secret-and-long-postgres-password

POSTGRES_HOST=host.docker.internal
POSTGRES_DB=postgres
POSTGRES_USER=postgres
POSTGRES_PORT=5432

The host.docker.internal on POSTGRES_HOST makes it possible for the container to connect to the host machine. Further the docker-compose.yaml has been changed ti include the line:

extra_hosts:
      - "host.docker.internal:host-gateway"

at the end of each service to allow them to connect with the host. And the depends-on db removed everywhere in the file. The final file wil look as show below:

Migrations

You are now ready to run database migrations. Migrations are essentially table schemas that supabase will need to initialize and set up everything it needs. These also include postgres extensions; Whilst supabase self hosted, now supports graphql, I will not include its configurations has it only works with postgres 14, which I have not yet tried.
While in the same supabase project, navigate to:

cd docker/volumes/db/init

In that directory you'll find files that need to be run in their order.

And you should be good to go.