DEV Community: Beppe

The Home Server Journey - 6: Your New Blogging Career

Beppe — Tue, 08 Oct 2024 22:48:36 +0000

Hello, folks!

One thing that bothers me when writing about self-hosting, to have greater control of my data, is that I don't apply those principles to the articles themselves. I mean, there is no immediate risk of Minds or Dev.to taking down my publications, but the mere fact that they could do that leaves me concerned

The Right Tool for the Job

First I've looked at the tools I was already familiar with. I have some old blog where I've posted updates during my Google Summer of Code projects. It uses Jekyll to generate static files, automatically published by GitHub Pages. It works very well when you have the website tied to a version-controlled repository, but it's cumbersome when you need to rebuild container images or replace files in a remote volume even for small changes

When looking for something more dynamic, I initially though about using Plume, since it's easy to integrate with some applications I plan to deploy later, but unfortunately it's not well maintained anymore. As Ghost or Wordpress seem overkill, I ended up opting for the conveniences of WriteFreely: it lets me create and edit posts in-place, with Markdown support and no need to upload new files. However, that comes with a cost: it requires a MySQL[-compatible] database

Contrarian Vibes

It seems easy enough to just deploy a MySQL container and use it, right? Well... It seems that there are some concerns about its licensing and development direction ever since the brand has been bought by Oracle (remember OpenOffice?). That was the motivation for the MariaDB fork, distributed under the GPLv2 license, which nowadays is not even a 100% drop-in replacement for MySQL, but still works for our case

Reputation-related shenanigans aside, one great advantage of picking MariaDB is the ability to use a Galera Cluster. Similarly to what we did for PostgreSQL, I wish to be able to scale it properly, and Galera's replication works in an even more interesting manner, with multiple primary (read-write) instances and no need for a separate proxy!:

(Man, I wish PostgreSQL had something similar...)

Of course that requires a more complex setup for the database server itself, but thanks to Bitnami's mariadb-galera Docker image and Helm chart, I've managed to get to something rather manageable for our purposes:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mariadb-config
  labels:
    app: mariadb
data:
  BITNAMI_DEBUG: "false"                 # Set to "true" for more debug information
  MARIADB_GALERA_CLUSTER_NAME: galera
  # All pods being synchronized (has to reflect the number of replicas)
  MARIADB_GALERA_CLUSTER_ADDRESS: gcomm://mariadb-state-0.mariadb-replication-service.default.svc.cluster.local,mariadb-state-1.mariadb-replication-service.default.svc.cluster.local
  MARIADB_DATABASE: main                      # Default database
  MARIADB_GALERA_MARIABACKUP_USER: backup     # Replication user
---
# Source: mariadb-galera/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mariadb-secret
  labels:
    app: mariadb
data:
  MARIADB_ROOT_PASSWORD: bWFyaWFkYg==             # Administrator password
  MARIADB_GALERA_MARIABACKUP_PASSWORD: YmFja3Vw   # Replication user password
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mariadb-cnf-config
  labels:
    app: mariadb
data:                                             # Database server configuration
  my.cnf: |
    [client]
    port=3306
    socket=/opt/bitnami/mariadb/tmp/mysql.sock
    plugin_dir=/opt/bitnami/mariadb/plugin

    [mysqld]
    explicit_defaults_for_timestamp
    default_storage_engine=InnoDB
    basedir=/opt/bitnami/mariadb
    datadir=/bitnami/mariadb/data
    plugin_dir=/opt/bitnami/mariadb/plugin
    tmpdir=/opt/bitnami/mariadb/tmp
    socket=/opt/bitnami/mariadb/tmp/mysql.sock
    pid_file=/opt/bitnami/mariadb/tmp/mysqld.pid
    bind_address=0.0.0.0

    ## Character set
    ##
    collation_server=utf8_unicode_ci
    init_connect='SET NAMES utf8'
    character_set_server=utf8

    ## MyISAM
    ##
    key_buffer_size=32M
    myisam_recover_options=FORCE,BACKUP

    ## Safety
    ##
    skip_host_cache
    skip_name_resolve
    max_allowed_packet=16M
    max_connect_errors=1000000
    sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
    sysdate_is_now=1

    ## Binary Logging
    ##
    log_bin=mysql-bin
    expire_logs_days=14
    # Disabling for performance per http://severalnines.com/blog/9-tips-going-production-galera-cluster-mysql
    sync_binlog=0
    # Required for Galera
    binlog_format=row

    ## Caches and Limits
    ##
    tmp_table_size=32M
    max_heap_table_size=32M
    # Re-enabling as now works with Maria 10.1.2
    query_cache_type=1
    query_cache_limit=4M
    query_cache_size=256M
    max_connections=500
    thread_cache_size=50
    open_files_limit=65535
    table_definition_cache=4096
    table_open_cache=4096

    ## InnoDB
    ##
    innodb=FORCE
    innodb_strict_mode=1
    # Mandatory per https://github.com/codership/documentation/issues/25
    innodb_autoinc_lock_mode=2
    # Per https://www.percona.com/blog/2006/08/04/innodb-double-write/
    innodb_doublewrite=1
    innodb_flush_method=O_DIRECT
    innodb_log_files_in_group=2
    innodb_log_file_size=128M
    innodb_flush_log_at_trx_commit=1
    innodb_file_per_table=1
    # 80% Memory is default reco.
    # Need to re-evaluate when DB size grows
    innodb_buffer_pool_size=2G
    innodb_file_format=Barracuda

    [galera]
    wsrep_on=ON
    wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so
    wsrep_sst_method=mariabackup
    wsrep_slave_threads=4
    wsrep_cluster_address=gcomm://
    wsrep_cluster_name=galera
    wsrep_sst_auth="root:"
    # Enabled for performance per https://mariadb.com/kb/en/innodb-system-variables/#innodb_flush_log_at_trx_commit
    innodb_flush_log_at_trx_commit=2
    # MYISAM REPLICATION SUPPORT #
    wsrep_mode=REPLICATE_MYISAM

    [mariadb]
    plugin_load_add=auth_pam
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mariadb-state
spec:
  serviceName: mariadb-replication-service       # Use the internal/headless service name
  replicas: 2
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      securityContext:              # Container is not run as root
        fsGroup: 1001
        runAsUser: 1001
        runAsGroup: 1001
      containers:
        - name: mariadb
          image: docker.io/bitnami/mariadb-galera:11.5.2
          imagePullPolicy: "IfNotPresent"   
          command:
            - bash
            - -ec
            - |
                exec /opt/bitnami/scripts/mariadb-galera/entrypoint.sh /opt/bitnami/scripts/mariadb-galera/run.sh
          ports:
            - name: mdb-mysql-port
              containerPort: 3306                # External access port (MySQL's default)
            - name: mdb-galera-port
              containerPort: 4567                # Internal process port
            - name: mdb-ist-port
              containerPort: 4568                # Internal process port
            - name: mdb-sst-port
              containerPort: 4444                # Internal process port
          envFrom:
            - configMapRef:
                name: mariadb-config
          env:
            - name: MARIADB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mariadb-secret
                  key: MARIADB_ROOT_PASSWORD
            - name: MARIADB_GALERA_MARIABACKUP_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mariadb-secret
                  key: MARIADB_GALERA_MARIABACKUP_PASSWORD
          volumeMounts:
            - name: previous-boot
              mountPath: /opt/bitnami/mariadb/.bootstrap
            - name: mariadb-data
              mountPath: /bitnami/mariadb
            - name: mariadb-cnf
              mountPath: /bitnami/conf/my.cnf               # Overwrite any present configuration
              subPath: my.cnf
            - name: empty-dir
              mountPath: /tmp
              subPath: tmp-dir
            - name: empty-dir
              mountPath: /opt/bitnami/mariadb/conf
              subPath: app-conf-dir
            - name: empty-dir
              mountPath: /opt/bitnami/mariadb/tmp
              subPath: app-tmp-dir
            - name: empty-dir
              mountPath: /opt/bitnami/mariadb/logs
              subPath: app-logs-dir
      volumes:
        - name: previous-boot
          emptyDir: {}                   # Use a fake directory for mounting unused but required paths
        - name: mariadb-cnf
          configMap:
            name: mariadb-cnf-config
        - name: empty-dir
          emptyDir: {}                   # Use a fake directory for mounting unused but required paths
  volumeClaimTemplates:                  # Description of volume claim created for each replica
    - metadata:
        name: mariadb-data
      spec:
        storageClassName: nfs-small
        accessModes: 
          - ReadWriteOnce
        resources:
          requests:
            storage: 8Gi
---
# Headless service for internal replication/backup processes
apiVersion: v1
kind: Service
metadata:
  name: mariadb-replication-service
  labels:
    app: mariadb
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: mariadb-galera-service
      port: 4567
      targetPort: mdb-galera-port
      appProtocol: mysql
    - name: mariadb-ist-service
      port: 4568
      targetPort: mdb-ist-port
      appProtocol: mysql
    - name: mariadb-sst-service
      port: 4444
      targetPort: mdb-sst-port
      appProtocol: mysql
  publishNotReadyAddresses: true
---
# Exposed service for external access
apiVersion: v1
kind: Service
metadata:
  name: mariadb-service
spec:
  type: LoadBalancer                        # Let it be accessible inside the local network
  selector:
    app: mariadb
  ports:
    - port: 3306
      targetPort: mdb-mysql-port
      appProtocol: mysql

Incredibly, it works. My deployment has been running without issue for some time now:

$ kubectl get all -n choppa -l app=mariadb                                                                                                                                                
NAME                  READY   STATUS    RESTARTS       AGE
pod/mariadb-state-0   1/1     Running   2 (3d1h ago)   5d3h
pod/mariadb-state-1   1/1     Running   2 (3d1h ago)   5d3h

NAME                                  TYPE           CLUSTER-IP     EXTERNAL-IP                 PORT(S)                      AGE
service/mariadb-replication-service   ClusterIP      None           <none>                      4567/TCP,4568/TCP,4444/TCP   5d3h
service/mariadb-service               LoadBalancer   10.43.40.243   192.168.3.10,192.168.3.12   3306:31594/TCP               5d3h

NAME                             READY   AGE
statefulset.apps/mariadb-state   2/2     5d3h

(The 2 restarts were due to a power outage that exceeded the autonomy of my no-break's battery)

Solving one Problem to Reveal Another

I just started typing my first self-hosted blog post to realize something was missing: images. On Jekyll I had a folder for that, but on Minds and Dev.to they are hosted somewhere else, e.g. https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v2221vgcikcr05hmnj4v.png

If complete self-hosting is a must, I now need some file server capable of generating shareable links, to be used in my Markdown image components. In summary, Syncthing is great for Dropbox-style backups, but can't share links, NextCloud is too resource-heavy and Seafile is interesting but apparently has proprietary encryption, which left me with the lightweight Filebrowser

I don't expect or intend my file server to ever deal with a huge number of requests, so I've ran it as a simple deployment with a single pod:

kind: PersistentVolumeClaim       # Storage requirements component
apiVersion: v1
metadata:
  name: filebrowser-pv-claim          
  labels:
    app: filebrowser
spec:
  storageClassName: nfs-big       # The used storage class (1TB drive)
  accessModes:
    - ReadWriteOnce
    #- ReadWriteMany               # For concurrent access (In case I try to use more replicas)
  resources:
    requests:
      storage: 200Gi               # Asking for a ~50 Gigabytes volume
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebrowser-config
  labels:
    app: filebrowser
data:                       # Application settings file
  .filebrowser.json: |
    {
      "port": 80,
      "baseURL": "",
      "address": "",
      "log": "stdout",
      "database": "/srv/filebrowser.db",
      "root": "/srv"
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: filebrowser-deploy
spec:
  replicas: 1              
  strategy:
    type: Recreate                  # Wait for the old container to be terminated before creating a new one
  selector:
    matchLabels:
      app: filebrowser
  template:
    metadata:
      labels:
        app: filebrowser
    spec:
      # Run this initial container to make sure at least an empty 
      # database file exists prior to the main container starting,
      # as a workaround for a know bug (https://filebrowser.org/installation#docker)
      initContainers:
        - name: create-database
          image: busybox
          command: ["/bin/touch","/srv/filebrowser.db"]
          volumeMounts:
          - name: filebrowser-data
            mountPath: /srv
      containers:
        - name: filebrowser
          image: filebrowser/filebrowser:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: file-port
              containerPort: 80                 
              protocol: TCP
          volumeMounts:
            - name: filebrowser-readonly
              mountPath: /.filebrowser.json
              subPath: .filebrowser.json
            - name: filebrowser-data
              mountPath: /srv
      volumes:
        - name: filebrowser-readonly
          configMap:
            name: filebrowser-config
        - name: filebrowser-data                        # Label the volume for this deployment
          persistentVolumeClaim:
            claimName: filebrowser-pv-claim           # Reference volumen create by the claim
---
apiVersion: v1
kind: Service
metadata:
  name: filebrowser-service
spec:
  type: NodePort               # Expose the service outside the cluster with an specific port
  selector:
    app: filebrowser
  ports:
    - protocol: TCP
      port: 8080                                 
      targetPort: file-port
      nodePort: 30080

(That's what I did here, make it makes the filebrowser.db file end up visible inside the root folder. It's probably a good idea to use subpaths and mount them separately e.g. srv/filebrowser.db and srv/data for root)

We can't upload or access the files from the Internet yet, but using NodePort an external port in the range 30000-32767 can be used to reach it locally. Use the default username admin and password admin to login and then change it in the settings:

Click on each file you wish to share and the option to generate links will appear on the top. In Markdown syntax, shared images may be annexed with the statement ![Image description](https://<your host>/api/public/dl/<share hash>?inline=true)

One Step Forward. Two Steps Back

All set to deploy WriteFreely, right? As you might guess, no

The application doesn't have an official Docker image, and the custom ones available are either too old or not available for the ARM64 architecture. The repository provided by karlprieb is a good base to build your own, but it lead to crashes here when compiling the application itself. In the end, I found it easier to create one taking advantage of Alpine Linux's packages:

Dockerfile

FROM alpine:3.20
LABEL org.opencontainers.image.description="Simple WriteFreely image based on https://git.madhouse-project.org/algernon/writefreely-docker"
# Install the writefreely package
RUN apk add --no-cache writefreely
# Installation creates the writefreely user, so let's use it
# to run the application
RUN mkdir /opt/writefreely  && chown writefreely -R /opt/writefreely
COPY --chown=writefreely:writefreely ./run.sh /opt/writefreely/
RUN chmod +x /opt/writefreely/run.sh
# Base directory and exposed container port
WORKDIR /opt/writefreely/
EXPOSE 8080
# Set the default container user and group
USER writefreely:writefreely
# Start script
ENTRYPOINT ["/opt/writefreely/run.sh"]

Entrypoint script (run.sh)

#! /bin/sh

writefreely -c /data/config.ini --init-db
writefreely -c /data/config.ini --gen-keys

if [ -n "${WRITEFREELY_ADMIN_USER}" ] && [ -n "${WRITEFREELY_ADMIN_PASSWORD}" ]; then
    writefreely -c /data/config.ini --create-admin "${WRITEFREELY_ADMIN_USER}:${WRITEFREELY_ADMIN_PASSWORD}"
fi

writefreely -c /data/config.ini

Here I've published the image to ancapepe/writefreely:latest on DockerHub, so use it if you wish and have no desire for alternative themes or other custom stuff. One more thing to do before running our blog is to prepare the database to receive its content, so log into the MariaDB server on port 3306 using you root user and execute those commands, replacing username and password to your liking:

CREATE DATABASE writefreely CHARACTER SET latin1 COLLATE latin1_swedish_ci;
CREATE USER 'blog' IDENTIFIED BY 'my_password';
GRANT ALL ON writefreely.* TO 'blog';

Now apply a K8s manifest matching previous configurations and adjusting new ones to your liking:

apiVersion: v1
kind: ConfigMap
metadata:
  name: writefreely-config
  labels:
    app: writefreely
data:
  WRITEFREELY_ADMIN_USER: my_user
  config.ini: |
    [server]
    hidden_host          =
    port                 = 8080
    bind                 = 0.0.0.0
    tls_cert_path        =
    tls_key_path         =
    templates_parent_dir = /usr/share/writefreely
    static_parent_dir    = /usr/share/writefreely
    pages_parent_dir     = /usr/share/writefreely
    keys_parent_dir      = 

    [database]
    type     = mysql
    username = blog
    password = my_password
    database = writefreely
    host     = mariadb-service
    port     = 3306

    [app]
    site_name         = Get To The Choppa
    site_description  = Notes on Conscious Self-Ownership
    host              = https://blog.choppa.xyz
    editor            = 
    theme             = write
    disable_js        = false
    webfonts          = true
    landing           = /login
    single_user       = true
    open_registration = false
    min_username_len  = 3
    max_blogs         = 1
    federation        = true
    public_stats      = true
    private           = false
    local_timeline    = true
    user_invites      = admin
# If you wish to change the shortcut icon for your blog without modifying the image itself, add here the configmap entry generated by running `kubectl create configmap favicon-config --from-file=<your .ico image path>`
binaryData:
  favicon.ico: <binary dump here>
---
apiVersion: v1
kind: Secret
metadata:
  name: writefreely-secret
data:
  WRITEFREELY_ADMIN_PASSWORD: bXlfcGFzc3dvcmQ=
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: writefreely-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: writefreely
  template:
    metadata:
      labels:
        app: writefreely
    spec:
      containers:
        - name: writefreely
          image: ancapepe/writefreely:latest
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 8080
              name: blog-port
          env:
            - name: WRITEFREELY_ADMIN_USER
              valueFrom:
                configMapKeyRef:
                  name: writefreely-config
                  key: WRITEFREELY_ADMIN_USER
            - name: WRITEFREELY_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: writefreely-secret
                  key: WRITEFREELY_ADMIN_PASSWORD
          volumeMounts:
            - name: writefreely-volume
              mountPath: /data/config.ini
              subPath: config.ini
            # Use this if you set the custom favicon.ico image above
            - name: writefreely-volume
              mountPath: /usr/share/writefreely/static/favicon.ico
              subPath: favicon.ico
      volumes:
      - name: writefreely-volume
        configMap:
          name: writefreely-config
---
apiVersion: v1
kind: Service
metadata:
  name: writefreely-service
spec:
  publishNotReadyAddresses: true
  selector:
    app: writefreely
  ports:
    - protocol: TCP
      port: 8080
      targetPort: blog-port

(You may add your own favicon.ico to the image itself if you're building it)

Almost there. Now we just have to expose both our blog pages and file server to the Internet by adding the corresponding entries to our ingress component:

apiVersion: networking.k8s.io/v1                    
kind: Ingress                                     # Component type
metadata:
  name: proxy                                     # Component name
  namespace: choppa                               # You may add the default namespace for components as a paramenter
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: traefik
status:
  loadBalancer: {}
spec:                      
  ingressClassName: traefik                       # Type of controller being used
  tls:
  - hosts:
      - choppa.xyz
      - talk.choppa.xyz
      - blog.choppa.xyz
      - files.choppa.xyz
    secretName: certificate      
  rules:                                          # Routing rules
  - host: choppa.xyz                              # Expected domain name of request, including subdomain
    http:                                         # For HTTP or HTTPS requests
      paths:                                      # Behavior for different base paths
        - path: /                                 # For all request paths
          pathType: Prefix
          backend:
            service:
              name: welcome-service               # Redirect to this service
              port:
                number: 8080                      # Redirect to this internal service port
        - path: /.well-known/matrix/
          pathType: ImplementationSpecific
          backend:
            service:
              name: conduit-service
              port:
                number: 8448
  - host: talk.choppa.xyz
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: conduit-service
              port:
                number: 8448
  - host: test.choppa.xyz                         # Expected domain name of request, including subdomain
    http:                                         # For HTTP or HTTPS requests
      paths:                                      # Behavior for different base paths
        - path: /                                 # For all request paths
          pathType: Prefix
          backend:
            service:
              name: test-service                  # Redirect to this service
              port:
                number: 80                        # Redirect to this internal service port
  - host: blog.choppa.xyz
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: writefreely-service
              port:
                number: 8080
  - host: files.choppa.xyz
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: filebrowser-service
              port:
                number: 8080

If everything went accordingly, you now have everything in place to log into your blog and start publishing. To get an idea of how your self-hosted articles will look like, pay a visit to the first chapter of this series that I'm starting to publish on my server as well:

Thanks for following along. See you next time

The Home Server Journey - 5b: A Bridge Too Far?

Beppe — Mon, 07 Oct 2024 10:13:49 +0000

Hi all. This is a late addendum to my last post

As I have found out, like the Allies had for those river crossings at Operation Market Garden, stateful sets are not as trivial as they initially appear: most guides will just tell you what's their purpose and how to get them running, which leaves a false impression that synchronized data replication across pods happens automagically (sic)

Well, it doesn't

Crossing that River. No Matter the Costs

That special type of deployment will only give you guarantees regarding the order of pods creation and deletion, their naming scheme and which persistent volume they will be bound to. Anything else is on the application logic. You may even violate the principle of using only the first pod for writing and the other ones for reading

When it comes to more niche applications like Conduit, I will probably have to code my own replication solution at some point, but for more widely used software like PostgreSQL there are solutions already available, thankfully

I came across articles by Bibin Wilson & Shishir Khandelwal and Albert Weng (we've seen him here before) detailing how to use a special variant of the database image to get replication working. Although a bit outdated, due to the Docker registry used I'm pretty sure that's based on the PostgreSQL High Availability Helm chart

I don't plan on covering Helm here as I think it adds complexity over already quite complex K8s manifests. Surely it might be useful for large-scale stuff, but let's keep things simple here. I have combined knowledge from the articles with the updated charts in order to created a trimmed-down version of the required manifests (it would be good to add liveliness and readiness probes though):

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
data:
  BITNAMI_DEBUG: "false"                  # Set to "true" for more debug information
  POSTGRESQL_VOLUME_DIR: /bitnami/postgresql
  PGDATA: /bitnami/postgresql/data
  POSTGRESQL_LOG_HOSTNAME: "true"         # Set to "false" for less debug information
  POSTGRESQL_LOG_CONNECTIONS: "false"     # Set to "true" for more debug information
  POSTGRESQL_CLIENT_MIN_MESSAGES: "error"
  POSTGRESQL_SHARED_PRELOAD_LIBRARIES: "pgaudit, repmgr"    # Modules being used for replication
  REPMGR_LOG_LEVEL: "NOTICE"
  REPMGR_USERNAME: repmgr               # Replication user
  REPMGR_DATABASE: repmgr               # Replication information database
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-scripts-config
  labels:
    app: postgres
data:
  # Script for pod termination
  pre-stop.sh: |-
    #!/bin/bash
    set -o errexit
    set -o pipefail
    set -o nounset

    # Debug section
    exec 3>&1
    exec 4>&2

    # Process input parameters
    MIN_DELAY_AFTER_PG_STOP_SECONDS=$1

    # Load Libraries
    . /opt/bitnami/scripts/liblog.sh
    . /opt/bitnami/scripts/libpostgresql.sh
    . /opt/bitnami/scripts/librepmgr.sh

    # Load PostgreSQL & repmgr environment variables
    . /opt/bitnami/scripts/postgresql-env.sh

    # Auxiliary functions
    is_new_primary_ready() {
        return_value=1
        currenty_primary_node="$(repmgr_get_primary_node)"
        currenty_primary_host="$(echo $currenty_primary_node | awk '{print $1}')"

        info "$currenty_primary_host != $REPMGR_NODE_NETWORK_NAME"
        if [[ $(echo $currenty_primary_node | wc -w) -eq 2 ]] && [[ "$currenty_primary_host" != "$REPMGR_NODE_NETWORK_NAME" ]]; then
            info "New primary detected, leaving the cluster..."
            return_value=0
        else
            info "Waiting for a new primary to be available..."
        fi
        return $return_value
    }

    export MODULE="pre-stop-hook"

    if [[ "${BITNAMI_DEBUG}" == "true" ]]; then
        info "Bash debug is on"
    else
        info "Bash debug is off"
        exec 1>/dev/null
        exec 2>/dev/null
    fi

    postgresql_enable_nss_wrapper

    # Prepare env vars for managing roles
    readarray -t primary_node < <(repmgr_get_upstream_node)
    primary_host="${primary_node[0]}"

    # Stop postgresql for graceful exit.
    PG_STOP_TIME=$EPOCHSECONDS
    postgresql_stop

    if [[ -z "$primary_host" ]] || [[ "$primary_host" == "$REPMGR_NODE_NETWORK_NAME" ]]; then
        info "Primary node need to wait for a new primary node before leaving the cluster"
        retry_while is_new_primary_ready 10 5
    else
        info "Standby node doesn't need to wait for a new primary switchover. Leaving the cluster"
    fi

    # Make sure pre-stop hook waits at least 25 seconds after stop of PG to make sure PGPOOL detects node is down.
    # default terminationGracePeriodSeconds=30 seconds
    PG_STOP_DURATION=$(($EPOCHSECONDS - $PG_STOP_TIME))
    if (( $PG_STOP_DURATION < $MIN_DELAY_AFTER_PG_STOP_SECONDS )); then
        WAIT_TO_PG_POOL_TIME=$(($MIN_DELAY_AFTER_PG_STOP_SECONDS - $PG_STOP_DURATION)) 
        info "PG stopped including primary switchover in $PG_STOP_DURATION. Waiting additional $WAIT_TO_PG_POOL_TIME seconds for PG pool"
        sleep $WAIT_TO_PG_POOL_TIME
    fi
---
apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
data:
  POSTGRES_PASSWORD: cG9zdGdyZXM=         # Default user(postgres)'s password
  REPMGR_PASSWORD: cmVwbWdy               # Replication user's password
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-state
spec:
  serviceName: postgres-service
  replicas: 2
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      securityContext:                   # Container is not run as root
        fsGroup: 1001
        runAsUser: 1001
        runAsGroup: 1001
      containers:
        - name: postgres
          lifecycle:
            preStop:                     # Routines to run before pod termination
              exec:
                command:
                  - /pre-stop.sh
                  - "25"
          image: docker.io/bitnami/postgresql-repmgr:16.2.0
          imagePullPolicy: "IfNotPresent"           
          ports:
            - containerPort: 5432
              name: postgres-port
          envFrom:
            - configMapRef:
                name: postgres-config
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: POSTGRES_PASSWORD
            - name: REPMGR_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: REPMGR_PASSWORD
          # Write the pod name (from metadata field) to an environment variable in order to automatically generate replication addresses   
            - name: POD_NAME                   
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          # Repmgr configuration
            - name: REPMGR_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: REPMGR_PARTNER_NODES          # All pods being synchronized (has to reflect the number of replicas)
              value: postgres-state-0.postgres-service.$(REPMGR_NAMESPACE).svc.cluster.local,postgres-state-1.postgres-service.$(REPMGR_NAMESPACE).svc.cluster.local
            - name: REPMGR_PRIMARY_HOST           # Pod with write access. Everybody else replicates it
              value: postgres-state-0.postgres-service.$(REPMGR_NAMESPACE).svc.cluster.local
            - name: REPMGR_NODE_NAME              # Current pod name
              value: $(POD_NAME)
            - name: REPMGR_NODE_NETWORK_NAME
              value: $(POD_NAME).postgres-service.$(REPMGR_NAMESPACE).svc.cluster.local
          volumeMounts:
            - name: postgres-db
              mountPath: /bitnami/postgresql
            - name: postgres-scripts
              mountPath: /pre-stop.sh
              subPath: pre-stop.sh
            - name: empty-dir
              mountPath: /tmp
              subPath: tmp-dir
            - name: empty-dir
              mountPath: /opt/bitnami/postgresql/conf
              subPath: app-conf-dir
            - name: empty-dir
              mountPath: /opt/bitnami/postgresql/tmp
              subPath: app-tmp-dir
            - name: empty-dir
              mountPath: /opt/bitnami/repmgr/conf
              subPath: repmgr-conf-dir
            - name: empty-dir
              mountPath: /opt/bitnami/repmgr/tmp
              subPath: repmgr-tmp-dir
            - name: empty-dir
              mountPath: /opt/bitnami/repmgr/logs
              subPath: repmgr-logs-dir
      volumes:
        - name: postgres-scripts
          configMap:
            name: postgres-scripts-config
            defaultMode: 0755               # Access permissions (owner can execute processes)
        - name: empty-dir                   # Use a fake directory for mounting unused but required paths
          emptyDir: {}
  volumeClaimTemplates:                     # Description of volume claim created for each replica
  - metadata:
      name: postgres-db
    spec:
      storageClassName: nfs-small
      accessModes: 
        - ReadWriteOnce
      resources:
        requests:
          storage: 8Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-service
  labels:
    app: postgres
spec:
  type: ClusterIP                 # Default service type
  clusterIP: None                 # Do not get a service-wide address
  selector:
    app: postgres
  ports:
  - protocol: TCP 
    port: 5432
    targetPort: postgres-port

As the Bitnami container runs as a non-root user for security reasons, requires a "postgres" administrator name for the database and uses a different path structure, you won't be able to mount the data from the original PostgreSQL deployment without messing around with some configuration. So, unless you absolutely need the data, just start from scratch by deleting the volumes, maybe doing a backup first

Notice how our ClusterIP is set to not have an shared address, making it a headless service, so that it only serves the purpose of exposing the target port of each individual pod, still accessible via <pod name>.<service-name>:<container port number>. We do that as our containers here are not meant to be accessed in a random or load-balanced manner

If you're writing your own application, it's easy to define different addresses for writing to and reading from a replicated database, respecting the role of each copy. But a lot of useful software already around assumes a single connection is needed, and there's no simple way to get around that. That's why you need specific intermediaries or proxies like Pgpool-II for PostgreSQL, that can appear to applications as a single entity, redirecting queries to the appropriate backend database depending on it's contents:

(From the PostgreSQL-HA documentation)

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-proxy-config
  labels:
    app: postgres-proxy
data:
  BITNAMI_DEBUG: "true"
  PGPOOL_BACKEND_NODES: 0:postgres-state-0.postgres-service:5432,1:postgres-state-1.postgres-service:5432
  PGPOOL_SR_CHECK_USER: repmgr
  PGPOOL_SR_CHECK_DATABASE: repmgr
  PGPOOL_POSTGRES_USERNAME: postgres
  PGPOOL_ADMIN_USERNAME: pgpool
  PGPOOL_AUTHENTICATION_METHOD: scram-sha-256
  PGPOOL_ENABLE_LOAD_BALANCING: "yes"
  PGPOOL_DISABLE_LOAD_BALANCE_ON_WRITE: "transaction"
  PGPOOL_ENABLE_LOG_CONNECTIONS: "no"
  PGPOOL_ENABLE_LOG_HOSTNAME: "yes"
  PGPOOL_NUM_INIT_CHILDREN: "25"
  PGPOOL_MAX_POOL: "8"
  PGPOOL_RESERVED_CONNECTIONS: "3"
  PGPOOL_HEALTH_CHECK_PSQL_TIMEOUT: "6"
---
apiVersion: v1
kind: Secret
metadata:
  name: postgres-proxy-secret
data:
  PGPOOL_ADMIN_PASSWORD: cGdwb29s
---
apiVersion: v1
kind: Secret
metadata:
  name: postgres-users-secret
data:
  usernames: dXNlcjEsdXNlcjIsdXNlcjM=
  passwords: cHN3ZDEscHN3ZDIscHN3ZDM=
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-proxy-deploy
  labels:
    app: postgres-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-proxy
  template:
    metadata:
      labels:
        app: postgres-proxy
    spec:       
      securityContext:
        fsGroup: 1001
        runAsGroup: 1001
        runAsUser: 1001
      containers:
        - name: postgres-proxy
          image: docker.io/bitnami/pgpool:4
          imagePullPolicy: "IfNotPresent"
          envFrom:
            - configMapRef:
                name: postgres-proxy-config
          env:
            - name: PGPOOL_POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: POSTGRES_PASSWORD
            - name: PGPOOL_SR_CHECK_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: REPMGR_PASSWORD
            - name: PGPOOL_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-proxy-secret
                  key: PGPOOL_ADMIN_PASSWORD
            - name: PGPOOL_POSTGRES_CUSTOM_USERS
              valueFrom:
                secretKeyRef:
                  name: postgres-users-secret
                  key: usernames
            - name: PGPOOL_POSTGRES_CUSTOM_PASSWORDS
              valueFrom:
                secretKeyRef:
                  name: postgres-users-secret
                  key: passwords
          ports:
            - name: pg-proxy-port
              containerPort: 5432
          volumeMounts:
            - name: empty-dir
              mountPath: /tmp
              subPath: tmp-dir
            - name: empty-dir
              mountPath: /opt/bitnami/pgpool/etc
              subPath: app-etc-dir
            - name: empty-dir
              mountPath: /opt/bitnami/pgpool/conf
              subPath: app-conf-dir
            - name: empty-dir
              mountPath: /opt/bitnami/pgpool/tmp
              subPath: app-tmp-dir
            - name: empty-dir
              mountPath: /opt/bitnami/pgpool/logs
              subPath: app-logs-dir
      volumes:
        - name: empty-dir
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-proxy-service
  labels:
    app: postgres-proxy
spec:
  type: LoadBalancer                        # Let it be accessible inside the local network
  selector:
    app: postgres-proxy
  ports:
    - protocol: TCP
      port: 5432
      targetPort: pg-proxy-port

I bet most of it is self explanatory by now. Just pay extra attention to the NUM_INIT_CHILDREN, MAX_POLL and RESERVED_CONNECTIONS variables and the relationship between them, as their default values may not be appropriate at all for your application and result in too many connection refusals (Been there. Done that). Moreover, users other than administrator and replicator are blocked from access unless you add them to the custom lists of usernames and passwords, in the format user1,user2,user3,.. and pswd1,pswd2,pswd3,..., here provided as base64-encoded secrets

With all that configured, we can finally (this time I really mean it) deploy a useful, stateful and replicated application:

$ kubectl get all -n choppa -l app=postgres                                                                                                                                               
NAME                   READY   STATUS    RESTARTS   AGE
pod/postgres-state-0   1/1     Running   0          40h
pod/postgres-state-1   1/1     Running   0          40h

NAME                       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/postgres-service   ClusterIP   None         <none>        5432/TCP   40h

$ kubectl get all -n choppa -l app=postgres-proxy                                                                                                                                         
NAME                                         READY   STATUS    RESTARTS   AGE
pod/postgres-proxy-deploy-74bbdd9b9d-j2tsn   1/1     Running   0          40h

NAME                             TYPE           CLUSTER-IP     EXTERNAL-IP                 PORT(S)          AGE
service/postgres-proxy-service   LoadBalancer   10.43.217.63   192.168.3.10,192.168.3.12   5432:30217/TCP   40h

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/postgres-proxy-deploy   1/1     1            1           40h

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/postgres-proxy-deploy-74bbdd9b9d   1         1         1       40h

(Some nice usage of component labels for selection)

Into the Breach!

You should be able to view your database you the postgres user the same way we did last time. After informing the necessary custom users to Pgpoll, now not only I can get my Telegram bridge back running (using the proxy address for the connection string), but also install WhatsApp and Discord ones. Although they're written in Go rather than Python, configuration is very similar, with the relevant parts below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: whatsapp-config
  labels:
    app: whatsapp
data:
  config.yaml: |
    # Homeserver details.
    homeserver:
        # The address that this appservice can use to connect to the homeserver.
        address: https://talk.choppa.xyz
        # The domain of the homeserver (also known as server_name, used for MXIDs, etc).
        domain: choppa.xyz
        # ...
    # Application service host/registration related details.
    # Changing these values requires regeneration of the registration.
    appservice:
        # The address that the homeserver can use to connect to this appservice.
        address: http://whatsapp-service:29318

        # The hostname and port where this appservice should listen.
        hostname: 0.0.0.0
        port: 29318

        # Database config.
        database:
            # The database type. "sqlite3-fk-wal" and "postgres" are supported.
            type: postgres
            # The database URI.
            #   SQLite: A raw file path is supported, but `file:<path>?_txlock=immediate` is recommended.
            #           https://github.com/mattn/go-sqlite3#connection-string
            #   Postgres: Connection string. For example, postgres://user:password@host/database?sslmode=disable
            #             To connect via Unix socket, use something like postgres:///dbname?host=/var/run/postgresql
            uri: postgres://whatsapp:mautrix@postgres-proxy-service/matrix_whatsapp?sslmode=disable
            # Maximum number of connections. Mostly relevant for Postgres.
            max_open_conns: 20
            max_idle_conns: 2
            # Maximum connection idle time and lifetime before they're closed. Disabled if null.
            # Parsed with https://pkg.go.dev/time#ParseDuration
            max_conn_idle_time: null
            max_conn_lifetime: null

        # The unique ID of this appservice.
        id: whatsapp
        # Appservice bot details.
        bot:
            # Username of the appservice bot.
            username: whatsappbot
            # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty
            # to leave display name/avatar as-is.
            displayname: WhatsApp bridge bot
            avatar: mxc://maunium.net/NeXNQarUbrlYBiPCpprYsRqr

        # Whether or not to receive ephemeral events via appservice transactions.
        # Requires MSC2409 support (i.e. Synapse 1.22+).
        ephemeral_events: true

        # Should incoming events be handled asynchronously?
        # This may be necessary for large public instances with lots of messages going through.
        # However, messages will not be guaranteed to be bridged in the same order they were sent in.
        async_transactions: false

        # Authentication tokens for AS <-> HS communication. Autogenerated; do not modify.
        as_token: <same as token as in registration.yaml>
        hs_token: <same hs token as in registration.yaml>
    # ...
    # Config for things that are directly sent to WhatsApp.
    whatsapp:
        # Device name that's shown in the "WhatsApp Web" section in the mobile app.
        os_name: Mautrix-WhatsApp bridge
        # Browser name that determines the logo shown in the mobile app.
        # Must be "unknown" for a generic icon or a valid browser name if you want a specific icon.
        # List of valid browser names: https://github.com/tulir/whatsmeow/blob/efc632c008604016ddde63bfcfca8de4e5304da9/binary/proto/def.proto#L43-L64
        browser_name: unknown
        # Proxy to use for all WhatsApp connections.
        proxy: null
        # Alternative to proxy: an HTTP endpoint that returns the proxy URL to use for WhatsApp connections.
        get_proxy_url: null
        # Whether the proxy options should only apply to the login websocket and not to authenticated connections.
        proxy_only_login: false

    # Bridge config
    bridge:
        # ...
        # Settings for handling history sync payloads.
        history_sync:
            # Enable backfilling history sync payloads from WhatsApp?
            backfill: true
            # ...
            # Shared secret for authentication. If set to "generate", a random secret will be generated,
            # or if set to "disable", the provisioning API will be disabled.
            shared_secret: generate
            # Enable debug API at /debug with provisioning authentication.
            debug_endpoints: false

        # Permissions for using the bridge.
        # Permitted values:
        #    relay - Talk through the relaybot (if enabled), no access otherwise
        #     user - Access to use the bridge to chat with a WhatsApp account.
        #    admin - User level and some additional administration tools
        # Permitted keys:
        #        * - All Matrix users
        #   domain - All users on that homeserver
        #     mxid - Specific user
        permissions:
            "*": relay
            "@ancapepe:choppa.xyz": admin
            "@ancompepe:choppa.xyz": user
        # Settings for relay mode
        relay:
            # Whether relay mode should be allowed. If allowed, `!wa set-relay` can be used to turn any
            # authenticated user into a relaybot for that chat.
            enabled: false
            # Should only admins be allowed to set themselves as relay users?
            admin_only: true
            # ...
    # Logging config. See https://github.com/tulir/zeroconfig for details.
    logging:
        min_level: debug
        writers:
        - type: stdout
          format: pretty-colored
  registration.yaml: |
    id: whatsapp
    url: http://whatsapp-service:29318
    as_token: <same as token as in config.yaml>
    hs_token: <same hs token as in config.yaml>
    sender_localpart: SH98XxA4xvgFtlbx1NxJm9VYW6q3BdYg
    rate_limited: false
    namespaces:
        users:
            - regex: ^@whatsappbot:choppa\.xyz$
              exclusive: true
            - regex: ^@whatsapp_.*:choppa\.xyz$
              exclusive: true
    de.sorunome.msc2409.push_ephemeral: true
    push_ephemeral: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whatsapp-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whatsapp
  template:
    metadata:
      labels:
        app: whatsapp
    spec:
      containers:
        - name: whatsapp
          image: dock.mau.dev/mautrix/whatsapp:latest
          imagePullPolicy: "IfNotPresent"
          command: [ "/usr/bin/mautrix-whatsapp", "-c", "/data/config.yaml", "-r", "/data/registration.yaml", "--no-update" ]
          ports:
            - containerPort: 29318
              name: whatsapp-port
          volumeMounts:
            - name: whatsapp-volume
              mountPath: /data/config.yaml
              subPath: config.yaml
            - name: whatsapp-volume
              mountPath: /data/registration.yaml
              subPath: registration.yaml
      volumes:
      - name: whatsapp-volume
        configMap:
          name: whatsapp-config
---
apiVersion: v1
kind: Service
metadata:
  name: whatsapp-service
spec:
  publishNotReadyAddresses: true
  selector:
    app: whatsapp
  ports:
    - protocol: TCP
      port: 29318
      targetPort: whatsapp-port

apiVersion: v1
kind: ConfigMap
metadata:
  name: discord-config
  labels:
    app: discord
data:
  config.yaml: |
    # Homeserver details.
    homeserver:
        # The address that this appservice can use to connect to the homeserver.
        address: https://talk.choppa.xyz
        # The domain of the homeserver (also known as server_name, used for MXIDs, etc).
        domain: choppa.xyz

        # What software is the homeserver running?
        # Standard Matrix homeservers like Synapse, Dendrite and Conduit should just use "standard" here.
        software: standard
        # The URL to push real-time bridge status to.
        # If set, the bridge will make POST requests to this URL whenever a user's discord connection state changes.
        # The bridge will use the appservice as_token to authorize requests.
        status_endpoint: null
        # Endpoint for reporting per-message status.
        message_send_checkpoint_endpoint: null
        # Does the homeserver support https://github.com/matrix-org/matrix-spec-proposals/pull/2246?
        async_media: false

        # Should the bridge use a websocket for connecting to the homeserver?
        # The server side is currently not documented anywhere and is only implemented by mautrix-wsproxy,
        # mautrix-asmux (deprecated), and hungryserv (proprietary).
        websocket: false
        # How often should the websocket be pinged? Pinging will be disabled if this is zero.
        ping_interval_seconds: 0

    # Application service host/registration related details.
    # Changing these values requires regeneration of the registration.
    appservice:
        # The address that the homeserver can use to connect to this appservice.
        address: http://discord-service:29334

        # The hostname and port where this appservice should listen.
        hostname: 0.0.0.0
        port: 29334

        # Database config.
        database:
            # The database type. "sqlite3-fk-wal" and "postgres" are supported.
            type: postgres
            # The database URI.
            #   SQLite: A raw file path is supported, but `file:<path>?_txlock=immediate` is recommended.
            #           https://github.com/mattn/go-sqlite3#connection-string
            #   Postgres: Connection string. For example, postgres://user:password@host/database?sslmode=disable
            #             To connect via Unix socket, use something like postgres:///dbname?host=/var/run/postgresql
            uri: postgres://discord:mautrix@postgres-proxy-service/matrix_discord?sslmode=disable
            # Maximum number of connections. Mostly relevant for Postgres.
            max_open_conns: 20
            max_idle_conns: 2
            # Maximum connection idle time and lifetime before they're closed. Disabled if null.
            # Parsed with https://pkg.go.dev/time#ParseDuration
            max_conn_idle_time: null
            max_conn_lifetime: null

        # The unique ID of this appservice.
        id: discord
        # Appservice bot details.
        bot:
            # Username of the appservice bot.
            username: discordbot
            # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty
            # to leave display name/avatar as-is.
            displayname: Discord bridge bot
            avatar: mxc://maunium.net/nIdEykemnwdisvHbpxflpDlC

        # Whether or not to receive ephemeral events via appservice transactions.
        # Requires MSC2409 support (i.e. Synapse 1.22+).
        ephemeral_events: true

        # Should incoming events be handled asynchronously?
        # This may be necessary for large public instances with lots of messages going through.
        # However, messages will not be guaranteed to be bridged in the same order they were sent in.
        async_transactions: false

        # Authentication tokens for AS <-> HS communication. Autogenerated; do not modify.
        as_token: <same as token as in registration.yaml>
        hs_token: <same hs token as in registration.yaml>

    # Bridge config
    bridge:
        # ...
        # The prefix for commands. Only required in non-management rooms.
        command_prefix: '!discord'
        # Permissions for using the bridge.
        # Permitted values:
        #    relay - Talk through the relaybot (if enabled), no access otherwise
        #     user - Access to use the bridge to chat with a Discord account.
        #    admin - User level and some additional administration tools
        # Permitted keys:
        #        * - All Matrix users
        #   domain - All users on that homeserver
        #     mxid - Specific user
        permissions:
            "*": relay
            "@ancapepe:choppa.xyz": admin
            "@ancompepe:choppa.xyz": user

    # Logging config. See https://github.com/tulir/zeroconfig for details.
    logging:
        min_level: debug
        writers:
        - type: stdout
          format: pretty-colored
  registration.yaml: |
    id: discord
    url: http://discord-service:29334
    as_token: <same as token as in config.yaml>
    hs_token: <same hs token as in config.yaml>
    sender_localpart: KYmI12PCMJuHvD9VZw1cUzMlV7nUezH2
    rate_limited: false
    namespaces:
        users:
            - regex: ^@discordbot:choppa\.xyz$
              exclusive: true
            - regex: ^@discord_.*:choppa\.xyz$
              exclusive: true
    de.sorunome.msc2409.push_ephemeral: true
    push_ephemeral: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: discord-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: discord
  template:
    metadata:
      labels:
        app: discord
    spec:
      containers:
        - name: discord
          image: dock.mau.dev/mautrix/discord:latest
          imagePullPolicy: "IfNotPresent"
          command: [ "/usr/bin/mautrix-discord", "-c", "/data/config.yaml", "-r", "/data/registration.yaml", "--no-update" ]
          ports:
            - containerPort: 29334
              name: discord-port
          volumeMounts:
            - name: discord-volume
              mountPath: /data/config.yaml
              subPath: config.yaml
            - name: discord-volume
              mountPath: /data/registration.yaml
              subPath: registration.yaml
      volumes:
      - name: discord-volume
        configMap:
          name: discord-config
---
apiVersion: v1
kind: Service
metadata:
  name: discord-service
spec:
  publishNotReadyAddresses: true
  selector:
    app: discord
  ports:
    - protocol: TCP
      port: 29334
      targetPort: discord-port

Logging into your account will still change depending on the service being bridged. As always, consult the official documentation

We may forget for a while the multiple opened messaging windows just to communicate with our peers. That river has been crossed!

The Home Server Journey - 5: Rebuilding Burned Bridges

Beppe — Sun, 29 Sep 2024 16:34:21 +0000

Hello again. It's been a while (in relative terms)

Having our own Matrix rooms feels amazing indeed, but left us too isolated, at least until we manage to bring people in. I know that in the beginning I've warned about making sacrifices, but it doesn't have to be that hard. In life some metaphorical burned bridges are better left unrepaired, but here let's try reconnecting to our old Internet life in an interesting way, right?

Notes on Infrastructure

I have waited a bit to resume publishing because there were 2 things bothering me about my setup: power reliability and storage coupling

I don't live near a big urban center, and power oscillations or outages are pretty common here. Having those causing frequent downtime on my server would be quite annoying, so I went after a nobreak to ensure more energy stability for my devices

With that out the way, as I have said on the previous article, ideally your K8s system would have an external and dedicated storage provider. I certainly wasn't satisfied with mixing K8s node and NFS server functionality in the same machines, but the ones I was using were the only units with high-bandwith USB 3.0 ports, desirable for external drives. So I've decided to invest a bit more and acquire an extra Raspberry Pi 5 for my cluster, leaving the RPi4 for data management

Not only the RPi5 gives me more performance, but it also is more equivalent to the Odroid N2+. As it comes with a beefier power supply (5V-5A), intended for video applications that I'm not doing, I've tried using it for the RPi4 connected to the current-hungry hard drives. However, its circuit is simply not made to use all that juice, so as a last piece of the puzzle I had to get an externally powered USB3 hub:

(The resulting Frankenstein is not a pretty sight)

If that mess of a wiring is too hard to understand (I bet it is), have a quick diagram of how it works below:

At first I was worried that using a single USB port for 2 disks would be a bottleneck, but even in the best case, sequential reading, HDDs cannot saturate the 5Gbps bandwidth the RPi4 can handle on that interface, and the 1Gbps Ethernet will be a greater constraint anyway

Keeping the State, Orchestrated Style

One thing that you notice when deploying all sorts of applications is how much they rely on databases, relational or non-relational, in order to delegate internal data management and remain stateless. That's true to the point that we can hardly progress here without having some DBMS running first

It's common to have databases running as off-node services just like NFS, but the ability to escalate them horizontally is then lost. Here, we'll resort to Kubernetes stateful features in order to keep data management inside the cluster. For that, instead of a Deployment we require a StatefulSet:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  namespace: choppa
  labels:
    app: postgres
data:
  POSTGRES_DB: choppadb                   # Default database
  POSTGRES_USER: admin                    # Default user
---
apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
  namespace: choppa
data:
  POSTGRES_PASSWORD: Z290eW91             # Default user's password
---
apiVersion: apps/v1
kind: StatefulSet                         # Component type
metadata:
  name: postgres-state
  namespace: choppa
spec:
  serviceName: postgres-service           # Do not give a service name to each replica
  replicas: 2
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16.4              # Current stable version at 17.0, but let's be a bit conservative
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
              name: postgres-port
          envFrom:
            - configMapRef:
                name: postgres-config
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: POSTGRES_PASSWORD
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgres-db
  volumeClaimTemplates:                     # Description of volume claim created for each replica
  - metadata:
      name: postgres-db
    spec:
      storageClassName: nfs-small
      accessModes: 
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-service
  namespace: choppa
spec:
  type: LoadBalancer                        # Let it be accessible inside the local network
  selector:
    app: postgres
  ports:
    - protocol: TCP
      port: 5432
      targetPort: postgres-port

(At first we're using a PostgreSQL image from the official DockerHub repository, but different databases such as MySQL/MariaDB or MongoDB will follow a similar pattern)

The biggest difference from previous deployments is that we don't explicitly define a Persistent Volume Claim to be referenced by all replicas, but give the StatefulSet a template on how to request a distinct storage area for each Pod. Moreover, the serviceName configuration defines how stateful pods will get their specific hostnames from the internal DNS, in the format <pod name>.<service-name>:<container port number>

(I have tried using a stateful configuration for Conduit as well, but also wasn't able to increase the number of replicas due to some authentication issues)

Notice how those pods follow a different naming scheme. ReplicaSets with persisting data changes have to be created, deleted and updated/synchronized in a very particular order, therefore they cannot be as ephemeral as stateless ones, with asynchronously managed and randomly named containers. Those constraints are the reason we should avoid StatefulSets wherever possible

A Telegram from an Old Friend

As I've promised, today we're learning how to keep in touch with people in the centralized world from our new Matrix home. And that's achieved through the power of Mautrix bridges. As I'm a big user of Telegram, that's what I'll be using as an example, but the process is mostly the same for other services, like Discord or WhatsApp

Long story short, the bridge needs to be configured on 2 ends: how to interact with your third-party service account and how to register onto your Matrix server. The templates for those settings may be automatically generated using the same container image we'll me pulling into our K8s cluster:

# Create a directory for configuration files and enter it
$ docker run --rm -v `pwd`:/data:z dock.mau.dev/mautrix/telegram:latest                                                                                                
Didn't find a config file.
Copied default config file to /data/config.yaml
Modify that config file to your liking.
Start the container again after that to generate the registration file.
# Change the configuration file to match your requirements and preferences
$ docker run --rm -v `pwd`:/data:z dock.mau.dev/mautrix/telegram:latest                                                                                               
Registration generated and saved to /data/registration.yaml
Didn't find a registration file.
Generated one for you.
See https://docs.mau.fi/bridges/general/registering-appservices.html on how to use it.

Now you either manually add the contents of config.yaml and registration.yaml to a ConfigMap manifest file or automatically generate/print it with:

sudo kubectl create configmap telegram-config --from-file=./config.yaml --from-file=./registration.yaml --dry-run=client -o yaml

(Running as super user is required as the folder permissions are modified. --dry-run prevents the configuration from being directly applied, so that you may adjust possible mistakes, pipe the results to a YAML file or add the result to a bigger manifest)

Bridge configuration is quite long, so I'll reduce it below to only the most relevant parts. For information on all options, consult the official documentation:

apiVersion: v1
kind: ConfigMap
metadata:
  name: telegram-config
  labels:
    app: telegram
data:
  config.yaml: |
    # Homeserver details
    homeserver:
        # The address that this appservice can use to connect to the homeserver.
        address: http://conduit-service:8448
        # The domain of the homeserver (for MXIDs, etc).
        domain: choppa.xyz
    # ...
    # Application service host/registration related details
    # Changing these values requires regeneration of the registration.
    appservice:
        # The address that the homeserver can use to connect to this appservice.
        address: http://telegram-service:29317
        # When using https:// the TLS certificate and key files for the address.
        tls_cert: false
        tls_key: false
        # The hostname and port where this appservice should listen.
        hostname: 0.0.0.0
        port: 29317
        # ...
        # The full URI to the database. SQLite and Postgres are supported.
        # Format examples:
        #   SQLite:   sqlite:filename.db
        #   Postgres: postgres://username:password@hostname/dbname
        database: postgres://telegram:mautrix@postgres-service/matrix_telegram
        # ...
        # Authentication tokens for AS <-> HS communication. Autogenerated; do not modify.
        as_token: <same as token from registration.yaml>
        hs_token: <same hs token from registration.yaml>
    # ...
    # Bridge config
    bridge:
        # ...
        # Shared secrets for https://github.com/devture/matrix-synapse-shared-secret-auth
        #
        # If set, custom puppets will be enabled automatically for local users
        # instead of users having to find an access token and run `login-matrix`
        # manually.
        # If using this for other servers than the bridge's server,
        # you must also set the URL in the double_puppet_server_map.
        login_shared_secret_map:
            choppa.xyz: <your access token>
        # ...
    # ...
    # Telegram config
    telegram:
        # Get your own API keys at https://my.telegram.org/apps
        api_id: <id you have generated>
        api_hash: <hash you have generated>
        # (Optional) Create your own bot at https://t.me/BotFather
        bot_token: disabled
        # ...
  registration.yaml: |
    id: telegram
    as_token: <same as token from config.yaml>
    hs_token: <same hs token from config.yaml>
    namespaces:
        users:
        - exclusive: true
          regex: '@telegram_.*:choppa\.xyz'
        - exclusive: true
          regex: '@telegrambot:choppa\.xyz'
        aliases:
        - exclusive: true
          regex: \#telegram_.*:choppa\.xyz
    url: http://telegram-service:29317
    sender_localpart: HyMPlWT_552RAGkFtnuy_ZNNpkKrSSaHDndS_nmb9VIZ4RiJLH0uiSas3fi_IV_x
    rate_limited: false
    de.sorunome.msc2409.push_ephemeral: true
    push_ephemeral: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: telegram-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: telegram
  template:
    metadata:
      labels:
        app: telegram
    spec:
      containers:
        - name: telegram
          image: dock.mau.dev/mautrix/telegram:latest
          imagePullPolicy: "IfNotPresent"
          # Custom container initialization command (overrides the one defined in the image)
          command: [ "python3", "-m", "mautrix_telegram", "-c", "/data/config.yaml", "-r", "/data/registration.yaml", "--no-update" ]
          ports:
            - containerPort: 29317      # Has to match the appservice configuration
              name: telegram-port
          volumeMounts:
            - name: telegram-volume
              mountPath: /data/config.yaml
              subPath: config.yaml
            - name: telegram-volume
              mountPath: /data/registration.yaml
              subPath: registration.yaml
      volumes:
      - name: telegram-volume
        configMap:
          name: telegram-config
---
apiVersion: v1
kind: Service
metadata:
  name: telegram-service
spec:
  publishNotReadyAddresses: true
  selector:
    app: telegram
  ports:
    - protocol: TCP
      port: 29317               # Has to match the registration url
      targetPort: telegram-port

After applying the changes, the bridge pod is deployed, but it keeps failing and restarting... Why?

Visualizing the logs give us a clearer picture of what's happening:

$ kubectl logs telegram-deploy-84489fb64d-srmrc --namespace=choppa                                                                                                                         ✔  default ⎈ 
[2024-09-29 15:28:34,871] [INFO@mau.init] Initializing mautrix-telegram 0.15.2
[2024-09-29 15:28:34,879] [INFO@mau.init] Initialization complete in 0.19 seconds
[2024-09-29 15:28:34,879] [DEBUG@mau.init] Running startup actions...
[2024-09-29 15:28:34,879] [DEBUG@mau.init] Starting database...
[2024-09-29 15:28:34,879] [DEBUG@mau.db] Connecting to postgres://telegram:password-redacted@postgres-service/matrix_telegram
[2024-09-29 15:28:34,946] [CRITICAL@mau.init] Failed to initialize database
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/mautrix/bridge/bridge.py", line 216, in start_db
    await self.db.start()
  File "/usr/lib/python3.11/site-packages/mautrix/util/async_db/asyncpg.py", line 71, in start
    self._pool = await asyncpg.create_pool(str(self.url), **self._db_args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/pool.py", line 403, in _async__init__
    await self._initialize()
  File "/usr/lib/python3.11/site-packages/asyncpg/pool.py", line 430, in _initialize
    await first_ch.connect()
  File "/usr/lib/python3.11/site-packages/asyncpg/pool.py", line 128, in connect
    self._con = await self._pool._get_new_connection()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/pool.py", line 502, in _get_new_connection
    con = await connection.connect(
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/connection.py", line 2329, in connect
    return await connect_utils._connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 991, in _connect
    conn = await _connect_addr(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 828, in _connect_addr
    return await __connect_addr(params, True, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 876, in __connect_addr
    await connected
asyncpg.exceptions.InvalidPasswordError: password authentication failed for user "telegram"
[2024-09-29 15:28:34,950] [DEBUG@mau.init] Stopping database due to SystemExit
[2024-09-29 15:28:34,950] [DEBUG@mau.init] Database stopped

If you pay attention to the configuration shown, there's a database section where you may either configure SQLite (local file, let's avoid that) or PostgreSQL usage, required by the bridge. I wouldn't go on a tangent about databases for nothing, right?

The problem is that the connection string defined, which follows the format postgres://<user>:<password>@<server hostname>/<database name>, is referencing a user and a database that don't exist (I wouldn't use the administrator account for a simple bot). So let's fix that by logging as the main user into the DBMS with a PostgreSQL-compatible client of your choice. Here I'm using SQLectron:

From there, use this sequence of SQL commands that are pretty much self-explanatory:

CREATE DATABASE matrix_telegram;
CREATE USER telegram WITH PASSWORD 'mautrix';
GRANT CONNECT ON DATABASE matrix_telegram TO telegram;
-- Personal note: ENTER THE DATABASE !!
GRANT ALL ON SCHEMA public TO telegram;
GRANT ALL ON ALL TABLES IN SCHEMA public TO telegram;
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO telegram;
-- Give priveleges for all newly created tables as well
ALTER DEFAULT PRIVILEGES FOR USER telegram IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO telegram;

(A bit overkill but, hey, works for me... If some commands fail you may need to wait a bit for the pods to synchronize or even reduce the number of replicas to 1 while making changes)

Errata: Once again we cannot use stateful features to the fullest. Replicating data across pods is not done automatically by the K8s system and dependends on the application. It took me a while to figure that out and I'll have to study how to do it properly for each case. For now we'll keep it at only one instance and address proper replication later

If you still get some log errors about an unknown token, go into the admin room of your Conduit server and send this message:

Finally, start a private chat with @telegrambot<your server name> (default name, may be changed in the config) and follow the login procedure:

Invites for your Telegram groups and private chats should start appearing, and now you can be happy again!

Thanks for reading. See you next time

The Home Server Journey - 4: Enter The Matrix

Beppe — Fri, 20 Sep 2024 23:24:25 +0000

Hello

Time to forget about simple demo containers and struggle [not so much] with the configuration details of fully-featured applications. Today we not only put Kubernetes to production-level usage, but find out about some strengths and weaknesses of distributed computing

Notes on storage

Code nowadays, more than ever, works consuming and producing huge amounts of data, that usually has (or "have", since "data" is plural for "datum"?) to be kept for later. It's no surprise then that the capacity to store information is one of the most valuable commodities in IT, always at risk of being depleted due to poor optimization

In the containerized world, where application instances can be created and recreated dynamically, extra steps have to be taken for reusing disk space and avoiding duplication whenever possible. That's why K8s adopts a layered approach: deployments don't create Persistent Volumes, roughly equivalent to root folders, on their own, but set their storage requirements via Persistent Volume Claims

Long story short, administrators either create the necessary volumes manually or let the claims be managed by a Storage Class, to provide volumes on demand. Like ingresses have multiple controller implementations, depending on the backend, storage classes allow for different providers, offering a variety of local and remote (including cloud) data resources:

(Screenshot from Nana's tutorial)

Although local is an easy to use option, it quickly betrays the purpose of container orchestration: tying storage to a specific machine's disk path prevents multiple pod replicas from being distributed among multiple nodes for load balancing. Besides, if we wish to avoid cloud providers (like Google or AWS) to have full control of our data, we're left with [local] network storage solutions like NFS

If you remember the first article, my setup consists of one USB hard disk (meant for persistent volumes) connected to each node board. That's not ideal for NFS, as decoupling storage from the K8s cluster, using a dedicated computer, would prevent node crashes compromising data availability to the rest. Still, I can at least make the devices share disks with each other

In order to expose external drive's partition to the network, begin by ensuring that they're mounted during initialization by editing /etc/fstab:

# Finding the partition's UUID (assuming a EXT4-formatted disk)
[ancapepe@ChoppaServer-1 ~]$ lsblk -f
NAME        FSTYPE FSVER LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                     
└─sda1      ext4   1.0              69c7df98-0349-4c47-84ce-514a1699ccf1  869.2G     0% 
mmcblk0                                                                                 
├─mmcblk0p1 vfat   FAT16 BOOT_MNJRO 42D4-5413                             394.8M    14% /boot
└─mmcblk0p2 ext4   1.0   ROOT_MNJRO 83982167-1add-4b6a-ad63-3479493a4510   44.4G    18% /var/lib/kubelet/pods/ae500dcf-b765-4f37-be8c-ee20fbb3ea1b/volume-subpaths/welcome-volume/welcome/0
                                                                                        /
zram0                                                                                   [SWAP]
# Creating the mount point for the external partition
[ancapepe@ChoppaServer-1 ~]$ sudo mkdir /storage
# Adding line with mount configuration to fstab 
[ancapepe@ChoppaServer-1 ~]$ echo 'UUID=69c7df98-0349-4c47-84ce-514a1699ccf1 /storage ext4 defaults,noatime        0       2' | sudo tee -a /etc/fstab
# Check the contents
[ancapepe@ChoppaServer-1 ~]$ cat /etc/fstab
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
PARTUUID=42045b6f-01  /boot   vfat    defaults,noexec,nodev,showexec     0   0
PARTUUID=42045b6f-02   /   ext4     defaults    0   1
UUID=69c7df98-0349-4c47-84ce-514a1699ccf1 /storage ext4 defaults,noatime        0       2

Reboot and check if you can access the contents of the /storage folder (or whatever path you've chosen). Proceed by setting up the NFS server:

# Install required packages (showing the ones for Manjaro Linux)
[ancapepe@ChoppaServer-1 ~]$ sudo pacman -S nfs-utils rpcbind
# ...
# Installation process ...
# ...
# Add disk sharing configurations (Adapt to you local network address range)
[ancapepe@ChoppaServer-1 ~]$ echo '/storage    192.168.3.0/24(rw,sync,no_wdelay,no_root_squash,insecure)' | sudo tee -a /etc/exports
# Check the contents
[ancapepe@ChoppaServer-1 ~]$ cat /etc/exports
# /etc/exports - exports(5) - directories exported to NFS clients
#
# Example for NFSv3:
#  /srv/home        hostname1(rw,sync) hostname2(ro,sync)
# Example for NFSv4:
#  /srv/nfs4        hostname1(rw,sync,fsid=0)
#  /srv/nfs4/home   hostname1(rw,sync,nohide)
# Using Kerberos and integrity checking:
#  /srv/nfs4        *(rw,sync,sec=krb5i,fsid=0)
#  /srv/nfs4/home   *(rw,sync,sec=krb5i,nohide)
#
# Use `exportfs -arv` to reload.
/storage    192.168.3.0/24(rw,sync,no_wdelay,no_root_squash,insecure)
# Update exports based on configuration
[ancapepe@ChoppaServer-1 ~]$ sudo exportfs -arv
# Enable and start related initialization services
[ancapepe@ChoppaServer-1 ~]$ systemctl enable --now rpcbind nfs-server 
# Check if shared folders are visible
[ancapepe@ChoppaServer-1 ~]$ showmount localhost -e
Export list for localhost:
/storage 192.168.3.0/24

(Remember that your mileage may vary)

Phew, and that's just the first part of it! Not only you have to do the same process for every machine whose storage you wish to share, but afterwards it's necessary to install a NFS provisioner to the K8s cluster. Here I have used nfs-subdir-external-provisioner, not as-it-is, but modifying 2 manifest files to my liking:

Modified deploy/rbac.yaml (Access permissions):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-storage
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs-storage
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-storage
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-storage
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs-storage
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

(Here just set the namespace to your liking)

Modified deploy/deployment.yaml (Storage classes):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-storage
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        # 1TB disk on Odroid N2+
        - name: nfs-client-provisioner-big
          image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-root-big
              mountPath: /persistentvolumes   # CAN'T CHANGE THIS PATH
          env:     # Environment variables
            - name: PROVISIONER_NAME
              value: nfs-provisioner-big
            - name: NFS_SERVER                # Hostnames don't resolve in setting env vars
              value: 192.168.3.10             # Looking for a way to make it more flexible
            - name: NFS_PATH
              value: /storage
        # 500GB disk on Raspberry Pi 4B
        - name: nfs-client-provisioner-small
          image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-root-small
              mountPath: /persistentvolumes   # CAN'T CHANGE THIS PATH
          env:     # Environment variables
            - name: PROVISIONER_NAME
              value: nfs-provisioner-small
            - name: NFS_SERVER
              value: 192.168.3.11
            - name: NFS_PATH
              value: /storage
      volumes:
        - name: nfs-root-big
          nfs:
            server: 192.168.3.10
            path: /storage
        - name: nfs-root-small
          nfs:
            server: 192.168.3.11
            path: /storage
---
# 1TB disk on Odroid N2+
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-big
  namespace: choppa
provisioner: nfs-provisioner-big
parameters:
  archiveOnDelete: "false"
reclaimPolicy: Retain
---
# 500GB disk on Raspberry Pi 4B
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-small
  namespace: choppa
provisioner: nfs-provisioner-small
parameters:
  archiveOnDelete: "false"
reclaimPolicy: Retain

(Make sure that all namespace settings match the RBAC rules. Also notice that, the way the provisioner works, I had to create on storage class for each shared disk)

If everything goes according to plan, applying a test like the one below will make volumes appear to meet the claims and the SUCCESS file to be written on each one:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim-big
spec:
  storageClassName: nfs-big
  accessModes:
    - ReadWriteMany          # Concurrent access
  resources:
    requests:
      storage: 1Mi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim-small
spec:
  storageClassName: nfs-small
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
---
kind: Pod
apiVersion: v1
metadata:
  name: test-pod-big
spec:
  containers:
  - name: test-pod-big
    image: busybox:stable
    command:               # Replace the default command for the image by this
      - "/bin/sh"
    args:                  # Custom command arguments
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: test-volume
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: test-volume
      persistentVolumeClaim:
        claimName: test-claim-big
---
kind: Pod
apiVersion: v1
metadata:
  name: test-pod-small
spec:
  containers:
  - name: test-pod-small
    image: busybox:stable
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: test-volume
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: test-volume
      persistentVolumeClaim:
        claimName: test-claim-small

(Thanks to Albert Weng for the very useful guide)

Your Very Own Messaging Service

Now that we have real storage at our disposal, some real apps would be nice. One of the first things that come to mind when I think about data I'd like to protect is the history of my private conversations, where most spontaneous interactions happen and most ideas get forgotten

If you're a old school person wishing to host your own messaging, I bet you'd go for IRC or XMPP. But I'm more of a late millennial, so it's easier to get drawn to fancy stuff like Matrix, a protocol for federated communication

"And what's federation?", you may ask. Well, to this day, the best video about it I've seen is the one from Simply Explained. Even though it's focused on Mastodon and the Fediverse (We'll get there eventually), the same concepts apply:

As any open protocol, the Matrix specification has a number of different implementations, most famously Synapse (Python) and Dendrite (Go). I've been, however, particularly endeared to Conduit, a lightweight Matrix server written in Rust. Here I'll show the deployment configuration as recommended by its maintainer, Timo Kösters:

kind: PersistentVolumeClaim       # Storage requirements component
apiVersion: v1
metadata:
  name: conduit-pv-claim          
  namespace: choppa
  labels:
    app: conduit
spec:
  storageClassName: nfs-big       # The used storage class (1TB drive)
  accessModes:
    - ReadWriteOnce               # For synchronous access, as we don't want data corruption
  resources:
    requests:
      storage: 50Gi               # Asking for a ~50 Gigabytes volume
---
apiVersion: v1
kind: ConfigMap                   # Read-only data component
metadata:
  name: conduit-config
  namespace: choppa
  labels:
    app: conduit
data:
  CONDUIT_CONFIG: ''              # Leave empty to inform that we're not using a configuration file
  CONDUIT_SERVER_NAME: 'choppa.xyz'                   # The server name that will appear after registered usernames 
  CONDUIT_DATABASE_PATH: '/var/lib/matrix-conduit/'   # Database directory path. Uses the permanent volume
  CONDUIT_DATABASE_BACKEND: 'rocksdb'                 # Recommended for performance, sqlite alternative
  CONDUIT_ADDRESS: '0.0.0.0'                          # Listen on all IP addresses/Network interfaces
  CONDUIT_PORT: '6167'                                # Listening port for the process inside the container
  CONDUIT_MAX_CONCURRENT_REQUESTS: '200'              # Network usage limiting options
  CONDUIT_MAX_REQUEST_SIZE: '20000000'                # (in bytes, ~20 MB)
  CONDUIT_ALLOW_REGISTRATION: 'true'                  # Allow other people to register in the server (requires token)
  CONDUIT_ALLOW_FEDERATION: 'true'                    # Allow communication with other instances
  CONDUIT_ALLOW_CHECK_FOR_UPDATES: 'true'   
  CONDUIT_TRUSTED_SERVERS: '["matrix.org"]'
  CONDUIT_ALLOW_WELL_KNOWN: 'true'                      # Generate "/.well-known/matrix/{client,server}" endpoints
  CONDUIT_WELL_KNOWN_CLIENT: 'https://talk.choppa.xyz'  # return value for "/.well-known/matrix/client"
  CONDUIT_WELL_KNOWN_SERVER: 'talk.choppa.xyz:443'      # return value for "/.well-known/matrix/server"
---
apiVersion: v1
kind: Secret                      # Read-only encrypted data
metadata:
  name: conduit-secret
  namespace: choppa
data:
  CONDUIT_REGISTRATION_TOKEN: bXlwYXNzd29yZG9yYWNjZXNzdG9rZW4=    # Registration token/password in base64 format
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: conduit-deploy
  namespace: choppa
spec:
  replicas: 1                     # Keep it at 1 as the database has an access lock
  strategy:
    type: Recreate                # Wait for replica to be terminated completely befor creating a new one
  selector:
    matchLabels:
      app: conduit
  template:
    metadata:
      labels:
        app: conduit
    spec:
      containers:
        - name: conduit
          image: ancapepe/conduit:next # fork of matrixconduit/matrix-conduit:next
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 6167                 # Has to match the configuration above
              name: conduit-port
          envFrom:                                # Take all data from a config file 
            - configMapRef:                       # As environment variables
                name: conduit-config
          env:                                    # Set environment variables one by one
            - name: CONDUIT_REGISTRATION_TOKEN
              valueFrom:
                secretKeyRef:                     # Take data from a secret
                  name: conduit-secret            # Secret source
                  key: CONDUIT_REGISTRATION_TOKEN # Data entry key
          volumeMounts:
            - mountPath: /var/lib/matrix-conduit/ # Mount the referenced volume into this path (database)
              name: rocksdb
      volumes:
        - name: rocksdb                           # Label the volume for this deployment
          persistentVolumeClaim:
            claimName: conduit-pv-claim           # Reference volumen create by the claim
---
apiVersion: v1
kind: Service
metadata:
  name: conduit-service
  namespace: choppa
spec:
  selector:
    app: conduit
  ports:
    - protocol: TCP
      port: 8448                                  # Using the default port for federation (not required)
      targetPort: conduit-port

One thing you should know is that the K8s architecture is optimized for stateless applications, which don't store changing information within themselves and whose output depend solely on input from the user or auxiliary processes. Conduit, on the contrary, is tightly coupled with its high-performance database, RocksDB, and has stateful behavior. That's why we need to take extra care by not replicating our process in order to prevent data races when accessing storage

There's actually a way to run multiple replicas of a stateful pod, that requires different component types to create one volume copy for each container and keep all them in sync. But let's keep it "simple" for now. More on that in a future chapter

We introduce here a new component type, Secret, intended for sensible information such as passwords and access tokens. It requires that all data is set in a Base64-encoded format for obfuscation, which could be achieved with:

$ echo -n mypasswordoraccesstoken | base64                                                                                                                                                            
bXlwYXNzd29yZG9yYWNjZXNzdG9rZW4=

You might also have noticed that I jumped the gun a bit and acquired a proper domain for myself (so that my Matrix server looks nice, of course). I've found one available at a promotional price on Dynadot, which now seems even more like a good decision, considering how easy is to bend Cloudflare. Conveniently, the registrar also provides a DNS service, so... 2 for the price of one:

The CNAME record is basically an alias: the subdomain talk.choppa.xyz is simply redirected to choppa.xyz and just works as a hint for our reverse proxy from the previous chapter:

apiVersion: networking.k8s.io/v1                    
kind: Ingress                                     # Component type
metadata:
  name: proxy                                     # Component name
  namespace: choppa                               # You may add the default namespace for components as a paramenter
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: traefik
status:
  loadBalancer: {}
spec:
  ingressClassName: traefik
  tls:
  - hosts:
      - choppa.xyz
      - talk.choppa.xyz
    secretName: certificate      
  rules:                                          # Routing rules
  - host: choppa.xyz                              # Expected domain name of request, including subdomain
    http:                                         # For HTTP or HTTPS requests
      paths:                                      # Behavior for different base paths
        - path: /                                 # For all request paths
          pathType: Prefix
          backend:
            service:
              name: welcome-service               # Redirect to this service
              port:
                number: 8080                      # Redirect to this internal service port
        - path: /.well-known/matrix/              # Paths for Conduit delegation
          pathType: ImplementationSpecific
          backend:
            service:
              name: conduit-service
              port:
                number: 8448
  - host: talk.choppa.xyz               # Conduit server subdomain
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: conduit-service
              port:
                number: 8448

If your CONDUIT_SERVER_NAME differs from the [sub]domain where your server is hosted, other Matrix federated instances will look for the actual address in Well-known URIs. Normally you'd have to serve them manually, but Conduit has a convenient automatic delegation feature that helps your set it up with the CONDUIT_WELL_KNOWN_* env variables

Currently, at version 0.8, automatically delegation via env vars is not working properly. Timo has helped me debug it (Thanks!) at Conduit's Matrix room and I have submitted a patch. Let's hope it gets fixed for 0.9. Meanwhile, you may use the custom image I'm referencing in the configuration example

Update: Aaaaaand... it's merged!

With ALL that said, let's apply the deployment and hope for the best:

(Success !?)

With your instance running, you can now use Matrix clients [that support registration tokens] like Element or Nheko to register your user, login and have fun!

See you soon

The Home Server Journey - 3: An Actually Global "Hello"

Beppe — Wed, 18 Sep 2024 12:48:32 +0000

Hi again

For starters, I apologize for setting up the wrong expectations: chapter 1 anticipated all that stuff about external IPs and name servers [like it would be immediately necessary], but event in chapter 2 we still only accessed applications inside the local network

In my defense, I wished to make all the prerequisites clear from the get-go, so that there where no surprises about what you were getting into. Anyway, let's address that today, sooner rather than later, shall we?

Your Virtual Home Receptionist

The reason why you can't simply type http://<my WAN IP>:8080 or http://mybeautifuldomain.com:8080 to view the test page you're running is due to K3s (and Kubernetes implementations in general) not exposing pods by default. And that's the sensible decision, considering that not every application is supposed to be reached from outside, like a database providing storage for your backend service

Those access restrictions are created with the container runtime using iptables to set network rules. I've learned it the hard way when K3s conflicted with the nftables system service I had already running (fixed by disabling the latter). Time to update to new tools, containerd!

Moreover, while it's obvious where external requests should go to when we have a single application, how to handle many containers exposed to Internet access? For instance, it's not practical or sometimes even possible to demand an explicit port to select different Web services (e.g. mydomain.com:8081 and mydomain.com:8082), being the standard to use the default 80 and 443 ports for HTTP and HTTPS, respectively

With such requirement, it's common to share a single IP address (host and port) using paths (e.g. mydomain.com/app1 and mydomain.com/app2) or subdomains (e.g. app1.mydomain.com and app2.mydomain.com) to address a particular process. To achieve that, messages have to be interpreted for rerouting [to a local port] before reaching their intended target, a task performed by what is known as a reverse proxy

In K8s terminology, the component responsible for that functionality is called Ingress. It's actually a common interface for different implementations named Ingress Controllers, of which it's usually recommended (if you're not doing anything fancy or advanced) to use the NGINX-based one, as it's officially maintained

I emphasize usually because K3s is different and comes with a Traefik-based ingress controller by default. Taking that into account, as much as I like NGINX outside the container's world, I'd rather keep things simple and use what's already in place

It's totally fine to use Ingress-Nginx, tough. Just get into the GitHub repository and follow the instructions. I've used it myself before running into problems with upgrades (but you're not that dumb, right?). Be aware that all components will be created in their own predefined namespace, valid system-wide

Now we may use the provided controller to set our ingress rules:

apiVersion: networking.k8s.io/v1                    
kind: Ingress                                     # Component type
metadata:
  name: proxy                                     # Component name
  namespace: test                                 # You may add the default namespace for components as a paramenter
status:
  loadBalancer: {}
spec:
  ingressClassName: traefik                       # Type of controller being used    
  rules:                                          # Routing rules
  - host: choppaserver.dynv6.net                  # Expected domain name of request, including subdomain
    http:                                         # For HTTP or HTTPS requests (standard ports)
      paths:                                      # Behavior for different base paths
        - path: /                                 # For all request paths
          pathType: Prefix
          backend:
            service:
              name: welcome-service               # Redirect to this service
              port:
                number: 8080                      # Redirect to this internal service port

As now the ingress exposes the service to the external network, it's not required to set the configuration type: LoadBalancer for welcome-service, as done on the previous chapter, which changes its behavior to the default ClusterIP

Save it to a file, apply the manifest with kubectl, and in the case everything is correct (including the network instructions from the first article), the test page should be accessible via domain name OR you get something like this:

Which is just the browser being picky with non-secure access, using http:// instead of https:// or not having valid certificates (more on that later). If you want to check the contents of the page, just be a pro for now and use curl:

$ curl "http://choppaserver.dynv6.net"                                                                                                                                                                
<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="welcome.txt">welcome.txt</a></li>
</ul>
<hr>
</body>
</html>
$ curl "http://choppaserver.dynv6.net/welcome.txt"                                                                                                                                                     
Hello, world!

(Almost the same... Who needs browsers anyway?)

Papers, please

Seriously, though, SSL certificates are important nowadays [with HTTPS being ubiquitous], and we need to get one. The easiest way is to let them be automatically generated by cert-manager. To cut it short, get the latest version of the manifest file from GitHub or simply install directly from the download link:

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/<desired version>/cert-manager.yaml
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created

(Quite a few pieces, huh?)

Wait for the manager pods to complete their initialization:

The manifest not only creates many standard K8s resources but also defines new custom ones, like the ClusterIssuer we have to manually add now for each environment (only one in our case):

apiVersion: cert-manager.io/v1      # API service created by cert-manager
kind: ClusterIssuer                 # Custom component type
metadata:
 name: letsencrypt
 namespace: cert-manager
spec:
 acme:
   # The ACME server URL
   server: https://acme-v02.api.letsencrypt.org/directory
   # Email address used for ACME registration
   email: <your e-mail here>
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt
   # Enable the HTTP-01 challenge provider
   solvers:
   - http01:
       ingress:
         class:  traefik            # Ingress controller type

(As always, save the .yaml and apply. A ClusterIssuer is not namespace-scoped and can be used by Certificate resources in any namespace)

Here we're using the ACME challenge with an HTTP01 solver to get a valid result from the Let's Encrypt certificate authority, and I honestly can't explain to you properly what all that means. However, one thing that I know is that this particular solver requires the ability to open port 80 to receive requests. It's common for system ports (below 1024) to be restricted by your ISP, and if that's your case a DNS01 solver might be useful. Please consult the documentation for more

Finally, we can edit our ingress manifest to define the domains requiring a certificate:

apiVersion: networking.k8s.io/v1                    
kind: Ingress                                     # Component type
metadata:
  name: proxy                                     # Component name
  namespace: test                                 # You may add the default namespace for components as a paramenter
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt   # Reference the utilized ClusterIssuer name 
    kubernetes.io/ingress.class: traefik          # Ingress controller type (yes, again)
status:
  loadBalancer: {}
spec:
  ingressClassName: traefik                       # Ingress controller type
  tls:                                            # Certificate options
  - hosts:                                        # List of valid hosts
      - choppaserver.dynv6.net
    secretName: certificate                       # Secret component that will store the certificates      
  rules:                                          # Routing rules
  - host: choppaserver.dynv6.net                  # Expected domain name of request, including subdomain
    http:                                         # For HTTP or HTTPS requests
      paths:                                      # Behavior for different base paths
        - path: /                                 # For all request paths
          pathType: Prefix
          backend:
            service:
              name: welcome-service               # Redirect to this service
              port:
                number: 8080                      # Redirect to this internal service port

After a while, a secret with the name defined in secretName should appear in the namespace you're using:

And the browser should stop complaining about the lack of security in your Web page:

Congrats! Now you the actual world can hear you say "hello". For more information on setting up cert-manager, see this guide (or this one for Ingress-Nginx)

See you next time

The Home Server Journey - 2: The Control Room

Beppe — Tue, 17 Sep 2024 00:47:49 +0000

I confess that not adding some application deployment to our first article left me a bit frustrated, but I was unsure about it's length, even more so for an author whose reputation alone wouldn't [yet] make people pay more attention that what's natural. But let's fix that right away and get something running.

Hello, World!

As we're talking about Internet servers, what better than a Web page? I've found a very fitting test container in jdkelley's simple-http-server, but there was an issue: no image available for the ARM architecture. That's the only reason why I had to build one and publish it to my own repository, I swear

If those concepts related to containers are still confusing to you, I recommend following Nana Janashia's tutorials and crash courses. She's an amazing teacher and I wouldn't be able to quickly summarize all the knowledge she provides. Her Docker lectures compilation is linked below:

With that out of the way (thanks, Nana), let's use our test image in our first Kubernetes deployment, where containers are encapsulated as pods. From now on I'll be abusing comments inside the YAML-format manifest files as it's an easier and more compact way to describe each parameter:

apiVersion: v1                        # Specification version
kind: ConfigMap                       # Static/immutable data/information component
metadata:                             # Identification attributes
  name: welcome-config                  # Component name
data:                                 # Contents list
  welcome.txt: |                        # Data item name and contents
    Hello, world!
---
apiVersion: apps/v1
kind: Deployment                      # Immutable application management component
metadata:
  name: welcome-deploy                # Component name
  labels:                             # Markers used for identification by other components
    app: welcome                                      
spec:
  replicas: 2                         # Number of application pod instances
  selector:
    matchLabels:                        # Markers searched for in other components
      app: welcome                                    
  template:                           # Settings for every pod that is part of this deployment
    metadata:
      labels:
        app: welcome
    spec:
      containers:                         # List of deployed container in each pod replica
      - name: welcome                                 # Container name
        image: ancapepe/http-server:python-latest     # Image used for this container
        ports:                                        # List of exposed network ports for this container
        - containerPort: 8000                           # Port number
          name: welcome-http                            # Optional port label for reference 
        volumeMounts:                                 # Data storages accessible to this container
          - name: welcome-volume                        # Storage name
            mountPath: /serve/welcome.txt               # Storage path inside this container
            subPath: welcome.txt                        # Storage path inside the original volume
      volumes:                            # List of volumes available for mounting
      - name: welcome-volume                # Volume name
        configMap:                          # Use a ConfigMap component as data source
          name: welcome-config              # ConfigMap reference name
---
apiVersion: v1
kind: Service                         # Internal network access component
metadata:
  name: welcome-service                 # Component name
spec:
  type: LoadBalancer                    # Expose service to non-K8s processes
  selector:                             # Bind to deployments with those labels
    app: welcome
  ports:                                # List of exposed ports
    - protocol: TCP                       # Use TCP Internet protocol
      port: 8080                          # Listen on port 8080
      targetPort: welcome-http            # Redirect trafic to this container port

(Copy it to a text editor and save it to something like welcome.yaml)

Among other things, Kubernetes are a solution for horizontal scaling: if your application process is bogged down by many requests, you may instantiate extra copies of it (replicas) in order to server a larger demand, even dynamically, but here we'll work with a prefixed amount. Moreover, see how services not only centralize access to all pods of a deployment, but allow us to use a different connection port than the one defined in the Docker image

A concept you should have in mind when working with K8s is idempotence: manifests such as above don't describe a sequence of operations to set up your pods and auxiliary components, but the desired final state, from which the operations (to either create or restore the deployment) are automatically defined. That way, trying to re-apply an already [successfully] applied configuration will result in no changes to the cluster

The time has come to get our hands dirty. Surely you may submit your YAML files by copying each new version to one of your nodes and invoking k3s kubectl via SSH, but how about doing that from the comfort of your desktop, where you have originally edited and saved the manifests?

It is possible to install kubectl independently of the K8s cluster (follow the instructions for each platform), but it won't work out-of-the-box, as your local client doesn't know how to find the remote cluster yet:

$ kubectl get nodes                                                                                                                                                                                  
E0916 18:45:31.872817   11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0916 18:45:31.873268   11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0916 18:45:31.875167   11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0916 18:45:31.875837   11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0916 18:45:31.877524   11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

In order to get that information, log into one of your cluster machines and get the system configuration:

[ancapepe@ChoppaServer-1 ~]$ k3s kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED

(Here authentication data is omitted for safety reasons. in order to show the complete configuration, add the --raw option to the end of the command)

Copy the raw output of the command, change the server IP (just the 127.0.0.1) to match your master node's one, and overwrite your desktop's kubeconfig file (on Linux, the default location is <user home directory>/.kube/confg) with these modified contents. Now you should be able to use the control client successfully:

$ kubectl get nodes                                                                                                                                                                                
NAME              STATUS     ROLES                  AGE   VERSION
odroidn2-master   Ready      control-plane,master   9d    v1.30.3+k3s1
rpi4-agent        Ready      <none>                 9d    v1.30.3+k3s1

Finally, apply our test deployment:

$ kubectl apply -f welcome.yaml
configmap/welcome-config created
deployment.apps/welcome-deploy created
service/welcome-service created
$ kubectl get pods                                                                                                                                                        
NAME                             READY   STATUS    RESTARTS   AGE
welcome-deploy-5bd4cb78b-hvj9z   1/1     Running   0          75s
welcome-deploy-5bd4cb78b-xb99p   1/1     Running   0          75s

If you regret your decision (already?), that operation may be reverted using the same manifest:

$ kubectl delete -f welcome.yaml                                                                                                                                           
configmap "welcome-config" deleted
deployment.apps "welcome-deploy" deleted
service "welcome-service" deleted
$ kubectl get pods                                                                                                                                                        
NAME                             READY   STATUS        RESTARTS   AGE
welcome-deploy-5bd4cb78b-hvj9z   1/1     Terminating   0          114s
welcome-deploy-5bd4cb78b-xb99p   1/1     Terminating   0          114s
# A little while later
$ kubectl get pods 
No resources found in default namespace.

As you can see, pods of a given deployment get a random suffix attached to their names, in order to differentiate them. Also, all Kubernetes are organized in namespaces to help with resource management. If not informed as a kubectl command option or inside the file itself, the default namespace is used

In order to perform the same operation with a particular namespace, create it first (if not created already):

$ kubectl create namespace test                                                                                                                                          
namespace/test created
$ kubectl apply -f welcome.yaml --namespace=test                                                                                                                           
configmap/welcome-config created
deployment.apps/welcome-deploy created
service/welcome-service created
$ kubectl get all --namespace=test                                                                                                                                         
NAME                                 READY   STATUS    RESTARTS   AGE
pod/welcome-deploy-5bd4cb78b-5df7d   1/1     Running   0          16s
pod/welcome-deploy-5bd4cb78b-9cpdj   1/1     Running   0          16s

NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP                 PORT(S)          AGE
service/welcome-service   LoadBalancer   10.43.194.209   192.168.3.10,192.168.3.11   8080:31238/TCP   16s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/welcome-deploy   2/2     2            2           16s

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/welcome-deploy-5bd4cb78b   2         2         2       16s

(See how namespaces allow to easily select all components that we wish to monitor)

With a successful deployment, our Web page should be displayed by accessing one of the external IPs from the LoadBalancer service (which are the node IPs) on any browser (generally you have to make the usage of HTTP over HTTPS explicitly):

(Opening the listed file works too!)

Congrats! Go add it to your list of LinkedIn's skills (just kidding)

Again, if you're interested in understanding all those concepts in more depth, in order to follow along this and the next chapters of our guide, Nana comes to the rescue again with her Kubernetes tutorial:

A nicer view

That's all well and good, but I bet you can imagine how, as you're developing and running more and more components, issuing commands and keeping track of stuff using kubectl gets quite repetitive. One may open multiple terminals running the get command with --watch as option for automatic updates, but switching between windows is not that great either

Thankfully, for whoever likes that approach, there are friendlier GUIs to help manage yours clusters and deployments. Rancher, developed by the same folks from K3s, is one of them, but honestly I've found too complicated to set up. In my previous job I came across an alternative that I consider much more practical: Lens

After installing it, if you're interested, there's not much to do to get it running for your cluster. Select the option to add a new one from a kubeconfig and paste the contents from your local kubectl configuration:

If everything is correct, you'll be able to access it, visualize your resources, edit manifest files, and even use the integrate terminal to create new components with kubectl (There doesn't seem to be a widget for creation, only modification or deletion of existing components):

(I don't know about you, but that's what I'll be using from now on)

That's it for now! Thanks for reading and let's start making some real-world stuff next time

Chapter 3

The Home Server Journey - 1: Motivation and Approach

Beppe — Sun, 15 Sep 2024 12:22:52 +0000

Inspired by the recent Twitter/X situation here in Brazil (no strong opinions on Elon Musk and all this drama, but it surely sucks to lose access to an useful service) and by an important article by Ceadda of Mercia, I've decided to set an old project of mine in motion.

Truth is, we grow so used to many readily accessible (even if not completely free) platforms on the Internet that we end up taking this ability to communicate and inform ourselves for granted. We easily forget how liberty has to be conquered every day, and that has to be backed by power i.e. the capacity to do what we want to do

Anyway, there are plenty of speeches and commentary around that would do a much better job than me in convincing you of that reality, not to mention the risks of providing all your data directly to third-parties. So here I intend to concentrate where I can add something meaningful: give you some ideas and guidance on how you may own your Internet life a bit more

Living in reality is knowing where things come from and where they go to

Social media, messaging, videos, photos, e-mail, blogs like this one and every other thing that we access on the Internet are all hosted out there, on The Cloud. However, as the saying goes, "'The Cloud' is just someone else's computer". And why couldn't it be YOUR computer? Well, it could, if you're willing to make some sacrifices

You won't be able to run e.g. a personal Facebook or Instagram server from home, but if you care more about the functionality they provide, well, you may be able to actually own something close to it. Of course your friends would have to come over to your place, and many probably won't. That's part of what I meant by "make some sacrifices": your personalized Web presence will be a bit lonely until it gets some traction

But, let's be real, if you're looking into building homeservers having almost no friends is probably the case already... I say it from experience

Ok, ok, maybe you DO have quite some friends [on the Internet] and don't want to give up interacting with them. You'd still find usefulness in having your own "cloud" storage/backup, or a less censorable place to publish your ideas

I'll try to show a bit of everything. Pick whatever suits you

The meat and potatoes of the issue

I don't plan on being condescending to the point of assuming you don't know the distinction between hardware as software. You might have already figured out that those personal software services would need dedicated hardware to be run, and it would be a good thing to have a relatively low-cost solution, so that affording a brand new Intel x86 PC (calm down, Apple M1 users) to keep turned on 24/7 is not a requirement

A common approach that I have adopted is resorting to ARM-based single board computers. They are cheaper, consume less power and take less space inside my modest house. Below I display my humble initial setup of an Hardkernel Odroid N2+ and a well-know Raspberry Pi 4:

(Size doesn't matter if you know how to use it, or so they say)

I have also attached one spare hard drive to each one of them using external USB 3.0 to SATA adapter cases, so that we have larger and more reliable storage. Besides, as you might have noticed, there is no screen, keyboard or mouse there, because I'll be managing everything software-wise from my desktop, over the local network (LAN)

Please make sure you have the correct/recommended power supply connected to each board, specially when using external devices powered through USB. I've just had some undervoltage issues with the Raspberri Pi for not using the ideal one

Both machines are running the minimal ARM version of Manjaro Linux. As a long-time user of Manjaro on my PC, I found it easier to let manjaro-arm-installer choose the right configuration for each board, download the image and write it to its respective SD card. There are plenty of options for operating systems, with it's own installation instructions, but generally you'll be taking and .img file and flashing it to the card with some tool like Etcher or dd. Take a look at what you like the most (preferably some Linux CLI version) or just use the same I did, you'll have to consult extra documentation regardless, as I can't give you all the details without making this article too long for my taste

Notes on networking

Needless to say, your server machines need Internet connection, preferably Ethernet, so that it can be detected without logging into each one locally and configuring Wi-Fi. Another [not so obvious] requirement is that at least one of them (your master) has to be exposed outside of your LAN, so that external devices can reach it (and you can imagine why that's not possible by default). If you have access to your router configuration (I DO hope you have), usually your NAT or Firewall settings, that's achieved by forwarding the ports used by the services you got running e.g. port 80/TCP for HTTP and 443/TCP for HTTPS (Web pages):

(If you're using standard IPv4, the machine's host address is valid exclusively inside your LAN, and only the router/gateway has also a global WAN IP. Even with all the IPv6 goodies enabled, reachability from outside might be limited for security reasons)

I'd also recommend reserving static IPs for each one of your server's network interfaces, so that we don't get any unpleasant surprises with changing addresses:

After redirecting incoming traffic from router WAN address to server machine LAN address on the given ports, it's time to make yourself easier to find. Passing your WAN IP around so that people may type it into the browser bar would already work, but that's both hard to remember and unreliable, as addresses attributed to your machine usually change from time to time (lucky you if a static global IP is available).

Then why not give your virtual place a proper memorable name, like "mybeautifulsite.com", that everybody could know? Well, that what name service (DNS) is for.

Sadly domains are one aspect of Internet centralization that still cannot be trivially avoided (I know about blockchain DNS, but who supports it?), so you have to rely on a third-party service (e.g. Cloudflare) in order to propagate your hostname across the Web. Generally you have to pay a registrar for the right to use a certain name (it's a way to prevent the ambiguity from 2 hosts using the same one), but for testing purposes you may rely on free DNS services (such as No-IP or Dynv6) that provide you a somewhat uglier subdomain alternative:

(They were so kind to tell the entire world my router's IP)

If you don't manage to get your hands on a static WAN IP, it's important that your DNS service has support for being automatically updated about changes in your dynamic address. Your router itself might be able to notify the DDNS, but if the proper protocol option is not available you may run a refresh client from one of your computers:

(Again, there's too much variation across routers and services for me to cover it here)

Just don't go trying to reach your server via DNS right after naming it. Propagation takes a while, as many name servers across the world have to be informed. If getting anxious, you may try to check the state of that process using some Web service or nslookup command on your Linux system (also available under the same name on Windows, apparently, but who in his/her right mind still uses it?)

Becoming the helmsman

If you've ever tried using a non-mainstream Linux distro, you probably know the pain of looking for a package, not finding it even in a third-party repository, trying to compile it from source and failing miserably over and over (Been there. Done that). Manjaro, as an Arch-based OS, at least has the very welcome access to the AUR, a community repository for compilation scripts that automate the process for us mere humans, but not all are regularly maintained and many still break, for a plethora of reasons that you eventually have to figure out and fix in a case-by-case basis

A way to avoid those headaches would be welcome

For a long time, I've been the kind of person that sees Linux's Dependency Hell as a feature rather than a bug: "It's efficient for storage", "It forces developers to keep their software updated and in sync with others", "Library duplication is r*tarded", I've been telling myself. I still find it somewhat true, and don't see sandboxed solutions like Flatpak, Snap and AppImage with good eyes, or even virtual environments for more than prototyping... Ok, maybe I'm still THAT kind of person, but I've found containers to be pretty neat

Having worked a bit with Docker in a previous job, I was convinced of how useful this level of isolation is for automated testing and deployment of applications, being able to change system settings and replicate them across countless [physical or virtual] machines. You really start visualizing containers as loosely coupled independent entities exchanging messages, which in turn naturally evolves into the main focus of my approach here: orchestration

And to cut it short, I'd guess in 9 of 10 cases that word would mean using one of the many implementations of Kubernetes (K8s for short). Not that there aren't other solutions, but I'm not knowledgeable enough to evaluate them properly and I went for what is more popular i.e. with more available documentation. Let me make it clear that this guide is a mish-mash of a number of tutorials and Stack Overflow threads, compiled, updated and adjusted by an amateur who is really stubborn about making all work together. So please bear in mind it might not be the most elegant solution

For tiny single-board computers, there is a lightweight implementation of K8s called K3s, which has worked well for me. If you're using Arch-based systems (assuming that you have actually gotten the machines set up), keep following my steps for K3s installation. On Debian-based systems like Ubuntu or Raspbian, you'd be probably better off following NetworkChuck's guide linked below. Otherwise, you'll have to do your own research:

As already mentioned, here I have accessed my servers from a desktop, via SSH, and installed K3s using yay, but you may use any other package manager with AUR access like pikaur or pamac:

$ ssh ancapepe@192.168.3.10    
ancapepe@192.168.3.10's password:  
Welcome to Manjaro ARM 
~~Website: https://manjaro.org 
~~Forum:   https://forum.manjaro.org/c/arm 
~~Matrix:  #manjaro-arm-public:matrix.org 
Last login: Wed Sep 11 18:41:48 2024 from 192.168.3.8 
[ancapepe@ChoppaServer-1 ~]$ sudo pacman -S yay 
[sudo] password for ancapepe:  
resolving dependencies...
looking for conflicting packages...

Packages (1) yay-12.3.1-1

Total Download Size:   3.31 MiB
Total Installed Size:  8.88 MiB

:: Proceed with installation? [Y/n] y
:: Retrieving packages...
 yay-12.3.1-1-aarch64                                                                                 3.3 MiB  2.33 MiB/s 00:01 [#############################################################################] 100%
(1/1) checking keys in keyring                                                                                                  [#############################################################################] 100%
(1/1) checking package integrity                                                                                                [#############################################################################] 100%
(1/1) loading package files                                                                                                     [#############################################################################] 100%
(1/1) checking for file conflicts                                                                                               [#############################################################################] 100%
(1/1) checking available disk space                                                                                             [#############################################################################] 100%
:: Processing package changes...
(1/1) installing yay                                                                                                            [#############################################################################] 100%
Optional dependencies for yay
    sudo: privilege elevation [installed]
    doas: privilege elevation
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[ancapepe@ChoppaServer-1 ~]$ sudo pacman -S containerd fakeroot
...
(More of the same)
...
[ancapepe@ChoppaServer-1 ~]$ yay -S k3s-bin 
AUR Explicit (1): k3s-bin-1.30.3+k3s1-1
...
(Too much to show here. Just follow along)
...

(Do that for each board. Teaching how to use a terminal is outside the scope of this guide)

Hopefully everything went well and you have K3s installed on all your servers, but it's still not doing anything. Kubernetes applies the concepts of master and worker nodes: masters coordinate workers so that all can work as a single system, namely a Cluster; both may dot the heavy lifting of running applications; Node is simply the abstraction used for each [physical or virtual] computer

[AFAIK] With K3s things are kept simple and only your first node will run as a master, or [K3s] server in this case (yes, terminology gets confusing). In order to initialize it as such when your main machine boots, open your K3s init file for modification with:

[ancapepe@ChoppaServer-1 ~]$ sudo systemctl edit k3s --full

(The file will be opened with the application defined in your $EDITOR environment variable, be it vim, emacs or even nano, a competition I don't want to be dragged into. Talking about undesired fights, I know systemd is not the only init system out there, but I haven't tried this with distros that use OpenRC or Upstart)

Edit the line starting with ExecStart to look like this:

ExecStart=/usr/bin/k3s server --write-kubeconfig-mode=644 --node-name <your chosen master node name>

And finally enable the initialization service with:

[ancapepe@ChoppaServer-1 ~]$ sudo systemctl enable containerd
[ancapepe@ChoppaServer-1 ~]$ sudo systemctl enable k3s --now

Not quite finally, actually. Before you restart your board to bring up K3s, you have to make sure that the cgroups feature, required by all containerized applications, is enabled and available in your system:

[ancapepe@ChoppaServer-1 ~]$ grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
[ancapepe@ChoppaServer-1 ~]$ cat /proc/cgroups 
#subsys_name    hierarchy       num_cgroups     enabled 
cpuset  0       51      1 
cpu     0       51      1 
cpuacct 0       51      1 
blkio   0       51      1 
memory  0       51      0 
devices 0       51      1 
freezer 0       51      1 
net_cls 0       51      1 
perf_event      0       51      1 
net_prio        0       51      1 
pids    0       51      1 
rdma    0       51      1

(TL;DR: Cgroups system v2 running, memory cgroup disabled)

If v1 system or none at all appears, you're probably using an old kernel, so update it. If the memory cgroup is not enabled, you have to find the particular way in which you switch it on for your device (for Raspberry Pi, that means adding cgroup_enable=memory at the end of the single line in the /boot/cmdline.txt file)

Moreover, we must tell containerd, the daemon that manages your containers behind the curtains, to use the right cgroup for its main process, runc, by running the commands below:

# Create a containerd configuration directory
[ancapepe@ChoppaServer-1 ~]$ sudo mkdir -p /etc/containerd
# Write the default settings to a configuration file
[ancapepe@ChoppaServer-1 ~]$ sudo containerd config default | sudo tee /etc/containerd/config.toml
# Find and replace the given string inside the file
[ancapepe@ChoppaServer-1 ~]$ sudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml

(The # character at the beginning of a line denotes a comment, which is not supposed to be typed. For more information on what you're doing there consult this article and the sed command documentation)

Now you may reboot the master node, and hopefully everything will start running properly. Your SSH connection will be terminated so it should be established again when the device finishes starting up:

[leonardo@ChoppaServer-1 ~]$ sudo reboot 
[sudo] password for leonardo:  

Broadcast message from root@ChoppaServer-1 on pts/1 (Sat 2024-09-14 20:31:07 -03): 

The system will reboot now! 

[leonardo@ChoppaServer-1 ~]$ Read from remote host 192.168.3.10: Connection reset by peer 
Connection to 192.168.3.10 closed. 
client_loop: send disconnect: Broken pipe 
$ ssh leonardo@192.168.3.10    
leonardo@192.168.3.10's password: 
Welcome to Manjaro ARM 
~~Website: https://manjaro.org 
~~Forum:   https://forum.manjaro.org/c/arm 
~~Matrix:  #manjaro-arm-public:matrix.org 
Last login: Sat Sep 14 17:30:32 2024 from 192.168.3.8 
[leonardo@ChoppaServer-1 ~]$

(At this point you may check the status of the enabled services using systemctl status)

With the master system set up, let's try connecting our worker nodes to it. That connection is established by sharing a common key or token generated by master and stored in a predetermined file:

# Displays content of the token file
[leonardo@ChoppaServer-1 ~]$ sudo cat /var/lib/rancher/k3s/server/token 
[sudo] password for leonardo:  
K10d4e8b232cbf03832752443a07cc6206e092733c8ae55c0e64407dcb2e1775f45::server:44a1a8b25c4a8f1d8a0d48164eff2303

Copy that file, guard it well, and repeat the same process (since the first ssh command) for each one of your workers, only changing the way the K3s service file is edited, as now it should contain:

ExecStart=/usr/bin/k3s agent --server https://<your master node IP>:6443 --token <your master token> --node-name <your chosen worker node name>

Take your time in repeating all that stuff, just make sure that you follow through. If everything works as intended, from any one of your nodes you'd be able to run the command below successfully:

[leonardo@ChoppaServer-2 ~]$ k3s kubectl get nodes 
NAME              STATUS   ROLES                  AGE    VERSION 
odroidn2-master   Ready    control-plane,master   7d5h   v1.30.3+k3s1 
rpi4-agent        Ready    <none>                 8d     v1.30.3+k3s1

(Yay!)

That's it for now. I hope it wasn't very tiring

Next time we'll actually deploy applications and get a proper way to manage them

Chapter 2