DEV Community: Alex Wellnitz

Highly scalable Minecraft cluster

Alex Wellnitz — Sat, 04 Nov 2023 09:21:05 +0000

Are you planning a very large Minecraft LAN party? Then this article is for you. Here I show you how to set up a highly scalable Minecraft cluster.

What is a Minecraft cluster?

A Minecraft cluster is a Minecraft server network that consists of multiple Minecraft servers. These servers are connected to each other via a network and can therefore be shared. This means that you can play with your friends on a server that consists of multiple servers.

How does a Minecraft cluster work?

A Minecraft cluster consists of several components.

Master database

First, there is the master database. This database allows servers to store data in a central location that all servers can access. Servers store chunks, maps, level.dat, player data, banned players, and more in this database. This database also records which chunk belongs to which server and coordinates communication between servers.

Server

The master database is great for storing data, but not so good at synchronizing data in real time between servers. This is where peer-to-peer communication comes in. Each server establishes a connection to another server so that data between them can be updated in real time. When a player on server A attacks another player on server B, server A sends this data directly to server B so that server B can damage the player and apply any knockback.

Load Balancer

The load balancer is the last component of the cluster. A load balancer is required to distribute players evenly across your servers. A load balancer automatically distributes players between servers to distribute the load evenly across the individual servers.

Why do I need multiple servers?

By having multiple servers, we can distribute the load across multiple servers. This means that we can have more players on our servers without the servers becoming overloaded. With this setup, we can also easily add new servers if we get more players. If the number of players decreases again, the server can be removed again.

Preparation

You should be familiar with Kubernetes and have set up a Kubernetes cluster. I recommend k3s.

You should also be familiar with Helm. I recommend Helm 3.

Installation

First, you should clone the repository.



git clone git@github.com:alexohneander/MultiPaperHelm.git
cd MultiPaperHelm/

I installed the entire setup in a separate namespace. You can create this namespace with the following command.



kubectl create namespace minecraft

Next, we install the Minecraft cluster with Helm.



helm install multipaper . --namespace minecraf

Once the Helm chart is installed, you can view the port of the proxy service.



kubectl describe service multipaper-master-proxy -n minecraft

This port is the port that you need to enter in your Minecraft client.

Configuration

The Helm chart creates several ConfigMaps. In these ConfigMaps, you can customize the configuration of your cluster.

For example, you can set the number of maximum players or change the description of the server.

For more information on the individual config files, see MultiPaper.

Conclusion

With this setup, you can easily set up a highly scalable Minecraft cluster. You can easily add new servers if you get more players and remove them again if the number of players decreases again.

You can test this setup under the following Server Address: minecraft.alexohneander.de:31732

If you have any questions, feel free to contact me on Email or on Matrix.

Writing Backup Scripts with Borg

Alex Wellnitz — Mon, 19 Sep 2022 07:18:46 +0000

Since we all know that the first rule is "no backup, no pity", I'll show you how you can use Borg to back up your important data in an encrypted way with relative ease.

If you do not want to use a second computer, but an external hard drive, you can adjust this later in the script and ignore the points in the instructions for the second computer.

Requirements

2 Linux Computers
Borg
SSH
Storage
More than 5 brain cells

Installation

First we need to install borg on both computers so that we can back up on one and save on the other.

sudo apt install borgbackup

Then we create a Borg repository. We can either use an external target or a local path.

External Target:

borg init --encryption=repokey ssh://user@192.168.2.42:22/mnt/backup/borg

Local Path:

borg init --encryption=repokey /path/to/backup_folder

If you are using an external destination, I recommend that you store your SSH key on the destination.
This way you don't have to enter a password and is simply nicer from my point of view.

Once you have created everything and prepared the script with your parameters, I recommend that you run the script as a CronJob so that you no longer have to remember to back up your things yourself.

crontab example:

#Minute Hour    Day    Month  Day(Week)      command
#(0-59) (0-23)  (1-31)  (1-12)  (1-7;1=Mo)
00 2 * * * /srv/scripts/borgBackup.sh

Automated script

#!/bin/sh

# VARS
BACKUPSERVER="192.168.2.42"
BACKUPDIR="/mnt/backup/borg"

# Here you can either use your external destination or the local path.
# External target
export BORG_REPO="user@$BACKUPSERVER:$BACKUPDIR"

# Local path
# export BORG_REPO=/path/to/backup_folder

# Your repository password must be stored here.
export BORG_PASSPHRASE='S0m3th1ngV3ryC0mpl1c4t3d'

info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }
trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERM

info "Start backup"

#Here the backup is created, adjust it the way you would like to have it.
borg create                    \
    --stats                    \
    --compression lz4          \
    ::'BackupName-{now}'        \
    /etc/nginx                  \
    /home/user

backup_exit=$?

info "Deleting old backups"
# Automatic deletion of old backups
borg prune                          \
--prefix 'BackupName-'              \
    --keep-daily    7              \
    --keep-weekly  4                \
    --keep-monthly  6

prune_exit=$?

# Information on whether the backup worked.
global_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))

if [ ${global_exit} -eq 0 ]; then
    info "Backup and Prune finished successfully"
elif [ ${global_exit} -eq 1 ]; then
    info "Backup and/or Prune finished with warnings"
else
    info "Backup and/or Prune finished with errors"
fi

exit ${global_exit}

Get your data from the backup

First, we create a temporary directory in which we can mount the backup.

mkdir /tmp/borg-backup

Once our mount point is created, we can mount our backup repo.
At this point you must remember that you can use an external destination or a local path.

borg mount ssh://user@192.168.2.42/mnt/backup/borg /tmp/borg-backup

Once our repo is mounted, we can change into the directory and restore files via rsync or cp.

Conclusion

I hope you could understand everything and now secure your shit sensibly. Because without a backup we are all lost!

Why Docker isn't always a good idea Part 1

Alex Wellnitz — Thu, 15 Sep 2022 13:01:24 +0000

To briefly explain the situation:
We have a HAProxy running on a Debian server as a Docker container. This is the entrance node to a Docker Swarm cluster.

Now, in the last few days, there have been several small outages of the websites running in the Docker Swarm cluster. After getting an overview, we noticed that no new connections can be established.

As soon as we restarted the HAProxy, everything went back to normal. After that I did some research on TCP connections and found out that there is a socket limit.

In Linux we have a limit of sockets that can be opened at the same time. At this point, I unfortunately did not understand that this limit refers to a client connection. So we note that a client can establish a maximum of 65535 socket connections to a server.

This limit refers to a range of ports that you release. We had about 35k sockets available on our server (HAProxy). Now the pages are always down when this limit is reached. Thinking back for a moment, we should never get to that limit as it relates to a client. But the problem with us was that Docker's softlayer network didn't route the client address cleanly through the NAT, so everything was coming from one client.

After stopping the Docker container and installing HAProxy natively on the server, we were able to cross that boundary as well.

Assumption

Because we NAT all the requests through the Docker network, the source address is always the same. This is how we reach the socket limit. If we omit the NAT and use HAProxy natively, we do not reach this limit, because the source address is no longer always the same.

Conclusion

With this setup, we get overhead into the system that we don't need. We have an extra abstraction layer, every request has to go through the Docker network and we reach the socket limit. All these points fall away when we use it natively.

If we use a lot of micro services it is important that we use something like Docker, because then we can share the kernel and it makes the deployment much easier.

But if we have only one application that is very important, it is better to keep it simple.

Baremetal CNI Setup with Cilium

Alex Wellnitz — Thu, 20 Jan 2022 13:04:48 +0000

In a freshly set up Kubernetes cluster, we need a so-called CNI. This CNI is not always present after installation.

What is a Container Network Interface (CNI)?

CNI is a network framework that allows the dynamic configuration of networking resources through a group of Go-written specifications and libraries. The specification mentioned for the plugin outlines an interface that would configure the network, provisioning the IP addresses, and mantain multi-host connectivity.

In the Kubernetes context, the CNI seamlessly integrates with the kubelet to allow automatic network configuration between pods using an underlay or overlay network. An underlay network is defined at the physical level of the networking layer composed of routers and switches. In contrast, the overlay network uses a virtual interface like VxLAN to encapsulate the network traffic.

Once the network configuration type is specified, the runtime defines a network for containers to join and calls the CNI plugin to add the interface into the container namespace and allocate the linked subnetwork and routes by making calls to IPAM (IP Address Management) plugin.

In addition to Kubernetes networking, CNI also supports Kubernetes-based platforms like OpenShift to provide a unified container communication across the cluster through software-defined networking (SDN) approach.

What is Cilium?

Cilium is an open-source, highly scalable Kubernetes CNI solution developed by Linux kernel developers. Cilium secures network connectivity between Kubernetes services by adding high-level application rules utilizing eBPF filtering technology. Cilium is deployed as a daemon cilium-agent on each node of the Kubernetes cluster to manage operations and translates the network definitions to eBPF programs.

The communication between pods happens over an overlay network or utilizing a routing protocol. Both IPv4 and IPv6 addresses are supported for cases. Overlay network implementation utilizes VXLAN tunneling for packet encapsulation while native routing happens through unencapsulated BGP protocol.

Cilium can be used with multiple Kubernetes clusters and can provide multi CNI features, a high level of inspection,pod-to-pod connectivity across all clusters.

Its network and application layer awareness manages packet inspection, and the application protocol packets are using.

Cilium also has support for Kubernetes Network Policies through HTTP request filters. The policy configuration can be written into a YAML or JSON file and offers both ingress and egress enforcements. Admins can accept or reject requests based on the request method or path header while integrating policies with service mesh like Istio.

Preparation

For the installation we need the CLI from Cilium.
We can install this with the following commands:

Mac OSx

curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-darwin-amd64.tar.gz{,.sha256sum}
shasum -a 256 -c cilium-darwin-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-darwin-amd64.tar.gz /usr/local/bin
rm cilium-darwin-amd64.tar.gz{,.sha256sum}

Linux

curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}

Install Cilium

You can install Cilium on any Kubernetes cluster. These are the generic instructions on how to install Cilium into any Kubernetes cluster. The installer will attempt to automatically pick the best configuration options for you.

Requirements

Kubernetes must be configured to use CNI
Linux kernel >= 4.9.17

Install

Install Cilium into the Kubernetes cluster pointed to by your current kubectl context:

cilium install

If the installation fails for some reason, run cilium status to retrieve the overall status of the Cilium deployment and inspect the logs of whatever pods are failing to be deployed.

Validate the Installation

To validate that Cilium has been properly installed, you can run

$ cilium status --wait
   /¯¯\
/¯¯\__/¯¯\    Cilium:         OK
\__/¯¯\__/    Operator:       OK
/¯¯\__/¯¯\    Hubble:         disabled
\__/¯¯\__/    ClusterMesh:    disabled
   \__/

DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Containers:       cilium-operator    Running: 2
                  cilium             Running: 2
Image versions    cilium             quay.io/cilium/cilium:v1.9.5: 2
                  cilium-operator    quay.io/cilium/operator-generic:v1.9.5: 2

Run the following command to validate that your cluster has proper network connectivity:

$ cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [k8s-cluster] Creating namespace for connectivity check...
(...)
---------------------------------------------------------------------------------------------------------------------
📋 Test Report
---------------------------------------------------------------------------------------------------------------------
✅ 69/69 tests successful (0 warnings)

Congratulations! You have a fully functional Kubernetes cluster with Cilium. 🎉

Site to Site VPN for Google Kubernetes Engine

Alex Wellnitz — Thu, 06 May 2021 06:37:08 +0000

In this tutorial I will try to explain you briefly and concisely how you can set up a site-to-site VPN for the Google Cloud Network.

Prerequisites

We need 2 virtual machines. The first one on the side of our office and the other one on the side of Google.

Setup OpenVPN Clients

Site-to-Site Client Office Side

We need to install OpenVPN, we do it as follows:

apt install openvpn -y

After that we add our OpenVPN configuration under this path /etc/openvpn/s2s.conf.

s2s.conf

# Use a dynamic tun device.
# For Linux 2.2 or non-Linux OSes,
# you may want to use an explicit
# unit number such as "tun1".
# OpenVPN also supports virtual
# ethernet "tap" devices.
dev tun

# Our OpenVPN peer is the Google gateway.
remote IP_GOOGLE_VPN_CLIENT 

ifconfig 4.1.0.2 4.1.0.1

route 10.156.0.0 255.255.240.0            # Google Cloud VM Network
route 10.24.0.0 255.252.0.0               # Google Kubernetes Pod Network

push "route 192.168.10.0 255.255.255.0"   # Office Network 

# Our pre-shared static key
#secret static.key

# Cipher to use
cipher AES-256-CBC

port 1195

user nobody
group nogroup

# Uncomment this section for a more reliable detection when a system
# loses its connection.  For example, dial-ups or laptops that
# travel to other locations.
 ping 15
 ping-restart 45
 ping-timer-rem
 persist-tun
 persist-key

# Verbosity level.
# 0 -- quiet except for fatal errors.
# 1 -- mostly quiet, but display non-fatal network errors.
# 3 -- medium output, good for normal operation.
# 9 -- verbose, good for troubleshooting
verb 3

log /etc/openvpn/s2s.log

We also have to enable the IPv4 forward function in the kernel, so we go to /etc/sysctl.conf and comment out the following line:

net.ipv4.ip_forward=1

We can then start our OpenVPN client with this command:

systemctl start openvpn@s2s

On the Office side we have to open the port for the OpenVPN client that the other side can connect.

Site-to-Site Client Google Side

When setting up the OpenVPN client on Google's site, we need to consider the following settings when creating it. When we create the machine, we need to enable this option in the network settings:

Also on this side we have to install the OpenVPN client again and then add this config under the path /etc/openvpn/s2s.conf:

# Use a dynamic tun device.
# For Linux 2.2 or non-Linux OSes,
# you may want to use an explicit
# unit number such as "tun1".
# OpenVPN also supports virtual
# ethernet "tap" devices.
dev tun

# Our OpenVPN peer is the Office gateway.
remote IP_OFFICE_VPN_CLIENT 

ifconfig 4.1.0.2 4.1.0.1

route 192.168.10.0 255.255.255.0          # Office Network

push "route 10.156.0.0 255.255.240.0"     # Google Cloud VM Network
push "route 10.24.0.0 255.252.0.0"        # Google Kubernetes Pod Network

# Our pre-shared static key
#secret static.key

# Cipher to use
cipher AES-256-CBC

port 1195

user nobody
group nogroup

# Uncomment this section for a more reliable detection when a system
# loses its connection.  For example, dial-ups or laptops that
# travel to other locations.
 ping 15
 ping-restart 45
 ping-timer-rem
 persist-tun
 persist-key

# Verbosity level.
# 0 -- quiet except for fatal errors.
# 1 -- mostly quiet, but display non-fatal network errors.
# 3 -- medium output, good for normal operation.
# 9 -- verbose, good for troubleshooting
verb 3

log /etc/openvpn/s2s.log

We also have to enable the IPv4 forward function in the kernel, so we go to /etc/sysctl.conf and comment out the following line:

net.ipv4.ip_forward=1

Connection test

Now that both clients are basically configured we can test the connection. Both clients have to be started with systemctl. After that we look at the logs with tail -f /etc/openvpn/s2s-log and wait for this message:

Wed May  5 08:28:01 2021 /sbin/ip route add 10.28.0.0/20 via 4.1.0.1
Wed May  5 08:28:01 2021 TCP/UDP: Preserving recently used remote address: [AF_INET]0.0.0.0:1195
Wed May  5 08:28:01 2021 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed May  5 08:28:01 2021 UDP link local (bound): [AF_INET][undef]:1195
Wed May  5 08:28:01 2021 UDP link remote: [AF_INET]0.0.0.0:1195
Wed May  5 08:28:01 2021 GID set to nogroup
Wed May  5 08:28:01 2021 UID set to nobody
Wed May  5 08:28:11 2021 Peer Connection Initiated with [AF_INET]0.0.0.0:1195
Wed May  5 08:28:12 2021 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Wed May  5 08:28:12 2021 Initialization Sequence Completed

If we can't establish a connection, we need to check if the ports are opened on both sides.

Routing Google Cloud Network

After our clients have finished installing and configuring, we need to set the routes on Google. I will not map the Office side, as this is always different. But you have to route the networks for the Google network there as well.

To set the route on Google we go to the network settings and then to Routes. Here you have to specify your office network so that the clients in the Google network know what to do.

IP-Masquerade-Agent

IP masquerading is a form of network address translation (NAT) used to perform many-to-one IP address translations, which allows multiple clients to access a destination using a single IP address. A GKE cluster uses IP masquerading so that destinations outside of the cluster only receive packets from node IP addresses instead of Pod IP addresses. This is useful in environments that expect to only receive packets from node IP addresses.

You have to edit the ip-masq-agent and this configuration is responsible for letting the pods inside the nodes, reach other parts of the GCP VPC Network, more specifically the VPN. So, it allows pods to communicate with the devices that are accessible through the VPN.

First of all we're gonna be working inside the kube-system namespace, and we're gonna put the configmap that configures our ip-masq-agent, put this in a config file:

nonMasqueradeCIDRs:
  - 10.24.0.0/14  # The IPv4 CIDR the cluster is using for Pods (required)
  - 10.156.0.0/20 # The IPv4 CIDR of the subnetwork the cluster is using for Nodes (optional, works without but I guess its better with it)
masqLinkLocal: false
resyncInterval: 60s

and run kubectl create configmap ip-masq-agent --from-file config --namespace kube-system

afterwards, configure the ip-masq-agent, put this in a ip-masq-agent.yml file:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: ip-masq-agent
  namespace: kube-system
spec:
  template:
    metadata:
      labels:
        k8s-app: ip-masq-agent
    spec:
      hostNetwork: true
      containers:
      - name: ip-masq-agent
        image: gcr.io/google-containers/ip-masq-agent-amd64:v2.4.1
        args:
            - --masq-chain=IP-MASQ
            # To non-masquerade reserved IP ranges by default, uncomment the line below.
            # - --nomasq-all-reserved-ranges
        securityContext:
          privileged: true
        volumeMounts:
          - name: config
            mountPath: /etc/config
      volumes:
        - name: config
          configMap:
            # Note this ConfigMap must be created in the same namespace as the daemon pods - this spec uses kube-system
            name: ip-masq-agent
            optional: true
            items:
              # The daemon looks for its config in a YAML file at /etc/config/ip-masq-agent
              - key: config
                path: ip-masq-agent
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      - key: "CriticalAddonsOnly"
        operator: "Exists"

and run kubectl -n kube-system apply -f ip-masq-agent.yml.

Now our site-to-site VPN should be set up. You should now test if you can ping the pods and if all other services work as you expect them to.

Kubernetes Storage Backup Discussion

Alex Wellnitz — Mon, 12 Apr 2021 07:55:05 +0000

I would be interested to know how you back up your storage and what tools you use.

In our development environment we use a NFS which is backed up via a cronjob every 4 hours.
MySQL and MongoDB are also backed up.

In the live environment we don't use NFS for many reasons.
Here I ask myself how I can backup the single read write volumes.
I have thought of it this way until now: I shut down the deployment with a cronjob, then mount the storage into a backup image, when that is done the deployment is started again.

How do you do that?

Backup MySQL Databases in Kubernetes

Alex Wellnitz — Wed, 03 Mar 2021 14:22:42 +0000

In this post, we will show you how to create a MySQL server backup using Kubernetes CronJobs.

In our case, we do not have a managed MySQL server. But we want to backup it to our NAS, so that we have a backup in case of emergency.
For this we first build a container that can execute our tasks, because we will certainly need several tasks to backup our cluster.

CronJob Agent Container

First, we'll show you our Dockerfile so you know what we need.

FROM alpine:3.10

# Update
RUN apk --update add --no-cache bash nodejs-current yarn curl busybox-extras vim rsync git mysql-client openssh-client 
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/linux/amd64/kubectl && chmod +x ./kubectl && mv ./kubectl /usr/local/bin/kubectl

# Scripts
RUN mkdir /srv/jobs
COPY jobs/* /srv/jobs/

# Backup Folder
RUN mkdir /var/backup
RUN mkdir /var/backup/mysql

Backup Script

And now our backup script which the container executes.

Our script is quite simple, we get all tables with the mysql client, export them as sql file, pack them in a zip file and send them in a 8 hours interval to our NAS.

#!/bin/bash

############# SET VARIABLES #############

# Env Variables
BACKUPSERVER="8.8.8.8" # Backup Server Ip
BACKUPDIR=/var/backup/mysql
BACKUPREMOTEDIR="/mnt/backup/kubernetes/"
HOST="mariadb.default"
NOW="$(date +"%Y-%m-%d")"
STARTTIME=$(date +"%s")
USER=mysqlUser
PASS=mysqlPassword


############# BUILD ENVIROMENT #############
# Check if temp Backup Directory is empty
mkdir $BACKUPDIR

if [ "$(ls -A $BACKUPDIR)" ]; then
    echo "Take action $BACKUPDIR is not Empty"
    rm -f $BACKUPDIR/*.gz
    rm -f $BACKUPDIR/*.mysql
else
    echo "$BACKUPDIR is Empty"
fi

############# BACKUP SQL DATABASES #############
for DB in $(mysql -u$USER -p$PASS -h $HOST -e 'show databases' -s --skip-column-names); do
    mysqldump -u$USER -p$PASS -h $HOST --lock-tables=false $DB > "$BACKUPDIR/$DB.sql";
done

############# ZIP BACKUP #############
cd $BACKUPDIR
tar -zcvf backup-${NOW}.tar.gz *.sql

############# MOVE BACKUP TO REMOTE #############
rsync -avz $BACKUPDIR/backup-${NOW}.tar.gz root@$BACKUPSERVER:$BACKUPREMOTEDIR

# done

Kubernetes CronJob Deployment

Finally we show you the kubernetes deployment for our agent.

In the deployment, our agent is defined as a CronJob that runs every 8 hours.
In addition, we have added an SSH key as a Conifg map so that this can write to the NAS and a certain security is given.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: backup-mariadb
  namespace: default
spec:
  schedule: "0 8 * * *"
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: cronjob-agent
              image: xxx/cronjob-agent
              command: ["bash",  "/srv/jobs/backup-mariadb.sh"]
              volumeMounts:
                - mountPath: /root/.ssh/id_rsa.pub
                  name: cronjob-default-config
                  subPath: id_rsa.pub
                - mountPath: /root/.ssh/id_rsa
                  name: cronjob-default-config
                  subPath: id_rsa
                  readOnly: true
                - mountPath: /root/.ssh/config
                  name: cronjob-default-config
                  subPath: config
          volumes:
            - name: cronjob-default-config
              configMap:
                name: cronjob-default-config
                defaultMode: 256
          restartPolicy: Never