[Homelab AI Project] Ep.2 - Rocky Linux 10.x OS Hardening & Kernel Tuning

[Core] Executive Summary

An in-depth systems engineering guide to turning a budget Mini PC running bare-metal Rocky Linux 10.x into a resilient, high-concurrency database/message-queue node. This post covers optimizing base OS RAM usage under 150MB, securing ports with Fail2ban, configuring physical disk partition isolation, installing base host runtimes (Docker Engine & Docker Compose), and tuning core kernel parameters for Redis and PostgreSQL.

---

1. Introduction: Between Cloud Mazes and Terminal Dark Worlds

Deploying virtualized servers on public clouds like AWS is the industry standard for production environments. However, that convenience comes with a cost: staring at five browser tabs, hunting for hidden options, and clicking dropdown selectors like you're playing a hidden-object game in the AWS console.

To escape that, we retreat to the terminal--an unforgiving, dark space where a single typo can reduce your entire system to ashes. In the end, it's just about what risks you are willing to own and manage. The moment that lone cursor starts blinking, the reality of bare-metal homelabs hits you with a barrage of systems-level constraints:

• IP Instability (DHCP): The home router changing the server IP on a whim, breaking all internal service links.
• Immediate Port Scans: Automated bots knocking on Port 22 within minutes of bringing the server online.
• Storage Disasters: Docker logging and DB data accumulating until the root filesystem hits 100% capacity, triggering a fatal OS kernel panic and locking you out of SSH.

If we don't isolate and secure these physical hardware risks at the OS layer, whatever shiny Node.js pipeline or RAG orchestration we build on top will just be a house of cards. Here is how I constructed the resilient foundation for my homelab server--along with a single script to automate the entire process.

---

2. Operating System Selection: "Why I Chose Rocky Linux 10.x Over Ubuntu"

Ubuntu Server is the absolute standard. It has the largest ecosystem, the most resources, and is a rock-solid choice for both home and enterprise production. I love Ubuntu and use it frequently.

However, for this project, I set aside Ubuntu's cozy ecosystem. Driven by personal familiarity, pure control, and a bit of extreme "hunger spirit" for efficiency, I opted for Rocky Linux 10.x (Minimal ISO).

• Speed of Familiarity: You troubleshoot faster when you use tools that match your muscle memory. The robust directory layout of RHEL-family distributions and the strictness of its package manager make me feel more in control.
• Extreme Memory Diet: By installing the Minimal ISO, I stripped out every unnecessary daemon and GUI component (like Gnome). The base RAM consumption at boot is a mere 140MiB.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       140Mi        14Gi       8.0Mi       480Mi        15Gi
Swap:          2.0Gi          0B       2.0Gi

This minimal footprint ensures that almost 99% of my Mini PC's 16GB RAM is reserved entirely for memory-intensive PostgreSQL (pgvector) calculations and the Redis 7.2 queue.

---

3. Storage Design: Split Partition Isolation (Disk A vs. Disk B)

My Mini PC has two 512GB SSDs. Simply merging them into a single logical volume (LVM) was out of the question. To prevent a storage bottleneck from taking down the entire system, I isolated the partitions physically.

graph TD
    subgraph Mini_PC["Mini PC (Rocky Linux 10.x)"]
        subgraph Hardware["Physical Hardware"]
            SSD_A["SSD A (512GB)"]
            SSD_B["SSD B (512GB)"]
        end
        subgraph OS_Layer["OS Layer (RAM < 200MB)"]
            Kernel["Kernel (vm.overcommit_memory=1, vm.swappiness=10, net.core.somaxconn=1024)"]
            Firewall["Firewalld (White-listed IPs)"]
            Fail2ban["Fail2ban (Bans brute-force IPs)"]
            SSH["SSH Daemon (Port xxxx, No Root Login)"]
        end
        subgraph Storage_Mounts["Storage Mounts"]
            RootMount["/ (Root Filesystem)"]
            DataMount["/data (Data Filesystem)"]
        end
        subgraph Docker_Layer["Docker Container Layer (SELinux :z context)"]
            PostgreSQL["PostgreSQL 16 + pgvector"]
            Redis["Redis 7.2 (Queue/Cache)"]
            Apps["Collector, AI Engine, Sender"]
        end
    end

    SSD_A --> RootMount
    SSD_B --> DataMount
    RootMount --> OS_Layer
    DataMount -.->|Volume Bind Mount :z| Docker_Layer

• Disk A (SSD 1 - 512GB): Mounted at / (Root). Contains only system configuration files and packages. Its size remains static.
• Disk B (SSD 2 - 512GB): Mounted at /data.
  • High-frequency read/write operations (Gitea repositories, workspace, PostgreSQL pgvector data directory, Redis AOF/RDB snapshots) are exiled to this /data mount.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        492G  2.1G  465G   1% /
/dev/sdb1        492G   24K  467G   1% /data

The benefit is clear:
If a database log overflows or backup files consume 100% of the /data partition, the OS on Disk A stays alive. Essential system logs (syslog, journald) and system processes continue to execute, ensuring SSH access remains available for root administrator recovery. If the OS completely crashes, I can wipe Disk A, reinstall the OS, remount Disk B, and restore the service with zero data loss.

---

4. Network Configuration & Security Hardening

First, I set up a static IP within the local network using NetworkManager CLI (nmcli):

# nmcli manual static IP assignment
sudo nmcli connection modify eth0 ipv4.addresses "192.168.0.100/24"
sudo nmcli connection modify eth0 ipv4.gateway "192.168.0.1"
sudo nmcli connection modify eth0 ipv4.dns "1.1.1.1,8.8.8.8"
sudo nmcli connection modify eth0 ipv4.method manual
sudo nmcli connection up eth0

To block brute-force scanners, I disabled Root login and changed the default SSH port in /etc/ssh/sshd_config:

Port xxxx
PermitRootLogin no

Next, I deployed Fail2ban to act as a sentinel. Any IP that fails login 3 times is banned for 24 hours:

# /etc/fail2ban/jail.local
[sshd]
enabled = true
port = xxxx
filter = sshd
logpath = /var/log/secure
maxretry = 3
findtime = 600
bantime = 86400  # 24-hour ban

Security Note: This firewall hardening is just a baseline protection for the local network. Opening port forwarding on your router to expose this SSH port directly to the public internet will still fill your Fail2ban logs with thousands of blocks. The proper way to connect remotely without exposing any ports is through Cloudflare Tunnel, which we will set up in Episode 3.

---

5. Kernel Optimizations

Configure /etc/sysctl.d/99-sysctl.conf to handle high-concurrency requests and Redis database snapshots.

# 1. Prevent background persistence failures in Redis (BGSAVE)
vm.overcommit_memory = 1

# 2. Prevent latency spikes by limiting aggressive swapping
vm.swappiness = 10

# 3. Prevent packet drops under high-concurrency requests
net.core.somaxconn = 1024

Apply immediately:

sudo sysctl -p /etc/sysctl.d/99-sysctl.conf

[Warning] Today's Major Gotcha: SELinux vs Docker

Rocky Linux enables SELinux in Enforcing mode by default. If you attempt a standard Docker volume mount (- /data/postgres:/var/lib/postgresql/data), the container process will throw a Permission Denied error when writing to the host directory.

• The Fix: Do not disable SELinux. Instead, append the :z flag in your Docker Compose volume bindings to trigger automatic SELinux security context labeling (e.g., - /data/postgres:/var/lib/postgresql/data:z).

---

6. Initialization Script (setup.sh)

To ensure this entire environment setup--including installing base host runtimes (Docker Engine, Docker Compose Plugin) and applying kernel/security hardening--is reproducible and safe from typos, use the following bash initialization script:

#!/bin/bash
# Rocky Linux 10.x Initialization & Optimization Script
# Execution: sudo bash setup.sh

set -e

echo "[+] Rocky Linux 10.x Optimization Script Started"

# 1. DNF Optimization
echo "[+] Optimizing DNF package manager..."
sudo tee -a /etc/dnf/dnf.conf <<EOF
max_parallel_downloads=10
fastestmirror=True
EOF

# 2. Update and Utilities Installation
echo "[+] Updating system packages & installing utilities..."
sudo dnf update -y
sudo dnf install -y epel-release dnf-plugins-core
sudo dnf install -y fail2ban git curl policycoreutils-python-utils

# 3. Docker Engine & Docker Compose Plugin Installation
echo "[+] Installing Docker Engine & Docker Compose Plugin..."
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker

# 4. Physical Partition Isolation Check
echo "[+] Checking physical partition isolation..."
df -h | grep "/data" || echo "[!] WARNING: /data mount point not detected. Ensure SSD B is mounted before running containers."

# 5. Firewall Configuration
echo "[+] Configuring firewalld for secure access..."
sudo systemctl enable --now firewalld
sudo firewall-cmd --new-zone=homelab-zone --permanent
sudo firewall-cmd --zone=homelab-zone --add-source=192.168.0.0/24 --permanent
sudo firewall-cmd --zone=homelab-zone --add-port=xxxx/tcp --permanent
sudo firewall-cmd --zone=homelab-zone --add-masquerade --permanent
sudo firewall-cmd --reload

# 6. SSH Daemon Hardening
echo "[+] Hardening SSH daemon settings (Port xxxx, No Root)..."
sudo sed -i 's/#Port 22/Port xxxx/' /etc/ssh/sshd_config
sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config

# Register custom port to SELinux
echo "[+] Registering custom SSH port to SELinux..."
sudo semanage port -a -t ssh_port_t -p tcp xxxx || sudo semanage port -m -t ssh_port_t -p tcp xxxx
sudo systemctl restart sshd

# 7. Fail2ban Setup
echo "[+] Setting up Fail2ban brute-force protection..."
sudo tee /etc/fail2ban/jail.local <<EOF
[sshd]
enabled = true
port = xxxx
filter = sshd
logpath = /var/log/secure
maxretry = 3
findtime = 600
bantime = 86400
EOF
sudo systemctl enable --now fail2ban

# 8. Kernel Optimizations
echo "[+] Injecting Kernel parameter optimizations..."
sudo tee /etc/sysctl.d/99-sysctl.conf <<EOF
vm.overcommit_memory = 1
vm.swappiness = 10
net.core.somaxconn = 1024
EOF
sudo sysctl -p /etc/sysctl.d/99-sysctl.conf

echo "[+] System configuration successfully completed!"

---

7. Next Episode

Now we have a solid bare-metal platform running on under 150MB of idle RAM, hardened with a firewall, and isolated at the disk level. But coding strictly within the local home network is a bottleneck. We need a way to access and develop on this server securely from cafes or external workspaces.

In Episode 3, we will establish external access without port forwarding using Cloudflare Tunnel (Zero-Trust). We will cover both HTTPS routing for web applications and a secure SSH tunneling architecture using Cloudflare Access OTP and VS Code Remote-SSH.