DEV Community

giveitatry
giveitatry

Posted on

Your Own S3 on Kubernetes in Hetzner

#s3 #k8s #devops

This article is a documentation of my journey to build an S3-compatible object storage cluster using MinIO on a Kubernetes (K3s) cluster hosted in Hetzner.

🔧 Step 1: Provision Your Hetzner VPS Nodes

Log into Hetzner Cloud and create at least 5 VPS nodes.

Node Architecture Overview

For a robust High Availability K3s cluster:

  • Control Plane Nodes (2 nodes)
    • Manage cluster state and provide redundancy
    • These will be hidden behind a Load Balancer
  • Worker Nodes (3 nodes)
    • Run application workloads
Role Description IP Address
Control Plane 1 Manages cluster state Save IP Address
Control Plane 2 Provides HA redundancy Save IP Address
Worker Node 1 Runs workloads Save IP Address
Worker Node 2 Runs workloads Save IP Address
Worker Node 3 Runs workloads Save IP Address

🌐 Step 2: Setup Load Balancer in Hetzner

  • Create a Load Balancer (LB) and target Control Plane 1 & 2
  • Note down the IP (e.g. lb.lb.lb.lb)

Required Ports

Port Target Protocol
80 80 HTTP
443 443 TCP
6443 6443 TCP
  • Disable proxying on all ports
  • Enable health checks:
    • Port 80 should return status 404 to pass — this is required by Let's Encrypt

Ensure all ports are marked healthy before proceeding.

🚀 Step 3: Install K3s

3.1 Generate Cluster Token 

openssl rand -hex 16
Enter fullscreen mode Exit fullscreen mode

Save this token for cluster joining.

3.2 Initialize First Control Plane Node 

curl -sfL https://get.k3s.io | K3S_TOKEN=YOUR_SECRET sh -s - server \
  --cluster-init \
  --tls-san=lb.lb.lb.lb
Enter fullscreen mode Exit fullscreen mode

3.3 Join Additional Control Plane Nodes 

curl -sfL https://get.k3s.io | K3S_TOKEN=YOUR_SECRET sh -s - server \
  --server https://lb.lb.lb.lb:6443 \
  --tls-san=lb.lb.lb.lb
Enter fullscreen mode Exit fullscreen mode

3.4 Install Worker Nodes on all remaining 3 nodes 

curl -sfL https://get.k3s.io | K3S_TOKEN=YOUR_SECRET sh -s - agent \
  --server https://lb.lb.lb.lb:6443
Enter fullscreen mode Exit fullscreen mode

3.5 Verify Cluster 

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

All nodes should appear as Ready.

🔐 Step 4: Enable Traefik and Configure Let's Encrypt

Ensure Traefik is running:

kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik
Enter fullscreen mode Exit fullscreen mode

If not, reinstall K3s without the --disable traefik flag.

4.1 Configure Traefik for TLS

Create:

/var/lib/rancher/k3s/server/manifests/traefik-config.yaml
Enter fullscreen mode Exit fullscreen mode

Paste:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--certificatesresolvers.default.acme.email=you@example.com"
      - "--certificatesresolvers.default.acme.storage=/data/acme.json"
      - "--certificatesresolvers.default.acme.httpchallenge.entrypoint=web"
    ports:
      web:
        exposedPort: 80
      websecure:
        exposedPort: 443
Enter fullscreen mode Exit fullscreen mode

Replace with your real email.

4.2 Watch Logs 

kubectl logs -n kube-system -l app.kubernetes.io/name=traefik --tail=100 -f
Enter fullscreen mode Exit fullscreen mode

Look for:

  • Using HTTP challenge
  • Registering with ACME

🪣 Step 5: Install MinIO (S3 Compatible Storage)

5.1 Install the MinIO Operator 

kubectl apply -k "github.com/minio/operator?ref=v7.0.1"
Enter fullscreen mode Exit fullscreen mode

5.2 Create Namespace 

kubectl create namespace minio
Enter fullscreen mode Exit fullscreen mode

5.3 Create Deployment YAML

Create a deployment.yaml file with the provided config (includes secret, tenant definition).

apiVersion: v1
kind: Secret
metadata:
  name: storage-configuration
  namespace: minio
stringData:
  config.env: |-
    export MINIO_ROOT_USER="admin"
    export MINIO_ROOT_PASSWORD="password"
    export MINIO_STORAGE_CLASS_STANDARD="EC:2"
    export MINIO_BROWSER="on"
    export CONSOLE_TLS_ENABLE="on"
type: Opaque
---
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  annotations:
    prometheus.io/path: /minio/v2/metrics/cluster
    prometheus.io/port: "9000"
    prometheus.io/scrape: "true"
  labels:
    app: minio
  name: minio
  namespace: minio
spec:

  requestAutoCert: false  # Disable automatic cert generation

  certConfig: {}
  configuration:
    name: storage-configuration
  env: []
  externalCaCertSecret: []
  externalCertSecret: []
  externalClientCertSecrets: []
  features:
    bucketDNS: false
    domains: {}
  image: quay.io/minio/minio:RELEASE.2024-10-02T17-50-41Z
  imagePullSecret: {}
  mountPath: /export
  podManagementPolicy: Parallel
  pools:
  - affinity:
      nodeAffinity: {}
      podAffinity: {}
      podAntiAffinity: {}
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault
    name: pool-0
    nodeSelector: {}
    resources: {}
    securityContext:
      fsGroup: 1000
      fsGroupChangePolicy: OnRootMismatch
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 1000
    servers: 3
    tolerations: []
    topologySpreadConstraints: []
    volumeClaimTemplate:
      apiVersion: v1
      kind: persistentvolumeclaims
      metadata: {}
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: local-path
      status: {}
    volumesPerServer: 3
  priorityClassName: ""
  requestAutoCert: false
  serviceAccountName: ""
  serviceMetadata:
    consoleServiceAnnotations: {}
    consoleServiceLabels: {}
    minioServiceAnnotations: {}
    minioServiceLabels: {}
  subPath: ""
Enter fullscreen mode Exit fullscreen mode

Important: requestAutoCert: false since Traefik handles TLS termination.

5.4 Deploy the MinIO Tenant 

kubectl apply -f deployment.yaml -n minio
Enter fullscreen mode Exit fullscreen mode

It will install minio in minio namespace

🌍 Step 6: Configure Ingress with TLS for MinIO

Create ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cert-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls.certresolver: default
spec:
  ingressClassName: traefik
  rules:
    - host: s3.example.com  # Change to your domain
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: minio-console  
              port:
                number: 9090  # MinIO default API port (HTTPS unsigned cert)
        - path: /console
          pathType: Prefix
          backend:
            service:
              name: minio-hl
              port:
                number: 9000  # MinIO default API port
  tls:
    - hosts:
        - s3.example.com
Enter fullscreen mode Exit fullscreen mode

Replace s3.example.com with your domain.

Apply the Ingress 

kubectl apply -f ingress.yaml -n minio
Enter fullscreen mode Exit fullscreen mode

Point it in DNS to your LB IP.

To direct a domain e.g. s3.example.com to a specific IP address (e.g., your Hetzner load balancer), you need to modify the DNS records associated with your domain. This is typically done through the domain registrar or DNS provider where your domain is managed (such as Cloudflare, Hetzner, Namecheap, or others). You'll want to create an A record that maps the subdomain s3 to your load balancer's public IP address. For example, if your load balancer’s IP is 95.217.XX.XX, the A record should look like: s3.example.com -> 95.217.XX.XX.

In most DNS management panels, this process involves opening the DNS settings for your domain (In Hetzner click upper right icon with squares and select DNS)) , clicking “Add Record,” choosing “A” as the record type, entering s3 as the name or host, and providing your IP in the value field. TTL (Time To Live) can typically be left as default (e.g., 3600 seconds). Once saved, this DNS record will start propagating. DNS propagation can take from a few minutes to up to 48 hours, although it often resolves within an hour.

To verify that the domain is correctly pointing to your IP, you can use command-line tools such as dig s3.example.com or nslookup s3.example.com, which should return the correct IP address. You can also open http://s3.example.com in a browser once your services are up. If you configured everything correctly (including your load balancer and ingress controller), you should see a response from the MinIO or Traefik setup.

🧪 Step 7: Troubleshooting

Check Traefik logs:

kubectl logs -n kube-system -l app.kubernetes.io/name=traefik --tail=100 -f
Enter fullscreen mode Exit fullscreen mode

Verify:

  • Log line: INF Updated ingress status ingress=s3-ingress namespace=minio

Wrap up

When I first set up the load balancer in Hetzner, I ran into a series of small but frustrating issues that blocked progress. The load balancer was not forwarding all required ports correctly—specifically, port 6443 for the Kubernetes API server wasn’t properly exposed. On top of that, one of the ports was being proxied instead of passed directly to the node, which caused all kinds of unexpected behaviors, including broken health checks. I also forgot to configure the health check on port 80 to treat HTTP 404 as a valid response, which is crucial for Let’s Encrypt to validate the domain via the ACME challenge. Without that, the load balancer status remained unhealthy, and nothing downstream worked properly.

The consequences of this misconfiguration became clear when I tried to issue TLS certificates through Let’s Encrypt using Traefik. No matter what I did, certificates weren’t being generated. The ACME challenge endpoint at http://s3.example.com/.well-known/acme-challenge/test should return a 404 to signal readiness, but in my case, it either timed out or returned a 502 due to the load balancer not routing correctly. Once I updated the health checks to allow a 404 on port 80 and ensured that port 443 and 6443 were correctly routed (with no proxying), the load balancer turned green and certificates started issuing immediately.

Even after getting the certificates working, I ran into another issue with MinIO itself. By default, MinIO exposed the console on port 9443, but it used a self-signed certificate which Traefik didn’t like. This resulted in broken HTTPS connections and browser warnings. Since I wanted Traefik to manage TLS termination with Let’s Encrypt, I had to disable MinIO’s internal certificate management completely (requestAutoCert: false). After doing that, MinIO started exposing the console on port 9090 instead, which is unencrypted HTTP, making it compatible with how Traefik expects to route traffic internally. That change finally allowed me to access the MinIO console securely through the Traefik-managed s3.example.com endpoint.

Now you have your own S3-compatible storage cluster running on Hetzner Kubernetes under domain s3.example.com. 🚀

Top comments (0)