DEV Community

Dhiraj Shrotri
Dhiraj Shrotri

Posted on AI-assisted

Self-hosting Paperless-ngx on k3s with GitOps

#kubernetes #selfhosted #gitops #homelab

I almost always use the Adobe Scan app to scan any documents that I have. I recently noticed that by default, Adobe saves the documents to their own cloud, instead of storing them on my device. I had a beelink mini pc and a 1TB external HDD lying around. I started looking for solutions which would let me scan a document using my phone's camera, tag the document by content and then upload it to my beelink and eventually to the external HDD.

I ended up with Paperless-ngx running on k3s on a Beelink mini PC. Documents live on an external HDD, and Flux deploys every config change I push to GitHub. This post covers the setup and, more usefully, every error I hit along the way.

Why Paperless-ngx

Paperless-ngx is a document archive, not a file cabinet. It OCRs everything you add, so searching "electricity bill March" finds the scan. It also tags and sorts documents automatically.

Nextcloud was the other contender. It's a better general Google Drive replacement, but its document scanner isn't available in every Android build, and it organises by folders rather than content. For paperwork, Paperless won.

There's no official mobile app, but the community ones are good:

App Platform Notes
Swift Paperless iOS Native, actively maintained, camera scanning, Face ID
PaperNext Android, iOS Built-in scanner, offline cache, supports Paperless-ngx 2 and 3 (open beta)
Paperless Mobile Android The longer-standing Android client

All three let you scan, tag and upload in one flow. The server then OCRs and files the document.

The architecture

Everything runs on one Beelink mini PC with k3s. Paperless-ngx uses the same components as the official Docker Compose file: PostgreSQL, Redis, and Tika plus Gotenberg for Word, Excel and email files.

The database and search index stay on the fast internal SSD. Documents and nightly backups go to the external HDD. My phone reaches Paperless through k3s's bundled Traefik, and config changes arrive from GitHub through Flux.

Using an external HDD as a Kubernetes volume

I wanted documents on the big external HDD, not the Beelink's small internal SSD. The trick is a static local PersistentVolume pointing at a folder on the HDD.

First, format the disk as ext4 (NTFS and exFAT can't store Linux ownership) and mount it by UUID:

sudo mkfs.ext4 -L paperless-hdd /dev/sdb1     # erases the disk
sudo mkdir -p /mnt/hdd
echo 'UUID=<uuid>  /mnt/hdd  ext4  defaults,nofail,noatime  0 2' | sudo tee -a /etc/fstab
sudo mount -a
sudo mkdir -p /mnt/hdd/paperless/{media,export}
sudo chown -R 1000:1000 /mnt/hdd/paperless
Enter fullscreen mode Exit fullscreen mode

nofail lets the machine still boot without the disk. Then make k3s wait for the mount, via sudo systemctl edit k3s:

[Unit]
RequiresMountsFor=/mnt/hdd
Enter fullscreen mode Exit fullscreen mode

Then the Kubernetes side: a StorageClass with no provisioner, and one PersistentVolume per folder, pinned to the node:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hdd-local
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: paperless-media-hdd
spec:
  capacity:
    storage: 500Gi
  accessModes: [ReadWriteOnce]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: hdd-local
  local:
    path: /mnt/hdd/paperless/media
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values: ["my-node"]   # kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

The PVC then binds to it with storageClassName: hdd-local and volumeName: paperless-media-hdd.

Why local and not hostPath? Because the volume points at a subfolder that only exists on the HDD. If the disk isn't mounted, the pod refuses to start, instead of quietly writing your documents to the internal SSD. Retain means deleting the PVC never deletes your files.

Every error I hit, and the fix

The manifests were the easy part. Here's what actually took the afternoon.

1. k3s died halfway through kubectl apply 

dial tcp 127.0.0.1:6443: connect: connection refused
Enter fullscreen mode Exit fullscreen mode

The API server went down mid-apply. kubectl apply is idempotent, so once k3s is back you just re-run it. The real question was why k3s died.

2. bind: address already in use on port 10257 

failed to listen on 127.0.0.1:10257: bind: address already in use
Enter fullscreen mode Exit fullscreen mode

sudo ss -ltnp | grep 10257 showed the culprit: kubelite, which is MicroK8s. I had a forgotten MicroK8s install on the same machine, and both distributions want the same controller-manager ports. Snap had probably auto-restarted it mid-install.

sudo microk8s stop
sudo snap disable microk8s     # or: sudo snap remove microk8s
sudo systemctl restart k3s
Enter fullscreen mode Exit fullscreen mode

3. failed to find interface with specified node ip

k3s had registered with an IP the machine no longer had, because my router handed out a new DHCP lease. Pin the current IP in /etc/rancher/k3s/config.yaml:

node-ip: 192.168.1.50
Enter fullscreen mode Exit fullscreen mode

Then set a DHCP reservation in the router so it never drifts again.

4. Pod stuck in ContainerCreating 

MountVolume.NewMounter initialization failed ... path "/mnt/hdd/paperless/media" does not exist
Enter fullscreen mode Exit fullscreen mode

This was the HDD safety check working as designed: I hadn't created the folders. Check findmnt /mnt/hdd first. If you create the folders while the disk isn't mounted, they land on the SSD.

5. Paperless crash-looping: 'tcp://10.43.x.x:8000' is not a valid integer

My favourite one. Kubernetes injects environment variables for every Service in the namespace. A Service named paperless produces PAPERLESS_PORT=tcp://10.43.220.175:8000, and Paperless reads PAPERLESS_PORT as its own listen port. One line on the pod spec fixes it:

spec:
  enableServiceLinks: false
Enter fullscreen mode Exit fullscreen mode

6. CSRF verification failed after login

Paperless only accepts logins from the exact origin in PAPERLESS_URL, and my phone's browser was silently upgrading to https. Trust both schemes:

PAPERLESS_CSRF_TRUSTED_ORIGINS=http://paperless.192.168.1.50.nip.io,https://paperless.192.168.1.50.nip.io
Enter fullscreen mode Exit fullscreen mode

I used nip.io for the hostname: paperless.192.168.1.50.nip.io resolves to 192.168.1.50, so phones work with no local DNS.

GitOps: deploy with a git push

Once it worked, I didn't want to SSH in and kubectl apply every tweak. Flux runs inside the cluster and pulls from GitHub, so the home server never needs an open inbound port. That's why I chose it over a push-based CI job. It's also lighter than Argo CD on a mini PC, and it decrypts SOPS secrets natively.

Repo layout 

clusters/home/
  apps.yaml            # Flux: deploy ./apps, decrypt with SOPS
  flux-system/         # created by flux bootstrap
apps/
  kustomization.yaml
  paperless/
    kustomization.yaml
    paperless.env      # day-to-day settings
    secret.yaml        # SOPS-encrypted
    paperless.yaml
    backing-services.yaml
    storage.yaml
    namespace.yaml
Enter fullscreen mode Exit fullscreen mode

The repo lives on my laptop. The server never holds a copy.

Config changes restart Paperless automatically

A pod reads env vars only at startup, so editing a ConfigMap does nothing to a running pod. Kustomize's configMapGenerator fixes that:

configMapGenerator:
  - name: paperless-config
    envs:
      - paperless.env
Enter fullscreen mode Exit fullscreen mode

The ConfigMap gets a content hash in its name (paperless-config-tg67g2tb44). Kustomize rewrites every reference to it, so any edit changes the pod spec and triggers a rollout.

Secrets with SOPS + age 

age-keygen -o ~/.config/sops/age/keys.txt
sops --encrypt --in-place apps/paperless/secret.yaml
Enter fullscreen mode Exit fullscreen mode

.sops.yaml (committed, since it only holds the public key) encrypts just the data and stringData fields. The private key goes into the cluster once:

kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=keys.txt
Enter fullscreen mode Exit fullscreen mode

Then the Flux Kustomization points at it:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops
    secretRef:
      name: sops-age
Enter fullscreen mode Exit fullscreen mode

Don't let prune delete your data

prune: true means removing a file from Git removes the resource from the cluster. That's great for Deployments and terrifying for storage. I annotated the namespace, PVCs, PVs and StorageClass so Flux never deletes them:

metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled
Enter fullscreen mode Exit fullscreen mode

Bootstrap gotchas

  • Push first. flux bootstrap doesn't push your local commits, only its own flux-system files. Push the repo, run bootstrap on the server, then git pull on the laptop.
  • Run it where kubectl works. For me that was the server over SSH.
  • x509: certificate signed by unknown authority. flux reads ~/.kube/config, which was stale from MicroK8s. k3s's kubectl reads /etc/rancher/k3s/k3s.yaml. Copy that file to ~/.kube/config.

Wrapping up

Now I scan a bill on my phone and it's OCR'd and searchable on my own hardware within a minute. Changing a setting is one edit in paperless.env and a git push.

What's next on my list:

  • Observability and alerting. I am thinking of integrating Prometheus + Grafana for metrics and configuring alertmanager to send me alerts if a pod goes down via Telegram.
  • Tailscale. It gives me access from outside the house and real HTTPS on a *.ts.net name, without exposing anything to the internet.
  • Pinned image tags. :latest plus GitOps means a restart can upgrade you silently. Pin the tag and bump it with a commit.

If you're doing this on a single-node k3s box, the two lines that would have saved me the most time are enableServiceLinks: false and "check for a second Kubernetes install before anything else."

Sources

I'm open to full-stack and platform software engineering roles. If your team needs someone who builds the app and the infrastructure it runs on, reach me on LinkedIn or Email Me.

Top comments (0)