DEV Community: Dhiraj Shrotri

Self-hosting Paperless-ngx on k3s with GitOps

Dhiraj Shrotri — Mon, 26 Oct 2026 09:58:26 +0000

I almost always use the Adobe Scan app to scan any documents that I have. I recently noticed that by default, Adobe saves the documents to their own cloud, instead of storing them on my device. I had a beelink mini pc and a 1TB external HDD lying around. I started looking for solutions which would let me scan a document using my phone's camera, tag the document by content and then upload it to my beelink and eventually to the external HDD.

I ended up with Paperless-ngx running on k3s on a Beelink mini PC. Documents live on an external HDD, and Flux deploys every config change I push to GitHub. This post covers the setup and, more usefully, every error I hit along the way.

Why Paperless-ngx

Paperless-ngx is a document archive, not a file cabinet. It OCRs everything you add, so searching "electricity bill March" finds the scan. It also tags and sorts documents automatically.

Nextcloud was the other contender. It's a better general Google Drive replacement, but its document scanner isn't available in every Android build, and it organises by folders rather than content. For paperwork, Paperless won.

There's no official mobile app, but the community ones are good:

App	Platform	Notes
Swift Paperless	iOS	Native, actively maintained, camera scanning, Face ID
PaperNext	Android, iOS	Built-in scanner, offline cache, supports Paperless-ngx 2 and 3 (open beta)
Paperless Mobile	Android	The longer-standing Android client

All three let you scan, tag and upload in one flow. The server then OCRs and files the document.

The architecture

Everything runs on one Beelink mini PC with k3s. Paperless-ngx uses the same components as the official Docker Compose file: PostgreSQL, Redis, and Tika plus Gotenberg for Word, Excel and email files.

The database and search index stay on the fast internal SSD. Documents and nightly backups go to the external HDD. My phone reaches Paperless through k3s's bundled Traefik, and config changes arrive from GitHub through Flux.

Using an external HDD as a Kubernetes volume

I wanted documents on the big external HDD, not the Beelink's small internal SSD. The trick is a static local PersistentVolume pointing at a folder on the HDD.

First, format the disk as ext4 (NTFS and exFAT can't store Linux ownership) and mount it by UUID:

sudo mkfs.ext4 -L paperless-hdd /dev/sdb1     # erases the disk
sudo mkdir -p /mnt/hdd
echo 'UUID=<uuid>  /mnt/hdd  ext4  defaults,nofail,noatime  0 2' | sudo tee -a /etc/fstab
sudo mount -a
sudo mkdir -p /mnt/hdd/paperless/{media,export}
sudo chown -R 1000:1000 /mnt/hdd/paperless

nofail lets the machine still boot without the disk. Then make k3s wait for the mount, via sudo systemctl edit k3s:

[Unit]
RequiresMountsFor=/mnt/hdd

Then the Kubernetes side: a StorageClass with no provisioner, and one PersistentVolume per folder, pinned to the node:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hdd-local
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: paperless-media-hdd
spec:
  capacity:
    storage: 500Gi
  accessModes: [ReadWriteOnce]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: hdd-local
  local:
    path: /mnt/hdd/paperless/media
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values: ["my-node"]   # kubectl get nodes

The PVC then binds to it with storageClassName: hdd-local and volumeName: paperless-media-hdd.

Why local and not hostPath? Because the volume points at a subfolder that only exists on the HDD. If the disk isn't mounted, the pod refuses to start, instead of quietly writing your documents to the internal SSD. Retain means deleting the PVC never deletes your files.

Every error I hit, and the fix

The manifests were the easy part. Here's what actually took the afternoon.

1. k3s died halfway through `kubectl apply`

dial tcp 127.0.0.1:6443: connect: connection refused

The API server went down mid-apply. kubectl apply is idempotent, so once k3s is back you just re-run it. The real question was why k3s died.

2. `bind: address already in use` on port 10257

failed to listen on 127.0.0.1:10257: bind: address already in use

sudo ss -ltnp | grep 10257 showed the culprit: kubelite, which is MicroK8s. I had a forgotten MicroK8s install on the same machine, and both distributions want the same controller-manager ports. Snap had probably auto-restarted it mid-install.

sudo microk8s stop
sudo snap disable microk8s     # or: sudo snap remove microk8s
sudo systemctl restart k3s

3. `failed to find interface with specified node ip`

k3s had registered with an IP the machine no longer had, because my router handed out a new DHCP lease. Pin the current IP in /etc/rancher/k3s/config.yaml:

node-ip: 192.168.1.50

Then set a DHCP reservation in the router so it never drifts again.

4. Pod stuck in `ContainerCreating`

MountVolume.NewMounter initialization failed ... path "/mnt/hdd/paperless/media" does not exist

This was the HDD safety check working as designed: I hadn't created the folders. Check findmnt /mnt/hdd first. If you create the folders while the disk isn't mounted, they land on the SSD.

5. Paperless crash-looping: `'tcp://10.43.x.x:8000' is not a valid integer`

My favourite one. Kubernetes injects environment variables for every Service in the namespace. A Service named paperless produces PAPERLESS_PORT=tcp://10.43.220.175:8000, and Paperless reads PAPERLESS_PORT as its own listen port. One line on the pod spec fixes it:

spec:
  enableServiceLinks: false

6. CSRF verification failed after login

Paperless only accepts logins from the exact origin in PAPERLESS_URL, and my phone's browser was silently upgrading to https. Trust both schemes:

PAPERLESS_CSRF_TRUSTED_ORIGINS=http://paperless.192.168.1.50.nip.io,https://paperless.192.168.1.50.nip.io

I used nip.io for the hostname: paperless.192.168.1.50.nip.io resolves to 192.168.1.50, so phones work with no local DNS.

GitOps: deploy with a git push

Once it worked, I didn't want to SSH in and kubectl apply every tweak. Flux runs inside the cluster and pulls from GitHub, so the home server never needs an open inbound port. That's why I chose it over a push-based CI job. It's also lighter than Argo CD on a mini PC, and it decrypts SOPS secrets natively.

Repo layout

clusters/home/
  apps.yaml            # Flux: deploy ./apps, decrypt with SOPS
  flux-system/         # created by flux bootstrap
apps/
  kustomization.yaml
  paperless/
    kustomization.yaml
    paperless.env      # day-to-day settings
    secret.yaml        # SOPS-encrypted
    paperless.yaml
    backing-services.yaml
    storage.yaml
    namespace.yaml

The repo lives on my laptop. The server never holds a copy.

Config changes restart Paperless automatically

A pod reads env vars only at startup, so editing a ConfigMap does nothing to a running pod. Kustomize's configMapGenerator fixes that:

configMapGenerator:
  - name: paperless-config
    envs:
      - paperless.env

The ConfigMap gets a content hash in its name (paperless-config-tg67g2tb44). Kustomize rewrites every reference to it, so any edit changes the pod spec and triggers a rollout.

Secrets with SOPS + age

age-keygen -o ~/.config/sops/age/keys.txt
sops --encrypt --in-place apps/paperless/secret.yaml

.sops.yaml (committed, since it only holds the public key) encrypts just the data and stringData fields. The private key goes into the cluster once:

kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=keys.txt

Then the Flux Kustomization points at it:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Don't let `prune` delete your data

prune: true means removing a file from Git removes the resource from the cluster. That's great for Deployments and terrifying for storage. I annotated the namespace, PVCs, PVs and StorageClass so Flux never deletes them:

metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled

Bootstrap gotchas

Push first. flux bootstrap doesn't push your local commits, only its own flux-system files. Push the repo, run bootstrap on the server, then git pull on the laptop.
Run it where kubectl works. For me that was the server over SSH.
x509: certificate signed by unknown authority. flux reads ~/.kube/config, which was stale from MicroK8s. k3s's kubectl reads /etc/rancher/k3s/k3s.yaml. Copy that file to ~/.kube/config.

Wrapping up

Now I scan a bill on my phone and it's OCR'd and searchable on my own hardware within a minute. Changing a setting is one edit in paperless.env and a git push.

What's next on my list:

Observability and alerting. I am thinking of integrating Prometheus + Grafana for metrics and configuring alertmanager to send me alerts if a pod goes down via Telegram.
Tailscale. It gives me access from outside the house and real HTTPS on a *.ts.net name, without exposing anything to the internet.
Pinned image tags. :latest plus GitOps means a restart can upgrade you silently. Pin the tag and bump it with a commit.

If you're doing this on a single-node k3s box, the two lines that would have saved me the most time are enableServiceLinks: false and "check for a second Kubernetes install before anything else."

Sources

I'm open to full-stack and platform software engineering roles. If your team needs someone who builds the app and the infrastructure it runs on, please feel free to reach out to me on LinkedIn or Email Me.

How do you scale a frontend application to a million users ?

Dhiraj Shrotri — Thu, 18 Jun 2026 20:23:45 +0000

I was recently in a job interview (yes, the kind you have to do when you get laid off) and the interviewer asked me: "How would you scale a frontend application from 100,000 users to a million plus?"
I gave a decent enough answer at the moment. But afterwards, I kept thinking about everything I didn't say. So here we are. Consider this my full answer, delivered about three months too late, to actually matter for that job.

First, Let's Calibrate

Scaling a frontend isn't just about throwing more servers at the problem. That's your backend team's headache. The frontend has its own scaling concerns, and a lot of them are invisible until you're already on fire. The jump from 100k to 1M+ users is less about one dramatic architectural overhaul and more about fixing the 20% of things causing 80% of your slowness. The trick is knowing which 20%.

Performance & Asset Delivery: The Biggest Bang For Your Buck

If you're not serving your static assets through a CDN at this scale, stop reading this and go fix that first. Cloudflare, CloudFront, Fastly; pick one. Your origin server should never be sweating over a CSS file.

Pair your CDN with content-hashed filenames. This lets you set aggressive long-term cache TTLs without worrying about users getting stale assets after a deployment. It's one of those things that feels like a small detail until you realise it's quietly saving you a fortune in bandwidth costs and keeping your load times fast for repeat visitors.

The other non-negotiable is code splitting and lazy loading. Users should only download the JavaScript they need for the page they're actually on. Nobody visiting your landing page needs the bundle for your admin dashboard. With Vite this is largely handled for you at the route level, but you still need to be intentional about it.

State Management: Where Things Get Messy At Scale

At 100k users, messy state management is annoying. At a million plus, it becomes a performance crisis. The most important distinction to get right is separating server state from client state. These are fundamentally different things and treating them the same way is a trap I've fallen into personally.

Server state: data that lives on your backend and needs to be fetched, cached, and kept in sync, should be handled by something like React Query or SWR. These libraries give you stale-while-revalidate caching out of the box, which means your UI feels snappy while fresh data loads in the background.

Client state: things like UI toggles, modal open/close, user preferences etc belong in something lean like Redux or even just local component state if it doesn't need to be shared widely. Not everything needs to be in a global store. Redux is a perfectly good tool that a lot of people use as a hammer for every nail.

The API & Data Layer

A few principles that matter a lot at scale:

Virtualize large lists: Never render 10,000 rows in the DOM. Libraries like TanStack Virtual handle this elegantly, only the rows visible in the viewport actually exist in the DOM. This sounds obvious until you're staring at a performance profile wondering why scrolling feels like wading through concrete.

Debounce and throttle user-triggered requests: Search inputs that fire an API call on every keystroke are a classic scaling antipattern. Debounce them. Your backend team will thank you.

Paginate everything: Infinite scroll or traditional pagination - pick your poison, but loading entire datasets upfront is a sin at scale.

You Cannot Optimize What You Cannot Measure

This is probably the lesson I wish I'd learned earlier in my career. At scale, gut feel and local testing will actively mislead you. You need Real User Monitoring — tools like Datadog or Sentry that show you what actual users on actual devices on actual network conditions are experiencing, not just what your M2 MacBook on gigabit ethernet thinks is happening.

Track your Core Web Vitals ( LCP, CLS, and INP specifically): These aren't just Google SEO metrics, they directly correlate with whether users stay on your page or leave. If your LCP is over 2.5 seconds, a non-trivial percentage of users are already gone before they've seen anything meaningful.

Error tracking with context is also non-negotiable: At a million users, you will have errors you never saw in testing. You need to know about them before your users start tweeting about them.

Bundle Size: The Silent Killer

Run a bundle analysis on your app right now. I'll wait. Tools like rollup-visualizer (for Vite) or webpack-bundle-analyzer will show you a treemap of everything you're shipping to the user. There is almost always something in there that makes you say "why is that so big?"

Common culprits:

A date library you imported for one utility function that weighs 70kb.
A component library where you're importing the entire thing for three components.
Polyfills targeting browsers that represent 0.2% of your user base.

Tree shaking handles a lot of this automatically, but it's not magic. Named imports, side effectful modules, and certain CommonJS packages can defeat it. The bundle analyzer doesn't lie.

Architecture Patterns For The Long Game

Micro-frontends become a real conversation at a very large scale. Different teams owning different slices of the UI independently, deploying without coordinating with everyone else. The tradeoff is real though. Shared dependency management becomes its own full-time job and the coordination overhead can eat the productivity gains if you're not careful. I wouldn't reach for this unless you have multiple large teams stepping on each other's toes.

Edge rendering tools like Next.js Edge Runtime, Cloudflare Workers etc can dramatically reduce time-to-first-byte for a globally distributed user base. If your users are spread across continents and you're rendering from a single region, you're leaving a lot of performance on the table.

The Stuff Nobody Talks About Enough

Images. Consistently the most underestimated performance problem. Use modern formats (WebP, AVIF). Lazy load anything below the fold. Serve correctly sized images for each viewport. An image that looks fine on your screen might be a 4MB monster getting served to someone on mobile data.
Fonts. font-display: swap and preloading your critical fonts goes a long way. A font that blocks rendering is a font that's costing you users.

Third party scripts. Analytics tools, chat widgets, A/B testing libraries audit these ruthlessly. Every third party script is code you don't control running on your page, and at scale their performance cost adds up. Some of them are genuinely heavier than your entire application.

So, What's The Actual Answer?

If I could go back and answer that interview question more completely, I'd say this: scaling a frontend to a million users is mostly about discipline. Discipline in measuring before optimizing, discipline in keeping your bundle lean, discipline in not letting state management become a free-for-all, and discipline in treating performance as a feature rather than a cleanup task you'll get to eventually.

The technical solutions - CDNs, virtualization, code splitting, RUM - are well documented and not particularly exotic. The hard part is building a culture and a codebase where performance doesn't quietly degrade every sprint until you're suddenly wondering why conversion dropped 15%.

Anyway. If anyone's hiring a frontend engineer who thinks a lot about this stuff, you know where to find me.

K3S troubleshooting: Fixing K3s permission denied and port binding errors on Ubuntu Server

Dhiraj Shrotri — Sun, 30 Nov 2025 20:09:01 +0000

Recently, while setting up K3s on a Beelink mini PC running Ubuntu Server, I ran into two frustrating errors while trying to run:

kubectl get nodes

The first issue was a permissions error loading the kubeconfig, and once that was fixed, a second error appeared indicating that the Kubernetes control plane wasn’t running. After debugging, here’s the full process and solution in case you run into the same problem.

Problem 1: kubeconfig Permission Denied

The initial error looked like this:

WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode or --write-kubeconfig-group to modify kube config permissions
error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied

This happens because K3s generates the Kubernetes config file owned by root with restrictive permissions (600). When we run kubectl as a regular user, kubectl can't access it.

To fix the issue, we create another directory, copy the kubectl config to it and change the ownership of the directory so that kubectl can access it.

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

Now running kubectl get nodes threw a different error

The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

Finally some progress!

This means kubectl can now read the config file, but the K3s API server isn’t running or reachable.

Problem 2: K3s controller-manager port conflict

Checking the K3s service:

sudo systemctl status k3s

The logs showed this error:

failed to listen on 127.0.0.1:10257: bind: address already in use

This indicates another process is already using port 10257, which is required by the Kubernetes controller-manager. We verify what process is using the port by running the lsof command with the port as argument.

sudo lsof -i :10257

kubelite 188849 root 120u IPv6 ... TCP *:10257 (LISTEN)

kubelite is K3s’s internal Kubernetes binary. That means a previous crashed instance left it running, preventing the new instance from starting.

To fix this, stop k3s completely:

sudo systemctl stop k3s

and kill leftover processes:

sudo pkill -f k3s
sudo pkill -f kubelite

Now to verify the ports are free, we run the lsof command again and that should give us no output.

Finally we just restart k3s:

sudo systemctl start k3s
sudo systemctl status k3s

Now when we run kubectl get nodes, it should run without any issues!

K3s is incredibly powerful and lightweight, but diagnosing startup issues can be intimidating if you’re new to Kubernetes internals. Understanding the logs and system service behavior helped uncover the root cause quickly.

Hopefully this helps someone trying to set up K3s on bare metal or home lab hardware!