DEV Community: Minor Keith

Kubernetes, Part Three: GitOps with Argo CD, let Git drive your cluster

Minor Keith — Wed, 05 Aug 2026 02:58:58 +0000

In Part One I deployed nginx on a single k3s node and killed a Pod to watch a replacement appear. In Part Two I pulled the config and the passwords out of the image. Both parts had the same weak spot, and it's the one nobody mentions when they teach you kubectl: I was still standing at a terminal typing kubectl apply at a cluster.

That works fine until it doesn't. A month later nobody can tell you what's actually running, or who changed it, or what the cluster looked like before someone "just quickly fixed" something at 11pm. The YAML on your laptop and the YAML in the cluster quietly drift apart, and the only way to find out is to go and look.

Argo CD fixes that by taking the terminal away from you. You put your manifests in Git, you tell Argo CD where that repo is, and from then on the repo is the truth. Change the cluster by hand and Argo CD changes it back. I ran the whole thing on the same 4GB box that already hosts my Forgejo instance, and I'm writing it down because six separate things bit me and I'd rather not rediscover any of them.

TL;DR Put your manifests in a Git repo first. Install Argo CD with kubectl apply --server-side (plain apply fails on one oversized CRD). Create an Application object that names three things: which repo, which folder, where it goes. Sync it once by hand and watch OutOfSync become Synced. Then set syncPolicy.automated with selfHeal: true and prune: true, and try to fight it: scale a Deployment, delete a Service, swap an image. It puts all of them back within fifteen seconds. Deploy by pushing to Git, roll back with git revert. And remember that Synced means "matches Git", not "works".

The one idea worth holding onto

Part One's idea was that you declare the state you want and Kubernetes makes reality match it. Part Three is the same sentence moved one level out:

You declare the state you want in Git. Argo CD makes the cluster match it.

Kubernetes watches its own objects. Argo CD watches your repo. That's the entire product, and every feature below is a variation on it.

What you get for free is the part I didn't expect. Because every change to the cluster is now a commit, Git becomes your audit log, your rollback mechanism, and your review gate without you building any of those things. "Who scaled this to five replicas and why" stops being an unanswerable question.

Before you start

You need a working cluster and a kubectl that reaches it without sudo. That's Part One, Step 0, the KUBECONFIG fix that catches everyone. You also need a GitHub account, and about forty minutes.

Mine is a single k3s node, v1.36.2, on a 4GB Hetzner box in Falkenstein that was already running Forgejo in Docker. Argo CD cost me about 500MB across seven Pods, taking the box from 2.3GB free to 1.8GB free. It fits on a small machine comfortably. Nothing got OOM killed.

One warning that matters if your box has a public IP like mine does. Argo CD ships with an admin account and a bootstrap password, and Step 6 is where I make sure the web UI is reachable from my devices and from nowhere else. Don't skip it and don't reorder it.

Step 1: the repo comes first

This ordering is the whole philosophy in miniature, so do it in this order even though it feels backwards: the repo exists before Argo CD does.

Create a new public repo. Mine is peculiarengineer-gitops, and it holds exactly two files under apps/hello/.

Save the first as apps/hello/deployment.yaml:

# A Deployment says "keep N copies of this container running, forever."
# Argo CD's job is to make sure this file and the cluster always agree.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app: hello
spec:
  replicas: 2 # change this number, push, and watch Argo CD notice
  selector:
    matchLabels:
      app: hello # which Pods this Deployment owns
  template:
    metadata:
      labels:
        app: hello # must match the selector above, or nothing happens
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          resources:
            # Small box, so be explicit about what this is allowed to eat.
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 64Mi

And the second as apps/hello/service.yaml:

# A Service gives the Pods above one stable address inside the cluster.
# ClusterIP means "reachable from inside the cluster only", so nothing is
# published to the internet by this file.
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  type: ClusterIP
  selector:
    app: hello # sends traffic to any Pod carrying this label
  ports:
    - port: 80
      targetPort: 80

Nothing new here. This is Part One's Deployment and Service with resource limits added, because a 4GB box deserves them. Commit and push both.

💡 What happened: You wrote down what you want running without running any of it. From here on, editing these files is how you change the cluster. That's the habit the rest of the post builds.

Step 2: install Argo CD

Argo CD installs into the cluster as ordinary Kubernetes objects in its own namespace. No operator, no Helm required, no packages on the host.

▶ Do:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

❌ And that fails. Most of it applies, then right at the end:

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes

This one confused me for a minute because the error blames the CRD, and the CRD is fine. kubectl apply stashes a copy of the entire manifest it just applied into a last-applied-configuration annotation so it can compute diffs later. The ApplicationSet CRD is bigger than the 256KB limit Kubernetes puts on annotations, so the copy can't be stored.

Server side apply doesn't use that annotation at all. It hands the whole document to the API server and lets it track ownership properly.

▶ Do:

kubectl apply -n argocd --server-side -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

✅ You should see three CRDs land:

customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io serverside-applied

Now wait for the Pods. Seven of them come up, and on my box that took about forty seconds:

kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

💡 What happened: Argo CD is now a workload in your cluster like any other, watching for Application objects that don't exist yet. Worth noticing that it installed by declaring a pile of YAML, which is the same trick it's about to do on your behalf.

Step 3: the CLI and the admin password

Grab the CLI on the node. This is the Linux x86 build, so if you'd rather run it from a Mac or an ARM box, take the matching binary from the releases page.

▶ Do:

sudo curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo chmod +x /usr/local/bin/argocd
argocd version --client --short

✅ You should see a version line and nothing else:

argocd: v3.4.5+564b949

The installer generates a random admin password and leaves it in a Secret. Reading it is a callback to Part Two, base64 and all:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ; echo

Same reminder as Part Two: base64 is encoding, not encryption. You just decoded a password with a pipe. Step 12 replaces this one properly.

Step 4: the Application, and the moment it clicks

This is the object the whole product hangs off, and it's smaller than you'd think. An Application answers three questions and nothing else. Which repo. Which folder inside it. Where the result goes.

You can create it with argocd app create, but write the YAML instead. Seeing the object is what makes the idea stick. Save it as hello-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello
  namespace: argocd # Applications live in the argocd namespace
spec:
  project: default
  source:
    repoURL: https://github.com/peculiarengineer-mk/peculiarengineer-gitops.git
    targetRevision: main # which branch to track
    path: apps/hello # which folder in the repo to apply
  destination:
    server: https://kubernetes.default.svc # this same cluster
    namespace: default # where the manifests land
  # No syncPolicy yet. This first one syncs only when we tell it to.

Swap in your own repo URL. Leaving syncPolicy out is deliberate, and it's the best decision in this whole tutorial, because it lets you see the next bit.

▶ Do:

kubectl apply -f hello-app.yaml
sleep 5
kubectl get application hello -n argocd \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status

✅ You should see two words that are the entire pitch:

NAME    SYNC        HEALTH
hello   OutOfSync   Missing

💡 What happened, and this is the paragraph I'd underline. Nobody asked it to do that. In the five seconds since you created the object, Argo CD cloned your repo, read apps/hello, compared both files against the live cluster, and reported the gap: OutOfSync means Git and the cluster disagree, Missing means those objects don't exist yet. It will keep doing that comparison forever, whether or not you ever sync anything.

The sync is almost an afterthought. The diff is what you're actually buying.

Step 5: the first sync

The argocd CLI has two modes and they behave differently, which caused me the second gotcha. --core talks straight to the Kubernetes API with no Argo CD server in the middle. It's the quickest way to work from the node.

▶ Do:

argocd app sync hello --core

❌ And it fails with a message that is actively misleading:

{"level":"fatal","msg":"configmap \"argocd-cm\" not found"}

Go and look and argocd-cm is sitting right there in the argocd namespace. The problem is that core mode has no server session telling it where Argo CD lives, so it reads the namespace from your kubeconfig context. On k3s that context has no namespace set, so it looks in default and finds nothing.

Setting ARGOCD_NAMESPACE does not help. I tried, twice.

The obvious fix is kubectl config set-context --current --namespace=argocd, and that's the third gotcha, because it poisons every kubectl command you run on that box afterwards. Suddenly kubectl get pods shows you Argo CD's internals instead of your app, forever, and you will not remember why. Use a separate kubeconfig for the argocd CLI instead.

▶ Do:

cp ~/.kube/config ~/.kube/argocd-core.yaml
KUBECONFIG=~/.kube/argocd-core.yaml kubectl config set-context --current --namespace=argocd
KUBECONFIG=~/.kube/argocd-core.yaml argocd app sync hello --core

That copies the kubeconfig Part One had you set up, so you're not going back to the root owned /etc/rancher/k3s/k3s.yaml that Step 0 told you to stop touching.

✅ You should see:

Sync Status:        Synced to main (d1395f1)
Phase:              Succeeded
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME   STATUS  HEALTH       MESSAGE
       Service     default    hello  Synced  Healthy      service/hello created
apps   Deployment  default    hello  Synced  Progressing  deployment.apps/hello created

Check what landed:

kubectl get deploy,pods -l app=hello -n default

deployment.apps/hello   2/2   2   2   22s
pod/hello-5d6b7fbfc4-f6kxf   1/1   Running   0   23s
pod/hello-5d6b7fbfc4-p4x64   1/1   Running   0   23s

That KUBECONFIG= prefix is needed on every argocd command from here on, and typing it each time gets old fast. Set an alias for the rest of the session:

alias argocd='KUBECONFIG=~/.kube/argocd-core.yaml argocd'

Every argocd command below assumes you did that. If one of them ever comes back with configmap "argocd-cm" not found, you're in a new shell and the alias is gone.

💡 What happened: You never ran kubectl apply on those manifests. Argo CD read GitHub and created both objects itself. That's a small thing on two files and a very large thing on two hundred.

Step 6: reach the UI without publishing it to the internet

The CLI does everything, but the web UI is genuinely good and the resource tree is the bit that makes Kubernetes legible to people who don't live in it.

Here's the trap. A k3s NodePort binds to 0.0.0.0, which on a box with a public IP means the open internet. Argo CD has a known admin username and a bootstrap password. So the firewall goes on first and the service gets exposed second. If you get that order backwards you spend a few minutes with an unauthenticated door open, and that's plenty.

My box wasn't on my tailnet at all. Only the Forgejo container was, through its sidecar. So the host joins properly:

curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --hostname=k3s-1 --accept-dns=false --accept-routes=false

Both of those flags are deliberate. --accept-dns=false stops Tailscale rewriting /etc/resolv.conf, and --accept-routes=false stops it touching the routing table. On a box already running k3s and Docker I want Tailscale to add an interface and change nothing else.

Now the firewall, and here's the fourth gotcha, which is the one that can genuinely take your services down. Ubuntu ships this in /etc/default/ufw:

DEFAULT_FORWARD_POLICY="DROP"

Enable ufw with that set and you break pod to pod networking in k3s and Docker's bridge. On my box that would have taken Forgejo down as collateral. Fix it before you enable anything:

sed -i 's/^DEFAULT_FORWARD_POLICY=.*/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw

Before you turn the firewall on, arm a dead man switch. Enabling a firewall over SSH on a box whose only access is SSH is how people lose servers, and a Hetzner rescue console at midnight is a bad time.

▶ Do:

setsid nohup bash -c "sleep 300; ufw --force disable" >/dev/null 2>&1 < /dev/null &
echo $! > /root/ufw-deadman.pid

That undoes the firewall in five minutes unless you cancel it. Now the rules. SSH, the tailnet, and the k3s internal networks:

ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow in on tailscale0
ufw allow 41641/udp        # tailscale wireguard
ufw allow in on cni0       # k3s pods
ufw allow in on flannel.1
ufw allow in on docker0
ufw allow from 10.42.0.0/16   # k3s pod cidr
ufw allow from 10.43.0.0/16   # k3s service cidr
ufw --force enable

Open a second SSH session and confirm you can still get in. Check your cluster and anything else on the box is still healthy. Then cancel the timer using the PID file:

kill $(cat /root/ufw-deadman.pid)

Use the PID file, not pkill -f "sleep 300". I tried the pkill version and it killed my own shell, because the command line running that pkill contains the string sleep 300 and therefore matches itself. Exit code 255, dropped connection, and a few seconds of wondering whether I'd just locked myself out.

Now expose the UI:

kubectl -n argocd patch svc argocd-server -p '{"spec":{"type":"NodePort","ports":[
  {"name":"http","port":80,"targetPort":8080,"protocol":"TCP"},
  {"name":"https","port":443,"targetPort":8080,"protocol":"TCP","nodePort":30443}]}}'

If you go looking for a listening socket you won't find one, and that threw me for a second:

ss -tlnp | grep 30443    # returns nothing, and that's correct

kube-proxy implements NodePorts with iptables rules, not a bound socket. The test that actually matters is reaching it from two places:

https://100.87.x.y:30443  ->  HTTP 200     # over the tailnet
https://203.0.113.88:30443  ->  HTTP 000     # public IP, blocked

💡 What happened: The UI is reachable from every device on my tailnet, including the iPad, and invisible from everywhere else. The cert is self signed so your browser will complain once. Log in as admin with the password from Step 3, click into the hello app, and you get the resource tree: Application, Deployment, ReplicaSet, Pods, each with a health dot. That view is why people keep the UI around.

Step 7: break the cluster on purpose

Now the interesting part. Do the exact thing GitOps is supposed to prevent.

▶ Do:

kubectl scale deployment hello -n default --replicas=5
sleep 15
kubectl get application hello -n argocd \
  -o custom-columns=SYNC:.status.sync.status,HEALTH:.status.health.status

✅ You should see:

SYNC        HEALTH
OutOfSync   Healthy

Five Pods running, and Argo CD spotted it within seconds and did nothing at all.

That's not a bug, it's the lesson. Detecting drift and correcting drift are separate features, and right now you only have the first one. Seeing Argo CD notice and deliberately sit on its hands is what makes the next step land.

Healthy sitting next to OutOfSync is worth its own thought too. Health and sync are different axes. Health means the five Pods are fine. Sync means this matches Git. An app can be in perfect health and completely wrong.

Ask what's wrong and it tells you exactly:

argocd app diff hello --core

===== apps/Deployment default/hello ======
120c120
<   replicas: 5
---
>   replicas: 2

< is the live cluster, > is Git. Four lines that answer "how does reality differ from what I said I wanted". You can run that against anything, at any time, and get a real answer.

Step 8: turn on self-heal and try to win

Three settings. selfHeal corrects drift, prune deletes objects whose files vanish from the repo, and automated means it acts without being asked.

▶ Do:

kubectl -n argocd patch application hello --type=merge -p '{
  "spec": {"syncPolicy": {"automated": {"prune": true, "selfHeal": true}}}
}'

Then watch, without running any sync command:

 5s  OutOfSync Healthy   replicas=5  pods=5
10s  Synced Healthy      replicas=2  pods=2

Five Pods became two on their own. I changed a policy, not the cluster, and Argo CD dragged reality back to what the repo says. For my money this is the moment the whole idea clicks, more than the first sync was.

Now try to win. I made three attempts and lost all three inside fifteen seconds:

What I did	What happened
`kubectl scale --replicas=4`	back to 2
`kubectl delete svc hello`	Service recreated
`kubectl set image nginx=nginx:1.25`	back to `nginx:1.27`

💡 What happened, plus the fifth gotcha. Look closely at that recreated Service and it has a different ClusterIP and an age of fifteen seconds. Argo CD did not restore the object you deleted. It created a new one that matches the file. Anything that cached the old address is now talking to nothing. Self-heal keeps your cluster matching your repo, and that is not the same as a backup. Don't let it talk you out of having real backups.

Step 9: deploy by pushing to Git

The whole point. Edit the repo, not the cluster.

▶ Do change replicas: 2 to replicas: 3 in apps/hello/deployment.yaml, then:

git commit -am "Scale hello to 3 replicas"
git push

Now touch nothing and watch:

  0s  revision=d1395f1  replicas=2
...
124s  revision=d1395f1  replicas=2
136s  revision=e6c57cb  replicas=3

136 seconds. That isn't Argo CD being slow, it's the default polling interval. With no webhook configured it asks GitHub "anything new?" every three minutes. In production you point a repo webhook at it and deployments become instant. While you're writing a tutorial and don't want to wait, force it:

argocd app get hello --hard-refresh --core

💡 What happened: Check the Pod ages afterwards and you get 28m, 28m, 2s. It scaled up. It did not redeploy. Argo CD works out the difference between the repo and the cluster and applies only that, so a one line change doesn't churn your running workload.

Step 10: delete a file, delete the object

prune is the setting that decides whether removing a file removes the thing it described. I tested it with something disposable.

Add apps/hello/temp-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: hello-temp
data:
  note: 'delete this file from git and prune should remove the object'

Commit, push, hard refresh, and the ConfigMap appears in the cluster. Now git rm the file, commit, push, hard refresh again:

kubectl get configmap hello-temp -n default

Error from server (NotFound): configmaps "hello-temp" not found

💡 What happened: Deleting a file deleted the object. Worth knowing that prune: false is the default, and with it off Argo CD will happily add and update things but never remove them. That sounds safer and mostly it is, but you end up with orphaned objects in your cluster that no file in any repo describes and nobody remembers creating. Turn prune on early, while your cluster is small enough that a mistake is obvious.

Step 11: ship something broken, then roll it back

This is the most important section in the post, so I deliberately shipped a release that could not possibly work. In deployment.yaml:

image: nginx:9.9.9-doesnotexist

Commit, push, refresh, wait thirty seconds:

=== app health ===
Synced   Progressing

=== pods ===
hello-5d6b7fbfc4-f6kxf 1/1 Running       32m
hello-5d6b7fbfc4-p4x64 1/1 Running       32m
hello-5d6b7fbfc4-vzq7x 1/1 Running       6m
hello-6d96c85587-lgxml 0/1 ErrImagePull  31s

=== is the site still up? ===
HTTP 200

Read that first line again. Synced, and completely broken.

Synced means the cluster matches Git. It does not mean the app works. I put something wrong in the repo, so Argo CD faithfully made the cluster wrong. GitOps guarantees fidelity to your repo, not correctness, and if you remember one sentence from this post make it that one. Your repo is now the thing that needs code review, because it's the thing that ships.

The site stayed up for a reason worth knowing: Kubernetes' rolling update will not kill a healthy Pod until its replacement reports Ready, so three good Pods kept serving while the fourth failed to pull. It also sits in Progressing for a full ten minutes before it admits to being Degraded, which is progressDeadlineSeconds defaulting to 600 rather than Argo CD dragging its feet. I stared at Progressing for a while assuming something was stuck.

Now the fix, and notice what the fix is:

git revert --no-edit <bad-sha>
git push

Synced   Healthy
image: nginx:1.27
HTTP 200

Pod ages afterwards: 32m, 32m, 6m. Not one Pod died. The bad release never took hold, the old Pods served throughout, and recovery was a normal commit.

💡 What happened: Your rollback procedure is the same command as every other change. No special runbook, no "quick fix" applied straight to prod that nobody writes down. The revert is right there in the log next to the mistake it undoes.

Small thing that cost me a minute: git revert -q is not a flag. It quietly prints the usage text and reverts nothing, and if you're not reading closely you'll push an empty change and wonder why nothing recovered.

Step 12: rotate that admin password

The bootstrap password from Step 3 is meant to be temporary. The documented command is argocd account update-password, and here's the sixth gotcha:

$ argocd account update-password --account admin ... --core
failed to get issue time: unable to extract token claims

Core mode has no server session, so there's no token to authenticate a password change. You either log in through the API server properly, or you write the hash yourself:

NEW="pick-something-better-than-this"
HASH=$(argocd account bcrypt --password "$NEW")
kubectl -n argocd patch secret argocd-secret \
  -p "{\"stringData\":{\"admin.password\":\"$HASH\",\"admin.passwordMtime\":\"$(date +%FT%T%Z)\"}}"
kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd delete secret argocd-initial-admin-secret

Confirm both directions. The old password should be refused:

$ argocd login <host>:30443 --username admin --password "<old>" --insecure --grpc-web
Invalid username or password

$ argocd login <host>:30443 --username admin --password "<new>" --insecure --grpc-web
'admin:login' logged in successfully

Worth understanding that those two CLI modes really are different tools wearing the same name. --core skips the server and talks to the Kubernetes API, which is fast and needs no login but can't do anything that depends on a session. --server <host> --grpc-web goes through argocd-server with a real login, which is what you want for anything touching accounts, tokens, or RBAC.

Gotchas I hit

Plain kubectl apply fails on the install manifest. The ApplicationSet CRD exceeds the 256KB annotation limit that apply needs for its last-applied-configuration copy. Use --server-side. The error blames the CRD, which sends you looking in the wrong place.
argocd --core reads its namespace from the kubeconfig context. Not from ARGOCD_NAMESPACE, which does nothing. On k3s the context has no namespace, so you get configmap "argocd-cm" not found while the ConfigMap is plainly sitting in the argocd namespace.
Fixing that by repointing your main kubeconfig poisons everything else. kubectl config set-context --current --namespace=argocd means every later kubectl get pods on that box shows Argo CD internals. Keep a separate kubeconfig for the argocd CLI.
DEFAULT_FORWARD_POLICY="DROP" breaks k3s and Docker when you enable ufw. Set it to ACCEPT in /etc/default/ufw first, and allow the pod and service CIDRs. Skip this and you take down pod networking and every container on the box at the same time.
Self-heal is not a backup. Delete a Service and Argo CD creates a new one matching the file, with a new ClusterIP. It restores the description, not the object. Anything holding the old address is now pointing at nothing.
argocd account update-password does not work in core mode. No session, no token, no password change. Patch the bcrypt hash into argocd-secret instead.
Synced does not mean working. It means the cluster matches Git. Put a broken image tag in the repo and you get a proudly Synced broken app.
Don't cancel a background timer with pkill -f "sleep 300". The shell running that command contains the string, matches itself, and kills your session. Use a PID file.

Quick command reference

Goal	Command
Install Argo CD	`kubectl apply -n argocd --server-side -f <install.yaml>`
Read bootstrap password	`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" \
App status	{% raw %}`kubectl get application <app> -n argocd`
See Git versus cluster	`argocd app diff <app> --core`
Sync now	`argocd app sync <app> --core`
Skip the 3 minute poll	`argocd app get <app> --hard-refresh --core`
List apps through the server	`argocd app list --server <host>:30443 --insecure --grpc-web`
Turn on self-heal and prune	patch `spec.syncPolicy.automated` with `selfHeal` and `prune`
Roll back	`git revert --no-edit <sha> && git push`

Where this series goes next

You now have a cluster that argues with you, which is the correct behaviour. Change something by hand and it changes it back. Deploy by pushing a commit. Roll back with git revert and lose nothing.

There's one loose thread I left deliberately, and it's the obvious next part. That hello-app.yaml from Step 4 is still sitting on the node, applied by hand with kubectl. The thing driving all my GitOps is itself not under GitOps, which is a slightly embarrassing place to stop. The fix has a good name, app of apps, and it answers the question of who deploys the deployer.

After that: repo webhooks to kill the three minute poll, private repo authentication so this can point at my own Forgejo rather than GitHub, and Helm and Kustomize as sources instead of plain YAML. The Kubernetes series hub tracks the lot, and Labels, Volumes, health probes and Ingress are all still on the list.

Go and scale something by hand, then watch it change back while you're still looking at it. That's the bit that made it real for me. [ synced ]

Originally published at peculiarengineer.com.

Self-host Forgejo on your tailnet with Docker Compose and Tailscale

Minor Keith — Thu, 30 Jul 2026 02:53:00 +0000

You want your own Git host. Maybe you're getting off GitHub, maybe you just want somewhere private to keep the repos that shouldn't be on someone else's servers. So you go looking, and every guide hands you the same shape: a public DNS record, ports 80 and 443 open to the entire internet, a reverse proxy, a certificate, and a login page that anyone on earth can now knock on.

For a private code host, that's a strange trade. Nothing about "my repos, for me and three collaborators" requires a public address. If the only people who should reach it are people you already trust, put it on your tailnet and the whole category of internet facing problems stops existing. No open ports, and no login page getting scanned at three in the morning.

The Compose file for this is short. What makes it worth writing down is the sidecar pattern it uses, which is not obvious the first time, and a handful of settings that decide whether it works at all. I built this from nothing on a fresh Ubuntu 26.04 box to check it, and the settings that went wrong were not the ones I expected, so those get their own section at the end.

TL;DR. Run Tailscale as its own container and give Forgejo network_mode: service:ts-forgejo so Forgejo has no published ports and its own tailnet identity. Point TS_SERVE_CONFIG at a serve JSON file and you get real HTTPS on forgejo.your-tailnet.ts.net with no port 80 and no HTTP-01 challenge. Persist /var/lib/tailscale or every restart creates a brand new node. Tag the device so its node key never expires. Do not set START_SSH_SERVER, because the image already runs sshd on port 22.

Prerequisites

A box with Docker installed. Mine runs Ubuntu 26.04, but nothing here is version specific.
A Tailscale account and a tailnet you can already log into. If you're starting cold, install Tailscale on the host first so you have something to test from.
Admin access to the Tailscale console, because two of the steps happen there and not on the box.
MagicDNS turned on. It's on by default for new tailnets, and without it you get an IP instead of a name.

You do not need a domain, a DNS record, a certificate, or a single open port in your firewall. That's the point.

The one thing to get straight

Forgejo is not sitting behind a reverse proxy here. Forgejo is the Tailscale node.

That distinction is the whole post. In the usual setup you'd run Caddy or Nginx on the host, publish ports, and forward traffic to a container. Here, the Tailscale container owns a network namespace, and Forgejo is placed inside that same namespace with network_mode: service:ts-forgejo. The two containers share one network stack the way two processes on one machine do.

Once that clicks, the rest follows on its own:

Forgejo publishes no ports to the host, so docker ps shows nothing listening and the host firewall has nothing to do.
The machine appears in your tailnet as its own device, with its own name and its own ACL rules, separate from the host it happens to run on.
Port 22 inside that namespace belongs to Forgejo, and the host's real sshd is somewhere else entirely as far as the network stack is concerned. The port collision that makes the public version of this setup annoying never happens.
HTTPS arrives without opening port 80. The certificate still comes from Let's Encrypt, but through a DNS-01 challenge that Tailscale completes for you by publishing the _acme-challenge TXT record under ts.net for your node. Nothing has to be reachable from the public internet for validation to pass.

The result is a real, publicly trusted certificate on a machine with no public presence at all. Your browser sees a normal padlock. Nothing is exposed.

1. Turn on HTTPS for your tailnet

Do this before anything else. Skipping it does not stop the stack coming up, which is exactly why it wastes your time later.

In the Tailscale admin console, go to DNS, and under HTTPS Certificates click Enable HTTPS. Note the tailnet name it shows you, something like tail1234.ts.net. Every device in your tailnet gets a name under it, so your Forgejo node will end up at forgejo.tail1234.ts.net.

If you skip this, the sidecar still starts and still says it's running. It logs a line telling you HTTPS is not enabled and links the docs, which is fair enough, but tailscale serve status just answers No serve config with no reason attached. The command that gives you a straight answer is:

docker compose exec ts-forgejo tailscale cert forgejo.tail1234.ts.net

With HTTPS off you get your Tailscale account does not support getting TLS certs. With it on you get two files written and you can move on. I use that as the check before touching anything else, because every other symptom of this is ambiguous.

2. Make a tagged auth key

The container needs a key to join the tailnet unattended, and the way expiry works here catches people out.

Auth keys cap out at 90 days, but that expiry only stops new devices from joining. A node that already registered keeps working until its own node key expires, and that defaults to 180 days. So a plain reusable key gets you roughly six months before your Git host quietly drops off the tailnet on a day you weren't touching anything.

The fix is a tag. Key expiry is disabled by default for tagged devices, because they're owned by the tailnet rather than by a user.

First define the tag in your ACL file, under Access controls:

{
  "tagOwners": {
    "tag:container": ["autogroup:admin"]
  }
}

Then go to Settings, Keys, and generate an auth key with Reusable on and the tag tag:container applied. Copy it somewhere safe now, because the console shows it exactly once.

Put it in a .env file next to your Compose file:

# .env
TS_AUTHKEY=tskey-auth-xxxxxxxxxxxx

And keep that file out of Git:

echo ".env" >> .gitignore

3. The Compose file

Two services. The first is the network, the second is the application.

services:
  ts-forgejo:
    image: tailscale/tailscale:latest
    container_name: ts-forgejo
    hostname: forgejo                      # becomes forgejo.your-tailnet.ts.net
    environment:
      - TS_AUTHKEY=${TS_AUTHKEY}
      - TS_EXTRA_ARGS=--advertise-tags=tag:container
      - TS_STATE_DIR=/var/lib/tailscale    # persisted, see the volume below
      - TS_SERVE_CONFIG=/config/serve.json
      - TS_USERSPACE=false                 # use the kernel networking path
    volumes:
      - ts-forgejo-state:/var/lib/tailscale
      - ./ts-config:/config
    devices:
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - net_admin
      - sys_module
    restart: unless-stopped

  forgejo:
    image: codeberg.org/forgejo/forgejo:15
    container_name: forgejo
    network_mode: service:ts-forgejo       # the whole trick, one line
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - FORGEJO__server__ROOT_URL=https://forgejo.tail1234.ts.net/
      - FORGEJO__server__SSH_DOMAIN=forgejo.tail1234.ts.net
      - FORGEJO__server__SSH_PORT=22
      - FORGEJO__service__DISABLE_REGISTRATION=true
    volumes:
      - forgejo-data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - ts-forgejo
    restart: unless-stopped

volumes:
  ts-forgejo-state:
  forgejo-data:

A few of those lines are doing more work than they look like they are.

network_mode: service:ts-forgejo is the pattern. Notice what is absent: there is no ports: block anywhere in this file. Forgejo listens on 3000 and 22 inside the shared namespace, reachable over the tailnet and nowhere else.

TS_USERSPACE=false matters more than it looks. The Tailscale image defaults to userspace networking, and in that mode the TUN device and the net_admin capability sit there unused. Turning it off puts the container on the kernel networking path those lines exist for, which is what you want when something in the namespace needs to answer on a real port.

hostname: forgejo is what decides the name. Tailscale registers the container under it, so this is the value that ends up in your URL. Pick it before first boot, because renaming a node later means fixing ROOT_URL too.

TS_STATE_DIR with a real volume behind it is the difference between one node and a hundred. More on that in the gotchas below.

FORGEJO__server__ROOT_URL uses Forgejo's double underscore convention, where FORGEJO__section__KEY maps to a key in app.ini. Get this value right before you ever start the container. Forgejo bakes it into the clone URLs it shows you, the links in its emails, and its webhook targets. Wrong here means every clone command your users copy points at the wrong place, and it looks fine on screen right up until someone tries it.

There is also something deliberately missing. You will see START_SSH_SERVER=true in a lot of Forgejo examples, and it does not belong here. The rootful image runs OpenSSH on port 22 by itself, unconditionally. Setting that variable starts Forgejo's own Go SSH server as well, and now two SSH servers want the same port in the same namespace. Leave it out and the image does the right thing on its own. That variable is for the rootless image, which is a different setup with SSH on 2222.

DISABLE_REGISTRATION=true from the first boot means nobody gets to sign up while you're still setting things up. You'll create your admin account through the installer instead.

4. The serve config

Create ts-config/serve.json next to your Compose file:

{
  "TCP": {
    "443": {
      "HTTPS": true
    }
  },
  "Web": {
    "${TS_CERT_DOMAIN}:443": {
      "Handlers": {
        "/": {
          "Proxy": "http://127.0.0.1:3000"
        }
      }
    }
  },
  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": false
  }
}

${TS_CERT_DOMAIN} is substituted by the Tailscale container at startup with the node's real name, so you don't have to hardcode your tailnet name in a second place.

The proxy target is 127.0.0.1:3000, and that is only correct because of the shared namespace. Forgejo really is on localhost from Tailscale's point of view. If you ever pull these two apart into separate networks, this line breaks first.

AllowFunnel is set to false on purpose. It's the switch that would put this node on the public internet, and having it present and off is better than having it absent, because you can see what the answer currently is.

Bring it up:

docker compose up -d
docker compose logs -f ts-forgejo

Watch for the node registering and the certificate being issued. Then check it from another machine on your tailnet:

tailscale status | grep forgejo
curl -I https://forgejo.tail1234.ts.net/

A 200 and a valid certificate means the hard part is done.

5. First boot and locking the installer

Open https://forgejo.tail1234.ts.net/ in a browser on any device in your tailnet and you get Forgejo's setup page.

Two things matter here. Leave the database as SQLite unless you have a reason not to. For a personal or small team instance it's genuinely fine, it's one less container to run and one less thing to back up, and you can migrate later if the instance grows into something that needs Postgres. And check that the URL fields on the form match the ROOT_URL you set, because the installer will happily write different values into app.ini and leave you with two sources of truth.

Create your admin account on that same form. Do not skip it and do it later.

The reason for the urgency is smaller here than on a public box, but it still applies: until the installer is completed, whoever reaches that page can complete it and become the administrator. On a tailnet that's limited to devices you've already authorised, which is a much shorter list than "the internet". It's still a list. Finish the form.

Once you're in, confirm the install lock landed:

docker compose exec forgejo grep INSTALL_LOCK /data/gitea/conf/app.ini
# INSTALL_LOCK = true

6. Cloning over SSH

This is the part that would have been a whole section of fighting in the public version, and here it's almost nothing.

git clone git@forgejo.tail1234.ts.net:yourname/yourrepo.git

Port 22, no custom port, no -p 2222 to remember, no conflict with the host's own sshd. The image's SSH server owns port 22 inside its namespace, and the host's sshd owns port 22 on the host, and they never meet.

Add your public key in the Forgejo UI under Settings, SSH / GPG Keys, the same as you would anywhere else.

One thing to check: the clone URL Forgejo displays comes from SSH_DOMAIN and SSH_PORT. If those disagree with reality, the button copies a command that fails, and the error the user sees is a connection timeout that tells them nothing about why.

Who can actually reach it

Right now, every device in your tailnet can reach it. That's fine for a tailnet of one person and three laptops. It stops being fine the moment you add a contractor's machine or a server that runs someone else's code.

ACLs fix that. This is a fragment to merge into your existing policy, not a whole policy file. If you paste it over the top of everything, you delete the default rule that lets your tailnet talk to itself, and every other connection you have stops working at the same moment.

{
  "groups": {
    "group:devs": ["you@example.com", "collaborator@example.com"]
  },
  "tagOwners": {
    "tag:container": ["autogroup:admin"]
  },
  "acls": [
    // ... your existing rules stay here ...
    {
      "action": "accept",
      "src": ["group:devs"],
      "dst": ["tag:container:443,22"]
    }
  ]
}

If you already worked through the ACL section of Tailscale for private networking, this is the same machinery pointed at a container instead of a server.

When you want it public after all

Sometimes you need to hand someone a link without adding them to your tailnet. Tailscale Funnel does that, and it's less of a one liner than it looks.

Two things have to be true first. Your tailnet policy needs the funnel node attribute, which for a tagged container node you will be adding by hand:

"nodeAttrs": [
  {
    "target": ["tag:container"],
    "attr": ["funnel"]
  }
]

Then flip the switch in serve.json and restart, which is the durable way to do it since TS_SERVE_CONFIG reapplies that file every time the container starts:

  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": true
  }

Restart with docker compose restart and no service name after you edit that file. Restarting only the Tailscale container is what leaves Forgejo answering 502 behind a network stack that moved out from under it, which is the gotcha below and the one that wastes the most time.

Funnel only works on ports 443, 8443, and 10000, so 443 is the one you want anyway.

The node attribute is not optional, and skipping it is the worst kind of failure. I flipped AllowFunnel to true without it and everything told me it had worked. tailscale funnel status printed Funnel on with the URL under it, tailscale serve status agreed, and the logs said nothing at all. The name simply never appeared in public DNS, so from outside the tailnet it did not resolve, let alone serve. The node did not have the capability and no part of the tooling mentioned it.

If you turn Funnel on, verify it from something that is not on your tailnet. A phone with WiFi off is enough:

dig +short @1.1.1.1 forgejo.tail1234.ts.net

An empty answer means Funnel is not really on, whatever the CLI told you.

Once it's on, you have inherited every problem this post was avoiding. Registration lockdown and rate limiting become your concern again. It's a good escape hatch and a bad default. Set it back to false when you're done.

The honest limitation: if the people you collaborate with will not join your tailnet, this setup is not for you. Tailnet only means tailnet only. Everyone who touches these repos needs Tailscale on their machine and a place in your ACLs. For a solo developer or a small team that already uses Tailscale, that cost is zero. For an open source project taking drive by contributions, it's a wall.

Gotchas I hit

No state volume, so every restart mints a new node. This is the big one, and I watched it happen. If TS_STATE_DIR has no volume behind it, the container loses its identity, registers again as a fresh device, and Tailscale appends a suffix to keep the name unique. Mine came back as forgejo-1 on a new address while the old forgejo sat there marked offline.

What makes it nasty is how healthy the result looks. The new node got its own certificate within seconds and served Forgejo on 443 without complaint. But ROOT_URL and SSH_DOMAIN live in app.ini inside the data volume, so they still held the old name, and the API cheerfully handed out clone URLs pointing at a node nothing is listening on. The server is fine. Every clone command it gives your users times out. Persist /var/lib/tailscale.

What losing that state does to you depends on your auth key. With a reusable key you get the duplicate node above. With a single use key the container cannot register at all, fails with invalid key: API key ... not valid, and never comes up. Two completely different mornings from the same missing volume.

Auth failures loop instead of stopping. Every registration failure I hit, whether a tag the policy did not allow or a spent key, left the container restarting rather than exiting. With restart: unless-stopped that continues forever. docker compose ps shows restarting, not an error, so if you run up -d and walk away you come back to something that has been failing quietly for however long you were gone. The reason is only ever in docker compose logs.

An untagged node drops off at six months, not three. The auth key expiring at 90 days is the number everyone quotes, but that only blocks new registrations. The node itself runs until its node key hits the 180 day default, which is a much worse way to find out, because by then you've forgotten the setup entirely. Tag the device and key expiry is off.

The tag has to exist before you advertise it. tag:container is my example name, not a default. If it is not in tagOwners in your policy the node refuses to join with requested tags [tag:container] are invalid or not permitted, which is at least an honest error. Use whatever tag your tailnet already has if you have one.

Tailscale SSH swallows Git over SSH. If you add --ssh to TS_EXTRA_ARGS, tailscaled intercepts inbound tailnet connections to port 22 before they ever reach the SSH server in the container. There's no bind conflict and nothing looks broken in the logs. Your clones just stop working. Leave Tailscale SSH off on this node and use it on your other machines.

Restarting the sidecar alone breaks Forgejo in a way that looks like something else. This one cost me the most time, because I did it to myself repeatedly while testing and kept diagnosing the wrong thing. docker compose restart ts-forgejo gives Forgejo a network stack that has moved out from under it. Forgejo keeps running, docker compose ps says both are Up, and the tailnet name still resolves.

What you get is a broken instance that points nowhere useful. The web side answers 502, because tailscaled terminates TLS perfectly well and then cannot reach 127.0.0.1:3000 any more. Git over SSH gives you Connection refused, or just hangs with no banner. Neither symptom points at the container you actually restarted.

The fix is docker compose restart with no service name, so both come back together. I checked this twice: sidecar alone gives 502 and a refused clone, both together gives 200 and a clean clone. Any time you touch the Tailscale container, take Forgejo with it.

ROOT_URL left over from an earlier attempt. If you tried this once with a different hostname, changing the environment variable is not always enough, because the installer wrote the old value into app.ini and that file lives in the volume. Check /data/gitea/conf/app.ini and fix it there, or start from a clean volume.

Quick reference

Setting	What it controls	What breaks without it
`network_mode: service:ts-forgejo`	Puts Forgejo in the sidecar's network namespace	Forgejo is not on the tailnet at all
`TS_USERSPACE=false`	Kernel networking instead of userspace	The TUN device and `net_admin` do nothing
`hostname` on the sidecar	The node name, and so the URL	Random or wrong `ts.net` name
`TS_STATE_DIR` plus a volume	Node identity across restarts	New device on every restart, name drifts
`TS_EXTRA_ARGS=--advertise-tags`	Marks the node as tailnet owned	Node key expires at 180 days, node drops off
`TS_SERVE_CONFIG`	HTTPS and the proxy to port 3000	No TLS, nothing served on 443
`FORGEJO__server__ROOT_URL`	Clone URLs, emails, webhooks	Everything points somewhere wrong
`FORGEJO__server__SSH_DOMAIN`	The SSH clone URL shown in the UI	Copy button hands out a command that times out
no `START_SSH_SERVER`	Lets the image's own sshd own port 22	Two SSH servers want the same port

Useful commands:

docker compose logs -f ts-forgejo                      # node registration and cert issuance
docker compose exec ts-forgejo tailscale status        # is it on the tailnet
docker compose exec ts-forgejo tailscale serve status   # what is being served on 443

# backup, written somewhere writable and then copied out
docker compose exec -u 1000 -w /tmp forgejo \
  forgejo dump -c /data/gitea/conf/app.ini

# the archive is named forgejo-dump-<timestamp>.zip, so read the name back
DUMP=$(docker compose exec -T forgejo sh -c 'ls -1 /tmp/forgejo-dump-*.zip | tail -1')
docker compose cp "forgejo:${DUMP%$'\r'}" ./

That timestamp is the part that bites. forgejo dump prints the filename it wrote and then you are on your own, so a copy command with a fixed name in it fails every time and you find out when you need the backup.

[ no open ports · real certs · git@ over the tailnet ]

Originally published at peculiarengineer.com.

Your MCP server's search is bad: ranking, embeddings, and what each one fixes

Minor Keith — Tue, 28 Jul 2026 02:34:37 +0000

At the end of Part One I left the notes server with a search I described as dumb, and promised to come back for it. Here is where it stands today, on the same query:

search_notes("firewall")

build-your-first-mcp-server-python: Build your first MCP server in Python: give Claude your own notes
caddy-reverse-proxy-docker-compose-ubuntu-26-04: Reverse Proxy Your Containers with Caddy and Docker Compose on Ubuntu 26.04
create-sudo-user-ubuntu-26-04: Create a Sudo User on Ubuntu 26.04
enable-ssh-on-ubuntu-desktop: Enable SSH on an Ubuntu desktop
extend-azure-windows-disk-run-command: Extending C: on locked-down Azure Windows VMs without RDP

It got worse. The top hit is now Part One itself, the post complaining that this search is bad, which mentions "firewall" only because it was quoting this exact output. The blog is up to 51 posts, 17 of them mention firewalls somewhere, and UFW Firewall Basics is still not in the list.

This post fixes it in two stages, because there are two different problems here and only one of them is the one everybody reaches for. The first is a ranking bug and it costs about twenty lines. The second is a query that no amount of ranking will ever answer, and that one costs an embedding model.

TL;DR Score every hit and sort, instead of taking the first five in alphabetical order. That alone puts the right post on top. Then add local embeddings with ollama pull nomic-embed-text, chunked on ## headings and cached in an .npz keyed by path, mtime and model name. Prefix documents with search_document: and queries with search_query:, which doubled how often my index put the right post first. Build the index from a command, not on the first search, because 496 chunks took four minutes on CPU. And keep both searches, because they fail in different places: the exact match misses brute force over a hyphen, and the embeddings miss it because two words are not enough meaning to work with.

Before you start

The finished server from Part One. Everything below edits that server.py.
Ollama running locally. That post is about feeding a GPU for chat models, and none of that applies here: embedding models are small and a CPU is fine for this job.
Two dependencies: uv add ollama numpy

1. The bug was ranking, not matching

Part One already named this, and it is worth quoting because the diagnosis is the whole of Stage One:

The substring match did not fail. It found the UFW post fine. The problem is that there is no ranking at all.

Look at what the original loop does:

hits = []
for path in _posts():
    text = path.read_text(encoding="utf-8")
    if query.lower() in text.lower():
        hits.append(f"{path.stem}: {_title(text, path.stem)}")
    if len(hits) == 5:
        break

_posts() globs the .md files and the .mdx files separately and concatenates the two sorted lists, so the order is alphabetical within each extension and any .mdx file lands at the end no matter what it is called. The loop appends in that order and breaks at five. So search_notes is not returning the five best matches for "firewall," it is returning the first five files in that order that contain the string anywhere, code blocks and links included. u sorts after b, c, and e, so the post that is entirely about firewalls loses to four posts that mention them in passing.

That is worse than returning nothing, because five confident, plausible, wrong results are an answer the model will happily build on.

The fix is to stop treating a match as a boolean. Collect every hit, score it, sort, then truncate:

def _frontmatter(text: str) -> str:
    parts = text.split("---", 2)
    return parts[1] if len(parts) > 2 else ""


def _fm_line(fm: str, key: str) -> str:
    return next((l for l in fm.splitlines() if l.startswith(key)), "")


def _score(path: Path, text: str, q: str) -> int:
    low = text.lower()
    if q not in low:
        return 0

    fm = _frontmatter(low)
    score = 0
    if q in _fm_line(fm, "title: "\"):"
        score += 100
    if q in path.stem.lower():
        score += 60
    if q in _fm_line(fm, "tags:"):
        score += 40
    # Body mentions count for something, but cap them so a 4,000 word post
    # cannot win on length alone.
    score += min(low.count(q), 10) * 3
    return score

Four signals, weighted by how much intent each carries. A title match scores highest because a title is the strongest claim a post makes about its own subject. The slug is close behind, since I hand-write slugs on this blog and they say what the post is about. Tags sit below that, and body count is the tiebreaker, capped at ten so a long post cannot grind out a win on length alone.

Then pull the ranked lookup into its own function, because Stage Two is going to need it separately from the tool:

def _keyword_hits(query: str, k: int = 5) -> list[tuple[str, str]]:
    """Every post containing the phrase, best first, as (slug, title)."""
    q = query.lower()
    scored = []
    for path in _posts():
        text = path.read_text(encoding="utf-8")
        score = _score(path, text, q)
        if score:
            scored.append((score, path.stem, _title(text, path.stem)))

    scored.sort(key=lambda row: (-row[0], row[1]))
    return [(slug, title) for _, slug, title in scored[:k]]


@mcp.tool()
def search_notes(query: str) -> str:
    """Search my blog posts for a word or phrase.

    Matches the literal phrase anywhere in a post. Results are ranked: a match
    in the title, slug or tags outranks a passing mention in the body. Returns
    the 5 best hits with slug and title. Use get_note with a slug to read a
    full post.
    """
    hits = _keyword_hits(query)
    if not hits:
        return f"No posts mention {query!r}."
    return "\n".join(f"{slug}: {title}" for slug, title in hits)

Same query, same corpus:

ufw-firewall-basics-ubuntu: UFW Firewall Basics on Ubuntu
hardening-ubuntu-desktop: Hardening an Ubuntu Desktop
ssh-connection-refused-port-22-ubuntu: Troubleshooting "ssh: connect to host port 22: Connection refused"
build-your-first-mcp-server-python: Build your first MCP server in Python: give Claude your own notes
enable-ssh-on-ubuntu-desktop: Enable SSH on an Ubuntu desktop

The UFW post scores 230 against 61 for the runner up, because it hits the title, the slug, the tags and the body. Part One drops to fourth where it belongs.

The docstring changed too. Part One's said "There is no ranking" and told the model to search repeatedly with different wording. That was honest then and is wrong now, and left alone it would keep the model burning three calls to work around a limitation that no longer exists.

If you have fifty markdown files, you could stop reading here. Twenty lines, no new dependencies, no model, nothing to keep in sync. That is a real answer and I am not going to pretend otherwise to justify the rest of the post.

2. The query ranking cannot touch

Here is the one that sent me looking further. I have a post about stopping repeated SSH login attempts. Ask for it the way a person would ask:

search_notes("brute force")

No posts mention 'brute force'.

Not badly ranked, not buried at position nine. Zero, across 51 posts.

Two separate failures are stacked on top of each other there. UFW Firewall Basics says "throttle brute-force knocking," so the word is right there in the corpus, and it does not match because I typed a space where the post has a hyphen. Substring matching is not looking for a concept, it is looking for a byte sequence, and those are two different byte sequences.

The second failure is the one that matters. The post that actually answers this question is the Fail2ban post, and it does not contain the word in any spelling. It never says "brute" or "attack" at all. It opens like this:

A public SSH server starts collecting failed logins almost as soon as it gets an address. Key-only authentication makes those guesses useless, but it does not stop the same addresses filling the journal all day.

"Failed logins," "guesses," "the same addresses." Every one of those describes the thing better than "brute force" does, and not one of them is the phrase somebody would type into a search box. I wrote a whole post about defending against brute force attacks without ever using the term.

This is where ranking runs out. Ranking orders the matches you already have. It cannot manufacture one for a word you never wrote.

And the caller here is not me. It is a model turning somebody's half-remembered question into search terms, and it will not guess my vocabulary. Part One patched around this by telling the model in the docstring to try different wording, which works about as well as it sounds. What the server needs is a search that matches on meaning instead of spelling.

3. What an embedding actually is

A model reads a piece of text and hands back a fixed-length list of floats. That is all. nomic-embed-text gives you 768 of them. The useful property is that texts about similar things land near each other in that 768-dimensional space, so "brute force attack" and "the same addresses filling the journal" end up close together despite sharing no words.

"Near" is cosine similarity, which for unit-length vectors is just a dot product: multiply the two lists element by element, add up the results, get a number between -1 and 1. Higher is more related. That is the entire retrieval algorithm.

Here is what you do not need, despite what a search for this will tell you: a vector database. No Chroma, no pgvector, no Pinecone, no index structure of any kind. Fifty-one posts chunked by heading comes to 496 sections, so the entire index is a 496 by 768 array of float32, about 1.5 MB. Comparing a query against every single one of them is one matrix multiply. You need a real vector store somewhere in the tens of thousands of documents, when scanning everything stops being free. Below that it is a file you can delete when it gets weird, which is worth more than it sounds.

4. Embeddings locally with Ollama

ollama pull nomic-embed-text

The current SDK call is embed(), which takes a list and returns a list:

import ollama

r = ollama.embed(model="nomic-embed-text", input=["hello", "goodbye"])
print(len(r.embeddings), len(r.embeddings[0]))

Two vectors of 768 floats each. The older ollama.embeddings(prompt=...) is the deprecated single-string version. Use embed(): batching matters later.

Two things about this model are not optional, and both fail silently.

Prefix your inputs. nomic-embed-text is trained with task prefixes. Documents go in as search_document: {text} and queries as search_query: {text}. Ollama will not add them for you. Leave them off and nothing breaks: you get vectors, plausible similarity scores, and ranked results. There is no error to chase.

I nearly talked myself out of this one. Comparing raw cosine scores between two documents, prefixed and not, the numbers barely move, which looks like proof that the prefixes are folklore. They are not. Raw cosine is the wrong thing to measure, because a change that shifts every score equally changes no rankings. What matters is where the right post lands.

So I built the index twice over all 496 chunks and ran eight questions with known answers through both:

	Mean reciprocal rank	Correct post first	Mean rank
No prefixes	0.480	2 of 8	3.6
With prefixes	0.682	4 of 8	2.4

Prefixing doubles the number of questions answered correctly on the first result. "Something keeps trying to log into my server over and over" goes from seventh place to first. That is not folklore, and it costs one f-string.

Know where it truncates. Ollama serves this model with a 2,048 token context. Feed it a longer chunk and it drops the tail and tells you nothing, which is the second silent failure in a row.

ollama show will tell you, if you think to ask:

$ ollama show nomic-embed-text
  Model
    architecture        nomic-bert
    parameters          137M
    context length      2048
    embedding length    768
    quantization        F16

  Capabilities
    embedding

  Parameters
    num_ctx    8192

Read those last two numbers together, because between them they are the whole trap. The architecture stops at 2,048, and the model file already asks for 8,192. The setting you were about to reach for is set, and it is not doing anything.

The obvious move is options={"num_ctx": 8192}, since nomic's own model card describes an 8,192 window. It does not work, and it is worth showing how I know, because "the flag had no effect" is a hard thing to prove by staring at vectors.

Take one long document, copy it, and change only the last sentence. If the model reads the whole thing, the two copies must embed slightly differently. If it truncates, the two copies are byte-identical up to the cut and the vectors come back the same:

filler = "The firewall configuration is stored in the usual place. " * 400
a = filler + " ZEBRA QUASAR MARMALADE."
b = filler + " The cat sat quietly on the warm mat."

That is roughly 5,700 tokens of identical text with two very different endings. On Ollama 0.32.4:

num_ctx=None  -> cosine(a,b)=1.000000
num_ctx=8192  -> cosine(a,b)=1.000000

Identical to six decimal places, with and without the flag. Everything past the limit was thrown away in both cases. The clamp is in the Ollama log if you go looking, one WARN line and then the model loads at 2,048 regardless:

level=WARN source=server.go:114 msg="requested context size too large for model"
  num_ctx=8192 n_ctx_train=2048
llama_context: n_ctx = 2048

Treat num_ctx as a request, not a setting, and check it the way I did rather than trusting either the model card or me.

The better answer is to not need it. Chunking on ## headings does most of the work for you. Across this blog, 496 sections, the longest one in any prose post is 7,825 characters, which at the usual four-characters-per-token rule of thumb lands just under the limit. The only sections that clearly blow through it are the four complete-source dumps at the end of the LÖVE posts, and the largest of those are past 8,192 as well, so raising the window would not have rescued them either. A wall of Lua that size should not be competing for search hits anyway. Split those or skip them.

5. Wiring it into the server

Four pieces: chunk, embed, cache, search.

Chunking is not optional. One vector for a 4,000 word post averages every idea in it into mush. The server hardening post covers SSH, the firewall, unattended upgrades, fail2ban and livepatch; embedded whole, it is a vector for "Ubuntu things," equally mediocre at all five. Chunk it and each section gets to be about one thing.

Every post on this blog uses ## headings, so the chunk boundaries already exist. That also means a hit can report which section matched, which is strictly more useful to the model than a slug.

import numpy as np
import ollama

MODEL = "nomic-embed-text"
INDEX = Path(__file__).parent / "index.npz"
FLOOR = 0.5          # minimum cosine score worth returning, see below


def _chunks(path: Path) -> list[tuple[str, str]]:
    """Split a post into (heading, text) pairs on ## boundaries."""
    text = path.read_text(encoding="utf-8")
    title = _title(text, path.stem)
    body = text.split("---", 2)[-1]

    sections, heading, buf = [], "intro", []
    for line in body.splitlines():
        if line.startswith("## "):
            if buf:
                sections.append((heading, "\n".join(buf)))
            heading, buf = line[3:].strip(), []
        else:
            buf.append(line)
    if buf:
        sections.append((heading, "\n".join(buf)))

    # Carry the title into every chunk so a section knows what post it is from.
    return [
        (h, f"{title}\n\n## {h}\n\n{b}")
        for h, b in sections
        if b.strip()
    ]


def _embed(texts: list[str], prefix: str) -> np.ndarray:
    try:
        r = ollama.embed(model=MODEL, input=[f"{prefix}: {t}" for t in texts])
    except (ollama.ResponseError, ConnectionError) as e:
        # ConnectionError is the daemon not running. stderr, never stdout.
        print(f"ollama embed failed: {e}", file=sys.stderr)
        raise
    v = np.array(r.embeddings, dtype="float32")
    # Normalize once, here, so search is a plain dot product later.
    return v / np.linalg.norm(v, axis=1, keepdims=True)

Now the cache, and this is the part where my first design was wrong.

Embedding the whole blog takes real time. On my Mac, CPU only, 496 chunks indexed in 239 seconds. That is 482 ms per chunk, and it is four minutes of wall clock before the first search can return anything.

My instinct was to build the index lazily on the first search, so mcp.run() starts instantly and the handshake never blocks. That is the right instinct and the wrong conclusion: it moves a four minute wait from startup, where nobody is looking, onto a tool call the model is waiting on. Build the index ahead of time instead, from a command you run yourself, and have the server load a file that already exists. The lazy path stays as the fallback for a cold cache, but it should be the exception, not the plan.

So: build it once, keep it on disk, and only re-embed what changed.

def _load_cache() -> tuple[np.ndarray | None, list[tuple]]:
    if not INDEX.exists():
        return None, []
    data = np.load(INDEX, allow_pickle=True)
    if str(data["model"]) != MODEL:      # a model swap invalidates every vector
        return None, []
    return data["vecs"], [tuple(row) for row in data["meta"]]


def _build_index() -> tuple[np.ndarray, list[tuple]]:
    """Load the cached index, re-embedding only posts whose mtime moved."""
    cached_vecs, cached_meta = _load_cache()
    live = {path.stem: str(path.stat().st_mtime_ns) for path in _posts()}

    keep_rows, keep_meta = [], []
    for i, row in enumerate(cached_meta):
        slug, mtime = row[0], row[1]
        if live.get(slug) == mtime:      # unchanged file, reuse its vectors
            keep_rows.append(cached_vecs[i])
            keep_meta.append(row)

    done = {row[0] for row in keep_meta}
    todo_texts, todo_meta = [], []
    for path in _posts():
        if path.stem in done:
            continue
        title = _title(path.read_text(encoding="utf-8"), path.stem)
        for heading, text in _chunks(path):
            todo_texts.append(text)
            todo_meta.append((path.stem, live[path.stem], heading, title))

    if not todo_texts:
        vecs = np.array(keep_rows)
        if len(keep_meta) != len(cached_meta):   # a post was deleted, persist the prune
            np.savez(INDEX, vecs=vecs, meta=np.array(keep_meta, dtype=object), model=MODEL)
        return vecs, keep_meta

    print(f"embedding {len(todo_texts)} chunks", file=sys.stderr)
    new_vecs = _embed(todo_texts, "search_document")   # one batched call
    vecs = np.vstack([np.array(keep_rows), new_vecs]) if keep_rows else new_vecs
    meta = keep_meta + todo_meta
    np.savez(INDEX, vecs=vecs, meta=np.array(meta, dtype=object), model=MODEL)
    return vecs, meta


_INDEX_CACHE: tuple | None = None


def _index():
    """Loaded on first search. Warm cache is instant; a cold one costs minutes."""
    global _INDEX_CACHE
    if _INDEX_CACHE is None:
        _INDEX_CACHE = _build_index()
    return _INDEX_CACHE


if __name__ == "__main__":
    # This replaces Part One's block. Build the index from the shell before
    # starting the server:
    #     uv run --directory /abs/path server.py --reindex
    if "--reindex" in sys.argv:
        vecs, meta = _build_index()
        print(f"indexed {len(meta)} chunks", file=sys.stderr)
        raise SystemExit

    print(f"serving {len(_posts())} posts from {NOTES}", file=sys.stderr)
    mcp.run(transport="stdio")

Four things in there matter. The model name goes into the cache file, so swapping models throws the whole index out instead of comparing 768-dimensional rows against a 1024-dimensional query. st_mtime_ns is the freshness check, so editing one post re-embeds one post and leaves the other fifty alone, which turns a four minute rebuild into a two second one. Deleting a post is the quiet case: its slug never makes it into keep_meta, and the save on the early return writes the pruned index back to disk, because otherwise the dead vectors sit in index.npz forever and every run prunes them again. And the --reindex path means the expensive build happens when you ask for it, not in the middle of somebody's question.

Search itself is four lines:

def _similar(query: str, k: int = 5) -> list[tuple[float, str, str, str]]:
    vecs, meta = _index()
    q = _embed([query], "search_query")[0]
    scores = vecs @ q                      # both sides normalized, so this is cosine
    top = np.argsort(-scores)[:k]
    # meta rows are (slug, mtime, heading, title)
    return [(float(scores[i]), meta[i][0], meta[i][2], meta[i][3]) for i in top]

That vecs @ q is the whole search engine. One matrix multiply against every chunk on the blog.

6. What it actually retrieves

Here is where I found out my plan for this post was wrong.

The whole setup in Stage Two was that brute force should find the Fail2ban post, and that meaning-based search is the thing that gets you there. So I built the index, 496 chunks across 51 posts, and asked it:

query: "brute force"
  0.6237  ufw-firewall-basics-ubuntu        ## Rate limiting the front door
  0.6048  hardening-ubuntu-desktop          ## First, know what you are defending against
  0.5961  hardening-ubuntu-26-04-server     ## See what is actually listening
  0.5951  dev-handoff-claude-code-skill     ## How it works, section by section
  0.5950  hardening-ubuntu-26-04-desktop    ## Per-app permission prompts

The Fail2ban post is not in the top five. It is eighth. Fourth place is a post about handing work to AI sub-agents, which has nothing to do with any of this.

That is a better result than the substring match, which returned nothing at all, and the top hit is defensible: the UFW post's rate limiting section really is about throttling repeated connection attempts. But it is not the answer I promised, and the spread across those five is 0.03, which is another way of saying the index has no strong opinion.

Now ask the same thing as a sentence:

query: "how do I stop repeated failed SSH login attempts"
  0.7546  set-up-fail2ban-ubuntu-26-04      ## intro
  0.7425  set-up-ssh-keys-ubuntu            ## 5. Turn off password login
  0.7232  hardening-ubuntu-26-04-server     ## Order matters: do not lock yourself out
  0.7185  set-up-fail2ban-ubuntu-26-04      ## Before you install it
  0.7168  create-sudo-user-ubuntu-26-04     ## Now lock down root

First place, and second place is the post it tells you to read first. Phrase it more vaguely and it still holds: "something keeps trying to log into my server over and over" also puts the Fail2ban intro on top.

Two words is not enough to embed. That is the lesson, and I would not have believed it without the numbers. An embedding is a summary of meaning, and "brute force" on its own carries almost none: no target, no symptom, no context. The model has to guess whether you mean SSH, or a search algorithm, or a way of solving a puzzle. A sentence gives it something to summarize.

Which is fine, and it is worth understanding why. The caller is not a human typing two words into a box. It is a model turning somebody's question into a tool call, and models write sentences. The query shape this search is worst at is the one it will rarely be handed. But it does mean the docstring should tell it so, and mine now does.

7. Keep both

I expected this section to be about embeddings falling over on exact strings. It is not, because they do not. Ask the index for _ctypes and the top five hits are all sections of the right post, starting at 0.7004. That is the single cleanest result in this whole experiment, and it is the opposite of what I sat down to write.

So the case for keeping both is not that one is bad at literals. It is that they fail in different places, and you now have both failures on record. The substring match returned nothing for brute force over a hyphen. The embedding index returned nothing useful for the same two words because two words carry no meaning to summarize. Neither of those is a flaw you can fix from inside the other approach.

The other reason is cheaper and more practical. When I paste an error string into my own notes, I want to know that a post literally contains it, not that a post is thematically nearby. Exact match is a different kind of answer, not a worse one, and it costs one function call to keep.

So the answer is both, in one tool:

@mcp.tool()
def search_notes(query: str) -> str:
    """Search my blog posts.

    Runs two searches and merges them: an exact-phrase search ranked by where
    the phrase appears, and a meaning-based search that finds related posts
    even when they never use your words. Good for both error strings and
    plain-English questions. Returns up to 5 posts, best first, with the
    section that matched. Use get_note with a slug to read a full post.
    """
    exact = _keyword_hits(query)
    # Ask for more chunks than we need: several may come from one post and
    # collapse into a single line below.
    related = _similar(query, k=15)

    seen, out = set(), []
    for slug, title in exact:                    # literal matches win the top
        if slug not in seen:
            seen.add(slug)
            out.append(f"{slug}: {title}")
    for score, slug, heading, title in related:  # semantic fills the rest
        if slug not in seen and score > FLOOR:
            seen.add(slug)
            out.append(f"{slug}: {title}  (matched section: {heading})")

    return "\n".join(out[:5]) or f"Nothing found for {query!r}."

The merge rule is deliberately blunt: anything the keyword search found literally goes first, because if you typed an exact string you almost certainly meant it, and semantic hits fill whatever is left. Chunk hits collapse to one line per post, keeping the best-scoring section as the label, since the model wants a slug it can pass to get_note and not four near-duplicate rows from the same file.

FLOOR exists so a query with no real answer returns nothing instead of the five least-unrelated posts on the blog, which is the semantic version of the bug in Stage One. I set it to 0.5 by guessing, and the numbers say that was useless: on this corpus every hit that came back for every query I tried scored above 0.59, including the sub-agent post that has nothing to do with brute force attacks. A floor of 0.5 filters nothing at all.

Then I went looking for a number that would work, and did not find one. Real scores here sit between 0.59 and 0.76, and across five queries neither obvious rule separates the good answers from the bad ones:

Query	Top score	Gap to second	Right answer?
`brute force`	0.6237	0.0189	no
`how do I stop repeated failed SSH login attempts`	0.7546	0.0121	yes
`something keeps trying to log into my server over and over`	0.6540	0.0009	yes
`_ctypes`	0.7004	0.0443	yes
`my server forgets its IP address after reboot`	0.6985	0.0080	no

A floor of 0.70 would have thrown away a correct answer at 0.6540. And the gap between first and second place, which I assumed would be the tell, does not sort them either. The widest gap on that list is a success and the narrowest gap is also a success, a factor of fifty apart, with both of the failures sitting comfortably in between. Both of the tidy heuristics I expected to find are wrong on my own data.

So FLOOR stays at 0.5 as insurance, not as a demonstrated filter. It never fired on any query I tried, and I am keeping it for the query that is genuinely about nothing on this blog, which none of my test queries were. The honest position is that I do not have a reliable confidence signal for this yet, and anyone who tells you a fixed cosine threshold means "relevant" has not printed their own numbers.

One tool, not two. It is tempting to ship search_notes and search_notes_semantic and let the model choose. Do not. The model would have to know which search strategy suits a query it has not run yet, which is a harder problem than the one you are solving, and it will get it wrong in the direction of whichever docstring reads more confidently. Every tool you add is another thing it can pick wrong. Give it one door and do the routing yourself.

8. Ask it the way a person would

Testing the function at a terminal is not the point. The caller is a model, so restart Claude Code and ask it the way somebody would actually ask.

how do I stop people repeatedly trying to log into my server?

mcp-server-search-ranking-embeddings-ollama: Your MCP server's search is bad: ranking, embeddings, and what each one fixes
set-up-fail2ban-ubuntu-26-04: Set Up Fail2ban on Ubuntu 26.04 Server  (matched section: intro)
hardening-ubuntu-26-04-server: Hardening Ubuntu 26.04 Server (Resolute Raccoon)  (matched section: Order matters: do not lock yourself out)
create-sudo-user-ubuntu-26-04: Create a Sudo User on Ubuntu 26.04  (matched section: Now lock down root)
set-up-ssh-keys-ubuntu: Set up SSH keys for Ubuntu  (matched section: 5. Turn off password login)

The Fail2ban post is there in second place, matched on its intro, which is the section that never says "brute" or "attack." That is the result section 6 promised and the whole point of the exercise.

Now look at what is sitting on top of it.

That is this post. And look at what the top line is missing: every other row carries a matched section, because every other row came out of the embedding index. The top row does not, which means it came back from the exact-phrase half.

When I ran this, the sentence I had just typed existed in exactly one file on the blog. It was in this section, inside an HTML comment, in a note to myself that said to ask something like "how do I stop people repeatedly trying to log into my server?" I wrote the question down before I wrote the section. The literal search found my own unfinished to-do list and ranked it above the answer.

Part One opened with the search returning Part One. Two stages and a rewrite later, it opens with the search returning a comment I had not finished writing.

There is a real lesson under the joke, and it is about the merge rule in Stage One. Exact matches go first because if you typed a literal string you almost certainly meant it. That reasoning holds for _ctypes. It does not hold for a twelve word question, because a twelve word question matching a document verbatim does not mean the document answers it. It means the document quotes it. Length should have been part of that decision and it is not.

The contamination is not only literal, either. Ask something with no phrase in common with anything I have written:

someone keeps guessing passwords on my ssh server and filling the logs

hardening-ubuntu-26-04-server: Hardening Ubuntu 26.04 Server (Resolute Raccoon)  (matched section: Order matters: do not lock yourself out)
set-up-ssh-keys-ubuntu: Set up SSH keys for Ubuntu  (matched section: Gotchas I hit)
ssh-config-file-explained: The ~/.ssh/config file explained: aliases, jump hosts, and a key per host  (matched section: A key per host)
mcp-server-search-ranking-embeddings-ollama: Your MCP server's search is bad: ranking, embeddings, and what each one fixes  (matched section: 2. The query ranking cannot touch)
set-up-fail2ban-ubuntu-26-04: Set Up Fail2ban on Ubuntu 26.04 Server  (matched section: intro)

Fail2ban falls to fifth, and this post is fourth on the strength of section 2, which is three hundred words about failed SSH logins that solve nothing. Every measurement in section 6 was taken on an index of 51 posts. There are 52 now, and the new one is a long post about searching for the answer rather than a post containing it.

I do not have a clean fix for that and I am not going to invent one. A post about retrieval is a genuinely hard document to keep in a retrieval index, because it is made of other posts' vocabulary. What I have is the knowledge that my corpus now contains a decoy, which is worth more than a confidence score.

9. Gotchas I hit

Part One's worst bug was a print() that worked fine until a buffer filled weeks later, and the first two here have the same shape: no crash, no error, just an index that is quietly worse than you think.

I built the first index without the task prefixes. Nothing told me. Vectors came back, the scores looked reasonable, the results were ranked, and I nearly shipped it. It took building the whole index a second time and scoring both against eight questions to see that the first one was measurably worse. There is no error to chase here, only the version of the index you did not build.
I reached for num_ctx before I tested whether it did anything. Ollama serves this model at 2,048 tokens, drops the rest without a word, and clamps a larger num_ctx back down to the trained context. Two documents differing only after the cut embed to cosine 1.000000, which is how I found out. Measure your longest chunk instead of trusting the flag.
A four minute index build in the wrong place. 496 chunks took 239 seconds on CPU. Put that on the first search, the way I first designed it, and the model waits four minutes for a tool call. Build ahead of time and let the server load a file.
A model swap poisons a cache keyed only on mtime. No file changed, so nothing re-embeds, and now a 768-dimensional index meets a 1024-dimensional query. That one at least fails loudly, though in numpy's terms rather than yours:

  ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0,
  with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 1024 is different from 768)

Nothing in there says "you changed embedding models." Put the model name in the cache file and check it on load, which is what _load_cache() above is for.

The two halves of the merged tool can disagree about the corpus. _INDEX_CACHE loads once per process, and Part One's rule that config is read at session start applies to it too. Edit a post mid-session and the keyword half sees the change on the next call, because it reads the files live every time, while the embedding half keeps serving the old vectors until the server restarts. One tool, two answers about what the blog says.
Ollama being down fails at search time, not startup. The index builds lazily, so claude mcp list reports ✔ Connected, the tools list normally, and the failure waits until someone searches. The try in _embed covers both paths that need it, the index build and the query, because _similar embeds the query on every single search whether the index is warm or not. The daemon being down surfaces as a plain ConnectionError, and the reason goes to stderr, which Part One established is the only channel that will not corrupt the transport.
The post you are reading poisoned its own index. Section 8 has the whole story. A twelve word question matched this file literally, because I had written that question into a to-do comment before I had written the section under it, and the merge rule puts literal matches first. The lesson is not about comments. It is that the corpus is not a fixed thing you index once and reason about afterwards. It contains whatever you are currently writing, including the parts that are not finished, and a post about search is made almost entirely of other posts' vocabulary.
The docstring changed three times in two posts. No ranking, then ranking, then two searches at once. Every time it went stale the model kept working around a limitation that no longer existed. It is still the only description of this tool it will ever see.

10. Quick reference

Task	Command / code
Add the deps	`uv add ollama numpy`
Pull the model	`ollama pull nomic-embed-text`
Embed a batch	`ollama.embed(model=MODEL, input=[...])`
Keep chunks under	2,048 tokens, where Ollama truncates this model
Check that limit yourself	`ollama show nomic-embed-text`
Prefix documents	`search_document: {text}`
Prefix queries	`search_query: {text}`
Build the index	`uv run --directory /abs/path server.py --reindex`
Normalize	`v / np.linalg.norm(v, axis=1, keepdims=True)`
Search	`vecs @ q` then `np.argsort(-scores)[:k]`
Cache key	path + `st_mtime_ns` + model name
Force a rebuild	`rm index.npz`
Inspect the index	`np.load("index.npz", allow_pickle=True)["meta"]`
Reload the server	restart Claude Code, config is read at session start
Pre-allow headless	`claude -p "..." --allowedTools "mcp__notes__search_notes"`

Part One's closing line was that the protocol is the easy part and what you put behind the door is your problem. Two stages later that still holds. The ranking fix is twenty lines and should have shipped in Part One. The embeddings are a 768-float vector per section and a matrix multiply.

Neither is the interesting part. The interesting part is that I wrote a whole post about stopping brute force attacks and never once typed the words. Any search I put in front of these notes has to survive that, because the person asking will not know what I called it.

[ two searches, one door ]

Originally published at peculiarengineer.com.

Reverse Proxy Your Containers with Caddy and Docker Compose on Ubuntu 26.04

Minor Keith — Fri, 24 Jul 2026 02:15:03 +0000

Once you have Docker running on 26.04 and a couple of containers up, you hit the wall everyone hits: your apps live on a pile of random ports. Plex on :32400, something else on :8080, a dashboard on :9000, and none of them speak HTTPS. You want plex.example.com and books.example.com to just work, with a real certificate, without hand-rolling Nginx configs or running certbot on a cron job.

Caddy is the boring, dependable answer to that, and it is short. A hostname in a Caddyfile is all it takes for Caddy to go get a Let's Encrypt certificate, serve it on 443, and renew it forever without you thinking about it again. This is the setup I use to front my own containers, and this post is where I keep the parts that are not obvious the first time.

TL;DR Run Caddy in the same Compose project as your apps, put every container on one shared Docker network, and in the Caddyfile reverse_proxy to the container name and its internal port, never localhost. Caddy gets HTTPS automatically as long as your domain's DNS points at the box and ports 80 and 443 are reachable. While testing, set acme_ca to the Let's Encrypt staging endpoint so a broken config does not burn your weekly certificate quota.

Prerequisites

Ubuntu 26.04 with Docker and the Compose plugin. If you are not there yet, start with Install Docker on Ubuntu 26.04.
A domain you control, with an A record (and AAAA if you have IPv6) pointing at the box's public IP.
Ports 80 and 443 reachable from the internet: open in UFW and forwarded at your router if you are behind NAT.

The mental model

Two ideas carry this whole setup, and if you hold them straight nothing else is confusing.

First, Caddy and your apps have to share a Docker network so Caddy can reach them by name. Containers on the same user-defined network get automatic DNS: a service called whoami is reachable at the hostname whoami from any other container on that network. Caddy proxies to those names. Your apps then do not need to publish any ports to the host at all, which is the point. Only Caddy is exposed, everything behind it is private.

Second, automatic HTTPS is triggered by using a real hostname, not a port. When the Caddyfile says books.example.com { ... }, Caddy registers an ACME account, proves it controls that name over port 80, installs the certificate, and serves 443. When it says :8080 { ... } it does not, because there is nothing to get a certificate for. The hostname is the switch.

1. The Compose file

Here is a complete, working stack: Caddy plus one example backend (whoami, a tiny container that prints request info, perfect for confirming the proxy works before you point it at anything real).

services:
  caddy:
    image: caddy:2.9
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"        # HTTP/3
    volumes:
      - /opt/dockerapp/caddy:/etc/caddy   # mount the dir, not the file
      - caddy_data:/data                  # certificates live here, keep it
      - caddy_config:/config
    networks:
      - web

  whoami:
    image: traefik/whoami
    container_name: whoami
    restart: unless-stopped
    networks:
      - web
    # note: no `ports:` at all. Only Caddy is exposed.

networks:
  web:

volumes:
  caddy_data:
  caddy_config:

Two lines earn their comments. caddy_data is a named volume holding your certificates and ACME account key; if you delete it you throw away real Let's Encrypt certificates and have to fetch them again, which matters because of the rate limit below. And the config mount is the directory /opt/dockerapp/caddy, not the single Caddyfile. Mounting a lone file bites you when an editor replaces it on save: the container keeps pointing at the old inode and your edits appear to do nothing. Mount the folder and that problem disappears.

Create the config directory before you bring the stack up:

sudo mkdir -p /opt/dockerapp/caddy

2. The Caddyfile

Drop this at /opt/dockerapp/caddy/Caddyfile:

{
      email you@example.com
}

whoami.example.com {
      reverse_proxy whoami:80
}

That is the entire config for one site. The global block at the top sets the email Let's Encrypt uses for expiry notices. The site block says "for this hostname, hand the request to the whoami container on port 80." Bring it up:

cd /opt/dockerapp/caddy
docker compose up -d

Point whoami.example.com at the box, wait for DNS, and load it over HTTPS. You should see the whoami output with a valid padlock and no certificate warning. Adding a second app is another three lines:

books.example.com {
      reverse_proxy calibre-web:8083
}

The port you write is the container's internal port, the one the app listens on inside its own container, not any host port you might have published elsewhere. Caddy is talking to it over the Docker network, so the host port mapping is irrelevant here.

3. The shared network is the whole trick

This is the part that sends people to the search bar, so it gets its own section. If Caddy and the backend are not on the same Docker network, reverse_proxy whoami:80 fails with a DNS error in the Caddy logs:

dial tcp: lookup whoami on 127.0.0.11:53: no such host

That message means exactly what it says: Caddy asked Docker's internal DNS for whoami and got nothing, because from Caddy's network that name does not exist. The fix is always the same: put both services on the same network. In the Compose file above they share web, so it works.

The trap shows up when your apps live in a different Compose file (a common way to organize a homelab, one project per app). Compose names each project's default network after the project, and those networks are isolated. To let Caddy reach a container in another project, declare a shared external network and attach both sides to it. Create it once:

docker network create web

Then in every Compose file, Caddy's and each app's, mark that network external:

networks:
  web:
    external: true

and add - web to each service's networks: list. Now they all share one bridge and Caddy can resolve every container by name across projects.

One real exception worth knowing: a container running with network_mode: host is not on any bridge network and has no Docker DNS name. My Plex container uses host networking because it needs it for discovery, so Caddy cannot reach it as plex. For those, proxy to the host's LAN address and the published port instead:

plex.example.com {
      reverse_proxy 192.168.1.10:32400
}

4. How automatic HTTPS actually happens

Caddy's certificate magic is not magic, and knowing the steps tells you exactly what to fix when it fails. On first request for a hostname, Caddy:

Registers an ACME account with Let's Encrypt (once, stored in caddy_data).
Requests a certificate for the hostname.
Proves control by answering an HTTP-01 challenge on port 80, or a TLS-ALPN challenge on 443.
Installs the certificate and serves HTTPS on 443, redirecting HTTP to HTTPS.
Renews automatically around 30 days before expiry.

So the requirements are concrete: the DNS record must resolve to this box, and port 80 must be reachable from the public internet for the challenge. If 80 is closed at the firewall or not forwarded through your router, the challenge fails and you get no certificate, no matter how correct the Caddyfile is. Check the logs with docker compose logs caddy and you will see the ACME error spelled out.

If the box is LAN-only and has no public DNS, public certificates are not an option, because Let's Encrypt cannot reach it to run the challenge. Two ways out: use Caddy's built-in local CA with tls internal (you then trust its root on your devices), or use the DNS-01 challenge, which proves control by writing a TXT record instead of answering on port 80. DNS-01 needs a Caddy image built with your provider's DNS module, so it is a custom build, not the stock caddy:2.9. For a normal internet-facing box, none of that applies and the default just works.

5. Reloading without downtime

After editing the Caddyfile, you do not need to restart the container and drop connections. Caddy reloads its config in place:

docker exec -w /etc/caddy caddy caddy reload

The -w /etc/caddy runs the command from the config directory so Caddy finds the Caddyfile without a --config flag. If the new config has a syntax error, the reload is rejected and the old config keeps serving, so a typo does not take your sites down. Validate a change before reloading if you want the check without applying it:

docker exec -w /etc/caddy caddy caddy validate

Gotchas I hit

The rate limit that burns a real certificate. Let's Encrypt allows 5 certificates per exact hostname per week. Fight a broken config with the production endpoint and you can exhaust that quota fast, then you are locked out of new certificates for that name for days. While you are still getting things working, point Caddy at the staging CA, which issues certificates your browser will not trust but that never run out:

{
      email you@example.com
      acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

Your browser will warn about the staging certificate, that is expected. Once the setup is solid, delete that acme_ca line, reload, and Caddy fetches a real one.

Editing the mounted single file did nothing. This is the inode trap from earlier. I mounted ./Caddyfile:/etc/caddy/Caddyfile directly, edited it with an editor that writes a new file and renames it over the old one, and the container went on serving the original because it still held the old inode. Mounting the directory instead of the file fixes it for good.

Port 80 closed after I set up the firewall. I locked the box down with UFW, opened 443, and forgot 80. Everything looked fine until the certificate came up for renewal weeks later and silently failed the HTTP-01 challenge. Open both:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

Port 80 is not just for a redirect, Caddy needs it for the ACME challenge on renewal, so leaving it closed is a time bomb set for 60 days out.

Quick reference

Job	Command / config
Proxy to a container	`reverse_proxy servicename:INTERNALPORT`
Proxy to a host-networked app	`reverse_proxy 192.168.1.10:32400`
Shared network across projects	`docker network create web` + `external: true`
Reload config	`docker exec -w /etc/caddy caddy caddy reload`
Validate config	`docker exec -w /etc/caddy caddy caddy validate`
Staging CA while testing	`acme_ca https://acme-staging-v02.api.letsencrypt.org/directory`
Open the ACME ports	`sudo ufw allow 80/tcp && sudo ufw allow 443/tcp`
Read the logs	`docker compose logs -f caddy`

Put every container on one shared network, proxy to names and not localhost, keep port 80 open for the challenge, and test against staging so a bad afternoon does not cost you a week of certificates. Do that and Caddy turns a pile of random ports into clean hostnames with real HTTPS that renews itself while you forget it exists. That last part, forgetting it exists, is the whole reason to use it.

[ 443 up · certs on autopilot ]

Originally published at peculiarengineer.com.

Install Docker on Ubuntu 26.04 (the right way, with the docker-group truth)

Minor Keith — Wed, 22 Jul 2026 21:24:32 +0000

The wrong way to install Docker on Ubuntu is the one that looks easiest: sudo apt install docker.io. That package exists, it installs, and it runs a container. It is also whatever version happened to be frozen into the archive when 26.04 was cut, it lags the real releases by months, and it ships without the Compose and Buildx plugins you will want by the end of the week. Use Docker's own apt repository instead, and this is the post I keep open so I do not re-derive the repo setup from memory each time.

This is short on purpose. The steps are the official ones, and the only place worth slowing down is step 5, where adding yourself to the docker group quietly hands out root. That tradeoff is the part most guides skip, and it is the one thing here actually worth reading twice.

TL;DR Remove any distro docker.io/containerd packages, add Docker's GPG key and the deb822 .sources repo, then sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin. Verify with sudo docker run hello-world. Add yourself to the docker group to drop the sudo (it is root-equivalent, more below).

Prerequisites

Ubuntu 26.04 (Resolute Raccoon), server or desktop, on amd64 or arm64.
A user with sudo. If you are still on root, create a sudo user first.
Outbound HTTPS to download.docker.com.

1. Remove the distro Docker packages first

Ubuntu ships its own docker.io, docker-compose, and containerd packages, and any of them will fight the official ones over the same files and the same containerd socket. Clear them out before you add Docker's repo. This is safe on a fresh box because there is nothing to lose yet; on a box that already ran the distro Docker, it removes the packages but leaves your images and volumes in /var/lib/docker alone.

sudo apt remove $(dpkg --get-selections docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc | cut -f1)

The dpkg --get-selections wrapper is just so the command does not error out on packages you never had installed. If none of them are present, nothing happens, which is exactly what you want.

2. Add Docker's apt repository

Two steps: trust Docker's signing key, then point apt at their repo. Docker's current docs use the newer deb822 format (a .sources file), which is more readable than the old one-line .list entry and is what 26.04's apt prefers.

First, the key:

sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

The key lives in /etc/apt/keyrings/, not the deprecated apt-key store, and the repo file below points at it with Signed-By. That pairing is what tells apt "only trust packages from this repo if they are signed by this specific key," which is the whole reason you are not just piping a script into your shell.

Now the repo itself:

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

The Suites: line reads your release codename out of /etc/os-release, so on 26.04 it resolves to resolute and on 24.04 it would be noble. That final apt update pulls in Docker's package list, and you are ready to install.

3. Install the engine and plugins

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Five packages, and it is worth knowing what each one is rather than pasting them as a magic incantation:

docker-ce is the daemon, the thing that actually runs containers.
docker-ce-cli is the docker command you type.
containerd.io is the lower-level runtime the daemon drives.
docker-buildx-plugin is the modern builder, so docker build uses BuildKit.
docker-compose-plugin gives you docker compose as a subcommand.

That last one is the part people miss. Compose v2 is a plugin now, so the command is docker compose up with a space, not the old standalone docker-compose with a hyphen. If you have muscle memory for the hyphenated one, this is where it stops working, and installing this plugin is how you get the replacement.

The daemon starts and enables itself on install, so there is nothing to systemctl enable. Confirm it:

systemctl is-active docker    # -> active

4. Verify it actually runs

sudo docker run hello-world

This pulls a tiny image, runs it, and prints a paragraph confirming the daemon, the runtime, and the network path all work end to end. If you see "Hello from Docker!", the install is done. If it hangs on the pull, that is a network or DNS problem reaching Docker Hub, not a broken install.

5. Run Docker without sudo (and what that really costs)

Typing sudo before every docker command gets old fast. The fix is to add yourself to the docker group, which owns the daemon's socket:

sudo groupadd docker            # usually already exists; harmless if so
sudo usermod -aG docker $USER

Group membership is read at login, so it does not apply to your current shell. Log out and back in, or start a fresh session with:

newgrp docker

Then prove it without sudo:

docker run hello-world

Here is the honest caveat, because most guides drop you in the docker group and move on. Membership in the docker group is root-equivalent. The daemon runs as root, and anyone who can talk to its socket can mount the host filesystem into a container and walk straight out as root. There is no privilege boundary between "in the docker group" and "root," full stop. On your own laptop or a single-admin homelab box, that is a fine trade for convenience. On a shared server, do not hand out docker group membership as if it were a lesser permission, because it is not. If that trade bothers you, rootless mode runs the whole daemon as your user instead, at the cost of a few limitations around ports below 1024 and some networking.

Quick reference

Job	Command
Remove distro packages	`sudo apt remove $(dpkg --get-selections docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc \
Add GPG key	{% raw %}`sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc`
Install engine + plugins	`sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin`
Verify	`sudo docker run hello-world`
Drop the sudo	`sudo usermod -aG docker $USER` then re-login
Compose command	`docker compose up` (space, not `docker-compose`)
Check the daemon	`systemctl is-active docker`

Use Docker's repo, not the distro package, so you get real versions and the Compose and Buildx plugins. Clear out the old packages first, trust the key, and let apt pull the real thing. Then decide with open eyes whether the docker group is a trade you want, because it hands out root. With that done, the box is ready for the actual reason you installed Docker: running something. A natural next step is putting Caddy in front of those containers for automatic HTTPS.

Originally posted on Peculiar Engineer.

Build your first MCP server in Python: give Claude your own notes

Minor Keith — Wed, 15 Jul 2026 02:40:18 +0000

Build your first MCP server in Python: give Claude your own notes

A beginner guide to the Model Context Protocol. Build a real MCP server in about sixty lines of Python with uv and the official SDK, expose two tools over stdio, and wire it into Claude Code with one command. Includes the stdout footgun that hides behind block buffering and only bites you later.

peculiarengineer.com

There are 48 posts on this site. I wrote every one of them so I would never have to Google the same thing twice, and it works: when I need the Netplan syntax or the exact ufw incantation, I come here instead of wading through Stack Overflow.

The annoying part is that the model I am talking to has not read any of it. It knows the general shape of Netplan, not the version I settled on after the third time it bit me. I can paste a post into the chat window, and I have, plenty of times. But pasting is not a system. It is a thing you do again every session, forever, and it only works when you already know which post you needed.

What I actually want is a door. Let the model knock when it wants something, and let it read the notes itself.

That door is the Model Context Protocol, and the useful part is not the protocol. It is that you write the integration once. Without MCP, giving a model access to your notes means a bespoke integration per tool per client: one for Claude Code, another for the desktop app, another for whatever you use next year. With MCP you write one server that knows how to search your notes, and every client that speaks the protocol can call it. The server does not know or care who is asking.

This post builds that server. It is about sixty lines of Python.

TL;DR uv add "mcp[cli]", decorate two functions with @mcp.tool(), end the file with mcp.run(transport="stdio"), and register it with claude mcp add notes -- uv run --directory /abs/path server.py. Use an absolute path or it will not connect, and never print() to stdout, because stdout is the transport.

Before you start

You need uv and Claude Code. That is the whole list. There is no server to deploy, no Docker, no port to open. A local MCP server is just a program that Claude Code launches as a subprocess and talks to over stdin and stdout.

That last point is worth sitting with, because it is the thing people expect to be complicated and it is not. MCP defines two transports. Streamable HTTP is for remote servers that many clients connect to over the network, and it comes with the whole authentication story. stdio is for local servers, and it is a pipe. Your process, the client's process, JSON going back and forth. Nothing is listening on a port and nothing is exposed.

Everything below is stdio. It is the right place to start, and for a server that reads files on your own laptop it may be the right place to stay.

What you are building

Two tools:

search_notes(query) finds posts containing a phrase and returns their slugs and titles.
get_note(slug) returns one whole post.

That split matters more than it looks. Search returns a small list so the model can pick, then it fetches only what it wants. If search_notes returned full post bodies, a three word query would dump 40,000 words into the context window and the model would drown before it started.

I am pointing this at my blog because that is what I have. Point NOTES at any folder of markdown and it works identically. That is the whole idea.

Set up the project

mkdir mcp-notes && cd mcp-notes
uv init .
uv add "mcp[cli]"

mcp[cli] is the official Python SDK. The cli extra brings along the development tooling, which you will want later.

The server

Here is the whole thing, server.py:

"""An MCP server that lets Claude read my blog posts.

Two tools: search_notes finds posts containing a phrase, get_note returns one
whole post. Point NOTES at any folder of markdown and it works the same.
"""

import sys
from pathlib import Path

from mcp.server.fastmcp import FastMCP

NOTES = Path.home() / "projects/peculiarengineer/src/content/blog"

mcp = FastMCP("notes")


def _posts() -> list[Path]:
    return sorted(NOTES.glob("*.md")) + sorted(NOTES.glob("*.mdx"))


def _title(text: str, fallback: str) -> str:
    # Frontmatter title, e.g.  title: 'Set up SSH keys for Ubuntu'
    for line in text.splitlines()[:15]:
        if line.startswith("title:"):
            return line.removeprefix("title:").strip().strip("'\"")
    return fallback


@mcp.tool()
def search_notes(query: str) -> str:
    """Search my blog posts for a word or phrase.

    Matches the literal phrase anywhere in a post, including its frontmatter.
    Returns up to 5 hits with slug and title. There is no ranking: hits come
    back in filename order, so a post that merely mentions the phrase can crowd
    out the post that is about it. Search more than once with different wording.
    Use get_note with a slug to read the full post.
    """
    hits = []
    for path in _posts():
        text = path.read_text(encoding="utf-8")
        if query.lower() in text.lower():
            hits.append(f"{path.stem}: {_title(text, path.stem)}")
        if len(hits) == 5:
            break

    if not hits:
        return f"No posts mention {query!r}."
    return "\n".join(hits)


@mcp.tool()
def get_note(slug: str) -> str:
    """Return the full text of one blog post, by slug.

    The slug is the filename without its extension, as returned by search_notes.
    """
    for path in _posts():
        if path.stem == slug:
            return path.read_text(encoding="utf-8")
    return f"No post with slug {slug!r}. Use search_notes to find one."


if __name__ == "__main__":
    # stdout is the transport. A stray print() lands in the middle of the
    # JSON-RPC stream, and block buffering means it may not bite until the
    # buffer fills hours later. Diagnostics go to stderr, always.
    print(f"serving {len(_posts())} posts from {NOTES}", file=sys.stderr)
    mcp.run(transport="stdio")

That is a complete MCP server. Three parts are doing the work.

FastMCP("notes") is the server. The name is what shows up in clients.

@mcp.tool() is where the magic is, and it is worth understanding what it saves you. The protocol requires each tool to advertise a JSON Schema describing its inputs. FastMCP builds that schema from your type hints, and it takes the tool's description from your docstring. You write a normal Python function and the protocol paperwork is generated for you. This is why the post is short.

mcp.run(transport="stdio") starts the loop that reads JSON off stdin and writes JSON to stdout.

Notice the search is a case insensitive substring match. That is not a placeholder I am going to upgrade at the end. It is the point: the protocol is twenty minutes of work, and search quality is a different problem that would eat this entire post. More on where that falls over below.

Wire it into Claude Code

One command:

claude mcp add notes -- uv run --directory /Users/you/projects/mcp-notes server.py

The -- matters. Everything before it belongs to claude mcp add, and everything after it is the literal command Claude Code runs to start your server. Without the separator, claude tries to parse your command's flags as its own.

The --directory matters more, and this is the first thing that got me. Claude Code runs that command from whatever directory you launched claude in, not from where your server lives. I registered it with a bare uv run server.py, started Claude in my blog repo, and got this:

notes: uv run server.py - ✘ Failed to connect

Of course it failed. There is no server.py in my blog repo. uv run --directory /abs/path server.py pins it, and now it does not matter where you start Claude from.

Check it:

claude mcp list

notes: uv run --directory /Users/you/projects/mcp-notes server.py - ✔ Connected

✔ Connected means Claude Code launched the process, completed the handshake, and got a tool list back. For more detail, claude mcp get notes shows the scope, the exact command, and any error.

A word on scope

By default your server is added with local scope, which means it is private to you and tied to the directory you added it from. This confused me for a solid minute: I added the server while sitting in my blog repo, went over to the server's own directory, ran claude mcp list, and it was not there. Not broken, not an error, just not listed. Local scope is per project, and I was in a different project.

Your options:

--scope local (the default) for this project only, stored in ~/.claude.json.
--scope user for every project you work in. This is what you want for a notes server.
--scope project writes a .mcp.json in the repo so anyone who clones it gets the server too.

Use it

Start Claude Code and ask it something that is in your notes and not in its training data:

> What did I decide about Secure Boot when installing NVIDIA drivers?

The first time a tool runs, Claude Code asks your permission, once per tool. Approve it and you will not be asked again. The names are namespaced by server, so mine show up as mcp__notes__search_notes and mcp__notes__get_note. /mcp inside a session lists the server and its tools.

Then it works. It calls search_notes("Secure Boot"), gets a slug back, calls get_note on it, and answers out of my own post: if Secure Boot is on, enroll the MOK key when the driver install prompts you, or turn Secure Boot off in the BIOS before you start.

That is the correct answer, and it is correct because it came from my own post rather than from a general impression of how NVIDIA drivers work.

But the part I did not expect was what came next. Unprompted, it searched again, found my Hardening Ubuntu 26.04 Desktop post, and pointed out that the two contradict each other: on the desktop I recommend TPM backed full disk encryption, which relies on Secure Boot being on. So "turn Secure Boot off" is advice scoped to a headless GPU box, not a rule, and I had never written that down anywhere because I had never had both posts in my head at the same time.

That is the moment the door earns its keep. I did not ask it to reconcile two posts written a month apart. It just had access to both.

Where the dumb search falls over

I promised this, and it is more interesting than I expected.

Ask for something specific and it is great. search_notes("fail2ban") returns exactly the right three posts. Then try a general word:

search_notes("firewall")

create-sudo-user-ubuntu-26-04: Create a Sudo User on Ubuntu 26.04
enable-ssh-on-ubuntu-desktop: Enable SSH on an Ubuntu desktop
extend-azure-windows-disk-run-command: Extending C: on locked-down Azure Windows VMs
hardening-ubuntu-26-04-desktop: Hardening Ubuntu 26.04 Desktop (Resolute Raccoon)
hardening-ubuntu-26-04-server: Hardening Ubuntu 26.04 Server (Resolute Raccoon)

Every one of those is a real match. And the actual firewall post, UFW Firewall Basics, is not in the list.

The substring match did not fail. It found the UFW post fine. The problem is that there is no ranking at all: _posts() returns files in alphabetical order and I stop at five, so five posts that mention "firewall" once in passing crowded out the post that is entirely about firewalls, purely because u sorts after c, e, and h.

This is the real lesson, and it is why I did not paper over it with a better search. MCP got your notes to the model in twenty minutes. Deciding which notes are the right ones is the actual work, and it is the same search problem it always was. The protocol does not help you with it and was never going to.

There is a cheap fix that costs nothing, though, and it is the most MCP-shaped part of this whole post: I told the model about the limitation. Read the docstring on search_notes again. It says there is no ranking, and it says to search more than once with different wording. The model reads that and compensates by trying ufw after firewall comes back weak. You cannot fix bad search with a comment, but you can stop the model from trusting it.

Gotchas I hit

Never print() in a stdio server, and do not trust your own testing on this one. stdout is the transport. Anything you print goes into the JSON-RPC stream and the client fails to parse it. The catch is that it often looks fine: when stdout is a pipe rather than a terminal, Python block buffers it at around 8 KB, so your debug line sits in the buffer and never interleaves. Add flush=True, or print enough to fill the buffer, or just run long enough, and it corrupts. Mine failed exactly as advertised the moment I flushed: Failed to parse JSONRPC message from server ... input_value='DEBUG: searching for fail2ban'. A footgun that works in testing and detonates in week three is worse than one that fails immediately. Send diagnostics to stderr with file=sys.stderr, which the client ignores and you can still read.
Relative paths fail. The command runs from wherever you started claude, not where the server lives. Use uv run --directory /abs/path server.py.
Local scope is tied to a directory. Added from one project, invisible from another, with no error to explain it. Use --scope user for anything you want everywhere.
The docstring is your API contract. It is not a comment. It is the only description of your tool the model ever sees, and it decides how the tool gets called. Mine said "newest matches first" when the order was actually alphabetical, and no compiler was ever going to catch that. A stale docstring is a lying API spec.
✔ Connected does not mean the model can call it. Connected means the handshake worked. Each tool still needs your approval the first time it runs. This bites hardest in a headless session (claude -p ...), where there is nobody to approve anything and the call is simply refused. Pre-allow them with --allowedTools "mcp__notes__search_notes,mcp__notes__get_note".
Config is read at session start. Edit the server and the running session keeps the old one. Restart Claude Code.
Test the command standalone first. If uv run --directory /abs/path server.py does not start on its own, it will not start under Claude Code either, and the error is much easier to read in your terminal.

Quick reference

Task	Command
New project	`uv init . && uv add "mcp[cli]"`
Server object	`mcp = FastMCP("notes")`
Define a tool	`@mcp.tool()` above a typed function
Run over stdio	`mcp.run(transport="stdio")`
Log safely	`print(msg, file=sys.stderr)`
Register it	`claude mcp add notes -- uv run --directory /abs/path server.py`
Register everywhere	`claude mcp add --scope user notes -- ...`
List servers	`claude mcp list`
Inspect one	`claude mcp get notes`
Manage in session	`/mcp`
Pre-allow tools	`claude -p "..." --allowedTools "mcp__notes__search_notes"`
Remove it	`claude mcp remove notes`

Which LLM should I actually code with? I built a small benchmark to find out

Minor Keith — Sun, 12 Jul 2026 03:41:18 +0000

LLM code benchmark — Peculiar Engineer

A small, self-run coding benchmark: 3 models on 14 problems across 3 languages, scored on pass@k, cost, and speed. Last run 11 July 2026.

peculiarengineer.com

I kept going back and forth on which model to reach for in my actual day job. Every "which LLM is best at code" thread turns into vibes and screenshots, and none of it answered the question I had, which is which one to open when I have real work in the languages I use. So I stopped guessing and built a small benchmark to settle it for myself.

It is deliberately small. 14 problems across Python, C#, and Bash, three models, three attempts each at temperature 0.7 with a 10 second timeout. Every attempt runs in a sandboxed Docker container and gets scored on pass@k, cost, and latency. It is not an authoritative ranking and I am not pretending it splits hairs. It is enough to show the shape.

The thing that surprised me

Accuracy is not the differentiator anymore. All three models solved every problem they were allowed to answer, 100% pass@3. If I only looked at pass rates I would have learned nothing, because they all pass.

The catch hides in "allowed to answer." One model got content filtered out of four problems, so its perfect score covers 10 of the 14, not the whole set. A perfect score on the problems you answered and a perfect score on the whole bench are not the same result, which is why the leaderboard sorts on coverage first.

Where they actually differ

If they all pass, the decision comes down to what you pay and how long you wait. That is where the spread lives.

Cost was close, about 1.3x from cheapest to priciest across the suite.
Latency was not close. 6.6x between the fastest and the slowest.
The cheapest run that also covered all 14 problems came in around $0.028 per solved problem.

So the honest summary is boring in the best way. Pick on speed and price, because accuracy already agrees.

The gotcha worth writing down

One result is not like the others. One model was quick and cheap on Python and C#, a second or two per problem, then fell off a cliff on Bash at 90 to 168 seconds per problem. Four slow Bash problems dragged its average latency up to something that makes it look slow overall when it is really fine everywhere except Bash. If your day is mostly shell, that matters a lot. If it is Python, you would never notice.

That is why a single "which model is fastest" number is a lie. Fastest at what, in which language, is the only version of the question worth asking.

What I took away

For my work it came down to speed and cost per language. The benchmark did the one job I wanted. It turned a running argument in my head into a few numbers I can re-run whenever the models change.

Hardening a fresh Ubuntu 26.04 server without locking yourself out

Minor Keith — Sun, 12 Jul 2026 03:37:42 +0000

The hardening pass I run on a fresh Ubuntu 26.04 LTS box before it does anything useful. Keys only SSH, the firewall set before you cut the cord, unattended security updates with a reboot window, fail2ban, free kernel livepatch, and an honest look at what is actually listening.

Hardening Ubuntu 26.04 Server (Resolute Raccoon)

A practical hardening pass for a fresh Ubuntu 26.04 LTS server: SSH locked down to keys only, the firewall set before you cut yourself off, automatic security updates with a sane reboot window, fail2ban on the front door, kernel livepatch from the free Ubuntu Pro tier, and an honest look at what is actually listening.

peculiarengineer.com

I built Pong from scratch in LÖVE and Lua (a two part series)

Minor Keith — Wed, 01 Jul 2026 02:12:12 +0000

I wrote a two part series on building Pong from scratch in LÖVE and Lua. If you've never made a game before, I think it's a nice place to start.

Part one is the whole game in two short files: the loop, delta time,
collision, scoring, a title screen, and sound generated in code so there are no audio files to ship.

Part two adds a one player mode by writing a computer opponent.

Part 1: https://peculiarengineer.com/blog/make-pong-with-love2d-and-lua/
Part 2: https://peculiarengineer.com/blog/pong-computer-opponent-love2d-lua/