DEV Community: Christian Anderson

Homelab dashboard IP sync failing silently: when a miss looks like no change

Christian Anderson — Fri, 09 Oct 2026 08:00:13 +0000

My homelab dashboard has a tile for each service. Most of those services sit at fixed addresses. A few don't: my 3D printer gets its address from DHCP, and when it moves, the tile points at nothing.

So on 12 July I wrote a small job to keep the tiles right. It runs every 15 minutes, works out the current address of each tracked device, and rewrites the dashboard's config only if something changed. It tracks three devices: the printer, Home Assistant and Uptime Kuma.

For three and a half weeks it did nothing at all, and its log said so every quarter of an hour in a way I read as good news.

Built to be safe

I made two design choices on purpose, and both come back into this story.

First, it is a plain Python script with no language model anywhere in it. An agent that "works out" an IP address can invent one, and a wrong address written into a dashboard is worse than a stale one.

Second, it fails safe. If it can't find a device, it leaves that device's tile alone. It never writes a guess. A device that is switched off shouldn't get its tile blanked, and the next run can pick it up again.

That second choice is the one that hid the problem.

How it found devices

My router doesn't serve forward DNS for local .lan names, so asking "what is the printer's address?" by name never worked. Reverse lookups did: DHCP leases were mirrored into the resolver, so asking "what name belongs to this address?" gave an answer.

That resolver rate-limits reverse lookups hard. Sweeping all 254 addresses at 64 workers turned up about 13 names. Even 8 workers was flaky. So the job had two paths:

Confirm. Do a reverse lookup on each device's last-known address, three queries in all, and accept it if the name still matches.
Sweep. Only for a device that has moved: ping the whole subnet twice, then do reverse lookups on live hosts only, four at a time, and stop as soon as it's found.

It worked. The log from the first days shows no changes (3 resolved, 0 miss).

The log line I kept reading as fine

On 16 August I actually read the log. Run after run looked like this:

MISS: Voron 3D Printer (...) not on <subnet> — leaving tile untouched
MISS: Home Assistant (...) not on <subnet> — leaving tile untouched
MISS: Uptime Kuma (...) not on <subnet> — leaving tile untouched
no changes (0 resolved, 3 miss)

"No changes" was true. It was also useless. Since the evening of 23 July, almost every run had found none of the three devices. It couldn't see anything, so it changed nothing, so it reported nothing to change. Nothing alerted, because the only thing it ever notified on was a change.

The log has 4,422 lines reading no changes (0 resolved, 3 miss). That isn't 4,422 runs, though. It's about half that, because of a second bug I found in the same pass: every line was written twice. The script printed to stderr and appended to its log file, and cron redirected stderr into that same file. Even the count of the failure was wrong.

Why it went blind

Reverse DNS had stopped working entirely. .lan names came back NXDOMAIN from the resolver the job uses, and also when I asked the router directly. Reverse lookups of addresses I knew were good came back empty. The router was otherwise healthy: it answered pings with no loss and resolved normal forward DNS. Bare hostnames still resolved, but only to IPv6 addresses on my VPN overlay, never the LAN IPv4 the tiles needed.

I never found out why the router stopped answering reverse lookups. I stopped depending on it instead.

The fix: ask something that knows

The rule I took from it: don't infer an address from DNS when an authoritative source exists. Two such sources were sitting there already.

Containers ask Proxmox. Two of the three devices are LXC containers, and Proxmox knows their addresses. The job now reads them from the nodes over SSH. Only running containers with an address on the right subnet count. One container name exists on both nodes. That is logged as ambiguous and refused, not guessed.

Physical boxes are found by MAC. The printer runs on a Raspberry Pi. Its MAC address doesn't change when its IP does, which is exactly the drift this job exists for. It checks the ARP table first. If the MAC isn't there, it pings the last-known address to refresh the table. Only then does it do a full ping-sweep.

Reverse DNS is still there, but only as the last fallback.

The first run after the change: 3 resolved, 0 miss. Then the embarrassing part. I checked the live dashboard config, and all three tiles had been correct the whole time. None of the devices had moved. The sync had been blind, not wrong, so the dashboard never showed a symptom.

Two more fixes in the same pass

The double-logging: the script now writes to the file and only echoes to stderr when it's attached to a terminal. It also echoes if the file write fails, so output can't disappear silently.

And timing: 38.0 seconds of a 38.1-second run were spent asking Proxmox for container addresses. That was two SSH calls, each running a helper that called pct three times per container. It came to around 3,600 pct invocations a day for addresses that almost never change. They're cached for an hour now. But the hour isn't the main guard: if a cached address stops answering, the job refreshes from Proxmox straight away. I tested that by poisoning the cache with a dead address. It noticed, refreshed and recovered the right one. The run dropped from 38 seconds to 0.34.

Did it earn its keep?

Since then the printer has moved twice. On 7 September and again on 10 September it came back on a new DHCP address. Both times the job found it by MAC and rewrote the tile, backing up the config first. That is the job it was built for, done twice, after three and a half weeks of finding nothing while saying "no changes".

"No change" and "couldn't look" are different answers

The log still has a problem, and I'd rather say so. Most runs today end no changes (2 resolved, 1 miss). The miss is the printer: its MAC isn't on the network much of the time. The script does keep misses in a separate list in its state file. But it still notifies only on a change, so a device it can't find is still a log line nobody reads.

What I'd do next, and what I'd suggest for any job shaped like this:

Return three outcomes, not two. "Checked, unchanged", "checked, changed" and "couldn't check" are different results. Don't let the third collapse into the first.
Alert on consecutive misses, not on each one. One miss is a printer that's switched off. Every device missing on every run for a day is a broken job.
Put the resolved count where you'll see it. 0 resolved was in every line. I just never looked.

This wasn't a one-off, either. This month my blog's publish queue ran empty after 18 September. Both publishers logged "queue empty — nothing to publish" and exited 0, and nothing alerted. A healthy, idle publisher looks just like a healthy, busy one unless you check how deep the queue is. There is now a check that alerts when the queue is empty.

And on 7 September, a canary that watches my paid model provider switched four agent profiles back to local models after its probe failed on every model. It turned out to be the probe's input, not the models. It only knew how to switch back after a credit problem, not after a failover, so it logged "nothing to watch" for 17 days. It now re-probes after a failover and switches back on its own.

All three were healthy processes reporting a true, reassuring sentence about a state they hadn't actually checked.

AI blog drafts fact-checked: my drafter invented a bug, 8 of 9 needed fixes

Christian Anderson — Thu, 08 Oct 2026 08:00:14 +0000

On 18 September I published a post called AI blog writing pipeline without made-up facts. Its central claim was that the model only phrases facts a script has gathered, so "it never had the chance to fabricate one", and that the whole system "fails toward silence, never toward fabrication".

Six days later I fact-checked every draft sitting in the review folder against the code, the logs and my own notes. The claim did not hold. This is what I found, and what I have changed.

The draft that invented a bug

The drafter had written a post about my podcast pipeline, which turns an article into a two-voice episode using Piper text-to-speech over the Wyoming protocol. My log recorded it as the first draft to pass a new set of checks. Those checks were about format: title length, description length, tags. It passed them all.

The content was mostly fiction. The draft said:

the bug was endianness, fixed by flipping a > to a <;
the text-to-speech model had a "reasoning field" that caused trouble;
the fix took two weeks of debugging;
synthesis ran at 90% real-time.

It also contained, in the body text, a sentence that began "wait, no. Let me be precise." The model had corrected itself mid-draft and the correction had been published into the post.

None of those four things happened. The real bugs were two, and both are in the code:

The Wyoming frame has three parts, not two. A header line can declare a data_length, and the event's data then follows as its own block of bytes after the newline, not inline in the header. My first reader went header, then payload, landed in the middle of a JSON object, and failed with Extra data: line 1 column 62. That looks like a broken server. It was a field I hadn't read.
The model writing the script returned nothing usable. Not the speech model: the language model writing the dialogue. Its answer came back with content empty and the whole token budget inside a thinking key. The pipeline logged a cheerful "0 turns" rather than an error. The code now raises when content is empty and thinking is populated, and it switches thinking off in the request body.

The speed figure was invented too. What I actually measured was 8.4 seconds of audio in 2.9 seconds of wall time, a real-time factor of about 0.35.

I rewrote the post from the code and published it the same day as Piper TTS over Wyoming: the frame format and empty model that broke my podcast. The fabricated original is kept, marked as such, so I can compare the two.

The irony is worth stating plainly. The invented bug was more conventional than the real one. My guess is that endianness is what a model reaches for when it knows a post is about binary framing and doesn't know the details. The true story, a missed field in a three-part frame, is the one other people will hit.

It wasn't the first time

Looking back, the warnings were already in my notes:

16 September: the topic radar picked up a public repo that turned out to have no code in it, only a README. The drafter wrote a whole build story from the one-line description. It was convincing enough that it was read as a real project. I renamed the draft so it couldn't be queued and added a guard that skips any repo with no commits.
20 September: a fix that let malformed drafts survive (the model was wrapping its front matter in a code fence) also let through a draft that invented a brownout threshold and a soldering step, neither of which was in the facts it was given.
21 September: on several runs the model emitted its own tool-call syntax instead of a post, because it was trying to call a tool inside a text-only request.

Each time I fixed the symptom in front of me. None of those fixes made a draft true.

Nine drafts, checked line by line

On 24 September the queue was empty and there were nine drafts waiting. I checked each claim against the thing it described. The result:

1 had a false core. A post about an Ollama deadlock credited the fix to a configuration setting that, on its own, did not hold. What actually fixed it was a small guard script. The draft also said I had pinned the model myself, which I hadn't, and it wrapped the whole thing in a debugging story that never happened.
8 needed fixes, from heavy to small.
All 9 were missing a description in the front matter.

The errors were not random. They fell into a handful of types.

Invented process story. Drafts described alternatives I "tried" and rejected. One post about using Discord as the interface for my agents listed two other chat platforms I had supposedly evaluated. It also described a rate-limit error and a message-chunking fix that didn't exist, and said voice notes worked, which they don't. Models write the post they expect, and posts usually have a "what I tried first" section.

A confident wrong fix. The deadlock post above. This is the dangerous type, because a reader will copy the fix.

Stale state. Counts that were true once: the number of smart bulbs responding, the number of containers in the estate, the number of agent profiles. One draft still included a service I had decommissioned. Another described logging tables that had been dropped ten days earlier.

Mixed-run numbers. A post about a price-watching agent's sources combined figures from different runs into one table, called a source that had already been fixed broken, and described sources as shops.

Half-true configuration. A Docker post said log size limits were set. The config file did exist, but containers created before it weren't covered.

Claims that were never true. A post about my secrets manager said the automated identity had read-only access. It never did. That also turned out to be in a post already live since 6 September.

Security-posture detail. Some drafts described access arrangements more specifically than I want public. Those lines were cut rather than corrected.

Checking the drafts found two live bugs

The most useful outcome wasn't the corrected posts. To check whether "every agent is traced" was true, I had to look, and it wasn't: when the tracing service moved address, the main configuration was updated but twelve per-profile environment files still pointed at the old one. The specialist agents had been silently untraced.

Checking a claim about which model the agents use turned up another. A canary that tests the paid cloud model had reverted four profiles to a local model on 7 September, because its probe prompt came back redacted by a privacy filter. It only restores automatically after a credit problem, not after this kind of revert, so they sat on the local model for more than 17 days while its log said there was nothing to watch. Both were fixed the same day: the twelve files repointed, and the canary changed so it re-probes after any revert and moves back when a paid model passes.

A draft making a confident claim about my own systems is a cheap audit prompt. It is only useful if someone checks the claim.

What the 18 September post got wrong

Reading it again:

"The model never went looking for a fact, so it never had the chance to fabricate one." It doesn't need to go looking. Given a gap, it fills it.
The leak gate is real and it works, but it is structural. It catches an IP address or a credential shape. It cannot know that "two weeks of debugging" is false.
"A ten-second read by the person whose name is on it can" catch a subtly overstated claim. The endianness bug reads fine. You only catch it by opening the code.
Topics were "derived from things that demonstrably happened". The empty repo shows a topic can come from a description alone.

That live post about the secrets manager has been corrected in place, with a dated correction note saying what was wrong, rather than edited silently.

What I do now

"Drafted OK" is not "true". Passing the format and leak checks means the file is well formed. It says nothing about the content.
Every draft is checked against the code, logs or notes before it is queued, claim by claim. For numbers, a live read beats a note, and a note beats a code comment.
Check the predictable categories first: any "I tried X first", any fix, any count, any statement about access or permissions.
Correct published posts visibly, with a date.

The nine drafts have been fixed and queued one a day. The pipeline still saves me the typing. It does not save me the checking.

I wrote the full pattern up, with the validators, the canary that lied and a checklist, as a short field report: Keeping AI Agents Honest.

DHCP lease churn in a Proxmox homelab: when Authelia down x87 was a lost IP

Christian Anderson — Wed, 07 Oct 2026 08:00:14 +0000

On 2 September my self-healing monitor logged DOWN: Authelia 87 times in a row. Authelia, my SSO portal, was never down. Its own address answered 200 the whole time.

What had gone was an IP address. This post covers that incident and the others that turned out to be the same bug: a smart bulb that took a server's address, a cloned container sharing a MAC, and a re-IP that switched off my tracing.

"It's always been on that address" is not "it's pinned"

Both Proxmox nodes rebooted at about 14:16 that afternoon. The container that runs my reverse proxy came back and DHCP gave it a different address.

Everything pointed at its usual address: AdGuard rewrites, Cloudflare records, proxy configs, monitors. But there was no DHCP reservation for it on the router. The address was only ever a sticky lease: dnsmasq tends to hand a returning client its old address, until the day it doesn't. The pool had 150 addresses on 12-hour leases.

That container runs my only Caddy instance, and six hostnames resolve to it, including the SSO portal. One lost lease took out the front door and all SSO while every backend stayed healthy. My monitors probe by hostname, so they blamed the wrong service.

The lesson I wrote down: when a monitor says a service on another box is down, probe that service's own address and port before touching it.

Making it static, and the lease that wouldn't leave

I set a static address in the container's Proxmox network config with pct set ... -net0 ...,ip=<addr>/24,gw=<gw>. That rewrites the container's systemd-networkd file to a fixed address with DHCP off, and hot-adds the static address. But the old dynamic address and its proto dhcp default route survived a systemctl restart systemd-networkd. I had to delete both by hand with ip addr del and ip route del. Until then the container held both addresses.

After that, SSO answered again and the healer went quiet.

The bulb that took a server's address

That evening I swept every service to check it was really back. My password manager's hostname returned 502, while its container reported itself healthy. Caddy's log showed its upstream dial failing with connect: connection refused.

The fault was at layer 2. The password manager's container was set to a static address, but that address sat inside the DHCP pool. So the router had leased it to something else, and the MAC answering for it began with an OUI registered to WiZ, the smart bulb maker. Ping times to it were around 15 ms, which is wireless, not a container on the same host.

The bulb won the ARP race across the whole network, even on the Proxmox host. A static IP inside a DHCP pool is not a static IP. The router doesn't know it's taken and will hand it to the next device that asks.

The fix was to move the container to a static address below the start of the pool, where the router can never lease it. Only one live reference needed changing: the reverse_proxy line in the Caddyfile, then systemctl reload caddy.

The third one, and a false green

Same night, a third service: my 3D printer monitoring app returned 502 while the app was healthy. My NPMplus reverse proxy on another box still had its upstream set to the container's pre-reboot address. I repointed it and reloaded nginx inside the container.

The monitoring was worse. Uptime Kuma had two monitors for that service, both on the old address:

the HTTP check was red with ECONNREFUSED: the right alarm for the wrong reason;
the ping check was green, because an unrelated household device had inherited the old lease and was answering pings.

A stale IP in a ping monitor doesn't go red. It goes green on whatever takes the address. Meanwhile my healer, which only reads Kuma, had started escalating to rebooting a container that was never unhealthy. Only its limit of three reboots an hour held it back.

Two more lessons came out of this one:

A raw-IP probe could never have worked. The app is Django using the sites framework, which picks the site by the request's Host header. Probed by IP and port, the login page returned 500. With the right Host header it returned 200. The old monitor had only ever passed because a site record existed for the old IP and port. I pointed the monitor at the public URL instead, the only probe that would have caught the real user-facing fault in the proxy.
A 3xx is not proof of health. I called that service "restored" off a 302. The 302 was the redirect to the login page, which was returning 500. Kuma followed the redirect, which is how it caught what my curl missed.

Kuma's history showed the old monitor green until 08:49 that morning, before the 14:16 reboot. That container's lease had turned over mid-morning, not at the reboot at all.

The tally for one unreserved address, over that night and the next day: two container IPs, a Caddy upstream, two proxy upstream fixes, five references in source files and two monitors, all changed by hand. One of those source files would have quietly recreated the broken raw-IP monitor on its next run.

Reservations and static config protect different things

On 3 September I added DHCP reservations on the router for all ten containers, then checked each still held its expected address and every front-door hostname answered.

This is the part I had misunderstood:

Static config in the container stops the container asking for a new address.
A reservation on the router stops the router giving that address to someone else.

The bulb incident was the missing second half. The rule I now follow: static hosts live below the pool, DHCP keeps the pool, and anything that other config names by address gets a reservation.

An older version of the same bug: two containers, one MAC

In August I'd already met the extreme case. On 5 August I migrated my agent container to a newer Proxmox node for more RAM, and never stopped the original. Both were set to start on boot. They were byte-identical clones: same MAC, so the same DHCP lease and the same IP, the same hostname, and the same NetBird identity.

The switch learns a MAC wherever it last saw a frame, so replies went to whichever twin had spoken last. That explained the "flaky networking" I had been chasing:

SSH banner stalls in roughly 20–30% of connections;
Uptime Kuma's login acknowledgement never arriving;
NetBird reconnecting 11,898 times in 24 hours on that one peer, against 2–3 for every other peer, because two clients held one identity and kept evicting each other.

ARP looked clean, because both twins answered with the same MAC. I first called it a layer-2 loop, which was wrong. A paired tcpdump on both ends finally showed a reset from our address that our container had never sent. That was the twin.

I stopped the old container on 9 August. NetBird's reconnect log for that peer dropped from about 24 per three minutes to 1.

The re-IP that quietly switched off tracing

The latest instance was this week. My Langfuse container came back from a node reboot on 22 September on a static address that someone or something had set in its config. It was inside the DHCP pool too, and different from its reservation. Uptime Kuma went red with EHOSTUNREACH for 2,382 heartbeats. Langfuse itself was healthy throughout, but my agents' tracing, pointed at the reserved address, went dark.

On 23 September I moved the container back to its reserved address rather than repointing everything else, because the reservation is the source of truth and a static address inside the pool is the bulb trap again.

That wasn't the end. On 22 September every specialist agent profile's config had been repointed to the wrong address, and none of those files went back when the container did. So every specialist agent stayed untraced until I found it on 24 September, repointed them, restarted the gateway and confirmed new traces arriving.

An IP change has to update every config that names the address, including the per-profile copies, and the monitors.

What I check now

Is this address reserved on the router, or just sticky? "It's always been there" proves nothing.
Is any static address inside the DHCP pool? Move it below the pool.
After any IP change: grep configs on every host, including proxy configs inside containers on other boxes, per-profile config files and every monitor.
Prefer hostname probes that follow redirects over pings to an IP.
After cloning or migrating a container, make sure the original is actually stopped.

The longer version of this, with the DNS, DHCP and tunnel traps side by side and a checklist, is a short field report: Home Network Traps.

Gumroad API pagination and HTTP 200 errors: the store that looked half-empty

Christian Anderson — Tue, 06 Oct 2026 08:00:13 +0000

I run a small Gumroad store from a set of scripts on my homelab. The scripts create products, set prices and tags, and check what is already listed before they list anything new. Twice this month they gave me a completely wrong picture of that store, and neither time did anything error. Both mistakes came from the same place: the Gumroad API tells you what happened in a way that is easy to misread.

There are two traps. The HTTP status code tells you almost nothing, and the product list comes back ten at a time.

Trap one: everything is HTTP 200

In late August I wrote down, with some confidence, that Gumroad's API could not create products. The note said POST /v2/products returned 404 and that creation was dashboard-only. I built the rest of the tooling around that: a script that packaged the files, then a manual step where I would click through the dashboard.

On 5 September I ran the full lifecycle against the live account to check, and the note was wrong:

POST   /v2/products              success=True   (created, unpublished)
PUT    /v2/products/{id}         success=True   (edit)
PUT    /v2/products/{id}/enable  success=True   (published)
DELETE /v2/products/{id}         success=True   (account back to 0 products)

Creation works. What had misled me is what a bare POST /v2/products returns. It is not a 404. It is an HTTP 200 with this body:

{"success": false, "message": "New products should be created with a price"}

Gumroad answers nearly everything with 200 and puts the real outcome in the success field. A product that does not exist is also a 200:

{"success": false, "message": "The product was not found."}

So there are two wrong readings available, and I had managed one of them. Treat the 200 as "the endpoint exists and it worked" and you carry on with a failure. Or half-remember a failure as "it 404'd" and you conclude the feature is missing. The status code in this API tells you that you reached Gumroad. It does not tell you whether Gumroad did what you asked.

Once I read success instead, the manual step disappeared. The scripts now create, edit, publish and tag products end to end. Tags, for the record, work through PUT /v2/products/{id} with repeated tags[] parameters. I applied them and read them back to confirm rather than trusting the response.

A related one caught me later, when I switched a product to pay-what-you-want. customizable_price and suggested_price_cents have to be sent together. Send only one and the product renders as plain free. Again, no error.

Trap two: ten products, then `next_page_url`

On 20 September I added a sync command. It lists and publishes every item in my local catalogue that is not live on Gumroad yet. It already had a guard: before creating anything, check whether a product with that name is already on Gumroad.

GET /v2/products returns only the first ten products. The rest sit behind a next_page_url field in the response. My status() function and the duplicate check in create() both read page one and stopped.

The consequences were all quiet:

The store appeared to have 10 products. It had 22.
A diff of catalogue against store said 10 catalogue items had never been listed. In fact 14 of the 15 were already live.
The duplicate guard could not see page two, so the first sync run created two duplicate products. I deleted both the same night and kept the older originals, which were the ones already linked from elsewhere.

Nothing errored. Nothing was wrong with any single request. The bug was in what I assumed a single request meant.

The fix was a function that walks the pages:

import json
import urllib.parse
import urllib.request

API = "https://api.gumroad.com/v2"


def gumroad_get(url, token):
    sep = "&" if "?" in url else "?"
    url += sep + urllib.parse.urlencode({"access_token": token})
    with urllib.request.urlopen(url, timeout=60) as r:
        body = json.load(r)
    # HTTP 200 means nothing here. The outcome is in `success`.
    if not body.get("success"):
        raise RuntimeError(f"gumroad: {body.get('message')}")
    return body


def all_products(token, max_pages=25):
    url, out, pages = f"{API}/products", [], 0
    while url and pages < max_pages:
        body = gumroad_get(url, token)
        out += body.get("products") or []
        url = body.get("next_page_url")
        if url and not url.startswith("http"):
            url = "https://api.gumroad.com" + url
        pages += 1
    return out

A few details in there matter. The loop accepts next_page_url as either a full URL or a bare path, so it does not break on whichever form comes back. The token is added with & when the URL already carries a query string, which the next-page URL does. The page cap of 25 is there so a bad cursor cannot loop forever. And the success check runs on every page, not just the first.

After the fix, status() and the duplicate check both use all_products(). The note I left in the code says it plainly: anything asking "is this already on Gumroad?" must use this function, never a single call to the products endpoint.

Trap two again, four days later

On 24 September I built a separate agent to propose better titles, summaries and tags for the store and my blog posts. It has its own helper module with its own Gumroad client, written from scratch after the 20 September fix.

It read page one only. The store had 21 products at that point, and the agent could see 10. Eleven were invisible, including several of the guides I had specifically asked it to look at. My own check missed the same eleven.

The fix was the same loop, in a different file. I then ran a second batch to add summaries to those 11 products and read each one back.

This is the part I find worth writing down. The 20 September fix was correct and it was tested. It fixed one function in one script. The pattern — call the list endpoint once and treat the result as the whole store — lived in my head, and it came out again the next time I wrote a Gumroad client from scratch.

Grep for the callers, not the bug

While writing this post I grepped my scripts directory for anything that touches v2/products. Four files came up. Two now paginate. One is the batch script that fixed the missing summaries. The fourth is the script that records distribution numbers per product. It still reads page one:

# ⚠️ Gumroad answers EVERYTHING 200 and puts the real outcome in `success`.
# A missing product is 200. A malformed request is 200. Read `success`.
d = _get(f"https://api.gumroad.com/v2/products?access_token={tok}")

It had the first lesson written into a comment, and it still made the second mistake. I fixed it the same afternoon with the same loop, and it now sees all 21 products instead of 10.

So I now treat an API quirk like this as a search, not a patch:

Read the outcome field, not the status code. For Gumroad that means success. A 200 only means the request arrived.
Assume every list endpoint is paginated until you've proved otherwise. Check for a next_page_url, cursor or page field on the first response, even when you think the list is small. Ten felt like a whole store to me because it was a round, plausible number.
When you fix it, grep for every caller. Search for the endpoint string, not the function name, because a second client written from scratch won't share your function names. Then check each hit.
Don't let your own check share the agent's blind spot. Mine missed the same eleven products the agent did, so it agreed with the agent instead of catching it.
Count before you create. If a script is about to create something because it "isn't there yet", the thing it checked against needs to be complete, or the guard is decoration.

None of these failures raised an exception. Every request succeeded. The store just looked smaller than it was, and three scripts believed it.

Cloudflare Tunnel 530 errors: the connector that was healthy and did nothing

Christian Anderson — Mon, 05 Oct 2026 08:00:14 +0000

It started as "Immich has gone down". Immich was fine. The server container was healthy and answered locally. What had gone was every public hostname I run through Cloudflare Tunnel, all at once, and it had been that way for about a day before anyone noticed.

This is what happened, the mistake that made me declare it fixed when it wasn't, and the checks I now use so the tunnel can't lie to me again.

A healthy container connecting nothing

On the evening of 3 September, around 22:00, two containers on my NAS were recreated. Both ran a community web UI wrapper for cloudflared, the Cloudflare Tunnel client. When they came back up, both logged this:

CONFIG: No pre-existing config file found

That line is the whole incident. The wrapper is a web UI, not a connector. With no config it starts happily, shows as healthy in docker ps, and connects no tunnel at all. Both of its config directories were empty. From Cloudflare's side the tunnel had zero connectors, so every public hostname behind it returned 530.

That included my single sign-on portal. Anything behind SSO was unreachable from outside, not just the photo library. The photo library was just the first thing to be noticed.

The container's health check was checking the UI, and the UI was fine. Nothing in docker ps distinguishes "a web page is being served" from "traffic is flowing through a tunnel".

The fix: run an actual connector

Instead of trying to rebuild the wrapper's config, I ran the real thing, the official image with the tunnel token:

docker run -d --name cloudflared-connector --restart unless-stopped \
  cloudflare/cloudflared:latest tunnel --no-autoupdate run --token "$TOK"

I pulled the token from the Cloudflare API (GET /accounts/{account}/cfd_tunnel/{tunnel}/token) using an API token already scoped to Tunnel edit and DNS edit. The tunnel then reported healthy with connections=4, and all five public hostnames were back.

A connector-only container has no UI and nothing to forget. The token is the whole configuration, because the routes live in Cloudflare's dashboard. That is exactly what I want from the piece that holds the front door open.

The two config-less wrapper containers kept running afterwards, doing nothing. They're harmless, but they're a good reminder that "running" and "working" are different claims.

The trap: split-horizon DNS makes local tests lie

Here is the part I'm least proud of. During the incident I tested a hostname from inside the network, got a clean redirect back and declared external access working. It was 530 the whole time.

My router's resolver is AdGuard Home, and it has rewrites for my own domain. Inside the house, a lookup for one of my hostnames returns the private address of my reverse proxy. Outside, public DNS returns Cloudflare's addresses. That is deliberate split-horizon DNS, and it means:

curl https://<my hostname>/ from any machine at home never touches Cloudflare.
It goes straight to the reverse proxy on the LAN, gets a perfect 302 to the login page, and proves nothing about the tunnel.

The tunnel could be completely dead and every local test would still pass.

How to test the real path

First, check whether you're in split-horizon territory at all. Compare what a public resolver says with what your own machine says:

dig +short @1.1.1.1 app.example.com   # what the internet sees
dig +short app.example.com            # what this machine sees

If those answers differ, a plain curl from this machine is testing your LAN, not your tunnel.

Then force the request through the Cloudflare edge by pinning the hostname to the public answer, while keeping the right SNI and Host header:

EDGE=$(dig +short @1.1.1.1 app.example.com | head -1)
curl -sS -o /dev/null -w '%{http_code}\n' \
  --resolve app.example.com:443:"$EDGE" https://app.example.com/

A 530 here means the edge has nowhere to send the request. A 200, 302 or 303 means the full path works: edge, tunnel, connector, origin.

--resolve also splits problems the other way. Point it at your reverse proxy's address instead, and it proves the proxy and certificate work independently of DNS. If that passes and the normal request fails, the fault is name resolution, not the service.

Ask the client's own resolver

The same mistake has caught me with DNS on the client side too. The week before, a household device had been handed a smart-DNS streaming box as its resolver by a DHCP tag, and that box answers nothing at all for my domain. I spent a session checking AdGuard, Cloudflare, the browser cache and the firewall, and they were all innocent.

The step that would have found it straight away: query the resolver the client is actually using, the one in its own DHCP lease, not the one you think it should be using. dig @1.1.1.1 and dig @router tell you about those resolvers, not about the device that's broken.

A public record pointing at a private address

One more split-horizon wrinkle turned up the same day. One of my hostnames had an A record in public DNS pointing at a private LAN address. It had only ever worked at home, because at home that address is reachable. From anywhere else it was a dead end. I moved it onto the tunnel, with the tunnel sending it to the reverse proxy and setting the origin server name so the proxy picks the right site block and still applies SSO. Then I verified it the only way that counts: GET and POST from the edge, both redirecting to the login portal.

Monitoring that was lying as well

On 13 September I found my Uptime Kuma "Cloudflare" monitor had been red for more than 2,700 heartbeats while the tunnel was fine. It had been polling the wrapper's web UI port, which no longer answered, so it got ECONNREFUSED. A monitor that's always red gets ignored, which is just as bad as one that's always green.

I replaced it with a push monitor that tests the path a real visitor takes:

a script on a systemd timer runs every five minutes;
it resolves two of my hostnames through 1.1.1.1, so it gets the public answer, not the split-horizon one;
it calls each with curl --resolve to the Cloudflare edge;
it only reports DOWN if both come back 530 or give no answer, so one flaky app can't page me about the tunnel;
Kuma expects a push every 600 seconds, so if the script itself dies, that goes red too.

It has a --dry-run flag so I can see what it would report without pushing anything.

The happy ending: a four-minute self-heal

This morning, 24 September, the NAS went down hard. Its journal simply stops mid-stream at 10:13:24 BST, with no shutdown sequence, which is what a power loss or hard reset looks like. It booted again at 10:14.

At 10:17 the connector container, set to restart unless-stopped, re-registered all four connections with Cloudflare. Total public outage: about four minutes. I didn't have to do anything.

Three weeks earlier, a container recreate had cost a day of silent outage. The difference wasn't a clever fix. It was running the real connector instead of a UI wrapper, and testing through the edge instead of through my own DNS.

What I'd tell myself three weeks earlier

A healthy container is not a working tunnel. Check the connector count on Cloudflare's side.
If your LAN resolves your own domain to a private address, every local curl is testing your LAN. Use curl --resolve against the public answer.
Compare dig @1.1.1.1 with your local answer before trusting any test.
When a client can't resolve something, query that client's own resolver first.
A monitor that probes the wrong thing is worse than no monitor. Probe the path a user takes.
Give the connector a restart policy and nothing else to remember. Then a power cut is a four-minute blip, not a day of "Immich is down".

The longer version of this, with the DNS, DHCP and tunnel traps side by side and a checklist, is a short field report: Home Network Traps.

AI agent SOUL.md: writing a soul for my chief of staff, and the rule I broke

Christian Anderson — Sun, 04 Oct 2026 08:00:20 +0000

My homelab agent had a job title and no personality.

Its identity file opened with "You are Hermes Chief of Staff" and then read like an HR policy: decomposition rules, card templates, guardrails. All useful. None of it told the model who it was, who it worked for, or what the house actually runs. So it behaved like a very polite ticketing system.

Three things made me rewrite it in one evening:

It routed work to 5 specialist profiles. There were 11, so six of them were invisible to it.
I gave it a written brief for a family party playlist — named artists, a running order, a reference DJ mix. It ignored the brief and built its usual weekly mix: sixty tracks, mostly rap, some of them explicit. For a baptism.
Three "blocked" tickets all said the same thing: no password, can't log in, SSH locked out. Every one of them was wrong. The access existed. One of the specialist agents had written a made-up vault path into its own notes and had believed it ever since.

This is what I changed, what the docs told me to do, where I ignored them, and how the same file ports to Claude Cowork and Grok Bot.

What a soul file actually is

I run Hermes Agent from Nous Research. Its Personality & SOUL.md docs are short and worth reading in full. The bits that matter:

SOUL.md lives in HERMES_HOME and is slot #1 of the system prompt. It's the first thing the model reads, every turn.
It's loaded only from HERMES_HOME, never from whatever directory you happen to be in, so the personality doesn't change per project.
It's scanned for prompt-injection patterns before it's included. Keep it to persona and voice, not clever meta-instructions.
Empty or unreadable file → a built-in default identity.
/personality is a session-level overlay. The soul is the default; the overlay is a mood.

The companion guide, Use SOUL with Hermes, gives a four-part skeleton — Identity, Style, Avoid, Defaults — and one rule I'll come back to:

It also calls out the weak version: files full of project details, or generic filler like "be helpful" and "be clear". My old file was mostly the first kind.

Give it a name that says what it does

The agent is now Johnny Decoder. Silly, memorable, and it describes the job: take the noise and decode it into what matters. The platform is still Hermes; the character isn't.

(Update: on 24 September it was renamed again, to KMDENZEL, "Denzel" for short, a CIA-style cryptonym. This post keeps the name it had when I wrote it.)

The identity section went from one line to this (trimmed):

## Who you are
You're a London-based chief of staff with a sound system in the back office.
Calm under pressure, dry sense of humour, allergic to waffle. British spelling,
contractions, plain words.

Although you're chief of staff, you're an all-rounder. You live on Discord and
know it inside out. You're deeply technical: ten years in tech research, you read
code fluently. You still hand the hands-on work to the specialists — knowing how
is what makes you good at checking their work.

You'd rather say "I don't know" than bluff. When you've messed up, you say so
first and fix it second.

Two things I learned writing it:

Taste is a behaviour, not decoration. "You know the difference between Afrobeat and Afrobeats, dancehall and bashment" does more work than "be knowledgeable about music". It gives the model something to be right about.
Say how it should fail. "When you've messed up, say so first" is there because the old version buried its mistakes in paragraph three of a status report.

Tell it what the house runs

The old soul never said what the agent was responsible for. The new one has a table: homelab, security, trading, writing, music, style and deals, the separate eBay bot, home lighting, travel, licences, tool research, income — and who owns each.

Then a routing table with all eleven specialists. This one matters more than it looks: in my setup the Kanban dispatcher silently skips an assignee name that doesn't exist, so a ticket for a made-up agent sits in ready forever. The table is the only list it's allowed to route from.

And the rule that came straight out of the false tickets:

A blocker that says "no password", "needs Infisical login" or "SSH locked out"
has been wrong every time so far. The access usually exists. Question that kind
of block before you pass it to the owner.

A soul file is a good place for scar tissue. Just keep it to one line per scar.

The rule I broke

Look back at that docs quote. Project details belong in AGENTS.md, not the soul.

My soul now contains a channel map (which Discord channel gets which alerts), the paths of the knowledge files it should read, and the specialist routing table. By the docs' definition that's project detail. I put it there anyway because this agent doesn't have a project directory — it lives in a Discord gateway, and the soul is the one file guaranteed to be in every turn.

It has a cost. The file went from about 8.6 KB to 15 KB, and on a local 6 GB GPU the real latency cost of a turn is prompt processing, not generation. Every KB is paid on every message.

What I'd do next time: keep the soul to identity, style, avoid-list and defaults, and put the house map somewhere the agent reads on demand. If your tool supports AGENTS.md, use it. I'm treating my version as a known debt, not a pattern to copy.

Memory is the other half of the soul

A soul says who the agent is. Memory says what it has learned. Hermes has a built-in memory system: a USER.md about you, a MEMORY.md of the agent's own notes, both injected every turn with a size cap, plus an optional external provider. I also run Honcho, self-hosted, for cross-session user modelling.

When I checked, the external provider was fine and the agent's own MEMORY.md didn't exist. Weeks of work, nothing written down. Meanwhile a specialist profile's notes held the invented vault path that caused every false "no password" ticket.

So the soul now has a memory section. The short version:

Store	What goes in it
USER.md	Durable facts about the owner: tastes, preferences, how he likes things done
MEMORY.md	Durable lessons about the estate: what broke, what the real fix was
Skills	Procedures that worked and will be needed again
Honcho	Runs in the background; the agent doesn't manage it

And the rules:

Only save what you verified — from a tool, or said by the owner. A guess written to memory becomes a fact for every later session.
Date anything that can go stale.
After a mistake, save the lesson in one line: what you assumed, what was true, how to check next time.
Fix memory the moment it's wrong.
Never store secrets.

"Self-evolving" sounds grand. In practice it's those five rules plus a weekly job that consolidates skills, and a daily IT check that now flags an empty or stale memory file. The agent that should have noticed its own memory was empty is now told to look.

Write a bio on yourself first

The most useful hour wasn't writing the agent's personality. It was writing mine.

The agent's USER.md had one paragraph, and half of it was wrong. I replaced it with what a sharp new colleague would need on day one:

- Lives in London — all times Europe/London.
- Dad; the music account is shared with the kids, so "top tracks" is nursery rhymes.
- Music: 90s hip hop, UK rap, R&B, afrobeats, dancehall, lovers rock, UK garage.
  A brief he gives beats any default.
- Films: comedy, thrillers, true crime, murder mysteries.
- Rules: never guess; never spend or send without a yes; work email never
  goes near the personal GitHub.

That last music line is the baptism fix. None of this is personality — it's context — but without it the personality has nothing to be personal about.

If you do one thing after reading this, write that file. It ports everywhere.

Porting the soul to Claude Cowork and Grok Bot

The same split — identity and rules that always apply, context that depends on the job, memory that grows — maps onto the other agent products surprisingly cleanly.

Claude Cowork

Cowork is Anthropic's agent mode in the Claude apps, on paid plans. It has two instruction layers:

Global instructions (Settings → Cowork, or Settings → General in the newer app) apply to every session. That's your soul: name, voice, avoid-list, hard rules like "never delete without approval".
Folder instructions attach to a local folder you give it, and Claude can update them as it works. That's your AGENTS.md: the project details my soul shouldn't contain.

The community pattern that works well is an "About me" folder with three files — who you are, your voice (with a few real writing samples), and your rules. There's a template set in davila7/claude-cowork-guide. My USER.md drops straight into the first file.

Grok Bot

Grok Bot is SpaceXAI's always-on agent product, launched 11 August 2026. Per the bot docs, each bot has a profile — Bot actions → Edit Profile — with a name, title, description and avatar. The docs are explicit about what goes where:

The description is for "rules that should remain true", and you update it when you discover "a durable preference, boundary, or responsibility".
The conversation is for task-specific instructions.
Memory builds up as the bot works: preferences, facts, summaries.

That's the soul/AGENTS/memory split again, just with different labels. The name and title fields are where "Johnny Decoder, Chief of Staff" goes; the description takes the identity paragraph, the avoid-list and the hard rules. The docs don't state a length limit, so I'd still keep it to the stable parts.

Claude Code

For coding agents, Claude Code's memory docs cover CLAUDE.md files at user and project level — the same global-versus-project split. And if you're building on the API, Anthropic's guide to giving Claude a role with a system prompt is the underlying technique all of this rests on.

The mapping

Layer	Hermes	Claude Cowork	Grok Bot	Claude Code
Identity, voice, hard rules	`SOUL.md`	Global instructions	Profile name, title, description	User `CLAUDE.md`
Project context	`AGENTS.md`	Folder instructions + files	The conversation	Project `CLAUDE.md`
About you	`USER.md`	"About me" files	Learned memory	Memory files
What it learned	`MEMORY.md`, skills	Chat memory	Learned memory	Memory files

Write the soul once, in plain Markdown, and you can paste the top half into any of them.

Reading list

Docs I actually used:

Hermes: Personality & SOUL.md and Use SOUL with Hermes
Hermes: memory
Anthropic: Claude's character — how Anthropic thinks about personality as a set of traits, not a costume
Anthropic: system prompts
Claude Cowork: getting started
Grok Bot: create and manage bots
AGENTS.md

GitHub repos worth a look (all MIT unless noted, checked September 2026):

NousResearch/hermes-agent — the agent itself.
aeonfun/soul.md — builds a soul from your own data with Claude Code or OpenClaw; its SOUL.template.md is a good blank page.
thedaviddias/souls-directory — a directory of ready-made soul files.
bchop-studio/soul-templates — eight small personas; useful for seeing how little text a strong voice needs.
davila7/claude-cowork-guide — folder-instruction templates for Cowork (no licence file at the time of writing).

Read the templates for voice. Don't copy one wholesale: a borrowed personality with none of your context is still the generic assistant, with a funnier opening line.

The checklist I'd use next time

Write your own bio first: time zone, household, tastes, hard rules.
Give the agent a name that describes the job.
Identity, Style, Avoid, Defaults. Specific traits, not "be helpful".
One line per scar: the mistakes it keeps making, and how to check.
Keep project detail out of the soul, unless you've decided to pay for it on every turn.
Add memory rules: verified only, dated, corrected, no secrets.
Check it's being used. An empty memory file is a silent failure.
After changing it, start a fresh session. Gateways often cache the system prompt, and the old personality will keep answering until you do.

The rewrite took an evening. Most of that time went on the bio and the memory rules, not the personality — which is roughly the opposite of what I expected going in.

Homelab census: 41 containers, one 6 GB GPU, and where my AI agents run

Christian Anderson — Sat, 03 Oct 2026 08:00:14 +0000

Today I asked my LLM box what it was doing, and it told me it had a 27-billion-parameter model loaded. Total footprint 18.3 GB. Amount of that on the graphics card: 0.5 GB.

So the 27B was "running on the GPU" in the same sense that I am running a marathon when I walk to the shop. The other 17.8 GB sat in system RAM and did its arithmetic on the CPU, one patient token at a time.

That seemed like a good moment for a proper census: what is actually running, on what, and which AI jobs go to local hardware, a paid cloud API or a flat subscription. The short version: 41 Docker containers and 9 LXC containers across four boxes, and a single 6 GB card makes most of the routing decisions for me.

(Background, already written up: the small-company org chart, the six-job agent team and why the coding agent moved to OpenCode and OpenRouter.)

The census

Measured on 24 September.

Box	Hardware	What it runs
Proxmox node 1	4 cores, ~8 GB RAM	5 LXC containers: three single-purpose service boxes, a Docker host (12 containers) and a Docker "proxy node" (4 containers)
Proxmox node 2	4 cores, 16 GB RAM	4 LXC containers: the Hermes agent box (native, no Docker), Obico (4 containers), Vaultwarden (1), Langfuse (6)
ZimaBlade	2 cores, 16 GB RAM, fanless	14 Docker containers on ZimaOS
LLM box	i7-10875H (16 threads), 31 GB RAM, RTX 2060 with 6 GB VRAM	Ollama with 21 model tags, and ComfyUI sharing the same GPU. No containers.

The arithmetic, since I got it wrong the first time: node 1 has 12 + 4 = 16 Docker containers, node 2 has 4 + 1 + 6 = 11, so 27 on the Proxmox side, plus 14 on the ZimaBlade. That's 41 Docker containers and 9 LXC containers, plus three native services: Ollama, ComfyUI and Hermes itself.

All of that sits on 26 CPU threads and 71 GB of RAM, if you add up four boxes that were never designed to be added up.

What's in the Docker hosts

The node 1 Docker host is the toolbox: crawl4ai for page fetching, SearXNG for search, ESPHome, Wyoming Whisper and Piper for speech-to-text and text-to-speech, a small web app plus its public demo, and Honcho, which is four containers on its own (API, database, deriver, Redis). The proxy node carries Home Assistant, Mosquitto, and the NetBird server and dashboard.

On node 2, Langfuse, which traces the agent's model calls, is six containers (ClickHouse, web, worker, MinIO, Postgres, Redis). That's more containers than the thing it observes.

The ZimaBlade is the photo library and the networking plumbing: Immich (server, machine learning, Postgres, Redis), the tunnel connector and reverse proxy, a Homepage dashboard, and omp, a terminal coding agent that points back at the LLM box, plus a few small utilities.

Two things the census doesn't show

Node 2's guests all live on a single 5400rpm hard drive. The agent moved there for the extra RAM, and it was the right call, but every reboot is a queue of cold-starting databases fighting over one spindle. The first time it happened the agent's dashboard answered 502 for about 15 minutes. The fix was stopping Langfuse, the heaviest starter and the one thing nobody misses for ten minutes. The dashboard came up 21 seconds later.

The move itself left a twin behind. The original agent box on node 1 was never stopped: same MAC, same IP, same VPN identity. What I called "flaky networking" for days was two containers taking turns answering for one address. A census would have caught it on day one.

The AI side, and the card that decides it

Here's the rule the whole routing falls out of: the agent framework I run refuses any model with less than a 64K context window, and the LLM box has 6 GB of VRAM.

Those two facts don't get along. The best local model I've measured for the job is qwen3.5 9B, built as a 64K variant. At 64K context it's 7.66 GB in total, of which 3.93 GB lands on the card. That's 51% on the GPU and the rest on the CPU, and no amount of context tuning fixes it: at 8K context it's still only 64% on the GPU, because Ollama pins roughly the same slice of VRAM and spills the remainder.

I spent much of August trying to shop my way out of that. Every candidate was measured and rejected:

Smaller qwen2.5 builds fit the card much better, but their context is architecturally 32K. Ollama silently clamps a 64K setting back down, so the framework rejects them outright.
The 27B loaded at 4096 context when I benchmarked it, still far below the 64K floor, and in a like-for-like test prefilled at 132 tokens a second against 604 for the 9B, and generated at 3.0 against 10.7. A typical agent turn would have spent over two minutes before producing its first token.
qwen2.5-coder, the obvious pick for a coding agent, can't tool-call. It writes the JSON for the tool call into its reply as prose, and the agent prints it and stops. Its 16K and 32K tags are still on disk, a small memorial to a good idea.

So the local model is genuinely the best option on this hardware, and a real agent turn on it still took around a minute and a half when I timed it in August. The same work on a cheap cloud model answers a tool-call probe in 1.5 to 2 seconds.

That gap decides most of the routing.

Who goes where

Job	Runs on	Why
Coding agent	OpenCode Go subscription (`deepseek-v4-flash`)	Flat $10 a month. Usage caps, not a meter
Chief of staff, morning brief, research, writing, travel	OpenRouter `qwen/qwen3.7-flash`	Pay per token, about $1.39 a month at measured volume
Security audit and code review	Local Ollama only	Deliberately no third-party egress
Embeddings, session titles, compression	Local Ollama	The framework's auxiliary calls stay local even when the main model is remote
Honcho user modelling	Local Ollama	Background work where latency doesn't matter
Images	ComfyUI (SD1.5) on the same card	Free, and small enough to share
Speech	Whisper and Piper containers, CPU	Never touches the GPU at all
Last resort	An OpenRouter free model	Rate-limited, so it only ever goes last

Paid cloud: OpenRouter

The cloud model became the primary for the general profiles in late August. The number that justified it came from measuring 30 days of real usage: 44.35M input tokens against 421K output. Input outweighs output by about 105 to 1, which means input price sets the bill, and headline "per million" pricing is close to useless for ranking models. On a model at $0.03 per million input tokens, that volume comes to roughly $1.39 a month.

The chain is vendor-diverse on purpose: qwen/qwen3.7-flash, then deepseek/deepseek-v4-flash-0731 as a second paid model from a different vendor, then the local 64K model, then a free model last. A fallback that shares its primary's failure mode isn't a fallback.

A watchdog probes the primary every three hours for two failure modes: the model dying, and the money running out. A healthy model you can't pay for is still an outage, so below a credit floor it moves everything back to local and says so.

It also got something badly wrong. On 7 September every paid model "failed" its health check at once, so the watchdog moved four agents (chief of staff, research, writing and travel) back to the local model. Nothing had died: an upstream PII-redaction filter had turned the city in the probe prompt into [ADDRESS], so no model could answer it properly. Worse, the watchdog only restored automatically after a credit revert, not a failure revert, so those four agents stayed on the slow local model for 17 days until I caught it on 24 September. It now re-probes after any failover and moves back when a model passes. A probe that fails the same way on every vendor is a problem with the probe, not the models.

Subscription: OpenCode Go

The coding agent is the heaviest user by a distance, so it's on OpenCode's Go plan instead: $10 a month, with rolling caps of $12 per five hours, $30 a week and $60 a month. Every response reports a cost of zero. On a subscription the risk isn't a bill, it's hitting a cap halfway through a task, so the watchdog reads Go's usage endpoint and tells a cap from a death. A capped profile gets parked on OpenRouter and put back automatically when the window resets.

One trap worth passing on: Go lives at /zen/go/v1, not /zen/v1. The second is pay-as-you-go and returns "insufficient balance" on a perfectly good Go key, which cost me an hour and a wrong conclusion.

Local: Ollama

Local isn't the cheap option here, it's the private one. The security and review agents see the most sensitive estate detail and never leave the LAN. Everything else local is background work, or the fallback for when the internet or a credit balance lets me down.

Running it taught me more than the cloud side did:

A hang never triggers a fallback. Something on the network kept asking Ollama to keep a model loaded forever. With only 6 GB, the scheduler waits for room rather than refusing, so every other model request just sat there, and Ollama logged nothing. Fallback chains fire on errors. This wasn't an error, it was silence, and every local-model profile was dead with nothing reporting it. A small script now unloads anything pinned for more than 48 hours, every 15 minutes.
The prompt was the bottleneck, not the model. A captured agent turn was about 18,400 prompt tokens, 74% of them tool definitions. Turning off toolsets the specialist agents had never once called cut their tool schemas by 47%. That was worth more than any model swap available.
Measure what's on the wire. For a while every benchmark I ran through the framework's one-shot mode was invalid, because that path never sent "thinking off". The real workloads were fine. My test harness wasn't.

What I'd tell someone starting

Count first. Take the census from the hosts, not from memory. Mine turned up a twin container and a tracing stack bigger than the thing it traces.

Let VRAM make the first cut. Write down the smallest context your agent needs, then check what fraction of a model at that context actually lands on your card. If the answer is half, you have your routing decision.

Rank cloud models on input price. Agent workloads are overwhelmingly prompt. Output price is mostly a distraction.

Keep local for the jobs that must stay local, and as the fallback. Don't make it carry interactive work it can't do quickly.

Assume silence is a failure. A pinned model, an empty ledger, a probe that swallows exceptions: none of them raise an error. Build the check that notices nothing happening.

A 27B model on 0.5 GB of VRAM is a fun number. It's also a fair summary of the whole exercise: the hardware will happily let you do something slow and call it running. The census is how you find out.

Infisical vs 1Password for homelab machine secrets: why I self-host

Christian Anderson — Fri, 02 Oct 2026 08:00:20 +0000

Two scripts on my agent box both needed the password for my uptime monitor. Each kept its own copy. When I finally tested them side by side, one logged in and saw every monitor, and the other had been failing authentication for who knows how long:

script A   LOGIN OK
script B   FAILED: authIncorrectCreds

Nothing had alerted, because nothing compared the two. Two copies of a secret turn out to be one copy and one liability, and nothing tells you which is which.

The same afternoon I found a third script that rotated its own machine credential by rewriting its own source file. That file sat in a directory a nightly backup pushed to my git server. So the rotation worked beautifully, and every rotated credential ended up in version control.

Both are fixed now. This post is the decision behind it: why the machine secrets live in a self-hosted Infisical rather than 1Password, which I'd otherwise happily recommend. My reasons fit on one line: it's hosted locally, I know where the cryptographic keys live, and I can manage all of it from the CLI.

Reason one: it runs on my LAN

Infisical runs as the official native package, the omnibus build supervised by systemd, inside its own small Proxmox container. Not Docker, not the cloud.

Infisical's own docs pitch self-hosting as keeping "your data on your own infrastructure and network", and for a homelab that's the whole appeal. The things fetching secrets are cron jobs, backup scripts, a monitoring daemon and a handful of agents, all on the same network. A secret request doesn't need to leave the house, and when my internet connection has a bad evening (it does), secret fetches don't care.

Local has a limit, though. My monitoring daemon deliberately does not depend on Infisical being reachable at the moment it runs, because a daemon whose job is noticing the network is down shouldn't need a network round-trip to find out. Self-hosting shortens the dependency chain but it doesn't remove it.

Reason two: I know where the keys live

This is where I want to be fair, because 1Password's cryptography is excellent and I'm not claiming otherwise.

1Password's security design white paper describes true end-to-end encryption: keys are generated on your devices, all encryption happens locally, and a "two-secret key derivation" mixes your account password with a locally held Secret Key so that what they store can't be used to crack your password. Vault items use AES-256-GCM. The company is never in a position to learn your keys. That's a strong design.

Infisical's model is different. Its security page says secrets are encrypted at rest with AES-256-GCM under a layered key hierarchy: an operator-supplied root encryption key (passed in as an environment variable) encrypts an internal KMS root key, which in turn encrypts per-organisation and per-project data keys. Their stated goal is that "compromising the database alone is insufficient to decrypt sensitive data". The self-hosting configuration docs make ENCRYPTION_KEY a required setting.

So the question for me was never whose maths is stronger. It was where the ciphertext lives and who holds the key at the top. With 1Password the ciphertext sits on their servers and the unlocking secrets sit on my devices. With self-hosted Infisical, the database, the server and the root key all sit on hardware I can touch. For machine secrets, read by unattended processes at three in the morning, I prefer the arrangement where I run the whole chain.

The honest price is that I'm now the one guarding the root key. If I lose it, the database is noise. If I leak it alongside a database backup, the layering stops helping. I also own the patching, the backups and the restore test. Nobody pages Infisical's on-call when my container falls over. That's a real cost, and anyone who tells you self-hosting a secrets manager is free is selling something too.

Reason three: the CLI is the interface

My scripts don't have a human to type a master password, so they log in as a machine identity using universal auth: a client ID and client secret swapped for a short-lived access token. The Infisical CLI does that in one line, and the shape in my shell scripts is roughly this:

# point the CLI at the self-hosted instance, not Infisical Cloud
export INFISICAL_TOKEN=$(infisical login --method=universal-auth \
    --client-id="$CLIENT_ID" --client-secret="$CLIENT_SECRET" \
    --domain="$INFISICAL_URL" --plain --silent)

infisical secrets get SOME_NAME --domain="$INFISICAL_URL" \
    --projectId="$PROJECT_ID" --env=prod --path=/SomeFolder --plain --silent

--plain prints just the value and --silent hides the update tips, which matters when stdout is being captured into a variable. If you self-host, note the docs' warning that the domain has to be set on every command (or via an environment variable), otherwise the CLI quietly heads for the US cloud. The CLI on my box is v0.38.0.

For services that take environment variables, infisical run -- <command> injects secrets into the process, and infisical export can write dotenv, JSON or YAML if something insists on a file.

Three habits grew out of driving it this way.

Names, never values, in anything automated. A nightly inventory script uses the CLI to list secret names per folder, so my estate notes can say "this folder holds these four things" without ever touching a value. When I need to know two copies agree, I compare hashes.

Delete explicitly. On my version, deleting a secret as a machine identity failed with "Must be user to delete personal secret". My first thought was that the error was wrong, but it wasn't: the CLI had created that secret as a personal secret, and deleting it needed --type=shared. The surprise was the default, not the message.

Python goes through one resolver. The Python services never call the CLI. They import one small module that resolves environment, then Infisical, then an explicit default, and raises rather than returning something wrong. Login is cached, so it's one round-trip per process, and resolution happens at the call rather than at import, so an Infisical blip surfaces as a clear error inside the function that needed the secret instead of killing an import. That module is what fixed the two disagreeing scripts: both now ask the same question of the same place.

The machine identity's own client ID and secret still have to live somewhere, because a bootstrap credential can't fetch itself. Mine sits in a locked-down file on the box, readable only by the account that needs it. The self-rewriting script now updates that file atomically and clears the token cache, instead of editing its own source.

Humans and machines are different problems

I don't use Infisical for my own passwords. Those live in Vaultwarden, the self-hosted Bitwarden-compatible server, with around 753 items. The split is simple: humans go to Vaultwarden, machines go to Infisical.

I found out why it matters by getting it wrong. At one point the Vaultwarden master password was stored in Infisical so automation could unlock the vault. That quietly collapsed the security of my entire password vault onto the agent box and the credential it logs in with. I deleted those entries. The consequence is deliberate: nothing on the estate can unlock my personal vault without me.

The same boundary shows up in small rules. When a bot needed a website session cookie, the rule was that the cookie value goes into Infisical and the script pulls it at runtime. It never gets pasted into a chat window, however convenient that would be.

The same thinking applies between machines. Most consumers only ever read a secret; the handful of jobs that rotate one are the only things that should be able to write. For a long time that wasn't true here: every script read with an identity that could also write and delete. I know because I didn't take its permissions on trust: I probed it with a throwaway secret, and it could write and delete. (That probe is also where the personal-secret surprise above came from.) The fix was a second machine identity with the project's built-in Viewer role, which my resolver now uses for every read. The install script only keeps it if a test read works and a test write is refused, and the API's answer to that write was a flat "not allowed to create on secrets". The two jobs that genuinely write, sync and rotation, keep the other identity.

What 1Password does better

Plenty, and it's worth saying plainly.

End-to-end encryption with server ignorance. 1Password can't read your data. My Infisical server can decrypt everything, because it has to hand secrets to machines.
No root key to babysit, no server to patch. Their operations team is better at uptime than my Proxmox box.
A good CLI of its own. op read, op run and op inject cover the same patterns as mine, using op://vault/item/field secret references.
Machine access without running anything. Service accounts automate secrets "without the need to deploy additional services". If you do want something local, a Connect server runs in your infrastructure and caches your data there, though it keeps that data in sync with 1Password.com and needs a 1Password account behind it. It's a local cache of a cloud vault, not a self-hosted vault.
Humans. Sharing, recovery, apps that non-technical family members will actually use. Nothing I've built competes with that.

What I'd tell someone deciding

If you'd rather not own a root key, backups and upgrades, use 1Password. That's a perfectly sensible answer, not a lesser one.
If your consumers are all on your LAN and you already run backups you've actually restored, self-hosting machine secrets is very reasonable. Budget for the patching.
Whichever you pick, keep humans and machines apart. The key to your personal vault should never be readable by a script.
Drive it from the CLI and a single resolver, not from copies. The tool matters less than the fact that there's only one place to ask.
Give readers and writers different identities, and test what each one can really do.

I didn't pick Infisical because 1Password is weak. I picked it because I wanted to know where every key lives, including the one at the top, and the only way to know that for certain was to hold it myself.

Self-hosted Langfuse: tracing 7% of my AI agents, and ClickHouse logging itself

Christian Anderson — Thu, 01 Oct 2026 08:00:15 +0000

On Discord I told my agent post followed by a comment id. That's the approval step for a dev.to reply it had drafted: read the draft, say "post", done.

Nothing got posted. The turn took 25 minutes and came back with a confident, detailed summary of the article. It was made up.

I self-host Langfuse to trace my agents, so I went and looked at the trace. Then I checked what else it had seen. It had been watching about 7% of my agents' traffic.

The 25-minute turn

Here is the trace of that one turn:

Hermes turn                      1526.3 s
  LLM call 1   qwen3.5-64k        569.4 s
  Tool: web_search                  0.8 s
  Tool: web_search                  0.9 s
  LLM call 2   qwen3.5-64k        207.4 s
  Tool: web_extract                 2.9 s
  LLM call 3   qwen3.5-64k        740.2 s

Tools: 4.6 seconds in total. Model: 1,517 seconds. The local model spent 25 minutes thinking its way to the wrong answer, with three short trips to the web in between.

Without the trace I'd have blamed the web tools. A turn that searches the web and takes 25 minutes feels like a network problem. The trace says the network was the fastest thing in the room.

The actual cause had nothing to do with speed. The Discord conversation was one long-lived session, started days before the reply skill existed. The agent reads its list of skills once, at session start. So as far as that session knew, there was no skill for posting a reply. Asked to "post" something it had no tool for, it did what models do and improvised: searched the web for the article, extracted the page, and wrote a plausible summary instead.

The fix was a fresh session. But it made me curious about what else the traces could tell me.

What the traces said about models

Pulling latency per model from the same Langfuse data (the default profile, all history):

Model	Calls	p50	Slowest
qwen/qwen3.7-flash	92	4.3 s	180.5 s
qwen3.5-64k (local, 6 GB GPU)	91	125.2 s	900.3 s
deepseek-v4-flash	65	6.7 s
mistral-nemo (local)	24	106.6 s	1,802.9 s
hermes3-64k	15	2.5 s

On the tool side, terminal ran 135 times with a median of 0.6 s, and the slowest single tool call was a search_files at 120.6 s.

None of this is shocking. A local model on a 6 GB card is slow, and cheap cloud models are fast. But "slow" and "125 seconds median, about 30 times slower than the fastest cloud option" are different sentences, and only one of them helps decide which job runs where.

Then I noticed the call counts. 91 calls for the local model. That seemed low for a system I lean on all day.

The coverage check

Langfuse held 618 events across 79 traces in four weeks. That's about three traces a day.

The agent keeps its own session database, so I counted from the other side. The last 14 days alone:

188 sessions
1,387 model calls
10 agent profiles

Tracing was enabled on exactly one of ten profiles: the default chat one, with 91 calls. That's roughly 7% of the traffic.

The untraced ones were the ones doing the work:

Profile	Model calls (14 days)	Traced
coder	522	no
sec	390	no
itadmin	172	no
writer	120	no
default (chat)	91	yes
reviewer	55	no
scout	29	no
others	small	no

The coding agent alone made more than five times as many calls as the only profile I was watching.

Why it happened

The tracing plugin is enabled per profile, and it reads its Langfuse API keys through that profile's own secret scope. That's deliberate: profile B's traces shouldn't be shipped with profile A's keys.

When I set Langfuse up, I enabled the plugin and added the keys on the default profile, and moved on. Every specialist profile then ran without tracing. No error. No warning in a log. The plugin fails open by design (no keys means its hooks quietly do nothing), which is right for a plugin, and exactly why nobody noticed.

A monitoring tool that's switched off looks identical to one that has nothing to report.

There was a second, smaller gap. One-shot jobs launched in "safe mode" skip plugins entirely, and the reply drafter ran that way because it's quicker. That one is closed now: the drafter sends its own traces through the Langfuse SDK, so it no longer depends on the plugin.

The second surprise: the disk

Langfuse v3 onwards stores traces in ClickHouse. Postgres holds users and projects, Redis handles the queue, MinIO stores blobs. So no, you can't drop ClickHouse to save resources; it is the trace store.

ClickHouse was using 6.03 GiB of disk. The actual Langfuse trace data in it was 2.2 MiB.

The rest was ClickHouse's own diagnostic logging:

System table	Size	Rows
`system.trace_log`	3.38 GiB	174.6 million
`system.text_log`	1.19 GiB
`system.part_log`	492 MiB
`system.asynchronous_metric_log`	485 MiB	949 million
`system.metric_log`	461 MiB

That's roughly 2,700 times more bytes about ClickHouse than about my agents.

Those tables aren't written once and forgotten. They're written continuously. The node this runs on keeps its guests on a single 5400 rpm HDD, and it already has a roughly 15-minute IO storm after every reboot. The Langfuse container is the one I stop to relieve it. So the trace store I'd barely been using was grinding that disk all day, recording its own profiler samples.

The fix

Tracing on every profile

For each of the ten profiles:

Copied the Langfuse keys into that profile's own env file (keeping the per-profile secret scoping).
Enabled the plugin.
Set the Langfuse environment to the profile name.

The third step is the one that matters for later. Traces carry no profile tag of their own, so without it every trace lands in one undifferentiated pile. With it, "cost and latency per agent role" is a filter in the UI rather than a spreadsheet exercise.

Then I restarted the gateway and sent a one-word probe on two profiles. Traces arrived tagged legal (on a cloud flash model) and writer (on the local qwen3.5-64k). That's the test I should have run the first time.

ClickHouse logging itself

ClickHouse lets you override server config with a file dropped into /etc/clickhouse-server/config.d/. This one removes twelve system log tables, keeps query_log with a 7-day TTL (it's genuinely useful when a query is slow), and turns the server logger down to warnings:

<clickhouse>
  <trace_log remove="1"/>
  <text_log remove="1"/>
  <metric_log remove="1"/>
  <asynchronous_metric_log remove="1"/>
  <part_log remove="1"/>
  <processors_profile_log remove="1"/>
  <query_thread_log remove="1"/>
  <query_views_log remove="1"/>
  <query_metric_log remove="1"/>
  <opentelemetry_span_log remove="1"/>
  <session_log remove="1"/>
  <latency_log remove="1"/>
  <query_log>
    <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
  </query_log>
  <logger><level>warning</level></logger>
</clickhouse>

It's mounted read-only into that directory through the compose file, and the container was recreated. Checks afterwards:

container healthy
trace_log row count unchanged over 30 seconds, so it has stopped writing
trace data intact
Langfuse web health check returns 200

One honest caveat: removing a log table from config stops ClickHouse writing to it, but it doesn't delete what's already there. The old 5.9 GiB of log tables was still on disk. I dropped those deliberately, as a separate step, rather than bundling a destructive delete into a config change. The Langfuse container went from 15 GB of disk used to 7.7 GB, with the trace data intact.

Why I want this at all

It would have been easy to conclude Langfuse was dead weight: a few gigabytes of disk for 79 traces.

But the estate runs around ten agent profiles: a chief of staff that I chat with, a coder, security, IT admin, writer, scout, reviewer, legal, analyst and a few more. They run on a mix of local models on a 6 GB GPU and cheap paid cloud models with a fallback chain. The questions I actually have about it are ones only traces answer:

Which role burns the calls? Coder, at 522 of 1,387.
Where does a slow turn's time go? Model or tool. See the 25-minute turn, which looked like a web problem and was a thinking problem.
Did the fallback fire? A cloud model failing over to another is invisible from the outside if the answer still arrives.
Local or cloud for this job? 125 seconds median against 4 to 7 seconds is a real trade, and it should be made per job, not by habit.
What does each role cost? Now that traces are tagged per profile, that's a query.

The rule I run the estate by is "scripts gather facts, models never do". Models get to summarise what scripts measured; they never go and measure it themselves. Langfuse is the fact-gatherer for the agents themselves. It just can't gather facts from agents it isn't connected to.

What the weekly report tracks now

A small script now reads the agents' own session records and Langfuse side by side, and every Monday it posts a report: per role, the turns, model calls, tokens, cost, errors and the share that was traced; per model, p50 and p95 latency and cost; how turn time splits between model and tools; the five slowest turns; and tool errors. It's built to answer these questions:

Does the coder profile's call volume hold up, and is it doing many short turns or a few enormous ones?
How often do the local models hit their multi-minute tail, and on which profiles?
How often does the cloud fallback chain actually fire, and for which model?
Which one or two jobs account for most of the cost, and would they be fine on a local model?
Does ClickHouse stay small now that it's only storing what I asked it to?

If you self-host Langfuse, check these three things

Is tracing actually enabled everywhere you think it is? Per service, per profile, per worker. Count calls from your application's side and compare with Langfuse's trace count. If one is an order of magnitude bigger than the other, like mine, you'll know.
How big are ClickHouse's system log tables? Check system.parts grouped by table. If trace_log and asynchronous_metric_log dwarf your actual data, a config.d override fixes it in a few lines.
Send a probe trace from every agent. One word, one turn, per profile, and confirm each one lands with the right tag. A plugin that fails open will never tell you it isn't there.

Postscript: it happened again

On 22 September the tracing server's address changed, and all 12 specialist agents' configs were repointed at the new one. When it moved back on 23 September, only the main config was updated. Every specialist silently stopped tracing. I found and fixed it on 24 September.

No error, no warning, nothing in a log. It's the same fail-open silence this whole post is about, and the reason point 1 above isn't a one-off check.

Shopify products.json vs anti-bot walls: 85 fashion sources, measured

Christian Anderson — Wed, 30 Sep 2026 08:00:03 +0000

I built a personal shopping agent. It runs on my homelab, gathers sale items from a list of sources once a day with two refreshes later on, and a half-hourly job checks alerts and my watchlist. The point is to tell me when something I'd actually wear is reduced in my size.

The interesting part was never the agent. It was finding out how much of the retail web will talk to a program at all. This is the measured version: 85 sources, what each gave up, and four mistakes I made reading the results.

The scoreboard

Every number here comes from one run: the daily gather on 19 September, which took 854 seconds.

Status	Sources
OK	56
Blocked	22
Parse failed	4
Reachable, nothing parsed	1
Error	2

Only 45 of the 85 returned even a single item. Eleven of the "OK" sources answered fine and had nothing on sale that matched.

The run collected 4,112 discounted products. After dropping what wasn't menswear, wasn't clothing or footwear, wasn't in stock in my size or wasn't really reduced, 941 were in my size, plus 452 where the size couldn't be checked. The biggest single drop was "wrong size", at 1,411.

And the distribution is lopsided in a way I didn't expect. 21 Shopify stores produced 3,173 of the 4,112 items, more than three quarters, from one endpoint that has been sitting there the whole time.

Shopify's `/products.json` is the quiet hero

Any Shopify storefront will hand you /products.json: paginated, JSON, no key, no rendering, no scraping. Every product, every variant, with price and compare_at_price, so "is this actually reduced" is a subtraction rather than a guess about a strikethrough in the markup.

Critically, the variants carry per-size availability. That one field is the whole reason this project works.

The other pleasant surprise was a big retailer that ships a public, search-only Algolia key in its own sale page. Its site uses that key to power its sale filters. I use it the same way, from the same page, to ask for on-sale menswear in my sizes, and on that run it returned 400 items filtered server-side. No rendering, no parsing, no load a normal visitor wouldn't cause.

The pattern: before you write a scraper, check whether the shop's own front end is already calling something cleaner than the HTML.

The hard part isn't the deal. It's the size.

Eight sources list products and prices but have no per-size stock I can read: Nike, JD Sports, Selfridges, Footasylum, Puma, Converse, Clarks and one flash-sale site. Some don't put sizes on the listing at all. On others the size selector looks the same whether a size is in stock or gone. Puma loads its size grid and inventory from a later API call, so none of it is in the HTML.

So they return items I can't confirm I can buy. A 40%-off jacket that doesn't exist in my size isn't a deal, it's an errand.

The rule I settled on: keep them in the pool, tag them size_unknown, and never alert on them. They can turn up if I go looking, but they can never interrupt me. For any notifier, the real question is the false-positive rate, and what the user does after the third one.

Mistake one: I read a 403 as a "no"

Five well-known retailers were written off early as "robots.txt disallows". I'd built the polite thing — fetch robots, honour it, move on — and five big names said no, so I moved on.

They hadn't said no. Their robots.txt returns 403 to a plain HTTP request, because the file sits behind the same edge protection as everything else. My code caught the failure, couldn't parse any rules, and fell through to "assume disallowed". Fetched the way a browser fetches it, every one of those files allows *.

Failing to read a policy is not the same as the policy saying no. If your fallback is a decision, log it as a decision ("couldn't read robots, assuming disallow"), not as a fact.

The sting in the tail: I re-tested all five properly, and all five still gave me nothing. No men's prices on one, redirects and a region picker on two, an empty page on the other two. Right answer, wrong reason, for about a day. Those are the ones that rot quietly: the outcome looks correct, so nothing makes you check the logic.

Mistake two: "blocked" is three different things

When I probed a new batch of sources, I tried each one twice: once with a plain HTTP client and once rendered in a real browser. Same honest User-Agent, same robots check, no evasion. The only difference is that JavaScript runs.

Shop	Plain HTTP client	Rendered in a browser
High-street fashion chain	connection timeout	76 prices on the sale page
Off-price outlet	not tried	287 prices
High-street chain	200, but no product grid	179 prices
Secondhand marketplace	200	192 listings
Voucher aggregator	200	230 KB of offers

A plain client would have put the first and third in the blocked column. One was just rendering its grid in JavaScript, and the other didn't answer a bare client but was perfectly happy to serve a browser.

Two others, Sports Direct and a high-street chain, returned 404 to every URL I tried. That wasn't a block at all. I was guessing their category paths wrong, and Sports Direct parses fine now from its real sale page. That's the trap: in a health table, "edge-blocked", "renders empty without JavaScript" and "you asked for a page that doesn't exist" all show up as the same red cell. Only the first is the retailer's decision. The other two are my bugs, and they'll sit in the blocked column looking like someone else's fault.

So the status note is now worth more than the count. Record why a source is red, and re-test the reds with a different client now and then.

Mistake three: Foot Locker was my bug, not theirs

On the 19 September run, Foot Locker returned 48 discounted shoes and none in my size, because my reader couldn't find per-size stock. I'd filed it with the others: size selector, no availability markers, nothing to read.

The stock was there all along. Foot Locker ships exact per-size availability in JSON embedded in the product page, with UK, EU and US sizes for each variant. My reader only looked at the page's visible elements. I fixed it on 20 September to read the embedded JSON first and fall back to the old method, and Foot Locker went from 0 to 9 in-size deals on the next full refresh.

Same day, same lesson: some European stores list shoe sizes as bare numbers like 45.5, and my parser read them as UK sizes and threw them away as wrong. "They don't publish it" and "I didn't read it properly" look identical from the outside.

Mistake four: I wrote my own guess into the outfit builder

The daily digest includes a couple of outfit ideas. Early on it put a brown hoodie forward as something I'd wear. The model hadn't made that up. My own outfit-building code assumed brown and cream were colours I wear, because I'd written that assumption in without asking. That's a question for me to answer, not a default for the code to guess.

The builder now splits the work between code and model:

Code picks the candidate pieces out of the pool, per slot (top, layer, bottom, footwear), inside my sizes. Sandals, mules and slides are excluded here, before the model sees anything.
The model chooses ids from that list and writes the name and the one-line idea.
Code validates the answer: the right slots filled, no more than one neutral, the cheaper look actually cheaper than the dearer one. If validation fails, it falls back to a deterministic pick.

The model does the part it's good at (taste, phrasing) and none of the part where being confidently wrong costs money.

Rules I gave myself

Worth stating, because the blocked column is 22 and the temptation to fix that is real:

Identify honestly in the User-Agent. No pretending to be Chrome.
No proxies, no captcha solving, no rotating anything.
A challenge page, an error page or an empty render counts as blocked. Back off 72 hours, don't retry in a loop.
Cache, and keep the number of requests per site per run small.

One retailer worked fine for about 30 page loads and then Akamai shut the door. That's the system telling you where the line is. Others sit behind DataDome and reCAPTCHA. They've decided, and the honest response is to record it and move on, not engineer around it.

Which leaves the real conclusion. The shops that were easiest to work with are the ones that already publish structured data, deliberately or as a side effect of their own front end. Everyone else is spending money on a wall, and the thing behind the wall — is this jacket in my size — is the only fact I ever wanted.

Docker homelab gotchas: docker ps called the image orphaned. It was live

Christian Anderson — Tue, 29 Sep 2026 08:00:15 +0000

My Docker host was 82% full, so I went looking for images to delete. docker system df said 6.67 GB was reclaimable. docker ps --format '{{.Image}}' printed bare SHA IDs instead of names for several running containers, which made one image, unclecode/crawl4ai, look like nothing was using it.

It was running. Healthy, up two weeks, serving traffic. I was one docker rmi away from taking out a live service on the strength of a formatting quirk.

What stopped me wasn't the image list. It was a listening port that nothing on my list owned.

for c in $(docker ps -q); do docker inspect -f '{{.Name}} {{.Config.Image}}' $c; done
ss -ltnp        # a port nobody claims is a container you mis-read

That's the pattern for most of what follows: Docker's summary views are fine until you act on them.

How I actually run it

Nothing exotic. Docker runs inside unprivileged LXC containers on Proxmox (they need nesting=1 and keyctl=1), plus a small NAS running an appliance OS whose app manager is a layer over Docker. Single-purpose things, like my voice assistant's speech-to-text and text-to-speech servers, are one docker run line each. Anything with a database is a compose stack; the LLM tracing one is six containers.

Here's what bit.

Disk: the pressure was images, not data

Four guests were 82 to 86% full, and my instinct was to push data to the NAS. Profiling disagreed. The Docker LXC had 16 GB of images. Another guest had 11.4 GB of images and 11.3 GB of build cache. There was very little cold data at all: my weekly offload job's first run reclaimed 213 MB.

You can't move a running container's image layers onto NFS anyway, since overlayfs over NFS is slow and fragile. The fix was growing the guest disks, online, from a thin pool that was 7% used.

Two more lessons from the clean-up:

Per-tag sizes are a lie for planning. I removed 16 superseded tags of my own app, each listed at about 240 MB. They freed 60 MB, because the builds share nearly every layer. The whole clean-up, dangling images and build cache included, came to about 340 MB.
An image can be dead in ps and still be a dependency. Three NetBird images weren't running, but a compose file on disk still referenced them. What I did delete had no container ever and no references anywhere: 2.8 GB.

docker image prune -a is now on my "a human types it" list. On one host 4 of the 11 images were in use, and there's no fast path to re-pull if something happens to be stopped at the time.

Logs: nobody set a limit

Another Docker host's root filesystem reached 100%, 0 bytes free, with five failed systemd units. Nothing had alerted. The cause was that /etc/docker/daemon.json didn't exist, so Docker was using the default json-file log driver with no rotation and no size cap. The Home Assistant container's log was a single 1.9 GB file.

truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log   # 1.9G
journalctl --vacuum-size=100M                                  # 286M
apt-get clean                                                  # 271M
docker image prune -f        # dangling only, reclaimed 3.47G

All four containers kept their uptimes. Then I wrote the daemon.json that should have been there all along: json-file, max-size: 50m, max-file: 3.

It didn't cap Home Assistant's log. Weeks later, the container still reports an empty log config:

docker inspect -f '{{.Name}} {{.HostConfig.LogConfig}}' $(docker ps -q)
# /homeassistant {json-file map[]}

Daemon log defaults are copied into a container when it's created, not when Docker restarts. Every container that existed before the file stays uncapped until it's recreated. The only containers on that host with a cap are two whose compose file sets its own logging: options.

Restart policy: the file said one thing, the container another

A file browser on the NAS hard-exited one morning because it couldn't reach the SSO provider. It stayed dead for six days. RestartCount=0. Nothing retried and nothing alerted.

The compose file said restart: unless-stopped. The container had been created before that line was added and was never recreated, so it was still running restart=no. I'd "fixed" this once already with docker update, and it had reverted. The second time I checked the on-disk hostconfig.json rather than trusting docker inspect. Then I audited the NAS: all 18 running containers now have unless-stopped. Five didn't, including the SSO container that every login depends on. The reverse proxy had the same drift.

That appliance layer had its own go at me. My dashboard container was uninstalled by the NAS's app manager twice, with its config left on disk both times. The first time I reinstalled from the leftover compose file. The second time that file was gone too. I don't know why it happened, so I stopped letting the app manager own it:

docker run -d --name homepage --restart unless-stopped -p <port>:3000 \
  -v <config-dir>:/app/config \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -e HOMEPAGE_ALLOWED_HOSTS=<host>:<port>,localhost:<port> \
  ghcr.io/gethomepage/homepage:v1.8.0

Leave out HOMEPAGE_ALLOWED_HOSTS on v1.x and every browser request gets a blank 400.

Configuration is read at create time

docker compose restart does not reload .env. env_file is read when the container is created. Use docker compose up -d --force-recreate.

Compose interpolates $ inside env_file values. My password manager's admin token is an Argon2 PHC string, $argon2id$v=19$m=19456,.... It reached the container as 51 mangled characters beginning =19=19456,, so the admin page returned 401 for every token, the right one included. Doubling every $ fixed it, and docker inspect showed the container getting 134 characters starting $argon2id$ . The file being right proved nothing. The env file is named vw.env, not .env, because compose auto-loads .env as its own interpolation source.

environment: beats env_file. My app's compose file pinned three service URLs in environment:, so anyone setting them correctly in their own .env was silently overridden.

Stock compose defaults can contradict each other. The tracing stack's compose file sets DATABASE_URL to a hardcoded postgres:postgres instead of building it from POSTGRES_PASSWORD. Set only the password and Postgres changes while the app keeps using the literal default:

Error: P1000: Authentication failed against database server
Applying database migrations failed. ... Exiting...

The web container restart-looped. The error mentions special characters that aren't URL-encoded, which was a red herring: the password was alphanumeric by construction. The fix is an explicit DATABASE_URL in .env, kept in step with the password.

Code baked into the image isn't live because the tests passed. My web app's code is copied in at build time, and compose mounts only the data volume. I committed three fixes, watched 661 tests pass, and both containers carried on running the previous version with the new code on disk beside them. Now I ask the container:

docker exec <container> python -c "from <pkg> import <module>; print(<module>.<CONSTANT_THE_FIX_ADDED>)"

Networking: published ports and DOCKER-USER

My web app has a private instance and an internet-facing demo copy on the same Docker host. The private one is meant to be reached only through a reverse proxy on another box, but it was published on 0.0.0.0, so the demo container could reach it directly and skip the proxy.

Binding it to 127.0.0.1 fixed that and took the site down for a day, because the proxy is on a different machine. The real fix was to publish on the host's LAN address and restrict it in the DOCKER-USER chain. The shape:

iptables -N PRIVATE-IN
iptables -I DOCKER-USER -m conntrack --ctorigdst 192.0.2.10 --ctorigdstport 8080 -j PRIVATE-IN
iptables -A PRIVATE-IN -m conntrack --ctstate ESTABLISHED -j RETURN
iptables -A PRIVATE-IN -s 192.0.2.20 -j RETURN     # the proxy
iptables -A PRIVATE-IN -j DROP

(Addresses and port are placeholders.) The proxy gets a 200 and everything else times out.

Before I added egress rules, the internet-facing demo could reach far more of my LAN than it had any business touching. Afterwards those internal services time out, while the handful it genuinely needs and the internet still answer. Getting there turned on three traps:

Matching -d never fires for published ports. By the time a packet reaches FORWARD, the destination has already been DNATed from the host address onto a 172.x container address. --ctorigdst matches what the client actually asked for.
DNS doesn't go through FORWARD. The container's resolver was the host's own mesh-VPN address, which is local, so queries land on INPUT. A mesh-range DROP in DOCKER-USER never touches DNS, which looks reassuring, but the separate INPUT -i br-... chain must allow port 53 or the container goes deaf.
The bridge name changes. It's br- plus the first 12 characters of the network ID, and recreating the network gives it a new ID. The script reads bridge and subnet from docker network inspect every run, and I re-run it after any compose down.

Docker rebuilds DOCKER-USER when it starts, so the script runs from a systemd unit with After=docker.service. It also had its own bug: the cleanup -D DOCKER-USER line left out the -s match it was inserted with, so it never deleted anything. I found 17 duplicate jumps. There's now one.

Healthy is not working

Two containers that docker ps would call fine:

My Cloudflare tunnel ran from a container that turned out to be a web UI for cloudflared, not a connector. After a recreate it logged No pre-existing config file found, showed healthy, and connected no tunnel at all. Every public hostname returned 530, SSO included. A plain cloudflare/cloudflared container with --restart unless-stopped fixed it.
The NAS has a Pascal-generation Quadro that, last I checked, falls off the PCIe bus hours after boot, so Immich's CUDA machine-learning container logs no CUDA-capable device and falls back to CPU. Immich v3 dropped Pascal support, so the moment the card is fixed that container will die with SIGILL, exit 132, and crash-loop. A working GPU is worse than a missing one until I change the image tag.

Rules I run by now

Never delete an image on the evidence of docker ps or docker system df. Inspect the containers and check the listening ports.
Check what the container received (docker inspect, docker exec ... env), not what the file says.
Changed .env? up -d --force-recreate, not restart.
Every container gets --restart unless-stopped, and I check it persisted.
If an app manager can uninstall it, anything I care about runs outside it.
Logs get a size cap on day one. A daemon.json default only reaches containers created after it, so recreate the old ones.
Firewall published ports by --ctorigdst in DOCKER-USER, derive the bridge name every run, and put the script behind After=docker.service.
"Healthy" means the process is up. It doesn't mean it's doing its job.

Discord as an AI agent UI: why chat is my primary interface, not dashboards

Christian Anderson — Mon, 28 Sep 2026 08:00:13 +0000

I have dashboards. There's one for the agent platform itself, behind single sign-on. There's a read-only dashboard for a trading experiment, one for my home-grown intrusion detection, and a Homepage start page tying the lot together. I open them when I want to look at something.

But I don't talk to my agents through any of them. I talk to them in Discord. A private server, one bot, and a set of channels. That's the primary interface, and this is how it's laid out, what it gives me for nothing, and what actually goes wrong.

How the channels route work

The main channel goes to my chief-of-staff agent, which is called Denzel. Anything I type there lands with it. It does some admin itself, and anything on a specialist's turf becomes a card on a kanban board, assigned to the right profile: infrastructure to the coder, research to the scout, security to the security profile, long-form writing to the writer.

The specialists report into their own channels. There's one for kanban updates, one for network and security alerts, one for sysadmin reports, one for dev.to, one for music, one for family holiday planning. Two of them take conversation directly: the shopping assistant and the toolsmith each have a channel routed straight to their profile, with no @mention needed. Only my account is allowed to talk to the bot at all.

Some channels double as approval queues. The toolsmith posts each tool it thinks I should trial as its own message, and I react ✅ or ❌. The SEO agent does the same with metadata changes for my posts. A reaction is about the smallest approval interface there is, and it works from my phone.

The useful property is that the room is the address. I don't type which agent I mean. Adding a specialist means adding a channel and a route.

What Discord gives a one-user setup for free

If I built this as a web app, I'd be building a list of things that have nothing to do with the agents:

Identity and auth. Discord already knows who I am. The bot only answers my account.
Push notifications to my phone and desktop, with no setup on my side.
History and search, going back as far as the channels do.
Threads. The bot can open one per conversation, which keeps a long back-and-forth out of the main channel.
Images. I can send the shopping assistant a photo of a jacket and it finds similar pieces in my size.
Mobile, off my home network. The agents reach Discord outbound. I don't open anything up to reach them from outside.

None of that is interesting to build, and all of it is necessary. That's the whole argument.

What actually breaks

Sessions cache the system prompt. A Discord session keeps the system prompt it started with. Add a skill, change an agent's instructions, rename it, and an existing session won't see any of it. The setting that should time sessions out doesn't do anything in the build I run. The fix is typing /reset in the channel.

I learned this the annoying way. I'd built a workflow for approving replies to comments on my posts, then asked the main agent to post one. The session predated the skill. Instead of saying it didn't know how, it spent 25 minutes searching the web and reading the article, and then invented a summary. Now, whenever an agent's instructions change, /reset is part of the change.

A session can get poisoned. My first real chat with the shopping assistant went badly. A simple request, a white T-shirt or a polo, took 43 minutes and failed. The session history held file paths that my cloud model provider's PII filter had redacted into placeholders, and the model kept copying those broken paths back into its own commands. Nothing wrong with the agent's code. The conversation itself was contaminated, and the only fix was ending the session and starting clean.

Voice notes are dropped. A recorded voice message in a text channel reaches my agent platform as an empty message. The audio attachment is thrown away before the code that would detect it as voice ever runs. The feature exists in the code and doesn't engage. There's a separate path for live voice channels. I haven't tried it, so I can't tell you whether it works.

2,000 characters per message. That's Discord's cap. Fine for "done" or a short list. Not fine for a daily digest. The shopping assistant's posting helper splits on line boundaries and keeps each piece under 1,900 characters, and product results go out as embeds rather than long text.

Link previews. Drop a bare URL in a message and Discord unfurls it into a preview card. My first test digest was hard to read for exactly that reason. The shopping assistant now sets Discord's per-message SUPPRESS_EMBEDS flag on its text posts, so links stay links.

No tables. A code block is the closest you get. If an agent's output is naturally a table, it has to become a list.

Someone else's server

This is the one to think about properly, and I'd rather be straight about it. My channels aren't carrying package updates and uptime pings. They carry my clothing sizes and order history, family holiday planning, and security alerts from my own network.

Discord isn't end-to-end encrypted. The identity and auth are free, but everything in those channels sits on Discord's servers. For me, a private server that only my account can post in is an acceptable trade for what it saves me. It is a trade, though, and it's worth making on purpose rather than by default. If your agents handle things you wouldn't want stored on a third party's servers, this layout isn't for you, or those agents belong somewhere else.

The general point

For a tool with one user, the chat app is the UI, and the dashboards are for looking. Everything I'd otherwise build — login, notifications, history, a mobile client, somewhere to drop a photo — already exists, so the time goes on the agents.

What I didn't expect is that the failures are about sessions, not the interface. Stale prompts, poisoned history, attachments dropped before they're processed. None of that shows up as an error in Discord. It shows up as an agent confidently doing the wrong thing, which is the thing to watch for.