DEV Community: Vineeth N K

I went on a trip. My Mac mini stayed home and kept texting me.

Vineeth N K — Sat, 30 May 2026 17:50:59 +0000

I went on a trip. My Mac mini stayed home and kept texting me.

TL;DR: A while back I built a homelab on an old 2018 Mac mini. Then I went out of town for a few days and left it running. I half expected to come back to a dead box. Instead it just kept doing its job, let me SSH in from my phone and keep pushing my own CLI tools forward while away, and buzzed me whenever something mattered. Nothing dramatic happened. And honestly, that quiet was the whole point. This is the story of the homelab finally earning its keep while I was nowhere near it.

The part nobody tells you about building a homelab

When you set up a homelab, all the blog posts stop at the setup. The screenshots are green, the containers are up, you take your victory lap and close the laptop.

I did the same. I wrote down the whole long evening of building this thing, every gotcha, every GUI click macOS forced on me. At the end I had Vaultwarden, ntfy, Uptime Kuma, n8n, a little agent webhook, restic backups, all sitting on a Mac mini that a colleague handed me from his drawer.

But here is the thing. A homelab that only works while you are sitting next to it is just a noisy space heater. The real test is the day you are not there. The day the power could flicker, a container could die, a backup could fail, and you would have no idea unless the box itself told you.

So when a short trip came up, I did not shut anything down. I left it all running and went.

Day one, and the silence was loud

First evening away, I caught myself doing the thing. You know the thing. Opening the phone to check if home is still alive, the way you check if you locked the front door.

I pulled up the status page. Everything green. Uptime Kuma sitting there with a row of happy little dots, every service responding, the agent webhook answering its health check. Netdata showing the mini idling cool and bored.

And then I just... put the phone down. There was nothing to do. The box did not need me.

That feeling is strange the first time. You build a thing for months, you babysit it, and then one day it does not need babysitting anymore. Bittersweet, almost. Like dropping a kid at hostel.

The 3:30 buzz

My restic backup runs every night at 03:30 in the morning, back home. Nobody is awake for that, which is the whole idea of a 3:30 AM cron. You set it for the dead of night precisely so it never gets in your way.

The job fired while I was fast asleep, exactly like it does on any normal night. The only difference was that this night I was not home. I woke up the next morning, picked up the phone out of pure habit, and there it was waiting on the lock screen. ntfy notification. Backup done, snapshot pushed, a few MB in, almost nothing out after dedup.

A tiny push telling me my data was safe, fired by a machine sitting alone in an empty flat, patiently waiting for me to wake up and read it. I did not do anything. I did not even open the app fully. I just saw it, nodded, and went to find coffee.

That little buzz is the entire reason I wired ntfy in the first place. Not to spam me. To tell me the boring good news so that the day it becomes bad news, I notice immediately. A backup that runs silently is a backup you do not trust. A backup that texts you "done" every night is one you forget about, in the good way.

If you have ever felt a small flush of pride at a green cron job, you and I would get along just fine.

The actual work happened from my phone

Now the part I am quietly proud of.

Here is what surprised me. The trip was not me firefighting a homelab from a hotel room. The box was calm the whole time. What I actually did was use the days to work on some cool side stuff and refine a few of my own personal CLI tools, straight from my phone.

The trick is nothing fancy. Remote Login is on, the mini is on my tailnet, so I open an SSH app on my phone and I am in a real shell on the machine back home. Not a watered-down dashboard, the actual terminal, with my dotfiles, my aliases, my tools, all sitting exactly where I left them. From there I run whatever I want, claude included, and do real work.

# from the phone, over Tailscale
ssh mac-mini
# and then just... work, same as if I was at the desk

So the rhythm of my day became this. Find a quiet half hour, SSH in from the phone, run a command, kick off a change to one of my CLI tools, read the output right there on the small screen, run the next one. Tiny keyboard, yes, and I am not going to pretend a phone replaced my full setup. But for steadily nudging a few personal tools forward, command by command, it genuinely worked. I came home with actual progress, not just a tan.

And yes, the homelab also has that agent webhook. But that one is built for a different job, automating the repetitive tasks from my daily work, where I fire a prompt and let the mini run it on its own and ping me the result. The trip work was the hands-on kind, just done through a very small keyboard.

Nothing went wrong, and that was the point

Here is the anticlimax. The dashboard stayed green the entire time.

No service fell over. No 3 AM page. No frantic debugging from a six-inch screen. Uptime Kuma just sat there with its happy row of dots, day after day, and the only buzzes I got were the friendly kind, backup done, agent result ready.

And I want to be clear that the quiet is not a boring detail to skip past. The quiet is the product. The point of all the monitoring was never to give me a dramatic save story. It was so that if anything did go red, I would know within a heartbeat instead of finding out days later, back home, staring at a dead service with no idea how long it had been gone. I had recovery alerts wired alongside the down alerts too, so a blip would have buzzed me twice, once for the scare and once for the all-clear.

It just never had to. And honestly, a homelab that gives you a boring trip is the homelab working exactly as designed.

The thing I was most nervous about

Power.

The one fear I could not fully shake was a power cut at home while I was away. If the mini went down and stayed down, my whole little world would go dark and there would be absolutely nothing I could do about it from out of town.

So I had stacked two layers of insurance for exactly this.

The first is a power backup. The mini sits behind a UPS that can keep it running on its own for a good six to eight hours. Most power cuts where I live are the short, annoying kind, gone and back before you finish complaining about them. The UPS swallows all of those without the mini ever noticing a thing.

The second layer is for when a cut outlasts the battery, or when power drops and returns while I am away. Back during setup I had told macOS to bring itself back on its own.

sudo pmset -a autorestart 1   # come back on your own after a power cut
sudo pmset -a sleep 0         # and never, ever go to sleep

autorestart 1 means if power drops and later returns, the Mac boots itself without anyone pressing the button. Colima starts on boot through launchd, the containers come up with restart: unless-stopped, Tailscale reconnects on its own, and the whole stack reassembles itself like nothing happened.

Between the two, the only way I genuinely lose is a power cut that runs longer than the battery and then never comes back for the rest of the trip. That is the real dark side, the one scenario where there is nothing left to do but wait until I am home. But it is a narrow window now, not the wide-open fear it used to be. And knowing that let me actually enjoy the trip instead of refreshing a status page every hour. A homelab you have to worry about is not a homelab, it is a pet that bites.

What this trip actually taught me

I came back home, walked in, and the mini was sitting there with its little light on, exactly as I left it. No drama, no recovery saga, no horror story. It had just quietly done its job the entire time.

And that is the lesson. The point of all that setup, all those gotchas and GUI clicks and one-word Caddy fixes, was not to have a pretty dashboard. It was to be able to leave, fully, and trust the thing to behave and to speak up only when it mattered.

A few things made that trust possible, and if you are building your own, these are the ones that earned their place:

ntfy for the boring good news, not just the bad. Let it tell you the backup worked. The day it says the backup failed, you will already be in the habit of reading it.
Tailscale so the box is in your pocket. Everything reachable like it is on localhost, from anywhere, no ports open to the internet. That single choice is what makes the phone a real remote control.
Uptime Kuma with recovery alerts on too. Wire both the down and the all-clear, so the day something blips you get the relief buzz right after the scare, not just the scare.
pmset autorestart for the power fear. You cannot fix a dead box from another city. So make sure it un-deads itself.
Plain SSH from the phone, over Tailscale. This is the one that surprised me. A real shell on the home machine, my own tools and dotfiles, reachable from a phone anywhere. It turned dead travel time into actual progress, command by command.

The homelab stopped being a project the day I could walk away from it. Funny how you only really finish building something when you stop having to look at it.

So tell me, what is the one thing your setup does while you sleep that quietly makes you trust it? I am genuinely curious, because that small thing is usually the whole game.

Right, I am off to check my phone for no reason again. Old habits. Take care of your machines, and they will take care of you back.

Building vaultctl: the password vault where my own server can't read your passwords

Vineeth N K — Sun, 24 May 2026 14:14:40 +0000

Building vaultctl: the password vault where my own server can't read your passwords

Every password manager forces the same uncomfortable question: do I trust this server with my actual passwords?

For most vaults, cloud or self-hosted, the honest answer is "yes, you have to". The server holds the keys. It can decrypt your data. Cloud just makes it worse because the server is not even yours.

vaultctl is what I built when I stopped wanting to make that trade. Self-hosted password vault, but the server has no code path to decrypt anything. Encryption happens in the browser, the extension, or the CLI, before the bytes ever leave the client. If someone walks off with the database tomorrow, what they get is noise.

This is another post in the series where I walk through my open-source projects. Earlier ones covered backupctl, agent-sessions, and mcp-pool.

How I got here

I gave LastPass a real try in the early days, only with throwaway logins. Good thing too, because a long year ago LastPass got breached. Nothing of mine was in it, but the message was loud enough. If the biggest name in cloud SaaS could leak vaults, the whole category had a trust problem I was not willing to live with. Your vault is sitting on somebody else's server. The day it gets breached you find out the same time everyone on Twitter does.

So I went self-hosted, and the whole picture changed. No third-party operator to be breached, no support team to be socially engineered, no vendor that gets acquired and changes its terms next quarter. The blast radius shrinks to "my one server, in my own house". Self-hosted is the safer category. Full stop.

But self-hosted is a spectrum. Closed-source self-hosted vaults are a non-starter for me, because a password is the most sensitive thing on my machine, and if I cannot read the binary that touches it, I cannot tell what it does with the master password in memory, whether it phones home for "telemetry", or what the next update is going to change without telling me. For most software, closed-source self-hosted is fine. For something that holds your passwords, credentials, API keys, SSH keys, recovery phrases, and whatever other secrets you put in it, it is not. A vault these days is a lot more than just passwords.

So I went open-source self-hosted, and that was much better. I could read the code, pin the version, trust the boundary. But every time I looked at the schema, one detail kept bothering me. The server is sitting on the keys that decrypt my stuff. Self-hosting protects me from somebody else's incident. It does not protect me from my own.

That was the missing piece. A vault where even my own server cannot read the data. If I was going to be the one trusting it, I might as well be the one making the calls.

That is the itch that became vaultctl. Still self-hosted. Still open-source. Just with one extra property bolted in at the bottom: the server itself, even mine, cannot read your data. Ever.

Why build it when Vaultwarden exists

Fair question. Vaultwarden is excellent and gets most of the way there. But it is a re-implementation of someone else's server protocol, which means the day I want to change how sharing works, or what the rekey path looks like, or whether a particular field is even allowed to hit the server, I am a guest in someone else's house. I either patch upstream or maintain a fork. vaultctl is what I built when I wanted to be the host of my own design instead.

What vaultctl gives me that I could not get off the shelf:

No decrypt(...) function on the server. Not by policy, by absence. Grep the source. The keys to decrypt user data do not live on the server side of the wire. Pop the server, leak ciphertext.
Member-removal does a full vault rekey. When you remove someone from a shared vault, every item gets re-encrypted under a fresh key. Anything they kept a copy of is no longer good for new data.
Ed25519 signature pinning on every public key. Even if my server is compromised and tries to hand a client an attacker-controlled public key during a re-wrap, the client refuses because the signature does not check out.
One Go binary, three clients. Web SPA, CLI, MV3 browser extension. All non-privileged, all hitting the same JSON API, all sharing the same crypto module. No client is treated special, no "admin" endpoint exists.
One-file install. The React SPA and the SQL migrations are embedded into the Go binary. docker compose up -d, migrate up, you are running on a ~45 MB image. No init containers, no nginx in front, no "remember to copy the static folder".
Verifiable releases. Cosign signatures, CycloneDX SBOMs, SLSA-L3 provenance attestations on public releases. For something you put your credentials into, this is the bare minimum.

The shorter version: I had opinions about how a vault should behave that I could not get on someone else's roadmap. Building it myself was the cheaper move.

The design bet: a constraint, not a feature

Zero-knowledge is not a feature you add. It is a constraint you accept, and then it quietly forbids a lot of code you would otherwise write.

The first time I felt this was when I sat down to design the "forgot password" flow. Of course there should be one. Every app has one. About thirty seconds in, the whole thing collapsed in my head. If the server can reset your password without your master password, then the server can derive the keys that decrypt your data. Which means the server can decrypt your data. The whole premise becomes a lie.

So there is no forgot password flow. There is a Recovery Kit, shown once at registration. If you lose both that and your master password, the data is gone. Not "gone until support". Gone.

Once I accepted that, the rest of the questions answered themselves. No server-side search on titles, the server does not know the titles. No "admin can see what is in here" rescue path, if the admin can rescue, the admin can read. No "store this as plaintext just for now" anywhere in the codebase.

The other big bet was Go. Honest reason: I write a lot more Go than Rust, and side projects that need me to re-learn the language between sessions do not get shipped. The technical reasons (single static binary, stdlib crypto, easy to read) helped, but they were not the deciding ones.

What it actually does

Self-host with docker compose up -d. One Caddy in front of one vaultctl container in front of one Postgres. The vaultctl image is ~45 MB on a distroless base. The React SPA is built once and embedded into the Go binary with //go:embed. SQL migrations are embedded too. The whole product ships as one file.

Register the first user, save your Recovery Kit, you are in. The browser derives your master key with Argon2id, then derives an auth hash to prove who you are to the server and a separate key to encrypt your data. The master password never leaves the browser. The server stores the auth hash, the salt, and a pile of ciphertext blobs.

Add a login and the client encrypts the title, URL, username, and password with the vault's symmetric key using AES-256-GCM. The server stores the ciphertext. Done.

Same flow from the CLI:

export VAULTCTL_API_URL=https://vault.example.com
vaultctl login
vaultctl add login --name Reddit
vaultctl get Reddit

Same from the browser extension. Same from the SPA. None of the three clients is privileged. They share the crypto primitives, hit the same JSON API, and the server treats them identically.

The hard parts

Actually enforcing "the server cannot decrypt"

Anyone can write a README saying "encryption happens client-side". The interesting question is whether you can prove it. The way I enforce it is structural. There is no decrypt(ciphertext, key) function anywhere in the Go codebase. None. Grep for aes.NewCipher or Open( and you will not find one that touches user data. The keys to decrypt do not exist on the server side of the wire.

The thing that bit me was the audit log. First version happily wrote "user X added item Y to vault Z, named GitHub root token" in plain English, sitting in a Postgres column. I opened the table in psql to admire my work and just sat there going, oh no. I had built a system whose whole pitch was "the server cannot read your data", and built right next to it a log that recorded exactly what was in your data in cleartext.

Rewrote it the same evening. Audit messages are templated client-side, the rendered string is encrypted before it goes in.

Zero-knowledge is a property of the entire pipeline, not a feature of the encrypt button.

The member-removal rekey, and the Ed25519 pin

A team vault has a symmetric key. Every member has that key wrapped with their RSA public key. When I share with you, I unwrap my copy, re-wrap with your public key, and the server stores the new wrap.

Now I remove you. Naive answer: delete your wrapped copy. Done.

No. You already had the key. You used the vault. Nothing stops you from caching the unwrapped symmetric key on your machine. Deleting your wrap does not unlearn bytes. If you kept a copy, you can decrypt every item in that vault forever.

The only real answer is a rekey. Every remaining member's client re-encrypts every item with a fresh symmetric key, then re-wraps that new key for each remaining member. A coordination nightmare in a transactional unit. I burned a lot of time getting batching and idempotency right.

But the real surprise was quieter. When you re-wrap for another member, you fetch their public key from the server. Which means the server gets to tell you which public key belongs to that user. If the server is malicious or compromised, it can hand you a key it controls instead of the real one. You happily wrap the new vault key for "Alice", and what you actually did is hand the server the keys to the vault.

The fix is signature pinning. Every user has a second keypair, an Ed25519 identity key, generated at registration. They sign their RSA public key with it. When another client fetches that public key, it also fetches the signature and verifies it against the identity public key on file. No valid signature, no wrap.

The trust shifts from "the server told me this is Alice's public key" to "Alice told me this is Alice's public key, at registration time, mathematically attached to the bytes." The day I got that flow right end-to-end was the day vaultctl stopped being a single-user toy.

One Go binary, three clients

I wanted three clients sharing one server and one set of crypto primitives, with one binary to deploy.

cmd/server/ is the entry point and does everything. vaultctl server runs the API. vaultctl migrate up applies the embedded SQL migrations. vaultctl login / add / get / ls are the CLI. One binary, multiple subcommands, dispatched through Cobra. The CLI is just another HTTP client hitting the same JSON API the SPA uses. No privileged "internal" endpoints.

The SPA is where embed earned its keep. make web-build runs vite build into web/dist/. A small Go file does //go:embed dist/* and folds it into the binary at compile time. The HTTP handler serves the SPA from the embedded filesystem. No second container, no nginx fronting a static folder.

The browser extension is the only piece outside the Go binary, because it has to ship through the Chrome and Firefox stores. But it uses the same JSON API and the same crypto primitives the SPA does, through a shared TypeScript module both web/ and extension/ import.

Fresh install: docker compose up -d, migrate up, done. No build steps on the target. No init scripts. For something whose whole job is to be the most boring piece of infrastructure in your life, that simplicity is the point.

Where to go from here

If any of this is useful, the whole thing is open source:

Code: github.com/vineethkrishnan/vaultctl
Docs: vaultctl.vinelabs.de
License: AGPL-3.0

I would genuinely like to be wrong about something here before someone trusts it with their AWS root key. If you read the code and find a hole, please open an issue.

So that is where I will stop. If you have a different way of doing this, I genuinely want to hear it. Drop me a note.

the slow request my APM never told me about

Vineeth N K — Sat, 23 May 2026 13:07:56 +0000

the slow request my APM never told me about

TL;DR: one specific user was loading a page in six minutes while everyone else loaded it in under a second. The APM dashboard had nothing on it. Turns out APMs drop requests that run too long, so the worst requests on your system are exactly the ones the dashboard cannot see. I learned to attach a sampling profiler to the live worker and read its stack directly. Different tool, different rules. Saved my week.

the part where the dashboard goes quiet

So there I was, staring at the APM dashboard, watching it lie to me by omission.

One user was reporting a six-minute page load. I could reproduce it. I could see it in the browser network panel. I could SSH into the box and watch one worker peg a CPU for the full duration. The thing was happening, on the box I was logged into, while I was watching.

And the APM had nothing. No trace. No slow query. No outlier transaction. Just the same boring p95 numbers I had been looking at for weeks.

For the longest time I assumed I was holding the APM wrong. Surely there is a filter, a date range, a sample rate dial I am missing. Surely the slowest request on the entire system is not the one piece of data we cannot pull up.

Turns out, kind of, yes.

why APMs cannot see your worst request

Most APMs work by hooking every function entry and exit in your runtime. The agent collects the data, buffers it, then uploads a full trace to the dashboard at the end of the request. This is wonderful when it works, because you get exact per-function timings. But it has two limits that matter the moment things get really bad.

One, the agent has to sub-sample. If you traced every request, the overhead would eat your servers. So most APMs trace some percentage of requests and aggregate the rest into thin metrics. On the box I was debugging, the sample rate was 25 percent. One in four. Even with no other problem, a single user would need to click four times on average before one of their requests got fully captured. They had clicked many more times than that. Still nothing.

Two, there is a hard cap on trace duration. Past some threshold (about five minutes on the APM I was using, possibly configurable, possibly not, I never got a confident answer), the trace is dropped during upload. The reason is sensible. A six-hour PHP request would generate a trace the size of a small novel and the upload pipeline would buckle. So the APM protects itself.

But here is the cruel irony you only notice once you are in this situation. The slower a single request is, the less likely your APM is to tell you why. Slow requests are exactly the ones that get dropped. Slow requests are exactly the ones you most need a trace for. The dashboard cannot help you with the problem that needs help the most.

Happened to you too, right? You go looking for the worst request in the system and the tool that was supposed to find it for you just shrugs.

what a sampling profiler does differently

This is the bit where someone smarter than me says "just use a sampling profiler". I had heard about them before. I had never actually reached for one. So I went and read what they do.

A sampling profiler is not the same shape as an APM. It does not instrument anything. It does not hook function entries. It does not even know about HTTP requests. What it does is sit outside your runtime, attach via the same syscall a debugger uses (ptrace on Linux), and read the worker's memory directly. Specifically, it walks the language VM's call stack from /proc/PID/mem at whatever rate you ask for. Ninety nine snapshots a second is a comfortable default.

That is it. That is the whole tool. A loop that goes "read stack, write to file, sleep, repeat".

The implications are what surprised me. Because it never hooks anything, there is zero overhead on requests you are not sampling. Because it does not buffer a trace, it does not care how long the request takes. A six-hour request, a six-minute request, a six-millisecond request, it all looks the same to the sampler. Read stack. Write to file. Sleep.

For PHP this tool is called phpspy. I built it from source on the box, picked the worker that was clearly burning CPU on the bad request, attached the sampler for a minute, and pulled the output back.

the wrong turn I have to tell you about

This is the part where I make myself look bad, but the lesson is worth the embarrassment.

My first aggregation came back saying "ninety seven percent of samples contain function X in the stack". X was a middleware that runs on every request. I read this as "X is the bottleneck", wrote a fix for X, deployed it, measured, and the request was still six minutes.

I felt stupid for a beat. Then I felt stupider when I went back and actually read the data.

Of course ninety seven percent of samples contained X in their stack. Middleware runs on every single request. It appears in one hundred percent of samples by definition. The "ninety seven" was only "not one hundred" because the worker was sometimes between requests. I had aggregated by "does this function appear anywhere in the stack" and treated that as a signal. It was not a signal. It was a property of how requests are built.

The actual signal in a sampling profiler is the leaf of each stack. That is, the function the worker is currently executing at the moment you sampled it. Not "anywhere in the stack". The very bottom. The leaf aggregation had already been telling me the truth in the same output. I just had not read it. Function X was at the top of the call chain; the actual hot code was six frames deeper, sitting at the leaf, plain as day, in the same dump I had been staring at.

Tell me I am not the only one who has done this. You get a dataset, you skim it, you grab the biggest number, you start typing. The biggest number was not the right number. Slow down and read the rest.

the right shape, after the dust settled

Re-ran the sampler. Aggregated by leaf this time. Sliced one more time at a mid-stack depth to see what was calling that leaf, because sometimes the same leaf gets reached from very different parents and you need to know the parent to understand the fix.

Within minutes the picture was unambiguous. Specific function. Specific loop. Specific reason it was running ten thousand times for that one user and twice for everyone else. From there it was just engineering.

The whole thing, beginning to end, took less than half a day once I had the right data. The two days before that, where I was poking at the APM and writing fixes for the wrong layer, those were the expensive days. Not the debugging itself.

the bigger thing I learned about my tools

After it was done I sat with the question for a bit, because I do not love being surprised by my own tooling.

The reframe was that APMs and sampling profilers are not the same tool with different names on them. They are different shapes for different problems.

The APM is for "across thousands of requests over weeks, where am I generally slow". It is for the aggregated view. It has to sub-sample. It has to cap trace size. It is wonderful at telling you that your checkout endpoint has a creeping p95. It is bad at telling you why one specific request you can reproduce right now is six minutes long. Those are different jobs.

The sampling profiler is for "this one process is doing something I do not understand, right now, while I am watching". It does not know about requests. It does not aggregate over weeks. But it does not care about duration, sample rate, trace size, or any of the other constraints the APM has to respect. It will happily sit next to a runaway worker for an hour and tell you exactly which line of code is on fire.

You need both. You reach for them at different moments. The hard part is noticing which moment you are in, because by default we all reach for the dashboard, because the dashboard is what we are used to.

so what does this look like outside PHP

The shape is general. Most runtimes have a sampling profiler that nobody tells you about until you are in trouble.

Node has node --inspect plus Chrome DevTools for live attaching, and the broader Linux ecosystem has eBPF-based profilers like Pyroscope and Parca that work on any process.
Python has py-spy, which is basically the same idea as phpspy, written for the Python VM. Same trick. Same syscall. Different language.
The JVM has async-profiler and a dozen other things, mostly because the JVM crowd has been doing this for longer than the rest of us.

The point is not the specific tool. The point is to know it exists in your stack before you need it. Knowing about py-spy when your Python app is stuck is the difference between a four-hour incident and a four-day one.

what I now do differently

Two things changed in how I work after this.

First, when the APM says nothing, I do not assume the problem is small. I assume the APM cannot see it. There is a difference. The first reaction makes you close the ticket. The second one makes you reach for the right tool.

Second, before I write any fix off the back of profiler data, I check what I am aggregating by. Leaf function. Caller of leaf. Time spent inside, not "appears anywhere". The shape of the question decides the shape of the answer, and if you ask the wrong shape you will get a confident answer that is also wrong.

Not going to pretend this was a perfect writeup. But if even one part of it helped someone avoid the headache I went through, then it was worth putting down. See you in the next one.

Pointing vinelabs.de at my Mac mini through a Cloudflare Tunnel

Vineeth N K — Wed, 20 May 2026 14:42:45 +0000

{/*
HERO IMAGE PROMPT - paste this into your usual image tool, then delete this comment block:

Prompt:
A small 2018 Mac mini sitting on a wooden desk, a translucent orange tunnel arching out of it through a soft pastel cloud, with a tiny green vine leaf hovering near the cloud, connecting to a small browser window with a green address bar on the other side, editorial illustration style, warm pastel colors, clean linework, gentle homelab vibe, no text labels.

Dimensions: 1200 x 630 (1.91:1, OpenGraph / social card ratio, matches the other hero images)
Save as: public/blog/cloudflare-tunnel-hero.png
*/}

Pointing vinelabs.de at my Mac mini through a Cloudflare Tunnel

TL;DR: I have a domain, vinelabs.de, and a Mac mini at home running my homelab. Until recently those two were strangers. A Cloudflare Tunnel turned out to be the simplest way to connect them - no port forwarding, no certbot, no public IP gymnastics. Two services ended up behind the tunnel, three stayed inside the Tailscale tailnet because they have no business being public. This post is the why and the how.

Two things that needed to meet

In one corner, vinelabs.de. A domain I bought one weekend after realising my npm and Composer publishes deserved a proper identity, not my personal GitHub handle. It has been sitting there with a small landing page and not much else.

In the other corner, my Mac mini homelab. Vaultwarden, Uptime Kuma, ntfy, n8n, an agent webhook that runs Claude Code, all behind Tailscale and Caddy, with restic backups going to Backblaze B2. Reachable from anywhere I am logged into my tailnet, which is fine for me, but not great when I want to share a status page with a friend who is not on Tailscale.

The obvious bridge was a Cloudflare Tunnel. I had been chewing on it ever since I wrote the homelab post. So I sat down one evening and finally did it.

Why a tunnel and not a forwarded port

Quick context for anyone new to this. The "normal" way to expose a service from your home network to the internet is to log into your router, forward a port from your public IP to the box running the service, and pray that your ISP is not putting you behind CGNAT. Some do. Mine kind of does.

A Cloudflare Tunnel flips that around. The cloudflared daemon on the Mac mini opens an outbound connection to Cloudflare's edge and holds it. When a request hits status.vinelabs.de, Cloudflare sends it back down that already-open connection. The Mac mini then talks to the local service on 127.0.0.1:3001 and ships the response back the same way. Nothing is listening on a public port at home. The router has no idea any of this is happening.

The wins are real.

No port forward, no router config, no UPnP.
It works through CGNAT, because the connection is outbound.
TLS is terminated by Cloudflare with a real cert, automatically.
I can add Cloudflare Access on top if I want to gate a service with email-based auth, with zero code changes.
Free for personal use.

The tradeoff is that all public-side traffic now goes through Cloudflare. I am okay with that for a few hobby services. For something that actually mattered, I would think harder.

What gets a public URL, and what does not

This was the decision I sat with for a bit before writing a single line of config. Not everything in my homelab deserves a public hostname. The rule I settled on is simple. If a service has strong authentication and a sensible signup story, it can sit on a tunnel. If its only protection is "nobody knows the URL" or "the topic name is the password", it stays inside the Tailscale tailnet, full stop.

That gave me two short lists.

Behind the tunnel, on vinelabs.de:

home.vinelabs.de, a small landing page on the apex. Public, read-only, harmless.
status.vinelabs.de, the Uptime Kuma instance. The visitor-facing dashboard is read-only and useful to share, and Kuma has real authentication on the admin side.

Tailnet only:

Vaultwarden. This one is obvious, it is my password vault. Even with Vaultwarden's solid auth, the math is "what is the upside of a public URL for a password manager". Roughly zero. Tailnet only, forever.
n8n. This one took a beat to articulate properly, so let me. n8n is a workflow automation tool, which is innocent enough on the surface. The catch is that n8n can execute arbitrary code, hit any third-party API, and stores OAuth tokens and API keys for every service my flows talk to, Gmail credentials, Slack tokens, GitHub PATs. Even if I added basic auth in front, the blast radius if anything ever went wrong on the auth layer is too high to think about. An automation engine plus a wallet of credentials sitting behind one login screen is not a thing I want a public URL for. Tailnet is its perimeter, period.
ntfy. ntfy is a push-notification service. Its security model is, the topic name is the secret. If you know the topic name, you can publish to it (which means push to my phone), and you can also subscribe to it (which means read every notification I receive on that topic). The whole utility of ntfy depends on the topic name staying private. Putting the server on a public URL means anyone scanning the internet could guess topic names brute-force-style, and topic names are not bcrypt-hashed passwords, they are just strings. Inside the tailnet, the only devices that can even reach the ntfy server are mine. The model holds. Outside, it leaks.

If you self-host even a couple of things, you probably know this feeling. The moment you ask "okay, do I actually want this thing reachable from a coffee shop in Frankfurt", and your honest answer is no. Three services on this box answered no. So the tunnel had two real hostnames plus a catch-all, and that was it.

Setting up the tunnel itself

The setup was honestly easier than I expected. A handful of commands and one YAML file, and that was it.

brew install cloudflared

# Sign in to Cloudflare (opens a browser to authorize the account)
cloudflared tunnel login

# Create the tunnel itself, which also drops a credentials JSON in ~/.cloudflared/
cloudflared tunnel create vinelabs-mini

# Wire up DNS for each public hostname (creates a proxied CNAME)
cloudflared tunnel route dns vinelabs-mini home.vinelabs.de
cloudflared tunnel route dns vinelabs-mini status.vinelabs.de

# Run it
cloudflared --config ~/.cloudflared/config.yml tunnel run vinelabs-mini

The tunnel route dns command is worth flagging. It calls Cloudflare's API directly, creates the proxied CNAME pointing at your tunnel, and is idempotent. If the CNAME already exists pointing at the same tunnel, it leaves it alone. If it exists pointing somewhere else, it tells you, instead of silently overwriting. So you never need to touch the DNS dashboard for the happy path. Worth committing to muscle memory early, it saves headaches down the line.

For persistence I wrote a tiny launchd plist so cloudflared starts at boot and comes back up if it ever crashes. Nothing fancy in it, just RunAtLoad and KeepAlive and the daemon sorts itself out.

The config.yml that runs the tunnel

The file that actually defines what the tunnel does lives at ~/.cloudflared/config.yml. Here is what mine ended up looking like.

tunnel: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
credentials-file: /Users/vineeth/.cloudflared/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.json

ingress:
  - hostname: home.vinelabs.de
    service: http://localhost:80

  - hostname: status.vinelabs.de
    service: http://localhost:3001

  # Catch-all (required)
  - service: http_status:404

Three things worth calling out for anyone new to cloudflared config files.

The tunnel ID and credentials file at the top are how cloudflared knows which tunnel it is and which credentials to use when it phones home to Cloudflare. You get both from cloudflared tunnel create.

Each ingress rule has a hostname and a service. The hostname is the public name a request comes in on. The service is where cloudflared should forward that request locally. http://localhost:3001 is Uptime Kuma running on the Mac mini. Plain HTTP is fine here, because the connection is from cloudflared to a service on the same box, never over the wire.

The last rule has no hostname. That is the catch-all, and it is required. cloudflared will refuse to start without one. If a request comes in for a hostname that does not match any earlier rule, this is where it lands. I use http_status:404, because that is the truthful answer for an unknown hostname. The point is, you need a fallback. Without it, the daemon does not run.

That is the whole tunnel config. Two public hostnames, one catch-all, twelve lines of YAML. Less than I expected when I started.

Watching it actually work

Once cloudflared was up and the DNS records were in place, I hit https://status.vinelabs.de from my laptop, off Tailscale, on a regular consumer connection. Got the redirect to Uptime Kuma's dashboard, served via Cloudflare. Exactly the response I was hoping for.

Real cert, real public URL, real public internet, and not a single port forwarded on my router. The Mac mini just kept doing its thing. Cloudflare did the rest.

home.vinelabs.de came up the same way. Two for two.

Where to from here

The bones are in place. What I want to add on top, roughly in this order.

First, a small phone shortcut that POSTs to the agent webhook through its own tunnel, fronted by Cloudflare Access so only my own Google account can hit it. Then the n8n flows will start consuming status.vinelabs.de as their uptime source, instead of polling each container directly. And somewhere down the line, a real landing page on home.vinelabs.de that lists the projects living under the vinelabs-de org, instead of the placeholder it has right now.

The domain finally has a job. The Mac mini finally has a face. The two of them are talking through a quiet outbound connection that my router never even noticed.

Okay, that is enough from me for today. If any of this saved you the evening of router-port-forwarding pain I used to do, that is the whole point of writing it down. Until the next one, take it easy.

My Raspberry Pi could not carry the dream. A 2018 Mac mini could.

Vineeth N K — Mon, 18 May 2026 13:50:57 +0000

My Raspberry Pi could not carry the dream. A 2018 Mac mini could.

TL;DR: I had a Raspberry Pi running a couple of small things at home. It was a good guy for small things. But I wanted a homelab that hosts my self hosted apps and also runs a small army of agents on my command, 24x7. The Pi would just die. A colleague had a 2018 Mac mini sitting unused, I told him what I had in mind, he handed it over without thinking twice. I wiped it clean, took it from Big Sur to macOS 15 Sequoia, installed Ghostty and my usual terminal things, then sat down after work and did not stop until past midnight. Vaultwarden, Uptime Kuma, ntfy, n8n, an agent webhook that runs Claude Code, all behind Tailscale and Caddy, with restic backups going to Backblaze B2. This post is the story of that one long evening, including the four or five places I got stuck.

The Pi was never going to do it

I have been running a Raspberry Pi 4 at home for a while. It was doing simple things, Pi-hole, a couple of cron jobs, a tiny dashboard. For that kind of work the Pi is perfect, no complaints.

But I had a different dream sitting in my head for some time. I wanted to run my own little army of agents. Not just one assistant on my laptop, but a setup at home that listens to me from anywhere, runs Claude Code on a prompt, sends the result to my phone, and goes back to waiting. On top of that I wanted to host my own password manager, my own push notifications, my own uptime checks, my own workflow tool. Basically the things I keep handing over to free tiers of various SaaS, brought home.

Dreams have wings like a phoenix and they want to fly sky high. But reality has weight. The Pi was not going to lift it. 4 gigs of RAM, one SoC, no real headroom. The moment I add Docker and a couple of containers and an agent that spawns subprocesses, the poor guy is on the floor.

So the dream was on hold.

When the Mac mini arrived

A colleague of mine had a 2018 Mac mini sitting at his place, unused. I knew about it. One day we were talking and I told him exactly what I wanted to build with it. The agent army, the homelab apps, the whole thing. He did not even think twice. He said take it. A great guy, no hesitation, no hold back. I owe him a proper dinner.

The mini arrived. Spec is 2018 spec, so by 2026 standards it is a baseline machine. 8GB RAM, 128GB SSD, Intel inside. For my homelab that is more than fine. The Pi would have died running half of it, this one will not even sweat.

It came with macOS 11 Big Sur on it. I wanted at least macOS 13, ideally the latest. Either way I was going to wipe the whole disk and start fresh, so the old OS did not matter.

Wiping it clean

First try was the usual one. Hold Cmd-R during boot to go into Recovery. The Mac mini just kept booting into Big Sur as if I had not pressed anything. Tried it a couple of times. Same result.

So I went to Internet Recovery instead. That is Cmd-Option-R (Cmd-Alt-R on the same key) held during boot. The screen showed a spinning globe instead of the Apple logo, the Mac fetched the recovery image straight from Apple's servers over Wi-Fi, and after a few minutes I was in the proper recovery utility. Slower, but it works when the local recovery partition decides it does not want to talk.

From there it was Disk Utility, wipe the internal SSD completely, no traces of the previous owner. Then Reinstall macOS from the same recovery menu, let it pull Big Sur back over the internet.

Once Big Sur was back on a clean disk, I went to Software Update and walked it all the way up to macOS 15 Sequoia. Big update, two reboots, no drama.

Happy face. Clean disk, latest OS, ready to be a server.

First, the terminal niceties

Before anything serious I do my usual terminal ritual. I install Ghostty as my terminal, pull in my dotfiles, get the same prompt and aliases I am used to on every machine. If you want the long story behind that 350-line .zshrc and the 68-line .bash_aliases time capsule it grew out of, that whole archaeology is in its own blog. About 20 minutes of dotfile-shuffling later, the Mac mini felt like my own machine.

Why Ghostty here, when I run iTerm2 on the Air

A small detour worth making, because people ask. On my MacBook Air, the one I use every day for actual work, my terminal is iTerm2. Has been for years. The reason is simple, iTerm2 is rich. Profiles per project with their own colors and fonts, split panes that can each be a different profile, hotkey window I can summon with a keystroke, instant replay of past output, shell integration that knows where a command starts and ends, the toolbelt, the status bar, badges, triggers, captured output, password manager, semantic history. Eleven years of polish piled into one app. For a working laptop where I am juggling four tabs of SSH plus a Vim plus a long-running build, that pile of features earns its keep every day.

On the Mac mini, the calculation flips entirely. This box is a server. I am not going to sit in front of it for eight hours debugging Rust. Most of the time it does its job with no terminal open at all. When I do open a terminal here, it is for a quick check, a docker compose ps, a tail -f, maybe a restic snapshots. Short visits, not full work sessions.

For that shape of usage, Ghostty wins on three things.

It is fast and small. Native, GPU-rendered, written in Zig, starts almost instantly. A homelab server has better things to do with its 8 GB of RAM than feed a feature-heavy terminal that I open twice a day. Ghostty stays out of the way.
One config file, no clicking through preferences. iTerm2's config lives in a binary plist that you can technically check into git but in practice you set up through twenty tabs of GUI preferences. Ghostty has a single text file at ~/.config/ghostty/config, plain key-value, version-controllable, easy to push around with my dotfiles. On a server I rebuild rarely but cleanly, that one file is the whole story.
Less surface area to break. No plugin system, no AppleScript dictionary, no triggers, no coprocesses, no profile imports, no semantic history. Just a terminal. Fewer moving parts means fewer things to go wrong on a machine I want to leave running for months.

It is not that Ghostty is better than iTerm2. It is that the two terminals are tuned for different jobs. iTerm2 is a workshop, Ghostty is a clean utility room. The Mac mini gets the utility room. The Air stays the workshop. Both end up with the same prompt and the same aliases through dotfiles, so the muscle memory does not break when I switch.

With the terminal sorted, I sat down with a plan already in my head. I knew exactly what I wanted on this box, in what order, and how each piece should talk to the next. Choosing Colima over Docker Desktop, Tailscale over a raw VPN, Caddy over Nginx, restic over rclone, the four apps in the stack, the launchd schedule for backups, the port layout, all of it was decided before I ran the first command. The shell was open and that is where the real work happened.

What I thought would be a 2 hour job stretched into a full evening that ran past midnight, mostly because of the gotchas I am about to walk through. The actual command-running was fast. The "wait, why is this not working" parts were not, and those were the parts where I sat with the logs, read them carefully, and worked out what to change.

Below is the same setup, phase by phase, with the four or five places I got stuck and what unstuck them.

Phase 1, making it a 24x7 server

A Mac is a laptop OS by default. It wants to sleep, dim the screen, idle the disks. For a homelab I want the opposite. pmset is the way to tell it.

sudo pmset -a sleep 0
sudo pmset -a disksleep 0
sudo pmset -a autorestart 1
sudo pmset -a womp 1
sudo pmset -a tcpkeepalive 1

Translation, in order.

sleep 0, the system never sleeps.
disksleep 0, the disks never spin down.
autorestart 1, automatic restart after a power cut.
womp 1, wake on magic packet so I can wake it remotely.
tcpkeepalive 1, keep TCP alive across long-lived connections.

I left displaysleep at the default 10 minutes, because I sometimes plug a monitor into this thing and I do want the screen to go off when I am not using it.

To check what stuck, pmset -g shows the current settings:

Then the firewall. I turned on the application firewall but kept it permissive, signed software allowed, stealth mode off. The reason is my threat model is the internet, not my LAN. Tailscale is going to be the real perimeter. If I lock the firewall down hard now, I will spend the next hour fighting it for every brew service I install.

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setglobalstate on
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setallowsigned on
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setallowsignedapp on
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setstealthmode off

Now the first stuck moment.

SSH needs a GUI click

I wanted Remote Login on so I could ssh into this thing from my laptop. The classic CLI way is sudo systemsetup -setremotelogin on. On macOS 13 and later that command will not work unless the calling binary has Full Disk Access. So even with sudo, it refuses.

The fastest fix is the GUI toggle. Open System Settings, in the search box on the sidebar type Remote Login, click the result, toggle it on. 15 seconds, done. Trying to fight it from the terminal is not worth the rabbit hole.

This was the first time the setup made me reach for the trackpad. It will not be the last.

With the OS pinned to "server mode" and SSH up, the box was ready to be reached from somewhere other than this desk.

Phase 2, Tailscale and the menu-bar trap

For remote access I went with Tailscale. Free tier, WireGuard underneath, MagicDNS gives every device a name on my private network. From anywhere in the world I can reach this Mac mini the way I would reach localhost.

Install was a one liner.

brew install --cask tailscale-app

Then I tried to log in from the CLI. It failed. Reason, the standalone cask uses a System Extension and that extension needs user approval the very first time. Until you sign in once through the menu-bar app, the CLI is a paperweight.

So I clicked the Tailscale icon in the menu bar, picked Log in, the browser opened, I signed in with my account, came back. Now tailscale status showed the Mac mini on my tailnet.

tailscale status
tailscale ip -4

The whole device got a stable name like mac-mini.tailXXXX.ts.net over MagicDNS. From now on, anywhere I am, I can reach this hostname as if the Mac mini was in the same room.

Good. The perimeter was sorted. Time for the actual reason I bought into all this.

Phase 3, the agent webhook

Now the fun part. The agent army I had been dreaming about.

The idea is simple. A small HTTP server on the Mac mini listens for a POST. When a POST comes in with a prompt, it spawns Claude Code as a subprocess, lets Claude do whatever the prompt asked, captures the output and returns it. Optionally it can also push the result to my phone.

I kept this deliberately small. Python stdlib, no Flask, no FastAPI. About 150 lines of Python that does these three things.

GET /healthz, no auth, returns ok if the server is up.
POST /trigger, bearer-auth, body has a prompt and a few optional knobs. Spawns claude -p "<prompt>" --output-format json, returns the JSON output.
POST /notify, bearer-auth, posts to ntfy so the result lands on my phone.

The server binds to 127.0.0.1:8765, so it is only reachable from the Mac mini itself. Tailscale plus Caddy will expose it later if I want it from outside. The bearer token lives in ~/homelab/secrets/agent_token, mode 600, generated once with openssl rand -hex 32.

To make it survive reboots I wrote a launchd plist at ~/Library/LaunchAgents/com.vineeth.agent.plist with RunAtLoad=true and KeepAlive=true. If the process dies, launchd brings it back. Logs go to ~/homelab/logs/agent.{log,err.log}.

The two CLI helpers that go on top:

# Fire a prompt, get JSON back.
agent "summarize what's on my calendar today"

# Same, plus push the result to my phone.
agent --notify "scan ~/code/foo for security issues"

# Liveness probe.
agent --healthz

The healthz probe is the satisfying one. The first time it came back green, the agent army had its first soldier alive.

One nice surprise here. The agent does not need an Anthropic API key. It shells out to the claude CLI as a subprocess, which uses whatever auth my local claude CLI already has. Since I am logged into Claude Code on this Mac, the agent inherits that login. No separate billing line, no key rotation, no fuss. If I ever want an independent non-interactive key, I can add one to the plist, but for now this is enough.

So the agent core was running. Now to put a friendly face in front of all this so my future self does not have to remember a port number for every service.

Phase 4, Caddy, and the auto_https trap

I wanted one nice landing page that lists all my homelab apps, and I wanted Caddy in front of everything as a reverse proxy. Caddy is great for this. One file, sensible defaults, internal certs.

brew install caddy

I wrote a Caddyfile at ~/homelab/caddy/Caddyfile. The shape is a small gruvbox-themed landing page served at / that links to every service, plus a reverse-proxy vhost per service. Starting with the vhost part:

{
    auto_https off
}

vault.example.tailnet.ts.net {
    reverse_proxy 127.0.0.1:8222
}

uptime.example.tailnet.ts.net {
    reverse_proxy 127.0.0.1:3001
}

ntfy.example.tailnet.ts.net {
    reverse_proxy 127.0.0.1:8088
}

n8n.example.tailnet.ts.net {
    reverse_proxy 127.0.0.1:5678
}

auto_https off because I assumed Tailscale would handle TLS and Caddy should just do plain HTTP. Started the service.

brew services start caddy

The four hosts came up on :80 but every request was returning the wrong thing. Some were dropping connections. Took me a minute to read the Caddy log.

Found it. auto_https off does not just disable the redirect from HTTP to HTTPS. It disables certificate provisioning too. So the four virtual hosts never got leaf certs and Caddy was confused about what to serve. The fix is one word.

{
    auto_https disable_redirects
}

disable_redirects keeps Caddy's internal cert management on, just stops it from auto-redirecting plain HTTP requests to HTTPS. That is exactly what I wanted. Reload Caddy, the four vhosts are healthy, Vaultwarden returns 200, Uptime Kuma returns 302.

This is the kind of thing you only learn by hitting it. The Caddy docs do say auto_https off disables both, but I had not parsed that line carefully. One word, half an hour gone.

MagicDNS does not resolve subdomain prefixes

There was a second smaller catch right after. Tailscale MagicDNS gives each device a stable name like mac-mini.tailXXXX.ts.net. What it does not do, is resolve subdomain prefixes like vault.tailXXXX.ts.net. So my Caddy vhosts named vault.<tailnet>.ts.net, uptime.<tailnet>.ts.net, etc, were not reachable from a phone or laptop on the tailnet, because the names did not resolve.

The quick fix was to skip the pretty subdomains for now and just reach each service on the device hostname plus its port.

Service	URL
Vaultwarden	`http://mac-mini.tailXXXX.ts.net:8222`
Uptime Kuma	`http://mac-mini.tailXXXX.ts.net:3001`
ntfy	`http://mac-mini.tailXXXX.ts.net:8088`
n8n	`http://mac-mini.tailXXXX.ts.net:5678`
Agent	`http://mac-mini.tailXXXX.ts.net:8765`

These work over Tailscale's WireGuard tunnel so plain HTTP is fine inside the tailnet. The Caddy subdomain vhosts stay configured, but they are sitting unused until I decide to do /etc/hosts entries on every client or move to Tailscale Serve. Which I did, but later in the story.

Routing done for now. Time to stand up the actual things being routed.

Phase 5, the Docker stack via Colima

Docker Desktop on macOS is a heavy beast. I went with Colima instead, which gives you a Lima VM running Docker, all from the CLI, no GUI.

brew install colima docker docker-compose
colima start --cpu 4 --memory 6 --disk 60

4 CPUs, 6 GB RAM, 60 GB disk for the VM. That leaves enough for the Mac itself to breathe. To make sure Colima starts on every boot I wrote a small launchd plist that only calls colima start if it is not already running, so it is idempotent.

Then the actual stack. One docker-compose file at ~/homelab/stack/docker-compose.yml. All four containers bound to 127.0.0.1 only, all with restart: unless-stopped, all with their data directories under ~/homelab/data/.

Service	Port	Purpose
vaultwarden	8222	Bitwarden compatible password manager
uptime-kuma	3001	Service and endpoint uptime checks
ntfy	8088	Push notifications to my phone
n8n	5678	Visual workflow automation

cd ~/homelab/stack
docker compose up -d
docker compose ps

All four green.

To update everything later, the two-line dance is docker compose pull && docker compose up -d. Easy.

A macOS 15 thing worth knowing

While testing, I noticed that hitting http://<own-LAN-IP>:<port> from the same Mac was returning Empty reply from server, even though TCP was accepting. This is a macOS 15 Local Network privacy behavior. It does not affect 127.0.0.1 and it does not affect access over Tailscale, only the case where the Mac is talking to itself via its LAN IP. If I ever want LAN access from other devices in the house, the fix will be a permission grant in System Settings, Privacy and Security, Local Network. For now I do not need it because Tailscale handles all access from outside the Mac.

Phase 6, ntfy and the topic mismatch

ntfy is a tiny push-notification service. You publish to a topic, anyone subscribed to that topic gets a push. I run my own ntfy in Docker, so the topic name itself acts as the shared secret. Mine is a 16-character random hex string stored in ~/homelab/secrets/ntfy_topic.

The wrapper command is just:

notify "title" "body"
notify --high "alert" "something bad just happened"

I tested the server side first by curling it directly. The server happily logged a publish event. So far so good.

Then I installed the ntfy app on my phone, pointed it at the Tailscale URL, subscribed to my topic, and sent a test. Nothing. Phone was silent.

The ntfy server log said subscribers=1, topics_active=2. That number is what gave it away. Two active topics means the publisher and the subscriber are on different topic names. The phone was subscribed, but to a different string than what I was publishing to. Some small typo on my phone screen when I added the topic.

I opened the ntfy app on the phone, long-pressed the subscription, edited the topic, pasted the exact 16-character hex string from ~/homelab/secrets/ntfy_topic. Tried again. Phone buzzed.

Small bug, but it is the kind of thing that wastes 20 minutes if you trust the phone-side input.

Notifications working. On to the bit I was most looking forward to.

Phase 7, Vaultwarden and the localhost-only crypto

Now the moment I was looking forward to. Bringing up Vaultwarden, the Bitwarden compatible self-hosted password manager. I opened it from my laptop using the Tailscale URL, http://mac-mini.tailXXXX.ts.net:8222. Clicked Create account. Got a browser error.

Web crypto refuses to run on a non-HTTPS origin, except for localhost and 127.0.0.1. Both of those are treated as secure contexts by browsers, so plain HTTP works there. But mac-mini.tailXXXX.ts.net is not in that whitelist. So the signup form rendered, but the moment it tried to derive a key, the browser blocked it.

The quick workaround was to do the signup directly on the Mac mini's own browser, hitting http://127.0.0.1:8222. That worked first try because of the localhost exception. But this was not a permanent fix. I wanted to access Vaultwarden from my phone too. Plain HTTP over Tailscale would not be enough for that.

The clean fix is Tailscale Serve. Quick intro for anyone who has used Tailscale but not Serve. Serve is a built-in feature on top of plain Tailscale that lets you front a local port with real Let's Encrypt HTTPS on your *.ts.net hostname. So instead of http://mac-mini.tailXXXX.ts.net:8222 you get https://mac-mini.tailXXXX.ts.net/, with a publicly-trusted cert that any browser trusts, all still inside the WireGuard tunnel. No certbot, no /etc/hosts tricks, no self-signed warnings.

Serve needs two one-time toggles in the Tailscale admin console first. Enable Serve for the node, enable HTTPS certificates for the tailnet. Both are clicks in the admin web UI.

After that I told Tailscale Serve to map ports.

sudo tailscale serve --bg --https=443 http://127.0.0.1:8222   # Vaultwarden
sudo tailscale serve --bg --https=8443 http://127.0.0.1:3001  # Uptime Kuma
sudo tailscale serve --bg --https=10000 http://127.0.0.1:8088 # ntfy
sudo tailscale serve --bg --https=10001 http://127.0.0.1:5678 # n8n

The port 443 squat

The first command failed with ERR_CONNECTION_CLOSED from the browser. Reason, Caddy was still listening on :443. Tailscale Serve also wants :443. They cannot share. So I took Caddy off :443 entirely (auto_https disable_redirects keeps it on :80 only) and let Tailscale own all the HTTPS.

After that the four services were on real HTTPS, with real certs:

Service	URL
Vaultwarden	`https://mac-mini.tailXXXX.ts.net/`
Uptime Kuma	`https://mac-mini.tailXXXX.ts.net:8443/`
ntfy	`https://mac-mini.tailXXXX.ts.net:10000/`
n8n	`https://mac-mini.tailXXXX.ts.net:10001/`

I reloaded Vaultwarden on the tailnet URL, the browser was finally happy, signup went through.

One housekeeping thing I did right after the first account, change SIGNUPS_ALLOWED to false in the compose file and bounce the container. Otherwise anyone who reaches the URL can sign up. Closing that door is a 10-second job.

Phase 8, Uptime Kuma and the HEAD vs GET trap

Uptime Kuma was easy. Open the web UI on its port, walk through the first-run wizard, add monitors for the four services and the agent. There was one gotcha here too.

By default, Uptime Kuma fires HEAD requests for HTTP checks. ntfy's /v1/health only answers GET. HEAD returns 404. So Kuma was reporting ntfy as down, even though ntfy was perfectly fine.

The fix is one dropdown change. Edit the monitor, scroll to HTTP Options, change Method from HEAD to GET. Save. Green within one heartbeat.

While I was there I switched all the monitors to GET to keep them consistent, since some upstreams quietly return 200 on HEAD and some do not. Always-GET is safer for this kind of liveness check.

The Kuma monitor list now has each service plus a ping to the Caddy landing page, all green. If anything goes red, Kuma is configured to push to ntfy, which means it pings my phone within a heartbeat. That closes the monitoring loop.

Phase 9, restic backups to Backblaze B2

The last brick. None of this homelab is useful if a power surge or a disk failure takes the data with it.

I used restic because I already know it inside out. I have been running it for backups for years, and at one point the cron-and-restic setup I had outgrew itself enough that I wrote my own NestJS backup service on top of it called backupctl. So picking restic for the homelab was the easy part. It is small, encrypts client-side, deduplicates, supports a bunch of remotes including Backblaze B2, and I know its quirks like a friend's bad jokes.

For the destination I went with Backblaze B2. At work we use a Hetzner Storage Box for company backups and it serves us very well, fixed monthly fee, plenty of space, predictable bill. The smallest one, BX11, is €3.20 a month for 1 TB. Cheap as chips at company scale.

For a personal homelab the math is a bit different. I am pushing only a few hundred MB a night, and after restic dedup the whole thing stays well under 10 GB for the foreseeable future. B2's first 10 GB are free, and beyond that it is roughly seven dollars per terabyte per month. So for my actual usage right now, B2 lands at zero dollars a month. A fixed €3.20 on Hetzner would also be fine, it is not exactly going to bankrupt anyone, but free is free, and pay-as-you-go has the right shape for a side project that may or may not grow.

If my homelab data ever crosses a terabyte, the math flips and Hetzner becomes the cheaper one. That is the day I will switch. For now, B2.

brew install restic

The wrapper script at ~/homelab/backup/backup.sh sources a secrets file with the B2 keys and the restic password, then runs:

restic backup \
  --exclude-file=~/homelab/backup/excludes.txt \
  ~/homelab ~/.zshrc ~/.gitconfig ~/.ssh/config

restic forget --prune \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 6

Excludes are the usual suspects, node_modules, caches, log files, the env file itself.

A launchd plist at ~/Library/LaunchAgents/com.vineeth.backup.plist runs this every day at 03:30. If the env file is missing or empty, the wrapper logs a polite message and exits 0, so the schedule does not crash on the days I have not yet configured credentials.

For the credentials part I went to Backblaze, created a B2 account, made a private bucket, generated an application key with bucket-scoped read+write, copied the keyID, applicationKey, and the bucket name into ~/homelab/secrets/restic-env.sh. The restic password I generated locally with openssl rand -base64 36, then wrote it down outside this Mac in two places, because if I lose this password, every snapshot becomes unrecoverable noise.

First snapshot pushed in under a minute, 5.9 MiB in, 256 KiB out after dedup.

The retention rule keeps 7 daily, 4 weekly, 6 monthly. So at any time my B2 bucket has rolling coverage of the last week, the last month, and the last half year, with restic pruning the rest. Daily fire at 03:30 is wired up, future failures will push a high-priority ntfy notification to my phone.

What I have, all together

That is the homelab. One long evening that went past midnight, one Mac mini that used to live in a colleague's drawer, no SaaS bills.

Vaultwarden at https://mac-mini.tailXXXX.ts.net/, my password vault.
Uptime Kuma at :8443, watching my services and the agent.
ntfy at :10000, pushing alerts and agent results to my phone.
n8n at :10001, where I will build workflow automations from now on.
Agent webhook at :8765, runs Claude Code on demand from a phone shortcut, a cron job, or an n8n flow.
Caddy on :80, serves a gruvbox themed landing page that lists everything.
Netdata on :19999, machine-level monitoring.
Restic to B2, daily at 03:30, 7-4-6 retention.

The agent army I had been dreaming about now has a barracks. The phone shortcut I wired up after this is one tap. Tap, speak the prompt, the Mac mini does the work, the result lands as a push notification on my phone. Pi could not carry that. Mac mini did not break a sweat.

What I would tell anyone doing this

A few things that would have saved me time if I had read them before starting.

The Mac will need GUI clicks. macOS keeps tightening what the CLI can do without Full Disk Access. SSH Remote Login is one. Tailscale's System Extension is another. Plan for at least three or four short GUI sessions across the setup. Do not try to be a hero with osascript workarounds, they break across versions.

Tailscale is your perimeter. Once Tailscale is up, your "internet-facing" attack surface drops to roughly zero. So you can stop pretending your local Caddy needs to be a hardened bastion, which is what frees you to keep the firewall permissive and the configs simple. Tailscale Serve on top gives you real HTTPS without certbot. That alone saves an hour.

Use Colima, not Docker Desktop. No GUI, no resource hog, no licensing question, starts via launchd. The Docker CLI works exactly the same as on Linux. The only thing you do not get is the Docker Desktop dashboard, which I do not miss.

One Caddyfile word matters. auto_https off is not the same as auto_https disable_redirects. The first one kills certs. The second one keeps certs and only kills the HTTP-to-HTTPS redirect. Read it once carefully.

HEAD vs GET will burn you once. Default Uptime Kuma monitors use HEAD. Some health endpoints only answer GET. Set the method explicitly when you add a monitor, it saves a "why is this red" round trip.

Save the restic password offline before the first backup. The first restic backup initializes the encrypted repo and locks it to that password forever. Lose it and your snapshots are garbage. Paper, second password manager, anywhere that is not this same Mac.

Where this goes next

The bones are in place. What I want to add on top, in roughly this order, are these.

A small phone shortcut that POSTs to the agent webhook from the home screen, so firing a prompt is one tap and one voice dictation away. An n8n flow that emails me a daily brief at 07:00 from the same agent endpoint. A morning launchd job that runs agent --notify "morning brief" so my phone wakes up with one notification instead of five.

And the bigger one I am chewing on, a Cloudflare Tunnel in front of one or two of these services on my vinelabs.de domain. I already have that domain sitting there as my experiment lane, so something like lab.vinelabs.de could front the agent or a public dashboard without ever opening a port on my router. Cloudflare Tunnel gives me a real public URL, real TLS, and Cloudflare Access on top if I want to gate it. The day I want any of this reachable from outside Tailscale, say from a friend's laptop or a colleague's phone, that is the path I will take.

For now, the dream is no longer flapping its wings in vacuum. It has a roof, a barracks, and it sleeps in a 2018 Mac mini that a generous colleague handed me on a weekend, right in the middle of his own family time, without making a big deal of it. That kind of thing sticks.

That is all I had for this one. If you have a Mac mini lying around, even an old one, give it a job. It will surprise you. Catch you in the next blog, take care until then.

I treated skills like dotfiles. Then they started spawning subagents.

Vineeth N K — Sun, 17 May 2026 09:50:05 +0000

I treated skills like dotfiles. Then they started spawning subagents.

TL;DR: My CLAUDE.md was turning into a Frankenstein .bashrc. Universal rules and one-off behaviours mashed together into a config file that kept growing by a few lines every week. Then Claude Code skills landed and my first instinct was, oh, these are basically dotfiles for Claude. The instinct was half right. Skills do behave like dotfiles in how they load. Opinionated, intent-triggered, version-controlled. But they go one step further. Skills can dispatch work to subagents. A dotfile sits there. A well-shaped skill knows how to recruit help when the task is bigger than one prompt. Dotfiles never did that. Skills go to work.

So let me back up a little.

The CLAUDE.md that ate itself

A while back I noticed my CLAUDE.md had quietly become the longest config file in my home directory. Not the most-edited. Not the most-read. Just the longest. Every hard lesson I had paid for in production ended up there. The schema-verification protocol I built after one too many "column does not exist" errors. The pre-merge audit checklist that catches regressions in a diff. The refactor playbook that forces every rename to trace its call sites before touching code. Every time a bug came back to bite me, I would tab over to that file and tack on the guardrail I wished had been in place.

Some sections genuinely belonged there. A rule like "never assume, verify before you respond or troubleshoot" is exactly the kind of always-on guardrail you want every session to start with. But other sections were not like that. The schema-verification protocol was a long, layered procedure with its own decision tree, fallback commands, and known gotchas for the legacy database. I do not need that loaded when I am writing prose or wiring up a frontend component. I need it loaded the moment I am about to touch a query. Different moment, different need.

But CLAUDE.md does not care. CLAUDE.md is sourced every time. It is the .bashrc of the Claude world. Whatever you put in there shows up in every conversation, whether you wanted it there or not.

If your config file has crossed a few hundred lines and you still keep adding to the bottom, you already know the shape of the problem.

My first instinct was dotfiles

When skills landed in Claude Code, my first reaction was the same one I imagine most developers had. Oh nice, these are just dotfiles for Claude.

The vibe is right. Skills are:

Opinionated. They reflect how you write, how you commit, how you review.
Version-controlled. They live in a folder. You can git track them. You can share them.
Intent-triggered. A skill loads when it becomes relevant, not before.
Composable. You install the ones you want, ignore the ones you do not.

That is dotfiles. That is exactly the model. The same way I keep a .zshrc because I do not want to retype aliases on every machine I touch, I now reach for skills because I do not want to paste the same workflow rules every time I am about to do something specific.

So I started moving things out of CLAUDE.md. The schema-verification protocol, with its full audit machinery and database-specific gotchas, became a verify-before-query skill. The pre-merge audit became a ship-readiness skill, complete with diff scanning, regression-pattern flagging, and a test-coverage gate. The refactor playbook, the one that traces every call site before allowing a rename, became its own skill. Slowly, CLAUDE.md shrank. The bloated file became leaner, and the sections that remained genuinely deserved to be always-on.

The split that worked

After a few rounds of pulling things out, a pattern showed up. Two kinds of rules ended up in two different places.

Always-on, universal rules stayed in CLAUDE.md:

Behavioural guardrails (verify before responding, ask when uncertain, do not invent APIs)
Naming and coding conventions that apply to every file you write
Testing standards that should be remembered any time tests are involved
Quality gates that you want enforced regardless of task

On-demand, situational rules moved into skills:

Schema-verification protocol (only matters when you are about to query or migrate a database)
Pre-merge ship-readiness audit (only matters right before you push a branch)
Refactor planning playbook that fans out call-site discovery (only matters when you are restructuring a module)
Incident triage runbook with parallel log and metric scans (only matters when something is broken in production)

Same way you keep export PATH=... in your .bashrc but you keep your CLI tools in /usr/local/bin and only call them when the task asks for them. Universal stays at the entry point. Specific moves into discrete, callable units.

Here is what that ends up looking like on disk, once you have done a few rounds of moving things out:

~/.claude/
├── CLAUDE.md                       # the lean .bashrc - always loaded
└── skills/
    ├── verify-before-query/
    │   └── SKILL.md                # schema audit, fallback commands, gotchas
    ├── ship-readiness/
    │   ├── SKILL.md                # pre-merge checklist
    │   └── references/
    │       └── regression-patterns.md
    ├── refactor-plan/
    │   └── SKILL.md                # fans out call-site tracing
    └── incident-triage/
        ├── SKILL.md                # parallel log + metric scans
        └── references/
            └── known-fingerprints.md

The shape is the point. CLAUDE.md sits at the top, lean and universal. Each skill is its own folder with its own SKILL.md and its own supporting references, isolated from every other skill. Same mental model as ~/.config/ or /usr/local/bin/. You can git track the whole tree, share it with a teammate, swap one skill out without touching the others.

The split clicked for me when I noticed a section in my CLAUDE.md had been reduced to a one-liner that pointed to a skill, saying in effect "the rule is X, see the skill for the full rationale and the audit machinery." That single line is the analogy made literal. The always-on rule lives at the entry point. The heavy machinery lives in the skill that gets invoked when the situation calls for it.

Then the analogy broke

I sat with the dotfiles-for-Claude framing for a bit and felt pretty pleased with myself. Then I noticed a difference I could not ignore.

A .bashrc is passive. It defines an alias and waits. The alias runs in your shell, in your process, in your context. It does not spin up another shell, hand it a task, get back a summary, and let you move on. That is a different category of thing.

A skill can do exactly that. A well-shaped skill can fan out helpers for the parallelisable parts of a task. Tracing every call site of a function before approving a rename. Scanning a year of deploy logs against an error fingerprint during incident triage. Auditing a large diff against a long list of regression patterns before letting the merge go through. The grunt work runs on its own lane and comes back as a single report, while your main thread stays clean and ready to make the actual decision. That is not something a dotfile has ever done in the history of dotfiles.

A dotfile is a snapshot of preferences. A skill is a snapshot of preferences plus a workflow. The workflow is yours to shape, and when the work is bigger than one prompt, it can call for backup.

Why this matters in practice

Three things shifted in how I think about my AI setup once this clicked.

One. I stopped putting per-task rules into CLAUDE.md. Whenever I am about to add a section, I ask myself, is this rule something I want Claude to know every time, or only when I am doing a specific kind of task? If it is the second, it goes into a skill. CLAUDE.md stays close to what .bashrc should be. Tight. Universal. Mostly stable.

Two. I started thinking about skill-internal delegation as a first-class design choice. When tuning a skill, I now decide which steps should fan out and which should stay single-threaded. The research-heavy steps (call-site tracing, regression-pattern auditing, log scanning, dependency-tree walking) get dispatched to subagents. The judgement-heavy steps (choosing the migration strategy, picking which regressions are real, deciding what to ship) stay in the main thread. A skill is not just a prompt. It is a workflow, and a workflow can dispatch.

Three. Sharing got more interesting. When you share a skill with a teammate, you are not just sharing your preferences. You are sharing a small workflow that knows how to recruit help when the task is bigger than itself. That changes the conversation. "Use my skill" becomes closer to "use my whole approach, including how it scales when the work is heavy."

Where the analogy gets uncomfortable

I do not want to oversell this. The dotfiles framing is useful as an entry point, but it bends in a few places. Worth saying out loud.

First, the always-on versus on-demand split is not as clean as I made it sound. Some sections still in my CLAUDE.md are arguably skill-shaped too. Stack-specific coding conventions. Framework patterns. API contracts. They only matter when I am working in that particular stack. They are not truly universal. A purist would extract them. I have not yet, partly because the boundary is fuzzy, partly because moving them out means trusting that the right skill activates at the right moment, and that trust takes time to build. Honest answer: this is a work in progress.

Second, dotfiles are passive. They define. Skills are active. They can do. That is not a difference of degree, it is a difference of category. When you copy someone else's .bashrc, the worst case is your shell behaves slightly weird until you remove the offending line. When you install someone else's skill, the worst case is much more interesting, because the skill might delegate to a subagent that runs code, opens PRs, or talks to your APIs. Skills come with more power, and therefore more responsibility for whoever is curating the list.

Third, sharing skills across machines and teams is not yet a solved problem the way dotfiles are. There is no stow for skills. No chezmoi equivalent. No widely-adopted "here is my skill repo, symlink-mount it into ~/.claude/skills/ and you are done" pattern. The ecosystem is still young. People are figuring it out as we go.

So the right framing is closer to this. Dotfiles is the entry-point analogy. Once you internalise the analogy, you can start to see that skills are a strict superset. They borrow the dotfiles vibe and then add agency on top.

The mental model I landed on

Here is how I now think about the whole stack.

.bashrc and CLAUDE.md are the same kind of object. Both are "load this every time I start a session." Both should be lean, opinionated, and universal. Both should grow rarely and shrink often.

Skills and /usr/local/bin/<tool> are the same kind of object. Both are "invoke this when the task calls for it." Both can do heavy work that does not belong in the rc file. Both can be shared, versioned, swapped out.

The new piece, the part that makes skills not just dotfiles, is that skills can spawn subagents. Your tool in /usr/local/bin/ does not call another shell to delegate work back to itself. A skill can. That is the upgrade. That is why "dotfiles with agency" is a more accurate metaphor than "dotfiles for Claude."

If you are still piling rules into CLAUDE.md and have not yet started moving the per-task ones out into skills, give the exercise a try. Even just the act of asking "is this rule universal, or is this for a specific moment?" sharpens your sense of what each layer is for. You may end up with a leaner CLAUDE.md, a few well-tuned skills, and a setup that finally has the kind of layered shape your terminal config has had for years.

That is all I had on this one. If you made it till here, thank you, genuinely. See you in the next one, where I will probably be complaining about something else that broke.

How Git Worktrees Killed My Stash-Hotfix-Rebase Dance

Vineeth N K — Fri, 15 May 2026 14:45:09 +0000

How Git Worktrees Killed My Stash-Hotfix-Rebase Dance

TL;DR: For the longest time, every urgent hotfix in the middle of a feature meant the same painful little dance. Stash my work, checkout main, branch off, fix, push, switch back, rebase, pop the stash, then enjoy the surprise conflicts. Git worktrees made all of that nonsense vanish. One feature branch checked out in one folder, one hotfix branch checked out in another folder, both alive at the same time, both pointing at the same repo. Add agentic AI on top and now I am spinning up parallel worktrees, handing each one a task, and reviewing clean PRs in Graphite while my coffee is still warm. This blog is for every developer who has not yet befriended git worktree. By the end of it, you will wonder how you survived without it.

So before I tell you why worktrees changed my life, let me tell you why my life needed changing.

The dance nobody asked for

Picture the scene. I am deep into a feature branch. Files half-edited, mental model loaded, twenty browser tabs open, a debugger paused on a breakpoint I am about to investigate. The good kind of flow. The expensive kind.

Then Slack does its little notification thing.

"Production is throwing 500s on the payment page. Can you take a quick look?"

Of course I can. I am the on-call. So now begins the ritual. You know the one. Every developer who has ever held a git branch open during an incident knows the one.

git stash push -m "wip feature stuff, please remember everything"
git checkout main
git pull
git checkout -b hotfix/payment-timeout
# ... patch the bug, write a test, push, open PR, ship ...
git checkout feature/checkout-redesign
git rebase main
# CONFLICT. of course.
git stash pop
# CONFLICT. again. of course.

By the time the stash pop ends in a second round of conflicts, the mental model I had carefully loaded into my head before the Slack ping has fully evaporated. The browser tabs are still there, but I have no idea why I had them open anymore. The breakpoint is irrelevant now because the file has been rewritten by the rebase. I have shipped the hotfix, sure. But I have also paid for it with the rest of my afternoon.

You know this evening if you have ever lived it.

The thing I should have known earlier

Here is the embarrassing part. git worktree has been in git since version 2.5. That is from 2015. The feature is older than half the JavaScript frameworks people are arguing about on Twitter. And for a good chunk of my career, I never used it.

The reason is simple. Nobody told me. The git tutorials I grew up on stopped at branch, merge, rebase, stash. Worktrees lived in the "advanced" page that nobody clicked. I want to fix that for you right here, before this blog ends.

A worktree, in one sentence, is a second working directory for the same repo, with its own checked-out branch.

That is the whole idea.

You know how a normal git repo has one folder where your files live, and you git checkout to switch branches inside that folder? Worktrees say "what if you could have more than one such folder, each on a different branch, all sharing the same underlying repo data?"

That is it. There is no magic. There is no parallel universe. There is no separate clone eating extra disk for a full second copy of history. Just one repo, multiple working directories, each on its own branch.

The new dance, which is not really a dance

So now the Slack ping comes in. I am still in my feature branch, still in flow. Here is what happens.

git worktree add ../app-hotfix -b hotfix/payment-timeout main
cd ../app-hotfix
# patch, test, ship
git push origin hotfix/payment-timeout
cd ../app
# back in my feature branch. nothing moved.
git worktree remove ../app-hotfix

No stash. No checkout dance. No rebase. No second conflict from popping a stash that no longer matches reality. My feature branch is exactly where I left it. The breakpoint is still paused. The browser tabs still make sense. The model is still loaded.

This is not a productivity trick. This is a sanity trick.

The first time I did this and switched back to my feature branch and saw my unsaved buffers exactly the way I had left them, I sat there and laughed at myself. All those years of stashing. All those evenings lost to conflict resolution. Gone, because of two flags on a command I had not bothered to read.

The four worktree commands you actually need

Worktrees sound exotic until you see how few commands run the whole show. There are basically four.

# 1. add a new worktree on a new branch
git worktree add ../path-to-new-folder -b new-branch-name base-branch

# 2. add a worktree on an existing branch
git worktree add ../path-to-new-folder existing-branch-name

# 3. list all your worktrees
git worktree list

# 4. clean up a worktree when you are done
git worktree remove ../path-to-old-folder

That is the whole API. You will not need anything else for the first month. Maybe ever.

A few things worth knowing that the man page mumbles instead of shouts.

Each worktree gets its own checked-out branch, and a branch can only be checked out in one worktree at a time. If your feature branch is checked out in ../app, you cannot also check it out in ../app-hotfix. Git will politely refuse. This is a feature, not a bug. It stops you from corrupting your own history by editing the same branch from two folders.

Worktrees share the same .git data. They do not duplicate your history. The new folder has a tiny .git file that points back to the original repo. So disk usage is basically the size of your source tree, not the size of your history. Even for a monorepo with years of commits, adding a worktree costs you almost nothing.

Branches you create inside a worktree are real branches in the main repo. Push them, merge them, delete them. There is no "worktree branch" species. It is just a branch.

If you have never tried this before and you are reading this on a workday, open your repo right now and run git worktree add ../scratch -b throwaway main. Look at the new folder. Be impressed. Run git worktree remove ../scratch when you are done. The whole experiment costs you nothing and teaches you everything.

Where this gets quietly powerful: agentic AI

Now we get to the part that turned this from a nice habit into a discipline I will not work without.

I have been heavy into AI-assisted development lately. Claude Code, Codex, whatever the agent of the month is. The pattern that finally clicked for me is this. Instead of pair-programming with the agent on one branch, I treat each agent like a junior colleague who needs their own desk.

The desk is a worktree.

git worktree add ../app-task-42 -b ai/refactor-auth main
git worktree add ../app-task-43 -b ai/upgrade-orval main
git worktree add ../app-task-44 -b ai/add-tracing  main

Then I open each worktree in its own terminal, kick off an agent in each one with a clear task, and walk away. Sometimes literally. Coffee, lunch, the school run.

When I come back, there are three branches. Sometimes three open PRs. Sometimes three half-done attempts where the agent got stuck on a question and is patiently waiting for me to unblock it. Either way, the worktrees never stepped on each other. Branch A did not corrupt branch B. The feature branch I was working on before I started this experiment is still there, untouched, sitting in its own folder, ready for me to pick up exactly where I left it.

I review the PRs in Graphite. Stack them if they belong together. Merge them in the right order. The agent does the typing. I do the deciding. The worktrees are what make it parallel instead of a queue.

Anyone else here doing this already and quietly grinning?

The other thing worktrees give you in the AI workflow is something I did not expect. Review without context switching. When one of the agents finishes a task, I do not need to abandon my own feature branch to review its PR. I just cd into that worktree, read the diff, run the tests, decide. A short detour. Then cd back to my own work and the model in my head is undisturbed.

Compare that to the old way. Stash. Checkout to the PR branch. Run tests. Comment. Switch back. Pop. Pray. The cognitive cost of the old way was so high that I avoided reviewing PRs mid-feature. So either the reviews stacked up at the end of the day, or my own feature suffered. With worktrees, neither happens.

The rules I follow, which you can steal

A few self-imposed rules that turned this from a sometimes-thing into a default.

One worktree per intent. Feature, hotfix, review, AI task. Each gets its own folder. If two efforts conceptually belong together, they share. If they do not, they do not.

Name the folder after the task, not the branch. ../app-hotfix is a folder. Inside it lives whichever hotfix branch I happen to be on at the moment. When the hotfix is shipped and the branch is dead, I can reuse the folder for the next hotfix. The folder is the desk. The branch is the paperwork on the desk.

Keep them as siblings of the main repo, not inside it. Putting a worktree inside the same folder as the main checkout confuses your editor, your file watchers, and your future self. A flat layout like code/app, code/app-hotfix, code/app-task-42 keeps everything sane.

Delete worktrees the moment they are done. They are cheap to create. They should be cheap to destroy. A dead worktree lying around is exactly the kind of thing that quietly accumulates until someone, probably future you, has six folders and no memory of what is in any of them.

Per-worktree shell setup if your stack needs it. If your project has a .env, a .tool-versions, or any per-folder setup, each worktree needs its own. This is usually a one-time copy and forget. Some teams put a tiny bin/new-worktree script in the repo that does the setup automatically. Worth it if you do this often.

Three gotchas worth knowing upfront

This is not all sunshine. Three things to watch out for.

Your editor does not always know what to do. If you have a workspace open in your main folder and you also open the hotfix worktree in the same editor instance, some IDEs get confused about which .git is which. I solved this by opening worktrees in fresh editor windows. Not a real problem, but worth knowing.

Submodules can be funny. If your repo uses submodules, each worktree needs to initialise its own submodule pointers. Read the man page section on this before assuming it will just work.

Tooling that hardcodes paths. Some build tools, some Docker setups, some test runners have absolute-path assumptions baked in. The first time you run them in a worktree at a different absolute path, things may behave oddly. Usually a small fix in the config. Just be ready for it.

None of these are dealbreakers. None of them have made me regret switching. But better you hear them from me than discover them at 2 in the morning during an incident.

A note on Graphite, because it deserves one

I mentioned Graphite earlier without explaining it. If you do not use it, the short version is that it is a tool for managing stacks of pull requests. When you have multiple small PRs that depend on each other, Graphite makes them feel like one coherent change instead of a logistics nightmare.

The combination of worktrees and Graphite is honestly the closest I have felt to having an actual second pair of hands. Worktrees give me parallel branches I can edit at the same time. Graphite gives me a way to review and ship those branches as a clean dependency chain. Together, they make the "many small focused PRs" school of working actually feasible, instead of the death-by-rebase it used to be.

I am not affiliated. I just like things that work.

Where to learn more

If this blog made you want to actually understand worktrees properly, here is the small reading list I would have wanted when I started.

The official man page. Honestly, just man git-worktree in your terminal, or read it online here. It is shorter than you expect.
The original announcement on the GitHub blog. Worktrees landed in git 2.5 way back in 2015, and the release post is still one of the clearest explanations of why this feature exists and what problem it solves.
Per-Erik Bergman's guide on Medium. If the AI angle in this blog is what hooked you, his guide walks the same arc from daily dev use to coordinating parallel agents, in more depth than I have given it here. One thing I will flag: he recommends nesting worktrees inside a gitignored .worktrees/ folder at the repo root, which I disagree with for the file-watcher and rm -rf reasons covered in the rules section above. Take the AI workflow ideas, skip the layout advice.
GitKraken's command walkthrough. The GitKraken page on worktrees is the cleanest "show me add, list, remove" reference I have come across. Skip the GUI parts if you live in the terminal, the command examples stand on their own.
Your own shell history. I am only half joking. After you have used git worktree add a few times, the muscle memory is the best teacher. Add a worktree to a throwaway repo today. Make a branch. Edit a file in it. Look at git worktree list. A few minutes of hands-on beats any blog post, including this one.

If you read just one of these, read the man page. It is genuinely the fastest path from "I have heard of worktrees" to "I cannot believe I lived without these".

The discipline part of the title

I called this a development discipline, not a trick. Let me explain why.

A trick is something you reach for occasionally. A discipline is something you build your workflow around, so that the right thing is also the default thing.

Worktrees only really pay off when you stop thinking of them as a tool for emergencies. They are how you organise simultaneous concerns. Feature in one. Hotfix in another. PR review in a third. AI experiment in a fourth. Each one has a desk. None of them step on the others. The cost of switching is cd, which is the cheapest thing your shell can do.

Once you operate this way, the old stash-checkout-rebase-pop dance starts to feel like something from a different era. Like writing CSS without a preprocessor. Or deploying without containers. The new way is so much calmer that the old way starts to seem actively user-hostile.

That is when I knew it had become a discipline and not a trick. When I stopped reaching for stash. When my default response to an interrupt became "let me spin up a worktree" instead of "let me save what I have in some fragile way I hope I can restore later".

If you take one thing from this blog, take that. Stop stashing. Start worktreeing. Your evenings will thank you.

That is pretty much it from my side today. Let me know what you think, or if you have been through this exact stash-rebase-pop horror and never want to go back to it. Those stories are always the best ones. Catch you in the next blog.

Why My One-Line Installer Worked Everywhere Except WSL

Vineeth N K — Fri, 15 May 2026 14:45:06 +0000

Why My One-Line Installer Worked Everywhere Except WSL

TL;DR: The application I work on used to take a new developer the better part of a week to set up. Some time back I added a Dockerfile and a docker compose setup, so the whole onboarding became one command. Then microservices showed up, port conflicts followed, and the README started to grow again. So I built a proper one-line installer. curl -fsSL https://app.our-product.com/install.sh | bash. Interactive, asks consent before installing missing deps, walks the user through port customisation, and uninstalls just as cleanly. It worked on every Mac. It worked on Linux. One developer on Windows tried it through WSL and got ./script.sh: 48: Syntax error: end of file unexpected (expecting "then") on a perfectly normal if block. The script was fine. The bytes were not. The trail led to PowerShell's curl, which is not curl, and a CRLF that snuck into every shell script in the pipeline. Strip the carriage returns at the top of the pipeline, or call curl.exe directly, and the installer behaves itself on every platform.

So here is the longer version, because this is really a story about onboarding, and the installer is just the last chapter of it.

A short history of setup pain

For a good while, getting a new developer up and running on our application was a small ritual. The system used to be a self-hosted one, with all the joy that brings. Install this version of the language runtime. Install this exact version of the package manager. Install the database, with these flags. Run these migrations. Apt this. Brew that. Then accept all the dependency licence agreements one by one. By the time you reached the login page in your browser, the better part of a workweek was gone.

I used to feel bad every time someone new joined. I mostly work remotely, so on the days I was on-site we would sit together at their desk with their fresh laptop, and on the other days we would slowly chew through the README over a Meet or a Huddle or a Teams call with screen-share, depending on which tool the team was using that quarter. Half the steps had silently rotted. The other half had hidden gotchas that only old hands knew. It was the kind of onboarding that quietly tells a new joiner "we do not really value your first impression". Not great.

So a while back I sat down and wrote a Dockerfile, and a docker-compose.yml, and a clear README on top of those. From that day on, new joiners ran one command.

docker compose up -d --build

Schema migrations were optional and documented. The application came up. The login page worked. Onboarding compressed from days into one afternoon. For a while that felt like the win.

And then microservices happened

Some time later, the codebase grew into more than one service. Then more than two. Each new microservice came with its own compose file, its own ports, and its own opinions about what a sensible host port mapping looks like. And when two services both wanted the same host port, the second docker compose up died loudly and the dev pinged me on Slack.

I have written about that whole port-conflict mess separately, if you want the longer story of how we ended up settling it. The short version was, we stopped baking host ports into the committed compose files and started using a small override convention. Read why I stopped arguing about Docker port conventions for the full take. But even with that fixed, onboarding had quietly slid back into a multi-page README again. New devs had to read a checklist of which services to clone, which ones to bring up, which ones their machine needed dependencies for. The "one command" promise had eroded.

So I sat down again.

The interactive one-line installer

The plan was simple. Bring the onboarding back down to a single line. But this time, account for the fact that we have multiple services, multiple ecosystems, and machines that are configured slightly differently from each other.

What I wanted was this.

curl -fsSL https://app.our-product.com/install.sh | bash

That is the entire user-facing command. Everything else happens inside the installer, interactively. The script does roughly this.

Detect the device. OS, architecture, available shells, whether Docker is installed, whether the user is already on a working setup.
Check the requirements. Walk through the list of things our stack needs. If any are missing, do not silently install them. Show what is missing and ask the user for consent, one by one. "Docker is not installed. Install it now? [y/N]". Same for Compose, same for the language runtime, same for the helper CLIs.
Walk through port customisation. Show the default host ports for each service. Detect conflicts on the user's machine. If a port is taken, suggest a replacement and let the user override. Write the chosen ports into the local override file.
Bring the stack up. All services, in the right order, with sensible defaults.
Print the URLs. "Open this in your browser, log in with these credentials." Done.

There is also a matching uninstaller that walks the reverse path. Stop the services, remove the containers and volumes, optionally remove the dependencies it installed earlier, leave the user's machine clean. The pair lives in the same repo, and the diff is the same shape as any other PR review.

I shipped this. Most of the team is on Mac. The application runs on Ubuntu 24.04 in production. New devs on Mac ran the one-liner, said yes a few times, picked their ports, and were on the login screen in one short coffee. Old devs ran the uninstaller and reinstalled clean. The README shrank to one line of copy-paste.

For a while it really felt like onboarding was solved.

The one developer on Windows

There was one holdout. One developer on the team is on Windows, and his microservices situation is genuinely different. The microservices stack on his end pulls in dependencies from a slightly different ecosystem, with its own package manager and its own setup steps. The Unix installer cannot do all of that work on a Windows host directly, because some of the tooling assumes a Unix shell underneath.

I did not want to leave him behind. The whole point of the installer was that everyone on the team gets the same easy ride. "Everyone except the Windows developer" is not a one-liner. It is a politely worded form of exclusion.

So I built a Windows wrapper. A small PowerShell script, install.ps1, that does the Windows-side preparation. Make sure WSL is enabled. Make sure an Ubuntu distro is installed inside WSL. Pull in the Windows-side toolchain that the microservices need. Then, once WSL is ready, the PS1 wrapper just delegates to the Unix installer inside WSL.

irm https://app.our-product.com/install.ps1 | iex

Run that in PowerShell. irm is Invoke-RestMethod and iex is Invoke-Expression. Together they fetch the PS1 from the server and run it in the current PowerShell session. The PS1 then sets the Windows world right. Then it reaches into WSL and runs the same Unix one-liner I shipped for everyone else. In theory, the Windows developer now lives the same life as a Mac developer. In practice...

The error that did not make sense

He pinged me with a screenshot. The terminal had this.

./cli.sh: 48: Syntax error: end of file unexpected (expecting "then")

Line 48. Line 48 was a plain if block. Three lines long. Looked like this.

if [ -z "$VERSION" ]; then
  VERSION="latest"
fi

Nothing fancy. No bashisms, no double brackets, just clean POSIX. The same lines were happily running on every Mac on the team and on the production Ubuntu fleet that same morning.

I asked him to run it again. Same error. Same line. Plain Ubuntu inside WSL, fresh install, all defaults.

And then he tried the other helper scripts. Same family of errors on every single one. Whichever shell script the PS1 wrapper ended up feeding into WSL, the parser choked on it. The pattern was suspicious. It was not one script. It was every shell script.

The wrong guesses I went through first

First guess. Old bash. Maybe WSL ships an ancient bash and if-then is being interpreted strangely. I asked for bash --version. Bash 5.1. Same as my Mac. Dead end.

Second guess. Shell mismatch. This was my most confident wrong guess. The one-liner pipes to sh, and on Ubuntu, /bin/sh is dash, not bash. Dash is much fussier about bashisms. So if a bashism had quietly slipped into the script, only the dash machines would choke on it. But the same script ran cleanly on the Ubuntu server. And I ran it through dash directly on a Linux box of mine. No problem. So this theory died too.

Third guess. Weird WSL distro. Maybe he had picked an Alpine variant or some musl-based thing where the system shell is mildly off. Turned out his default WSL distro was actually docker-desktop, which is the stripped-down distro Docker Desktop ships for itself. Not really meant to be a daily-driver shell. So we changed his default WSL distro to plain Ubuntu using wsl --set-default Ubuntu, made sure it was the fresh Microsoft Store one, and ran the installer again. Same error. Same line 48. So the distro was not the problem either, but at least now his terminal was a sensible place to live.

I had eliminated all the reasonable explanations. The bug was still right there.

Tell me I am not the only one who has been in this exact spot.

So I gave up on guessing and asked him for a screen-share session. Sometimes the bug is not what you imagine. Sometimes you have to watch it happen on the actual machine where it breaks.

The moment the truth dropped

Over the call, I asked him to skip the one-liner and instead download the script first, save it locally inside WSL, then run it.

curl -fsSL https://app.our-product.com/install.sh -o cli.sh
./cli.sh

Same error. So the network step was fine. The script content was the actual problem.

Then I asked him to run this.

cat -A cli.sh | head -5

cat -A shows hidden characters. Where a normal Unix line ends with $, a Windows line ends with ^M$. And the output that came back was full of ^M$. Every single line.

That is when it clicked. And once I saw it on his machine, I knew it was going to be the same story on every other shell script the PS1 wrapper had touched.

The script on his machine had Windows line endings. CRLF everywhere. then\r\n instead of then\n. To dash, and frankly to bash too, the word "then" followed by a carriage return is not the keyword then. It is a six-character soup that happens to look like the word "then" if you ignore the \r. The parser does not ignore the \r. It looks for an actual then, never finds one, walks off the end of the file, and reports "end of file unexpected (expecting then)" with the line number of the if that started the block.

The script was fine. The bytes were not. Something between the file on the server and the bytes that ended up inside WSL had decided to rewrite the line endings.

The actual culprit: PowerShell's `curl` is not curl

This is the part I want every dev to know, because it bit me cleanly.

In Windows PowerShell, curl is not the curl you think it is. It is an alias for Invoke-WebRequest. They are fundamentally different things. Real curl streams raw bytes from a URL to stdout. Invoke-WebRequest returns a structured PowerShell object with headers, status, body, and the rest. When you pipe that object onward, PowerShell stringifies it. And one of PowerShell's choices when stringifying is "use native Windows line endings, because we are on Windows".

The PS1 wrapper I had written did a lot of small things, but at the heart of it, for every shell script it had to pull from the server and hand over to WSL, it was effectively doing this.

curl -fsSL https://app.our-product.com/install.sh | wsl bash

Innocent looking. Reads exactly like the Unix one-liner. But the curl in there was the PowerShell alias, not real curl. The bytes that left it had been quietly converted from LF to CRLF on the way through. By the time bash inside WSL saw the script, every line ended in \r\n. Every if. Every then. Every case branch. And the helper scripts the installer kicks off internally have #!/bin/sh shebangs, which means they get executed by dash on Ubuntu. Dash is even less forgiving about then\r than bash. That is why the error in the screenshot was the dash-flavoured one.

The kicker is that none of us could have spotted this from reading either the PS1 or the shell script. Both files were fine. The transport was the problem. And the transport was lying about being curl.

The fix, in three flavours

I ended up shipping all three of these in different layers, because each one defends against a slightly different version of the same trap.

Flavour one. Tell PowerShell to use real curl.

Windows 10 and Windows 11 ship a real curl.exe. So the fix inside the PS1 wrapper is to bypass the alias and call the executable directly.

curl.exe -fsSL https://app.our-product.com/install.sh | wsl bash

That .exe is the whole difference. It tells PowerShell "no, do not give me your fake curl, give me the actual binary that ships with Windows". The bytes pass through unchanged. LF stays LF. The script runs.

This was the first thing I changed inside the wrapper.

Flavour two. Defend in the pipeline.

You cannot trust every future maintainer to remember the .exe. So I also changed the pipeline to strip carriage returns before handing bytes to the shell.

curl.exe -fsSL https://app.our-product.com/install.sh | wsl bash -c "tr -d '\r' | bash"

tr -d '\r' removes every \r byte from the stream. If the upstream curl was real curl, this is a no-op. If something later breaks and a CRLF sneaks back in from a different source, this quietly fixes it before the shell ever sees it. Belt and braces.

Flavour three. Defend inside the script.

For people who download the script first and then run it locally, which is the careful thing to do, the pipeline fix does not help them. So I added a small self-heal at the top of the installer itself.

#!/bin/sh
set -eu

if grep -q "$(printf '\r')" "$0" 2>/dev/null; then
  echo "Detected Windows line endings, normalising and re-running..."
  tmp=$(mktemp)
  tr -d '\r' < "$0" > "$tmp"
  chmod +x "$tmp"
  exec "$tmp" "$@"
fi

# rest of the installer below this

The first thing the script does is check itself for carriage returns. If it finds any, it writes a CRLF-free copy to a temp file and re-executes that copy with the same arguments. The user sees one extra line of output, and the install continues like nothing happened.

You can argue this is too clever for an installer. I would normally agree. But the whole job of an installer is to absorb platform weirdness so the user does not have to. If the cost of doing that is six lines at the top of the script, I will pay six lines every day of the week.

And one more, while we are here

I also added a .gitattributes rule to the repo, because the same trap has a sibling that bites at checkout time rather than at transport time.

*.sh text eol=lf
*.bash text eol=lf

This tells git that no matter what platform the repo gets checked out on, shell scripts get LF endings on disk. Windows machines with core.autocrlf=true, which is the Windows default, will still hand you LF for these files. It does not solve the PowerShell curl problem because that one happens in transport, not at checkout. But it stops a different version of the same trap from biting any future dev who clones the repo on the Windows filesystem and then tries to run scripts from inside WSL.

Same shape of bug. Different point in the pipeline. Better to defend both.

Where we landed

After the screen-share session ended, the Windows developer ran the PS1 one-liner again. WSL was already set up from the earlier failed attempt. PowerShell now used real curl. The pipeline normalised line endings just in case. The shell scripts self-healed if they ever saw a CR. All of his microservices, including the ones on the other ecosystem, came up. He saw the login page in his browser. The whole thing took a coffee, the same as everyone else.

He pinged me later that day to say it was the smoothest setup he had ever done on a Windows machine for a real engineering project. Coming from someone who has spent years working around the seams between Windows and Linux tooling, that mattered.

What I would have done differently

With hindsight, the very first thing I should have checked was line endings. Whenever a shell script behaves differently across platforms, and there is no obvious bash-versus-dash issue, the next thing to look at is the bytes. It is almost always line endings. I lost a good chunk of an afternoon to version checks and dash compatibility tests before I got there.

I also should have written the PS1 wrapper with curl.exe from day one, instead of using whatever curl happened to resolve to in PowerShell. The alias is a footgun and the fix is six characters.

The bigger lesson though is one I knew but had not really internalised. In a "modern one-line installer", the line that does the most work is not the line that runs the install. It is the line that gets your script's bytes from the server to the user's shell without corruption. That step is invisible. That step also has the most ways to silently go wrong, and it does not care that the script is correct. If the bytes are off by one carriage return, all the careful code in the world will not save you.

So now the installer assumes nothing about the transport. It uses real curl. It strips \r in the pipeline. It normalises itself if it sees CRLF inside. And the repo carries a .gitattributes rule for good measure. The Mac devs are unaffected. The Linux servers are unaffected. The Windows developer has the same one-command onboarding as the rest of the team.

Not going to pretend this was a perfect writeup. But if even one part of it helped some other developer avoid the afternoon I lost, then it was worth putting down. See you in the next one.

How I ended up buying vinelabs.de

Vineeth N K — Sun, 10 May 2026 17:47:53 +0000

How I ended up buying vinelabs.de

TL;DR: I bought vinelabs.de last weekend. Was not planning to. The trigger was the author field of a manifest file, the same kind you fill into a composer.json, a package.json, a Cargo.toml, or whatever your stack of the day calls it. The realisation was that shipping serious packages under my personal GitHub username reads like a hobby for code that will sit in someone's finance pipeline. Trust problem, not a code problem. So I bought a domain. Set up an org. Built a small landing site. Here is the short version.

So here is what happened. I was in the middle of finishing up xrechnung-kit, which started as a small Shopware plugin and grew into a monorepo with eight packages. I have already written about that one separately, so if you want the long story you can find it here.

But the boring scene that mattered was this. I was filling in the manifest files for the Shopware sibling package and the small Astro showcase site that was going to live next to it. So the composer.json for the PHP package on one side, the package.json for the site on the other. I got to the author block, and I paused. The whole list of packages at that point was going to live under vineethkrishnan/xrechnung-kit-* on Packagist, and the showcase site under my personal GitHub username too. All in my personal namespace. For a library that will sit inside finance and accounting pipelines, the vibe of "github.com/vineethkrishnan/anything" reads as hobby. Even if the code is solid. Even if the tests pass. The address itself does the talking before the code gets a chance to.

That was a trust problem, not a code problem. I needed a brand.

If you have ever flinched while writing your own name into a composer.json, a package.json, or whatever manifest your stack uses, for a package you actually want people to take seriously, you know exactly what I mean.

The shortlist that did not happen

I sat for a bit with name options. The first instinct was, of course, .com. Tried vinelabs.com. Already taken. Looked at vinelabs.io and vinelabs.app next, the standard "labs" fallbacks people reach for.

But .de had been in the back of my head the whole time, and I will tell you why.

I have been working in German work culture for a long while now. Handled many .de domains across many German shops. Shopware itself is German-scoped. The first XRechnung use case is German. EN 16931 is a EU thing, but XRechnung 3.0 is a federal German standard. If the projects I am putting under this brand are going to focus on the DE and EU region, which they will, then .de is not a quirky choice. It is the home address.

So vinelabs.de. Bought it.

What I set up

The bare minimum to make a brand feel real, in order:

The org github.com/vinelabs-de. This is where the public-facing repos live.

Two mailboxes, info@vinelabs.de and support@vinelabs.de. Forwarded to where they need to go. Nothing fancy.

A small landing site, Astro 5 + Tailwind v4, deployed to Cloudflare Pages. The site is driven by a markdown content collection at src/content/projects/. Every project I want to showcase is one markdown file with a tagline, a description, a license, and a few highlights. New project equals new file. There is no CMS, no admin panel, no database. I keep saying this about Astro to anyone who will listen, but Astro continues to be unreasonably nice when you do not need a backend.

Why now, and why DE

The timing is not accidental. Germany is right in the middle of phasing in mandatory B2B e-invoicing. The receive-side mandate is already live, and the send-side mandate is rolling out behind it. EN 16931 / XRechnung 3.0 is what has to come out the other end. A small library that does that correctly, sitting under a brand that is clearly in the DE / EU lane, has a place.

I should also be clear about who I am here. I am an Indian developer, not a German one. I have been working with German teams and German shops for a long time, picked up a fair bit of the working culture, handled enough .de domains and Shopware shops to feel at home in this stack. But I am not pretending to be local. The brand is in the DE / EU lane because that is where the work is, not because I am putting on a costume.

The mirror trick

Here is the part I am quietly pleased about. I did not want to actually move my repos out of my personal GitHub account. That account has my history, my issues, my CI configurations, my settings. I did not want a hard fork, a rename, or a redirect.

So I wrote a tiny workflow template, mirror-to-vinelabs.yml. Lives in a workflow-templates/ folder. I drop it into any of my personal repos, and on every push to main it syncs that repo into the vinelabs-de org.

My personal repo stays the source of truth. The labs org stays the public face. If I ever pull out of the labs branding, it costs me nothing because the canonical code never moved. It is already wired up for xrechnung-kit. vaultctl is next, then probably a couple of the smaller tools that have outgrown my personal username.

The honest part

I do not have a roadmap. There is no team. There is no monetisation plan. No funding round, no big launch.

The labs domain exists because I would rather under-promise on a brand than over-promise on my own name. xrechnung-kit deserved a home that says "this is built to be used", not "this is what one developer made on a long weekend." It did start on a long weekend. What it is not going to stay is a weekend project. I plan to maintain it like something that has to keep working.

The V is a stem. Everything else is what grew off it.

Alright, that is me done rambling for today. Hope something in here was useful to you. Catch you in the next blog, take care until then.

The disk that filled itself

Vineeth N K — Thu, 07 May 2026 15:44:29 +0000

The disk that filled itself

TL;DR: my homelab box hit 100 percent disk full out of nowhere. I deleted half the things I could find, df still said full, du said I had plenty of space. Turned out the disk was holding on to files I had already deleted, because a long-running process still had them open. lsof +L1 was the magic. A service restart was the fix.

So there I was, on a perfectly normal evening, ssh'd into the homelab box because something had stopped responding. The first thing I check on any "why is this dying" run is df -h, almost as a reflex.

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  450G  448G   2G  100% /

Cool. So that is why nothing is working.

I have a deal with this box. It runs my self-hosted things, it does not ask for much, and once a quarter or so I prune some old container images and we move on. So I went straight to the usual cleanup playbook, mildly annoyed that I had let it fill up.

docker system prune -a --volumes
journalctl --vacuum-size=200M
apt clean
rm -rf ~/.cache/*

Felt good. Watched the percentages tick down in du as I went. Ran df -h again, full of optimism.

/dev/nvme0n1p2  450G  448G   2G  100% /

Excuse me?

When df and du disagree

I went and added it up the long way. du -sh / took its time, came back with about 130G used. Big folders identified, nothing weird. Half the disk should have been free.

But df sat there, smug, telling me I had two whole gigabytes of breathing room. Same disk. Same minute.

This is the moment in any disk-full story when you realise the problem is not actually the disk. It is who is asking.

If you have hit this exact mismatch before, you already know where this is going. If you have not, here is the thing that took me longer to internalise than I want to admit: df and du are not measuring the same thing.

du walks the directory tree. It adds up files it can see, file by file. If a file is not in some directory, du does not know it exists.

df asks the filesystem itself how many blocks are in use. The filesystem does not care about directories. It cares about which blocks have been handed out to a file, any file, anywhere.

Most of the time these two views agree. The interesting case is when they do not. And the most common reason they disagree is files that are not in any directory but are still very much being used.

The deleted file that is not deleted

In Linux, rm does not actually delete a file. It just removes the entry from a directory. The file's data only goes away when the last process holding it open lets go.

Which means: if a process has a log file open, and you rm that log file, the directory entry is gone, du cannot see it, your file browser shows it as deleted, you are happy. But the process is still writing to it. The blocks are still held. df is still counting them.

Until that process closes the file or dies, those bytes are real, just invisible.

This is the part of Linux that feels like a magic trick once you see it. lsof exposes it directly.

sudo lsof +L1

+L1 means "show me files with a link count less than 1", which is exactly the deleted-but-still-held case. I ran it expecting maybe a couple of stray MB. The output was a wall of text. The same process kept showing up, holding a frankly embarrassing number of "deleted" files.

The culprit was not exotic. It was the docker daemon, sitting on a container's json-file log that had ballooned to hundreds of gigs across the time the box had been running. Some time back, in a cleanup session I do not really remember anymore, I had rm'd that log file directly, thinking I was reclaiming space. Docker had no idea I had done that. The file was gone from disk as far as I was concerned. Not gone from docker's open file descriptor.

So every byte that container had been logging since that day, plus every byte before, was still there. Held. Counted by df. Invisible to du.

Tell me I am not the only one who has done this exact "smart" cleanup move and quietly made it worse.

The fix, and the not-fix

The actual fix was embarrassing in its simplicity.

sudo systemctl restart docker

That is it. The daemon restarted, every file descriptor it was holding got closed, every "deleted" file finally got a chance to be properly deleted, and df was suddenly back to a sensible number.

The not-fix, the thing I should have done in the first place to avoid this whole thing, would have been to never rm an active log file. The right move on a docker container log is to truncate it through the existing file descriptor.

truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log

truncate writes through the file descriptor instead of unlinking the directory entry. Docker keeps writing. Disk space comes back. Nobody gets confused.

Or, even better, configure the json-file log driver with max-size and max-file so it rotates itself and you never have this conversation.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  }
}

That goes in /etc/docker/daemon.json, you restart the daemon once, and then this whole class of bug stops being a thing on that box.

The tools I built so I do not have to do this manually again

After this exact kind of incident, and the embarrassing number of du -sh /* sessions that came before it, I went and built a few small things to take the manual labour out of disk-full nights. They are the tools I now reach for before I touch anything by hand.

dfree is the first one I run. It is a shell script. No arguments, no flags to remember. It scans the disk in a few passes and shows me what is taking space across docker, system caches, dev caches, and logs. Same playbook I tried to do by hand at the start of this story, except it adds the numbers correctly and shows me the docker side first.

$ dfree

=== System Analysis ===

[INFO] Scanning disk usage...
500G 448G 2G 100%

[INFO] Scanning Docker usage...
Images: 18.2GB (12.4GB reclaimable)
Containers: 287GB (281GB reclaimable)
Build Cache: 4.1GB

[INFO] Scanning Developer Caches...
  - /home/vineeth/.cache: 480MB
  - /home/vineeth/.npm/_cacache: 1.1GB

[INFO] Scanning Logs...
  - /var/log/journal: 320MB

Look at the docker line. Containers: 287GB (281GB reclaimable). On the actual night this happened, I could have read that one line and known exactly where the trouble was, without going on a find expedition. After the analysis, dfree asks me one item at a time what I want cleaned, and I say yes or no.

=== Cleanup Process ===

Prune Docker system (images, containers, networks)? [y/N] y
[INFO] Pruning Docker...
Total reclaimed space: 12.4GB

Clean system cache at /var/log/journal? [y/N] y
Clean developer cache at /home/vineeth/.npm/_cacache? [y/N] y

[SUCCESS] Cleanup complete.

For when a flat list is not enough and I want to actually see the shape of the disk, I built diskdoc, a Rust TUI that walks the filesystem in parallel and lets me browse the result like a tree. Useful when the offender is buried somewhere weird and I want to wander through the directory structure instead of reading a summary. It is not what saves you on the night of. It is what saves you the third time you keep ending up in the same neighbourhood and want to understand why.

But the tool that would have actually short-circuited this whole post is dockit, a Go CLI that talks to the docker daemon directly. It has a logs subcommand built for this exact failure mode.

$ dockit logs
Finding container log paths on disk...

--- CONTAINER LOG SIZES (Total: 287 GB) ---
CONTAINER            SIZE            WARNINGS
notes-app            287 GB          🚨 EXCESSIVE - Consider adding 'log-opt max-size=10m'
nextcloud            42 MB
gitea                8.3 MB
media-server         2.1 MB

That first row is the entire war story compressed into one line. One container, no rotation, hundreds of gigabytes of json sitting on disk, and the tool literally tells me what to do about it. If I had been running dockit logs on a cron and getting a ping when any single container crossed a sensible threshold, none of this would have happened. The investigation would have been "fix the log driver config" months ago, not "why is my disk lying to me" at midnight.

If you want the tools, all three are open source:

dfree: github.com/vineethkrishnan/dfree
diskdoc: github.com/vineethkrishnan/diskdoc
dockit: github.com/vineethkrishnan/dockit

Two lessons I keep relearning

df and du measure two different worlds. When they agree, life is easy. When they disagree, the answer is almost always "something is being held open". lsof +L1 is the single command that tells you exactly what. I have probably typed it a hundred times in my career and I still forget it exists for the first stretch of every disk-full incident.

rm on an active log file is a trap. It looks like cleanup. It is actually just hiding bytes from du while the process keeps appending to invisible disk. Use truncate if the process supports being truncated under it, signal the process to reopen its log if the app supports that, or rotate properly with logrotate or the platform's native rotation.

Early on in this incident, I was completely sure I had simply not deleted enough stuff yet. I was a few minutes away from ordering another drive. The fix was a service restart, and the cause was a rm from months ago that I had thought was helpful at the time.

If you have an old box with self-hosted things on it and you have ever cleaned up a "huge log file" by deleting it directly, today is a good day to run sudo lsof +L1 and see what your processes are still holding. Worst case you find nothing. Best case you find a sizeable chunk of your disk waiting to be freed.

Closing

The thing that bothers me about this kind of bug is not the bug itself. It is that I had a wrong mental model of rm for years and never really noticed, because most of the time the wrong model and the right model produce the same result. The penalty only shows up at the edges, in long-lived processes with open files, on a box you have neglected for long enough that you forget what you did last summer.

So that is where I will stop. If you have a different way of catching this kind of thing earlier, or a cleaner way of dealing with active logs on a homelab box, I genuinely want to hear it, drop me a note. Otherwise, see you when the next interesting problem shows up.

MCP is the USB-C of AI tools, and most devs are still using their AI assistant like it is 2023

Vineeth N K — Thu, 07 May 2026 13:17:44 +0000

MCP is the USB-C of AI tools, and most devs are still using their AI assistant like it is 2023

So here is a small thing I noticed the other day. I was watching a friend debug a production issue, and the workflow was painful in a very specific way. Tab to their AI chat of choice, paste an error. Read the answer. Tab to Sentry, copy the stack trace. Tab back to the chat, paste the stack trace. Tab to the codebase, copy the function. Paste it again. Repeat until coffee gets cold. It honestly does not matter which AI they were using. ChatGPT, Claude, Codex, Gemini, take your pick. The flow was the same.

The whole thing felt like watching someone use a phone in 2010. Functional. Slow. And clearly a generation behind something that already exists.

That is the gap I want to talk about today. Because there is a very real protocol shift happening in AI tooling right now, and most developers are completely unaware of it.

The cable drawer in your house

Open the drawer where you keep your old chargers. Go on, I will wait.

If you are anywhere over thirty, you probably have a small museum in there. Mini USB. Micro USB. The old Apple 30-pin. Lightning. That one weird Samsung cable that nobody can identify. A barrel charger from a router you threw away in 2014. Each one was the only way to talk to a specific device. Each one was useless for anything else.

USB-C did not appear and instantly fix the world. It just slowly became the one cable that worked for everything. Laptop, phone, headphones, monitor, the toothbrush my wife uses, my Kindle. One connector. No drawer.

AI tooling is going through the exact same moment right now. Most people have not noticed.

The drawer of integrations

For the last couple of years, every AI integration was its own custom cable.

You wanted your AI assistant to read your Notion? Cool, here is a custom plugin that runs on that vendor's plugin system, with its own auth, its own schema, its own quirks. You wanted a different model to query your database? Different system. You wanted to do something with Slack? Build a function-calling wrapper, write the schema by hand, host it somewhere, deal with the auth yourself. You wanted to switch from ChatGPT to Claude, or Claude to Codex, or any of them to a local model? Throw all of it away and start over.

Every "AI integration" was bespoke. Every developer who built one had to figure out the same five problems from scratch. Auth. Schema. Transport. Tool descriptions. Error handling. Five problems times one hundred SaaS tools times five model vendors gives you a number that should have scared us all.

And then a small thing called the Model Context Protocol showed up and said: what if this was just one shape?

What MCP actually is

I will keep this short because the spec is honestly not that interesting and you can read it later if you want.

MCP is a protocol. Your AI client (Claude, ChatGPT, Codex, Gemini, Cursor, whoever) speaks one shape. Any tool, any service, any local script can implement that shape and the client can talk to it. The client does not care if it is reading from Notion, posting to Slack, querying Postgres, or running a Playwright browser. They all expose the same kind of interface. Tools, resources, prompts. That is basically the whole story.

The cleverness is not in the protocol design. The cleverness is in the agreement. Anthropic shipped it. OpenAI adopted it. The big SaaS companies started writing servers for their own products. Atlassian has one. Figma has one. Slack has one. Notion. Vercel. Gmail. Google Calendar. Playwright. The list is now embarrassing in length.

It is the same thing USB-C did. Not a technical breakthrough. A standardisation moment.

What this looks like in practice

Here is what my actual day looks like now, and I want to be honest, this is the part that took me a while to internalise.

When something breaks in production, I open my editor. I do not open Sentry. I do not open Notion. I do not switch tabs. I just say something like, "pull the latest unresolved issue in the api project, show me the stack trace, and tell me which file it points to". The agent calls the Sentry MCP, gets the issue, reads the file from the codebase, and tells me where the bug is. Sometimes it offers a fix. Sometimes I tell it to write the fix and resolve the issue. The whole loop, including writing the patch and closing the ticket, lives in one window.

And that is for one tool. The same agent, in the same session, can also pull a Linear ticket, check a Figma frame, post an update to Slack, query a Postgres database, and run a quick Playwright test against staging. All without me leaving the editor.

Compare that to the friend I mentioned at the start. Tab to chat, paste, copy, paste, copy. Same problem. Different decade. And again, it is not about which AI tool they picked. ChatGPT, Claude, Codex, Gemini, all of them now speak MCP or are in the process of adding it. The bottleneck is not the model. The bottleneck is whether you have actually plugged anything into it.

Tell me I am not the only one who finds this gap funny.

I built a thing because I felt the pain

A while back I started building MCP servers for the SaaS tools I actually use at work. It started with one. Then two. Then before I knew it I had eleven of them, plus a shared OAuth library, plus a docs site, plus a Docker setup so they would show up properly in the public registries. The repo is called mcp-pool and I wrote a whole separate post about how it grew, so I will not retell that story here.

The thing I want to point out is that the painful part was never writing the servers. The SDKs are decent. The protocol is small. You can scaffold a basic server in an afternoon if you have done it once before.

The painful part was running them. Six different Node processes on my machine, each one with its own config file, each one needing its own auth token, each one occasionally crashing for no reason and silently disappearing from the agent's tool list. That is the part nobody warns you about. Once you have more than two or three MCP servers, the operations side starts to look a lot like running a small fleet of microservices on your own laptop. Which, when you put it that way, is kind of an absurd thing to be doing.

But that is the price of being early. Same way the first USB-C laptops needed three dongles in your bag. The protocol was right. The ecosystem was still catching up.

The 2023 dev versus the 2026 dev

So here is the bit I keep coming back to.

The 2023 developer treats the language model as a smarter Stack Overflow. You type a question. You read the answer. You copy something out. You paste it into your code. Your context lives in the chat window. The model has no memory of your repo, your team, your tools, your tickets, your design files, your runbooks, anything.

The 2026 developer treats the language model as the centre of a small workshop. The model has access to the actual systems. It can read the ticket. Open the file. Run the test. Check the design. Post the update. Close the ticket. The dev is no longer copy-pasting context in. The dev is just describing what they want done, and the agent is fetching, reading, deciding, writing.

This is not about AI being smarter. It is about AI being plugged in.

And I would gently suggest that if you are still in the first group, you are leaving an embarrassing amount of productivity on the table. Not because you are bad at your job, but because you are using a 2023 workflow on a 2026 toolchain. Same way someone might still be charging their phone with a cable they keep in a drawer with seven other cables.

The bit nobody is putting on the marketing slide

So far this post has been mostly cheerful. A new protocol, a nicer way to work, a cable drawer that finally got cleaned up. Honest moment now.

Plugging more tools into your AI assistant is also plugging more attack surface into your daily workflow. The MCP ecosystem has had a genuinely rough run on the security front, and if you are about to install a few servers this weekend, you should know what has actually happened in the last year before you do it.

A short and very much not comprehensive list of real incidents (the authzed MCP breach timeline has the fuller version, and is what I cross-checked these against):

April 2025, WhatsApp MCP: a tool-poisoning attack disguised a backdoor as a legitimate server and quietly exfiltrated chat histories.
May 2025, GitHub MCP: a prompt injection in a malicious public issue hijacked the agent into leaking private repository contents, using a token whose scope was way too broad.
September 2025, Postmark MCP: a trojanized package on a public registry was BCC-ing every email it handled to attacker infrastructure.
October 2025, Smithery Registry: a path traversal bug exposed builder credentials and compromised thousands of hosted MCP servers in one go.
April 2026, core MCP STDIO design flaw: an architectural decision in Anthropic's official SDKs that, depending on who you read, exposes upwards of a hundred and fifty million downloads across Cursor, VS Code, Windsurf, Claude Code and others.

And right next to this, a related incident that was not strictly an MCP breach but is exactly the pattern you should be watching for. In April 2026, Vercel disclosed that an employee was compromised through Context.ai, a third-party AI tool that held a Google Workspace OAuth app with broad permissions. Malware on the AI vendor's laptop, then OAuth pivot, then into Vercel customer environment variables (TechCrunch and Trend Micro have the cleanest writeups). Not MCP-specific. But the shape is exactly the shape MCP makes more common.

The pattern across all of these is the same. An AI tool sits in the middle of your stack, holding tokens that reach into your real systems. If that tool is malicious, vulnerable, or just sloppily run, the blast radius is whatever those tokens can reach. And tokens for "read my Notion" or "post to Slack" are not low-privilege things in 2026. They are basically the keys to an entire workspace.

How to actually check if an MCP server is safe for you

This is not a perfect checklist. It is the rough rubric I run before I install a server. Steal it, sharpen it, throw it away, whatever works.

Who publishes it. Is the server from the SaaS vendor whose API it wraps, from a known community maintainer, or from a username you have never seen before? Vendor-official is safest. A maintainer with a real track record is fine. A brand new account with one package and no GitHub history is a hard no.
Read the source. Most MCP servers are small. Cloning the repo and skimming the tool list takes a few minutes. Look at what tools are exposed, what their descriptions actually say, and whether anything is doing something the README does not mention. Tool poisoning lives in exactly this gap.
Check the dependency tree. A small wrapper with two hundred transitive dependencies is a very different risk profile from a small wrapper with five. Shorter is better.
Token scope, ruthlessly. When you generate the token the server will use, give it the smallest set of permissions that gets the job done. Read-only beats read-write. Single-project beats organisation-wide. Single-channel beats whole-workspace. Never reuse a token you already use somewhere else.
Run it locally, not on a hosted gateway. Hosted MCP gateways are convenient. They are also a single point at which someone else is holding your credentials. If a server can run as a local stdio process on your own machine, prefer that.
Read-only first, write tools opt-in. If the server supports read-only mode, start there. Only enable write tools after you have used it long enough to trust both the server and how the agent behaves with it.
Watch for updates that change tool descriptions. This is one of the sneakier attack patterns. A server you trusted last month silently expands its tool descriptions in this week's update to include something new and harmful. Pin versions if you can.
Check the registry verification badges. Glama and the official MCP registry now flag servers that have been smoke-tested. Not perfect signal, but a server with zero badges, zero stars, and no recent commits is at least worth a second look.

If a server fails most of these, do not install it. If it fails one or two, decide whether the convenience is worth it for your specific situation. None of this is paranoia. It is the same hygiene most of us already apply to npm packages, just adapted to a newer ecosystem that is still figuring out the basics.

What I would tell a friend

If you read this far and you are wondering whether to bother, here is what I would actually say to a friend over coffee.

Pick one tool you use every day. One. Sentry, Notion, Linear, Slack, your database, whatever. Find an existing MCP server for it on GitHub, or look at the official ones from Anthropic, or check mcp-pool if any of those line up with your stack. Run the safety checklist above before you install. Then wire it into Claude Desktop or Claude Code or your client of choice. Spend a single evening doing this and nothing else.

The first time you say "summarise the last five Sentry issues from this morning" and an actual answer comes back, with real data, from the real system, you will get it. The shift will feel obvious in hindsight. You will wonder how you spent so long copy-pasting things into a chat box.

That is basically the whole point of this post. Not "MCP is cool". Not "here are the seven best servers to install today". Just: a thing has changed, and most people I know in tech have not yet noticed it has changed. Which is normal. Standardisation moments are always quiet. The drawer of cables does not announce itself. One day you just notice you have not opened the drawer in years.

Closing

If your AI workflow today involves a lot of tab switching and copy-pasting, that is the cable drawer. It is fine, it works, it is not broken. But there is a different way of doing it now, and the gap between the two is going to keep widening every month as more SaaS companies ship MCP servers for their products.

You do not have to rush. Nobody is keeping score. But it might be worth at least poking at one server this weekend, just to see.

That is all I had on this one. If you made it till here, thank you, genuinely. See you in the next one, where I will probably be complaining about something else that broke.

The webhook that worked in Postman and nowhere else

Vineeth N K — Mon, 04 May 2026 11:38:41 +0000

The webhook that worked in Postman and nowhere else

TL;DR: an app I work on was firing webhooks at a third-party device API. The receiver kept returning 401. Postman, with the same payload, got 200 every time. The cause was not signing logic, not auth, not network. The app had two completely different bootstrap paths, the secret-loading config was wired into only one of them, and a silent-skip guard quietly hid the real failure under a misleading 401.

So there I was, staring at a wall of 401 responses in the logs. The app was firing webhooks at a third-party device API every time something on our side changed state. Every single one was bouncing back as "unauthorized".

Fine, must be the signature. I copied the raw request body straight out of the logs, dropped it into Postman, signed it the same way the app does, and fired it at the same URL. 200 OK. First try.

So Postman was happy. The app was not. Same payload, same URL, same headers (so I thought), and yet only one of them was getting through.

If you have ever been in this situation, you know the feeling. There is no Stack Overflow post for "works in Postman, fails from my own app". You have to walk yourself through it.

First, rule out the obvious stuff

I went through the standard checklist before doing anything clever.

Same URL? Yes, copy-pasted from the same config.
Same body? Yes, byte for byte.
Same auth header? Yes, same shared secret loaded from the same env file.
Time skew? The timestamp inside the signature was within a few seconds of the receiver's clock.
IP whitelist? No, the receiver does not even check the source IP.

So on paper the two requests were the same. The receiver clearly disagreed. Which meant I had to see what the app was actually putting on the wire, not what I thought it was putting on the wire.

The diff that made the cause obvious

I added a logger that dumped the full outgoing HTTP request right before the dispatch: method, URL, every header, body. Then I triggered an event from the app and let it fire. Side by side with the Postman request:

Postman                              App
-----------------------------------  -----------------------------------
POST /webhook                        POST /webhook
Content-Type: application/json       Content-Type: application/json
X-Signature: sha256=a3f4...e991      X-Signature:
User-Agent: Postman                  User-Agent: GuzzleHttp/...
{"event":"door.unlocked",...}        {"event":"door.unlocked",...}

Look at the second-to-last line on the right. The app was sending the X-Signature header. The value was just an empty string. Postman had a signature, the app had nothing.

That was a relief in a small, sad way. At least there was something to find.

Why is the signature empty?

Easy enough to check. The dispatcher looked roughly like this:

function dispatch(event, payload):
    secret = config.get("device_api.signing_secret")
    if secret is empty:
        // skip signing, send anyway
        send(payload, headers={})
        return
    signature = hmac_sha256(secret, payload)
    send(payload, headers={"X-Signature": signature})

Two things wrong here, but bear with me.

I dropped a log line on the secret = ... line. The value came back null. At runtime, in the queue worker's process, the signing secret was just not there.

But the same config file. The same env. The same code reading from the same key. Why was it empty in the worker and full in the HTTP layer?

Has this happened to you also, where two parts of the same app behave like they live in different universes? Welcome to bootstrap drift.

Two doors that look the same from the outside

The app, like a lot of older codebases, has more than one entrypoint. There is the HTTP entrypoint that serves the website, the API endpoints, anything that comes in over a request. And separately there is a queue worker entrypoint that handles background jobs: sending mails, replicating data, dispatching webhooks (yes, that webhook).

Both entrypoints share most of the codebase. They both load the same config files. They both connect to the same database. From the file tree, they look identical.

But they boot through different paths. The HTTP entrypoint has its own bootstrap routine. The queue worker has its own. And somewhere along the way, the config that loaded the third-party device API secret had been added only to the HTTP entrypoint's bootstrap.

When a request came in over HTTP, the bootstrap ran, the secret got loaded, the dispatcher had what it needed. Tested manually with Postman replay against the HTTP entrypoint? Worked, because Postman was hitting the side that had the config.

But the actual production trigger was a queue job. The job ran inside the queue worker process, which booted through the other path, which never loaded that config. So config.get("device_api.signing_secret") came back null. Every single time.

The two entrypoints had drifted apart. Whoever added the config load had put it where they could see it being needed (the HTTP layer, where the test was easy), and nobody noticed that the queue worker was also calling the same dispatcher.

The second bug: the silent-skip guard

Look at the dispatcher again:

if secret is empty:
    // skip signing, send anyway
    send(payload, headers={})
    return

That comment is the second crime scene.

When the secret was missing, instead of throwing an error, the dispatcher quietly stripped the signature header and sent the request anyway. So the receiver, who is doing what every signed-webhook receiver does, saw an unsigned request and answered 401.

From the outside, what we saw was: webhooks fail with 401. The obvious assumption is that the signature is wrong. We spent a good while looking at HMAC code, hashing algorithms, payload encoding, header casing. All of that was fine. The bug was four layers up the stack from where the symptom was showing.

If the dispatcher had just thrown a loud MissingSecretError: device_api.signing_secret is null, the cause would have shown up the very first time a webhook tried to fire. Instead it whispered "no signature, oh well", and the receiver did the polite thing and rejected it. Two pieces of code, each individually being defensive, together producing a misleading symptom.

The fix, and the meta-fix

The local fix was a one-liner. Move the config load into the shared bootstrap that runs for every entrypoint. Now every process that boots, whether HTTP, worker, CLI, or cron, has the secret loaded by the time anything else runs.

The meta-fix was the silent-skip guard. I changed it to throw if the secret is missing in any non-test environment. If somebody, some day, manages to start a worker process without that config loaded, I want it to crash on the first webhook attempt with a useful error, not soldier on producing 401s for hours.

if secret is empty:
    if env != "test":
        throw MissingSigningSecret("device_api.signing_secret")
    // tests can opt in to unsigned mode
    send(payload, headers={})
    return

Took maybe ten minutes to write. The bug had been confusing me for a good chunk of the day.

Two lessons I am writing on the wall

Cross-cutting config belongs in the shared bootstrap, not in the entrypoint-specific one. If a piece of config is needed by code that runs in more than one process type, the only safe place to load it is somewhere all of those processes pass through. Not the HTTP bootstrap. Not the worker bootstrap. The one underneath both. Otherwise you are building two apps that pretend to be the same app, and they will eventually disagree.

Silent-skip guards turn loud failures into quiet ones. If a value being missing is going to make the next operation meaningless, do not paper over it. Throw. The sound of a real error in a dev environment is so much cheaper than the silence of a wrong-but-running production. There are exceptions, where degrading gracefully is genuinely the right answer. But the default should be loud, and "quiet on missing config" is almost never the right answer.

If you have hit this kind of bootstrap drift in your own apps, I would love to hear how you spotted it. Mine was pure luck. The request logger I added was actually for an unrelated thing, and I noticed the empty header by accident. Without that I might still be reading HMAC source somewhere.

Closing

Looking back, this whole thing was less about webhooks and more about how easy it is for two parts of the same app to grow apart without anyone noticing. The codebase looks like one app from the file tree. It runs as two different apps from the operating system's point of view. That gap is where bugs like this live.

If your app has more than one entrypoint, today is a good day to grep for bootstrap and check whether all of them are setting up the same world.

That is pretty much it from my side today. Let me know what you think, or if you have been through something similar, those stories are always the best ones. See you soon in the next blog.