DEV Community: Hoelee

Why I Still Bought a GPU to Run AI When Claude and GPT Are Obviously Stronger

Hoelee — Sat, 14 Nov 2026 04:25:40 +0000

Originally published on my blog — read it there with the full context, code and updates.

Across the self-hosted production infrastructure I manage — Docker stacks, NAS
storage, tunnels, and automated cron pipelines — I still pay for frontier models.

So when people find out I run local models too, the question is always the same:
Claude and GPT are stronger. Local models are a pain to set up, slower, and
often just worse. So why did you spend money on a GPU?

If you are only comparing raw model capability, I agree with you completely.
When I hit a genuinely new problem, do complex reasoning, plan a product
architecture, or kick off a long agent run, I still reach for Claude or GPT
first. I did not deploy local models because I think they have beaten the
closed frontier.

I did it because of a different problem that only showed up after AI got good
enough to depend on. Everything I built started sitting on a service I do not
control.

The problem is not capability — it is dependency

How many tokens you get, what a plan costs, which regions can buy the commercial
service, when the model version changes under you — none of that is the user's
decision. Anthropic has regional restrictions of its own; some markets cannot
buy Claude's commercial service directly. OpenAI's plans, quotas and model
lineup keep shifting. These are commercial services, so of course they change.

That was fine when ChatGPT was something I opened occasionally to ask a
question. If it was unavailable today, I would ask tomorrow and lose nothing.

It stopped being fine once I started using AI to write code, run agents,
analyze documents, handle support, organize company data, and wire models into
workflows that run every day. When AI moves from chat tool to infrastructure,
the question changes. It is no longer "which model is smarter?" It becomes:

"If I depend on this model today, and its terms change tomorrow, does my work
still run?"

That is why I think the stronger the closed models get, the more valuable a
local model becomes. Precisely because they are so good, people hand them more
and more work. And the deeper the dependency, the more of your operation someone
else controls.

There is a second-order version of this that rarely gets said plainly. Every big
lab talks about AI safety, model risk, and restricting certain content, regions
or use cases — while all of them keep racing to make models stronger, because
capability is the competitive advantage. Nobody stops, because if you stop,
someone else does not.

So what actually gets controlled is rarely whether AI keeps developing. It is
who may use it, how, how much, and how far the vendor is willing to go. For an
individual, that reads like a terms-of-service page. For a business, it is
supplier risk. If your core process and every agent, tool and data pipeline
around it are pinned to one vendor, you are not just buying tokens — you are
handing over part of the steering wheel.

"But local models are worse" — yes, and that is the wrong comparison

This is the most honest objection, and it is true. Put a model that fits on a
24GB or 32GB consumer card against Claude Opus and it loses on complex tasks. No
argument.

But the comparison most people run is wrong. The question is not can Qwen beat
Claude. It is:

Is the capability gap large enough to justify using the most expensive model
for this task, forever?

Because the trade is not symmetric. A 20–30B model is meaningfully behind the
frontier — and costs one to two orders of magnitude less per token, or runs on
hardware you already own. Different ratio, different decision.

I treat capability as a threshold, not a ranking

If a task is below some capability line, it cannot be done at all — use the
strong model. But once a model crosses that line, the things that decide how I
actually work stop being benchmark scores and become cost, speed, whether it can
run for hours, and whether I control the system.

Concretely, here is how it splits for me:

Task	What I reach for
Vague problem, no clear direction yet	Frontier model — a wrong early call invalidates everything downstream
Product architecture, data model, acceptance criteria	Frontier model
Scaffolding a project to a fixed spec, editing files, running tests, fixing failures	Local model
Long-running agents, retries, batch jobs, bulk content	Local model

The important shift is the second block. Once direction is settled and the agent
is executing a spec, a single task may call a model 20–50 times. Do all fifty
calls need the most expensive model on the market? Usually not. The agent is not
deciding what the product should be — it is following specifications that are
already written. If a local model understands those specs and knows how to use
the tools, it has crossed the working threshold.

Agents turn a small price difference into a large one

With chat, you ask once and get one answer. If Claude costs many times more than
a local model, you barely notice.

Agents are not one call. A task can invoke the model twenty times, fifty times;
it fails and reruns. The per-token price gap gets multiplied by every call.

And anyone who has actually built these pipelines knows the expensive part is
not the final run — it is the testing before it. You rewrite the prompt, swap
tooling, re-chunk the data, adjust the workflow, and run the same thing ten or
twenty times. That is normal.

On metered API calls, I feel it: I am burning money right now. On my own card,
there is still electricity and hardware cost, but running an agent five times or
fifty times does not add a line to an invoice. That difference changes my
behavior — it decides whether I dare keep testing. Most stable AI workflows are
not written correctly the first time. They get there after failing enough times
that you find out where they break.

The value of local compute is that you move that trial-and-error into a fixed
cost. That is a different question from "how much does a million tokens cost."

Here is the math on a real engineering workload: a typical agent loop with tool-use iterating through tests and refactoring takes 30–50 turns. Because each turn carries previous tool outputs and conversation history, an average turn easily loads 15k–25k tokens of context — consuming 600k to 1.2M tokens in a single agent run. If you are tuning a new autonomous workflow and test it 20 times a day, you burn 15M to 24M tokens. On frontier APIs like Claude 3.5 Sonnet ($3/$15 per million tokens), that single day of debugging costs $50–$150; on Opus, it crosses several hundred dollars.

On my own self-hosted GPU node (a dedicated 12 GB VRAM card running quantized Qwen 2.5 via LM Studio / llama.cpp), the entire rig pulls under 200W at full load. Running 20 iterations or 100 iterations adds cents in electricity and $0 in marginal API bills. The real math is not just "cost per million tokens" — it is whether you can afford to let an agent fail enough times to become reliable.

Open models crossed the useful line, and the line keeps moving

Three years ago, running a model on a consumer card that could genuinely do work
for you was a research project. Today, 20–30B models — quantized — handle real
work on 24GB and 32GB cards. Go up in VRAM and language, image, and now portions
of video generation become self-hostable too.

The important part is not "open source finally won." It is that work which used
to be closed-model-only keeps falling inside the range that local hardware can
do acceptably. If three years ago two of ten jobs could run locally, and today
four or five can, that ratio alone has changed the value of owning the hardware.

And when I buy a GPU, I am not buying a specific model. Models change. If a
better coding model ships next month, I swap weights. If a better quantization
scheme lands in six months, the same card suddenly fits a model it could not run
before. What I actually bought is compute I control; the models on top of it
are disposable.

That is the structural difference. A closed model is stronger — and also exactly
the version one company currently chooses to offer you, at terms it currently
chooses. Local hardware is weaker — but as long as the card holds up, I decide
which model runs today, which version I switch to, and how long it runs.

Fixed compute vs. renting compute

If AI stays a question-and-answer tool, I do not think most people need a GPU at
all. But that is clearly not the direction. Agents run for long stretches. Image
and video models want bulk generation. Companies hand more fixed processes to
AI. We are not just using models more often — models are becoming a continuously
working compute unit.

At that point, owning part of the capacity and renting every execution become
two different cost structures. Low usage: renting is obviously more
convenient. High, steady usage: owning lets you control and predict cost.

It also buys something no API can sell you — control. When the machine is mine,
I can point it at internal data, connect my own tools, and run agents for hours
without checking whether a vendor's plan, quota or limits changed overnight.
Once AI is inside real company processes, this stops being about saving tokens.
Some data should not leave the environment it lives in. Some agent permissions
should not be handed wholesale to an external service. If the model runs on my
own hardware, data, tools, permissions and execution can be designed together.
That is what I mean by control — not merely having the weights on disk.

What I actually evaluate now

I no longer judge a local model by "it is dumber than Claude, therefore
worthless." Four questions, in order:

Does this model clear my working threshold for this specific job?
How many times will this task execute?
Does it need heavy trial-and-error, or long agent runs?
Does the data or system need to stay in my own environment?

When those stack up, a local model — even one that is not the strongest — can be
the better tool for that slice of work. And as open models improve, that slice
keeps getting bigger.

So what I am bullish on is not one model or one GPU. It is the trend: more AI
work becomes doable on hardware you own. A consumer card that used to run only
language models now runs agents, image generation, video. Better models and
better inference frameworks keep giving the same metal new jobs.

Hardware depreciates, new cards get more expensive — that is real. But the model
is not locked in. As long as the machine has enough compute, I can keep swapping
in newer open models.

The short version

I no longer compare "which model is smarter." I compare how much capability
this job needs, how many times it will run, and how much control I am willing to
leave in someone else's hands.

Where I need frontier reasoning, I pay for the frontier. But when work becomes
high-volume, continuous and predictable — and a local model has crossed the
usable threshold — there is no reason to keep paying frontier prices for every
single step.

In one line:

Use the frontier model to decide what to build. Use local compute to build it
a thousand times.

That is why I bought the GPU.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

The Five-Step AI Roadmap Everyone Shares — and What I Actually Run in Production

Hoelee — Fri, 13 Nov 2026 04:26:19 +0000

Originally published on my blog — read it there with the full context, code and updates.

There is a graphic that makes the rounds every few weeks, and by now it has
reached my feed in translated form, reposted by accounts I don't follow. It
lays out the path to working with large models in five steps: learn the
fundamentals — Transformer, neural networks, attention — and stop as soon as
you can read the diagrams; learn prompt engineering, which takes days; learn
RAG so the model answers from your own data instead of making things up;
learn agents, the biggest opportunity in the field; and then learn
fine-tuning, where the advice is to skip pretraining entirely (that belongs
to the big labs) and get comfortable with multimodal fine-tuning instead.
There is a salary band attached to each step, and the band at the end is the
reason the thing spreads.

I am not going to argue with the list. It is not wrong — it is a syllabus.
And a syllabus is the kind of advice that behaves perfectly in a classroom
and then dies on contact with a system, because it tells you what to learn
and in what order, and never tells you which of those items you will still
be doing years later when somebody is paying you for the result. I have been
running AI workloads of my own for a while now — not research, not model
training, but real services other people depend on. So here is the honest
version of that graphic, checked against what is actually running on my
hardware tonight.

Why this matters

The reason to distrust the roadmap is not that the items are wrong. It is
that a roadmap tells you what to study, while employment asks what you can
operate. Those are different skill sets, and the gap between them is where
most people who "learned AI in five steps" stall: they can describe a RAG
pipeline, but they have never had one break on them at 11pm in a way that
made them understand retrieval, cost, ingestion and evaluation in a single
evening.

The rest of this post maps each step to something I actually run, states
plainly which steps I have never done, and gives you the order I would pick
if I had to start again. Every claim below is measured on a dedicated
workstation with an RTX 3060 (12 GB) and a private NAS node.

Step 1: Fundamentals — enough to be dangerous, then stop

The graphic's framing here is closer to right than most engineers admit:
learn the Transformer until the diagrams make sense, then move on. My
version stopped earlier than that. I can hold a real conversation about
attention, tokens, context windows, quantization schemes and the KV cache,
because those are the levers I actually turn when a model has to fit into 12
GB of VRAM and answer fast enough to be useful in a chat window. I could not
derive backpropagation on a whiteboard, and I would not trust a number I
produced from memory.

What that costs me: nothing measurable so far, in systems that ship. What it
would cost me: the day I train something. If your goal is "own the AI stack
for a small business", step 1 is a reading list, not a degree — and the
honest reason to buy local hardware in the first place is not that a 12 GB
card beats the frontier models, because it does not; it is what running a
model yourself teaches you that an API never will. I wrote about that trade
honestly in
why I still bought a local GPU to run AI.

Step 2: Prompt engineering — the step with the lowest ceiling

"Days to learn" is accurate, and that is the problem with it. Prompting is
the cheapest step to enter and the first one to get automated, so it is
where enormous numbers of people stop and start describing themselves as AI
engineers. The honest skill is not writing a better instruction; it is
deciding when an instruction should not exist at all.

If a task has one correct answer, a deterministic input and a known output
shape, then a script is faster, cheaper and cannot hallucinate. I have
replaced agent prompts with forty lines of Python more than once, and
wrote up the case for that.
What survives on the prompt side is interface design: what the model is
allowed to see, what shape its answer must take, how much of your own
context you are willing to hand it, and — the part the graphic never
mentions — how you would know the output got worse. That last item is
evaluation, and in my experience it is the difference between a demo and a
service.

Step 3: RAG — the hard half is everything except the retrieval

The one-line version everyone quotes is "give the model your data so it
stops lying". I run exactly that: a self-hosted memory layer built from
mem0, LiteLLM and PostgreSQL 17 with pgvector, all on my own NAS, with
memories scoped to a user_id so one customer's history cannot surface in
another's answer —
the whole stack is written up here,
including the container layout and the fact that the API is two small HTTP
calls.

Three things the tutorial version of RAG skips, and each of them cost me
real time:

Ingestion is the boring half that breaks. Retrieval quality gets all the attention; getting the data in reliably is where the work is. The most recent input I pushed through my own speech-to-text stack was a 57-second video sitting on my desktop — it came back as text in about eight seconds on the same 12 GB card that already serves live dictation for my phone and laptop. That speed is what makes the stack usable instead of a demo, and the pipeline is documented end to end here.
Byte-level mistakes look like model mistakes. My first attempt to feed that same service through a workflow tool reported a clean 502 "conversion failed" from the audio layer. The real cause was a multipart upload arriving without a Content-Type header, which made the broker decode raw audio as UTF-8 text and destroy it before any of my code ran. Nothing about that bug is AI. It is exactly the kind of failure the five-step graphic cannot prepare you for.
Retrieval is not a model problem. When an answer is wrong, the useful question is whether the right chunk was ever retrieved, whether it survived chunking, and whether the model was given room to use it. Swapping to a bigger model has never once fixed a retrieval bug for me.

Underneath that entire layer there are two calls, which is worth seeing
once:

POST http://127.0.0.1:20015/memories    # store a fact or a conversation, tagged by user_id
GET  http://127.0.0.1:20015/search      # semantic search, scoped to the same user_id

Nothing exotic — which is the point. A memory layer you can describe in two
lines is a memory layer you can debug, and the whole thing stays local.

Step 4: Agents — the opportunity is real, and the agent is the smallest part

I agree with the graphic's ranking: this is where the work is. I disagree
with what it implies by the word "agent", because an agent is a model plus a
loop plus tools — and the part that decides whether it survives contact with
reality is the checking around it, not the model at the centre. I've written
about this as
loop engineering,
and the finding I keep coming back to is uncomfortable: most of my scheduled
automation has no AI in it at all, and those are precisely the jobs that
have never needed babysitting. The model is the part that adds judgement.
The rest of the system is what makes the judgement safe to trust.

Three things that are actually running here, so the claim is concrete rather
than aspirational:

A Telegram support bot for my own digital-goods shop that keeps long-term memory per customer, so a returning buyer is not a stranger on their second message. The memory layer it writes into is the same self-hosted mem0 stack from step 3.
A scheduled job that manages discount settings in my Shopee seller account, which is the least glamorous useful agent I own: it reads the current state, decides, acts, and reports — and the entire value comes from the fact that it checks the result afterwards instead of assuming its click landed.
A research run that compared twenty VPS providers in parallel by fanning the work out across subagents, which is a pattern worth learning even if you never build a "chatbot": the write-up is here.

Step 5: Fine-tuning — the step I have never done, on purpose

The graphic says skip pretraining, which is correct, and then suggests
learning multimodal fine-tuning. For anyone whose job looks like mine, I'd
push back on the second half.

I have never fine-tuned a model. Not because I couldn't, but because in the
time I have been running these services, none of my failures have been
model-behaviour failures. They were retrieval failures, licensing surprises,
byte-level failures and operational failures — a job that reported success
while the file it claimed to replace was untouched, a mount whose metadata
lied about which file it had swapped, an API whose real model name differed
from the one it advertised. A LoRA does not fix a schema drift.

Where I would fine-tune: one narrow, well-defined task; thousands of
labelled examples rather than dozens; an inference bill or a latency ceiling
that a smaller tuned model would genuinely fix; and the willingness to own
an evaluation set, because a fine-tune you cannot measure is a regression
you cannot see. And I would rent the GPU for it long before I bought a
second card — the training run is a project, not a workstation purchase.

What I'd do differently

If I were starting from zero with that graphic in front of me, I would not
walk it in numerical order. I would walk it in the order my problems
arrived, and accept that the first real lesson will not come from a course:

Treat step 1 as a reading list you finish in public documentation, not a curriculum. Enough theory to know why quantization and context limits hurt; no more until something forces you deeper.
Spend a week on step 2, then immediately automate the parts of it that have one correct answer, and move the rest of your attention to evaluation — which the graphic omits entirely.
Live in steps 3 and 4, because that is where nearly all employed AI work currently sits: wiring models to real data, and making the loop around them trustworthy.
Keep step 5 in your back pocket until a specific problem names it. Fine-tuning is a technique with a precondition, not a rung on a ladder.

And notice the roadmap's real omission, which is not a step but half the
job: identity, storage, deployment, cost, monitoring, and who gets woken up
when it breaks. That is the half that decides whether a team can use you,
and it is the half this blog mostly writes about.

The result

Three of those five steps run in production on one desktop GPU and one NAS,
and the honest scorecard is this:

A 57-second video becomes text in about eight seconds, on the same 12 GB card that serves live dictation — and the speech server drops from 4,906 MiB to 1,321 MiB of VRAM when I toggle it idle, because a GPU that is always resident is a GPU I cannot use for anything else.
The memory layer answers in two small HTTP calls, scoped per user, backed by PostgreSQL with pgvector on the NAS — mine, not a vendor's.
Most of the scheduled automation that keeps this stack alive contains no model at all, and it is the most reliable part of the system.
Step 5 stays empty, and I would rather say that plainly than pad a roadmap. The gap is real, and it is a choice.

If the five-step graphic is the syllabus, this is the transcript. The
syllabus tells you what to learn. The transcript is what a recruiter or a
client can actually check.

Want an AI stack that runs in your business?

I run Hoelee Enterprise, a website design and development business for
Malaysian SMEs — and the AI work above runs on the same infrastructure I
build for clients. If you want a model answering from your data, on your
hardware, without a per-seat subscription attached, I'd be glad to talk:

📱 WhatsApp: +60 12-797 2969
📧 Email: me@hoelee.com
🌐 Website: hoelee.com

And if you are hiring for a platform, DevOps or AI-infrastructure role, the
same channels reach me.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Your Own Speech-to-Text Server: Faster Than Typing, Private by Default

Hoelee — Thu, 12 Nov 2026 04:25:51 +0000

Originally published on my blog — read it there with the full context, code and updates.

Typing is the slowest thing most offices do all day. An average person types
40 words per minute; they speak 130. Every email, quotation, report, support
reply and chat message in your business pays that tax — and the tax is usually
paid by whoever is fastest at the keyboard.

Speech-to-text removes it. But the version most people adopt has two problems:
it costs a monthly subscription per seat, and it ships your voice — your
internal discussions, customer names, pricing, contracts — to someone else's
cloud.

So I built the other version: one speech-to-text API running on a single
desktop PC in my office, with my own GPU doing the work. My development PC,
my iPhone, an iPad and Android phones all dictate through it. It's faster than
typing, it's unlimited, and nothing leaves my network.

This is how it works — including the four traps that cost me most of a day.

Why this matters

Speech-to-text is the highest-leverage office automation that isn't an AI
chatbot. Concretely, what a self-hosted setup buys you:

Over 3x the throughput of typing (65%–70% time reduction). At 130–150 wpm spoken versus ~40 wpm typed, dictating a 500-word email is about 3.5 minutes of talking instead of 12 minutes of typing. Someone who writes 10 emails a day gets over an hour back — every day.
Cost scales with hardware, not headcount. Cloud dictation is priced per user per month, forever. This one runs on hardware you own. Add the tenth employee and the marginal cost is zero.
Unlimited length, no quota anxiety. No minute caps, no "you've reached your monthly limit" at 4pm on a Friday.
Your audio stays yours. Medical notes, legal drafts, HR conversations, customer pricing — voice is sensitive by default. Self-hosted means the transcription never leaves the building.
It works in whatever app already has focus. Not a separate transcribe-then- paste website — a keyboard you use inside Outlook, WhatsApp Web, your CRM, or your own internal tools.

If you run an office where people write all day, this is the same class of win
as moving from dial-up to broadband, and it costs a GPU you may already own.

What you need

Piece	What I used	Notes
GPU machine	Desktop PC with an RTX 3060 (12 GB)	Any NVIDIA card with ≥6 GB VRAM works; it can be a normal work PC
Whisper build	`whisper.cpp` with CUDA	Free, open source
Model	`large-v3` (~2.9 GB)	Best accuracy; smaller models use less VRAM
Delivery layer	n8n + nginx gateway	Adds auth, so the endpoint can be shared safely
Clients	Desktop app, iPhone, iPad, Android	Any client that speaks OpenAI's transcription API

The PC doesn't have to be dedicated. Mine also runs development work,
local LLM inference and image generation — the model loads when a request
arrives and unloads when idle, so it isn't permanently holding VRAM.

The architecture

The important design decision is that the phones never talk to the GPU
directly. There's a gatekeeper in between that handles authentication,
so the GPU itself stays on a private network.

   iPhone / iPad / Android / Work PCs
                 │
                 │  HTTPS + per-device API key
                 ▼
     ┌───────────────────────────┐
     │  https://stt.example.com  │   public HTTPS entry
     │  reverse proxy            │
     └─────────────┬─────────────┘
                   ▼
     ┌───────────────────────────┐
     │  nginx gateway container  │   strips/forwards auth, fixed routes
     └─────────────┬─────────────┘
                   ▼
     ┌───────────────────────────┐
     │  n8n workflow             │   validates the key → 401/403 if wrong
     │  (no audio ever logged)   │
     └─────────────┬─────────────┘
                   ▼
     ┌───────────────────────────┐
     │  whisper.cpp on the PC    │   GPU transcription
     │  :20129  →  {"text": …}   │
     └───────────────────────────┘

Four layers, each doing one job: the proxy terminates TLS, the gateway fixes
routing, n8n authorises, whisper transcribes. The result is
POST /v1/audio/transcriptions — the same shape OpenAI uses, which means any
OpenAI-compatible client works with zero custom code.

The build

Whisper needs CUDA and a CMake toolchain. On Windows that's three installs and
a build:

git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --config Release -j

Then fetch the model and run it:

# large-v3, ~2.9 GB
cd models && sh ./download-ggml-model.sh large-v3 && cd ..
./build/bin/Release/whisper-server.exe \
  -m models/ggml-large-v3.bin \
  --host 0.0.0.0 --port 20129 \
  --convert

--convert matters more than it looks: it lets the server accept MP3 and
other compressed formats by shelling out to ffmpeg, instead of forcing every
client to send raw WAV. Most mobile apps send compressed audio.

The server is now a working transcription API. Everything after this point
is about making it safe to reach from a phone.

Trap 1: CUDA Toolkit installs an incomplete compiler

The CUDA Toolkit installer's default component selection includes nvcc — but
not the pieces nvcc needs to actually compile. My first two CMake runs both
failed on missing headers, and the errors pointed at my build config rather
than at a partial toolchain.

The fix is to add three components explicitly:

crt_13.x      → C runtime headers (the missing <crt/host_config.h>)
nvvm_13.x     → contains cicc, the actual CUDA compiler backend
cublas_13.x   → cuBLAS, required to link at runtime

Symptom to watch for: nvcc --version succeeds, but the build fails immediately
with a missing-header error. A working nvcc is not a working toolkit.

Trap 2: it listens on localhost, so your firewall is innocent

Whisper's default bind is 127.0.0.1 — loopback only. Nothing else on the
network can reach it, no matter what your firewall says.

I lost real time here, because I assumed a firewall problem and tested the
firewall repeatedly (including turning it off entirely) while the actual cause
was the bind address. Loopback-only is defined to refuse every other
interface.

# what it looks like when the port is live but not exposed
netstat -an | grep 20129
#   TCP    127.0.0.1:20129    0.0.0.0:0    LISTENING     ← only you can reach it

Change the bind, and the picture changes:

--host 0.0.0.0

   TCP    0.0.0.0:20129     0.0.0.0:0    LISTENING      ← the network can reach it

Check the bind before you touch the firewall. The two failures present
identically — a connection that times out — and only one of them is a firewall
problem.

One related gotcha: Windows Firewall profiles can re-enable themselves. A rule
scoped to Private,Domain will silently stop working on a network Windows has
since reclassified as Public, and an update that re-enables a profile has the
same effect. Scope the rule to the LAN subnet rather than to a profile name:

New-NetFirewallRule -DisplayName 'Whisper STT' -Direction Inbound -Action Allow `
  -Protocol TCP -LocalPort 20129 -Profile Any -RemoteAddress LocalSubnet

-RemoteAddress LocalSubnet keeps it reachable from your office while staying
unreachable from the internet — which is the correct posture even with auth in
front.

Trap 3: n8n throws away your audio (twice)

Putting n8n in the path is deliberate — it's where the API key check lives — but
it has two behaviours that silently break a proxy.

First: Code nodes discard binary data. My workflow was Webhook → Code
(validate key) → HTTP Request (forward audio). The Code node returned JSON, and
n8n's binary payload — the audio itself — never made it past it. Subsequent
error: "Make sure that the previous node outputs a binary file."

n8n's Webhook node accepts multipart uploads and stores them as binary. To
preserve that through an auth check, the Code node must pass the binary
reference through explicitly, not just return JSON. And the key name is not
what you'd guess:

┌──────────────┬────────────────────────────┐
│ input field  │ binary key in n8n          │
├──────────────┼────────────────────────────┤
│ file         │ data0                      │
│ (second)     │ data1                      │
└──────────────┴────────────────────────────┘

The multipart field is named file, but n8n indexes it as data0. Forwarding
$binary.data fails; $binary.data0 works.

Second: execution logging stores the audio. By default n8n persists
execution data — including binary payloads — which means every dictation is
written to the workflow database. For audio that's unacceptable. Disabling it:

"settings": { "saveDataSuccessExecution": "none" }

⚠ This is the trap with teeth. If you edit a workflow via the n8n API and
omit the settings object, the update silently resets it and you go back to
logging audio. Nothing warns you. Verify after every workflow change:

curl -s -H "X-N8N-API-KEY: $KEY" \
  http://localhost:5678/api/v1/workflows/<id> \
  | jq '.settings'

Trap 4: the stock nginx config hijacks port 80

I run the gateway as an nginx container. It kept returning 404s for paths that
were definitely configured — because the stock default.conf also declares a
server block on port 80, and it wins as the default server. My config was
loaded and correct, and simply never saw the traffic.

# remove the stock config before starting
command: >
  sh -c "rm -f /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"

Two smaller notes from the same build, both worth knowing before you deploy to
a NAS:

You can't bind-mount a config file you haven't created yet — the deploy fails with "Bind mount failed: ... does not exist." On DSM the obvious workaround of writing the file from inside another container doesn't work either, because containers see only their own mounts, never the real host filesystem. I ended up base64-embedding the config in the compose command and writing it at startup — no host filesystem access needed at all.
The /volume1 mounts available to containers are read-only, so the obvious write paths are closed.

The authorisation model, honestly

Two layers, and it's worth being precise about what each one buys you.

Per-device API keys. Each device gets its own key, so revoking a lost phone
doesn't affect anyone else:

{
  "iphone":  "stt_<random>",
  "ipad":    "stt_<random>",
  "laptop":  "stt_<random>"
}

The n8n workflow checks the Authorization header against this list and
returns a real 403 when it doesn't match. One subtlety: an API key in a URL
query string lands in proxy logs and browser history — keep it in the header.

What this protects. It stops unauthorised use of your GPU and keeps the
endpoint from being an open transcription service on the internet. What it does
not do is encrypt the audio — that's TLS at the proxy layer, which is why the
public entry point is HTTPS-only.

This is access control for a private service, not a hardened public API.
There's no rate limiting and no per-key quota yet. For an internal office
deployment across a handful of known devices that's proportionate; if you were
opening it to third parties you'd want limits, logging and rotation on top.

What I'd do differently

Verify the bind address first, not the firewall. Two hours went into testing the wrong layer for a problem that netstat answers in one line. Both failures are "connection timed out", and I should have known that loopback-only refuses everything regardless of firewall state.
Check what the toolkit installer actually installed. I trusted nvcc --version as proof of a working toolchain. A compiler that runs but can't find its own headers is not installed, and the error message will not tell you that.
Set logging policy before wiring the pipeline, not after. I configured saveDataSuccessExecution: none during the build, but the same edit through the API would have silently re-enabled audio retention with no warning. Privacy settings that can be reset by an unrelated update need a verification step.

The result

Four devices — a Windows development PC, an iPhone, an iPad and Android — all
dictating through one GPU on my own network. Typical transcription of a short
sentence is well under a second, and the end-to-end request through all four
layers returns a real transcript in about 2 seconds.

The measured win that matters most is memory. Because the model unloads when
idle, a toggle drops GPU usage from 4,906 MiB to 1,321 MiB — about 3.6 GB
returned — so the machine can go back to development work, local LLM inference
or image generation, then load the model again on the next dictation.

And the honest framing on speed: over 3x the throughput of typing (65%+ time cut) is the
correct figure for prose (130–150 wpm spoken vs ~40 wpm typed). It's less dramatic for
code, where you're thinking more than typing. The gain is largest for exactly
the work offices drown in — email, documentation, notes, contracts, support
replies.

Could this work for your office?

The pieces are unglamorous and the payoff is immediate: one machine with a GPU,
open-source software, and a delivery layer for authentication. No per-seat
subscription, no minute quotas, no audio leaving your premises — and the whole
office dictating instead of typing.

I build exactly this kind of internal infrastructure: self-hosted services with
proper auth, GPU workloads that share hardware sensibly, and the integration
layer that makes them usable from the devices your staff already carry. If you
want speech-to-text (or another internal API) running on your own hardware —
or on a workstation you already own — that's the work I do.

Reach me at me@hoelee.com
or WhatsApp +60 12-797 2969, or see what I do at
hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day

Hoelee — Wed, 11 Nov 2026 04:25:48 +0000

Originally published on my blog — read it there with the full context, code and updates.

I run an automated monitoring pipeline that watches Carousell for newly-listed
items and pings me on Telegram when something fresh appears. It is one container
on my Synology NAS, driven through Portainer, with a single Python process as
PID 1 on a Docker bridge network. One day it stopped pinging. The container was healthy, the scraper was still archiving
listings into NocoDB — but every notification died with the same three
characters in the log:

telegram sendPhoto failed: 400 ... nginx/1.30.1

That string, nginx/1.30.1, became the villain of a day-long debugging
session. This is the story of how a deceptively small bug turned out to be
three separate problems stacked on top of each other — and how I'd find each
one faster next time.

The problem

The bot had two jobs: collect listings, and notify me. Collection
worked. Notification didn't. The log said 400 Bad Request and pointed at a
server identifying itself as nginx/1.30.1.

My first assumption was textbook: something in the network is intercepting
the request. nginx/1.30.1 is not a header I associated with Telegram's
API. A 400 with an HTML body (Telegram returns JSON even on errors) smelled
like a local reverse proxy or a VPN gateway rejecting traffic before it ever
left the machine.

Assumptions like that are cheap. They're also wrong a lot. Let me walk
through what I actually found.

What I tried, and why each thing failed

Round 1: it must be DNS / IPv6

I checked getMe from inside the container. It worked. Then I checked
sendMessage. It failed. Same token, same container, one method up, one down.

The giveaway was in how the two endpoints resolved:

# inside the container
$ getent hosts api.telegram.org
2001:67c:4e8:f004::9   api.telegram.org

The container resolved api.telegram.org to IPv6 only — a single AAAA
record and no A record. The container ran on a Docker bridge network with
no IPv6 connectivity. So every request that needed api.telegram.org was
trying to reach an IPv6 address it had no route to.

The fix looked obvious: pin the IPv4 address with extra_hosts. I did, and
the DNS answer became exactly what I expected:

149.154.167.220   api.telegram.org

And the notifications still failed. Same 400. My proud fix did nothing.

Round 2: I picked the wrong IP

Here's a mistake worth remembering: I hard-coded 149.154.167.220 from
memory. When I finally queried a public resolver, the real A record for
api.telegram.org was different:

149.154.166.110   api.telegram.org   # what DNS actually says
149.154.167.220   api.telegram.org   # what I hard-coded

167.220 is inside Telegram's IP range (149.154.160.0/20), so it's not
"wrong" in the sense of being somebody else's server — but it's not the
active endpoint, and bot-API traffic to it behaved unpredictably. I switched
to 166.110. It still failed. So the IP wasn't the whole story either.

Round 3: the same command, two different outcomes

This was the moment the whole thing got weird. Inside one container, at
almost the same second:

the long-running monitor process (PID 1) → sendPhoto failed, 400
a fresh docker exec process running identical code → sendPhoto succeeded, 200

Same container. Same code. Same payload. Same second.

I ruled out token, chat_id, emoji in the caption, Unicode quotes, image
format, environment variables, and the image URL itself — I download-tested
the photo and it was a perfectly valid JPEG. Nothing in the code differed
between the two paths.

When you hit a contradiction like that — "identical inputs, opposite
outputs" — the answer is usually that the inputs aren't identical. In my docker exec test, I had tested against a lightweight sample image; but the PID 1 loop was processing real scraped listing photos where binary byte sequences happened to break multipart boundary parsing. The binary payloads were not the same. But
finding the difference took one more frame change.

Round 4: `nginx/1.30.1` is Telegram

I re-read the raw 400 response instead of assuming. The response was an HTML
"400 Bad Request" page, and the Server header was nginx/1.30.1.

That nginx is Telegram's own edge. Telegram's API sits behind nginx
(version 1.30.1), and when a request is malformed before it reaches their
application layer, nginx itself answers with a plain 400 page — no JSON, no
friendly error code.

That reframed everything. The request wasn't being intercepted. It was
reaching Telegram, and Telegram's nginx was rejecting it as malformed.

And the malformed thing was the one part I hadn't scrutinized: the
sendPhoto multipart body.

The fix

My original helper assembled the multipart body by treating headers and binary payloads as a list of "lines" and joining them with b"\r\n".join(body_lines) — the kind of "obviously fine" shortcut that ships a framing bug:

# BROKEN: pre-terminated chunks or line-split binary data re-joined with b"\r\n"
body = b"\r\n".join(body_lines)

It fails in two ways, and neither is bytes.join() rewriting the JPEG — b"\r\n".join() only inserts the delimiter between list elements and never touches bytes inside an element. The damage happens upstream. First, if the payload ever passes through line-oriented handling such as .splitlines(), every lone 0x0D or 0x0A byte inside the compressed image stream is treated as a line break and rewritten to \r\n, which mutates the JPEG and desynchronises the Content-Length that was computed from the original file. Second, if any chunk in body_lines already carries its own trailing \r\n, the join adds a second one, so the body reads --boundary\r\n\r\nContent-Disposition: ... — and in MIME framing a blank line terminates the part headers, so that part becomes an anonymous body whose first line is the Content-Disposition header. Telegram's edge nginx/1.30.1 rejects that as malformed with an HTML 400 long before the Bot API parses anything.

That also closes the Round 3 contradiction. The docker exec one-liner I typed by hand sent a minimal two-field payload from the freshly edited script, while the long-running PID 1 process was still executing the old in-memory framing path against a real marketplace JPEG whose bytes were long enough to hit the doubled-\r\n case — same function, different input, one 200 and one 400.

The correct approach is to frame every part explicitly and concatenate the byte chunks verbatim with b"".join(parts):

boundary = "----tg" + token_hex(8)
parts = [
    f"--{boundary}\r\n".encode(),
    b'Content-Disposition: form-data; name="chat_id"\r\n\r\n',
    f"{chat_id}\r\n".encode(),
    f"--{boundary}\r\n".encode(),
    b'Content-Disposition: form-data; name="caption"\r\n\r\n',
    f"{caption}\r\n".encode(),
    f"--{boundary}\r\n".encode(),
    b'Content-Disposition: form-data; name="photo"; '
    b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n',
    image_bytes,                   # ← binary stays untouched
    b"\r\n",
    f"--{boundary}--\r\n".encode(),
]
body = b"".join(parts)             # concatenate raw byte chunks

The difference is b"".join(parts) instead of b"\r\n".join(...): each part
is already a complete chunk with its own framing, and we concatenate them
verbatim rather than inserting a delimiter between every element.

There's a second, higher-level fix worth naming: stop asking Telegram to
download the image for you. My earlier code passed the image as a URL and
let Telegram fetch it:

tg("sendPhoto", {"photo": image_url, "caption": caption})

That means Telegram's servers have to reach out to the source CDN — and if
that CDN is flaky or geo-blocks Telegram's crawlers, you get intermittent
failures that are impossible to reproduce locally. Downloading the image
yourself, then uploading the bytes, removes an entire class of flakiness and
gives you a chance to validate the bytes before you send them.

What I'd do differently

Read the raw response, not the assumption. nginx/1.30.1 sat in logs for hours while I chased phantom proxies. Look up who owns the header before inventing a man-in-the-middle.
Verify a hard-coded IP against a live resolver. Memory is not DNS. dig +short api.telegram.org takes three seconds and would have saved me a whole round.
Treat "identical inputs, opposite outputs" as a lie. The inputs were never identical — the two processes differed somewhere I wasn't looking. The honest move is to diff the bytes, not the code.
Never hand-join binary data with a text delimiter. b"\r\n".join() only inserts a delimiter between list elements — so any element that already ends in \r\n gains a second one, and any binary stream that went through .splitlines() is already rewritten. Frame each part yourself, then use b"".join().

The result

After the multipart fix, the monitor pushed a real notification through on
the next tick — image and all. I also folded in a retry queue so a failed
notification stays marked un-sent and is retried 30 seconds later instead of
silently vanishing. The whole episode cost the better part of a day, but it
turned into exactly the kind of bug I'll never trip over twice.

This monitoring pipeline was originally built for high-frequency internal inspection, but the troubleshooting lessons apply to any production service that relies on outbound notifications. If you've got a monitor, a scraper, or
an alert pipeline that needs to reach people reliably, let's talk.

I build reliable notification and monitoring pipelines — Telegram/WhatsApp
alerts, scrapers, and the infrastructure around them. Tell me what you need
to watch and where you need it delivered:

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed

Hoelee — Tue, 10 Nov 2026 04:25:46 +0000

Originally published on my blog — read it there with the full context, code and updates.

I maintain a numerology report generator: a CodeIgniter 4 application that takes a birth date and a name, runs them through a numerology engine, and renders a 19-page A4 report. It had run on Windows Server with IIS for years. In September 2026 I moved it to OpenLiteSpeed on CyberPanel, on Linux, on a different domain.

The migration itself was unremarkable — copy the files, point the docroot at public/, install dependencies. What made it worth writing about is that the first unauthenticated request to the new host fatally crashed on two routes that had been working fine on IIS for years.

Both bugs were real. Both were present on IIS the whole time. Neither was visible, because IIS had an authentik SSO gate in front of the exact routes that were broken.

Why this matters

If you run a self-hosted app behind an authentication gateway, your auth layer is doing more than protecting data — it is also hiding your bugs. Anything that only breaks for unauthenticated users never runs unauthenticated, so it never fails. The failure surfaces at the worst possible moment: when you migrate, remove the gate, or expose the route to the public internet.

In my case the app had been reporting "healthy" for as long as I'd owned it. The moment I pointed a new domain at it, two routes returned HTTP 500. Nothing had changed in the code. The only thing that changed was that the gate was gone.

The fix for both took about twenty minutes. Finding them took two hours, because the actual error messages were hidden behind CodeIgniter's CLI error renderer.

The setup

The app is a fairly ordinary CI4 project with an unusual architecture detail: the report engine is not a library, it is a set of HTTP endpoints on the same host. The controllers generate a request, curl it back to /api/single on their own domain, and the engine returns JSON with the computed numbers. The controller then renders that into the printable report view.

That self-call design is what made CONST_IIS_INTERNAL_BASE necessary on IIS. Let me come back to it — it turns out to be the interesting part.

	Before	After
OS	Windows Server	Ubuntu (CyberPanel)
Web server	IIS	OpenLiteSpeed 1.9.0
PHP	8.4 (Windows build)	8.4.25 (lsphp84)
Docroot	`C:\inetpub\calc.hoelee.com`	`/home/<domain>/public_html/public`
Auth gate	authentik SSO on `/lifecode`, `/api/*`	none

Step 1: Point the docroot at `public/`, not the project root

CI4 ships with a two-folder layout. app/, vendor/, writable/ and .env live in the project root. Only public/ is meant to be web-accessible.

CyberPanel creates the document root as public_html. My first instinct was to extract the whole project into public_html and leave the docroot alone.

That would have exposed app/, vendor/, the .env file, and writable/ session data over HTTP. The .env in this project contains a webhook credential. Anyone who guessed /../.env — or just fetched /.env, since the file sits directly under a served directory — gets it.

The correct layout is to extract into public_html and then repoint the vhost docroot one level deeper, at public_html/public:

# extract the project so that app/, vendor/ and .env sit UNDER public_html
cd /home/<domain>/public_html
tar -xzf /tmp/deploy.tar.gz

# the docroot must be public_html/public, NOT public_html
sudo sed -i 's#/home/<domain>/public_html$#/home/<domain>/public_html/public#' \
  /usr/local/lsws/conf/vhosts/<domain>/vhost.conf

sudo /usr/local/lsws/bin/lshttpd -t
sudo systemctl restart lsws

This is the single most important step in the migration, and it is the one most guides skip. If you do nothing else, do this.

Step 2: The two fatals IIS was hiding

Fatal 1 — `env()` called too early

The first 500 had no body at all. That is unusual — CI4 normally renders something. An empty 500 usually means PHP died before the framework's error handler was installed.

CI4's bootstrap loads app/Config/Constants.php very early in Boot::bootWeb(), via Boot::loadConstants(). That happens before the Common.php helper file is loaded, which is where env() is defined.

So this line:

// app/Config/Constants.php  — BROKEN
define('CONST_fullBase', env('hoelee.fullBase'));

fails with:

Fatal error: Uncaught Error: Call to undefined function env()
in app/Config/Constants.php:96

The rule is absolute: Constants.php may only contain plain constants. No env(), no getenv(), no config helper. If you need environment-dependent values there, define them further along the bootstrap — or make the constant a fallback and read the real value later.

That is exactly what I did. CONST_fullBase became a plain string, and the helper that consumes it checks the framework's own app.baseURL (which is .env-driven) first:

// app/Helpers/hoelee_helper.php
function getFullBase(bool $selfCall = false): string
{
    // the .env-driven baseURL wins; CONST_fullBase is only a fallback
    if ($selfCall && defined('CONST_IIS_INTERNAL_BASE') && CONST_IIS_INTERNAL_BASE) {
        return rtrim(CONST_IIS_INTERNAL_BASE, '/');
    }
    $appBase = config('App')->baseURL;
    if ($appBase) return rtrim($appBase, '/');
    if (defined('CONST_fullBase') && CONST_fullBase) return rtrim(CONST_fullBase, '/');
    return '';
}

This one was my own doing. I had introduced it during a secrets-cleanup refactor in the same week — I moved a hardcoded URL into .env and called env() in Constants.php to read it. It worked on my machine because the local .env was being read through a different path. It never worked on a clean boot. I caught it within an hour only because I deployed and tested; a code review would plausibly have waved it through.

Fatal 2 — `parent::__construct()` in a CI4 controller

The second failure was on /lifecode, and this one is a genuine pre-existing bug in the application — not something I introduced.

The error, once I got past the CLI renderer, was:

Error: Cannot call constructor

CI4's base CodeIgniter\Controller class has no constructor. It implements initController(), which the framework calls with the request, response, and logger objects. The standard pattern is:

// correct CI4 pattern
public function initController(
    RequestInterface $request,
    ResponseInterface $response,
    LoggerInterface $logger
) {
    parent::initController($request, $response, $logger);
    // your setup here
}

But Lifecode.php and ApiEn.php declared a classic constructor and called parent::__construct():

// BROKEN — CodeIgniter\Controller has no __construct()
public function __construct()
{
    parent::__construct();
    // ...
}

parent::__construct() on a parent class that does not define __construct() is a fatal in PHP 8. Converting both controllers to initController() fixed it — and required adding the three use statements for the interface types in the new signature.

I checked whether the base class really had no constructor rather than trusting the error message:

grep -n 'function __construct\|function initController' \
  vendor/codeigniter4/framework/system/Controller.php

Only initController() came back. Worth doing — "Cannot call constructor" also fires when a parent has a constructor that errors internally, and you want to know which case you are in before you rewrite the signature.

Why neither bug showed up on IIS

This is the part that changed how I think about the deployment.

On the IIS host, /lifecode and /api/* sat behind authentik SSO. An unauthenticated request to either returned HTTP 302 to auth.hoelee.com — it never reached the controller at all. The broken constructor was never executed. The route had presumably been broken since it was written, and it had never once been asked to serve a request.

I confirmed this by comparing the two hosts directly:

curl -sI https://calc.hoelee.com/lifecode | head -1
# HTTP/2 302   <- authentik redirect; controller never runs

curl -sI http://<new-host>/lifecode | head -1
# HTTP/1.1 500  <- no gate; controller runs and fatals

The 302 is why the bug survived. The app looked healthy because the unhealthy parts were unreachable.

The lesson, stated plainly: an auth gate in front of a route means that route has no working test coverage for its own code. If you migrate or remove the gate, budget time to hit every previously-gated route unauthenticated before you call the migration done. On a small app that is a ten-line curl loop. Here it would have found both bugs in seconds instead of two hours:

for p in / /read/single /read/partner /lifecode /api/date /api/single; do
  printf '%-16s %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' https://<host>$p)"
done

Step 3: The self-call that stops being a loopback

Now the interesting architectural consequence.

The report engine is reached by the controller curl-ing its own host. On IIS, CONST_IIS_INTERNAL_BASE pointed that call at http://localhost:7296 — the loopback interface — so the request never left the machine and never touched TLS or DNS.

On LiteSpeed that constant is deliberately left undefined, so getFullBase() falls through to app.baseURL. Which means every report generation now makes a real outbound HTTPS request to the app's own public URL and comes back in through Cloudflare.

Which raises the obvious question: does that work at all?

The answer is yes, but it is worth testing explicitly, because the failure mode is confusing. Here is the exact test — run it on the server, not from your laptop:

curl -s -o /dev/null -w 'HTTPS self-call: %{http_code}\n' \
  -X POST 'https://<domain>/api/single' -d 'nameCn=test&dob=1990-01-01'

Two things to watch for:

Run it from the server. From my Windows machine the same URL returned 404 and 500 — an artefact of DNS resolution and Cloudflare bot protection on my caller, not a server problem. Testing from the wrong host produces confident, wrong conclusions.
Watch the JSON, not just the status code. A 200 with a truncated body is worse than a clean failure. I checked the response was parseable before trusting it.

There is a real trade-off hiding here, and I have not fully resolved it: a public-URL self-call means report generation depends on Cloudflare being up, costs a TLS handshake per report, and occupies two PHP workers for the duration. On LiteSpeed with a small worker pool, a burst of concurrent reports could deadlock — each request waiting on another request that has no free worker. For the current traffic level it is fine. Before this scales, the self-call should move to an internal path that bypasses the CDN.

Step 4: Verify the actual output, not the status code

A 200 on / proves the landing page renders. It does not prove the report engine works, and the engine is the entire product. So the last step was generating a real report end to end:

curl -s -X POST 'https://<domain>/read/single' \
  -d 'nameCn=test&dob=1990-01-01&gender=m' \
  -o report.html -w 'HTTP:%{http_code} bytes:%{size_download}\n'

The result: HTTP 200, 99,189 bytes, 19 A4 pages. I ran it again with Chinese-character input to exercise the UTF-8 path through the engine — HTTP 200, 99,205 bytes, 19 pages, name rendering correctly, zero PHP warnings in the output.

Byte-comparable page counts between the old and new host is the check that matters. It proves the fonts, the DPI, and the pagination logic all survived the move.

A side note on the UTF-8 test: my first attempt at it returned a 500 and I nearly went bug-hunting. The cause was my own shell — the Chinese characters were being mangled by the terminal encoding on the way into curl, so the app received invalid bytes. Running the same request from a script on the server, where the encoding is under my control, returned 200. Before you debug an encoding failure, confirm the bytes actually arriving at the server are the bytes you meant to send.

What I'd do differently

Test unauthenticated routes before migrating, not after. The whole two-hour debugging session was avoidable with a curl loop over the route list. I now treat "list every route, hit it unauthenticated, record the status" as step zero of any migration.

Do not call env() in Constants.php. I have left a comment in the file itself saying so, because the next person — probably me in six months — will be tempted.

Check the framework version's project-space config before the move. The same week, I upgraded CI4 from 4.6.3 to 4.7.4, and it fataled twice on properties the upgrade guide did not mention: Config\App::$permittedURIChars, required by the 4.7 Router, and Config\Format::$jsonEncodeDepth, required by the JSONFormatter — the second of which broke the engine's JSON self-call specifically. Composer updates vendor/ but never merges app/Config/*.php, because those are project-space files. I ended up merging new properties into 14 config files. That is a separate post, but the migration lesson is the same shape: the framework tells you what changed in vendor/; nothing tells you what changed in app/.

Confirm the docroot before anything else. If I had extracted into public_html and stopped, the app would have worked — and quietly served .env over HTTP. A migration that works is not the same as a migration that is safe.

The result

One evening. Six routes verified 200, byte-comparable report output on both Chinese and ASCII input, zero PHP warnings, and two long-standing latent bugs removed from the codebase that IIS had been concealing behind an auth gate.

The app is now running on OpenLiteSpeed with a properly separated docroot, on a host I can automate, with the credential in .env rather than a constant.

Want this done for your own application? I migrate PHP applications between IIS, Apache, nginx and LiteSpeed — including the awkward parts: self-calling architectures, SSO gates that hide bugs, and framework upgrades that touch project-space config.

WhatsApp +60 12-797 2969 · me@hoelee.com · hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Authentik 2025.8 to 2026.8: The Breakable Parts Nobody Warns You About

Hoelee — Mon, 09 Nov 2026 04:25:45 +0000

Originally published on my blog — read it there with the full context, code and updates.

I run authentik as the single sign-on gate in front of my self-hosted stack —
email, dashboard, Synology apps, a remote-access outpost. For a long while it
sat on 2025.8.3, and a year of releases piled up. This is the story of
bringing it all the way to 2026.8.1 in one sitting, and everything that
broke along the way — especially the one mistake that took Single Sign-On
completely offline and made every internal app ask for a password again.

What is authentik, and why run it?

authentik is an open-source identity provider (IdP) — the thing that answers
one question over and over: "who is this person, and are they allowed in?"
Think of it as the front door to a whole building of apps. Instead of every
app running its own username/password screen (and its own list of bugs, its own
"forgot password" flow, its own 2FA), you make all of them ask authentik instead.

It speaks the protocols that matter for real deployments:

OIDC / OAuth2 — the modern single sign-on standard (what "Sign in with Google" uses under the hood).
SAML — the enterprise standard (for tools like Grafana, Jira, or any legacy line-of-business app).
LDAP — so older clients and NAS boxes that only know LDAP can join too.
Proxy / forward-auth — it sits in front of an app via a reverse proxy and gatekeeps access before a request ever reaches the app.

The practical benefits, in order of how much they actually matter:

One login for everything. Log in once, move freely between every app. Users stop juggling a dozen passwords (and the support tickets that go with them).
One place to lock things down. MFA, password policy, session limits, and account recovery live in authentik once — not re-implemented per app.
One place to audit. Every login, on every app, for every user, in one log. That's the difference between "we think nothing happened" and "we can prove it" when a question comes up.
Self-contained and self-hosted. You own the data. No per-user SaaS fees that scale with your headcount, no lock-in — it's AGPL, runs in Docker.

For a small business or a solo operator running twenty-something services
(email, dashboards, NAS apps, internal tools), authentik is the difference
between "every app has its own flimsy password" and "one hardened front door."

Why upgrade at all

Version 2025.8.3 wasn't broken. But it had fallen far enough behind that a
stack of CVEs had landed in the releases after it, and I was getting ready to
do per-application branding. authentik's own policy is that you can't jump
major versions — it enforces a stepwise path. So the plan was:

2025.8.3 → 2025.10 → 2025.12 → 2026.2 → 2026.4 → 2026.6 → 2026.8

Six hops, one at a time, with a migration and a health-check between each.
Before touching anything, the one non-negotiable step: back up the database.
authentik doesn't support downgrades. If a migration half-runs, you're restoring
from dump, not rolling back an image tag.

sudo docker exec authentik-postgres pg_dump -U authentik authentik > authentik-backup.sql

Pitfall 1: Portainer is the source of truth, not the compose file

My first instinct was to edit the docker-compose.yml on disk and up it.
Wrong. The stack is managed by Portainer (stack 143), which keeps the real
compose and the real environment variables in its own store. The .env on disk
was stale — its PG_PASS didn't match what Portainer actually ran.

The correct update path is via the Portainer API, not the filesystem:

Update the image tags in the compose content.
docker pull the new images first (so the API call doesn't time out mid-pull).
docker stop + docker rm the running containers (fixed container_name will otherwise collide on redeploy).
PUT /api/stacks/143?endpointId=2 with the new StackFileContent + Env.

I hit the colliding-container error, the pull-timeout error, and a network-attach
problem where the rebuilt authentik-server only joined one of its two networks
and couldn't resolve postgres-server. Each one is a five-minute fix once you
know what you're looking at, but together they ate the better part of the evening.

Pitfall 2: the storage mount moved (2025.12)

Up to 2025.12, brand assets lived under /media/ and were served at /media/....
After 2025.12, the storage layout changed: files moved to a /data/media
structure served under a new /files/media/public/<name>?token=... URL with a
JWT signature. My containers were still mounting ./media:/media, so every
logo, favicon, and background image 404'd the moment I crossed that version.

The fix is the documented migration:

mkdir -p data && mv media data/media

…and change the mounts to ./data:/data. The new file backend also refuses to
work unless /data is an actual mount point — the earlier version of my fix
used a symlink, which the backend's is_mount() check rejected with
No file management backend configured.

Pitfall 3: RBAC migration leaves a poisoned session table

2025.12 removed the old authentik_core.User_groups model in favor of the RBAC
rework. The migration ran clean, but old sessions still held references to
the deleted model. Result: the login page kept throwing
LookupError: App 'authentik_core' doesn't have a 'User_groups' model.

Not django_session — that was empty. The real culprit was authentik's own
authentik_core_session table. Clearing it (and the other session tables)
forced everyone to log in again and cleared the error:

TRUNCATE authentik_core_session; -- named authentik_core_authenticatedsession in some versions

One side effect worth knowing: this also invalidates every OIDC refresh token
your apps were holding. They'll bounce the user to a fresh login once, then
recover. It's a one-time annoyance, not a bug.

Pitfall 4: trusted proxies are now opt-in (2026.8)

2026.8 tightened the default forwarded-header handling. Previously authentik
trusted all the private ranges; now it only trusts what you list explicitly.
Behind a Synology reverse proxy forwarding to localhost, that means:

environment:
  AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS: 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,::1/128

Skip this and the proxy headers get rejected, which surfaces as auth failures
that look like almost anything except what they actually are.

Pitfall 5: the one that broke SSO — authorization_flow vs authentication_flow

This was the expensive mistake, and it's the kind of thing that's easy to get
wrong if you're doing per-application branding.

A provider in authentik has two flow fields, and they mean different things:

# "Flow used for authentication when the associated application is
#  accessed by an un-authenticated user."      ← the LOGIN page
authentication_flow = models.ForeignKey(...)

# "Flow used when authorizing this provider."  ← the OAuth consent page
authorization_flow = models.ForeignKey(...)

authentication_flow is the login page. This is where you put a per-application flow to customise the title and background.
authorization_flow is the consent/authorize step for a user who is already logged in.

I wanted different background images per app, so I created one flow per
application and pointed the provider's authorization_flow at it. Instant
breakage: a user who was already authenticated still got walked through the
full identification → password → MFA flow every time they opened another app.
Single Sign-On was, effectively, gone.

The fix was a single UPDATE to put the two fields back where they belong:

UPDATE authentik_core_provider
SET authorization_flow_id = '1d85b1b1-...',   -- explicit-consent flow
    authentication_flow_id = 'aa##-per-app-flow'
WHERE ...;

And the branding itself — the per-app background and title — goes on the
flow's own background and title fields (writable since 2026.8), not on
the brand's domain match.

There was a second, related gotcha hiding behind this one. When I created the
18 per-app flows, they all landed with designation=authentication, and my
auth.hoelee.com brand had flow_authentication set to NULL. authentik's
fallback when a brand has no explicit auth flow is to pick the first
authentication flow by slug, alphabetically — which happened to be
auth-agent, not default-authentication-flow. So the root login page started
showing my agent's background image. Setting the brand's flow_authentication
to the real default flow fixed it.

And one more that compounds with the RBAC pitfall: creating a flow sets its
background but not its stages — stages is read-only on the flow object.
An empty flow with no stage bindings is exactly what produced an infinite
redirect loop on the login page earlier in the migration. Stage bindings are
created separately:

POST /api/v3/flows/bindings/   # { target: "<flow pk>", stage: "<stage pk>", order: N }

What I'd do differently

The whole ordeal came down to three preventable patterns:

Never guess at a field's semantics — I treated authorization_flow as "the login flow" when the source-of-truth is the model definition, which spells out the difference in the field's own docstring.
Keep the API surface at arm's length — API tokens kept expiring mid-run with every server restart, so I ended up doing the critical fixes directly against the database with psql. Reliable, but worth scripting before the cluster is on fire, not during.
One breaking change per restart — I tried to reason about storage, RBAC, and proxy changes all in one go. Each would have been trivial if isolated and verified independently.

The result

authentik now runs 2026.8.1 — current, patched, all containers healthy —
with 18 applications each showing their own background and title on the login
page, and SSO working across every subdomain. Seventeen of the applications I
use daily went from "asks me to log in again every time I switch apps" back to
"log in once, move freely."

The lesson worth carrying: a major-version upgrade on auth infrastructure is
roughly 10% "change the image tag" and 90% "the data model, storage layout,
and proxy rules all shifted underneath you." Back up, go one version at a time,
and when something behaves in a way that makes no sense, read the field name
again before you reach for another config.

Want single sign-on for your business?

If you're running several internal apps — a dashboard, an email server, a help
desk, a file server — and your team is still logging into each one separately
(or reusing the same password everywhere), I set up and maintain exactly this
kind of infrastructure. I'll deploy authentik, wire it to your existing apps,
add MFA, and make sure "log in once" actually works — then leave you with a
handover so you're never locked in.

Reach me at me@hoelee.com or WhatsApp
+60 12-797 2969, or see what I do at
hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Upgrading n8n v1 to v2: Seven Deprecations in One Log File

Hoelee — Sun, 08 Nov 2026 04:25:21 +0000

Originally published on my blog — read it there with the full context, code and updates.

I run n8n as the automation backbone for my self-hosted stack — it handles
file-delivery permissions, database backups, and a text-to-speech API that a
reading app depends on. It had been sitting on the 1.123.x line for the
better part of a year, quietly doing its job.

Then I pulled n8nio/n8n:2.40.1 and restarted the container. The upgrade
itself took about ninety seconds. Understanding what it broke took the rest
of the evening — and almost all of it was already written down in a single log
file that n8n prints on boot. I just hadn't read it carefully enough the first
time.

This is that log file, decoded, so you can plan your own v1 → v2 jump instead
of discovering these at 11 PM.

Why it matters

Major-version upgrades of an automation platform are different from upgrading
a leaf service. n8n is the thing that runs everything else: if it comes up
broken, your backups, your access-control syncs, and your internal APIs all
stop with it. Worse, most of what breaks in v2 doesn't throw an error — it
logs a deprecation notice once and then quietly does something different.

The seven items below are the ones that actually applied to a real, messy,
production-shaped install. Three of them changed behavior in my stack. One of
them silently turned a feature off.

Start here: n8n tells you what's wrong at boot

Before touching a single workflow, read the container log from the top. On a
fresh v2 boot, n8n prints an explicit deprecation block:

There are deprecations related to your n8n setup. Please take the recommended
actions to update your configuration:
 - WEBHOOK_URL -> Use N8N_WEBHOOK_URL instead, which sets the base URL for
   both test and production webhooks.
 - N8N_UNVERIFIED_PACKAGES_ENABLED -> The default for this variable will
   change to `false` in a future version.
 - N8N_RUNNERS_MODE -> Internal task runner mode is deprecated and will be
   removed in a future version.
 - N8N_RUNNERS_TASK_TIMEOUT -> The default for this variable will be reduced
   from 300 (5 minutes) to 60 (1 minute) in a future version.
 - N8N_COMPRESSION_NODE_MAX_DECOMPRESSED_SIZE_BYTES -> The default will be
   reduced from 2 GiB to 256 MiB in a future version.
 - N8N_COMPRESSION_NODE_MAX_ZIP_ENTRIES -> The default will be reduced from
   5000 to 1000 in a future version.

That block is the core of your migration checklist. Five of my seven configuration and deprecation gotchas are right inside it, and the other two surfaced a few lines away in the same startup log.

1. Your env values can now fail schema validation

Here is the one that confused me most, because it looked like a nonsense error:

Telemetry event "Instance started" failed schema validation:
execution_variables.executions_data_save_on_error: Invalid option:
expected one of "all"|"none"

I had EXECUTIONS_DATA_SAVE_ON_ERROR=error set — a value that was perfectly
legal in v1, and that I'd chosen deliberately, because saving only failed
executions is the sane default for a busy instance. In v2 that value is no
longer in the allowed set, which is now all or none.

The failure mode is the interesting part. It didn't crash. It didn't even warn
in a way that reads like an error at a glance — it emitted a telemetry schema
validation message, which sounds like an n8n-internal problem, not a
configuration problem of mine. The setting was effectively ignored.

The fix is to move that intent somewhere coherent: pick a legal value and
control volume with pruning instead.

EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX_AGE=336
EXECUTIONS_DATA_PRUNE_MAX_COUNT=10000

Lesson: in v2, treat your environment variables as a typed interface with
a schema. An invalid value may be dropped silently rather than rejected loudly.

2. `WEBHOOK_URL` is deprecated for `N8N_WEBHOOK_URL`

If you publish webhooks behind a reverse proxy — which you almost certainly do,
because that's how they become reachable — the base URL variable is load-bearing.
It's what makes n8n report the public webhook path instead of
http://localhost:5678/....

The old name still works today, so this one won't bite immediately. But note
the wording: the new variable sets the base URL for both test and production
webhooks. In my setup those had drifted apart in behavior, which is exactly
the class of bug this consolidation is meant to eliminate.

# before (still functional, deprecated)
WEBHOOK_URL=https://auto.example.com/

# after
N8N_WEBHOOK_URL=https://auto.example.com/

3. Internal task runner mode is going away — and mine was already broken

This one was sitting in my logs the whole time, several lines above the
deprecation block, and I'd been reading past it for months:

Failed to start Python task runner in internal mode. because Python 3 is
missing from this system. Launching a Python runner in internal mode is
intended only for debugging and is not recommended for production.

If any workflow of yours uses a Python Code node, it has not been running
in internal mode at all — there's no Python in the stock image. JavaScript
Code nodes are fine (a JS runner registers normally), which is why this can go
unnoticed indefinitely: everything looks healthy.

v2 makes the direction of travel explicit: switch to external mode and share
an auth token with a separate launcher process.

N8N_RUNNERS_MODE=external
N8N_RUNNERS_AUTH_TOKEN=<a long random string>

(Note: switching to external mode means pairing the main container with the n8nio/runners sidecar image in your compose stack sharing that same token, which actually ships both the JS and Python runtimes.)

Lesson: "internal mode is deprecated" is the headline, but the real
finding is that a runner type can be silently non-functional for months.
Check docker logs for the runner registration line, not just for up/down.

4. Task timeout drops from 300s to 60s

This is the one I'd flag hardest for anyone with slow workflows:

N8N_RUNNERS_TASK_TIMEOUT -> The default for this variable will be reduced
from 300 (5 minutes) to 60 (1 minute) in a future version.

Be honest about your own workloads. Do you have a Code node that loops over
thousands of records, or an HTTP call to a slow upstream? Mine do — a nightly
reconciliation walks every customer and calls an external API per record. On a
future upgrade, that stops at sixty seconds with no config change on my side.

Set it explicitly now, while you're already in the file:

N8N_RUNNERS_TASK_TIMEOUT=300

The general principle for every deprecation of the form "the default will
change": if you rely on the current default, pin it explicitly. Otherwise
the upgrade is a silent behavior change, and you'll debug it as a bug rather
than recognise it as a stale default.

5 & 6. Two compression-node limits shrink (2 GiB → 256 MiB, 5000 → 1000 entries)

These two travel together and matter only if you use compression/decompression
nodes on large payloads — which is easy to end up doing when you're shuttling
database dumps or archives through a workflow.

N8N_COMPRESSION_NODE_MAX_DECOMPRESSED_SIZE_BYTES -> reduced from 2 GiB to
256 MiB in a future version.
N8N_COMPRESSION_NODE_MAX_ZIP_ENTRIES -> reduced from 5000 to 1000 in a
future version.

An eighth of the memory ceiling and a fifth of the entry limit. Nothing errors;
the node just refuses at a threshold you didn't set. Pin both if you're near
either.

7. The storage path renames in v3 — and you have a volume mounted at the old one

Not a v2 breakage, but v2 warns about it, and it's the one with real data-planning
implications:

Deprecation warning: The storage directory "/home/node/.n8n/binaryData" will
be renamed to "/home/node/.n8n/storage" in n8n v3. To migrate now, set
N8N_MIGRATE_FS_STORAGE_PATH=true. If you have a volume mounted at the old
path, update your mount configuration after migration.

Read that last sentence again: if you have a volume mounted at the old path,
update your mount configuration after migration. If you set the migration flag
and keep your old bind mount, you now have two directories and your binary data
lives in whichever one the container is actually pointed at. Do the rename and
the mount change in the same maintenance window — not one now, one "later".

The one outside the deprecation block: my AI sandbox disabled itself

Here's the finding that had nothing to do with a deprecation notice, and that
I'd never have caught without reading the full boot sequence:

Sandbox: enabled=false provider=n8n-sandbox (DB override; env was enabled=true
provider=n8n-sandbox)

My environment said enabled. The database said otherwise. The database won.

The env var was N8N_INSTANCE_AI_SANDBOX_ENABLED=true, and it was still set
correctly on the container. But a value persisted in n8n's own configuration
store overrode it at startup — and the only place that conflict is reported is
inside a parenthetical in a log line.

This is a genuinely valuable debugging lesson beyond n8n: when a feature is off
despite the env var being obviously right, suspect a persisted settings layer
that outranks your environment. The container config is not always the last
word. Grep for the feature name in the logs, and don't stop at the env value.

What I'd do differently

Read the boot log before declaring the upgrade done. Every deprecation that mattered to me was printed on startup, in one block, on the first run. My v1 habit was to check "is it up, do the workflows run" — which is exactly the check that misses all seven of these.
Treat "the default will change" as a to-do, not a warning. Five of the seven items are future-default changes. Pinning them now costs one edit and converts a mystery outage later into a config diff.
Diff the container config against n8n's own stored config. The sandbox override taught me that env is one of two inputs, not the source of truth. When behavior and configuration disagree, believe the behavior and go find the higher-priority layer.
Pin the image tag and keep the previous one. I jumped v1 → 2.40.1 directly. Having the old image on disk is what makes a rollback a docker run instead of a rebuild.

The result

n8n is on 2.40.1 with the full deprecation block resolved: the schema-invalid
value corrected, N8N_WEBHOOK_URL in place, the JS runner registering cleanly,
and the task timeout and compression limits pinned so the next upgrade is a
no-op rather than a surprise.

Everything downstream kept working — the file-permission syncs, the nightly
backups, the text-to-speech endpoint. That's the outcome worth aiming for with
an upgrade like this: not "it came back up", but "it came back up and the
next three upgrades are already paid for".

The uncomfortable part is how much of it I could have known in advance. n8n
handed me the entire list, unprompted, at startup. The upgrade was never the
hard part — reading the output was.

Running automation you'd rather not babysit?

I build and maintain self-hosted automation — n8n workflows, Docker stacks,
and the glue between apps that were never designed to talk to each other. If
you're facing a major version upgrade, or you have automation that works until
it doesn't, I plan the migration, do it in a maintenance window, and document
every config decision so the next upgrade is boring.

Reach me at me@hoelee.com or
WhatsApp +60 12-797 2969, or see what I do at
hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Upgrading CodeIgniter 4.6 to 4.7: The Breaking Changes the Guide Misses

Hoelee — Sat, 07 Nov 2026 04:25:54 +0000

Originally published on my blog — read it there with the full context, code and updates.

I upgraded a CodeIgniter 4 app from 4.6.3 to 4.7.4. I read the upgrade guide, checked every documented breaking change against the codebase, and confirmed none of them applied. Then I ran the app and it fataled twice.

Both fatal errors were the same kind of problem, and that kind of problem is not mentioned anywhere in the upgrade guide. This is a short post about what actually breaks and why the guide can't warn you.

Why this matters

Composer has a scope boundary that is easy to forget. composer update writes to vendor/ — the system scope, the framework's own files, which it owns and can replace freely.

Your app/Config/*.php files are project scope. They are yours. Composer will never touch them, and the upgrade guide describes them as "you may want to merge these changes" rather than "this will break".

But when 4.7's framework code reads a property that your 4.6-era config class does not define, you get a fatal — not a deprecation, not a warning. The guide cannot enumerate this, because the set of properties is the intersection of what the new framework reads and what your config file happens to contain, and it has no visibility into the second half.

If you are upgrading a CI4 app more than a minor version behind, budget for this. It took me about an hour and 14 config files.

What I checked first

The documented breaking changes in 4.7.0, and whether each applied:

Documented change	Affects this app?
`Model::insertBatch()` / `updateBatch()` return values	No — no Models used
Entity casting changes	No — no Entities
Validation rule changes (`regex_match`, `differs`)	No
Encryption handler defaults	No
Uploaded-file / image validation behaviour	No — no uploads
`IncomingRequest` internal changes	No
Removed `Session` class properties	No — see below
`PageCache` constructor signature	No — see below

Two of those needed a closer look, and both turned out to be false alarms worth describing, because they look identical to a real hit if you only grep:

"Removed Session class properties" — grepping for session in my configs lit up immediately. But the properties in question (sessionDriver, sessionCookieName, and friends) on my Config\App are Config\App's own properties, which were not removed. The guide refers to properties on the Session class. Same word, different class.

"PageCache constructor signature changed" — I reference pagecache — but only as a filter alias in Config\Filters. That does not extend or instantiate the class, which is the only thing that breaks. A reference is not a subclass.

Both checks taught me the same thing: grep finds mentions, not usages. Verify the class in the match is the class the guide means, and that you actually extend or call the thing that changed.

The two fatals that were not in the guide

Both surfaced on a clean boot. Both are properties that 4.7 framework code reads and 4.6-era config files do not define.

1. `Config\App::$permittedURIChars`

Undefined property: Config\App::$permittedURIChars

Required by 4.7's Router, which validates incoming URI characters against this property. Absent from a 4.6-era App.php, the Router throws before it routes anything.

// app/Config/App.php — append to the class
/**
 * CI4 4.7 compatibility: allowed characters in a URI.
 * Required by the 4.7 Router.
 */
public string $permittedURIChars = 'a-z 0-9~%.:_\-';

2. `Config\Format::$jsonEncodeDepth`

Undefined property: Config\Format::$jsonEncodeDepth

Required by 4.7's JSONFormatter. This one broke the app's report engine, because the engine's JSON response is decoded with json_decode() and the failure surfaced much further downstream as a confusing parse error rather than the real "undefined property" message.

// app/Config/Format.php — append to the class
/**
 * CI4 4.7 compatibility: json_encode() depth limit.
 */
public int $jsonEncodeDepth = 512;

That second one is worth dwelling on. The visible error was:

Failed to parse JSON string. Malformed UTF-8 characters

Which points at encoding. The actual cause was a missing config property. If a framework error names a symptom rather than a cause, go looking for the real exception in the log before acting on the message you were given — I lost time chasing UTF-8 that was never broken.

How to find the rest before they bite you

Both fatal errors were found by running the app. Running is the reliable method, but you can get ahead of it by diffing property coverage.

The approach: parse the framework's shipped config defaults from vendor/, parse your project's config classes, and list properties defined in the former but missing from the latter. That gives you the whole class of problem at once instead of one fatal per restart.

import re, pathlib

sys_ = pathlib.Path('vendor/codeigniter4/framework/system/Config')
app_ = pathlib.Path('app/Config')

def props(path):
    try:
        src = path.read_text(encoding='utf-8', errors='replace')
    except FileNotFoundError:
        return set()
    # public/protected/private $name = ...
    return set(re.findall(r'(?:public|protected|private)\s+(?:[\w\\\[\]|?]+\s+)?\$(\w+)', src))

# names whose properties 4.7 requires but a stale config may omit
for name in ['App', 'Cache', 'ContentSecurityPolicy', 'CURLRequest', 'DocTypes',
             'Email', 'Encryption', 'Exceptions', 'Format', 'Honeypot',
             'Migrations', 'Paths', 'Routing', 'Toolbar', 'View']:
    missing = props(sys_ / f'{name}.php') - props(app_ / f'{name}.php')
    if missing:
        print(name, '->', sorted(missing))

Run it against the new vendor/ after composer update and before you boot the app. Anything it prints is a candidate fatal.

In this project it flagged 14 config files. I merged the missing properties into each one, appended in a marked block at the end of the class so existing settings and behaviour were untouched:

    // ----------------------------------------------------------------
    // CI4 4.7 compatibility — properties the 4.7 framework reads.
    // Appended as a block so existing settings above are unchanged.
    // ----------------------------------------------------------------

Two config files the framework now ships had no counterpart at all and needed creating rather than merging: Hostnames.php and WorkerMode.php. Copy those straight from vendor/codeigniter4/framework/app/Config/.

Also worth knowing before you upgrade

Validate app.baseURL properly. 4.7 throws a ConfigException on values 4.6 accepted. 'http://localhost/' — which had been sitting in my local .env for months, working fine — now throws. Production was unaffected because it used a real URL, but any dev environment with a shorthand baseURL will refuse to boot.

The security case is real. 4.7.4 alone patched an uploaded-file extension bypass (is_image / mime_in), a SQL injection in deleteBatch(), and a path traversal in UploadedFile::move(). None were exploitable in this app — it does no uploads and uses no database — but all three become live the moment those features are added. Upgrading before you need those features is the cheaper order.

PHP floor is now 8.2. Check your host before starting, not after.

What I'd do differently

Run the app against the new vendor/ before doing anything else. composer update, then boot it, then fix what breaks. Reading the guide tells you what might change; only running tells you what did.

Do the property diff as a matter of course. It converts a serial debugging session into one report.

Do the upgrade as its own commit, on its own. I did, and it made the two fatals trivially bisectable — one git show told me exactly which change introduced each. Mixing a framework bump into a feature branch turns a five-minute diagnosis into an archaeology exercise.

Read the changelog, not just the upgrade guide. The guide lists what the maintainers judged likely to break you. The changelog lists what changed. They are not the same set, and the gap is exactly where these two fatals lived.

The result

4.6.3 → 4.7.4 on a 17,000-line CodeIgniter app: 14 config files merged, two new config files added, two fatals found and fixed, and every route verified returning 200 — including a real report generation producing 19 A4 pages with byte-comparable output to the pre-upgrade baseline. Zero errors in the application log.

Running an older CodeIgniter app? I do framework upgrades like this one — CI4 minor bumps, PHP version migrations, and the config-merging work that the upgrade guide assumes you will figure out yourself.

WhatsApp +60 12-797 2969 · me@hoelee.com · hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

The JPA Field That Turns a Silent Lost Update Into a 409

Hoelee — Fri, 06 Nov 2026 04:25:50 +0000

Originally published on my blog — read it there with the full context, code and updates.

Two people open the same post. Both edit it. Both save. One of the saves
disappears without a trace — no error, no warning, and the API answered
HTTP 200 for both of them.

That is the lost update problem, and it is the quietest data-loss bug in
read-modify-write APIs. I hit it while building my Spring Boot reference
service — a Spring Boot 3.5.16 (Java 21) REST API where editors create and
update posts from a browser page. The fix turned out to be one @Version
field on the JPA entity, one field on the request contract, and one
exception that the API layer maps to HTTP 409 Conflict. This post traces
that path through the actual code.

The problem: an edit that vanishes with no one to blame

Why does one editor's change disappear when two people save the same
record?

Follow the timeline. Editor A and editor B both fetch the same post.
A saves first: the row is updated and the API answers 200. B saves a
moment later: the row is updated again, this time over A's text, and
the API answers 200 again. Every request succeeded; every response told
its caller that their save is the state of the record. A's edit is
simply gone, and neither client has any way to know.

The failure is silent because it is normal read-modify-write behaviour,
not an anomaly. That is exactly why the bug survives in real systems, and
why it is worth fixing properly in something meant to demonstrate
engineering judgement.

What I tried first, and why it failed

My first sketch of the update flow was the obvious one: GET the record,
edit it, PUT it back with the new title and body and nothing else.

GET  /api/posts/{id}    # read the record
PUT  /api/posts/{id}    # write it back with a new title and body

The server loads the row, applies the update, and returns 200. It has no
way to notice that the row moved underneath the client between the GET
and the PUT — the read and the write are two unrelated requests, and
nothing in the contract connects them. The last write wins, and both
writers are told they succeeded.

A 200 is worse than an error here. An error at least tells the losing
client that something happened and gives them a reason to look. A 200
tells both clients that their save is the current state of the record,
which is a lie for one of them. The losing edit now exists nowhere — not
on screen, not in the database — and because the client believes it
saved, nobody goes looking for the data.

The fix: a @version field, a contract, and a conflict

The fix is five small parts, all of them in the reference service's source.

1. The entity carries a version column

@Entity
@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title"))
@EntityListeners(AuditingEntityListener.class)
public class Post {

    @Id
    @GeneratedValue
    @UuidGenerator
    private UUID id;

    @Column(nullable = false, length = 160)
    private String title;

    @Column(nullable = false, length = 10_000)
    private String body;

    @Version
    private long version;
    // ...
}

@Version tells Hibernate to treat version as an optimistic lock. Every
UPDATE it generates becomes conditional:

UPDATE posts
SET title = ?, body = ?, version = version + 1, ...
WHERE id = ? AND version = ?

If the row's version no longer matches the value the statement was built
with, zero rows change and Hibernate rejects the flush instead of quietly
overwriting. The entity never bumps the counter by hand — the update
method only touches title and body; the version moves with the write
itself.

2. The version is part of the API contract

A lock is useless if the client cannot take part in it, so the version
travels through the API in both directions. The response record exposes
it:

public record PostResponse(UUID id, long authorId, String title, String body,
                           long version, Instant createdAt, Instant updatedAt) {
}

and the update request requires it back:

public record UpdatePostRequest(
        @NotBlank @Size(max = 160) String title,
        @NotBlank @Size(max = 10_000) String body,
        @NotNull @Min(0) Long version) {
}

The learning note next to UpdatePostRequest says it plainly: "the
expected version is part of the update contract, making optimistic
locking visible to clients."

3. The service checks it inside the transaction that owns the write

@CachePut(cacheNames = "posts", key = "#postId")
@Transactional
public PostResponse updatePost(UUID postId, UpdatePostRequest request) {
    Post post = postRepository.findById(postId)
            .orElseThrow(() -> new PostNotFoundException(postId));
    if (post.getVersion() != request.version()) {
        throw new PostVersionConflictException(postId);
    }
    post.update(request.title().trim(), request.body().trim());
    return PostResponse.from(postRepository.saveAndFlush(post));
}

The load, the comparison and the save all happen inside one
@Transactional method, and that is the part that makes the check
honest: the version is compared against the row as it stands at write
time, not against a snapshot from an earlier request. If the read and
the write lived in different transactions, the check would compare
against a version the row has already left behind, and the whole scheme
would be theatre. The learning note on PostService names the rule:
"service methods centralize transaction boundaries, cache coherence, and
optimistic-locking rules." The service also runs with
spring.jpa.open-in-view: false, so no lingering session serves a stale
entity across requests — the row is re-read inside the writing
transaction.

The explicit check catches stale requests deterministically. The
@Version column still matters underneath: if another commit slips in
between the check and the flush, the conditional UPDATE affects zero
rows and Hibernate refuses the write anyway.

4. The handler maps the exception to a 409

The exception itself carries the message a client can act on:

public class PostVersionConflictException extends RuntimeException {

    public PostVersionConflictException(UUID postId) {
        super("Post %s has changed; fetch it again before retrying".formatted(postId));
    }
}

and the error boundary turns it into the right HTTP answer:

@ExceptionHandler({PostVersionConflictException.class, ObjectOptimisticLockingFailureException.class})
ProblemDetail handleConflict(RuntimeException exception) {
    return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage());
}

HttpStatus.CONFLICT is HTTP 409, and the response is a problem document
— a type URI, a stable machine code (POST_VERSION_CONFLICT) and a
human-readable detail. Mapping both our explicit exception and Spring Data JPA's ObjectOptimisticLockingFailureException (thrown when Hibernate detects a race condition) guarantees that concurrent write collisions always return 409 rather than an unhandled 500. The service enables the framework's problem-details
support in its configuration (spring.mvc.problemdetails.enabled: true),
so validation and every other error ride the same shape.

5. The whole path, end to end

stale PUT /api/posts/{id}
  -> PostService.updatePost re-reads the row inside its transaction
  -> post.getVersion() != request.version()
  -> PostVersionConflictException: "fetch it again before retrying"
  -> ApiExceptionHandler.handleConflict
  -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT
  -> the client knows its premise was stale

One @Version field, one version in the request contract, one
exception, one handler method. Nothing on the controller changed —
updatePost still looks like an ordinary PUT.

Why a conflict is the correct answer, not a retry

When two editors produce two different versions of the same record, the
server cannot know which intent is the one meant. Both saves are
legitimate from their authors' point of view, and replaying the stale
write just replays the same data loss.

The honest verdict is HTTP's own definition of 409: the request conflicts
with the current state of the resource, and the client is told exactly
how to resolve it — "fetch it again before retrying". The losing client
re-reads the record, sees the other writer's change, and a human decides
what to keep. The silent overwrite becomes a visible decision point,
which is the entire purpose of the exercise.

What the rest of the reference service proves

The version field is one layer of a small system, and the rest of it is
verifiable the same way — every row below is a class or a config file in
the repository, not a claim:

Layer	What it does	Where it lives
Caching	Bounded Caffeine cache on single-post reads (max 500, TTL 10 min, both from config); `@Cacheable` on read, `@CachePut` on update, `@CacheEvict` on delete; `GET /api/showcase/cache` exposes requests, hits, misses and hit rate	`CacheConfig`, `CacheProperties`, `ShowcaseMetricsController`
Security	Stateless HTTP Basic with BCrypt; pages and read APIs are public, every mutation needs the `EDITOR` role; local defaults are a throwaway `demo-editor`/`changeit` account overridable via env vars	`SecurityConfig`, `application.yml`
Errors	Request records validate at the boundary; one `@RestControllerAdvice` returns consistent problem documents with machine codes — `POST_NOT_FOUND` (404), `POST_VERSION_CONFLICT` (409), `VALIDATION_FAILED` (400) with a field-error map	`CreatePostRequest`, `UpdatePostRequest`, `ApiExceptionHandler`
Configuration	Local profile runs an in-memory H2 database in PostgreSQL mode with `ddl-auto: update` and seeded sample posts; `prod` selects PostgreSQL, takes every credential from `APP_*` env vars and sets `ddl-auto: validate`	`application.yml`, `application-prod.yml`
Tests	One `@SpringBootTest` suite asserts the rendered Thymeleaf pages, unauthenticated 401s versus editor 201s, the `VALIDATION_FAILED` problem response, public search, and cache hits after repeated reads	`DemoApplicationIntegrationTest`

Hardening This Pattern for Production Rollout

Two things change the moment this pattern moves to production. The project
README says both, and they are worth repeating as my own judgement.

First, schema management. I would add Flyway or Liquibase migrations
before ever setting ddl-auto to validate. The local development profile
uses update, which is convenient on a scratch database and dangerous as
a habit — production schema should be versioned code, not a side effect
of startup.

Second, secrets. The local editor account is intentionally a non-secret
demo account with default credentials. For anything real, I would keep
credentials in a managed secret store and refuse to boot without them,
instead of falling back to a default. The prod profile already takes
every credential from environment variables with no committed secrets —
that is the shape I would keep, minus the fallbacks.

The result

A stale update now returns HTTP 409 with code POST_VERSION_CONFLICT
instead of a 200 that silently discards the other writer's change. The
loss can no longer happen quietly: the failure is loud, actionable —
"fetch it again before retrying" — and the integration test suite asserts
the access and validation behaviour end to end.

The whole fix is one annotation on one field, plus plumbing that makes
the version part of the contract. If a record can be read by two people
at the same time, that field is the difference between a lost edit and a
conflict both clients can see.

I build Java/Spring back ends and full-stack web applications — REST APIs
with JPA, optimistic locking, caching, validation and integration tests,
from the browser page down to the database. If you have a form or a field
where edits keep quietly disappearing, tell me about it:
WhatsApp ·
me@hoelee.com ·
hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

When a Self-Hosted Model Gateway Tripped Trust & Safety

Hoelee — Fri, 06 Nov 2026 00:17:31 +0000

Originally published on my blog — read it there with the full context, code and updates.

A single self-hosted OpenAI-compatible gateway served every model tier: long-context reading, refactors, cheap background jobs. One key, one model picker, one dashboard where spend was visible in one place. Six independent workflow seats — repo indexing, CI triage, interactive refactors — sat behind it, each authenticating with its own subscription OAuth credential.

Then all six came back disabled inside one short detection window. None of them had been doing anything unusual. What tripped was a scheduled credential check, and what it found was a block already applied upstream.

The interesting part was not the block. It was the ten minutes I spent rotating credentials that were never wrong.

The error that sent me the wrong way

For the first ten minutes, every consumer of the gateway's error reported a credential failure — orchestrator, shell probe, CLI — because that is what the body said: 401 invalid_api_key. The upstream had actually returned a policy decision.

Three probes separated the layers, and they are worth keeping for any "the model stopped answering" ticket: confirm the gateway answers its own health endpoint; confirm a model from an unrelated provider succeeds on the same key; then loop the subscription models and compare the error codes. One model failing means a broken model. Every model on one provider failing identically, while other providers succeed on the same key, means the provider relationship is closed — and no key rotation moves that.

The real defect is the error contract, not the outage:

# Before: policy decisions disguised as auth failures
if upstream.status == 403:
    return jsonify(error="invalid_api_key"), 401

# After: distinct classes, distinct client behaviour
if "Terms of Service" in upstream.text or "disabled in this account" in upstream.text:
    return jsonify(error="upstream_policy_block",
                   upstream_status=403,
                   alert="credential-policy"), 502
return jsonify(error="invalid_api_key"), 401

A 401 that means "your key is wrong" and a 403 that means "the vendor closed this seat" must never collapse into the same response. Flattening them costs the responder the first ten minutes of every incident.

Why a shared token is not a shared key

An API key is a bearer credential: whoever holds it can use it. An OAuth token is bound to a client identity — client id, consent record, scopes. When a different process presents that token, the request keeps the claims and loses the behaviour: headers, TLS fingerprint, request cadence.

Two properties of the initial gateway topology made the pattern obvious to any detection system:

One egress. Every seat left through one container on a single public address.
One fingerprint. The gateway normalised headers and stripped client telemetry, so six workflow seats looked like one client at six times the volume.

Simultaneous failures across independent workflows are not six human reviews. They are one deterministic cluster rule firing. The vendor's terms had said it in advance: scripting the official CLI is acceptable, pooling seats or automating rotation across them is not.

Task dispatch instead of routing

The lesson is not "use fewer seats". It is that a subscription is a seat, not an API. The moment its token leaves the vendor's client, it becomes a shared credential.

So the gateway keeps serving models paid for by API key, and subscription work moved to task dispatch: the orchestrator runs isolated, single-task invocations of the official CLI bound to a dedicated workflow seat. No pooled credentials, no hijacked persistent session, no rotation. The token never leaves the vendor's binary.

orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
                       |
                       +- cwd, permission scope, timeout, tree-kill, no retries

Four dispatcher properties mattered more than any prompt:

Explicit working directory — each task starts inside one repository, never a user home directory or a filesystem root, because an agent with edit rights inherits the whole shell.
Closed stdin — interactive prompts in a headless run become errors instead of silent hangs. One forgotten -y otherwise blocks the parent forever.
Process-group kill on timeout — killing the parent leaves grandchildren running; kill the group and return a timeout code, not a half-finished diff.
No automatic retries — an inner agent loop times an outer retry policy is hundreds of upstream calls in minutes, the exact cadence that got the previous design flagged.

proc = subprocess.Popen(
    argv, cwd=repo, stdin=subprocess.DEVNULL,      # no interactive hangs
    stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    start_new_session=True,                         # own process group
)
try:
    out, err = proc.communicate(timeout=task_timeout)
except subprocess.TimeoutExpired:
    os.killpg(proc.pid, signal.SIGKILL)             # kill the whole tree
    raise

Context stopped travelling in prompts as well: the delegate reads repository instruction files and queries a self-hosted memory service over MCP, so a brief stays six lines instead of a transcript dump. Delegate writes are tagged status: draft and final archiving stays with the orchestrator.

The economics survived, because the workload splits

Nine turns in ten are routine orchestration — tool loops, retrieval, cron, log triage, short edits. Those run on a curated open-weight plan: $10/month carrying a $100 monthly allowance on the fast model, with per-model pools beside it. At $0.15 per million input tokens and $0.60 per million output, a 4k-in/500-out turn costs about $0.0009 — under a tenth of a cent.

The remaining tenth is heavy engineering: multi-file refactors, legacy archaeology, anything with a real build-and-test contract. That dispatches to an authorised CLI seat, where the vendor's client owns the credential and the allowance belongs to the subscription rather than to a pooled bucket.

Pay for frequency with API-key models, pay for depth with authorised seats, and never let the two share a token.

💬 Let's Compare Notes

If a credential is bound to a client, where should a platform team draw the execution boundary when metered APIs and client-bound CLI tools have to coexist: standardise on pure APIs and pay the premium, or run a dispatcher that governs official binaries? I have moved to the second, and I would like to hear from anyone who has run the first at team scale and made it survive an audit.

Two related posts from the same infrastructure work:

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Embedding Fonts So a PDF Looks the Same on Every Machine

Hoelee — Thu, 05 Nov 2026 04:25:03 +0000

Originally published on my blog — read it there with the full context, code and updates.

The requirement was one sentence

A report generator has to produce a document that looks the same everywhere:
Chinese headings in a serif, Chinese body text in a sans, Latin headings and
cover numerals in a display serif, Latin body text and tables in Inter.
Weights matter — medium for emphasis, semibold for headings — across three
languages and two PDF renderers.

And the fonts must travel with the document. Not "the server has them
installed", not "the reader probably has them" — embedded, so a print
shop in another country sees the same pages.

The old pipeline failed on the last clause in a way that was invisible: it
depended on fonts installed on the server by a package manager, and it only
ever embedded a single CJK face. The second weight did not exist, so "bold"
was the renderer's synthetic approximation, and the only way to notice was
to open the PDF and look at what was actually inside it.

Rule one: read the fonts back out of the finished PDF

"Embedded" is a claim about the output, so I measured the output. The same
document, rendered before and after:

	before	after
zh page count	23	23
en page count	28	29
bm page count	30	30
fonts actually embedded	one CJK face + Roboto	Source Han Serif Medium + SemiBold · Source Han Sans Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · Source Han Sans for the CJK runs inside Latin reports
render time / peak memory	baseline	same order (±10%, noise)
file size	baseline	roughly +3%

The glyph counts per face are the receipt: in the Chinese report, 15,988
glyphs land in the body sans, 2,247 in its medium weight, 634 in the heading
serif medium, 222 in its semibold. That is the difference between "the font
is on the server" and "the font is in the file" — a reader in another
country sees the same four faces, and a page count that moved by exactly one
page in one language (the Latin metrics are wider than what they replaced)
is the honest cost.

Rule two: static instances only, and the reason is in the parser

The obvious modern choice — ship one variable font per family — is wrong
here, for two measured reasons:

The PDF engine's font parser reads glyf/loca only. Feed it an OpenType file with PostScript/CFF outlines and it refuses: Fonts with postscript outlines are not supported. A variable font is not the problem; CFF outlines are.
It does not read the fvar table at all, so a variable font silently resolves to its default instance — and the default instances of the CJK families are the thinnest ones (Thin 100 and ExtraLight 200, visible in the name table). Every page would render in a hairline weight, with no error anywhere in the pipeline.

So the build step takes the upstream variable fonts and instantiates
static cuts: it runs the instancer, then rewrites name,
OS/2.usWeightClass and head.macStyle so the file identifies itself as
the weight it now is (the source's own name still says "Thin"). A manifest
records family, weight class, sha256, glyph count and code-point count for
every file, and the licence files ship next to them.

Rule three: the engine speaks two weights, so map onto two slots

The renderer's CSS handling honours exactly normal and bold — its source
says so in as many words ("Does not support: bolder, lighter, 100..900").
Numeric weights are therefore meaningless to it, while the headless-Chromium
renderer in the same pipeline honours them properly.

The mapping that works: each family declares an R slot and a B slot
(regular-ish and heavier), font-weight: 500 lands on the R slot of the
intended family, and 600 is rewritten to bold on the PDF side only,
which lands on the B slot. Chromium keeps 600 as written. Two tests pin
it: the B slot must be heavier than the R slot (read from usWeightClass,
not assumed), and font-weight: 600 must never appear in the stylesheet
handed to the PDF engine.

One subtlety worth the paragraph: a "medium" weight is not an emphasis
mechanism in this pipeline, it is a family member. Chinese headings use
the serif's medium as their regular and its semibold for emphasis; Latin
headings do the same with the display serif. Getting that wrong is invisible
on a screen and obvious in print.

Rule four: do not subset against a character set you do not control

The first attempt subset the CJK fonts to the union of two national
character sets and produced a ~29 MB bundle instead of ~50 MB. Then I ran
the name probe and found these were cut:

犇  骉  頔  祎  廸  玧  祾  赟  堃

Every one of them is a real given name character. The report prints a
customer's name on the cover, and the name is free input — so a subset-based
build will eventually put a tofu box on the most prominent page of the
product. Full coverage means keeping ~31k code points — roughly 50 MB of static font assets in the repository (adding only ~3% to each finished PDF after engine subsetting); that is a fine price for never showing a customer an empty rectangle.

Rule five: make missing glyphs a build failure, not a surprise

The gate runs three checks and exits non-zero on any of them:

every font file declared in the manifest is present, and its sha256 matches the file on disk;
every character in the content library (all three languages) plus the brand strings, plus probe classes — simplified, traditional, CJK punctuation, full-width forms, mixed-script, digits, Latin extended, symbols — plus the name characters, has a glyph in the face that will actually print it;
the lookup uses the PDF engine's own font parser rather than a third-party library, so the check cannot disagree with the renderer.

The known edges are written down instead of hidden. Dingbats and emoji (✅ ✔)
exist in none of these text fonts and the engine cannot draw colour emoji,
so they are kept out of the report body. The display serif has only 974
glyphs and does not draw symbols like ① ★ ℃, which is why the
content-library check covers all slots rather than just the body. And CJK
characters inside a Latin report must go through the dedicated CJK slot —
two tests keep the renderers from disagreeing about that.

What I would do differently

Choose faces and weights before styling anything. Retrofitting a second weight after the layout exists means re-checking every page count by hand.
Read the renderer's source before choosing a font format. The variable-font decision looked like a taste question and was actually a parser question — two files settled it in ten minutes.
Treat everything you ship inside a document as a licence artifact. Font files enter the repo with their licences, or they enter someone else's legal review.
Measure what got embedded. Page count, glyph-per-face counts, file size, render time: all four moved, and three of them moved in ways I would have guessed wrong.

The result

Before: one CJK face and a UI font, with synthetic bold, depending on what
the server had installed. After: four families and eight static faces
embedded in the file, weights that are real members of their families, page
counts within one page of the previous layout, and a lint gate that fails
the build when a character has no glyph — so the failure mode moved from "a
customer sees a box" to "CI refuses the release". Which is the same trade as
everywhere else in this pipeline: make the mistake impossible to ship rather
than easy to notice.

I build automated document generation and multi-lingual PDF publishing pipelines — from layout engine customisation and compliant font embedding to CI visual verification gates. If your business reports, invoices, or client deliverables need to look identical on every machine, get in touch: WhatsApp +60 12-797 2969 · me@hoelee.com · hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

How I Vet an Open-Source Dependency Before Betting On It

Hoelee — Wed, 04 Nov 2026 04:26:01 +0000

Originally published on my blog — read it there with the full context, code and updates.

I was handed a 1,306-line architecture spec for a partner program. It read like a
plan: consistent, cross-referenced, confident. It named a self-hosted
open-source engine as the attribution and commission layer, and it assumed three
things about that engine — that it spoke OIDC so my existing identity provider
could log partners in, that its commission rules covered the five shapes the
business needed, and that its white-label feature was shipped.

Ten minutes of curl and grep later, two of those were false and the third was
half-true. None of that is unusual. The README describes the product; the
repository describes the software. The gap between those two is exactly where
integration projects die — and it is cheap to measure before you write any code.

The problem, phrased as the question I'd actually search for

How do I know an open-source project does what its documentation claims before
I build on it?

A spec is a claim about other software. The more polished the spec, the more its
assumptions read like facts — this one cited sources, defined stable IDs, and had
a source-of-truth matrix. What it never did was check whether the engine down the
chain implemented the features being assigned to it. So before installing
anything, I went looking for those specific features.

What I tried first, and why it wasn't enough

1. Read the README end to end. It sells the pipeline well:
click → identity → event → attribution → commission → payout. It never once
says which authentication protocol the app speaks. For a feature you need,
silence isn't neutral — but an absent mention is easy to skim past when the rest
of the page is convincing.

2. Searched the web. Several of my queries returned nothing usable. The
project is a few months old and its SEO surface is thin. For software that's
fine — the source is the documentation. It just means the reading has to happen
in the repo, not in a blog post.

3. Found a 78 KB feature design doc and assumed the feature shipped. It was
labelled "FINAL, post-review", listed three reviewers, and carried a date. I took
that as proof. That was the worst inference of the three, and the easiest to check.

The six checks that actually settled it

1. Read the repo's vital signs before any feature claim

curl -s -H "User-Agent: hermes" https://api.github.com/repos/<owner>/<repo> \
  | grep -E '"created_at"|"pushed_at"|"stargazers_count"|"forks_count"|"open_issues_count"|"archived"|"spdx_id"'

What came back: MIT, created 2026-04-23, last push 2026-09-10, 8 stars, 3 forks,
15 open issues, not archived. Read the same as: real, still moving, and very
small — one vendor, which also sells a hosted tier. Now I know how much of my
business I want resting on it before I start grading its features.

2. Grep for the feature instead of reading for it

grep -in 'oidc\|sso\|saml' README.md ARCHITECTURE.md docs/*.md | wc -l

Zero hits. The app authenticates with its own magic-link sessions and
per-persona API keys; it does not speak OIDC at all. That one line of output
killed a phase of the plan — "put the portal behind our SSO provider" — which
would otherwise have surfaced weeks later, mid-integration, with a partner
already invited. Reading for a feature you can't find is how you lose an
afternoon; grepping makes the absence loud.

3. Read the target branch line at the top of a feature doc

head -5 docs/white-label-custom-domains.md

Target branch: multi-tenant. The doc is a design for a branch that is not the
default branch — it describes work in flight, so the feature is not in the code
you would install. A finished-looking document with reviewers and dates can
document something unshipped. That isn't dishonesty; it's what design docs are.

4. Read the data model, not the feature list

The commission section named exactly two rule types — percent (with a recurring
flag) and fixed. The business spec had enumerated five. That gap isn't a bug;
it's a modelling job I'd have to do in configuration instead of expecting from
the product:

Spec assumed	What the engine actually has
first-order bonus	a `fixed` rule on the first event
event-specific rules (`lead` vs `invoice_paid`)	one rule per campaign — so: two campaigns
recurring commission	`percent` + `recurring: true`
percentage	`percent` ✅
fixed	`fixed` ✅

Two rule primitives, five behaviours — workable, but only if you design for it up
front instead of discovering it during implementation.

5. Check which paid tier gates the feature you need

For the database tool in the same stack, the edition table was blunt:

Feature I needed	Community (free)	Enterprise (paid)
API tokens, so I can automate it	✅	✅
SSO / OIDC	❌	✅
Page designer, hide branding, 2FA	❌	✅

The thing I wanted for the partner-facing door (SSO) sits behind the commercial
gate, while the thing I wanted for automation (an API token) is free. Knowing
which side of the wall each requirement falls on changes the design — before
you've designed around it.

6. Check that the money rail works in your country

The payout model was stripe_connect | manual, and self-hosted mode was
documented as "operator manages out-of-band" — i.e. you move the money yourself.
Stripe Connect is not how Malaysian businesses pay partners, so that one line
decided the entire deployment shape: self-host mode, manual payouts, and event
ingestion pushed from my own order system rather than leaning on the provider's
payment integration. That line is never on a marketing page, and it mattered more
than any feature.

Then: ask the fallback question before you need it

"What if the feature I need isn't there?" For the missing SSO the answer was:
don't put login in the app, put it in front of the app. authentik's proxy
provider exists to protect "applications that do not support native
authentication protocols such as OIDC, SAML, or LDAP", and it injects
X-authentik-username and X-authentik-groups headers upstream, so the reverse
proxy can do the gating. That turned "no SSO support" from a disqualification
into an afternoon of config.

What I'd do differently

Write the requirement list first, then go look for it. Five lines — the features I need — and search for those. Starting from the README's feature list means grading the software on the vendor's exam.
Treat "a doc exists" as different from "a feature ships". Read the branch line.
Grep, don't read. Presence is cheap to prove; absence is the thing you're hunting.
Decide the fallback before the decision, so a missing feature becomes a design change rather than a surprise.
Pin the commit. This project's README says its API is "stable but unversioned" and there is no OpenAPI description anywhere in the tree. There's no contract to compile against, so the commit hash is the contract — the endpoints I use go into my own docs, and upgrades get a regression pass.

The result

Six checks and about ten minutes of curl and grep corrected all three faulty assumptions in the spec before a single line of integration code existed. The project stayed viable — but
reshaped: no OIDC, manual payouts, a pinned commit, and a plan that no longer
depends on a feature sitting on an unmerged branch.

The spec wasn't wrong to be optimistic. It was wrong to be unverified, and that's
the cheapest thing on the whole project to fix.

Building on a self-hosted stack, and want a second pair of eyes on whether
those features will actually be there when you integrate? That's the kind of
review I do.

WhatsApp +60 12-797 2969 ·
me@hoelee.com ·
hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

DEV Community: Hoelee

Why I Still Bought a GPU to Run AI When Claude and GPT Are Obviously Stronger

The problem is not capability — it is dependency

"But local models are worse" — yes, and that is the wrong comparison

I treat capability as a threshold, not a ranking

Agents turn a small price difference into a large one

Open models crossed the useful line, and the line keeps moving

Fixed compute vs. renting compute

What I actually evaluate now

The short version

The Five-Step AI Roadmap Everyone Shares — and What I Actually Run in Production

Why this matters

Step 1: Fundamentals — enough to be dangerous, then stop

Step 2: Prompt engineering — the step with the lowest ceiling

Step 3: RAG — the hard half is everything except the retrieval

Step 4: Agents — the opportunity is real, and the agent is the smallest part

Step 5: Fine-tuning — the step I have never done, on purpose

What I'd do differently

The result

Want an AI stack that runs in your business?

Your Own Speech-to-Text Server: Faster Than Typing, Private by Default

Why this matters

What you need

The architecture

The build

Trap 1: CUDA Toolkit installs an incomplete compiler

Trap 2: it listens on localhost, so your firewall is innocent

Trap 3: n8n throws away your audio (twice)

Trap 4: the stock nginx config hijacks port 80

The authorisation model, honestly

What I'd do differently

The result

Could this work for your office?

Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day

The problem

What I tried, and why each thing failed

Round 1: it must be DNS / IPv6

Round 2: I picked the wrong IP

Round 3: the same command, two different outcomes

Round 4: nginx/1.30.1 is Telegram

The fix

What I'd do differently

The result

Migrating a CodeIgniter 4 App From IIS to OpenLiteSpeed

Why this matters

The setup

Step 1: Point the docroot at public/, not the project root

Step 2: The two fatals IIS was hiding

Fatal 1 — env() called too early

Fatal 2 — parent::__construct() in a CI4 controller

Why neither bug showed up on IIS

Step 3: The self-call that stops being a loopback

Step 4: Verify the actual output, not the status code

What I'd do differently

The result

Authentik 2025.8 to 2026.8: The Breakable Parts Nobody Warns You About

What is authentik, and why run it?

Why upgrade at all

Pitfall 1: Portainer is the source of truth, not the compose file

Pitfall 2: the storage mount moved (2025.12)

Pitfall 3: RBAC migration leaves a poisoned session table

Pitfall 4: trusted proxies are now opt-in (2026.8)

Pitfall 5: the one that broke SSO — authorization_flow vs authentication_flow

What I'd do differently

The result

Want single sign-on for your business?

Upgrading n8n v1 to v2: Seven Deprecations in One Log File

Why it matters

Start here: n8n tells you what's wrong at boot

1. Your env values can now fail schema validation

2. WEBHOOK_URL is deprecated for N8N_WEBHOOK_URL

3. Internal task runner mode is going away — and mine was already broken

4. Task timeout drops from 300s to 60s

5 & 6. Two compression-node limits shrink (2 GiB → 256 MiB, 5000 → 1000 entries)

7. The storage path renames in v3 — and you have a volume mounted at the old one

The one outside the deprecation block: my AI sandbox disabled itself

What I'd do differently

The result

Running automation you'd rather not babysit?

Upgrading CodeIgniter 4.6 to 4.7: The Breaking Changes the Guide Misses

Round 4: `nginx/1.30.1` is Telegram

Step 1: Point the docroot at `public/`, not the project root

Fatal 1 — `env()` called too early

Fatal 2 — `parent::__construct()` in a CI4 controller

2. `WEBHOOK_URL` is deprecated for `N8N_WEBHOOK_URL`

1. `Config\App::$permittedURIChars`

2. `Config\Format::$jsonEncodeDepth`