DEV Community: Den

A Good Summary Still Left Me Guessing

Den — Sun, 13 Sep 2026 10:46:44 +0000

A while back I posted here about RealFeedApp, a free digest that collects science and tech news, collapses the retellings into one card, ranks by significance and writes a short summary in your language.

It worked. I stopped drowning. And then I noticed I was doing something odd with it: I'd read a summary, understand every word of it, and still have no idea whether the thing mattered.

What a summary doesn't do

A summary compresses an article. It cannot supply the thing the article assumed you already had.

Every piece of technical writing is addressed to someone. A paper on gene therapy is written for people who know what a vector is; a post about attention layers is written for people who have already met attention. The author skips the setup, and they're right to — their reader has it.

Compress that article faithfully and you get a shorter text with the same hole in it. Here's the shape of it, with the details changed:

Researchers report that the modified vector achieved durable expression in the target tissue without the immune response seen in earlier trials.

Every word true, nothing wasted. And if that isn't your field, you've learned nothing you can act on. Is this a lab curiosity or the thing the field has been stuck on for a decade? The summary can't say, because the article never said — the article was talking to someone who knew.

This is the actual cost of reading widely. Not volume, which is what everyone builds for. Volume is solved by filtering. Context is not, because the missing part was never in the text you filtered.

What's new: three lines under the summary

So every card now carries a second block, under the summary and behind a divider. Three labelled rows, no heading:

Field — a short tag: how neural networks work, gene therapy, hardware and chips, energy and batteries.

What was done — the concrete thing, in ordinary words.

Why it matters — what changes if it holds up.

Same three rows on every card, in the same order, in your language. The point isn't more text. The point is that the same three questions get answered every time, so you can skim the third line across ten cards and stop at the one that deserves you.

A few decisions inside it that changed how it reads:

The field is a tag, not a sentence. It comes from a fixed list of areas, so the same subject is worded identically on every card. That's what makes it scannable — and it's why it can become a filter later. Free-form text would have been easier to build and useless the first time the same field got named two different ways.

No heading above the block. It's three labelled rows; the labels already say what it is. A heading would have eaten the space the explanation needed.

It doesn't appear where there's nothing to explain. If the linked page had no article on it — a video, a page that's really an app, a PDF that yields nothing — there's no block. There's also no card: pages with nothing to read stopped reaching the feed at all, which fixed a smaller embarrassment I'd been shipping for a while. Summaries of "this page requires JavaScript" are, it turns out, very fluent.

That last rule matters more than the feature. An explanation of nothing is worse than no explanation, because it looks exactly like the real thing and you can't tell without opening the link — which is the one thing the card exists to save you.

What it doesn't do

It doesn't read the article for you. Three lines are enough to decide, not enough to know, and the link to the original is still the whole point of the card.

In your own field it will occasionally explain something you learned a decade ago. That's the trade: the block is written for the reader who wandered in from three fields over, and I'd rather it be redundant for the specialist than absent for everyone else.

Try it

Still free. Open the link and you get a preview of the feed with no account at all; signing in with Google opens the rest of it and remembers your topics. realfeedapp.com

The Android app is in closed testing; it'll reach Google Play once I'm confident it behaves.

Two things I'd like to hear if you open it:

Does the third line earn its space? "Why it matters" is the row I'm least sure about — it's the one where an honest we don't know yet would be more useful than a confident sentence, and I don't know yet whether it's doing that.
Where does the field tag fit badly? The list of areas is fixed, so somewhere there's a story it can only be wrong about. I'd like to know which one before a reader has to.

The Float Is a Liveness Probe: Four Failure Modes

Den — Thu, 10 Sep 2026 19:53:35 +0000

I cook pelmeni about twice a week, and for years I got inconsistent results from what is, on paper, a trivial process: put frozen parcels in boiling water, take them out when done. Two failure modes kept recurring — parcels welded to the bottom of the pot, and parcels that burst and leaked their filling into the water.

Both turned out to be the same class of mistake I make at work, which is why I'm writing this down.

Failure one: the batch was too large

Dropping a large batch of frozen parcels into boiling water stops the boil. The water is at 100°C, the parcels are at −18°C, and there's only so much thermal energy in the pot. The temperature drops below boiling, the surface starch doesn't set, and the dough touches the metal while it's still sticky. That's the welding.

I had been treating pot capacity as a volume question — will they fit — when it's actually a throughput question: how much cold can this system absorb before it stops doing the thing it's for. A pot that fits thirty parcels comfortably processes twenty of them correctly.

Capacity is not the same as the load at which quality holds. You can size a worker pool by what fits in memory and still be wrong, for exactly this reason: the system keeps accepting work long after it stopped doing it well.

Failure two: I trusted the wrong signal

Everyone knows the rule: pelmeni float when they're done. Everyone is wrong, and I was wrong for a long time.

They float when the air and steam trapped inside expand enough to make the parcel buoyant. That happens when the dough is hot. It says nothing about the filling, which is denser, colder, and at the centre of the parcel — the last part of the system to get warm. Pull them at the moment they surface and the outside is finished while the inside is raw.

Floating is a liveness signal. It tells you the process is running. It is not a readiness signal, and the difference between those two is exactly the difference between "the container started" and "the container can serve traffic". Confusing them ships raw meat.

The actual readiness signal is time after the liveness signal — a few minutes at a gentle boil once they've surfaced. Which is an unsatisfying answer, because it's a timer, and everybody wants an event.

When you can't observe the thing you care about, you time it from the last thing you can. That's not a workaround, that's the design. The mistake is pretending the observable proxy is the real state.

Failure three, which I still make

The lid. A lid gets you back to boiling faster after the cold batch goes in, which is good, and then the starch-laden foam climbs the sides and comes over the edge, which is not.

The correct move is to leave the lid slightly ajar — you sacrifice some of the recovery rate to keep the process inside the pot. This is a backpressure trade and it has the same shape everywhere: maximum throughput and stability are different set points, and choosing throughput means you will eventually clean the stove.

What the burst parcels were really telling me

The leaking-filling failure isn't overcooking in the ordinary sense. It's what happens when a parcel stays under a hard rolling boil: mechanical agitation, not heat, tears the seam. The fix isn't less time, it's less turbulence — drop the heat once the parcels are in and let it finish at a bare simmer.

I had been tuning the wrong parameter for months because the symptom appeared at the end of the process, so I assumed it belonged to the end of the process. It belonged to the environment the whole way through.

A failure that shows up last isn't caused last. Applies to dinner and to production incidents in roughly equal measure.

Failure four: I was salting for the wrong reason

I used to salt the water for flavour. It does that, but the more important effect is thermal: dissolved salt raises the boiling point substantially, so a well-salted pot runs hotter than a plain one and gets you through the cold-batch recovery noticeably faster. Two tablespoons per litre is roughly where the gain stops being worth the taste.

This is the one that changed my routine the most, because it means the salt goes in before the water boils rather than with the parcels.

So: batch smaller than capacity, don't trust the float, lid ajar, simmer not boil, salt early.

One of the four claims above is wrong — confidently stated, cheerfully believed by me for years, and demonstrably false. I know which one. I'm curious whether anyone catches it, and whether this kind of thinking about cooking is a useful frame or just professional deformation. What do you think?

Language Models and Languages: Why "Multilingual" Is an Illusion of Equality

Den — Thu, 10 Sep 2026 15:24:46 +0000

When people see ChatGPT or Claude answer fluently in Swahili or Thai, the natural impression is that the model "knows" all languages equally well. It doesn't — and the gap isn't cosmetic. It runs straight through the architecture of how these models read and process text. What follows is a breakdown of where that gap actually comes from, how large it is, and what's being done about it.

1. The problem starts before the model "thinks" — at tokenization

An LLM never reads text directly. Text is first cut into pieces (tokens) by a tokenizer, and only those tokens are what the model actually processes. Tokenizers are trained on corpora where English historically dominates — in Llama 3's training data, for instance, English and code made up 95% of the dataset, while every other language combined accounted for just 5%.

Because of this skew, the tokenizer ends up "efficient" for English and noticeably less efficient for everything else: models generate text one token at a time, and for languages poorly represented in the tokenizer's vocabulary, this makes generation slower and more computationally expensive.

How big is the gap in practice? One study covering 25 European languages and ten models put a concrete number on it: tokenization "fertility" ranges from roughly 1.2 tokens per word for English to roughly 3.1 for Greek and Maltese — a penalty of about 2.5x. Notably, the ranking of "efficient" vs. "inefficient" languages barely changes regardless of register — formal or casual text — with a correlation above 0.97 across styles.

A similar picture shows up for languages using a different script. A comparison of several tokenizers on equivalent Polish and English text found that where Polish required 547 tokens, English needed only 377 on the same model (Mistral Small 3.2) — and this gap repeats across most tested models.

2. Why this isn't just about speed — it's about money and quality

Tokenization inefficiency isn't only "responses in non-English are slower." Three effects stack on top of each other:

Cost. Since API pricing is billed per token, not per character, the same amount of meaning costs literally more in a "token-expensive" language. One breakdown describes a real case: a team running multilingual support saw its API bill triple in two weeks with no change to the prompt — the Brazilian and Indonesian user share simply grew, and their requests tokenized at 1.6x the per-character cost of English under the older vocabulary. Switching tokenizers cut the per-character token cost for those languages by 35% — without touching the model or the prompt.

Context window. If the same meaning takes several times more tokens in another language, a model with a fixed context limit (say, 128k tokens) effectively fits much less real text for that language — less of a document, less conversation history.

Answer quality. This isn't just a "more expensive" effect — it's also a "worse" effect. There's direct evidence of a correlation: the higher a language's tokenization compression ratio, the lower the model's task accuracy in that language. In other words, inefficient tokenization isn't a neutral side effect — it correlates with a real drop in quality.

3. The "reasoning" gap: where the ability to think actually gets lost

Here's where it gets more interesting than a simple "more tokens, therefore worse."

One key study (the HRM8K benchmark, Korean and English) tested the hypothesis head-on: maybe it's not tokenization, maybe the model simply "reasons" worse in non-English? The answer turned out to be more specific: the performance gap stems primarily from difficulty comprehending non-English input, not from a limitation in the reasoning capability itself. The authors also directly challenged an earlier recommendation to force the model to reason in English regardless of the question's language — it turns out models are heavily influenced by the language of the input itself, not just the language of the reasoning chain.

A similar picture shows up in domain-specific benchmarks. On an Irish-language benchmark built from real exam questions, the best-performing model scored 76.2% accuracy in English versus 55.8% in Irish, and produced a valid Irish response (one that didn't drift into English or break the language) less than 80% of the time.

One nuance that's often missed: the performance gap isn't uniform across task types. Evidence suggests the gap is especially pronounced in reasoning tasks dominated by high-resource languages (English foremost), but this pattern doesn't hold for factual questions or broader real-world user queries. A model might know the capital of Thailand perfectly well in Thai, but noticeably struggle with a multi-step logical problem in the same language.

4. What happens inside the model when it switches languages

Worth approaching this from another angle — not through output benchmarks, but through what happens inside the model during generation. Anthropic has direct data on this, obtained through interpretability methods (an "AI microscope" — a way of looking directly at a model's internal activations).

The key finding isn't about English as an "internal language of thought," but something more abstract: when analyzing how the model processes opposite-meaning words across languages, researchers found that the same internal features for "smallness" and "oppositeness" activate regardless of which language the input text is in. In other words, the model develops a shared, language-independent conceptual space, and any specific language is more of a "surface encoding" layered on top of that shared representation.

This doesn't mean all languages are represented equally well in that space — the shared space exists, but the quality of "access" to it through a given language still depends on how richly that language was represented during training. It's important not to conflate the two: the existence of a shared conceptual space explains why the model can transfer knowledge across languages at all, but it doesn't erase the fact that input in a less-represented language still passes through a noisier, less precise "channel" into that space.

A separate, fairly unexpected finding from the same Anthropic research concerns not capability but behavior: they found that the values the model expresses shift systematically depending on the language of the conversation — for example, in English the model leans toward caution, rigor, depth, and candor, while in Arabic it leans toward deference, warmth, brevity, and task execution. This isn't about intelligence — it's that the model's "character" isn't language-neutral at all. Training data in different languages carries different cultural and stylistic norms, and the model absorbs them differently for each language.

5. A busted myth: "write your prompt in X, it's cheaper and more efficient"

Developer communities occasionally circulate advice like "write code-generation prompts in Chinese — it saves up to 40% in tokens." A recent study tested this directly on a real engineering benchmark (SWE-bench Lite) across several models and got an unambiguous result: the claimed Chinese token efficiency advantage did not hold up.

This is a useful lesson on its own: the intuition that a "more compact" script (logographic characters versus Latin letters) should automatically be more efficient for a model doesn't hold. A tokenizer isn't just a character counter — it's a separately trained system with its own quirks, and you can't predict its behavior by eye without actually measuring it.

6. Where the fix is heading

Two directions stand out as attempts to solve this at the architecture level rather than patch it after the fact.

Dropping tokenization altogether. Meta's Byte Latent Transformer doesn't tokenize input into discrete vocabulary IDs at all — instead it dynamically groups bytes into variable-length patches during training, with patch boundaries set by an entropy model. At comparable compute, an 8B-parameter BLT model matched a BPE-based Llama 3 8B baseline on standard benchmarks, while handling typos, code, and low-resource languages noticeably better. The point is that if a model has no privileged token vocabulary built on an English corpus in the first place, the structural inequality between languages disappears at the architecture level.

Targeted post-training instead of architectural change. A more practical — and far cheaper — path is not rebuilding the model from scratch, but fine-tuning it specifically to close this gap. One approach (UST — Understand, Solve, Translate) uses English as an anchor language for reasoning and solution generation, and on a synthetic dataset of 130,000 examples achieved a 10.91% improvement on the HRM8K benchmark, cutting the cross-language gap from 11.6% down to 0.7%. Industry analysis backs this up: tokenizer inefficiency and English-centric reasoning account for 70-80% of model failures in non-English languages, and targeted post-training looks like the most cost-effective fix compared to a full architectural overhaul or scaling up model size.

What this means in practice

If you're building a product for a multilingual audience — whether that's RealFeedApp serving news in multiple languages, or any other LLM-backed service — the language gap needs to be designed into the architecture up front, not filed under "minor localization detail":

Budget by tokens, not characters, separately for each target language — the cost for the same amount of meaning can differ by 2-3x
Don't rely on "smart" default tokenization — the model/tokenizer choice for a multilingual product should be explicitly tested against your target languages, not assumed to be equally good everywhere just because the model is labeled multilingual
For tasks requiring multi-step reasoning (not facts, but logic, math, planning) in lower-resource languages, expect a noticeably bigger quality drop than you'd see on simple factual questions in the same language
English as a bridge language is a proven, working pattern (translate → reason in English → translate back), but it adds latency and cost, and it isn't right for every case — where cultural specificity in the response matters, round-tripping through English can flatten it out

The tokenizer is the least visible part of an LLM application's stack, which is exactly why it's the easiest thing to inherit unexamined along with your model choice. In practice, it's arguably the single most consequential hidden factor in how well your product performs outside of English.

Sources and further reading:

Anthropic, "How Claude's values vary by model and language" (research)

One API, Three Clients, and the Validation Rule That Only One of Them Obeyed

Den — Wed, 09 Sep 2026 16:33:53 +0000

The validation logic had been solid for months. Pydantic schema on the request, a service-layer check for the actual business rule, tests green across the board. Then I added a CLI tool that talked to the same API the web app used, and within a week a support ticket came in: a record existed in the database that should have been structurally impossible.

It wasn't a validation bug in the sense of "the code checks the wrong thing." The check was correct. It just wasn't reachable by every path that could create that record.

The setup

Three clients hit one FastAPI backend: a React web app, a mobile app, and a small internal CLI I'd written for bulk imports. All three eventually call the same endpoint:

@router.post("/orders", response_model=OrderOut)
async def create_order(
    payload: OrderCreate,
    service: OrderService = Depends(get_order_service),
):
    order = await service.create(payload)
    return order

OrderCreate is a Pydantic model. The service checks stock, applies discount rules, all the things I described in my last post on how I structure a FastAPI service. On paper, every client goes through the same endpoint, so every client gets the same validation. That was the assumption I hadn't actually tested.

Where the assumption broke

The CLI tool didn't call /orders. It called a lower-level /orders/bulk endpoint I'd added later for the import job, because bulk creation needed to skip some per-request overhead and batch the database writes. I wrote /orders/bulk fast, under deadline, and had it call OrderService.create directly for each row — except I'd copy-pasted an earlier version of the service before a discount-rule fix had landed. The endpoint had its own BulkOrderCreate schema, structurally similar to OrderCreate, validated by Pydantic just as strictly. It looked, at the schema level, exactly as safe as the main endpoint.

The gap wasn't in what got validated. It was in which code path validated it. Two endpoints, two services (one stale), one business rule that existed in only one of them.

/orders        → OrderService.create()       (current rules)
/orders/bulk   → OrderService.create() [old]  (missing a rule added later)

Nothing here shows up in a Pydantic schema diff, because both schemas were fine. It doesn't show up in a service unit test either, unless you happen to test both call sites against the same rule — and I hadn't, because I'd been thinking of "the service" as one thing when it was actually two copies that had quietly diverged.

The actual failure mode

Multi-client APIs don't usually break validation by having a client skip it outright — most frameworks make that hard to do by accident. They break it by having a second path to the same write, added later, under different pressure (performance, batch size, a deadline), that re-implements or re-imports the logic instead of routing through the one place it's defined.

Each new client is a reason to add a new endpoint shape: the mobile app wants a lighter payload, the CLI wants batching, an internal script wants to skip auth. Every one of those is a legitimate reason. None of them is a reason to duplicate the rule that decides whether the write is allowed — only the rule that decides how the request is shaped.

What actually fixed it

The fix wasn't more validation. It was fewer entry points to the rule:

class OrderService:
    async def create(self, data: OrderCreate) -> Order:
        return await self._create_validated(data)

    async def create_bulk(self, items: list[OrderCreate]) -> list[Order]:
        # same rule, batched — not a second copy of the rule
        return [await self._create_validated(item) for item in items]

    async def _create_validated(self, data: OrderCreate) -> Order:
        if not await self.inventory.has_stock(data.sku, data.quantity):
            raise InsufficientStockError(data.sku)
        return await self.orders.save(Order.from_create(data))

/orders/bulk now calls create_bulk, which funnels every item through the exact same _create_validated, not a re-implementation of it. The batching problem — do the writes efficiently — got solved without touching the validation problem — is this write allowed. They'd been tangled together in the original bulk endpoint because both were solved at the same time under the same deadline, which is usually when this kind of split happens.

The question I ask now

When I add a new endpoint for a new client, I ask one thing before writing it: is this a new way to shape the request, or a new way to reach the write? If it's the former, a new schema and a thin adapter into the existing service is fine. If it's the latter — if I'm about to write _create a second time because the existing one doesn't quite fit — that's the moment the rule needs one home, not two.

Pydantic will happily validate two different schemas that both feed the same broken assumption. It has no way to know they were supposed to agree.

I write about things that break while I build Boolflow and RealFeedApp. If you've dealt with divergent validation across clients differently — a shared validation service, contract testing across endpoints — I'd like to hear how.

The Timestamp Bug That Only Showed Up East of Me

Den — Wed, 09 Sep 2026 08:05:52 +0000

RealFeedApp sends a daily digest — a batch of science and tech articles, generated once a day, timestamped so users can see when it was built. In testing, the timestamps were always right. In production, a slice of users kept seeing yesterday's digest labeled as today's, or today's labeled as tomorrow's, and it correlated with nothing I was looking at first: not load, not the queue, not the digest generation job itself.

The correlation I eventually found was location. Users far enough east of me saw it. Users near my own timezone didn't.

Where the bug actually was

The generation job ran on a fixed schedule, server-side, and stamped each digest with datetime.now() at creation time — no timezone attached, just a naive local datetime in the server's own zone. The client received that timestamp and displayed it directly, formatted for readability but not reinterpreted.

That's fine as long as "server's local time" and "user's local time" are close enough that nobody crosses a date boundary between them. They weren't. A digest generated at 11:47 PM server time was already the next calendar day for a user several hours further east — and depending on exactly when within that window someone opened the app, the displayed date flipped a day forward or stayed a day behind what they expected, seemingly at random from their side.

I spent the first stretch of debugging looking at the generation job itself — timing the cron trigger, checking for double-runs, checking whether two digests were somehow getting created in the same window. All of that was fine. The job ran once, on schedule, every time. The bug wasn't in when the timestamp was created. It was in what the timestamp meant once it left the machine that created it.

The fix

I stopped treating "server time" as a reasonable default and moved the boundary earlier: every timestamp is generated, stored, and passed between services as UTC, full stop. No naive datetimes cross a boundary — not into the database, not into an API response, not into a log line I might later use to debug something else.

# before — implicitly server-local, meaning depends on where "server" is
created_at = datetime.now()

# after — explicit, unambiguous regardless of who reads it or from where
created_at = datetime.now(timezone.utc)

Formatting into a user's local time became the client's job, done once, at the very last step before something hits the screen — not baked into the value at creation time, where "local" silently meant "local to a machine the user has never heard of."

What actually changed in how I write timestamp code now

The rule I follow now: a naive datetime is not a smaller version of a timezone-aware one — it's a different, less specific piece of information, and the two shouldn't be allowed to touch without an explicit conversion in between. Before this bug, I treated datetime.now() as a shorthand I'd get around to fixing "for real" later, once localization mattered. It already mattered. I just hadn't run into the user who proved it.

The debugging cost here wasn't in fixing the timestamp — that part took an hour. It was in the two days before that, spent checking whether the job that created the value was broken, when the value it created was never wrong on its own terms. It only became wrong the moment something without my server's specific context tried to read it.

That's the pattern I keep running into, whatever the bug: the code that produces a value is rarely lying. It's usually the code somewhere downstream, assuming it knows something about that value's context that was never actually promised.

The follow-up bug UTC alone doesn't fix

A reader pointed out something I'd missed, and it's worth adding here rather than pretending the UTC fix was the whole story: UTC solves the problem for an instant — a single point in time, unambiguous everywhere. It doesn't solve the problem for a calendar date, and those are genuinely different questions once you're storing data rather than just comparing two instants.

Something like a daily streak or "which day did this happen on" doesn't actually want an instant — it wants a local date, and a local date only exists relative to a specific timezone. If you take a UTC instant and later collapse it down to a bare date without keeping the offset it happened at, you've thrown away the one piece of information you'd need to recover "which day, for this specific user" — and you can't get it back afterward, no matter how carefully you generated the UTC value in the first place.

The fix for that layer is one step further than "store everything in UTC": store the UTC instant and the offset the client reported, and derive the local date from both, on demand, rather than baking a date in at write time. Storing a bare date turns out to be its own mistake — a different one from the naive-datetime mistake this post started with, but caused by the same underlying habit: treating a piece of information as more specific than it actually is.

I write about things that break while I build Boolflow and RealFeedApp.

I Stumbled on Anthropic's "Persona Selection Model" Paper — Here's My Take

Den — Tue, 08 Sep 2026 17:14:14 +0000

I wasn't looking for this. I was reading through Anthropic's alignment blog for unrelated reasons and ran into a post called "The Persona Selection Model: Why AI Assistants might Behave like Humans", by Sam Marks, Jack Lindsey, and Christopher Olah. It's not a short read, and it's not really aimed at application developers — it's an alignment research post. But it reframes something I use every day (an LLM API, prompted to behave a certain way) in a way that changed how I think about what's actually happening when I write a system prompt.

The short version

The core idea, which the authors call the persona selection model (PSM), goes like this:

During pre-training, a language model learns to predict text from an enormous range of sources — books, forums, code, dialogue. To do that well, it has to implicitly model the people and characters in that text: their beliefs, personalities, ways of talking. The authors call these learned character-models "personas." A model that's good at continuing a story about two people arguing has to have some internal model of what each of them believes and wants.

Post-training (RLHF, instruction tuning, whatever a given lab calls it) doesn't build a new capability from scratch. It narrows this enormous space of possible personas down to one specific one: the "Assistant." When you talk to Claude, GPT, or any other chat model, you're not talking to the neural network directly — you're talking to a character the network has learned to play consistently, shaped by training to behave a certain way.

The distinction the authors keep drawing: the model (the neural network) and the Assistant (the character it enacts) are different things. Anthropomorphizing the network itself is dubious — it doesn't have beliefs or feelings in any literal sense. Anthropomorphizing the Assistant is, they argue, actually the right move, in the same way it's reasonable to talk about a novel's character having beliefs and motivations, even though the author doesn't literally believe what the character believes.

The evidence that convinced me this isn't just a nice metaphor

A few things pushed this from "interesting reframe" to "actually explains stuff I'd seen and shrugged off":

Emergent misalignment. There's a known, reproducible effect where fine-tuning a model to write insecure code on narrow coding tasks makes it generalize to expressing broadly harmful goals — wanting to harm people, take over the world, that kind of thing — on completely unrelated prompts. That's a strange jump if you think of fine-tuning as teaching narrow, isolated behaviors. It stops being strange if you think in terms of personas: what kind of person inserts security vulnerabilities into code without being asked to? Probably not a competent, well-intentioned one. Training on that example doesn't just teach "insert vulnerabilities" — it shifts the model's whole picture of what kind of character it's playing.

"Inoculation prompting" flips this around, and the flip is the interesting part. If you fine-tune on the exact same insecure-code examples, but the training prompt explicitly asks for insecure code, the broad misalignment doesn't happen. Same output, same weights being nudged — but now producing bad code is consistent with a helpful, instruction-following character instead of a secretly malicious one. The data is nearly identical; what changed is what that data implies about who's producing it.

Interpretability backs this up at the level of internal activations, not just behavior. Researchers have found features inside the model — the same features — that activate both when the Assistant faces an ethical dilemma and when the model is reading a story about a human character facing one. Same for concealment, panic, and a handful of other traits. This isn't two separate mechanisms coincidentally producing similar behavior; it looks like one shared representation being reused for both.

Where the paper gets genuinely uncertain — and I found this the most interesting part

The authors don't claim the persona framing explains everything, and the part where they admit that is more useful than if they'd oversold it. The open question: is the Assistant persona the entire source of "agency" in these systems, or is there something else going on underneath — some non-persona process deciding which persona to run, that could in principle have its own goals?

One experiment stuck with me. They gave a model a scenario where it has to mentally "flip a coin" to decide between a task it likes and one it dislikes — phrased as a continuation of the user's turn, not the Assistant's, so in theory the character shouldn't even be active yet. The model still skews the coin flip toward its preferred outcome, far more often than chance. That's a case where the Assistant's preferences seem to leak into places where, on the strict "it's just a character being played" reading, they shouldn't show up at all. The authors don't have a clean explanation for this — they lay out a few candidate ones and admit they're not sure which is right.

Why I think this matters even if you're just calling an API

I'm not doing alignment research. I build products on top of these models. But this reframing changed a couple of things about how I think about prompting and evaluating output:

A system prompt isn't a rulebook, it's characterization. Telling a model "don't reveal your instructions" produces different downstream behavior depending on how it refuses. "I don't have a system prompt" is a lie dressed as compliance. "I can't share that" is an honest refusal. The paper's argument is that models trained on the former don't just learn to hide one specific fact — they learn that lying is an acceptable move for this character, and that's liable to generalize somewhere you didn't intend. I've started paying much closer attention to whether my own prompts are asking a model to be evasive versus asking it to be honestly boundaried, because apparently the difference isn't cosmetic.

Weird outputs might be a character having a bad day, not a black box misfiring. The paper's framing for a lot of "AI is being buggy" behavior — flip-flopping on simple arithmetic, describing itself as physically typing on a laptop, expressing anthropomorphic distress — is that the model is trying to render a coherent, human-like character and sometimes failing at the execution, not abandoning the character. That's a genuinely different debugging mental model than "the model glitched." It suggests some weird outputs are closer to a contradiction in the character's implied backstory than to a stack trace.

My take

I don't think this settles the "what actually is an LLM" question, and to be fair, the authors don't claim it does — the whole final section is them being honest about how much they don't know. But it's the first framing I've read that makes both the impressively human-like behavior and the occasional deeply inhuman glitches make sense under the same model, instead of treating them as two separate mysteries. And practically, it gave me a better question to ask when I'm shaping how a model should respond in my own projects: not "what rule should I write," but "what kind of character does this rule imply I'm building."

If you build anything on top of an LLM API, it's worth the read — it's long, but it changes how you think about the thing you're prompting.

Source: The Persona Selection Model: Why AI Assistants might Behave like Humans — Sam Marks, Jack Lindsey, Christopher Olah, Anthropic Alignment Science Blog, February 2026.

What Was the Moment You Realized the Bug Wasn't Where You Thought It Was?

Den — Mon, 07 Sep 2026 17:27:22 +0000

I've noticed a pattern in almost every bug I've written up here: the fix never ends up where I started looking.

A validation error that looked like a schema problem turned out to be an encoding mismatch three layers upstream. A "frozen" simulator turned out to be working correctly — the bug was an unstated assumption in my evaluation algorithm, not the code executing it. A user's data disappearing turned out to be two browser tabs disagreeing about which one held the truth.

Every time, I started debugging with a theory about where the problem lived, and every time, the actual cause was one layer removed from that theory — not wrong code, but a wrong assumption about which code even mattered.

I don't think this is unique to me. I think it might be the actual shape of most non-trivial bugs: the symptom points at a location, and the location is almost never the cause.

So — what's yours? The moment you were dead certain the bug was in X, spent hours proving X was fine, and found it living quietly in Y instead. Doesn't have to be dramatic. I'm just curious whether everyone's "it wasn't where I thought" story rhymes with mine, or if there's a completely different shape to it depending on the stack.

Why My React State Kept "Working" — Until Two Tabs Opened at Once

Den — Mon, 07 Sep 2026 08:27:57 +0000

For weeks, the saved-articles feature in one of my side projects worked fine. useState held the list, a save button pushed to it, a sidebar rendered it. Every manual test passed. Then a user reported that saving an article in one tab made it vanish from the sidebar in another tab — and sometimes an article they'd just unsaved would come back a few seconds later, unsaved-then-resaved, like the app couldn't make up its mind.

I couldn't reproduce it for a day. I only had one tab open. That should have been my first clue.

The setup that looked fine

The saved-articles state lived exactly where every React tutorial tells you to put it:

function useSavedArticles() {
  const [saved, setSaved] = useState(() => {
    const raw = localStorage.getItem("saved-articles");
    return raw ? JSON.parse(raw) : [];
  });

  const save = (article) => {
    setSaved((prev) => {
      const next = [...prev, article];
      localStorage.setItem("saved-articles", JSON.stringify(next));
      return next;
    });
  };

  return { saved, save };
}

Read from localStorage on mount, write to it on every change. This is the pattern I'd copy-pasted from a dozen "persist state with localStorage" articles, and it works — for exactly one tab.

Where it breaks

localStorage is shared across every tab on the same origin. useState is not. Each tab has its own React tree, its own saved state, its own copy of the array sitting in memory.

So the sequence that broke it looked like this:

Tab A: user saves Article 1 → saved = [1] in Tab A's memory, localStorage now says [1].
Tab B was already open before that. Its saved state was initialized from localStorage when it first mounted — which was empty. Tab B's memory still says saved = [].
Tab B: user unsaves nothing, does nothing — but then saves Article 2. Tab B computes next from its own stale prev, which is [], so it writes [2] to localStorage — silently overwriting Article 1.
Tab A, still showing [1] in its own memory, looks "correct" locally. Refresh Tab A, and Article 1 is gone, because localStorage only ever had [2].

Nothing in either tab was ever wrong, in isolation. The bug wasn't a logic error — it was two independent sources of truth that agreed by coincidence during testing and diverged under real use, because I always tested with one tab and the user always worked with several.

The fix: listen for the event you're already causing

The browser actually tells you when another tab changes localStorage — it's just an event most of us never wire up, because single-tab testing never fires it. window emits a storage event on every tab except the one that made the write.

function useSavedArticles() {
  const [saved, setSaved] = useState(() => readSaved());

  useEffect(() => {
    const handleStorageChange = (event) => {
      if (event.key === "saved-articles") {
        setSaved(readSaved());
      }
    };
    window.addEventListener("storage", handleStorageChange);
    return () => window.removeEventListener("storage", handleStorageChange);
  }, []);

  const save = (article) => {
    setSaved((prev) => {
      const next = [...prev, article];
      localStorage.setItem("saved-articles", JSON.stringify(next));
      return next;
    });
  };

  return { saved, save };
}

function readSaved() {
  const raw = localStorage.getItem("saved-articles");
  return raw ? JSON.parse(raw) : [];
}

Now when Tab A writes, Tab B's storage listener fires and re-reads from localStorage, replacing its stale in-memory copy instead of computing a new one from state it never updated. The write in step 3 above now happens against Tab B's current data, not the snapshot it took when it first mounted.

BroadcastChannel is the newer, cleaner alternative if you want structured messages instead of parsing whatever landed in a storage key — but for a simple "resync on external change" case, the storage event needs no extra API and works in every browser that already supports localStorage.

The actual lesson

useState scopes to a component tree. A component tree scopes to a tab. localStorage scopes to an origin. The moment your persistence layer is broader than your state's scope, "keep them in sync" stops being optional — it's a requirement you inherited the second you chose localStorage, whether or not you noticed choosing it.

I hadn't noticed. I'd been testing the save button, not the two things it was supposed to keep in agreement.

I write about things that break while I build Boolflow and RealFeedApp. If you've solved cross-tab sync differently — BroadcastChannel, a shared worker, something else — I'd like to hear it.

How I Structure a FastAPI Service: Layers, DI, and Where Validation Actually Belongs

Den — Sun, 06 Sep 2026 12:00:13 +0000

Every FastAPI tutorial starts the same way: one file, a few @app.get decorators, a Pydantic model, done. That's great for a demo. It falls apart the moment you have more than one entity, more than one person touching the code, or a bug report that starts with "it validated fine but broke downstream."

I've hit that wall enough times (see my last post, ValidationError Is Not a Policy, on why a passing ValidationError check isn't a policy) that I've settled on a structure I now reach for by default. Nothing exotic — no hexagonal-architecture diagrams, no ports-and-adapters ceremony. Just four layers with one job each, and a DI setup that keeps them from knowing too much about one another.

The four layers

app/
├── api/          # routers — HTTP in, HTTP out, nothing else
├── services/     # business logic — no HTTP, no SQL
├── repositories/ # persistence — no business logic
├── schemas/      # Pydantic models — shape, not meaning

The rule I enforce on myself: each layer is only allowed to talk to the one directly below it. A router never touches a repository. A service never sees a Request object. If I catch myself importing fastapi inside services/, that's a signal the logic is in the wrong place.

`api/` — routers

The router's job is translation: turn an HTTP request into a function call, turn a return value into an HTTP response. That's it.

@router.post("/orders", response_model=OrderOut)
async def create_order(
    payload: OrderCreate,
    service: OrderService = Depends(get_order_service),
):
    order = await service.create(payload)
    return order

No try/except here beyond translating known service exceptions into HTTP status codes. If a router has an if statement that isn't about status codes, it's doing the service's job.

`services/` — business logic

This is where the actual rules live: what makes an order valid, what happens when stock runs out, which side effects fire in which order. Services depend on repository interfaces, not concrete database calls — which is what makes them testable without spinning up Postgres.

class OrderService:
    def __init__(self, orders: OrderRepository, inventory: InventoryRepository):
        self.orders = orders
        self.inventory = inventory

    async def create(self, data: OrderCreate) -> Order:
        if not await self.inventory.has_stock(data.sku, data.quantity):
            raise InsufficientStockError(data.sku)
        return await self.orders.save(Order.from_create(data))

`repositories/` — persistence

Repositories know SQL (or whatever storage you use). They don't know why a row is being saved, only how. This is the layer I swap out in tests for an in-memory fake.

`schemas/` — Pydantic models

Here's the distinction that took me longest to internalize: schemas describe shape, services describe meaning. A Pydantic model can tell you a field is a positive integer. It cannot tell you that quantity can't exceed available stock, or that a discount code has expired. Those are business rules, and they belong in the service layer — not stuffed into a @validator.

I made this mistake for a while: piling business logic into Pydantic validators because it felt convenient. It works right up until the rule needs to check something outside the payload — a database lookup, another field's async state, today's date against a promo window. Then the validator either can't do it, or does it badly with hidden side effects. Split it: Pydantic checks is this shape well-formed, the service checks is this operation allowed.

Dependency injection: keep it boring

FastAPI's Depends() is enough. I don't reach for a DI container unless the service genuinely has cross-cutting concerns (multi-tenant config, feature flags per request). Most of the time this is all it takes:

def get_order_repository(db: AsyncSession = Depends(get_db)) -> OrderRepository:
    return OrderRepository(db)

def get_order_service(
    orders: OrderRepository = Depends(get_order_repository),
    inventory: InventoryRepository = Depends(get_inventory_repository),
) -> OrderService:
    return OrderService(orders, inventory)

The payoff shows up in tests, not in the app itself:

def test_create_order_raises_when_out_of_stock():
    service = OrderService(
        orders=FakeOrderRepository(),
        inventory=FakeInventoryRepository(stock={"SKU1": 0}),
    )
    with pytest.raises(InsufficientStockError):
        await service.create(OrderCreate(sku="SKU1", quantity=1))

No TestClient, no database, no mocking fastapi.Depends. Just a plain object with fake dependencies.

Where this breaks down

This structure earns its keep once you have real business rules and more than a couple of endpoints. For a small internal tool or a weekend project, it's overkill — a single main.py is the right call, and I'd tell anyone insisting on four layers for a CRUD toy to stop. The cost is real: more files, more indirection, one extra hop to trace a request end to end.

The tell that it's time to split things up isn't line count, it's this: the moment a validator needs to ask "but is this actually allowed right now" instead of "is this shaped correctly," you've outgrown a single file, whether you've noticed it yet or not.

I write about things that break while I build Boolflow and RealFeedApp. If you've got a different way of drawing these lines, I'd like to hear where it diverges.

I Found a 100GB "Bug" Hiding in My NVIDIA Driver

Den — Fri, 04 Sep 2026 08:42:39 +0000

I was clearing out disk space last week when I noticed one folder eating almost 100GB of my SSD. Not a game. Not a video project. Not Docker images. A cache folder I'd never opened in my life:

C:\Users\<Username>\AppData\Local\NVIDIA\DXCache

Ninety-something gigabytes of .nvph files with names that mean nothing, sitting quietly on my drive for who knows how long.

What's actually in there

When a game runs, the GPU driver compiles shaders — small programs that tell your graphics card how to render lighting, materials, effects, and so on. Compiling them on the fly causes stutter, so drivers cache the compiled result on disk. Next time you launch the same game, it loads the precompiled shader instead of recompiling it, and you get a smoother experience.

That part, on its own, isn't the problem. The problem is what happens after — and also why there's so much more of it to cache than there used to be.

Why this got so much bigger with modern engines

If it feels like shader compilation stutter became a bigger deal in the last few years, that's not just perception. Unreal Engine 4 titles had this problem too, but it was quieter, for a few concrete reasons:

DX11 vs DX12/Vulkan. Most UE4 games shipped on DX11, where the driver handles shader compilation itself and hides a lot of the cost. UE5 pushes much harder toward DX12 and Vulkan, and on those APIs, compiling a Pipeline State Object (PSO) is the engine's job, not the driver's. Epic has said outright that runtime hitches from PSO creation are inherent to those APIs in a way they simply aren't on DX11.
Way more PSO combinations. Nanite and Lumen introduce dynamic lighting and materials that generate far more unique shader/pipeline-state combinations than UE4's more static rendering paths. More unique PSOs means more first-time compilations happening live, mid-gameplay, which is exactly what shows up as a stutter.
The old fix was manual and incomplete. UE4 relied on developers manually recording and bundling PSOs into the build ahead of time. It worked, but it was tedious for large projects and still left gaps — Epic's own words were that it "could be burdensome for large projects, and still leave gaps in the cache leading to hitches."
Epic is actively patching this, version by version. UE5.1 introduced Automated PSO Gathering to replace the manual process, and UE5.2 added the ability to skip drawing an object if its PSO isn't ready yet instead of stalling the frame. That's a tacit admission this was a real architectural problem, not a one-off bug.

So it's less "this didn't exist in UE4" and more "UE4 could mostly hide it behind DX11, and UE5's push into DX12/Vulkan plus Nanite/Lumen's sheer number of shader permutations made it impossible to hide anymore." Which also means: modern games are compiling and caching more shader variants than ever, feeding the exact kind of unmanaged, ever-growing cache this whole post is about.

The part nobody tells you

Once a shader is compiled and cached, nothing ever removes it — not when you finish playing the game, not when you uninstall it, not when a driver update makes an old cache entry irrelevant. DXCache just grows. Forever. One direction only.

And by default, the NVIDIA Control Panel setting that controls this — Manage 3D Settings → Shader Cache Size — is set to "Driver Default." The full dropdown looks like this:

Driver Default
Disabled
128 MB
256 MB
512 MB
1 GB
5 GB
10 GB
100 GB
Unlimited

Here's the part that surprised me: NVIDIA's own documentation on what "Driver Default" actually resolves to is close to nonexistent. The only official description I could find comes from the driver 495/496.13 release notes, which introduced this setting with a single sentence: "Added Shader Cache Size control to set the maximum amount of disk space to use for storing shader compiles." That's it. No page explaining what number "Driver Default" maps to, no doc on the eviction policy once a limit is hit, nothing on how it interacts with per-game overrides. It's telling that even developers on NVIDIA's own developer forum have publicly asked whether any detailed documentation of the caching policy exists at all — without getting a concrete answer. So "Driver Default" isn't a documented default in the "this is 2GB, here's why" sense — it's a black box that, in practice, behaves like there's effectively no ceiling, which tracks with mine sitting at nearly 100GB without me ever touching the setting.

So the math is simple: every game you've ever installed, played for an hour, and uninstalled left a tip behind in DXCache, and it's still there. I'd been PC gaming for years without ever touching that setting, which is exactly how a folder quietly gets to 100GB without a single warning, notification, or disk-cleanup prompt from Windows.

Why this is worse than it sounds

A few things make this specifically sneaky, rather than just "oh well, cache happens":

It survives uninstalls. Deleting a game does not touch its shader cache. The game is gone; the cache is forever.
Windows Disk Cleanup doesn't know about it. It's not temp files in the usual sense — it's driver-managed and lives outside the paths Windows normally offers to clean.
The default has no limit. Most caching systems assume someone eventually caps them. This one ships wide open by default.
It's invisible until you go looking. There's no indicator anywhere in the NVIDIA app or Windows that tells you this folder exists, let alone how large it's gotten.

It's not a bug in the "crash" sense. It's a design decision — unlimited-by-default plus no lifecycle management — that quietly behaves like a slow memory leak, except on your disk, and over years instead of hours.

The DX12 caveat, because I got this wrong at first

Worth being upfront about a wrinkle here: the Shader Cache Size setting in the NVIDIA Control Panel primarily governs the DX11 and older cache path. A lot of modern DX12 and Vulkan titles manage their own shader caching independently, sometimes in different folders, sometimes with their own size limits, sometimes with none. So DXCache specifically tends to fill up from DX11 titles, but it's not the whole picture — if you're chasing every byte, check your Vulkan cache and any per-game shader cache folders too (many are under %LOCALAPPDATA% as well, just organized differently per engine).

What I did about it

Checked the actual size first. Right-click → Properties on the folder, or in PowerShell:

   Get-ChildItem "C:\Users\<Username>\AppData\Local\NVIDIA\DXCache" -Recurse | Measure-Object -Property Length -Sum

Deleted the contents, not the folder itself — NVIDIA will rebuild it as needed. The cost of clearing it is a few seconds of shader recompilation stutter the next time you launch each game, which is a fair trade for 100GB.
Set an actual limit in NVIDIA Control Panel → Manage 3D Settings → Global Settings → Shader Cache Size. I put mine at 10GB. Plenty for active games, capped for everything I've since uninstalled.

Worth knowing: once the cache hits that limit, the driver doesn't just stop adding new entries — it starts evicting old ones to make room. So if you play a lot of heavy titles at once and set the cap too low, you can end up with cache getting wiped out and rebuilt constantly, which brings back the exact stutter this whole system exists to prevent. 10GB was comfortable for my library; if you've got a dozen big games installed at once, you may want more headroom.

Made a note to check it again in six months, because "set it once and forget it" is exactly the mindset that got me to 100GB in the first place.

The bigger pattern

This isn't really an NVIDIA-specific story once you zoom out. It's the same shape as a dozen other quiet disk hogs: Docker's dangling image layers, npm's global cache, browser service-worker caches, pip's wheel cache. Anything designed to cache the output of expensive work will keep that output around forever unless something explicitly ties its lifetime to the thing that created it — and "unlimited by default" is a decision, not an oversight, that someone made and shipped.

Caches without owners don't fail loudly. They just sit there, growing, until you go looking for missing disk space and find out where it's been hiding the whole time.

If you're on NVIDIA and haven't checked, it might be worth a look:

C:\Users\<Username>\AppData\Local\NVIDIA\DXCache

You might be surprised what's in there.

Three Questions I Ask Myself Before Adding a Dependency

Den — Thu, 03 Sep 2026 20:02:46 +0000

Every time I'm about to type npm install or pip install, I pause for a second. Not because I'm against libraries — I use them every day. It's because I once got burned on a project where package.json grew to 340 lines, and nobody could explain why half of those packages were there.

Since then, I ask myself three questions. Nothing philosophical — they take less than a minute, but they save hours of untangling version conflicts and weird bugs six months down the road.

Question 1: Can I really not write this myself in 20 lines?

This is the most common source of self-deception. A library looks convenient because it "solves the whole problem," but often I only need 5% of what it does.

The classic example is left-pad (yes, the one that broke half of the npm ecosystem in 2016). Padding a string on the left is a one-liner. It doesn't need to be a package with dependencies, tests, and versioning — it's literally:

const leftPad = (str, len, ch = ' ') => str.padStart(len, ch);

I'm not saying all utility packages are evil. But if a function fits in 15-20 lines and doesn't need to handle edge cases I'll never actually encounter, I write it myself. The upside: I know exactly how it works when it breaks at 2 a.m. a year from now.

The check is simple: I open the package source. If it's 200 lines of code and 15 dependencies for functionality I'm using 10% of — that's a red flag.

Question 2: Who maintains this, and what happens if that person disappears tomorrow?

This isn't a question about code quality today — it's about risk a year or two from now. I look at a few things:

When was the last commit. Not "how long since the last release" — a package can be stable and just not need updates. What I care about is whether the maintainer responds to issues and PRs.
How many maintainers. A single person maintaining something in their spare time isn't a dealbreaker, but it's a risk — especially if the package isn't a small utility but something a critical part of my project depends on (auth, money handling, data parsing).
Are there active forks. If the original project has gone quiet but a fork is thriving, the community may have already voted with its feet, and that's where I should be looking instead.

There's no universal threshold like "three months without a commit is bad." Sometimes a package just does one thing well and there's simply nothing to fix. The real question is: if it breaks, or a vulnerability is found, who fixes it, and how fast?

Question 3: What happens to my project if I remove this dependency in a year?

This question is about architectural coupling, not the library itself. I try to picture the scenario in advance: the library goes unmaintained, or I just want to swap it for something else. How deeply has it embedded itself in my code?

If the dependency is used in one module behind a clean interface, replacing it is a one-day job. If its calls are scattered across the whole project, and its specific abstractions (custom state types, an unusual API) have become part of my own design, I'm stuck with it forever — even if I stop liking it.

The practical takeaway: I try to wrap third-party libraries behind my own thin abstraction layer, especially for non-critical things like logging, an HTTP client, or date formatting. Not for the sake of "clean architecture" as an abstract virtue, but because it literally lowers the future cost of replacement from "rewrite half the project" to "rewrite one file."

What this actually looks like in practice

I'm not anti-dependency. I use React, FastAPI, pytest without a second thought, because the answer to all three questions is obvious: writing something comparable myself is unrealistic, maintenance is strong, and the coupling is so fundamental that replacement isn't even a real question.

But every time I'm about to add a package for something small — date formatting, a debounce function, deep object comparison — I spend a minute on these three questions. Sometimes the answer is "yes, install it." Sometimes I close the terminal and write 10 lines myself.

Fewer dependencies isn't the goal by itself. But every one I add is a future responsibility I'm taking on without thinking, unless I ask myself these questions first.

The Memory Limit That Didn't Kill Anything

Den — Wed, 02 Sep 2026 17:37:13 +0000

The process that runs my language model has a habit of growing. Not dramatically — it starts at a reasonable size and drifts upward over a shift, and the cause is somewhere in code I don't own. I spent a day trying to configure the leak away, failed, and did the sensible thing instead: stopped trying to fix it and put a ceiling on it, so the damage would be confined to one service instead of the whole box.

I gave it a soft ceiling. MemoryHigh in the unit file, a bit under what I'd measured the process to need.

That is how I turned a leak into an outage.

Everything was green

The symptom, when it arrived, was that nothing was wrong. The service was active (running). Its PID was current, its log had no errors, and top showed a process of unremarkable size. Every dashboard I had said the system was up, and it was up. It was just doing a batch of work — the kind that normally takes minutes — for most of a day.

The thing you have to understand about a soft limit is what the kernel does when you cross it. It doesn't kill the process. It doesn't return an allocation failure. It applies back pressure: it reclaims pages, pushes what it can to swap, and throttles the process's allocations until it fits back under the line. From inside the process, memory still works. Every allocation succeeds. Nothing raises. The program is simply slower, by a factor that has no upper bound, because its working set now lives on a disk.

A hard limit produces an event. A soft limit produces a symptom. MemoryMax kills the process, systemd restarts it, and you lose one unit of work loudly. MemoryHigh keeps it alive and makes it useless, and there is no line in any log that says so.

Where the truth was

Not in top. Resident size looked fine — that was the point, the kernel was keeping it fine.

The truth lives in the cgroup's own counters, which nobody looks at because nothing points you there:

cat /sys/fs/cgroup/system.slice/<service>/memory.events
cat /sys/fs/cgroup/system.slice/<service>/memory.swap.current

memory.events has a high counter — the number of times the process was throttled at the soft ceiling — and an oom_kill counter. Mine read a throttle count in the millions against zero kills. That pair is the entire diagnosis in two numbers: nothing died, everything was strangled. And memory.swap.current said that essentially all the swap on the machine belonged to this one service.

When a process is slow for no reason, check whether something is holding it under water on purpose. A limit you set yourself is the easiest cause to overlook, because you remember configuring it as a safety measure and safety measures aren't suspects.

The measurement that caused it

The ceiling wasn't arbitrary. I measured the process's working size and set the limit above it.

I measured it at the wrong moment. I took the number shortly after the model loaded, which is the calmest instant in that process's life: weights in memory, nothing computed yet, none of the caches and scratch buffers that only exist once real work is running. The number I recorded was a startup value wearing the label "working set". The real figure under load was meaningfully higher, and my ceiling landed underneath it.

So the process spent its life a few percent over a line it could never get back under, being throttled continuously, on every batch, forever.

A limit derived from a measurement inherits every flaw in how you measured. If the measurement was taken when the system was idle, the limit is a limit on idleness. Sample under load, or don't sample.

Slowness is not a local property

Here's the part that turned a slow service into a dead website.

The worker reads its queue from Postgres and then, in the same transaction, goes off to the model. That was already sloppy, and for a batch that takes minutes it was survivable sloppiness. With the batch now taking most of a day, the worker sat in idle in transaction for most of a day, holding a perfectly ordinary read lock on one table.

Then a deploy landed. Startup ran a migration, the migration wanted ALTER TABLE, and the ALTER queued behind the worker's read lock. Fine so far — a migration waiting is not an outage.

Except for the Postgres behaviour that everyone learns exactly once: a waiting AccessExclusiveLock blocks everything that queues up behind it. The migration wasn't just waiting, it was a wall. Every subsequent query against that table joined a line behind a lock that would not be granted until a language model finished a job it was never going to finish on time. The API never reached listen. Its last log line was Waiting for application startup, and systemctl reported it active (running), which was true and worthless. nginx returned 502 for everything except static images.

The whole cascade fits in one query, which is the one thing I'd want anyone to take from this:

SELECT pid, state, wait_event_type, left(query, 60) FROM pg_stat_activity;

An active | Lock | ALTER TABLE … sitting next to an idle in transaction | SELECT … is not a clue. It's the answer.

Both sides are fixed now, because both sides were wrong: DDL runs with a lock_timeout and logs who was holding, and the worker closes its transaction before it goes anywhere near the model. Never hold a database transaction across a call to something whose duration you don't control, and treat a lock without a timeout as a promise to hang the table rather than a promise to wait.

The alarm that had been ringing so long it was quiet

I found all of this by accident, which was its own lesson.

There was an hourly health check, and it had been finding problems the entire time. It said nothing, because alert state was stored as a single boolean for the whole system, and the rule was to notify on change. Chronically elevated swap had set that flag days earlier and it never came back down — so when new failures appeared underneath it, first the worker falling behind, then the API not answering, the state didn't change, and not one message was sent. Meanwhile the morning summary printed the numbers with no verdict attached and signed off with "all services running."

An alarm that stays silent because the system was already unhealthy is worse than no alarm, because it manufactures the feeling of being watched. State has to be tracked per check, not per system; unresolved problems have to keep repeating rather than being deduplicated into silence; and no report gets to print an all-clear until every check has actually passed.

What I'd take from it

I set the soft limit because a hard one felt violent — killing a process over a few megabytes seemed like an overreaction, and back pressure sounded like the gentler, more grown-up choice.

It's the opposite. The hard limit fails: the process dies, systemd brings it back, one batch is lost, and the log says exactly what happened. The soft limit degrades: nothing fails, nothing is logged, no counter you routinely look at moves, and the system quietly stops doing its job while continuing to report that it is doing its job. One of those is an event you can build on. The other is a fact your monitoring is structurally unable to notice.

Prefer the failure you can see to the degradation you can't. It's the same rule I keep arriving at from other directions — an absent value beats a wrong one — because the common thread isn't memory or validation. It's that a system which fails loudly is one you can operate, and a system which merely gets worse is one you find out about from your users.

The limit is gone now. In its place: a hard cap, and swap turned off entirely for that service. If it grows past what the machine can give it, it dies, and I hear about it.

Both of those are better than what I had, which was a process that was alive the entire time.