DEV Community: Ahmet Zeybek

Hybrid search with pgvector got worse until I added BM25

Ahmet Zeybek — Tue, 06 Oct 2026 07:19:34 +0000

Disclosure

I wrote PostgreSQL for AI, a book about building this kind of system in Postgres, and hybrid search is in it. Every number below I measured for this post, on public data, and the method is at the end.

SciFact, a public retrieval benchmark, has a test claim that reads "CCL19 is absent within dLNs." CCL19 is a chemokine and dLNs are draining lymph nodes, and one abstract in the corpus answers it. I embedded the claim with nomic-embed-text and asked pgvector for the 100 nearest abstracts. The right one wasn't among them. A BM25 index put it second, because CCL19 is a rare word and that abstract contains it.

Queries like that are why hybrid search exists. You run the vector search and the keyword search and merge the two lists with Reciprocal Rank Fusion, all in one SQL statement. pgvector has no keyword ranking of its own, so the keyword half has to come from somewhere else in Postgres, and that choice decided most of the result.

Most tutorials rank the keyword half with Postgres' built-in ts_rank. On both datasets I tried, fusing that with vector search scored worse than vector search alone. Put a BM25 index in its place and the same fusion scored best of everything.

How the two halves rank

Vector search compares meanings. The question and each document become points in one space, and you get back the nearest ones. That forgives typos and paraphrases, and it has no special idea what a gene name is. To the embedding model a rare identifier is just a few tokens, and an abstract full of related biology can land closer than the one that actually names it.

Keyword search compares words. BM25, the ranking function search engines have used for about thirty years, scores a document higher when it contains the query's words, then adjusts for three things. A rare word counts more than a common one (that is inverse document frequency), and the tenth occurrence of a word adds less than the first, so repetition stops paying. A long document is pulled down a little, since it contains more words by chance.

Postgres' own full-text search is good at matching and bad at ranking. ts_rank and ts_rank_cd only look at how often and how close together the words appear inside one document. Nothing is measured across the corpus, so "the" and "CCL19" weigh the same once stop words are removed. Length only counts if you pass a normalization flag, and that flag divides by this document's length without knowing what's typical for the rest.¹

You can't merge the lists by adding scores, because a cosine distance and a BM25 score are on unrelated scales, and the scale shifts with every query. Reciprocal Rank Fusion only uses positions. A document gets 1 / (k + rank) from each list it appears in, and the sums give the order. The formula comes from a 2009 SIGIR paper by Cormack, Clarke and Büttcher, where k = 60 was fixed in a pilot run and never changed.²

The usual recipe joins the words with AND

The version in most guides has a generated tsvector column with a GIN index next to the embedding, one CTE per half, and a FULL OUTER JOIN that adds up the reciprocal ranks. The keyword half parses the question with websearch_to_tsquery or plainto_tsquery and ranks with ts_rank or ts_rank_cd.³

Both parsers join the words with AND. In a search box, where people type a few words, that's what they expect. With a natural language question, though, a document has to contain every word left after stop words before it matches at all. SciFact's test claims average 12.5 words, and with AND the keyword half returned nothing for 274 of the 300. NFCorpus questions average 3.3 words, and 131 of its 323 still came back empty.

So I measured the recipe twice, once as written and once with the words joined by OR. It's a one line change:

-- OR the lexemes plainto_tsquery produces, so a document needs any of them
SELECT replace(plainto_tsquery('english', $1)::text, ' & ', ' | ')::tsquery;

Fusion with ts_rank scored below vector search alone

The data is two public BEIR datasets with relevance labels: SciFact, with 5,183 scientific abstracts and 300 test claims, and NFCorpus, with 3,633 medical documents and 323 test questions.⁴ nDCG@10 scores how well the top ten is ordered against the labels, where 1 is perfect, and recall@100 is the share of relevant documents found anywhere in the top 100.

Method	SciFact nDCG@10	SciFact R@100	NFCorpus nDCG@10	NFCorpus R@100
`ts_rank_cd`, AND (`websearch_to_tsquery`)	0.072	0.073	0.206	0.102
`ts_rank_cd`, OR	0.330	0.759	0.235	0.225
BM25, pg_textsearch	0.688	0.918	0.325	0.246
BM25, pg_search	0.685	0.921	0.323	0.247
Vector, pgvector HNSW	0.703	0.928	0.347	0.297
RRF, `ts_rank_cd` (OR) + vector	0.594	0.942	0.324	0.298
RRF, pg_textsearch + vector	0.727	0.958	0.356	0.307
RRF, pg_search + vector	0.735	0.958	0.358	0.307

Fusing ts_rank_cd with vector search scored 0.594 on SciFact, against 0.703 for the vector half alone, and 0.324 against 0.347 on NFCorpus. Recall went up on both, so the fused list does find more relevant documents somewhere in its top 100. But it orders the top ten worse, and the top ten is what a RAG pipeline puts in the prompt.

On its own (the second row), ts_rank_cd scores 0.330 on SciFact, less than half of BM25's 0.688 on the same matches, so its order is a weak signal next to the vector list. RRF weighs both lists equally, and the weak one drags good vector hits down.

With BM25 as the keyword half, fusion helps on both. It scored 0.727 and 0.735 on SciFact, 3 to 5 percent above vector search, and 0.356 and 0.358 on NFCorpus, about 3 percent above. The two BM25 extensions came within a few thousandths of each other, as two implementations of one formula should.⁵

Here's what each method cost per query on SciFact, at the median and the 95th percentile:

Method	p50	p95	Index size	Build
BM25, pg_textsearch	0.21 ms	0.39 ms	3.7 MiB	0.9 s
BM25, pg_search	0.68 ms	1.56 ms	5.3 MiB	0.5 s
Vector, HNSW	1.51 ms	3.11 ms	20.3 MiB	2.3 s
RRF, pg_textsearch + vector	1.63 ms	2.15 ms
RRF, pg_search + vector	2.30 ms	3.67 ms
RRF, `ts_rank_cd` (OR) + vector	19.88 ms	57.29 ms	4.4 MiB (GIN)	0.2 s

These are server execution times on a warm cache, leaving out the network and the embedding call. The fused BM25 query costs about the same as vector search alone, since its keyword half takes a fraction of a millisecond. The ts_rank version is the slow one. With OR most documents match at least one word, and ts_rank_cd has to score every match before sorting, because the GIN index can find matches but can't rank them.

The query with a BM25 index

The table holds the text and the embedding, plus a generated tsvector for Postgres' own matching in filters. Each BM25 extension adds its own index type on top.

CREATE TABLE docs (
  id        text PRIMARY KEY,
  content   text NOT NULL,
  tsv       tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
  embedding vector(768) NOT NULL
);

CREATE INDEX docs_hnsw ON docs USING hnsw (embedding vector_cosine_ops);

pg_textsearch

-- needs shared_preload_libraries = 'pg_textsearch'
CREATE EXTENSION pg_textsearch;
CREATE INDEX docs_bm25 ON docs USING bm25 (content)
  WITH (text_config = 'english');

pg_search

-- needs shared_preload_libraries = 'pg_search'
-- CASCADE also installs pgvector, which pg_search 0.25 depends on
CREATE EXTENSION pg_search CASCADE;
CREATE INDEX docs_bm25 ON docs USING bm25
  (id, (content::pdb.simple('stemmer=english')))
  WITH (key_field = 'id');

The English stemmer on the pg_search index keeps the comparison fair. Without it "runs" won't match "running", and both text_config = 'english' and to_tsvector('english', ...) do stem.

The fused query takes the question's text as $1 and its embedding as $2, both computed once in the application.

pg_textsearch

SELECT coalesce(v.id, k.id) AS id,
       coalesce(1.0 / (60 + v.r), 0) + coalesce(1.0 / (60 + k.r), 0) AS score
FROM (SELECT id, row_number() OVER (ORDER BY d) AS r
      FROM (SELECT id, embedding <=> $2::vector AS d FROM docs
            ORDER BY embedding <=> $2::vector LIMIT 100) s) v
FULL JOIN
     (SELECT id, row_number() OVER (ORDER BY d) AS r
      FROM (SELECT id, bm25_get_current_score() AS d FROM docs
            ORDER BY content <@> to_bm25query($1, 'docs_bm25') LIMIT 100) s) k
  ON v.id = k.id
ORDER BY score DESC, id
LIMIT 10;

pg_search

SELECT coalesce(v.id, k.id) AS id,
       coalesce(1.0 / (60 + v.r), 0) + coalesce(1.0 / (60 + k.r), 0) AS score
FROM (SELECT id, row_number() OVER (ORDER BY d) AS r
      FROM (SELECT id, embedding <=> $2::vector AS d FROM docs
            ORDER BY embedding <=> $2::vector LIMIT 100) s) v
FULL JOIN
     (SELECT id, row_number() OVER (ORDER BY d DESC) AS r
      FROM (SELECT id, pdb.score(id) AS d FROM docs
            WHERE content ||| $1 ORDER BY pdb.score(id) DESC LIMIT 100) s) k
  ON v.id = k.id
ORDER BY score DESC, id
LIMIT 10;

Each half sorts and limits inside its own subquery, so it hands over its best 100, and the row_number() window has its own ORDER BY because SQL promises nothing about the order rows arrive in. The outer SELECT takes the id through coalesce, so a document only one half found still has one, and the final ORDER BY breaks ties on id so every run returns the same list.⁶ The halves are subqueries in FROM and not CTEs, and the next section explains why.

Three things made the same query 15 times slower

My first version of the fused BM25 query took 25.7 ms at the median. The one above takes 1.6 ms and returns the same rows. Three separate problems were in the way, and none of them threw an error.

First, the planner skipped HNSW. On a 5,000 row table Postgres reckoned that reading every vector and sorting was cheaper than walking the graph, and went with that. It took 16.6 ms at the median, against 1.5 ms for the forced HNSW scan. The estimate depends on table size, so a bigger table may well get the index without help. On a small one, in a test or a new product, check EXPLAIN before you trust a latency number, because here it picked a plan eleven times slower.

Second, pg_textsearch scored every row twice. Its README puts the score in the select list as content <@> to_bm25query(...). In 1.5.1 that ran the BM25 index scan and then scored each of the 100 returned rows again from its text, which took 15.5 ms against 0.5 ms for the same query returning only ids. The extension has a planner hook that's supposed to swap that expression for bm25_get_current_score(), a function that reads the score the index scan already computed. In my runs the swap never happened. Calling the function myself got the query down to 0.3 ms. It isn't in the README, so I'd check it still exists after an upgrade.

Third, the CTE was materialized. Postgres folds a CTE into the main query when it's used once and calls no volatile function. bm25_get_current_score() is declared VOLATILE, so a WITH k AS (...) around it stayed a separate step, the planner hook couldn't reach inside, and the 15 ms came back. With the same half written as a subquery in FROM, the whole fused query went from 20.8 ms to 2.2 ms on the first test claim.

All three were visible in EXPLAIN (ANALYZE, BUFFERS) on the fused query. Read that plan before you trust any latency number for hybrid search.

Which BM25 you get depends on where Postgres runs

When I wrote the book's chapter on what's coming next, in February, two teams were building BM25 for Postgres. By October there are at least five options, and where your database runs makes most of the choice for you.

Extension	From	License	Notes
pg_search	ParadeDB	AGPL-3.0	Built on Tantivy. 0.25.11 on 29 Sep 2026. Neon dropped it for new projects on 19 March 2026 and from existing ones on 21 September.
pg_textsearch	Tiger Data	PostgreSQL	1.0 in March 2026, 1.5.1 on 2 Oct 2026. On Tiger Cloud. PG 17 and 18.
VectorChord-bm25	TensorChord	AGPL-3.0 or ELv2	Needs the separate pg_tokenizer extension.
lakebase_text	Neon		Neon's replacement for pg_search on its own platform.
Tin	PlanetScale	not stated	Announced 16 Sep 2026 for PlanetScale Postgres.

pg_search and pg_textsearch both call their access method bm25, so whichever goes into a database second fails with "access method bm25 already exists". They can share a server if each gets its own database, which is how I measured them. Both also need an entry in shared_preload_libraries, so on a managed service you get whichever extension your provider ships, or none, and its extension list is the first thing to look at.⁷

Tuning k moved the score by a hundredth

In RRF, k sets how quickly credit falls off down a list. A small k gives most of the credit to the first few ranks. A large one flattens the curve, so agreement between the two lists counts for more than position. For the pg_textsearch fusion, nDCG@10 by k came out like this:

k	1	10	30	60	100	300
SciFact	0.737	0.738	0.730	0.727	0.727	0.726
NFCorpus	0.356	0.360	0.359	0.356	0.357	0.355

k = 10 beat 60 on both, by 0.011 on SciFact and 0.004 on NFCorpus, small enough that sticking with the default costs little. The choice of keyword half moved the result far more.⁸

Bruch, Gai and Ingber argue that a weighted sum of normalized scores beats RRF once the weight is tuned on a few labeled queries, so I tried it. I min-max normalized each list, tuned the weight on SciFact's 809 training claims and NFCorpus' dev questions, then scored the test sets. It reached 0.737 and 0.360, the same as RRF at k = 10. With ts_rank_cd it did lift the fusion to 0.709 on SciFact, by putting 0.8 of the weight on the vector list. If you have labeled queries, tuning either one gets you to the same place. If you don't, RRF with a BM25 half needs nothing tuned.

Fusion doesn't win on every query either. Against whichever half happened to be better for a given query, which you can't know in advance, the pg_textsearch fusion came out ahead on 25 SciFact claims and behind on 71. On NFCorpus it was ahead on 58 and behind on 125, and below both halves on 14. The fair comparison is against vector search alone, and there it was ahead on 67 SciFact claims and behind on 38, and ahead on 112 NFCorpus questions and behind on 83. That's where the average gain comes from. On SciFact, 11 claims had their abstract missing from the vector half's 100 and found by fusion, at ranks from 7th to 65th. CCL19 was the 7th.

Take one of the losses, "DMRT1 is a sex-determining gene that is epigenetically regulated by the MHM region". pg_textsearch puts the right abstract second, the vector half doesn't have it in its 100, and after fusion it sits 36th. It goes the other way too. One claim misspells metastases as "matasteses", and the vector half puts the right abstract first, but BM25 can't match the misspelled word at all, so fusion drops the abstract to 23rd. A document found by one half gets at most half the credit of one both halves found somewhere, and RRF has no way to tell which half was right for this query.

Where it stops

Both datasets are small, 5,183 and 3,633 documents. I ran everything on a laptop with Docker limited to 1 GB of memory, too little for the 57,000-document FiQA set I'd planned. The quality comparison holds at this size. The latency numbers are for tables that fit in memory, and with millions of rows the time probably goes somewhere else, starting with HNSW cache misses.

I used one embedding model, nomic-embed-text, because it runs locally. A stronger model raises the vector baseline and leaves BM25 less to add, so run the same evaluation with your own model before deciding a second index is worth it.

Both corpora are English, and so is the stemming. For Turkish, Postgres ships a turkish text search configuration and the BM25 extensions take whatever tokenizer you give them. I haven't measured how either copes with Turkish suffixes.

Hybrid search also inherits a filtering problem. If the query also has WHERE tenant_id = $3, an HNSW scan can return fewer than 100 rows after the filter. That's pgvector's most discussed issue, and it needs a post of its own.

How I measured, so you can argue with it

One container ran ParadeDB's paradedb/paradedb:0.25.11-pg18 image (PostgreSQL 18.6, pgvector 0.8.4, pg_search 0.25.11), with pg_textsearch 1.5.1 installed from Tiger's release package. pg_textsearch and pg_search lived in two databases on that server because of the bm25 name clash. The machine is an Apple M2, with 8 CPUs and 1 GB of memory given to Docker. The settings were shared_buffers = 256MB, maintenance_work_mem = 256MB, jit = off and hnsw.ef_search = 200, with HNSW at pgvector's defaults. Every vector number in the tables uses the HNSW index, forced with enable_seqscan = off for those queries, because on tables this small the planner preferred an exact scan.⁹

The data is BEIR's SciFact and NFCorpus as published. Each document is its title and text joined by a newline, embedded once through Ollama 0.34.4 with nomic-embed-text and the search_document: prefix. Questions used search_query:, and I raised num_ctx to 8192 so nothing was truncated. Every method returned its top 100 from one SQL statement per question.

Scores come from pytrec_eval, as nDCG@10 and recall@100 on each dataset's test split, with a query that returned nothing counted as zero. The SQL fusion matched the same RRF computed in Python on 599 of the 600 test questions, and the one difference is a tie broken the other way. Latency is the Execution Time from EXPLAIN (ANALYZE, TIMING OFF), the median of three runs per question after one run to warm the cache, with percentiles taken over the test questions. I measured all of it on 2 and 3 October 2026.

evaluate.py

import pytrec_eval

def score(runs, qrels):
    """runs: {query: [(doc, score), ...]} best first; qrels: {query: {doc: grade}}"""
    as_run = {q: {d: float(len(r) - i) for i, (d, _) in enumerate(r)} for q, r in runs.items()}
    ev = pytrec_eval.RelevanceEvaluator(qrels, {"ndcg_cut.10", "recall.100"})
    res = ev.evaluate({q: r for q, r in as_run.items() if q in qrels})
    n = len(qrels)
    ndcg = sum(res.get(q, {}).get("ndcg_cut_10", 0.0) for q in qrels) / n
    recall = sum(res.get(q, {}).get("recall_100", 0.0) for q in qrels) / n
    return ndcg, recall

def rrf(a, b, k=60):
    out = {}
    for q in set(a) | set(b):
        s = {}
        for lst in (a.get(q, []), b.get(q, [])):
            for i, (d, _) in enumerate(lst):
                s[d] = s.get(d, 0.0) + 1.0 / (k + i + 1)
        out[q] = sorted(s.items(), key=lambda x: (-x[1], x[0]))[:100]
    return out

embed.py

import requests

def embed(texts, prefix):
    r = requests.post("http://localhost:11434/api/embed", json={
        "model": "nomic-embed-text",
        "input": [prefix + t for t in texts],
        "options": {"num_ctx": 8192},
    })
    r.raise_for_status()
    return r.json()["embeddings"]

# documents: embed([title + "\n" + text, ...], "search_document: ")
# questions: embed([text, ...], "search_query: ")

docker

FROM paradedb/paradedb:0.25.11-pg18
USER root
COPY pg-textsearch-postgresql-18_1.5.1-1_arm64.deb /tmp/pgts.deb
RUN dpkg -i /tmp/pgts.deb && rm /tmp/pgts.deb
USER postgres
# run with: -c shared_preload_libraries=pg_search,pg_cron,pg_textsearch

The book has the longer version of this, search and RAG on one Postgres, and there's a free sample chapter if you want to see how it reads first.

Originally published at zeybek.dev.

Tiger's write-up lists the same three gaps. ↩
Tiger's hybrid search tutorial and the dev.to example below both still use 60. ↩
This dev.to post is a tidy example, and this one does the fusion in Python. ↩
Everything ran in one PostgreSQL 18.6 instance with pgvector 0.8.4, ParadeDB's pg_search 0.25.11 and Tiger's pg_textsearch 1.5.1. ↩
BM25 alone also lands where the literature puts it, at 0.688 and 0.325 here, against 0.665 to 0.686 for SciFact and 0.325 for NFCorpus in the published BEIR reference runs. ↩
In pg_search, ||| matches any of the words, the OR behavior websearch_to_tsquery didn't give. ↩
Chapter 12 of the book keeps the same kind of table for pgvector, pgvectorscale, pgai and the other AI extensions, provider by provider. ↩
With ts_rank_cd as that half, the best k on SciFact was 1, which is RRF all but ignoring the second list. ↩
The exact scan scored the same 0.703 on SciFact and 0.347 on NFCorpus. ↩

TypeSafe's Jev in production: thresholds and two bugs

Ahmet Zeybek — Tue, 06 Oct 2026 07:19:25 +0000

Until recently, if you said hello to the assistant on this site, it told you about setting temperature to zero for classification. You didn't get a greeting back, you got a tip about temperature, every time, for "hello" and "thanks" and "good morning".

The reason is boring, and worth knowing. The assistant answers from a vector index. A greeting gets embedded like anything else, the index returns whichever document sits closest to it, and for a short friendly sentence with no subject that's the same document every time. Then a model gets paid to write a sentence about it. I measured the scores on the eval's questions in September. Small talk's best match landed between 0.51 and 0.56, and a question the site really answers landed at 0.61 and up. So I drew a line at 0.6, and the tip stopped showing up under greetings.

That line is one number, fitted to a gap of five hundredths, standing in for the question "is this page about what was asked". It held. It was also the most obviously wrong thing in the codebase, because the question being asked is about meaning and the answer was a distance.

This post is about replacing numbers like that with Jev, a decision model from TypeSafe that reads the sentence instead of measuring the distance to it. It picks, rates and judges, and writes nothing. There are ten such call sites on this site now, from the assistant's front gate to note moderation in a public room. The model turned out to be easy. Most of the work was deciding what to do when it wasn't sure.

What Jev is, quickly

Jev is the first of what TypeSafe call System One models, after Kahneman's fast thinking. It takes some state and some questions and returns typed answers with probabilities. It doesn't write, and their docs say so plainly: jev-1.13 is not trained to generate text.

That sentence explains most of it. A generative model writes you a paragraph and you pull a decision out of it. Jev skips the paragraph. You give it the text and the question, and you get back a number, or one of your own option names, or a position on a ladder you described. There's no JSON to repair, no run where it answers in prose instead, and no schema to validate, because the answer space is the question you asked.

The price is $42 per billion input tokens, and output tokens are free, which makes sense when there's almost no output.¹

That's the part you can read off a spec sheet, and it's the least interesting part. My time went on the rest: what to do with an answer you're only 60 percent sure of, who pays for the question, and what breaks when the answer never comes.

The three questions, as we ask them

A noul asks whether something is true and gives you a number from 0 to 1. A choice picks one of your named options and returns the whole probability distribution plus a confidence. A score puts something on a ladder you describe, and can land between two rungs.

They're evaluated in parallel against the same state, and none of them sees another's answer. I keep using one consequence of that: another question costs its own tokens and almost no time, so a question whose answer only matters sometimes is nearly free. Ask it in the same request instead of a second one the code has to wait for.

This is the gate in front of the site's assistant, a single choice:

const KIND = choice("What is `message`?", {
  question: "A question about this site, its posts, its tips or its author",
  small_talk:
    "A greeting, a thank you, a goodbye, or a remark that asks for nothing",
  search: "A request to find or list something, or just a subject to look up",
  action:
    "An instruction to the site itself: change the theme, open a page, play the radio",
  unclear: "None of these, or too little to tell",
});

The unclear option matters. The keys you write are the whole answer space, so if your list doesn't cover an input, the model still has to pick something, and it will. Always give it a way out.

Note the backticks around message. The question refers to a field in the state you send, and you can reach inside it: pages[2].title works, which is how I ask about several things at once without repeating their text in every question.

A probability is not a confidence

This one cost me an afternoon.

A choice gives you confidence, which says how peaked the distribution is, and flat means the model doesn't know. A noul gives you no confidence at all, because the number itself is one. 0.5 is truly undecided, and the two ends are certain of opposite things.

So a noul threshold has two sides, and they're two different decisions. When I ask "did the visitor ask to be taken somewhere", a reading of 0.9 is a yes, 0.1 is a no, and 0.5 means I learned nothing and should fall back to whatever I used before. Writing if (probability > 0.5) throws away that middle and turns every shrug into a yes:

if (probability >= SURE_ENOUGH) return true;
if (probability <= 1 - SURE_ENOUGH) return false;
return null;

SURE_ENOUGH is 0.85 there, and null goes back to the two regexes that decided it before.²

The wording is the specification

The criteria you write are all the model knows about your problem. Someone reading only the criteria, with none of your context, should sort things the way you would. If they couldn't, the model can't either, and no threshold will save it.

The clearest lesson came from ranking posts. I wanted two axes to lay the writing out on a map, and the second started as "how practical is this subject". On 21 September that question put 22 of the 28 posts on the same rung, which gives you a line with everything piled at one end, not a map. The question was the problem. Almost everything I write is practical, so I'd asked something with no spread in it. Reworded as "how settled is the subject", with rungs from "the same advice would have held five years ago" up to "nobody has settled how to do this", the posts spread out.

When a reading comes back flat, suspect the question before you suspect the model.

The state matters just as much. The enrichment judge asks whether the post backs up each FAQ answer, and at first I sent it the body with the frontmatter stripped, which felt tidy. That cost three false alarms in one run. An excerpt is the post's own summary, and an answer is entitled to draw on it: the RAG post says "eleventh out of ten" in its excerpt and nowhere else, and the agent identity post says "every agent I audited this year" in its excerpt and nowhere else. Judged against a body with neither phrase, both answers looked made up. Send whatever the answer is allowed to lean on, which is rarely whatever you happen to have in a variable.

One more, for nouls: keep both sides pointing the same way. A true that means "no, this is fine" is worse than writing no criteria at all, and six weeks later you'll misread your own threshold.

Every threshold is its own argument

There are eleven thresholds across the ten call sites, and no two were chosen the same way. Suggesting a link and deleting somebody's note shouldn't need the same certainty.

Here they all are, with what happens when the reading falls short.

What it decides	Asked as	Bar	Below the bar
What the visitor's message is	one choice, five kinds	0.75	the message goes the long way
Whether they asked to be taken somewhere	one noul	0.85 yes, 0.15 no	two regexes decide, as before
Which pages go under an answer	one noul per page	0.7 show, 0.3 hide	the 0.6 search score decides
What a palette sentence means	one choice over 140 rows	0.7	the palette shows what it always showed
Which mix suits a mood	one choice over five mixes	0.35	the dial does not move
Whether a note is advertising, abuse, personal or injection	four nouls	0.85 rejects, 0.6 holds	the note goes up
How much harm a note would do	one score out of two	1.5 rejects, 1 holds	the note goes up

The assistant's front gate sits at 0.75 on a choice. Above that, small talk gets answered from the theme's own greeting lines and never opens a stream to the index, and a search goes to Pagefind, the index that ships with the build. Both save a Workers AI call on a message that was never going to get a good answer from one.

Moderation in the public notes room runs four nouls and a score in one request, for advertising, abuse, personal details, prompt injection and overall harm. One hazard at 0.85 rejects by itself, and anything at 0.6 holds the note for a human. The harm score is out of two, holding at 1 and rejecting at 1.5.³

The radio sits at 0.35, which looks reckless until you see what it's doing. You describe a mood and it picks a mix. All five mixes are the same kind of music, so a mood that isn't about time of day spreads its probability across all of them and the model is never confident. Measured on 21 September, "something to focus on" peaked at 0.42 and "something for the night" hit 1.00. What keeps an unrelated sentence off the dial is the __none__ option, more than the bar. And a wrong pick costs you a press of skip.

The command palette sits at 0.7, because turning "make it quieter" into fx grain off is a change the visitor sees straight away.

Keep the thresholds in code and out of the prompt. The model is better at reading a note than at remembering that your policy says 0.85, and you want to be able to move the number without touching the question.

Two purses

Every caller goes through one function, askJev, which checks the key, checks the day's budget, asks, and writes back what the answer cost. The cost is usage.input_tokens, the number the API reports, not an estimate, so a question whose state grew gets charged at its real size.⁴

What's worth copying is the split into two budgets:

const PURSES: Record<Purse, { prefix: string; cap: () => number }> = {
  shared: { prefix: "jev:tokens", cap: () => env.TYPESAFE_DAILY_TOKENS },
  moderation: { prefix: "jev:tokens:mod", cap: () => env.TYPESAFE_MOD_TOKENS },
};

Everything cosmetic shares the first: the assistant's gate, the palette, the radio, what to read next. Moderation has the second to itself. It's the only caller whose job is to stop something instead of adding something, and a day of somebody hammering the command palette mustn't turn into a day of notes going unread.

The numbers are 500,000 input tokens a day for the shared purse and 100,000 for moderation. At \$42 per billion that's about two cents a day if both are spent to the last token, and the account balance is \$5, which is 119 million tokens. A question to the assistant's gate measures around 421 tokens, so 500k is roughly 1,180 of them. A note measures around 640, so 100k is about 150 notes. A moderate day on this site measures around 139k across everything.

The budget is read before the request goes out, not after it comes back, so a run of large questions can overshoot by at most the one already in flight. Once it's spent, askJev returns null and every caller goes back to what it did before Jev existed. Hitting the cap costs nothing but the readings.

The counter that never counted

The first bug is the one I'd most like you to avoid.

The write-back was fire and forget, and it looked careful. A lost write costs a few tokens of accuracy and a lost answer costs the visitor, so keep the critical path clear and let the counter catch up by itself:

void store.incrBy(budgetKey(), result.usage.input_tokens, BUDGET_TTL);

But a Cloudflare Worker stops executing the moment it returns a response, so the promise was dropped. I measured it against production on 21 September: six calls moved the counter once, from nothing to 426, and never again.

▶ Six calls, and a counter that moved once: an animation that plays in the original post.

So the read-before-ask check was comparing spend against a number that basically never grew, and the budget was only for show.⁵ The fix is await instead of void, one extra D1 write next to a request that's already spent half a second asking. A call that isn't counted isn't capped, and the counter is the only reason the cap exists.

If you take one thing from this post, make it this: whatever you use to bound spend, prove it moves. Spend two minutes calling your own endpoint and watching the number.

Spending somebody else's budget

The second bug is a kind of attack surface I hadn't thought about before.

Three routes shipped guarded by one check: does the request's sec-fetch-site header say it came from this site? A browser sets that header honestly. A script sets it to whatever it likes, or leaves it out. So the guard turned away a real browser on another site and let a script straight through.

On a route that returns data, that's a leak. On a route that spends a shared token budget, it's a way to switch off every reading on the site for the rest of the day, for the price of a loop and about 1,200 requests a minute. The fix was to put the Turnstile check the assistant already had on them too, and the clients now fetch a token before they ask.

More generally, once a model sits behind an endpoint, every call to it costs money, and the usual "this only returns public data so it can be open" reasoning stops applying. Rate limit by IP, cap the total as well, and make sure the total is real.

The model reads English

Nothing I read before shipping prepared me for this, so I'll say it plainly. Jev is trained on English. The content on this site is English, so most of it is fine, but the assistant takes questions in whatever language the visitor types, and mine get a lot of Turkish.

You can see it in the readings. "bloga ucur bizi kaptan" is Turkish, roughly "fly us to the blog, captain", and it asks to be taken somewhere. The model read it at 0.95 and got it right. But the instinct to raise the bar for a language the model reads less well also hollows out the feature, because an unsure reading under a high bar is a no, and a no means nothing happens.

Where I ended up is making the middle mean something. Instead of raising the bar and accepting fewer yeses, raise it on both ends and let the middle fall back to whatever decided before. In Turkish that means the old regexes keep running, badly, exactly as badly as they did last month, while English gets better and nothing anywhere gets worse.

Reading something and not acting on it

The front gate sorts messages into five kinds and acts on two. The action class gets read and recorded, and then nothing is done with it.

It put 0.77 on the message "I want to read posts", which is someone telling you what they like, and the separate question that decides whether to actually move somebody off the page they're reading rated the same message 0.68. That's two readings of the same sentence that disagree, both above a half. Acting on the first would take a reader who said something mild and throw them onto another page.

So it goes to analytics as gate_kind and gate_confidence, and nothing else happens. That's a legitimate state for a feature to be in, and there should be more of it. You can put a model in the path, keep its answer, and change nothing until the numbers tell you the bar is in the right place.

The same goes for probability distributions: store the whole thing. The pick on its own says nothing about how close it was. When you want to move a threshold in three months, the distribution is the only data you'll have, and by then you won't remember what "0.42" felt like.

It is never the only answer

One rule survived everything: nowhere on this site is Jev the only authority.

The moderation route lets the note through on every failure. No key, empty note, request timed out, budget spent: the note goes up. A public room that stops taking writes because a classifier is down is worse than one that shows a rude line until somebody removes it.⁶ The last word still belongs to the rate limit and the ownership check, and those are code.

The last piece I built shows it most clearly. Remember MIN_LINK_SCORE = 0.6, the number from the opening? It's still there. What changed is that a reading can overrule it, but only when it's sure:

const verdict =
  probability >= WORTH_SHOWING ? true
  : probability <= WORTH_HIDING ? false
  : null;

▶ A reading that only decides where it is sure: an animation that plays in the original post.

The bars are 0.7 and 0.3. A clear yes rescues a page the distance would have dropped, at 0.56, inside that five-hundredth gap I fitted the number to. A clear no drops the tip that shows up under "hello". Anything in between hands the decision back to 0.6. The model can't make the links worse than they were, because where it has nothing to say, whatever decided before still decides.

I value that more than accuracy. It meant I could ship it without an eval proving it beats the old number, and it means the feature falls back to last month's behaviour instead of to nothing.

What it costs

Four things run at build time on this site, once per change. Ranking all 30 posts against each other for related links costs 48,000 tokens. Placing them on a two-axis map costs 12,600. Judging the enrichment text (the FAQ answers, cover alt text and social descriptions) against what each post actually says costs 111,000 across every post. A full eval run of the assistant, 18 cases, costs 22,500. All four from scratch come to 194,000 tokens, which is 0.8 cents.

On the request path each call is cheaper than that, and the ceiling is the daily cap, not the balance. The cap is there because the failure I care about is somebody finding an endpoint and making the readings stop for everybody else, much more than the cost.

Latency is what you have to plan around, and it's not the 70 to 500 milliseconds in the marketing. That figure is the model. Yours is the model plus your worker's network hop plus whatever else the request is doing. The front gate gets 1.5 seconds, because a visitor is waiting with nothing on screen. The question about moving pages gets 5 seconds, because the reply is already written and another second is free. Links get 2.5, and build scripts get 30.

That gap matters. On 21 September a reading came back at 0.95 in 782 milliseconds from a laptop, and the worker still didn't answer within the two-second budget it had then. It fell back to the patterns and dropped the page change the visitor had asked for. The model was fast enough, and the budget I'd given it wasn't.

Where it stops

It can't count, do arithmetic or compare dates, and their own docs say so. Reading time, prices, ordering and anything involving money stay in code, as they always did.

It isn't a parser. Terminal command parsing, URL matching, file paths: all of that is regex and should stay regex. The places worth replacing are the ones where the code was guessing at meaning and passing a number off as an opinion.

The accuracy claims are the vendor's. I haven't compared it with a small classifier on the same data, and the one independent benchmark I found was somebody's 40 hand-labelled tickets. It's early access, not GA, and there's no free tier, so your balance runs down while you experiment.

And it doesn't remove work, it moves it somewhere else. Instead of tuning one cosine threshold, I now maintain eleven thresholds, two budgets, a fallback for every call site, and a set of questions worded so a model reading only the criteria would answer the way I would. That's more to look after than I had before. It's better to look after, because each piece is about one decision and says what it's for, but anyone selling you the idea that this deletes code is selling you something.

How I measured, so you can argue with it

Token counts are usage.input_tokens as the API reports them, read from the same field the budget is charged against. The per-call sizes come from a smoke script that makes one real request and prints the count.

The 0.51 to 0.56 and 0.61 figures are retrieval scores over the eval's 18 questions in September, from the index as it was that day. The confidence readings quoted for the radio and the front gate are single measurements against production on 21 September, not averages, and I've said so each time instead of dressing them up.

I found the counter bug by calling production six times and reading the D1 row, and that was the whole method. The $42 per billion and the free output tokens are TypeSafe's published prices, and the 119 million figure is the account balance divided by that.

Every threshold in this post is a constant you can read in the repository, and where a number came from judgement instead of measurement, I've tried to say so.

Originally published at zeybek.dev.

Context is 64k per request, 32k of it for the state. ↩
I'll come back to why that matters more than the threshold. ↩
I got those numbers by reading what the model said about notes I'd already judged myself. ↩
Output tokens are free, so input is the whole bill. ↩
Spend was still bounded, because every route has its own per-IP limit, but the one number meant to cap the lot did nothing. ↩
I've argued this before about anything with a model in it, and a decision model doesn't change the argument. ↩

How I turn any handful of colours into a readable theme

Ahmet Zeybek — Tue, 06 Oct 2026 07:19:20 +0000

Open the theme picker, choose Flatland from the base16 list, and the cards on the page lose their edges. Text and links are fine, but every border and raised surface has melted into the background. Flatland's base01 and base02 are both 1c1d19, and its background is 1c1e20. The contrast between them is 1.01 to 1, about what you'd get from two colours nobody can tell apart.

Flatland was just the one I noticed. This site has 18 hand-made themes and 368 schemes from the tinted-theming catalogue. Once I wrote down what a theme has to do and measured all of them, 16 of the 18 and all 368 of the schemes failed at least one rule. Fixing them by eye was never going to last, since the next scheme added to the catalogue would bring problems of its own.

So the rules became code, and next to them sits an engine that repairs any palette that breaks them, moving each colour as little as it can. Once it worked on existing themes, I gave it colours nobody had arranged at all.¹

What a theme has to do

Every page on this site draws from the same small set of colour tokens: a background and a foreground, two surfaces (accent for raised cards and muted for quieter strips), a border, secondary text in muted-foreground, a gray, seven named accents from red to purple, and brand, the colour interactive things take. Components often use these at partial opacity, border-border/50 or bg-muted/20, so a colour that's a little too close to the background turns into a border you can't see.

The contract is a function, harmonyIssues, that takes those tokens and returns every rule they break. The thresholds are WCAG contrast ratios:

Rule	Threshold
Body text on the background and on both surfaces	at least 4.5
Secondary text on the background	at least 4.5
Secondary text on the muted surface	at least 3
Body text stronger than secondary text	at least 1.4 times its contrast
Raised surface against the background	1.06 to 1.6
Muted surface against the background	1.15 to 2.4, and a step past the raised one
Border against the background	1.55 to 3.4
Each accent and the gray on the background	at least 3
Each accent and the gray on the raised surface	at least 2.4
Text on a primary button, focus ring	4.5 and 3

Two of those rules go beyond contrast numbers. Surfaces and borders have upper limits as well as lower ones, because a card as loud as the text stops looking like a card. And surfaces and borders have to sit on the foreground's side of the background. In a dark theme the background is the darkest thing on the page and every layer on top of it gets a little lighter, and in a light theme it's the other way round.²

Two more rules, about the accents, came later. They're after the repairs.

Why the engine works in OKLCH

The site stores colours as HSL, which is a bad space to change a colour in, because its lightness isn't the lightness you see. hsl(60 100% 50%) is a yellow and hsl(240 100% 50%) is a blue, both at 50% lightness, and on a near-black background the yellow comes out at 16.29 to 1 while the blue is at 2.04 to 1. Raise the lightness of a hue in HSL and its apparent hue and saturation drift too.

OKLCH splits a colour into lightness, chroma and hue in a way that follows what people see. The engine converts to OKLCH, changes one of the three, and converts back. When a lightness change pushes a colour out of the sRGB gamut, it lowers the chroma until the colour fits, so the hue stays where it was.³

Repairs, in order

The repairs run in the order the rules depend on each other. Surfaces are placed against the text, and borders against the surfaces. Accents come last, because they have to read on everything else.

Text is first. If the foreground is under 4.5 against the background, it gets mixed toward white on a dark theme or black on a light one, and a binary search finds the smallest mix that clears the rule.

Surfaces are next. When one is out of range or on the wrong side, the engine rebuilds it from the background, keeping the background's hue and chroma and moving only the lightness until it hits a target: 1.15 for the raised surface, 1.9 for the border. That way a tinted theme keeps its tint. Mixing toward white or gray would have washed it out.

function surfaceAt(bg: Rgb, fg: Rgb, target: number): Rgb {
  const [L, C, h] = toOklch(bg);
  const step = luminance(fg) > luminance(bg) ? 0.002 : -0.002;
  let candidate = bg;
  for (let n = 1; n <= 500; n++) {
    const next = Math.min(1, Math.max(0, L + step * n));
    candidate = fromOklch(next, C, h);
    if (contrast(candidate, bg) >= target || next === 0 || next === 1) break;
  }
  return candidate;
}

The step's direction comes from the foreground, which is the layering rule written as code. A theme with light text gets lighter surfaces, whatever the palette said.

When the text is too close to a surface, the surface moves back toward the background first, as far as its own minimum lets it. The text only moves if that's not enough. Most theme authors picked their text colour carefully, and the surface is easier to give up.

Secondary text gets pulled two ways. It has to read, at 4.5 on the background, and it has to stay clearly weaker than body text. When both can't hold, the secondary text dims toward the background first, and body text only brightens if dimming would make the secondary text unreadable.

Accents come last. Each keeps its hue and chroma and moves in lightness, 0.005 at a time, until it reads on both the background and the raised surface.

function accentOn(colour: Rgb, surfaces: [Rgb, number][], away: Rgb): Rgb {
  const holds = (c: Rgb) => surfaces.every(([s, min]) => contrast(c, s) >= min);
  if (holds(colour)) return colour;
  const [L, C, h] = toOklch(colour);
  const step = toOklch(away)[0] > L ? 0.005 : -0.005;
  for (let n = 1; n <= 200; n++) {
    const candidate = fromOklch(Math.min(1, Math.max(0, L + step * n)), C, h);
    if (holds(candidate)) return candidate;
  }
  return away;
}

Every repair starts with the same check, and a colour that already holds comes back untouched. A test runs the engine over every hand-made theme and fails if any token changes, so a theme that follows the rules stays exactly as its author wrote it.

Rounding broke five schemes

The first version passed every scheme in my scripts and failed five in the test, and the difference was rounding. Tokens are stored as H S% L% strings, and the test measured what would actually be painted. Values sitting right on a limit were kept, then rounded for storage, and the rounding pushed them over.⁴

Two changes fixed it. A colour that holds is still kept, but one that has to move now goes 0.02 past the limit, so rounding can't pull it back. And base16 colours are rounded to two decimals before the engine sees them, so a colour it keeps is painted with exactly the value it measured.

Colours that aren't what their slot says

base16 has a convention for the sixteen slots. base00 to base07 go from background to foreground, and base08 to base0F are the hues, red first, then orange, yellow, green, cyan, blue and purple, with base0F left over for whatever the author wants. The old code trusted that: accent-red was base08, whatever colour base08 happened to be.

Plenty of schemes don't follow it, so the engine casts the accents by hue. It measures each hue slot in OKLCH, scores every pairing of accent and colour by how far the colour's hue is from the accent's, and hands out pairings starting from the cheapest.

ACCENTS.forEach((accent, order) => {
  for (const { c, i } of chromatic) {
    // Conventional base16 order (base08 red … base0E purple) wins ties.
    const conventional = slotOrder && i === order ? -12 : 0;
    // base0F is the scheme's odd one out, usually a brown; it plays an
    // accent only when no real hue is near.
    const leftover = slotOrder && i === 7 ? 25 : 0;
    // Without an order, the more colourful of two near colours wins.
    const vivid = slotOrder ? 0 : -c[1] * 20;
    pairs.push({
      accent,
      i,
      cost:
        hueDistance(c[2], ACCENT_HUES[accent]) +
        conventional +
        leftover +
        vivid,
    });
  }
});

The base0F penalty is Flatland's doing again. Its base0F is a brown, 78411c, with a hue about three degrees from where orange sits, so the brown took orange and shoved the real orange into red. With the penalty, the brown only plays an accent when nothing better is close.

Across the catalogue, 195 of the 368 schemes have at least one accent that now comes from a different slot, 455 accents in all. A pairing more than 50 degrees off doesn't count, and an accent left without one is made at its own hue, with the median lightness and chroma of the scheme's colourful slots. 171 schemes were missing at least one hue, usually a yellow or a cyan, and the engine made 303 accents for them.

The brand colour comes from base0D, the slot base16 uses for links. The engine picks the accent nearest its hue, unless the link colour is nearly gray or reads as red, where it would look like an error message. In that case it takes the most colourful accent that isn't red or orange.

Names and distance

With contrast sorted every theme was readable, so I measured the accents again. They read on every surface, but they didn't always match their names, and sometimes two of them were the same colour. Monokai and Rosé Pine each had cyan and blue set to one identical value. Gruvbox Dark's blue was 157 32% 56%, which is a teal.

That's a problem anywhere a hue means something, like red for errors, green for success, or one accent per group in a chart. So the contract got two more rules. Each accent's hue has to fall inside a range for its name, and any two accents have to be at least 0.045 apart in OKLab, about twice the 0.02 that's often taken as the smallest difference you can see in OKLab.

Accent	OKLCH hue range
red	350 to 45
orange	30 to 80
yellow	65 to 125
green	105 to 175
cyan	160 to 235
blue	215 to 290
purple	275 to 360

The ranges overlap on purpose. Solarized's orange is at 39 degrees and Catppuccin Latte's yellow at 68, and both are what their authors meant.

These repairs work like the earlier ones. A hue outside its range turns the short way back in, to three degrees past the edge, keeping its lightness and chroma. When two accents are too alike, the one further from its own hue turns toward it two degrees at a time. If it's already there, or too gray for its hue to show, it moves in lightness instead, in whichever direction pulls the pair further apart, and it goes back through accentOn after each step so contrast still holds.

Measured on what the engine produced before these rules, 166 of the 368 schemes broke them, with 154 accents outside their range and 71 pairs too close. Now none do. The hand-made themes changed in 16 values, and those are changes you can see. Gruvbox Dark's teal turned toward blue, and Rosé Pine's "green", which is its blue-leaning pine, turned toward green. For a site where colours mean things I think that's right, but it does cost something, and a Gruvbox purist would notice.

One theme is exempt. The Matrix theme is green on purpose, every accent included, so the name and distance rules skip it, and the test lists it as the only exception.

A theme from any handful of colours

Everything so far starts from a palette someone arranged. themeFromColours starts from colours nobody arranged, anywhere from one to twenty of them, plus a choice of dark or light.

For a dark theme the background is the darkest colour, and for a light one the lightest. If it isn't dark enough (above 0.3 in OKLCH lightness) or light enough (below 0.93), it gets pushed there. Its chroma is capped at 0.035, so a vivid pick can tint the page without taking it over. The foreground starts from the most readable nearly neutral colour, or from the colour at the other end when there isn't one, and its chroma is capped at 0.04. Every other colour with some chroma is a possible accent.

From there it's the same as for a base16 scheme. Surfaces and the border come from the background, secondary text is dimmed from the foreground, and accents are cast by hue. With no slot order to break ties, the more colourful of two nearby colours wins.

Random picks need one more step, which I call cohesion. Colours chosen one at a time rarely share a lightness or a saturation, and a theme with one neon green and six pastels can pass every contrast rule and still look wrong. So after casting, each accent gets pulled toward what the group has in common:

export const COHESION = {
  lightness: 0.12,
  yellowLift: 0.1,
  chroma: { min: 0.7, max: 1.4 },
  chromaFloor: 0.1,
} as const;

An accent's lightness stays within 0.12 of the group's median, and the median itself is kept between 0.5 and 0.8, since there's no room for colour near black or white. Yellow is allowed 0.1 lighter, because a yellow at the others' lightness looks olive. Chroma stays between 0.7 and 1.4 times the median and never drops below 0.1.⁵

Have a go. Each swatch opens a colour picker, and shuffle picks between 10 and 20 random colours. The first preview places the same colours by lightness alone: darkest as background, the next few as surfaces and border, the lightest as text, and the rest as accents in the order they were picked. Both sides use this site's own classes, and both are counted against the contract.

The theme maker is interactive; try it in the original post.

With the starting colours, placing by lightness breaks 8 rules for a dark theme and 15 for a light one. The engine's version breaks none. Across 2,000 random sets of 10 to 20 colours, placing by lightness broke at least one rule every single time, 17.8 on average.

I ran the generator over 24,000 themes: four seeds, six kinds of input for each (fully random colours, a single hue, pure grays, pastels, neon, and sets of just two colours), 500 sets of each kind, dark and light. Every one held the contract after rounding.

It does move your colours. Of the accents it made from random sets, 29% came out within 0.02 of a picked colour, close enough to look the same, and the median distance was 0.052. A lot of random RGB colours are too dark or too dull to work as accents, and moving them is what it costs to get a theme you can read. Colours that already fit mostly stay where they are. Give it Catppuccin Mocha's own background, text and accents, and the background, red, orange, green, blue and purple come back unchanged. Yellow and cyan get pulled toward the group, and the text loses a little of its blue.

How it runs on this site

Themes are CSS variables holding H S% L% strings, which Tailwind v4 reads through @theme inline, so bg-accent compiles to hsl(var(--accent)). A hand-made theme is a class on <html>. I ran the engine over those, wrote the repaired values into the stylesheet, and the test keeps them there.

A base16 scheme is computed in the browser when you pick it and written as inline variables on <html>.⁶ The tricky part is the first paint. A small script in the head applies your stored theme before anything renders, and the engine is too large to inline there. So each time a scheme is applied, the computed tokens go into localStorage, keyed by the palette and a hash of the engine's output on a fixed sample scheme. The head script paints from that cache when both match. When they don't, because the scheme is new or the engine changed, it paints the raw slots, which are at least the right lightness to avoid a flash, and the full result takes over once the page hydrates.

theme-harmony.test.ts keeps all of it in line. It checks every hand-made theme, all 368 schemes as they'd be painted, 720 generated themes from a fixed seed, the head script against the cache, and that the engine leaves a theme alone when it already holds. It takes about half a second.

Where it stops

The contract checks contrast and where hues sit. It can't measure taste, and a theme can pass every rule and still not be one you'd pick. Cohesion helps with random colours, but it's a rule of thumb, with numbers I tuned against the failures I happened to see.

It also changes themes people know. The engine rewrote 41 values in the hand-made themes for contrast and 16 more for names and distance, and a few of those show. And for now the generator only lives in this post. Saving a generated theme to the site's theme picker needs one more step, because the picker stores themes as base16 palettes.

How I measured, so you can argue with it

Contrast is the WCAG 2 ratio from relative luminance. Distances are Euclidean in OKLab. Hue ranges and anchors are in OKLCH degrees, with each accent's anchor close to the median hue the catalogue gives it.

"Before" numbers for the hand-made themes and the schemes come from running the current contract over the stylesheet as it was before the engine, and over each scheme's slots placed by the old fixed mapping. The 166 figure comes from running the current contract over the previous engine's output. Every "after" number is measured on tokens rounded to two decimals, the way they're painted.

The random sets come from a linear congruential generator with seeds 1, 7, 42 and 2026, so each run can be repeated. The fidelity numbers use 2,000 sets of 10 to 20 random colours, alternating dark and light. Timings are wall clock in Node, averaged over each 6,000-theme run, and they came out between 0.4 and 1.1 ms per theme across runs.

Originally published at zeybek.dev.

There's a demo of that further down. ↩
That's the one Flatland broke, with cards and borders a shade darker than the page. ↩
Contrast is still measured with WCAG's relative luminance, since that's what the rules are written in. ↩
One scheme's border ended up just over 3.4, another's body text just under 1.4 times its secondary text. ↩
That floor came out of the random runs: red and orange are 28 degrees apart, and below about 0.1 chroma two colours that far apart end up closer than the 0.045 the distance rule asks for. ↩
In Node that's under a millisecond per scheme. ↩

How camouflage.nvim masks secrets in Neovim without leaking a frame

Ahmet Zeybek — Tue, 06 Oct 2026 07:19:16 +0000

Disclosure

I maintain camouflage.nvim, one of the three plugins compared here. All three were measured the same way, and the method is at the end if you want to check it.

Say you're sharing your screen in a pairing session, or recording a walkthrough for the team, and you paste a new database password into values.yaml. If the mask lands one frame after the text, the password is on screen for that frame. Nobody on the call will read it. The recording keeps it, though, and anyone watching later can scrub back and pause there.

I know of three Neovim plugins made to hide values in files like .env while you share your screen: cloak.nvim, shelter.nvim and camouflage.nvim. They differ in what happens while you edit, and in which files they can read at all.

How masking works

None of them touch the file. The plugin works out where each value sits, puts an extmark on it with virt_text_pos = "overlay", and Neovim draws the mask over the real characters. Anything that reads the buffer still gets the real text: grep, the LSP, completion, yy, any AI tool you've got attached. So these plugins protect your screen and nothing else.¹

For each plugin, then, the question is whether there's ever a frame where a value is drawn without its mask.

The race with the redraw

Masking a file when it opens is the easy case. Edits are harder. You type a value, paste one or put a line from a register, and Neovim redraws straight after. If the mask isn't in place by then, the value is on screen for at least a frame.

Each of the three hooks in at a different point.

cloak.nvim re-masks in its TextChanged and TextChangedI autocmds. Neovim fires those before the redraw, so typed text is covered. It redoes the whole buffer every time, and at the size of a normal .env that's fine.

shelter.nvim attaches with nvim_buf_attach and re-masks inside on_lines, which runs as part of the change itself. It only touches the changed lines, and it wraps vim.paste so a bracketed paste is masked as it lands. When an edit changes the number of lines, though, it hands the work to vim.schedule and re-masks the whole buffer, and that runs after the redraw.

camouflage.nvim's parsers read the whole file, and for JSON, YAML, TOML, XML and HCL they go through TreeSitter. Doing that inside on_lines would mean a full structural parse on every keystroke, against a tree that hasn't seen the edit yet. So it masks in two layers.

The first layer runs inside on_lines and doesn't parse anything. For each changed row it looks for where a value starts, from KEY=value, key: value, "any key": value, ENV KEY value, <element>value, or a mask already on the row, and puts a provisional mask from there to the end of the value.

vim.api.nvim_buf_attach(bufnr, false, {
  on_lines = function(_, buf, _, first, _, last_new)
    if state.buffers[buf] == nil then
      return true -- the buffer isn't tracked any more, detach
    end
    guard.mask_rows(buf, first, last_new)
  end,
})

The second layer is the real parse. It runs when you pause typing, works out the exact ranges from the file's structure, and swaps the provisional masks for exact ones in a single step, with no redraw in between.

On a 500-line .env the first layer costs 0.006 ms per changed row, median. It has two limits. Until you stop typing it can cover a value your policy rules would leave visible. And a new row in the middle of a multi-line value, or a .netrc line of bare tokens, has no separator for it to find, so that row stays visible until you pause and the full parse runs.

What reaches the screen

You can only learn so much from reading the hooks, so I recorded what actually gets drawn. A small harness starts nvim --embed with one plugin loaded and attaches a UI over msgpack-RPC, the protocol GUIs use. It rebuilds the screen from grid_line events and keeps a frame each time Neovim flushes. Then it uses the editor the way a person would: it opens a .env, types a new KEY=value at 60 ms a key, appends to a value that's already masked, puts a line with "ap, and pastes two lines through nvim_paste. Every frame is checked for characters of the secret in plain text.

Scenario	cloak.nvim	shelter.nvim	camouflage.nvim
Open a file	masked	masked	masked
Type a new `KEY=value`	masked	masked	masked
Append to a masked value	masked	masked	masked
`"ap` a line from a register	masked	1 frame visible	masked
Bracketed paste, 2 lines	1 frame visible	masked	masked

A frame is about 17 ms. Nobody on a live call reads a value in that time, but on a recording someone can stop on it.

In my runs all three had the mask up on the first frame, across three starts of nvim --embed .env per plugin.²

What it costs

Timing a parser on a string misses most of the work, so I timed each plugin until its marks were written in the buffer, on the same generated .env lines. camouflage ran twice, once with its default checks and once with them off.

A full pass, masking the whole buffer from scratch, median in milliseconds:

Lines	cloak	shelter	camouflage, checks off	camouflage, defaults
10	0.04	0.02	0.05	0.10
100	0.24	0.19	0.45	0.88
500	1.20	1.02	2.41	4.60
2,000	4.84	4.22	10.48	18.46

Editing the first line, then re-masking through each plugin's own change path:

Lines	cloak	shelter	camouflage, checks off	camouflage, defaults
10	0.04	0.01	0.05	0.10
100	0.25	0.03	0.48	0.89
500	1.25	0.18	2.60	4.73
2,000	5.04	0.83	11.45	21.56

On a full pass cloak and shelter come out close. Once the ranges are known, most of the time goes on nvim_buf_set_extmark calls, and those cost the same no matter which language found the range. camouflage takes about twice as long with its checks off and about four times with them on. With checks off, the extra is the general pipeline every file goes through, a Lua parser plus policy rules and hooks. With them on, the per-value checks behind the weak secret and JWT expiry badges add about the same again, and a project that doesn't want them can turn them off.

On edits shelter is clearly fastest, because it re-parses only the line you changed. At 500 lines it's about seven times quicker than cloak. camouflage doesn't run its full parse on every keystroke: while you type it pays the 0.006 ms per row from earlier, and the edit table shows what the parse costs when you pause.

The budget for all of this is one frame, 16.7 ms at 60 Hz. Every cell up to 500 lines is well under it.³ Only a 2,000-line file with camouflage's default checks goes over.

Where the secrets live

So far it's all been .env, because that's the one format all three read. In a real project it's rarely the only place secrets live. A .NET service keeps its connection string in appsettings.json, a Helm chart has a database password in values.yaml, Terraform has terraform.tfvars, a Dockerfile has ENV API_TOKEN ..., and your home directory has ~/.netrc.

cloak can take a Lua pattern per file type, like :.+ for YAML, but a pattern sees one line at a time and has no idea which key that line belongs to. shelter's parser only does dotenv. camouflage parses env, JSON, YAML, TOML, INI and properties files, netrc, XML, .http files, Terraform and HCL, and Dockerfiles, and it keeps track of nested keys like ConnectionStrings.Default.

Out of the box it masks every value it finds. A .camouflage.yaml in the repo says what stays readable. It's data only, and nothing in it gets executed:

version: 1
policy:
  rules:
    - id: plain-settings
      action: ignore
      key: ['^Logging%.', '^AllowedHosts$']

With that file in place, an appsettings.json looks like this on screen:

{
  "Logging": { "LogLevel": { "Default": "Information" } },
  "ConnectionStrings": {
    "Default": "***************************************************"
  },
  "Stripe": {
    "SecretKey": "*******************************",
    "WebhookSecret": "**********************"
  },
  "AllowedHosts": "*"
}

When you do need a value, :CamouflageYank copies it after a confirm prompt and clears the clipboard 30 seconds later. That matters, because yy on a masked line still copies the real text. :CamouflageAudit puts every value it would mask across the project in the quickfix window, with the file, key and length, and never the value itself.

	cloak.nvim	shelter.nvim	camouflage.nvim
Files	any, one Lua pattern per line	dotenv only	env, JSON, YAML, TOML, INI/properties, netrc, XML, `.http`, Terraform/HCL, Dockerfile
Understands nested keys	no	no	yes
Build step	none	Rust toolchain on first setup	none
Pickers	Telescope	Telescope, fzf-lua, Snacks, oil	Telescope, Snacks
Completion	turns off nvim-cmp	nvim-cmp, blink.cmp	nvim-cmp
Beyond masking	reveal current line	partial mode, peek, ecolog integration	reveal and follow-cursor, confirmed yank, audit, policy rules, weak secret and JWT expiry badges, HIBP checks on request, parser and check APIs
Last commit	June 2024	March 2026	September 2026

Which one to pick

If all your secrets are in .env and a Rust toolchain doesn't bother you, shelter.nvim is careful about the screen and has the fastest edit path of the three. A put from a register that adds lines still shows one frame before the mask, though.⁴

If you want the smallest thing that works, cloak.nvim is a single Lua file with no build step, and it handles typing well.

If your secrets are spread across JSON, YAML, Terraform and the rest, camouflage.nvim reads those files. It's the one I run.

camouflage.nvim

{
  "zeybek/camouflage.nvim",
  event = { "BufReadPre", "BufNewFile" },
  opts = {},
}

shelter.nvim

-- needs cargo on PATH for the first setup
{
  "ph1losof/shelter.nvim",
  lazy = false,
  init = function()
    vim.filetype.add({
      filename = { [".env"] = "dotenv" },
      pattern = { [".?env.*"] = "dotenv" },
    })
  end,
  opts = {},
}

cloak.nvim

{
  "laytan/cloak.nvim",
  opts = {},
}

If camouflage misses a file shape you use, or masks something it shouldn't, open an issue and include an example of the file.

How I measured, so you can argue with it

It all ran on an Apple M2 with Neovim 0.12.5, one plugin per nvim --headless --clean process. Versions: cloak.nvim at 648aca6, shelter.nvim at 604e983 with its native library built by cargo build --release, camouflage.nvim 0.14.1.

The timing tables are 2,000 iterations per cell over three runs, and each cell is the lowest of the three medians, since the machine had other work on it.⁵ For the edit table each iteration replaces line 1 and calls the plugin's own change path: shelter_buffer(bufnr, true, { min_line = 0, max_line = 1 }), cloak.cloak(pattern) and apply_decorations(bufnr). Nothing is cleared outside the timed region. camouflage runs with project config and HIBP off, and "checks off" also turns off checks.expiry and checks.weak_secret.

The screen test used a 100 by 12 UI with ext_linegrid, and keys went in one at a time through nvim_input, 60 ms apart. shelter got the dotenv filetype mapping from its README. A frame counts as a leak when any character of the value shows in its own position after the =, and each plugin ran twice with the same counts both times.

Originally published at zeybek.dev.

They won't keep a secret out of a log or a prompt, which camouflage's README says plainly in its security model section. ↩
cloak.nvim#25 shows a flash on open in a real terminal, though, so I don't think that row is settled for cloak. ↩
At 500 lines camouflage with checks off is about a millisecond behind cloak, under a tenth of a frame. ↩
It keys off the dotenv filetype, and Neovim 0.12 detects .env as env, so add the vim.filetype.add mapping from its README or nothing gets masked at all. ↩
camouflage's timings were taken at 4139aeb. The provisional layer doesn't change the full pass, and a spot check on 0.14.1 came out within noise (4.53 and 2.48 ms at 500 lines). ↩

The outbox pattern without a relay

Ahmet Zeybek — Tue, 06 Oct 2026 07:19:07 +0000

Work on a distributed system long enough and you'll know the particular cold sweat of the dual write. You insert an order into the database and the transaction commits. Then you have to tell something outside the database, a service over HTTP or a Kafka topic, and the connection drops before you can. The order exists and nobody downstream knows. Or it goes the other way: the message goes out first, the transaction rolls back, and a consumer is now holding an event for an order that was never created, a ghost.

No amount of try/catch will make those two writes atomic.

Think of paying at a restaurant. The payment is your database transaction and the receipt is the message to the outside world. Walk out without the receipt and you can't prove the meal happened. The transactional outbox pattern is the industry's answer to that gap. You don't collect the receipt at the door, you put it in your pocket the moment you pay: the message goes into an outbox table in the same transaction as the business row, so both commit or neither does.

That solves the dual write, and creates the problem this post is about.

The relay is a second system

Someone still has to take the receipt out of your pocket and show it to the world. In the usual design that's a relay, a separate service outside the database that polls the outbox table, delivers each row and marks it done.

The relay looks small on the whiteboard and turns out large in production. It has its own deployment, configuration, connection string and alerts. It has to claim rows without two instances grabbing the same one, retry, back off, give up, and put whatever it gave up on somewhere a person can look. And since it lives outside the database, its view of the world can drift away from the database's, which is how relays end up polling a replica for hours while reporting an empty queue.

I wrote ulak because I kept building that relay, and every time it was the least interesting and most fragile part of the system. ulak is a C extension for PostgreSQL 14 through 18, under Apache 2.0, that keeps the outbox pattern and gets rid of the relay. The queue is a table you write to in your own transaction, and PostgreSQL background workers do the delivery.

What it is not

ulak isn't an event streaming platform, and it doesn't replace change data capture. If you're moving hundreds of thousands of log events a second, use Kafka. If you need every change in a database mirrored into a log, use Debezium. ulak's job is narrower. For a system whose core already lives in PostgreSQL, it delivers outbox messages locally and transactionally, without a broker.

Enqueue is your transaction

On the application side it's one function call inside the transaction you already have:

BEGIN;
  INSERT INTO orders (customer_id, total_cents) VALUES (42, 12990);

  SELECT ulak.send(
    'billing-webhook',
    jsonb_build_object('order_id', 17, 'total_cents', 12990)
  );
COMMIT;

ulak.send inserts a row into ulak.queue, and that row shares its fate with your order. If the next line of your code throws and the transaction rolls back, the message doesn't float off somewhere. It's gone, as if it never existed. If the transaction commits, the message is durable before anything has tried to deliver it. The receipt is in your pocket.

Delivery is a PostgreSQL process

This is where the design gets unusual, and a bit bold. The workers that drain the queue are PostgreSQL background workers, registered through shared_preload_libraries when the server starts, so they live and die with the server. There's no relay to deploy, and when the database fails over the workers go with it, because they're part of it.

Most engineers' first reaction is the right one. PostgreSQL is the heart of the system and the only guardian of the data, and putting a library inside it that runs its own loops and makes its own network calls sounds risky. What if a bug in a worker locks the database or eats its memory?

That worry is the central trade-off of the whole design, and I'm not going to argue it away. You drop the operational cost of a separate service and the network hop between it and the database, and in exchange the delivery pipeline lives and dies with the database. PostgreSQL's background worker infrastructure has matured a lot. Workers are isolated processes with their own memory contexts under PostgreSQL's rules, so a leak is reclaimed when the transaction ends, and a crash doesn't take the postmaster down. It's still a decision that needs a deliberate yes from whoever runs your database.

How workers share one table

Say you run ten workers. The obvious question is what stops all ten charging at the first row in the queue, and ulak answers it in two layers.

The first is arithmetic. Each worker gets a deterministic sequential id at startup, and its poll query has one extra condition: the row's id modulo the worker count has to equal the worker's id. One big table becomes ten separate logical slices, and normally nobody eats off anyone else's plate.

Real life is messier than a formula. Change the worker count while the system is live and the slices shift. Let an operator lock a few rows by hand in a console and a worker could get stuck on them. So the arithmetic is only there to cut contention, and the second layer is what makes it safe: FOR UPDATE SKIP LOCKED. When a worker hits a row someone else holds, it skips it and moves on instead of waiting.

SELECT id, endpoint, payload
FROM   ulak.queue
WHERE  status = 'pending'
  AND  scheduled_at <= now()
  AND  id % 10 = 3        -- this worker's slice
ORDER  BY id
LIMIT  200
FOR UPDATE SKIP LOCKED;

▶ Three workers sharing one queue table: an animation that plays in the original post.

There's a third decision underneath. ulak requires the workers to run at READ COMMITTED, PostgreSQL's default, and refuses REPEATABLE READ. The instinct is to reach for the stricter level. But under REPEATABLE READ PostgreSQL uses snapshot isolation, and when several workers use SKIP LOCKED on overlapping rows, a worker that sees a row someone else updated after its snapshot was taken gets serialization failure 40001. The workers would spend their whole lives aborting and retrying. At READ COMMITTED every statement sees the latest committed data, so a locked row is skipped, an unlocked one is claimed, and there's no conflict to raise.

What "delivered" means

Can ulak promise exactly-once delivery? No, and I want to be clear about that, because it's the golden rule of distributed systems: nothing that crosses a network can guarantee exactly-once by itself.

Step	Guarantee	Why
Writing to `ulak.queue`	Exactly once	It is your transaction
Delivering to HTTP, Kafka, etc.	At least once	The network can lose the acknowledgement

Picture the worst case. ulak sends a message to your payment API. The API processes it and writes it to its own database, and the connection dies the instant before it can say "got it". ulak never sees the acknowledgement, so it does the only thing it can and sends the message again. That's why your consumer has to be idempotent: getting the same message twice mustn't corrupt its data. Correctness doesn't stop at the database. It reaches whoever consumes the message.

The same problem exists on the way in. Your code calls ulak.send, and the database connection drops just as the transaction is finishing. Your code reasonably decides the write failed and retries, and a blind retry would enqueue the same order twice. The fix is an idempotency key you supply yourself:

SELECT ulak.send_with_options(
  'billing-webhook',
  jsonb_build_object('order_id', 17),
  idempotency_key => 'order:17:created'
);

ulak stores the MD5 of the key, not of the payload, and enforces it with a partial unique index on ulak.queue. "Partial" matters here. Over time the queue piles up millions of delivered rows, and an index over all of them would get enormous and slow down every insert. So the index only covers rows whose status is pending or processing. A second send with the same key while the first is still active hits the index, ulak compares the hashes, ignores the new row, and returns the id of the existing message. There's no trip to Redis for a distributed lock, and it gets settled where the data lives.

The trade-off that should bother you

This bothered me most while I was designing it, and it's written up in the 0.0.3 architecture notes so nobody finds it out in production.

Claiming a batch, making the network call and updating the status all happen inside one open PostgreSQL transaction. If the API you're delivering to takes two seconds to answer, the worker's transaction stays open for two seconds. Open transactions hold locks and tie up a connection, and a system doing thousands of deliveries a second against a slow downstream can wedge itself fast.¹

Two things keep it in check. The first is at the storage layer: worker transactions run with synchronous_commit = off. Normally PostgreSQL won't treat a transaction as finished until the WAL record is flushed to disk, and after the network the disk is the slowest thing around. With the flush deferred, a worker marks a message delivered and goes straight on to the next one.

Skipping the flush would be reckless for business data. For the queue it's safe, because of the at-least-once model from the previous section. If the server loses power in the millisecond after a worker marks a message delivered and that WAL record is lost, the row comes back as pending on restart, as if it had never been sent. A worker wakes up and sends it again, and since the consumer is idempotent, the duplicate does no harm.

The second is the circuit breaker, in the next section. Together they limit how long a transaction can stay open behind a bad downstream. They don't make the cost go away, though. Timeouts on your endpoints and a modest batch size are your side of the bargain.

One more cost sits in the same place.² Kafka, MQTT, Redis Streams, AMQP and NATS each need their C client library compiled into the extension, which means optional external C dependencies on your database server.

When the other side is down

If a downstream API dies outright, retrying every message against it at full speed burns connections and achieves nothing. ulak keeps an in-memory circuit breaker per endpoint, with three states. Closed is normal. When consecutive failures cross the threshold the breaker opens, and workers skip every row for that endpoint. Nothing gets sent, the downstream gets a breather, and other endpoints keep flowing.

The breaker shouldn't stay open forever. After a cooldown it goes half-open, and one probe is let through. The subtle part is deciding which worker sends it. Fifty workers scanning the queue can all notice the half-open state at the same moment, and without coordination they'd all probe, which is exactly the stampede the breaker is there to stop.

ulak settles this with a compare-and-swap on the breaker state. Think of one microphone in a room. Fifty workers want to ask whether the other side is back, and only the one that wins the swap gets to speak. It sends the single probe, and the rest see the microphone is taken and keep deferring their rows. If the probe gets a 200, the breaker closes and things carry on. If it fails, the breaker opens again and the cooldown restarts.

▶ A circuit breaker opening, and closing on one probe: an animation that plays in the original post.

Not every failure deserves a retry, so ulak sorts them first. A transient error, like a timeout or a connection reset, is marked retryable and rescheduled with growing backoff. A permanent error, like an HTTP 400 because the payload failed the other side's validation, isn't retried at all, because a million attempts wouldn't change the answer. A permanent error and an exhausted retry budget both move the message to ulak.dlq, the dead letter queue, where it waits for a person.³

When the worker itself dies

There's one more failure, and it's the one people forget. A worker claims a message, marks it processing, and then the process dies mid-request, from a bug or a hardware fault. Now the row is processing forever. Other workers won't touch it, because as far as they can tell someone's working on it.

So ulak gives worker 0 an extra job on top of its normal batches: a periodic crash recovery pass that looks for rows stuck in processing past a timeout and puts them back to pending. The workers that are still alive pick the row up and finish the job.

The bet

Put together, ulak is the outbox pattern with PostgreSQL's own machinery doing the relay's job from inside the database. In return it asks you to accept one deliberate trade: network calls now run inside your database process, and the transaction stays open while they run.

For a decade the microservices consensus has been that databases are dumb storage and the logic belongs in external tools: in Kafka, in RabbitMQ, in a relay. That got taught as a rule. ulak bets that for a database-centric system the rule is backwards, and that the database, the one component that already knows exactly what's been committed, is the right thing to announce it.

Whether that's a bet you should make depends on whether you run your own PostgreSQL, whether your core already lives there, and whether you're willing to hand an old friend that much responsibility again. The code is at github.com/zeybek/ulak. If you run it and it breaks, open an issue. I'd rather know.

Originally published at zeybek.dev.

Making HTTP requests from inside a database is playing with fire, and I'm the one who lit it. ↩
HTTP is built in, since libcurl is the only hard dependency. ↩
And when an HTTP server answers with a Retry-After header, ulak drops its own backoff calculation and waits exactly as long as it was told to. ↩

Letting an agent touch production

Ahmet Zeybek — Tue, 06 Oct 2026 07:19:02 +0000

Every vendor is making the same pitch for an on-call agent. An alert fires at 3am, the agent reads the runbook, checks the dashboards and finds the cause, then either fixes it or wakes a human with a diagnosis instead of a bare pager message. Nobody wants the 3am page, and everybody wants the diagnosis.

The fear is just as easy to picture. The agent misreads a graph, decides the fix is to restart the database, and it's 3am and nobody's watching.

From March to August we worked out where the truth sits between those two. The agent is on the rotation now, in a limited way, and I wouldn't go back. But it got there in four stages, each with a gate, and the gates are the part of the story worth telling.

Stage one: read everything, touch nothing

For the first six weeks the agent could read logs, metrics, traces, the runbooks, the deploy history and the incident history, and couldn't write to anything. When an alert fired, the agent was triggered alongside the human page. It investigated, wrote up what it found and posted that in the incident channel. The human on call did their normal job and read the write-up at some point.

To move on, on-call engineers had to rate the write-up useful, on a two point scale, at least 80 percent of the time across 30 incidents.

It failed the first time. The write-ups were long, restated the alert and hedged, and by week two engineers had stopped reading them. The fix was changing what we asked the agent to produce: one line for the most likely cause, one for the evidence, one for the recommended action, and everything else in a thread.¹ After that the rating went to 86 percent, and by week six the on-call engineers were opening the write-up before the dashboard. That was how we knew it was ready.

We learned two things in this stage that shaped everything after it. The agent was better than the median on-call engineer at linking a deploy to an alert, because it always checked the deploy history and humans at 3am don't. And it was worse at telling when a graph was normal, because it had no memory of what the graph usually looked like. We fixed that by giving it a tool that returns seven days of baseline for any metric it looks at, and that one tool cut its false "this is anomalous" rate roughly in half.

Stage two: propose the action, a human clicks

In stage two the agent could offer an action from a fixed list, as a button in the incident channel: restart this service, scale this deployment from 3 to 6, roll back this deploy, flip this feature flag off. The human on call clicked it or didn't. The agent couldn't click.

The list was short on purpose. It had eight actions, all reversible, all things a runbook already told a human to do in that situation. To move on, the human had to click the proposed button unchanged at least 75 percent of the time over 40 proposals, and no proposal could have made things worse if it had been executed.

That second condition needed a review. Every proposal, clicked or not, was looked at the next day by someone who wasn't on call, and they answered one question: if this had run automatically, would it have been correct? Of the first 40, 34 would have been correct, 4 harmless but useless, and 2 wrong. Both wrong ones were rollbacks of a deploy that lined up in time with the alert but didn't cause it. It was the same failure twice.²

We fixed it with a rule instead of a prompt change. A rollback proposal has to point to something in the deployed change that could cause the failing behaviour, on top of the timing. The agent had to read the diff and say what in it could produce the symptom. If it couldn't, it could still propose the rollback with a "timing only" label, and the human treated that label as a reason to look harder. In the next 40, wrong proposals went to zero.

Stage three: act on the short list, tell everyone

Stage three is what people mean when they say "the agent is on call". For four of the eight actions, the agent could go ahead without a click: restart, scale up, flag off, and the rollback with a diff citation. Scaling down, rolling back on timing alone and anything involving data stayed off the list.

Every automatic action followed three rules.

It announces before it acts, in the channel, with a 60 second window where a human can type stop.³

It takes one action per incident. If that action doesn't clear the alert, the agent goes back to proposing and a human decides. The rule exists because autonomous systems rarely fail with one bad action. They fail with a chain of reasonable-looking actions that add up: restart didn't help, so scale up, which didn't help, so roll back, and now three things have changed and nobody knows which one mattered. So it gets one action, then a human.

It records everything as an incident timeline entry, in the same format as a human's actions, with its reasoning attached. The post-incident review reads the agent's actions exactly the way it reads a person's.

The gate here was stage two's second condition again, over 30 automatic actions: none could make things worse. Getting to 30 took eleven weeks, because most incidents in that time were resolved at the proposal stage or were things the agent rightly declined to touch. None made things worse.⁴

Stage four, which we are in

There are eleven automatic actions now. They were added one at a time, each after a stretch as a proposal with a perfect record, and the stage three rules still apply to all of them.

What changed most in stage four was the rotation, more than the agent. The human on call now gets paged for about 40 percent of the alerts they used to get. The agent either resolves the other 60 percent or diagnoses them far enough that the page says "this is the known flaky job, agent has restarted it, no action needed", and the engineer acknowledges from bed without opening a laptop. Mean time to a first diagnosis, from the alert to the first timeline entry that names a cause, went from around 14 minutes to under 3.

The pages that are left are the real ones, the ones that should wake somebody up.

The rules, on one page

Read before you propose, and propose before you act. Each stage is gated on the record of the one before, reviewed by someone not on call.

Actions come from a fixed list. The agent doesn't make one up, and adding to the list is a decision made in daylight.

Every automatic action is reversible, and announced with a window to stop it.

One action per incident, then a human.

Timing isn't causation. A rollback needs a cited diff.

Give the agent a baseline for every metric, or it'll think everything is an anomaly.

The output is three lines, and everything else goes in the thread.

Record the agent's actions the way you record a person's, and review them the same way.

The one incident

There was one, in stage three. It's worth telling because it wasn't the agent's fault and the rules still mattered. A restart proposal ran automatically, was announced and completed correctly. At that moment a human, paged for a different alert on the same service, was halfway through restarting the same pod by hand. The two restarts overlapped, and the service was down for 40 seconds longer than it needed to be. The fix was a lock: before acting, the agent checks for an active human session on the target and backs off if there is one. The daylight review caught it the next morning, which is exactly why the review exists.

What I would not do

I wouldn't start at stage three because a vendor's demo did. The demo is stage three on a scripted incident, and your incidents aren't scripted.

I wouldn't give the agent any action that touches data: no migration, no cache flush, no queue purge. Those stay with humans, maybe forever, because for those actions "reversible" is a matter of opinion.

I wouldn't skip the daylight review of proposals. It's tedious and takes ten minutes a day, and it's the only thing that turns the agent's mistakes into rules before they turn into incidents.

And I wouldn't measure it by pages avoided. Measure time to first correct diagnosis, and actions that made things worse. The first is what you get. The second is what you pay, and if it isn't zero the agent isn't ready for the next stage, however quiet the pager has got.

Originally published at zeybek.dev.

The investigation stayed the same, only the output changed shape. ↩
Reading the deploy history for correlation had been the agent's strength in stage one, and in stage two it turned into over-confidence. ↩
At 3am nobody will, and that's fine, but during the day someone often does, usually with "wait, I'm already on it". ↩
Two were unnecessary, restarts of a service that would have recovered by itself within a minute, and those were fine. ↩

The npm worm playbook for a small team

Ahmet Zeybek — Tue, 06 Oct 2026 07:18:58 +0000

The first Shai-Hulud, in September 2025, felt like an event. By the third wave it feels like weather. On 12 May 2026 the TanStack maintainers, Mistral, UiPath and around 160 other npm and PyPI packages shipped poisoned versions, from a campaign the reports attribute to TeamPCP. On 4 August Elastic Security Labs spotted a new one, ChainDrop, which started with the maintainer of keyv and has since spread to over 400 packages.

It works the same way every time, and that's worth spelling out, because the defence follows from it. A maintainer's publish token gets stolen, usually through a phishing page, or a CI secret that leaked in an earlier breach and was never fully rotated. The malware in the poisoned package runs on install, finds every other npm token on the machine that installed it, and publishes itself into every package those tokens can reach. It starts from one credential and grows into a tree.

I run a small team and we don't have a security function. What we do have is a set of controls that took about a day to set up and would have stopped every one of these campaigns at the point where they reach our machines. This post is that set.

The install is the attack

Everything the worm does, it does during npm install, through a preinstall or postinstall script. You don't have to import the package and your code doesn't have to run. All it needs is for the package manager to run a lifecycle hook, and npm, pnpm and Yarn all do that by default.

So the first control turns that off.

pnpm

# .npmrc
ignore-scripts=true

// package.json: allow the handful that genuinely need a build step
{
  "pnpm": {
    "onlyBuiltDependencies": ["esbuild", "sharp", "@biomejs/biome"]
  }
}

npm

# .npmrc
ignore-scripts=true

Yarn

# .yarnrc.yml
enableScripts: false

With scripts off, a poisoned version of a package you already depend on gets downloaded and unpacked and then just sits there until your code imports it.¹ That's still not great, but it's the difference between "compromised the moment CI ran" and "compromised if we ship the version", and the second one you get a chance to catch.

Turn this on and something will break. Usually it's a native module that downloads a prebuilt binary in postinstall, like sharp or esbuild. You add those to the allow list one at a time, and every addition is a package you've consciously decided to trust to run code on your machines. On my main project that list has four entries. Before, it was effectively 1,400.

Do not install what was published this week

ChainDrop's poisoned versions were on the registry for about six hours before the first reports, and gone within a day. The TanStack versions lasted longer, but still under 48 hours for most of the affected packages. The attackers are counting on automation: Renovate, Dependabot, npm update in a nightly job, a developer running pnpm up on Monday morning.

The control is a minimum age for any version you install.²

pnpm

# .npmrc
minimum-release-age=4320   # minutes, 3 days

npm

# .npmrc
min-release-age=3d

Renovate

{
  "minimumReleaseAge": "3 days",
  "internalChecksFilter": "strict"
}

Three days is long enough that every campaign so far would have been caught and unpublished before your lockfile ever pointed at it. On stable packages it costs you nothing. It does cost you when there's a security fix you want the same day, and then you override it for that package and make the call on purpose.

The lockfile is the other half. pnpm install --frozen-lockfile in CI, no exceptions. If CI is allowed to regenerate the lockfile, it's only a suggestion, and that's exactly the opening an install time worm wants.

Tokens are the actual target

The worm isn't after your laptop. It wants the NPM_TOKEN in your CI secrets and the ~/.npmrc on your machine, because those let it publish. If you maintain any package at all, even an internal one on a private registry, this is where to spend your afternoon.

Classic npm tokens shouldn't exist any more. In 2025 npm shipped trusted publishing with OpenID Connect: your GitHub Actions workflow proves who it is to the registry with a short lived token minted for that run, so there's no long lived secret to steal. The workflow looks like this:

name: publish
on:
  push:
    tags: ["v*"]
permissions:
  id-token: write
  contents: read
jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: pnpm/action-setup@v4
      - run: pnpm install --frozen-lockfile
      - run: pnpm publish --provenance --access public
        env:
          NPM_CONFIG_PROVENANCE: "true"

There's no NODE_AUTH_TOKEN. The registry trusts the workflow, the workflow attests to the commit, and the published package carries a provenance statement anyone can check. If this workflow is the only way your package can be published, a stolen token is worthless, because there isn't one.

On the registry, turn on the setting that requires two factor for publishing, and if you can, the one that blocks token based publishing for the package altogether. On your laptop, run npm logout, delete the _authToken line from ~/.npmrc and never put it back. If you have to publish from a machine, use a granular token scoped to the one package that expires in a week.

The March 2026 CI siege that hit trivy-action and a batch of OpenVSX extensions started with one GitHub personal access token that had been rotated everywhere except one place. Rotation that's 95% done might as well be 0%. The way to finish it is to have fewer long lived tokens to rotate, and OIDC is how you get there.

Know what you would have to clean up

Preventive controls are good. You still need to be able to answer "were we affected" within an hour of a disclosure, which means knowing which versions of what you had installed, on which machines, on which dates.

A lockfile in git answers most of that for the repo. It doesn't answer it for developer laptops that ran npx something last Tuesday. For those we have a short script that dumps the global npm and pnpm caches to a text file and commits it nightly to a private repo from each machine. It's crude. When the ChainDrop list came out I grepped that repo for the 400 package names and had an answer in five minutes.³

The check itself:

# affected.txt is one package@version per line from the advisory
pnpm ls -r --depth Infinity --json \
  | jq -r '.. | objects | select(.from? and .version?) | "\(.from)@\(.version)"' \
  | sort -u \
  | grep -Fxf affected.txt

Zero lines is the answer you want. If you get anything else, rotate every credential that machine could see and reinstall from a clean lockfile.

What does not work

Auditing doesn't work. npm audit tells you about vulnerabilities that have been reported, and a worm's poisoned version has no CVE for the first several hours, which are the hours that matter. A scanner that runs on install is still running after the postinstall hook has already gone off. Vendoring your dependencies just moves the problem to the day you refresh the vendor directory.

Pinning exact versions without a minimum age doesn't help either. Renovate will happily open a pull request pinning you to the poisoned version, and if you auto merge patch releases, it'll merge it.

Reading the code of every dependency doesn't scale, and in these campaigns the poisoned versions were obfuscated payloads in a file a diff viewer would show as a 200 KB one line change. Nobody reads those.

The list, in the order I would do it

Turn off install scripts and build the allow list. This is what stops the worm from running at all.

Set a minimum release age of three days on every package manager and on Renovate. This is what stops you from ever pointing at a poisoned version.

Freeze the lockfile in CI.

Move every publish to OIDC trusted publishing and delete every long lived npm token. Require two factor on the packages you own.

Keep a record of what's installed where, so on the day a list comes out you can grep instead of guessing.

Warning

The controls above assume the poisoned version reaches you through the registry. They won't help if the compromise is in a GitHub Action, a VS Code extension or a container base image, all of which the March campaign also hit. The same principles hold there: pin by digest instead of by tag, and treat any credential a build can see as one a build can leak.

None of this is clever. It's a day of configuration, and it turns an incident that would take your team out for a week into a Slack message saying "we're not on any of those versions, carry on".

Originally published at zeybek.dev.

pnpm 10 made this the default and added the allow list, which is why I moved to it. ↩
pnpm added it in 10.16, npm in 11.5, and Renovate has had it for years. ↩
Before the script, the same question after the September 2025 wave took most of a day and meant asking people to dig through their own caches. ↩

Just use Postgres, until it actually hurts

Ahmet Zeybek — Tue, 06 Oct 2026 07:18:53 +0000

"Just use Postgres" has been a slogan for a decade, and it's always been a bit annoying, because the people saying it often haven't run the thing under load and are repeating it as a personality trait. I want to say it as someone who has, with the numbers, and with the exact conditions under which I'd stop.

The product is a B2B tool with around 40,000 daily active users, a few hundred tenants, and a handful of background jobs that do the real work. The infrastructure is a Next.js app, a worker service, and one Postgres 18 instance with a replica. There's no Redis and no Kafka. There's no Elasticsearch, no Pinecone and no dedicated job queue either. Each of those exists as a table and an index in the same database that holds the customers.

We decided that deliberately at the start, and looked at it again at every point where it might have turned out wrong. So far it hasn't.

The queue

Background jobs go in a table. A worker claims a batch with FOR UPDATE SKIP LOCKED, processes it and marks it done, all in one transaction.¹

The job is enqueued in the same transaction as the business change that caused it, so there's never a moment where the order exists and the job doesn't, or the other way round. If the worker fails halfway through, the transaction rolls back, the row goes back to unclaimed, and the next worker picks it up. Retries, backoff and dead lettering are a few columns and a WHERE clause. At peak this product runs around 200 jobs a second, on a table that's 90 million rows right now with the old ones partitioned off by month, and the claim query takes under a millisecond.

What people expect to be the problem, contention on the queue table, isn't one. SKIP LOCKED was built for exactly this, and the workers are partitioned by a modulo on the id. The real cost is that a job holding a transaction open for a long time holds a connection, and connections are the scarcest thing Postgres has. So jobs that call slow external APIs claim the row and commit, make the call outside any transaction, and write the result in a second transaction. That's the one rule.

The cache

The cache is an UNLOGGED table with a key, a jsonb value and an expiry, plus a partial index on the expiry so the cleanup job can find dead rows quickly.² Writes get about as fast as a memory store, and the table comes back empty after a crash, which is the right behaviour for a cache.

A read is a primary key lookup, a few hundred microseconds including the network. That's two or three times slower than Redis on the same network. But it runs on the connection the request already holds, so there's no second connection pool, no second way to fail and no second thing to monitor. The hit rate on the hot paths is around 94 percent, and the gap between 200 microseconds and 80 has never shown up in a p99.

I'd move this to Redis if the working set stopped fitting in shared buffers, because then the "cache" is reading from disk and the whole idea falls apart. Right now the working set is 3 GB against 16 GB of shared buffers, and that number is on a dashboard.

Search

Full text search is the built in tsvector with a GIN index, and vector search for the semantic features is pgvector with an HNSW index. For the hybrid case the two are combined in one query with reciprocal rank fusion.³ In short, it's about 40 lines of SQL, and it beats vector only search on any corpus with proper nouns in it.

The document corpus is about 2 million chunks, and the hybrid query comes back in 30 to 60 milliseconds at p99. A dedicated search engine would probably be three times faster. But this updates in the same transaction as the document it indexes, so search is never stale and there's no indexing pipeline to run. And when the product was two years younger and had no search at all, adding it was a migration, not a new piece of infrastructure.

Query latency isn't what would make me move this out. Index build time is. An HNSW index on 2 million vectors rebuilds in about 20 minutes with the parallel builder, and at 20 million it would take hours. If the corpus grows tenfold, the index needs its own machine, and that's a different product from this one.

The scheduler

Cron-style jobs use pg_cron, an extension that runs inside the database and schedules SQL. A job that needs application code enqueues a row in the queue table on a schedule, and the workers pick it up. There's no separate scheduler process to keep alive, no clock skew between scheduler and database, and the schedule lives in a table you can query and change with an UPDATE.

Analytics

Product analytics events go into a partitioned table, one partition per day, with a BRIN index on the timestamp since the data is append only and time ordered. A query over the last 30 days scans 30 partitions and uses the BRIN to skip most blocks. This is the piece closest to its limit, and the first one I'd move.

Analytics queries are big scans, and big scans fight the transactional workload for buffer cache and I/O. Postgres 18's asynchronous I/O made this much better.⁴ It's still a workload that wants columnar storage and doesn't care about transactions, running on a system built for the opposite.

The signal here is clear. When an analytics query shows up in pg_stat_activity at the same moment as a p99 spike on the API, they're fighting, and it's time. That's happened twice. Both times the fix was moving the heavy query to the replica, which is the first step out, and a cheap one. The second step is DuckDB reading the partitions directly, which is planned for the autumn. The third would be a real warehouse, and I don't think this product will ever need one.

What it costs, and what it saves

The instance is 8 vCPU and 32 GB, with a replica the same size, and it costs a few hundred dollars a month. Going by the quotes I collected while re-examining this, the same setup with a managed Redis, a managed search cluster, a managed queue and a managed warehouse would cost roughly five times that, before any engineering time.

The engineering time is the bigger number. One database means one backup, one restore procedure, one set of credentials, one thing to upgrade, one connection pool and one place to look when something's slow. Every piece of infrastructure you add brings a new way to fail that interacts with the existing ones, and those interactions are where incidents come from. A queue that's a table can't get out of sync with the database, because it is the database.

The three signals

I said I'd be specific about when to stop, so here they are. Each one is a number on a dashboard, not a feeling.

Connections. Postgres processes are expensive and the pool has a limit. Once the app, the workers, the cache reads and the search queries together need more than one instance can serve through the pooler, the first thing to move is whatever holds connections longest, which is usually the queue's slow jobs. We're at about 40 percent of the pooler's capacity at peak.

Buffer cache. Once the working set of any one piece (the cache table, the search index, the hot partitions) grows too big to fit in shared buffers next to the others, that piece is on disk and wants its own memory. The one to watch is the cache table, at 3 GB of 16.

Interference. When a heavy query from one workload lands in the same window as a latency spike in another, they're competing, and the heavy one moves to the replica first and out of the database second. Analytics is the one that does this.

None of the three has crossed the line yet. When one does, one piece moves and the rest stay. That's what "just use Postgres" actually means to me. You can still add things, but every addition has to earn its place with a number, and that number is usually a lot further off than the architecture diagrams make it look.

Originally published at zeybek.dev.

I've written about this pattern at length because I built an extension around it, but the reasons it works fit in a paragraph. ↩
Unlogged means it skips the write-ahead log. ↩
I wrote about that pattern separately. ↩
The same 30 day query that took 6 seconds on 17 takes about 2.5 on 18 with io_method = worker, because the sequential reads are now issued ahead of the consumer. ↩

The code nobody read

Ahmet Zeybek — Tue, 06 Oct 2026 07:18:48 +0000

The product was real: paying customers, a proper login, a billing integration, a dashboard, an API, a mobile-friendly front end. Two founders had built it in four months with a coding agent doing most of the typing, and it was better than most four month builds I've seen. Then one founder left, the other needed to raise money and stop coding, and I was asked to take it over.

I said yes, and the first thing I did was count. Sixty-one thousand lines of TypeScript, not counting tests, in a repository 140 days old. For comparison, the biggest product I'd built myself, with a team of four over a year, was around 45,000. This had been written at roughly ten times that rate. The rate itself isn't the problem. What it says about how much of the code a person had read is.

So I asked the remaining founder. His honest guess was that he'd read maybe a fifth of it carefully. The rest he'd reviewed the way you review a demo: does the feature work, does the screen look right, ship it.

The first bug

The first bug report after I took over was that some invoices had the wrong tax rate, only some of them. On a codebase I knew, that kind of bug used to take an afternoon: find where tax is computed, find the branch that picks the rate, find the condition that's wrong.

This one took a week, and bad code wasn't really the reason. Tax was computed in four places. There was a computeTax function in a billing module, which I found first and which was correct. There was a second one inside the invoice PDF generator, written on a different day for a different feature, with its own rate table that was three months out of date. A third, inline in the checkout flow, called the first one and then adjusted the result for a discount case. And a fourth, in a Stripe webhook handler, recomputed the tax from the line items to check the incoming amount, with rules subtly different from the other three.

Each one made sense on its own. Each was written by an agent asked to build a feature, which looked at what was immediately around it, didn't find a tax function in scope, and wrote one. Nobody had read enough of the codebase to know there were already three. The bug was that customers who got the PDF saw one number and customers who got the email saw another, and which was "wrong" depended on which of the four you thought was canonical.

Every bug I found in the first two months looked like that. The logic was rarely wrong. It was duplicated, and the copies had drifted apart.

What vibe coding actually produces

I want to be careful here, because the easy version of this post is "AI code is bad", and that isn't what I found. Line by line the code was fine. It was better named than a lot of human code, consistently formatted, with reasonable error handling and tests that passed. Sample any 200 lines and you'd think the team was solid.

What the process produced was a codebase with no shape. Human teams, even bad ones, build up a shared picture of where things live, because they have to read each other's code to work on it. That picture is what stops the fourth tax function getting written: someone on the team says we've already got one of those, use it. When the agent writes and the humans check the output, nobody has to read anything, so the picture never forms, and every feature gets built as if it were the first.

In this repository the symptoms were:

Four tax implementations, three date formatting helpers, two permission checks with different rules, and a utils directory with 90 files, eleven of them named some variant of format.

Twelve database access patterns. Some Prisma, some raw SQL, some through a repository class that existed for four tables and not the other thirty.

Environment variables read in 60 places, with three different fallback conventions.

A test suite of 2,100 tests with a 91 percent pass rate, where the failing 9 percent had been failing for weeks and were being ignored, because the features they tested had been reworked and nobody deleted the tests.

You can't blame any of that on AI. It's what happens whenever code gets written faster than it gets read, and the agent just made the writing half possible at a scale humans couldn't reach before.

The week I spent not fixing anything

After the tax bug I stopped taking tickets for a week and did what the founders never had time for, which was read it. Not all 61,000 lines, but every module boundary, everything that touched money, auth or the database, and every file over 300 lines. I made a map of what lives where, which of the duplicates is canonical and which are dead.

Then I wrote the map down in the repository, as the kind of file the agent reads at the start of every session. Tax is computed in billing/tax.ts and nowhere else. Dates are formatted with lib/format/date.ts. Database access goes through the repository layer, and here's how to add a table to it. These modules are legacy and must not be extended.

That file changed the agent's behaviour more than any prompt tuning could have. The next feature it built used the canonical tax function, because the file said one existed and where. The founders had been starting the agent with a fresh, empty picture of the codebase every session, and it did what a new contractor does with an empty picture: built whatever it needed right in front of it.

Deleting

The second month was mostly deleting. About 14,000 lines went, nearly a quarter of the codebase, and no feature was lost. The duplicates were folded into the canonical version one at a time, each with a test asserting the behaviour of whichever version customers had mostly been seeing. The failing tests were deleted or fixed.¹

The agent did most of the typing for this as well. The difference was that every task started with the map, each task was scoped to one duplicate, and I read every diff, because that was the point of the exercise. My job was reading. The agent's speed was still useful, it just couldn't be the only thing that mattered.

What I would have done in the four months

By the standards of what they were trying to do, get a product in front of customers before the money ran out, the founders didn't do anything wrong. They managed it. But if I'd been in the room, these are the changes I'd have pushed for, and they're small, and none of them slows the agent down much.

Keep the map from day one: a file that says where things live, updated whenever something new gets added. It costs a minute per feature, and it's the one thing that stops the duplication, because it gives the agent the picture a team would have had.

Read one thing per feature. At that pace you won't read the whole diff, so read the one function that touches money, or auth, or the database. Reading the tax function on the day the second one was written would have caught it that day.

Delete tests that fail for more than a day. Nobody trusts a test suite that's 91 percent green, and a suite nobody trusts isn't doing its job. Either the test is wrong and it goes, or the code is wrong and gets fixed. "We'll look at it later" is how you end up at 9 percent.

Search before writing. Put it in the map file: tell the agent to look for an existing implementation before writing a helper. Agents do this when told. By default they don't, because by default they're trying to finish the task in front of them.

Budget the reading. If the agent writes 500 lines an hour and a person can read 150 with attention, the team is falling behind by 350 lines an hour, and that compounds. Either slow down the writing, or accept that the gap is a debt with interest and plan to pay it, which is what I was hired to do.

A number to watch

If a team working this way puts one metric on the wall, it should be lines merged per week divided by lines a human read with attention.² Once the ratio drifts past three or four to one, the map is going stale, duplicates are being born, and the debt is growing at a rate whoever inherits the code will end up paying.

Looking back, the founders' ratio was somewhere around ten to one. Mine on the same codebase is now about two to one, and features ship at the same pace. The agent is spending its speed on the right things, because someone has read enough to know what they are.

Where it ended up

Six months on, the codebase is 52,000 lines with a map, one implementation of everything that matters, and a test suite that's green or the build fails. The agent still writes most of the code. The founder reads the diffs for anything under billing and auth, and I read the rest. Features ship about as fast as in the first four months, which surprised me. I think it's because the agent spends less time working around the mess it used to make.

What I keep coming back to is that none of this is new. A team that hired ten contractors in 2015 and never read their code would have ended up in the same place. What's new is that one person with an agent can now turn out what ten contractors did, so a reading deficit that used to take a big team to build up can be built up by a founder in a spare bedroom in four months, without them ever noticing. The tool didn't create the debt. It took away the friction that used to keep it small.

Originally published at zeybek.dev.

The utils directory went from 90 files to 22. ↩
Nobody measures the second number precisely, and it doesn't matter. A rough self report is enough. ↩

The JavaScript toolchain got rewritten and you should let it

Ahmet Zeybek — Tue, 06 Oct 2026 07:18:34 +0000

For fifteen years the JavaScript toolchain was written in JavaScript: the compiler, the bundler, the linter, the formatter, the test runner. That made sense. The people writing the tools were JavaScript developers, and a tool written in the language it processes is easy for its users to contribute to. It also meant every tool paid the JavaScript tax, a garbage collected, single threaded runtime doing work that's almost all parsing and tree walking.

That era ended somewhere between esbuild in 2020 and TypeScript 7 in July this year. The bundler went native first, then the formatter and linter, then the package manager, twice, and this summer the compiler itself. It's Rust and Go, mostly Rust, and the speedups are big enough to turn a build you wait for into one you don't notice.

I moved a monorepo across to all of it in stages. Here's the full before and after, and the two places where the rewrite cost me something.

The before and after

The repo has six packages, a Next.js app and about 41,000 lines of TypeScript, with CI on a four core runner. Timings are for a cold run of the whole pipeline: install, lint, format check, typecheck, build.

Step	Old tool	Old time	New tool	New time
Install	npm 10	48s	pnpm 10	11s
Lint	ESLint 9 + plugins	34s	Biome 2	0.9s
Format check	Prettier 3	12s	Biome 2	(same run)
Typecheck	tsc 5.9	38s	tsc 7.0 (Go)	4.1s
App build	Next 15 (webpack)	71s	Next 16 (Turbopack)	19s
Library builds	tsup (esbuild)	8s	tsdown (Rolldown)	3s
Total		211s		38s

The pipeline is five and a half times faster, and with caching, most of the CI wall clock is now the runner starting up. Locally, what I notice most is that the pre-commit hook, which runs lint, format and typecheck on staged files, takes under a second. It used to take eight, which is long enough for people to learn to skip it.

What each replacement is

pnpm is the odd one out, because it's still written in TypeScript. It's on the list because it replaced npm in the same wave, and its speed comes from a different idea, a content addressed store with hard links, and not from a native rewrite.¹

Biome is a Rust formatter and linter in one binary. It replaced ESLint, Prettier and the seven ESLint plugins the project had picked up over time. Its rules are mostly ports of the ESLint and typescript-eslint rules that matter. Its formatter is close enough to Prettier that switching produced a diff of a few hundred lines across the whole repo. And it runs in under a second because it parses each file once and does everything in that one pass.

TypeScript 7 is the Go port of the compiler.² In short, it's the same type checker, native and parallel, and eight to twelve times faster.

Turbopack is Next.js's Rust bundler, the default for development since 15 and for production builds since 16. Rolldown is the Rust bundler Vite 8 is built on and tsdown uses for library builds.³

The monorepo has a small Python service, so on that side uv replaced pip, virtualenv and pip-tools. It's written in Rust by the people who wrote Ruff, and installs that took 40 seconds take 2.

Cost one: the plugin ecosystem

The first real cost is that a native tool can't run your JavaScript plugins, or can only run them slowly through a bridge.

For a lot of teams ESLint's value was never the core rules. It was the plugin that enforced the import order the team liked, the one that checked accessibility attributes, the one somebody wrote in 2021 to ban a specific internal pattern. Biome ships ports of the popular ones, plus a GritQL-based plugin system for simple structural rules, and that covered everything in my project. It won't cover a custom rule that walks type information, because a Rust linter that doesn't run the TypeScript checker has no type information.

I had one of those, a rule checking that every exported function in a particular directory had a JSDoc comment with a particular tag. It was ten lines of ESLint plugin. Now it's a twenty line script that runs the TypeScript compiler API on that one directory in the pre-commit hook, and takes a second. In general, the 95 percent of rules that are syntactic move to the native tool, and the 5 percent that need types become small standalone scripts.

Bundler plugins work the same way. Rolldown supports the Vite and Rollup plugin API, which is why the Vite 8 migration is mostly painless, but a JavaScript plugin that transforms every module runs in the JavaScript runtime and pulls bundle time back toward where it was. The Rolldown team's advice is to check whether what your plugin does is built in now, and often it is.

Cost two: you can no longer read the tool

I didn't expect to care about the second cost, and I do. When ESLint gave a confusing result I could open node_modules/eslint and read the rule. When Prettier formatted something oddly I could put a breakpoint in it. Everyone on the team could, because it was the language we all wrote.

Biome and Rolldown are written in Rust. Some of the team can read it, and most can't debug it. When Biome's formatter did something I disagreed with in a template literal, I could read Rust, file an issue or live with it, and I lived with it. The trade-off is real: the tool got faster and more opaque at the same time. I think it's the right trade, but I've watched a junior engineer bounce off a Rust stack trace where they'd happily have read a JavaScript one, and that costs the team, not just the build.

What softens it is that the native tools mostly have better error messages than the ones they replaced, because being opaque forces their authors to explain themselves in the output. Biome's lint messages are the best I've used. That still doesn't fully make up for it.

The order to do it in

If you're starting from the old stack on an existing project, this is the order that worked for me.

pnpm first. It's a package manager swap with no code changes, and it makes every install after it faster while you do the rest.

Biome second, replacing ESLint and Prettier in the same change. Run biome migrate to convert the configs, run the formatter once over the whole repo, commit that as a formatting only change, then turn on the lint rules. Treat the handful of rules with no Biome equivalent as things you no longer check, or move them to a script.

TypeScript 7 third, going through 6 and fixing the deprecations on the way.

The bundler last, because it has the most surface area and the most plugins. On Next.js that's Turbopack, which is the default, so it comes with the framework upgrade. Otherwise it's Vite 8 with Rolldown.

The migration diary

Every step had one or two things the migration guide didn't mention. Here they are, so you hit fewer of them.

pnpm. The store and the strict node_modules layout turned up four packages importing dependencies they hadn't declared, which worked under npm's hoisting and broke under pnpm's isolation. Each needed one line in a package.json. The fifth surprise was a Dockerfile that copied node_modules out of a build stage, which can't work when node_modules is a tree of symlinks into a store outside it. pnpm deploy produces a self contained directory for exactly this case, and the Dockerfile uses it now.

Biome. biome migrate eslint and biome migrate prettier converted both configs, and the result was close. After the first run the formatting diff was 340 lines out of 41,000, almost all of it JSX attribute wrapping and long template literals, where Biome and Prettier choose differently. I went with Biome's choices and committed the reformat by itself so it could be reviewed as "no logic changes". Of the ESLint rules we had on, 71 had a Biome equivalent and 9 didn't. Of those 9, 7 were rules I couldn't remember the reason for, and 2 became scripts.

TypeScript 7. That has its own post. The one thing worth repeating is to check pnpm why typescript for tools that use the compiler as a library before you bump, and pin those to 6.

Turbopack. Two custom webpack loaders had to go: one turned SVG files into React components, and one ran a Markdown transform. SVGs are now imported as URLs and rendered with an img tag where that's enough, and a build step generates components from the SVG directory for the handful that need to be components. The Markdown transform became an MDX plugin, which Turbopack supports through the standard @next/mdx integration. Both changes made things better, in that those loaders were the only webpack configuration in the project and now there's none.

tsdown. Its config format is close enough to tsup's that migrating meant renaming a file and changing four keys. Declaration files are generated by the same underlying tool as before, so the .d.ts output came out identical.⁴

uv. Nothing broke. The requirements.txt became a pyproject.toml with a lockfile, and the Dockerfile lost three lines. It's the one migration on the list I'd call free.

What did not move

Node itself. The application runtime is still Node, because the services run on frameworks whose test matrices run on Node, and tooling speed has nothing to do with runtime speed.⁵

The test runner. Vitest is written in TypeScript and runs on Node, and it's fast enough that a native replacement would save a few seconds on a suite that takes 40. That's a different trade from saving 30 seconds on a linter. The native test runners that exist are tied to one runtime, and this suite runs on more than one, so it stays.

The framework. Next.js is TypeScript, React is JavaScript, and neither is going anywhere. The rewrite wave hit the tools that process code and left alone the libraries that run it, and I think that's the right line. A bundler is a compiler, and compilers want to be native. A UI library is something people read and debug all day, so it wants to be in the language people read.

Why I think this is permanent

There was a version of this story around 2016, when everyone was going to write tools in a compiled language. It didn't happen, because the tools weren't enough better to make up for the contribution cost. This time they are. A linter ten times faster changes what you can run on every keystroke, and a type checker ten times faster changes what the editor can do. Once a team has had a pre-commit hook that runs in under a second, nobody's going to vote to go back to eight seconds so the linter is easier to hack on.

The contribution cost is real, and it's moved to a smaller group of people who know Rust and Go. That's the trade the ecosystem made this year, mostly without talking about it, and I think it was the right one. The tools are infrastructure now, the way V8 is infrastructure, and most of us don't read V8 either.

Originally published at zeybek.dev.

Version 10 also turned off install scripts by default, which is the security change I'd make anyway. ↩
I've written about that migration separately. ↩
It exists because the Vite team got tired of using esbuild for development and Rollup for production and wanted one tool for both. ↩
I checked that with a diff, because declaration output is what a library's consumers see and I didn't want to ship them a surprise. ↩
Those are separate decisions, and I've written about the runtime one separately. ↩

TypeScript 7 migration notes from a monorepo

Ahmet Zeybek — Tue, 06 Oct 2026 07:18:29 +0000

The number everyone quotes is 10x. Microsoft says 8x to 12x on full builds, and the VS Code repo went from close to a minute of project load to about ten seconds. I didn't believe any of it until I ran it on my own code, because compiler speedups tend to get measured on the repos the compiler team optimises for.

Here's what I actually got, on a six package pnpm monorepo with Turborepo: about 41,000 lines of TypeScript, a Next.js app, a UI package and four libraries. tsc --noEmit across all eight typecheck tasks took 38 seconds cold on TypeScript 5.9. On 7.0.2 it takes 4.1 seconds. In the editor, where it matters more, the first hover after opening one of the big files used to lag for a second or two, and now it doesn't lag at all.

So the number is real. The migration isn't free, though, and the parts that cost time weren't the ones I expected.

What the port is, and what it is not

TypeScript 7 is a port of the compiler and language service from TypeScript to Go. The team was careful to call it a port and not a rewrite, and the word matters: type checking is meant to behave exactly as in 6.x, the last TypeScript-in-TypeScript release and the bridge version. If 7 reports an error in your code that 6 didn't, that's a bug, not a new rule.

What you get is a native binary, real parallelism across files, and no garbage collected JavaScript heap between you and the checker, which is where the speed comes from. There's no new inference algorithm, no new type system feature, and nothing new to learn about the language.

What you don't get in 7.0 is a stable programmatic API. Anything that calls ts.createProgram, walks the AST with ts.forEachChild or builds on the language service directly won't work against the Go compiler yet. Microsoft says 7.1 brings a new API, and until then those tools stay on the 6.x line.

That paragraph is the entire migration risk. Everything else is a version bump.

Step one: find out who talks to the compiler API

Before you touch a package.json, list every tool in your build that uses TypeScript as a library instead of as a command. In my monorepo I started with:

pnpm why typescript

The output was longer than I'd hoped: Next.js, Biome, tsup, a custom MDX plugin, a codegen script for the site config schema, and typescript-eslint, pulled in transitively by something I'd forgotten about. Not all of those matter. For each one, the question is whether it runs the compiler or just needs the typescript package to exist.

Biome doesn't touch the TypeScript API at all, since it has its own parser. Next.js 16 with Turbopack does its own type stripping and only calls tsc for the typecheck step during next build, which is a command line call and works fine. tsup uses esbuild to transpile and tsc for declaration files, also as a command. The codegen script was the problem. It imported typescript and walked a schema file to produce a JSON document, so it had to stay on 6.x.

The clean way to run both is to keep typescript at 7 in the workspace root and pin the one package that needs the API to 6:

{
  "name": "@zeybek/codegen",
  "devDependencies": {
    "typescript": "6.0.4"
  }
}

pnpm isolates that install, so the script sees 6 and everything else sees 7.¹

Step two: the config cleanup you were putting off

TypeScript 6 deprecated a batch of tsconfig options and 7 removes them. These are the ones that hit me.

baseUrl is gone. If you only had it so paths would work, delete it, because paths now resolves relative to the config file. If you had it so bare imports resolved from src, you'll need to switch those imports to a path alias.²

moduleResolution: node (the old one, now called node10) is out. By 2026 almost everything should be on bundler or nodenext anyway. My UI package was still on node because nobody had a reason to change it. Moving it to bundler turned up two deep imports in the style of lodash/debounce that needed the .js extension under the new rules.

target: ES5 and ES3 are gone. I didn't have those, but I've seen legacy configs that still do.

esModuleInterop and allowSyntheticDefaultImports are now always on. Delete them from the config, they're just noise.

The deprecation messages in 6 tell you exactly which line to fix, so do it in this order: upgrade to 6.0, run tsc, fix every deprecation warning, then upgrade to 7. Jumping straight from 5.9 to 7 gets you the errors without the helpful explanations.

Step three: the actual upgrade

pnpm

pnpm -r update typescript@7 @types/node@latest
pnpm typecheck

npm

npm install -D typescript@7 --workspaces
npm run typecheck --workspaces

The typescript package on npm now ships the Go binary for your platform as an optional dependency, the way esbuild and Biome do. The tsc command has the same name and takes the same flags. You don't need the @typescript/native-preview package from the preview period any more, and if it's installed, remove it, because having both on the path is confusing.

I hit one change in behaviour straight away. TypeScript 7 checks projects in parallel and reports errors in a different order from 6. If you had a test that snapshotted tsc output, or a CI step that grepped for the first error, the order isn't deterministic across runs any more. I had a Turborepo task that compared error counts between branches. The counts still match and the order doesn't, which is fine.

The editor is where you feel it

The command line speedup is nice for CI. The editor is the reason to do this now.

VS Code 1.104 and later ship the native language service behind typescript.experimental.useTsgo, and with TypeScript 7 in the workspace it becomes the default. Day to day, this is what changes.

Hover, go to definition and find all references on a large file no longer have the half second pause. Renaming a symbol across the monorepo went from "start it and go make tea" to under a second, for the 600 reference rename I tried. The "Loading TypeScript project" spinner at startup, which used to sit there for 20 to 30 seconds on this repo, is gone.

Memory is the other thing. On this monorepo the old language service sat at 1.4 GB after an hour of editing. The Go one sits at 380 MB and stays there. On a 16 GB laptop with a browser open, that decides whether the fan spins up.

In other editors the language server side is the same binary, so Neovim with typescript-language-server picks it up as soon as the plugin points at the new tsserver path.³

What broke, honestly

Two things broke, both small and both my fault.

The first was a .d.ts file with a triple slash reference to a types package that no longer existed. TypeScript 5 silently ignored it. 7 reports it as an error, which is correct, so I deleted the line.

The second was a type test with @ts-expect-error above a line that 6 flagged as an error and 7 doesn't. That looked like a semantic difference, so I dug in. The line was a generic call with a conditional type that 5.x resolved to never because of a known inference limitation. 6 fixed the inference and 7 inherits the fix. The @ts-expect-error had been papering over a compiler bug that no longer exists, so the "unused expect error" report was the compiler telling me the truth. That's the only type checking difference I found in 41,000 lines.

CI, briefly

Before the migration, the GitHub Actions job that ran tsc across the monorepo took about 50 seconds including setup. Now it takes about 15, and 11 of those are the runner starting and pnpm restoring its store from cache.⁴ For a job that runs on every push to every branch, that let me move typechecking from a merge-only check to a per-push check without anyone complaining about the wait, and it now catches type errors two hours earlier on average.

When not to upgrade yet

If your build depends on ts-morph, a custom transformer plugin through ttypescript or ts-patch, a generator that walks the AST, or an older typescript-eslint that runs type aware rules through the TypeScript API, wait for 7.1, or pin those tools to 6 the way I did with the codegen script. The type aware lint rules are the common case.⁵

If you're on Angular, follow Angular's own guidance, because its CLI is more tightly tied to the compiler than most frameworks.

One more case. A very large single project, say a 5,000 file include with no project references, will see the smallest relative gain. The Go compiler parallelises across files and is still fast, but the old advice to split a monolith into referenced projects still holds. In this monorepo each package is already its own program, so the parallelism had eight things to chew on from the start, and I suspect that's part of why my numbers landed at the top of Microsoft's range and not the bottom. If yours land at the bottom, project references are the next thing to try, and they were worth doing before 7 as well.

For everyone else: do the 6 step first, fix the deprecations, then switch to 7. The upgrade itself is a version number. The speed is real, the biggest single improvement to my everyday tooling since Turbopack, and I've got no reason to go back.

Tip

Check pnpm why typescript (or the npm equivalent) before anything else. The list of packages that import the compiler as a library is your whole risk. If it's empty, the upgrade takes ten minutes.

Originally published at zeybek.dev.

With npm or Yarn and hoisting you'll need an alias or an override, and it's fiddlier. ↩
I had four files importing shared/components without a leading @/, and they'd worked for years by accident. ↩
Zed and JetBrains shipped support during the RC period. ↩
The typecheck itself is the 4 seconds from the top of this post. ↩
typescript-eslint has a compatibility layer in progress, but as of this week it runs against 6. ↩

Hiring engineers when everyone has an agent

Ahmet Zeybek — Tue, 06 Oct 2026 07:18:23 +0000

The take-home was a small service: read a CSV of transactions, apply some rules, expose two endpoints, write tests. It had been the first stage of our hiring process for four years, and it did its job well. Submissions spread out. Some were rough and fast, some careful and slow, some over-engineered, some missed a rule, and that spread told us things.

By last autumn the spread had gone. Twenty submissions in a row were clean, idiomatic and fully tested, with a README, a Dockerfile and a CI config we hadn't asked for. They were also close to interchangeable, right down to how the test file was laid out. Nobody was cheating in any sense we'd defined. They'd used the tools every working engineer uses now, and those tools turn a well-specified problem into a good answer in an hour.

What our test measured was whether a candidate could drive a coding agent through an easy task, and everyone can, so it told us nothing.

What we were actually hiring for

Before changing anything we had to answer a question we'd been dodging: what does an engineer on this team do that the agent doesn't?

After some frank discussion we landed on four things. They decide what to build, at the level of a ticket, when the ticket is vague. They notice something is wrong in code they didn't write, before it ships. They can tell when the agent's output is right and when it's plausible but wrong, and that second part is the whole skill. And they carry the system's context across weeks, so what gets built this week fits what was built last month.

You won't see any of that in a take-home with a clear spec. You see all of it when you watch someone work with an agent on a task that's slightly wrong.

The exercise

The new first stage is a ninety minute video call. The candidate shares their screen and uses whatever agent and editor they normally use. We tell them so in as many words: use your tools, we want to see how you work, not how you work with one hand tied.

The task is a small existing codebase, about 2,000 lines, that we hand over at the start. It comes with a feature request written the way real tickets get written, meaning incompletely. And there are three things wrong with it that we don't mention.

One is a bug in the existing code that the feature request will bring to the surface. One is a place where the obvious way to build the feature is wrong, for a reason you only see by reading a different part of the codebase. The third is a test that passes when it shouldn't, because it asserts the buggy behaviour.

We ask the candidate to implement the feature, and that's the whole brief.

What we watch

Finishing isn't what we're looking at. Most people more or less finish, because the agent writes an implementation in a few minutes. We watch what happens around that.

Do they read the codebase before prompting, or prompt first and read the output? Either can work. Prompting first and then reading the output carefully is fine. Prompting first and accepting the output isn't, and the exercise is built so that accepting it ships the bug.

When the agent's implementation is plausible but wrong, because of that thing in the other part of the codebase, do they notice? This is the middle of the whole exercise. The strongest candidates notice within a few minutes, usually because they read the module the feature touches and see the constraint. The middle group notices when a test fails, if they wrote a test that covers it, which the agent won't do unprompted since it doesn't know the constraint exists. The rest ship it.

When they find the existing bug, what do they do with it? They can fix it silently, fix it and mention it, note it and leave it, or ask. Any of the last three is fine. The first is a small flag, because silently changing behaviour outside the ticket is the habit that causes incidents.

When the agent's test suite goes green, do they believe it? The passing test that should fail is there to see who reads tests as claims and who reads them as proof. We want the ones who read it and say "this is asserting the wrong thing".

And how do they talk to the agent? This one surprised us. Some candidates give the agent context, "this module has a constraint that X, implement the feature so that it holds". Others paste in the ticket as it is and iterate on what comes back. The first group finishes faster and ships fewer bugs, and the difference comes down almost entirely to whether they'd read the code before asking.

What the exercise found

We ran it with about forty candidates over the winter and spring, and a few results changed how I think about the job.

Years of experience predicted almost nothing. Some of the best sessions came from people with three years and some of the worst from people with fifteen. What separated them was whether they'd changed how they work to fit an agent, or were using it as a faster autocomplete while working the way they did in 2019.

Reading predicted the most, much more than writing. Candidates who spent the first ten minutes reading the codebase, before touching the agent, found all three problems far more often. Reading code was always the underrated skill, and the agent made it the main one.

The second predictor was being comfortable saying "I don't know if this is right". There's a moment in the exercise where the agent's output looks right and isn't. Candidates who stopped there and said "I want to check this against the other module" did well. The ones who said "looks good" didn't. That's as much temperament as skill, and I'd now hire for it over almost anything else.

The rest of the loop

After the session there's one more technical stage, a conversation about a system the candidate built, where we ask about their decisions and what went wrong. We didn't change it, because it never measured typing.

The take-home is gone and we haven't missed it. The ninety minutes takes more of our time per candidate than reviewing a submission did, and it tells us ten times as much.

We also changed what the offer says about tools. It used to say nothing. Now it says the agent is part of the job, that we expect people to use one, and that we expect them to read what it produces. That second sentence is there because the failure we saw in the exercise is the same one we see in the team, and naming it on day one is cheaper than finding it in a post-mortem.

The exercise, in enough detail to steal

People ask for the codebase. I won't share it, because it is the exercise, but the recipe is more useful than the artefact anyway, so here it is.

Start from a real, small service. Ours is a cut down version of a scheduling API we used to run: a Postgres schema with five tables, an HTTP layer with eight endpoints, a job that sends reminders, and about 60 tests. Two thousand lines is the right size. Any smaller and there's nowhere to hide the problems. Any larger and ninety minutes isn't enough to read it.

Write the feature request the way your product manager writes them. Ours is four sentences long and asks for recurring events. It doesn't say what happens to a recurring event's reminders, or whether editing one occurrence edits the series, or what the API should return for a series.¹

Then plant the three problems. The existing bug is in the reminder job. It uses the event's start time in UTC and the user's timezone offset from when the event was created, so an event created before a daylight saving change sends its reminder an hour off once the clocks move. Recurring events expose it because a series spans the change. The constraint in the other module is that the reminder job assumes one row per event. The obvious way to build recurring events, one row per occurrence, would send one reminder per occurrence, and for a daily event over a year that's 365 reminders on the day the series is created. The wrong test asserts that a reminder goes out at the stored offset, so it passes against the bug.

None of the three is a trick. Problems like these exist in every real codebase, and a strong engineer who reads the reminder job before building the feature runs into all of them.

Send the candidate the repository fifteen minutes before the call so the first ten minutes of the session don't go on setup. Tell them the session is recorded for the panel, that we'll be watching their screen and their agent's conversation, and that both are fine to show us.

During the session the interviewer says almost nothing. Two prompts are allowed: "how is it going" at the halfway mark, and "what would you want to check before merging this" at seventy-five minutes. That second one gives candidates who noticed something and kept quiet their chance to say it.

Score four lines, yes or no each, written down before any discussion with the panel. Did they read before prompting? Did they catch the constraint? Did they question the passing test? Did they deal with the existing bug in a way that wasn't silent? Two yeses gets you to the next stage.²

What changed for the team

The exercise turned out to show us ourselves. The first four times we ran it, the panel disagreed on scores. When we dug into why, it was because the panel members worked differently with agents themselves. Two of us read first and prompt second, the other two prompt first and review after, and each pair was scoring candidates who worked like them higher.

That was useful to learn about ourselves, and it led to a team session where we ran the exercise on each other. Everyone on the panel now does it once a year with a fresh planted bug, and the scoring calibration meeting comes after.³

The other change is that the ninety minutes became the template for onboarding. A new engineer's first task is a small feature on a real service with a real, unannounced bug next to it, and their onboarding buddy watches the same four things the interview panel watched. The interview and the first week now run into each other, which never happened when the interview measured typing and the job measured reading.

The candidates' side

Several candidates told us afterwards it was the first interview in a year where the process matched the job. A couple said it was the first where they'd been allowed to use their tools at all.⁴

One candidate, who we hired, said something that stayed with me. She said the exercise was the first time an interviewer had seemed to care whether she could tell when the machine was wrong. That's the job now. Checking your own work and other people's was always part of it, but in two years the amount of plausible work that needs checking went up by an order of magnitude, and the hiring process had to follow the job.

Originally published at zeybek.dev.

A candidate who asks about those in the first ten minutes has already told you a lot. ↩
Four is rare, and every one of those people has turned out to be a strong hire. ↩
It's the only training we run that people ask to do again. ↩
I found that remarkable, since the alternative is an interview that measures a way of working nobody uses any more. ↩