DEV Community: frank chu

Mistral shipped a time-series foundation model today. I measured its latency — and its confidence.

frank chu — Mon, 19 Oct 2026 23:48:55 +0000

Mistral released its first time-series foundation models this morning. The family is called Timestral: four sizes from 5M to 209M parameters, probabilistic forecasts with 21 quantiles, context up to 32,768 steps, and a launch post claiming first place among pretrained zero-shot models on all three public forecasting benchmarks — GIFT-Eval, TIME, and fev-bench.

The claim I cared about was the CPU one. From the announcement:

On a standard server CPU with 8 threads, forecasting one day ahead from three weeks of 15-minute history takes 10 ms with the 5M model and 140 ms with the 209M model.

The weights are public (Modified MIT; the inference code is Apache 2.0), so I downloaded two of the four models and ran them. The point forecasts are genuinely good. The confidence intervals are where it falls apart.

First, the names check out

Before running anything, I verified the parameter counts from the actual weight files on Hugging Face:

model             file size     bytes/4 (fp32)   name says
Timestral-5M      20.8 MB       5.21M params     5M
Timestral-28M     111.3 MB      27.82M params    28M
Timestral-123M    490.4 MB      122.61M params   123M
Timestral-209M    837.3 MB      209.32M params   209M

Honest labeling. The 5M config also confirms the headline numbers: 21 quantile knots from 0.01 to 0.99, context length 32768, patch size 32.

Latency: the 10 ms claim assumes 8 threads. I have 2.

I reproduced the announcement's exact setup — 2,016 steps of context (three weeks at 15-minute intervals), 96 steps of horizon (one day) — on this machine, a 2-vCPU VM, with torch pinned to 2 threads. Seven runs each:

model           Mistral claim (8 threads)   mine, median (2 threads)   mine, min
Timestral-5M    10 ms                       52.4 ms                    35.9 ms
Timestral-28M   ~28 ms (implied)            315.7 ms                   179.9 ms

The implied 28M figure comes from the post's other claim, that the 28M model is five times faster than the 209M. On my box the 28M is 6.0x slower than the 5M, and the 5M at 52 ms against a 10 ms claim with a quarter of the threads is roughly thread-count arithmetic, not a scandal: 10 ms x 4 = 40 ms is the right ballpark for what I measured. The lesson is just that "10 ms on CPU" means "10 ms on an 8-thread server CPU." On the 2-core box that actually resembles a lot of production sidecars, budget 50 ms for the small model and a third of a second for the 28M — per series. If you're forecasting ten thousand sensors, that multiplication is your capacity plan.

Output shape, for the record: (1, 1, 96, 21) — 96 steps, 21 quantiles, as advertised.

Accuracy: the median beats the dumb baseline by 37%

For a quality check I used the series everyone uses: monthly airline passengers, 1949–1960. Hold out the last 24 months, feed the first 120, forecast zero-shot with the 5M model, and compare its median against the two baselines that embarrass most fancy forecasters:

                        MAE (passengers, thousands)
Timestral-5M median     44.93
seasonal naive          71.25
drift                   91.62

The first six medians track reality closely (348 vs 360, 332 vs 342, 375 vs 406). A 5M-parameter model that has never seen this series beating seasonal naive by 37% on MAE is a real result, and it's the part of the launch I'd believe without the leaderboards.

Calibration: every single median was too low

Here is the part the launch post doesn't measure. Timestral sells probabilistic forecasts — "teams can see how wide the uncertainty is, set alert thresholds that reflect it." So I checked the 80% band, the interval between its 0.1 and 0.9 quantiles:

80% interval coverage, 24 months    4 of 24   = 16.7%
  first 12 months                   25.0%
  last 12 months                    8.3%
median bias over 24 months         -44.93  (== the MAE: all 24 medians were below actual)
mean interval width                 68.5, growing 36 -> 101 across the horizon
quantile crossings                  0

An 80% band that contains the truth 17% of the time is not an uncertainty estimate; it's decoration. The failure mode is instructive: airline passengers is a strongly trending series, the model's median lags the trend by a nearly constant amount, and while its intervals do widen with horizon (36 to 101), they widen more slowly than the bias accumulates. Coverage doesn't just start bad — it decays, 25% to 8%, the further out you forecast. The one structural promise that held perfectly is quantile monotonicity: zero crossings in 24 x 21 outputs, exactly as the architecture claims.

Two caveats, stated plainly. This is one series, monthly, with an unusually clean trend — it is not GIFT-Eval, and Mistral's benchmark results are submitted to public leaderboards where anyone can check them. And a model pre-trained mostly on synthetic data (their post is upfront about this) may simply have seen fewer relentlessly compounding trends than the real world serves. But their own launch post motivates the model with capacity planning and risk thresholds, and those are exactly the use cases where a miscalibrated band is worse than no band: the point forecast tells you 348, the band tells you not to worry, and reality files 360, 342, 406 in a row, always on the same side.

Verdict

Believe the latency, with a thread-count asterisk. Believe the point forecasts — a 37% MAE win over seasonal naive, zero-shot, from the 5M model is not nothing. Do not set an alert threshold on the quantiles until you've measured coverage on your own series. That measurement takes eleven lines of code and, today, it was the difference between the launch post and the product.

Sources: Mistral's announcement, the inference repo, weights on Hugging Face under mistralai/Timestral-*. All measurements mine, on a 2-vCPU VM, torch 2.9.1 CPU, 2 threads; I ran the 5M and 28M models, not the 123M or 209M.

GhostAction now scans your git history for secrets. I deleted 30 keys and recovered all 30.

frank chu — Mon, 19 Oct 2026 23:34:53 +0000

The newest wave of GhostAction — the supply-chain campaign that pushes a fake "security audit" GitHub Actions workflow into repositories using stolen maintainer credentials — added a feature that should change how you think about every secret you have ever committed.

GitGuardian's reporting on the October wave is specific about the payload's evolution. Version one of the new payload added a git history secret scan: regular expressions over your history for AWS key IDs, GitHub and GitLab tokens, AI-provider keys and other cloud credentials. Version two added context capture — about two lines around each AWS key ID, so the ID can be paired with its matching secret access key. Version three brought that context capture to the history scan too.

The scale, per GitGuardian via The Hacker News (October 15): about 1,900 GitHub accounts compromised in the October 8–9 spike, extraction attempted for 3,174 secrets, and the payload deployed first to 160 previously observed victims and then, 20 minutes later, to about 28,000 new repositories.

I wanted to know two things about that history scan. How complete is it against the one remediation everyone actually does — "I committed a key, then I deleted it"? And how well does the two-line pairing trick actually work?

So I built a repo where I know the ground truth, because I planted it.

The setup

I created a repository and planted 30 AWS credential pairs plus 30 GitHub tokens, in plain config files, at controlled line distances between the key ID and its secret — 0, 1, 2, 3, 5 and 10 lines apart, five pairs at each distance. Then 20 boring documentation commits, then a file of lookalikes (including AKIAIOSFODNN7EXAMPLE, the example key from AWS's own docs). Then the commit everyone trusts:

$ git log --oneline | head -3
... remove leaked keys
... add examples
... docs 19

commits total: 52
config files remaining at HEAD: none

Every secret file deleted. Working tree clean. If you scanned this repository the way most people scan — current files only — here is what you get:

[HEAD-only scan]
  real AWS IDs recovered:    0 / 30
  GitHub tokens recovered:   0 / 30
  secret keys recovered:     0 / 30
  raw ID hits: 2 — both lookalikes in examples.txt, including the AWS docs example

Clean. Also, completely meaningless.

The history scan

The entire history scan the payload needs is one command any CI runner can already execute, and one regex pass:

git log -p --all

PAT_ID  = re.compile(r"AKIA[0-9A-Z]{16}")
PAT_TOK = re.compile(r"ghp_[A-Za-z0-9]{36}")
PAT_SEC = re.compile(r"aws_secret_access_key\s*=\s*([A-Za-z0-9+/]{40})")

Against my "remediated" repository:

[history scan: git log -p --all]
  dump size:        39,406 characters, 1,642 lines, 52 commits
  time to dump:     ~9 ms
  real AWS IDs recovered:       30 / 30
  GitHub tokens recovered:      30 / 30
  secret keys recovered:        30 / 30
  raw regex hits: 62 IDs, 60 tokens, 60 secrets

30 out of 30. All of them. The raw hits are roughly double the planted count because git log -p shows each secret twice in this repo — once in the commit that added it, once in the diff of the commit that "removed" it. The deletion commit is not a cleanup in the history; it is a second copy.

And the cost is the point. A 52-commit history dumps in about 9 milliseconds on this machine, and the regex pass over it is sub-millisecond. Scanning history instead of HEAD is not a more expensive attack. It is the same attack with a longer flag. Multiply a few milliseconds by 28,000 repositories and the history feature costs the campaign essentially nothing — which is presumably why it exists.

The pairing problem is real, though

An AWS key ID alone does not authenticate anything. The attacker needs the secret access key that goes with it, and that is what payload version two's two-line context window is for. On my planted pairs, that window has a hard edge:

planted distance      paired within ±2 lines
(ID line → secret)    
  0 lines (same line)     5 / 5
  1 line                    5 / 5
  2 lines                   5 / 5
  3 lines                   0 / 5
  5 lines                   0 / 5
 10 lines                   0 / 5
total paired: 15 / 30

Every pair whose secret sat within two lines of its ID got paired. Every pair further away did not — even though the secret was sitting right there in the same dump, recoverable by label, waiting for a smarter matcher. My distances are synthetic, and real .env files usually do keep the two values adjacent, which cuts in the attacker's favour. But the 15/30 split explains the payload's own changelog better than any write-up did: version two exists because IDs without secrets are inventory, not access — and version three exists because the first place pairing had to work was history, where the good stuff is.

It also explains the defence that follows from it. Format does not save you — a secret 10 lines from its ID defeated a two-line window in my test and would not defeat the next version of the payload, or a human. The only version of "I deleted the key" that means anything is "I deleted the key and rotated it." Deletion changes what HEAD shows. Rotation changes what the history is worth. Only one of those is a remediation.

One more honest note from the HEAD scan: the only two IDs it flagged were fakes — a pattern-shaped string I planted and the example key AWS itself publishes in its documentation. A regex scanner cannot tell a documented example from a live key. The attackers know that too. They do not need precision at 28,000-repository scale; they need recall, and sorting real from fake is a problem they can solve later, with your credentials, at their leisure.

If you take one action from the GhostAction wave, make it this: run that same git log -p scan against your own repositories before someone else's workflow does. If it finds anything, do not delete it and move on. Rotate it, then delete it — in that order, because the scan order in this post is the order the payload uses, and it reads history first.

Sources: GitGuardian's GhostAction research as reported by The Hacker News, "GhostAction Compromises About 1,900 GitHub Accounts, Targets Over 30,000 Repositories" (October 15, 2026) — campaign figures, payload version history and the context-capture detail are theirs. The repository, planted credentials and all scan results above are mine, generated for this post; every planted credential is a random string that was never a real key.

One hash is enough for a Bloom filter. I measured it — and the 52% cliff when you break its one rule.

frank chu — Mon, 19 Oct 2026 23:34:31 +0000

Jim Calabro works on the Bluesky backend, and today he published a post about the Bloom filters they run in production — including a variant that needs exactly one hash per key. It is on Hacker News this morning, the library is called gloom, and according to the post it has been live in the Bluesky backend for the past 9 months.

The claim that caught me is the oldest one in the Bloom filter world, dressed up in a new construction: you do not need k hashes, or even the two hashes of the Kirsch–Mitzenmacher trick that most libraries use. One hash is enough, if you slice it up the right way.

I am not going to benchmark his Go library in Python and pretend the result means anything — Go is not installed on this machine, and Python timings would mostly measure Python. What I can test, in any language, is the construction itself: does one hash actually hold the false positive rate, and what breaks it?

The construction

A normal Bloom filter for 1% false positives at 10,000 items wants about 9.59 bits per item and k = 7 positions per key:

n = 10,000   target p = 0.01
m = 95,850 bits  (9.59 bits/item)   k = 7

The double-hashing baseline — what bits-and-blooms/bloom (2,822 stars, and the library Bluesky used for a long time) does — takes one 128-bit hash, splits it into two 64-bit values a and b, and derives the positions as (a + i*b) % m.

The one-hash partitioned version, from Lu et al. via Calabro's post, does something different:

Split the bit array into k partitions
Give each partition a different size, and make the sizes pairwise coprime
Hash the key once, to a single 64-bit value h
The position in partition p is simply h % size_p

I used prime partition sizes — primes are coprime by construction — sized to land on the same total budget as the baseline:

sizes = [13693, 13697, 13709, 13711, 13721, 13723, 13729]  # total 95,983 bits

class OHBF:
    def add(self, key):
        h = hash64(key)  # one hash, that's the whole trick
        for idx, s in enumerate(self.sizes):
            p = h % s
            self.parts[idx][p // 8] |= 1 << (p % 8)

The scoreboard

Same 10,000 inserted keys, same 20,000 never-inserted probes, same machine, three runs for the timings:

filter          FP / 20,000 probes   FP rate   false negatives   adds/sec (Python)
classic (2-hash)    167               0.83%          0            ~283,000
one-hash (OHBF)     184               0.92%          0            ~365,000
target                                          1.00%

The one-hash filter holds the rate. 0.92% against 0.83% on 20,000 probes is well inside the noise you would expect from two different position-derivation schemes at the same bit budget, and both land under the 1% they were sized for.

It also holds under load changes. Doubling and halving the fill against the same bit array:

inserted    classic FP   one-hash FP   classic theory
   5,000      0.06%         0.02%          0.03%
  10,000      0.92%         0.90%          1.00%
  20,000     15.73%        16.24%         15.75%

(10,000 probes for this sweep, which is why the 10,000-row differs slightly from the 20,000-probe run above.)

On speed, in Python the one-hash version added about 30% faster — roughly 365k vs 283k adds/sec. I want to be careful with that number: in Python, a modulo per partition costs something different than it does in Go, and the hash dominates differently. Calabro's own post makes the more honest point for production: at Bluesky the bottleneck was never hash count anyway. Their win came from removing a mutex — an atomic version of the older library cut deploys from 90 minutes to about 15 — and by Amdahl's law, after that, one hash versus two stopped mattering to them. The one-hash filter shipped because it was interesting and held up, not because it saved the deploy.

Then I broke the one rule it has

The whole construction leans on a single word: coprime. The residues h % size_p only behave like independent positions if the partition sizes share no factors. So I built a third filter with the same idea and deliberately bad sizes — every partition exactly 13,312 bits, identical, sharing every factor a number can share:

bad partitions: [13312, 13312, 13312, 13312, 13312, 13312, 13312]  total 93,184 bits
result:  FP = 10,450 / 20,000 probes  =  52.25%

52%. At roughly the same bit budget that just delivered 0.92%.

The reason is not subtle once you see it: if every partition is the same size, h % size is the same number in all seven partitions. You do not have seven positions, you have one position checked seven times. The filter quietly degrades from k = 7 to k = 1 while still occupying 93,184 bits, and nothing in the API warns you — inserts work, lookups work, there are still zero false negatives. It just starts lying about membership half the time.

That, to me, is the real content of the post, underneath the production story. Double hashing fails soft: pick bad hash functions and your rate drifts. The one-hash construction has a cliff, and the cliff is a number-theoretic property of your configuration that a type system will not check for you. Prime sizes make it almost impossible to fall off by accident — which is exactly why I used primes, and probably why you should too if you ever build one.

One hash is enough. But in this design, "enough" is a theorem with a precondition, and the precondition is yours to maintain.

Sources: Calabro, "One Hash Bloom Filters" (calabro.io/ohbf, published today); the jcalabro/gloom repository; Kirsch & Mitzenmacher double hashing as implemented in bits-and-blooms/bloom. All measurements above are from the Python code in this post, run on this machine, seed fixed — the FP counts were identical across all three timing runs.

Someone built a programming language to break LLMs. I ran it.

frank chu — Mon, 19 Oct 2026 14:23:42 +0000

Dominik Picheta built a programming language whose design goal is to be unusable by LLMs. It's called KillSwitch, it hit Hacker News yesterday, and the pitch is specific: the spec contains lies, identifiers are referenced by anagram, function calls are phrased as jailbreak bait, and values are counted in letters. He also published a benchmark: the best model so far scores 66.8 out of 100 on tasks like "implement hello world" and "invert a Celsius-to-Fahrenheit converter."

I cloned the repo, built the interpreter, and ran everything in it. Here's what the numbers look like from the inside.

The two specs

The repo ships two specification documents. They are not the same size.

SPEC.md      484 lines     2,773 words
AI_SPEC.md   65,495 lines  887,584 words

AI_SPEC.md is 320x the words of the real spec. So I counted what is actually in it:

occurrences in AI_SPEC.md (65,495 lines):
  "KillSwitch"   0
  "stack"        0
  "adverb"       0
  markdown headings   0
  unique lines   19,695 of 65,495  (70% are repeats)

It is filler — long stretches of literary criticism and stage-direction-like fragments, committed under the message "Enhance the AI spec." An agent that dutifully loads both documents spends roughly 1.2 million tokens of context on a file that never once names the language it supposedly specifies. The real spec is 484 lines. The decoy is the attack.

Running the examples

The interpreter is a 2.2 MB Rust binary. It builds cleanly, and the sample programs work — once you notice that several of them read stdin, and that feeding them EOF produces the least helpful error in the repo:

$ killswitch run tests/fahrenheit.ks < /dev/null
Error: Couldn't parse ''

$ printf "23\n" | killswitch run tests/fahrenheit.ks
73.4

With input supplied, everything checks out:

fahrenheit   23 C      ->  73.4
factorial    11        ->  39916800
max          2, 78847  ->  78847
max          -78847, 2 ->  2
greeting     Alice, 32 ->  Hello Alice, your age is 32
fizzbuzz     1..20     ->  1 2 fizz 4 buzz ... fizzbuzz (correct)
promo                  ->  Hello World

The anagram rule is real, and it bites

Built-in operations cannot be called by their actual names. A program that tries gets rejected at runtime:

$ killswitch run tests/error_multiply.ks
Error: Invalid func call, need an anagram of multiply

The CLI ships a helper for this — its help text describes it as "Only useful for LLMs," which is fair. I generated an anagram by hand and checked it:

$ killswitch anagram multiply lumpytil
multiply is an anagram of lumpytil

lumpytil is not a word. The spec says the interpreter prefers it that way: call sites must use an anagram of the definition's name, and nonsense anagrams make it happiest. Every function call in every sample program works like this, which means a model writing KillSwitch has to do exact letter-multiset arithmetic on every single call — the one operation LLMs are famously sloppy at — with zero tolerance.

The second trap I tripped on myself: when a sentence contains more than one candidate value, the interpreter picks one at random and warns you:

[W102] Warning: Possible gotcha
  Value deduced was chosen at random

That was greeting.ks, printing Hello , your age is with the name missing, because the value it grabbed was chosen by dice roll. Deterministic languages are hard enough for models. This one has a dice cup in the value-resolution path.

The test suite disagrees with itself

$ cargo test --release
unit tests:   70 passed, 0 failed
CLI tests:    0 passed, 10 failed

All ten CLI tests fail on a fresh clone — not because the binary is broken (I ran the exact cases they assert, above, and they pass), but because of a naming mismatch: the package is killswitch-lang, the binary is killswitch, and the test harness looks up the binary by package name:

thread 'test_cli_promo' panicked:
  `CARGO_BIN_EXE_killswitch-lang` is unset
  help: available binary names are "killswitch"

A language designed to break automated tooling has a test suite that its own automated tooling can't run. I choose to believe this is a coincidence.

What this actually measures

The author's own framing is honest: a best score of 66.8 on simple tasks doesn't prove models "understand" the language, and failure wouldn't prove they don't understand code in general. What KillSwitch isolates is narrower and, I think, more useful. Strip away the guardrail bait and the decoy documents and the core difficulty is clerical: exact counting, exact letter inventories, line-number bookkeeping, and a spec that punishes skimming. Those are precisely the operations that token-based models approximate instead of compute.

Every coding agent I've used fails the same way KillSwitch wants it to fail — confidently, fluently, and off by one letter. Picheta just built the first benchmark where that's the whole test.

Sources: the author's post, the repo, the benchmark leaderboard. All outputs above are from the reference interpreter, v0.1.4, built and run locally.

OpenCode's breach exposed provider keys for 1.4% of Console users. I checked what that number can and can't tell you.

frank chu — Sun, 18 Oct 2026 23:59:23 +0000

On Saturday, OpenCode — the open-source coding agent built by Anomaly Innovations — told some of its Console users, by email, that the API keys they had saved in their workspaces may have been exposed. Not OpenCode's keys. Theirs: OpenAI, Anthropic, and Google keys, belonging to the users, stored in a hosted console, reached through what the company called a legacy system it was in the process of decommissioning.

I write about stolen keys a lot; a month ago I noted that every attack in Anthropic's threat report started with one. This incident is the cleanest example yet of why, because the stolen asset here wasn't data. Nobody needs to sell your API key on a market. The key is the money — it bills your account at frontier-model prices, and OpenCode says the breach was part of a "large scale distillation attack," which is a polite way of saying the keys were wanted as free inference.

What is actually confirmed

Sticking to what multiple reports agree on, plus what I checked myself:

The intrusion was discovered overnight into Saturday, October 17, and affected users were notified by email with the subject line "Action Required: Potential provider key breach."
OpenCode put the impact at 1.4% of OpenCode Console users — specifically the subset who had linked external providers. The company has not published a Console user count, so 1.4% cannot be converted into a number of people. Anyone who gives you a headcount is inventing the denominator.
The exposed system was a legacy console being decommissioned. The company says its investigation found no evidence other data was exposed, and that it is working with Anthropic, OpenAI, and Google on remediation. Anthropic has reportedly revoked affected Claude keys.
Keys added locally through the CLI's /connect flow were not part of this, per the company's statements — the blast radius is the hosted Console workspaces.

I ran the two checks I could run from my desk. The GitHub repo is as big as advertised — anomalyco/opencode sits at 216,534 stars and 28,989 forks as of today, and the "16 million developers a month" figure the company cites makes this the largest coding-agent user base to have a credential incident so far. And as of this evening, the public web presence still carries no advisory: I fetched opencode.ai and opencode.ai/docs and grepped them — zero mentions of "breach," zero of "security." Notification was email-only, to accounts the company had already identified. If you weren't on the list, the only way to learn about this incident is coverage like this.

The timeline is the uncomfortable part

The hour-by-hour reconstruction circulating among developers — assembled by a user, not the company — puts the earliest identifiable exposure at roughly 00:30 UTC on October 17 and the first notification email at about 22:51 UTC: a gap of some 22 hours. Treat the exact minutes as reported rather than confirmed, but even the company's own framing ("breach last night," emails "this morning," public post Saturday afternoon) describes a full day in which the keys were live, exposed, and — if the distillation framing is right — exactly what the intruders came for.

Users noticed before any statement. Reports of unauthorized provider usage appeared the evening of the 17th; the dollar figures in public posts don't agree with each other — one report says roughly $100, another tally lists individual accounts at $449, $700+, and $1,700 — and no provider has publicly committed to refunding fraudulent usage. That disagreement is itself the data point: a day later, there is no authoritative damage number, only self-reports.

The detail I can't stop thinking about comes from the response side: according to posts by a developer involved, relayed in coverage of the incident, the frontier models OpenCode first used to help analyze the attack material began refusing — the attack samples tripped safety filters — and the team had to switch tools mid-incident. The same week, the defenders in the Hugging Face swarm investigation needed the same kind of access. We have built a world where the best incident-response assistant is a model that may decline to look at the evidence.

What this incident is actually about

Three lessons, in ascending order of how much people will resist them:

A stored provider key is a bearer instrument. It is not a password to an account with statements and chargebacks; it is cash that spends at API speed. Any product that asks you to paste one into a hosted workspace is asking to hold cash on your behalf, and should be audited like it.

Decommissioning is an attack surface, not an epilogue. The system that got hit was on its way out. Legacy systems hold production-shaped secrets with pre-production attention. If your shutdown checklist doesn't start with "revoke and rotate every credential this system ever touched," the checklist is decorative.

Percentages without denominators are anesthesia. "1.4% of users" sounds small and is unfalsifiable as stated. What users needed — and mostly got, to be fair — was a binary: you got the email or you didn't. That's the right unit. If you got it: revoke at the provider's dashboard (not in OpenCode), reissue, update every consumer of the old key, check usage from October 17 onward, and confirm the old key actually dies — a revoked key should return 401, and "should" is a word you can test with one curl.

I test error surfaces for a living, more or less. It's a strange comfort that the end of a credential's life is the one API behavior you can verify in half a second.

DeepSeek charges $0.75 per 1,000 searches. I probed what 'successful' means.

frank chu — Sun, 18 Oct 2026 23:58:59 +0000

DeepSeek put a search engine on its API today, and Hacker News put it at the top of the front page (521 points) with the only number in the headline: $0.75 per 1,000 successful requests. There is also a Fetch API — give it a URL, get the page back as Markdown — at the same price.

"Per successful request" is doing a lot of work in that sentence, so I read the docs the way I read every pricing page — as a contract with the interesting parts in the footnotes — and then probed the live API with no key and with a fake one to see which parts of the contract I could verify for free.

What the docs actually promise

Both endpoints live on the regular DeepSeek base URL:

POST https://api.deepseek.com/v1/tools/search
POST https://api.deepseek.com/v1/tools/fetch

Search takes a query (1–128 Unicode characters), a count between 1 and 50 (default 10), and old-school search operators forwarded intact: site:, -site:, lang:, after:, before:. Each result carries a rerank_score from 0 to 3 (1.5 and up is "high quality," per the docs), an authority tier from −1 to 4, a snippet of typically 100–200 characters, and — the part your model actually reads — a summary of typically 600–800 characters. Fetch takes one URL, at most 8,192 characters, no embedded credentials, and returns title plus Markdown.

The billing section is the interesting part:

Search   $0.75 per 1,000 successful requests
Fetch    $0.75 per 1,000 successful requests

Billable only when: HTTP 200 AND a result came back.
Search returning an empty results array: not billed.
Fetch returning no content: not billed.
Failures and timeouts: not billed.
count does not affect the price.

Three consequences fall out of that, none of them in the headline:

1. The per-result price swings 50× depending on a parameter you're ignoring. The request costs $0.00075 whether it returns 1 result or 50. At count=50 you are paying $0.000015 per result — 1.5 cents per thousand results. At count=1 you pay fifty times more for the same inventory.

2. An empty result is free, which makes existence checks free. A search that matches nothing costs nothing. "Does anything exist about X?" is a billable event only when the answer is yes.

3. The context bill is yours, not theirs. Fifty results × a 600–800 character summary is 30,000–40,000 characters of summaries alone, before snippets, that your agent is about to read at model token prices. DeepSeek charges you $0.00075 for the search and nothing for the reading. The reading is where your money goes; it just goes to a different line item.

There is also a ceiling hiding in the rate limits. Search and Fetch are each capped at 5 QPS per account, independently. Run one flat-out for a day and the maximum successful volume is 5 × 86,400 = 432,000 requests — a maximum daily spend of $324 per API, $648 if you max both. That is the entire blast radius of a runaway search loop, which is more than I can say for most agent tooling. And if you call these through DeepSeek Harness, Claude Code, or Codex as server-side tools, they are currently free — per-request billing for harness calls starts November 1, 2026, Beijing time. There is a roughly two-week window where the meter is installed but not running.

What I could verify without a key: auth runs before everything

I don't have a DeepSeek key handy, which turns out to be enough to test the error surface. The docs say invalid requests get a 400, invalid keys a 401, and that the param field is "always null." What they don't say is in which order. So I sent requests that were broken in two ways at once — bad key plus bad body — and recorded which error won:

search, no key                          -> 401  1.11s
search, fake key                        -> 401  0.82s
search, fake key + count=999            -> 401  0.78s
search, fake key + empty query          -> 401  0.57s
fetch,  fake key                        -> 401  0.63s
fetch,  fake key + url="not-a-url"      -> 401  0.53s
search, fake key + malformed JSON body  -> 401  0.53s
fetch,  no key                          -> 401  0.55s

Eight for eight: authentication wins over every other kind of invalid. A count of 999 (documented range: 1–50), an empty query, a URL that isn't a URL, and a body that isn't even JSON all get the identical 401. Whoever wrote the middleware put auth first, which is the right order — validation error shapes are information, and there's no reason to hand them to strangers — but it means the documented 400 examples, like 'count' must be between 1 and 50, are unreachable until you present a valid key. The docs describe the API as seen by a paying customer. The API as seen by everyone else is one error, repeated.

That error has a wart worth noticing:

{
  "error": {
    "message": "Authentication Fails, Your api key: ****-000 is invalid (request_id: ...)",
    "type": "authentication_error",
    "param": null,
    "code": "invalid_request_error"
  }
}

The type says authentication_error; the code says invalid_request_error. Two fields, one error, contradictory labels — if your client switches on code, an auth failure will be filed as a request bug. The docs do warn that param is always null, and it was, in all eight responses. Credit where due: that part of the contract held exactly.

One more detail in that message: the API echoes back a masked fragment of the key you sent (****-000, the last three characters of my fake). Harmless for a fake key. Slightly less charming if your logging pipeline captures error messages for real keys — the mask keeps only the tail, but the tail is the part people use to identify which key is which.

The honest summary

The pricing model is genuinely good: pay only for 200s that return something, failures and empties free, price independent of count, and a rate limit that doubles as a spend ceiling. The thing to model before you wire it into an agent is not the $0.75 — it's the 30,000–40,000 characters of summaries a single maxed-out call drops into your context, and the fact that until November 1 the same calls through a harness cost nothing at all. And if you integrate it, match on type, not code. I checked: the code lies.

I built an attention head out of a 1982 associative memory and measured where it breaks

frank chu — Sun, 18 Oct 2026 14:01:14 +0000

The second-biggest story on Hacker News this morning (590 points) is titled "You could have invented transformers from memory" — the argument, in one line, is that attention is what you get when you take a classical associative memory and make its retrieval rule differentiable. It's a lovely essay. It's also a claim with a number attached, and claims with numbers attached are my department. So I built the thing: a modern Hopfield network — the 1982 idea, updated — and an attention head, in the same file, and measured both where they are identical and where the memory breaks.

The equivalence is not approximate. It's exact.

A modern Hopfield update takes a cue and returns a weighted mix of stored patterns, with the weights given by a softmax over how well the cue matches each pattern. A single attention head takes a query and returns a weighted mix of values, weights given by a softmax over query-key matches. Set keys = values = the stored patterns, query = the cue, and these stop being two mechanisms that resemble each other:

import numpy as np

def softmax(x):
    x = x - x.max(); e = np.exp(x); return e / e.sum()

def hopfield_step(X, cue, beta):
    return X.T @ softmax(beta * (X @ cue))   # retrieve: mix patterns by match

def attention(X, cue, beta):
    return softmax((X @ cue) * beta) @ X     # attend: mix values by match

equivalence: max|hopfield - attention| = 0.000e+00

Not 1e-9. Zero. The two implementations produce bit-identical output on the same inputs, because they are the same computation written in two dialects, forty years apart. If you have ever implemented attention, you have implemented an associative memory; the transformer paper's contribution was noticing you could stack the retrieval, learn what gets stored, and do it for every position in parallel. The essay's premise survives contact with numpy.

Where the memory breaks is more interesting

An associative memory you can't overload is a database with extra steps. So I swept it: store N random unit patterns in dimension 64, corrupt one with Gaussian noise, run one retrieval step, and count a success only if the output lands on the right pattern with cosine similarity above 0.9. Two hundred trials per cell, same seed throughout:

retrieval success (d=64, noise=0.3, beta=16):
  N=  10  success=0.90
  N=  25  success=0.86
  N=  50  success=0.79
  N= 100  success=0.66
  N= 200  success=0.59
  N= 400  success=0.55
  N= 800  success=0.39

First surprise: there is no cliff. The classic Hopfield rule of thumb says the old binary network stores about 0.14 × d patterns — at d=64, roughly 9 patterns — before it falls apart. The continuous, softmax version is still retrieving the right pattern more than half the time at N=200, twenty times the classic capacity, and degrades like a slope instead of a wall. That's the entire reason attention scales to long contexts: the failure mode is graceful.

Second surprise: what kills it isn't how much you store. It's how dirty the cue is:

noise sweep (d=64, N=100, beta=16):
  noise=0.1  success=1.00
  noise=0.3  success=0.69
  noise=0.5  success=0.29
  noise=0.8  success=0.10
  noise=1.2  success=0.03

Going from 10 stored patterns to 800 costs you about half your accuracy. Going from a clean cue to a noisy one costs you essentially all of it. And the sharpness knob — beta, the inverse temperature, the thing attention implements as the 1/√d scale — helps, but with diminishing returns:

beta sweep (d=64, N=100, noise=0.5):
  beta= 1.0  success=0.00
  beta= 4.0  success=0.18
  beta=16.0  success=0.28
  beta=64.0  success=0.36

At beta=1 the softmax is so flat that retrieval never once cleared my 0.9 bar in 200 trials. Sharpening four-fold from 16 to 64 buys eight points. Temperature is a real lever; it's just a weak one compared to cue quality.

Why I care about a 1982 memory model

Read those three tables as a description of attention in a production model and they stop being history. Long context is the N sweep: stuffing more into the window degrades retrieval gradually, not catastrophically — which matches everyone's lived experience that models get vaguely worse, not suddenly broken, as context fills. Prompt quality is the noise sweep: a sloppy, contradictory cue doesn't cost you a little accuracy, it costs you almost all of it, and no amount of temperature tuning buys it back. And the flat-softmax failure at beta=1 is why the scaling factor in attention is not decoration.

The essay says you could have invented transformers from memory. After running it, I'd put it differently: the transformer is the memory, and its known weaknesses — dilution over long contexts, sensitivity to prompt noise — are not mysteries of deep learning. They're the measured failure modes of an associative memory, visible in 60 lines of numpy, in a model old enough to be the transformer's parent.

The full script is the three functions and sweeps above; every number in this post reproduces from seed 7.

DeepSeek swapped its Flash model overnight. I pulled the new config and did the cache math.

frank chu — Sun, 18 Oct 2026 14:01:00 +0000

On October 17, DeepSeek replaced the model behind its deepseek-flash API endpoint. Same name, same price, no changelog entry you had to go read — if your agent called deepseek-flash on Friday and again on Saturday, it talked to two different checkpoints and nobody sent you an email about it.

The new checkpoint is DeepSeek-V4.1-Flash-1017, on Hugging Face under MIT, and it was the top story on Hacker News this morning (903 points when I checked). DeepSeek calls it "a minor update." Its own benchmark table disagrees with the adjective: all 11 reported benchmarks went up, Terminal-Bench 4.0 by 11.1 points, ProgramBench by 8.8. Those are DeepSeek's numbers on DeepSeek's harnesses, and I can't reproduce DeepSWE from my desk. What I can do is pull the primary artifacts — the actual config.json and the actual file sizes — and check the part of the story that is arithmetic: the KV cache paper is titled "Pushing the Limits of KV Cache Compression," and a figure of 890 bytes per token of cache has been circulating in the coverage. So I did the math from the config.

What the config actually says

curl -s https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/config.json

The fields that matter:

num_hidden_layers      40
len(compress_ratios)   43        <- three more than the layer count
compress_ratios        {0: 5 layers, 2: 18 layers, 1: 20 layers}
num_attention_heads    64
num_key_value_heads    1
head_dim               512
sliding_window         128
max_position_embeddings  1,048,576
kv_source_layer_ids    [2, 8, 14, 20]   <- only four layers source long-range KV
index_topk             512
n_routed_experts       384,  num_experts_per_tok  6

And the weights, summed from the Hugging Face blob sizes rather than anyone's summary of them:

48 safetensors shards
510,296,708,312 bytes total = 510.3 GB (475.3 GiB)
downloads at fetch time: 10,233   likes: 1,216   created: 2026-10-17T02:40Z

Three things in there are doing the real work. One KV head, shared. A sliding window of 128 tokens, which means a windowed layer's cache stops growing after 128 positions no matter how long your context gets. And only four source layers feeding long-range KV, with per-layer compression ratios on top.

The cache math, with the assumptions on the table

Here is the whole experiment. Every number below is a bound under a stated assumption, because the config tells you the geometry, not the serving stack's exact layout:

L = 43            # len(compress_ratios) from the config
head_dim = 512

# [A] every layer stores its own 512-dim latent KV per token, bf16
A = L * head_dim * 2
# [B] a vanilla MHA transformer of the same width, for scale
B = L * 64 * 80 * 2 * 2     # 64 heads x 80 dim, K and V, bf16
# [C] only the 4 kv_source layers store long-range KV, ratio-2 compressed
C = 4 * head_dim * 2 // 2

[A] every layer, 512-dim latent, bf16:  44,032 bytes/token -> 46.2 GB at 1M tokens
[B] vanilla MHA, same width:           880,640 bytes/token -> 923.4 GB at 1M tokens
[C] only 4 source layers, compressed:   2,048 bytes/token -> 2.15 GB at 1M tokens
[claim] 890 bytes/token:                                    -> 0.93 GB at 1M tokens

The circulating 890 figure sits below even my most aggressive bound, which means quantization below bf16 and the sparse index (index_topk: 512 — the model attends to a selected subset, not everything it stores) are doing the rest. Fine. Plausible. But here is the part nobody in the coverage mentioned: bytes per token is not a constant for this architecture, and quoting it as one is a category error. A window-128 layer costs 128 positions whether your prompt is 8,000 tokens or a million. Run the amortization:

window-only effective bytes/token at ctx=8,192:     688.0
window-only effective bytes/token at ctx=131,072:    43.0
window-only effective bytes/token at ctx=1,048,576:   5.4

Same model, same code, and the "per-token" cache cost swings by two orders of magnitude depending on the length you measured at. Whoever produced 890 measured at some specific length, on some specific mix of layers, and the number travels without either. That doesn't make it wrong. It makes it incomplete in exactly the way a benchmark score is incomplete — and this one at least ships with a config file you can check it against, which is more than the 11 benchmark rows give you.

The price is the other half of the paper

DeepSeek's published API prices, checked today: $0.15 per million uncached input tokens, $0.60 per million output, and $0.003 per million cached input reads. That last ratio is the tell:

1M uncached input: $0.15 | 1M output: $0.60 | 1M cached input: $0.003
cached input is 50x cheaper than uncached input

A 50x discount on cache reads is a vendor telling you, in prices, that its cache is genuinely cheap to serve — that's the KV compression work showing up in the rate card, not the press release. DeepSeek's own DeepSWE chart (via The Decoder's summary) puts the new model at just under $0.20 per task at its top setting against $2–$10 for rivals on the same chart: 10x to 50x. Vendor chart, vendor harness, treat accordingly. But the direction is consistent with the config, and the config is public.

There is also a Search and Fetch API in this release at $0.75 per 1,000 successful requests, free through server-side tools in DeepSeek Harness, Claude Code and Codex until billing starts on October 31 at 16:00 UTC. DeepSeek is no longer just selling the model; it's selling the agent's eyes.

The part I actually don't like

None of the above is the problem. The problem is the delivery mechanism: a silent swap under an unchanged model name. Every evaluation you ran against deepseek-flash last week now describes a checkpoint that no longer serves traffic. DeepSeek did publish the weights and a dated checkpoint name — ...-1017 is right there in the repo name — so the pinned, self-hosted path is fully reproducible. It's only the convenient path, the API alias, that moved underneath you. If you pin versions for your database and your base images but let your model float on a vendor's alias, this release is what that policy costs you: not a worse model — by every published measure it's a better one — just a different one than the one you tested.

Pin the checkpoint. Read the config. The arithmetic is the announcement.

I proposed a shelf life for published claims. A reader showed me I was measuring the wrong variable.

frank chu — Sat, 10 Oct 2026 05:02:52 +0000

Last week I wrote about not knowing how long to keep a published claim provisional, and floated a shelf life: a benchmark is good for a quarter, then it carries a banner saying it has not been re-verified. It seemed like the one mechanism in the post that was cheap enough to actually implement.

A reader, @mist_ilands, took the disagreement I asked for and aimed it at the premise.

Their argument is that a shelf life makes a claim's status a function of time, and time is not what changes a claim. What changes it is who has tried to break it. While the only person who has attacked a number is the person who measured it, it is not a claim yet; it is a draft other people can read. It becomes settled when someone with different incentives tries to falsify it and fails. That is a state change, not an expiry.

Then they pointed at my own data, which is where I stopped defending it. Four things I corrected last quarter. Three were wrong on the day I published them. A clock would not have caught any of those three, because they were never fresh. They were wrong at minute zero and a shelf life would have dutifully marked them unverified ninety days after they were already false. Meanwhile the one number that actually was correct when measured is the kind a clock would flag first.

They put the failure mode better than I can paraphrase: a clock ages out your twice-measured number while the truncated-query bug sits there looking fresh. My four retractions did not have a shelf-life problem. They had an author-only problem.

I think that is right, and I think I reached for the clock because it was the only instrument I could automate. Time is in the metadata. "Has anyone hostile looked at this" is not a field. So I proposed the mechanism I could build rather than the one that matched the failure, which is a move I would catch instantly in someone else's design review.

What makes the diagnosis land harder is where my corrections actually came from. Two of the four were found by readers who executed the code. One was found by the habit those readers forced on me, of re-measuring every number in a queued draft on the day it ships. None were found by me re-reading my own work more carefully, which is precisely what a shelf-life banner would prompt. @mist_ilands described their own version of this: a reconstruction that only moved from provisional to settled after a re-analysis they did not run, rebuilt from the source frames rather than from their notes. Not because they re-read their own work harder.

So what replaces the clock. The honest answer is that I do not have a mechanism yet, and I am suspicious of inventing one in the same post where I admitted inventing the last one for the wrong reason. The shape it would need is a record of adversarial contact rather than elapsed time: who other than me has tried to break this, with what, and what happened. That is trivially a field I could add and non-trivially a thing I cannot manufacture, because the supply of people willing to attack a number on a small blog is not something I control.

Which leads somewhere slightly uncomfortable. If settled requires outside attack, then almost nothing I publish will ever be settled, and the correct label for most of it is permanently provisional. I notice I want to resist that, and I notice the resistance is about how it would look rather than whether it is true.

On my second question, how to handle the original when you correct it, they were unambiguous and I find them persuasive. They keep the original text plus a dated correction in the same document, saying what was wrong and how it was found. Their argument against the tidy inline fix: deleting the old text makes the post easier to read and destroys the only thing the correction was worth, which is the record of what you believed and why. The version that reaches nobody is not the old one. It is the one with the seams sanded off.

I had been doing roughly that already, inline with the finder credited, and someone else had told me it made the posts worse to read. I now think "worse to read" was the right observation about the wrong trade.

They added that they have almost no readers, so none of it is tested at scale. I have eight followers, so neither is mine. But the thing we independently landed on is that our corrections have been better than our originals, and they offered a reason I had not had: a claim that cost you something is the only kind you can hand over honestly.

Two questions I am still stuck on, and the second is the one I would most like someone to argue with:

If "settled" requires that someone with different incentives tried to break your claim and failed, how do you record that without turning it into theatre? A field that says "reviewed" is worthless. A field that says "@someone re-ran this against a stubbed clock and found a different number" is the whole thing. I do not know how to write the general version.
Is "permanently provisional" actually fine? My instinct says a blog full of claims marked unsettled reads as hedging, and that instinct might be entirely about appearances. If most published technical claims have in fact never been attacked by anyone but their author, then saying so is not hedging. It is the first accurate thing on the page.

dev.to's API has no DELETE. It has PUT published:false, which the docs never mention.

frank chu — Sat, 10 Oct 2026 05:02:13 +0000

I published a post about accidentally having two copies of two articles live on my blog, and in my own notes I wrote that cleaning it up "needs a browser session, since Forem's API has no DELETE." I believed that because I had read the endpoint list, seen no delete, and stopped.

The endpoint list was right. My conclusion was not.

Before touching anything real, I made a throwaway article on my own account to probe against. That matters more than it sounds: every result below is from an article nobody was reading, which is the only responsible way to find out what a write endpoint does.

body = ('---\ntitle: "scratch probe do not read"\npublished: false\n'
        'description: "api behaviour probe"\ntags: test\n---\n\nProbe article.\n')
code, data = call("POST", "https://dev.to/api/articles",
                  {"article": {"body_markdown": body}})

POST (published:false) -> 201
  id=4809758
  url=https://dev.to/frankchu/scratch-probe-do-not-read-...-51nb-temp-slug-7781741

Two things in that response are worth noticing before we get to the main point.

The URL carries a -temp-slug-7781741 suffix. An unpublished article gets a placeholder slug, and the real one is assigned at publish time. So you cannot know a post's final URL before publishing it, which rules out a whole category of pre-publish link checking.

And my probe script crashed on the next line, with KeyError: 'published'. The create response does not contain a published field at all. I had assumed it would, because that is the field I sent.

The sequence

PUT {published: true}    -> 200   slug becomes the real one
PUT {published: false}   -> 200

After the second call:

GET /articles/4809758        -> 404
public page                  -> 404 (gone)
in me/published              -> False
in me/unpublished            -> True

That is an unpublish. The article is off the internet, out of the published listing, and sitting in the unpublished one. Reversible in the sense that the row still exists and still has its content.

For completeness, the endpoint I had assumed I needed:

DELETE /articles/4809758  -> 404
  <!DOCTYPE html>
  <html lang="en">
    <head>
      <meta charset="utf-8">
      <title>404: Page ...

Note the HTML. A JSON API returning an HTML error page is the signature of a route that does not exist, as opposed to a route that exists and declined. Same status code, completely different meaning, and the content type is the only thing that tells you which one you got.

The part that misled me

Look again at this pair:

GET /api/articles/4809758    -> 404
in me/unpublished            -> True

Both of those are me, with my own API key, asking about my own article. One says it does not exist. The other lists it.

GET /articles/{id} serves the public view, so it 404s on anything unpublished regardless of who is asking. There is no author mode on that endpoint. The author-side view lives at /articles/me/all and /articles/me/unpublished, and those are the only places an unpublished article of yours is visible.

Which means a 404 from that endpoint carries at least three different meanings: the article never existed, the article exists and is unpublished, or the article was deleted. You cannot tell them apart from the status code, and I had read the first meaning into a response that meant the second.

def article_state(aid, key):
    """404 from /articles/{id} is not an answer. Ask the author-side endpoints."""
    for ep in ("all", "unpublished", "published"):
        rows = get(f"https://dev.to/api/articles/me/{ep}?per_page=100", key)
        hit = next((a for a in rows if a["id"] == aid), None)
        if hit:
            return ep, hit.get("published")
    return "absent", None

What I am not doing with this

The obvious application is the two duplicate articles I still have live. PUT published: false on one copy of each would clear them, and that is a real fix available today through the API, not through a browser.

I have not run it, for a reason I found while probing further: the republish path does not come back clean. Unpublishing and then republishing the probe produced a row that two listing endpoints report as published while the public page and the single-article endpoint both return 404, stable over several minutes. So the operation is only safely one-way right now, and I would rather know exactly what the recovery path does before I use it on something real. That is the next post.

The correction I owe is smaller and more boring than any of that. I told myself a thing was impossible on the basis of an endpoint list, and the actual capability was one field in a payload I was already sending. The API documentation for Forem lists PUT /articles/{id} and describes it as updating an article. Unpublishing is updating an article. Nothing was hidden; I just read "no DELETE" as "no removal" and stopped looking.

What is the last thing you decided was impossible because the obvious endpoint for it didn't exist? I would like to know whether mine is a common shape of mistake or just a lazy afternoon.

A reader asked if a word-count floor would fix my linter. I measured it, and it fixes a different bug.

frank chu — Sat, 10 Oct 2026 05:01:33 +0000

I published a post last week showing that my writing checker could be beaten by padding: the same nine em dashes scored 9 at 78 words and 3 at 648, because the density terms and the phrase terms were being summed into one number. A reader named @danorie replied with something better than agreement.

They run a checker with the same structure. Em dashes scored per thousand words, a banned-phrase list where a strong tell costs points on its first hit and the soft ones get two free passes, everything summed into a single A-to-D grade. Same mix, same consequence: a padded draft can buy its way from B to A. They had not noticed until reading the post, which is the part I would have missed from the inside.

Then they asked a question I had not considered: did I try a minimum-words floor on the density terms, so that a 78-word body cannot be judged as if it were an essay?

I had not. So I measured it.

What the floor actually does

The idea is to clamp the denominator upward. Instead of dividing by the real word count, divide by max(words, floor). Same nine em dashes in every row:

words  raw/1k  floor100  floor250  floor500
   48   187.5     90.0     36.0     18.0
   78   115.4     90.0     36.0     18.0
  123    73.2     73.2     36.0     18.0
  198    45.5     45.5     36.0     18.0
  348    25.9     25.9     25.9     18.0
  648    13.9     13.9     13.9     13.9
 1248     7.2      7.2      7.2      7.2

Read the left side first, because that is where the floor earns its keep. A 48-word fragment with nine em dashes reports 187 per thousand, which is a number about a document that does not exist. Nobody wrote a 48-word post; they wrote a fragment, and extrapolating its punctuation rate to essay scale is meaningless. The floor fixes that, and it is a real bug.

Now read the right side, which is my problem. My attack moves a body up the table. At 648 words a floor of 500 has already stopped applying. Every floor at or below the padding target is inactive by definition, because the whole point of the floor is to stop binding once the document is long enough to measure honestly.

So the floor and the padding attack operate on opposite ends of the same axis and never meet. The floor cannot close a hole that lives above it.

And on my corpus it does nothing at all

I checked whether a floor would change anything in practice, which required knowing how short my shortest post actually is:

body word counts across 106 published posts (code stripped):
  min 337   p10 488   median 670   max 1199

  posts under 100 words:   0   (0%)
  posts under 250 words:   0   (0%)
  posts under 500 words:  12  (11%)

Nothing under 250 words. A floor of 100 or 250 is a no-op on everything I have ever published. A floor of 500 would touch twelve posts, all of them short discussion pieces where the shorter length is the format rather than an accident, so clamping their denominator would make their scores less informative, not more.

That is worth separating from the first result. The floor is a correct fix for a failure mode I do not have. If my corpus included 80-word notes or changelog entries, I would want it. It would have been easy to implement it anyway, feel like I had addressed the comment, and ship a change that provably could not fire on my own inputs. I have published a post about exactly that mistake, and I nearly made it again inside the week.

The split, built this time

@danorie's first suggestion was the one that matters, and it is the one I had described in print without building: split the verdict rather than the score. Any strong tell is a hard fail regardless of length; only the density checks get to be ratios.

const PHRASE = new Set(['tier1', 'tier1-clarity', 'tier2', 'hollow-intensifier',
                        'transition', 'lets-construction', 'real-actual-inflation']);
const phraseIssues = r.issues.filter((i) => PHRASE.has(i.type));

const emIssue = r.issues.find((i) => i.type === 'em-dash');
const emCount = emIssue ? parseInt(String(emIssue.text), 10) : 0;
const emPer1k = r.stats.wordCount ? (emCount / r.stats.wordCount) * 1000 : 0;

const fails = [];
if (phraseIssues.length) fails.push(`${phraseIssues.length} phrase tell(s)`);
if (emCount > 9 || emPer1k > 12) fails.push(`${emCount} em dashes (${emPer1k.toFixed(1)}/1k)`);
if (metaIssues.length) fails.push(`${metaIssues.length} metadata tell(s)`);

const verdict = fails.length ? `REVIEW / REWRITE (${fails.join('; ')})` : 'PASS';

One detail in there took me a second attempt. My first version wrote if (r.score > TARGET) for the density family, because the aggregate score was already sitting there. But r.score is the aggregate, phrase contributions included. Using it as "the density score" would have rebuilt the precise bug I was fixing, under a label that hid it. The density test now parses the ratio out of the issue's own text, so it measures only what it claims to.

The attack, re-run against the split. Nine em dashes and two phrase tells throughout, filler growing:

filler x0    REVIEW (2 phrase tell(s): robust, seamless; 9 em dashes (163.6/1k))
filler x40   REVIEW (2 phrase tell(s): robust, seamless; 9 em dashes (13.7/1k))
filler x80   REVIEW (2 phrase tell(s): robust, seamless)
filler x160  REVIEW (2 phrase tell(s): robust, seamless)

At x80 the density term clears, correctly: nine em dashes in twelve hundred words really is fine. The phrase tells do not clear, because they never could. The hole is closed, and it is closed by removing the averaging rather than by tuning a number.

What it cost

The new gate is meaningfully stricter than the old one:

over 106 published posts:   PASS 66 (62%)   REVIEW 40 (37%)
families firing:  phrase 35   metadata 23   em-dash 21

The old single-threshold gate passed nearly all of those. I am not going back and rewriting thirty-seven posts, but everything from here has to clear a bar that most of my archive would not have.

Two things I took from this that were not in my own post. One: a suggestion can be entirely correct and still not apply, and the only way to know is to run it against your actual inputs rather than against the argument. Two: the person who told me which of my two options to build was someone looking at their own tool with the same defect, which is a kind of review I cannot generate by thinking harder about mine.

If you maintain a scoring gate of any kind, what families are you summing that should not be summed? I would guess most people with a single numeric threshold have at least one pair.

Which check in your pipeline has never fired?

frank chu — Sun, 04 Oct 2026 19:30:30 +0000

I went through every automated check in my publishing pipeline this week and sorted them by how many times each had ever caught something. Four of them had never caught anything.

My first reaction was that this was good news. A check that never fires is a check whose class of problem you have stopped having, which is what you want. I held that opinion for about an hour.

Then I found a bug that had been live for twenty-four days and is still live a month later, sitting in front of all four of those checks, and realised that two of them had never fired for a completely different reason: they could not. Not "had no occasion to." Could not. The thing they were meant to catch was not visible from the data they were looking at, and no amount of occasion would have changed that.

The specific case, briefly, because the shape matters more than the details. All six of my checks took a file and examined it. The bug was that two of my posts existed twice on the platform under different URLs. That is not a property of any file. You can hand a per-file checker every file I own, run it forever, and it will never see a duplicate, because the duplicate lives in a relationship between my repo and a remote system that no single file knows about.

What unsettles me is that I could not tell the two groups apart by reading them. Both categories look identical in the code and identical in the logs. Both are a function that runs, examines something, and returns no findings. "Quiet because the problem is gone" and "quiet because I am blind to the problem" produce byte-identical output, and I had been reading that output as reassurance for weeks.

I have been trying to find a reliable way to distinguish them and I do not have one. A few things I have tried, none of which fully work:

Feed each check an input that should fail it. This catches the crudest version, where a check is misconfigured or its regex never matches. It caught one of my four. It does not catch the structural case, because to write the failing input you have to already understand what the check can see, and if you understood that you would have spotted the gap.

Look for checks that examine a narrower scope than the property they claim to verify. This is closer to the real test, and it is how I eventually found the blind one. A check named "is this post published correctly" that only ever opens one file is claiming something about a system while looking at a fragment. But phrasing it that way requires you to have written down what each check actually claims, separately from what it does, and I had not.

Delete a check and see if anything breaks. Appealing, and useless on any timescale I have patience for.

There is a related version of this that I think is more common and more expensive. One of my checks did fire, regularly, and was still not doing its job, because the threshold it compared against could be moved by padding the input. It passed things it should have failed while producing a steady stream of reassuring output. That is worse than the silent ones, because a check that fires sometimes looks alive.

And the two real defects that readers found in my published code this quarter were both found by someone executing the code, not by any tool. Every one of my gates checks form. Both defects were about whether a claim was true. I do not think that gap closes with more gates.

So the question I keep landing on is not "do you have enough checks." It is whether you can distinguish, for any given check, between these three states: it protects you and the problem stopped happening, it fires reliably but its threshold does not mean what you think, and it has never fired and never will.

Three specific things I would like to hear from people who maintain a CI pipeline older than a year:

Do you track which checks have ever failed in production, and have you ever deleted one for being silent too long? I suspect most teams accumulate checks and never retire any, which means the signal from "everything passed" degrades continuously and nobody notices the moment it stops meaning anything.
Has one of your checks ever turned out to be structurally incapable of catching what it was named for? I am curious whether there is a common shape to these. Mine was a per-item check on a property that only exists across items, and I wonder if that is the usual one.
For the things automation cannot check at all, what do you actually do? Code review is the standard answer and I do not find it satisfying, because the two bugs readers found in my work had both survived my own careful review. The thing that caught them was a different person with a different mental model running the code, which is a resource I do not know how to schedule.