DEV Community: Ofri Peretz

I Linted My Own Design System. All 8 Breaks Were Pasted In.

Ofri Peretz — Thu, 20 Aug 2026 09:23:07 +0000

Your design system has tokens. Your linter does not know they exist.

eslint-plugin-react has 104 rules and not one of them can tell you that #ffffff1f should have been var(--surface-highlight). That gap is invisible in review, because a hex value in a className looks exactly like every other string.

So I ran four rules over my own code: 401 .tsx files across a design system, a docs site, a blog, and a control-room app. 144 violations in 50 files. The design system defines 192 tokens. It also breaks them.

Raw color literals: 30 findings, and they cluster

The rule is no-raw-color-literal — any hex, rgb(), or rgba() in JSX source.

// packages/ui/src/effects/shimmer-button.tsx:102
"rounded-2xl px-4 py-1.5 text-sm font-medium shadow-[inset_0_-8px_10px_#ffffff1f]"

⚠️ Raw color literal in source — use a design token (R19) | MEDIUM
 Fix: Replace with a CSS custom property (var(--your-token)) or a Tailwind
      theme class wired to it.

Why this survives review: the hex is inside an arbitrary Tailwind value inside a class string. A reviewer scanning for color changes greps for color or bg-. Nobody greps for #.

Here is the part I did not expect. The design system package had 8 of these, and all 8 were in ambient/, effects/, and patterns/ — components I pasted in from component galleries. The core primitives had zero.

The system didn't drift. It leaked at the seam where other people's code came in.

`data-slot`: 67 findings, the biggest by far

⚠️ JSX element has data-testid but no data-slot (R6) | MEDIUM

require-data-slot fires when an element carries a data-testid but no data-slot. Both are hooks; only one is a contract. A data-testid says "a test grabs this." A data-slot says "consumers may style this." Ship the first without the second and every consumer targets your test IDs, and your next refactor breaks their CSS.

67 of my 144 findings were this one rule. It is the least dramatic and the most expensive — a data-testid standing in for a styling contract is a proxy metric with a refactor bill attached.

The pattern

A design system fails at its boundaries, not its center. The primitives I wrote while thinking about tokens obey them. The components I imported while thinking about shipping do not — and they arrive pre-broken, which means no diff ever shows the moment the violation entered.

That is why this has to be a lint rule and not a review convention. Review catches what a diff shows. Pasted code shows up as one clean green addition.

The config

// eslint.config.mjs
import reactFeatures from "eslint-plugin-react-features";

export default [
  {
    files: ["**/*.tsx"],
    plugins: { "react-features": reactFeatures },
    rules: {
      "react-features/no-raw-color-literal": "error",
      "react-features/no-arbitrary-token-class": "error",
      "react-features/require-data-slot": "warn",
      "react-features/no-inline-style": "warn",
    },
  },
];

npm i -D eslint-plugin-react-features

eslint-plugin-react-features on npm · all four rule docs

Start the last two at warn. On a codebase that has never had these rules, require-data-slot alone will be your largest number, and turning it red on day one just teaches people to disable it.

An honest caveat: 47 of this plugin's 61 rules overlap eslint-plugin-react by name. These four do not — nothing in the incumbent enforces a token system, because the incumbent cannot know what your tokens are.

Source at github.com/ofri-peretz/eslint. Every number here came from a run on 2026-08-11 against the four repos named above — four repos I own, which is a sample of one opinion about components, not a ground-truth corpus. I ran my own rule over my own code and reported what it found, which is the only version of benchmarking I trust myself to do. Re-run it on yours and you will get different numbers.

Run one of these on your own components — what's your data-slot number? Mine was 67, and I wrote the rule.

Your Token Is Not Safer in a Cookie. It Is Safer From JavaScript.

Ofri Peretz — Fri, 14 Aug 2026 09:43:04 +0000

Every team I have worked on has had the localStorage argument. Someone says tokens belong
in cookies. Someone else says cookies mean CSRF. Both cite a blog post. The token stays
where it was.

The argument is unwinnable as stated, because both sides are describing the container —
and the container is not what decides the outcome.

The axis everyone argues about

Ask why one mechanism is safer and you get answers about persistence, capacity, expiry,
whether it survives a tab close. Real differences. None of them has anything to do with an
attacker.

The axis that decides it

One question: can JavaScript running on your origin read it?

If yes, anything achieving script execution on your page reads it too. Not encryption —
you have nowhere to put the key. Not obfuscation — the attacker has your bundle. Not "we
only write it after login" — the attacker runs after login too.

Mechanism	Readable by page JS	Under XSS
`localStorage`	yes	gone
`sessionStorage`	yes	gone
IndexedDB	yes	gone
Cache API	yes	gone
Cookie without `HttpOnly`	yes	gone
Cookie with `HttpOnly`	no	survives

Five of those six rows are the same row. The debate spends its energy on a distinction
that exists only in the last line — and it is not the cookie that creates it, it is the
flag.

`HttpOnly` is the boundary

HttpOnly is unusual: a capability the browser withholds from your own code. You cannot
opt back in at runtime. No API reads it, no devtools trick your bundle can perform. The
value goes out on requests to its domain and is otherwise unreachable.

Almost nothing else in the browser works this way. Everything in the first five rows is a
filing cabinet in a public lobby — useful, convenient, not a vault, and never described as
one by anyone who built it.

The corollary people skip

HttpOnly protects the token from being read. It does not stop it being used.

Script on your origin can still fetch with credentials: "include", and the browser
attaches the cookie. The attacker does not hold your session; they operate it, from your
page, while it is open. Smaller blast radius — no exfiltration, no replay next week — but
not nothing.

The honest summary: HttpOnly converts permanent theft into temporary misuse. Real
and worthwhile. Not immunity.

So what counts as sensitive

Stop asking "is this sensitive" — a word that invites negotiation. Ask: if an attacker
had this value, what could they do, and for how long?

Session token — act as the user until expiry.
Refresh token — mint new sessions, far longer. Higher risk, and the one most often parked in localStorage because it needs to persist.
Cached API response with personal data — disclosure, not takeover. The Cache API is the mechanism people forget is storage at all.
Feature flag, theme, draft — nothing. Put it anywhere.

And the CSRF half?

Never on the same axis either. CSRF is the browser attaching a cookie to a request the
user did not intend — a property of ambient authority, not storage — and SameSite
addresses it directly.

That is why the debate feels unresolvable: one camp describes a read primitive, the other
a send primitive. HttpOnly answers the first, SameSite the second — orthogonal flags on
the same cookie.

Where this lands in the standards

Reading storage you should not have access to is
CWE-522 — insufficiently protected
credentials — under
A07:2021.
HttpOnly and Secure are specified in
RFC 6265 §4.1.2.5–6;
SameSite arrived later, in
RFC 6265bis, still
a draft despite universal implementation. Both are on
MDN's Set-Cookie page.
For severity language, use
CVSS and
the OWASP categories.

Why the argument keeps happening

All five equivalent rows are reached the same way: script execution on your origin. So
storage is downstream of the XSS question, and a team that has not settled how untrusted
data reaches a DOM sink is arguing about where to put the token while leaving the door
open. DOM sinks and sources, the
companion piece in this series, works through why those sources outnumber the sinks.

What is in your localStorage right now? Not what you think — open devtools and look. I
have found a refresh token, a full user object with an email, and a cached permissions
array in codebases I would have defended.

I write about measurement and static analysis at
dev.to/ofri-peretz.

innerHTML Has Five Doors. Most Reviews Only Watch One.

Ofri Peretz — Sat, 08 Aug 2026 08:19:02 +0000

I have reviewed the same line of code, written five different ways, and caught it once.

The line is element.innerHTML = something. I catch it when something is obviously a
URL parameter. I miss it when something arrived four frames ago through a message
event, got stored, and is written to the DOM by a function in a different file.

The sink was identical every time. My attention was not.

Sinks are few. Sources are many.

A sink is where data becomes execution: innerHTML, outerHTML,
insertAdjacentHTML, document.write, eval. You can list them on one hand, which is
why they feel solved. Grep for innerHTML. Ban it in review. Done.

A source is where untrusted data enters. Sources are not listable — they are a
property of your architecture, and they multiply with every feature.

That asymmetry is the whole problem. The sink is a syntactic fact. The source is a
trust fact, and trust facts do not survive being passed as a function argument.

The five doors

The same sink, reached five ways. Every one is code I would call normal.

// 1. Direct — the one everybody catches
element.innerHTML = new URLSearchParams(location.search).get("q");

// 2. postMessage — another frame speaks, event.data is a string like any other
window.addEventListener("message", (e) => { element.innerHTML = e.data; });

// 3. FileReader — the user picked the file, which feels like consent
reader.addEventListener("load", () => { preview.innerHTML = reader.result; });

// 4. WebSocket — the connection is authenticated, so the payload feels authenticated
socket.addEventListener("message", (e) => { feed.insertAdjacentHTML("beforeend", e.data); });

// 5. Worker — it is our own worker, running our own code
worker.addEventListener("message", (e) => { results.innerHTML = e.data.html; });

Five doors, one room. On the first, a reviewer says "that's user input." On the other
four, they say some version of "but that's ours."

The word "ours" is doing all the damage

Each of those four defences is a claim about provenance, and each is wrong the same way.

A message event fires for any frame with a handle on your window unless you check
event.origin — the listener does not filter by default. An authenticated WebSocket says
the connection is trusted, not what the other end sent, which may have been stored by a
different user hours earlier. A worker is your code, but the data it posts back is
whatever it was given. And a file the user chose is untrusted precisely because they
chose it.

The failure is not that developers think innerHTML is safe. Everyone knows. The failure
is that danger gets assessed at the sink, where the only visible fact is a string — and
the string looks the same whoever wrote it.

Why generated code lands here

Ask a model to render a preview and it reaches for innerHTML, because that is the
shortest correct-looking answer to "put this markup on the page" and the training corpus
is fifteen years of tutorials using it.

More to the point: a model completing a message handler has the reviewer's problem,
worse. It sees a local scope. It cannot know whether that listener lives in an app that
checks event.origin elsewhere. The sink is local; the trust decision is not.

Which is the interesting part — the information needed to get it right is not present at
the point of writing. A property of the problem, not of the author.

The taxonomy is the useful artifact

Source	Trust claim that fails
URL / query params	none — everyone catches this
`postMessage`	"another frame of ours"
WebSocket message	"the connection is authenticated"
`FileReader` result	"the user chose the file"
Worker message	"it is our own worker"

Read the right column. That is not five vulnerabilities. It is one — provenance assumed
from proximity — wearing five costumes. Sinks tell you where to look; pairs tell you what
to ask.

What it means for detection

Grepping a sink is a heuristic. Following a source to a sink is taint tracking —
the thing that decides your false-positive rate.
Full taint tracking across frames and sockets is not something a linter does; those
boundaries are
the line between linting and SAST.
What a linter can do is match source and sink inside one handler — which covers all four
cases, because the risky pattern is nearly always written in a single function.

Two branches, not one chain

The XSS outcome is CWE-79, under
CWE-74 Injection. The trust failure is
CWE-345 under
CWE-693 — with its child
CWE-346 applying to postMessage
alone, the only case with a page-side origin check. A WebSocket's Origin is validated
server-side at the handshake; FileReader and workers have no origin at all.

Different branches, not one chain. Filing all of it as "XSS" is how the actual defect goes
unrecorded. The taxonomy piece
has the map.

Worth reading in full:
OWASP's DOM-based XSS Prevention Cheat Sheet
and PortSwigger on sources and sinks.
To stop it being a review question at all,
Trusted Types moves
it to the browser — the only place that sees every assignment.

The fix is boring, which is the point

window.addEventListener("message", (event) => {
  if (event.origin !== "https://trusted.example") return;
  element.textContent = event.data;
});

Check the origin. Use textContent for text. Sanitize when you genuinely need markup.
None of that is the hard part.

The hard part is noticing you are in one of the five doorways — because four of them do
not look like doorways. They look like your own code talking to itself.

Which door caught you? I am most interested in postMessage — the origin check is one
line, documented everywhere, and I still find handlers without it in code I wrote myself.

I write about measurement and static analysis at
dev.to/ofri-peretz.

Nobody Writes Bad Crypto. They Write Correct Crypto at Four Layers.

Ofri Peretz — Sat, 01 Aug 2026 09:00:32 +0000

The code review comment I have written most often, and regretted most often, is
"use a stronger algorithm."

It is regrettable because it is almost never the actual defect. It is the defect I can see
from where I am standing. The team swaps SHA-1 for SHA-256, everyone agrees the review was
useful, and the thing that would have mattered stays where it was.

Four layers, each hiding the next

A crypto call has to be right four separate times, and being correct at one layer makes
the next one invisible. AES looks like a decision you already made. It is four.

Layer 1 — the primitive. Right kind of thing? A hash where you needed a
KDF. A cipher where you needed a MAC. Math.random() where you needed a CSPRNG. Everyone
reviews this layer, because it is visible in the function name.

Layer 2 — the mode. Primitive right, mode wrong. AES in ECB encrypts
identical blocks to identical ciphertext — which is why the famous penguin is still
legible after "encryption." Nothing about aes-256-ecb reads as wrong at a glance; it
contains aes and 256, and both are reassuring.

Layer 3 — the parameters. Right primitive, right mode, wrong inputs. A
static IV. An iteration count set in 2015 and never revisited. RSA where someone passed
RSA_PKCS1_PADDING explicitly — Node defaults to OAEP, so this one takes a deliberate
argument to get wrong, which is exactly why it survives review. Each is a single value,
usually a constant, and constants do not attract attention.

Layer 4 — the usage. Everything above correct, and the surrounding code
gives it away. Comparing an HMAC with === and leaking the answer through timing.
Deriving a key correctly, then logging it. This layer is not crypto code at all, which is
why crypto review misses it.

Why layers beat a rule list

A rule list tells you what to grep for. The layer tells you where your team is, and
therefore what you are not going to catch.

My working hypothesis — and it is a hypothesis, not something I have measured — is that
failures cluster by how much crypto attention a team has already had. Never thought about
it: layer 1. Just audited: layer 2, because the audit said "upgrade the primitives" and it
did exactly that. Careful for years: layers 3 and 4.

If that holds, the uncomfortable corollary is that layer 3 and 4 defects live longest
because the code passes every review asking "are we using strong algorithms." The
maturity that fixes the first two layers is the same maturity that stops looking further
down. That is the measurement I most want from this taxonomy.

Detection: the easy layers are the trap

Layers 1 and 2 live in a single call expression. createHash("sha1") and aes-256-ecb
are string literals; matching them is syntactic and any linter does it well.

Layers 3 and 4 are not. Whether an IV is static depends on where it came from — a
data-flow question, the difference between
matching a pattern and following a value.
Whether a comparison is timing-unsafe depends on whether the value is a secret, which no
analyzer knows from the expression alone.

So tool coverage is inversely correlated with how long a defect survives. The layers a
tool covers cleanly are the ones your team already fixes. Read any claim that a tool
"covers crypto" as "covers layers 1 and 2" until shown otherwise — and checking that needs
a corpus with known answers,
not a rule count.

Where this lands in the standards

The layers map to distinct CWEs, which is the useful part —
the taxonomy already encodes the
distinction most reviews collapse:

Layer	CWE
1 — primitive	CWE-327 risky algorithm; CWE-338 weak PRNG
2 — mode	CWE-327 — ECB has no dedicated CWE
3 — parameters	CWE-1204 weak IV generation; CWE-323 reusing a nonce or IV; CWE-916 insufficient computational effort
4 — usage	CWE-208 observable timing discrepancy

Filing all four as "weak crypto" is how the layer 3 and 4 instances disappear from your own
bug data, and then from your priorities.

Prior art worth reading: Egele et al., An Empirical Study of Cryptographic Misuse in
Android Applications (CCS 2013), which found the overwhelming majority of apps using
crypto APIs got at least one layer wrong. A decade old, different platform, and the
layering has not aged at all.
NIST SP 800-175B is the reference for
what each primitive is for, and Bernstein's cache-timing work on AES is the clearest
statement of why layer 4 is not paranoia.

Why generated code lands on layers 2 and 3

A model asked for "AES encryption in Node" produces something that runs. Running is the
constraint it optimizes, and every one of these layers is invisible to that constraint.
ECB runs. A static IV runs. An iteration count of 1000 runs. The code is correct at the
layer a reader checks, because it was trained on a corpus where that is what correct
looked like.

Same reason the human review misses it. The model read the same tutorials you did.

Which layer does your codebase fail at? The answer tends to say more about when the code
was written than who wrote it — layer 2 usually means "audited once, years ago."

I write about measurement and static analysis at
dev.to/ofri-peretz.

I Audited 203 of Our Own ESLint Security Rules. 16% Mislabel Their Own CVSS Score.

Ofri Peretz — Tue, 28 Jul 2026 04:59:46 +0000

Run eslint-plugin-node-security against a Server-Side Request Forgery
pattern and the terminal prints this:

src/fetch-avatar.ts
  4:7  warning  🔒 CWE-918 CVSS:9.1 | HTTP call whose URL argument name
              suggests user input. | LOW

CVSS 9.1. Read the number on the calculator and it's nine tenths of a point
from the maximum possible score a vulnerability can get. Read the word next
to it and it says LOW. Those two facts are printed four characters apart
on the same line, by the same rule, and they disagree with each other.

I write the CWE/CVSS/OWASP metadata that ships in every one of our lint
messages — it's meant to be the thing that tells you, at a glance, how
worried to be. So I went and checked whether it actually does that. I wrote
a 25-line script that reads every rule in the Interlace ESLint ecosystem,
pulls its CVSS score and its shipped severity label, and checks whether the
label matches the official band the score falls in.

33 of 203 rules — 16% — don't. Some of those are defensible once you
know why. Some aren't. All of them are a good excuse to explain what CWE,
CVSS, and OWASP actually measure, because the gap between them is exactly
where lint output stops being trustworthy and starts being decoration.

What the three labels actually measure

A single line of our lint output packs in three unrelated standards. Quick
definitions — there's a full reference for each if you need it:

CWE names what kind of bug it is. CWE-918 is SSRF; CWE-287 is Improper Authentication. A category, not a verdict.
CVSS is a computed 0.0–10.0 score. Its four bands are the ruler this audit uses: 0.1–3.9 Low, 4.0–6.9 Medium, 7.0–8.9 High, 9.0–10.0 Critical. The number is the primary source; the band name is derived from it.
OWASP Top 10 is a category bucket (A01–A10) — an address, not a severity measurement.

Only CVSS is actually a severity measurement. So when a lint rule prints a
severity word, that word has exactly one legitimate source: the CVSS score
next to it. If they disagree, the word is wrong.

Case 1: a 9.1 that prints LOW — and the honest reason it might be fine

no-ssrf flags HTTP calls where the URL argument's name looks
user-supplied (userUrl, req.query.endpoint, targetUri) — a classic
Server-Side Request Forgery
setup, the bug class behind more than one well-known cloud metadata-endpoint
breach:

// flagged — the *name* suggests user input, not a data-flow trace
async function fetchAvatar(userUrl) {
  return fetch(userUrl);
}

The rule's own metadata: cwe: 'CWE-918', cvss: 9.1 — squarely in the
Critical band. The message it prints: severity: 'LOW'.

Here's the part that's genuinely defensible: the rule's own doc comment
calls itself out as "a naming heuristic, not data-flow analysis." It
matches on identifier names, not on whether the value actually reaches an
attacker-influenced source. That's a real limitation — plenty of
userUrl-named parameters are perfectly safe, admin-configured constants.
A low-confidence match that could be a
false positive
does deserve a lower-urgency label than a rule that traced the actual
taint.

The part that isn't defensible: confidence and impact are two different
axes, and severity is only supposed to encode one of them. "How sure are
we this is real" and "how bad is it if it's real" are independent
questions — a low-confidence match on a 9.1-impact bug is still worth
more attention than a high-confidence match on a 3.0. Collapsing both into
one LOW/MEDIUM/HIGH/CRITICAL string means the reader can't tell
which axis they're looking at, and a grep-based triage pass (or an
LLM agent doing the same) will deprioritize a Critical-impact finding on the
strength of a label that was only ever describing detection confidence.

Case 2: a 9.8 with no heuristic excuse at all

require-audience-validation and require-issuer-validation aren't naming
heuristics — they check one deterministic fact: does a jwt.verify() call
pass an audience (or issuer) option? Yes or no, no ambiguity:

// flagged — no audience claim checked
jwt.verify(token, publicKey);

// not flagged
jwt.verify(token, publicKey, { audience: "https://api.example.com" });

Skip audience validation and a JWT minted for any service that trusts the
same signing key gets accepted by this service too — a textbook confused
deputy, CWE-287,
cvss: 9.8. Same story on the issuer side. Both rules print severity: 'MEDIUM'.

There's no false-positive story to hide behind here — the check is exact.
9.8 is a hair under the maximum a CVSS score can reach. MEDIUM sits two
whole bands below where that number lives. If a reader — or a CI gate that
filters on severity string instead of CVSS number — treats MEDIUM findings
as "fix this sprint," a near-maximum-severity auth bypass just got queued
behind a color contrast warning.

Case 3: the one that runs the other way

Not every mismatch under-states. no-unsafe-search-path
— the PostgreSQL search_path hijacking rule I wrote up in full elsewhere —
ships cvss: 7.5 (High band) but prints severity: 'CRITICAL', one band
above its own score. I won't re-run the attack walkthrough here — the
short version, if a label disagreeing with a number bothers you as much as
it bothers me: read that piece for the exploit, come back here for why the
label drifted in the first place.

Why this happens: `severity` is the one field nobody enriches

Every rule funnels its message through one formatter,
formatLLMMessage(), and every formatter call goes through the same
enrichment step first:

// packages/eslint-devkit/src/messaging/formatters.ts
function enrichFromCWE(options) {
  if (!options.cwe) return options;
  const cweData = CWE_MAPPING[options.cwe];
  if (!cweData) return options;

  return {
    ...options,
    owasp: options.owasp ?? cweData.owasp,
    cvss: options.cvss ?? cweData.cvss,
    compliance: options.compliance ?? CWE_COMPLIANCE_MAPPING[options.cwe],
  };
}

There's a canonical lookup table, CWE_MAPPING, and it's correct — its
entry for CWE-918 is { cvss: 9.1, severity: 'CRITICAL' }, internally
consistent, right band. CWE-287 maps to { cvss: 9.8, severity: 'CRITICAL' }. If every rule simply deferred to this table, none of the 33
mismatches would exist.

But look at what enrichFromCWE actually forwards: owasp, cvss,
compliance — and that's the whole list. severity isn't in it. Every
cvss number in our output either comes from this vetted table or from an
explicit override that a human cross-checked against it. Every severity
string is whatever word the rule's author typed into that one call site,
with nothing checking it against the table, the CVSS number, or any other
rule's choice for the same CWE. Two rules can share a CWE, share a CVSS
score, and print two different severity words, and nothing in the pipeline
would ever notice.

That's the actual bug: not "someone mislabeled 33 rules," but "the schema
has a field with no source of truth." Free-text fields drift; that's what
they do when nothing enriches them. The fix is the same one-line pattern
already applied to owasp and cvss — default severity from
CWE_MAPPING[cwe].severity unless a rule explicitly overrides it, the same
options.severity ?? cweData.severity fallback the other three fields
already get. I've filed it; it hasn't shipped as of this writing, and I'd
rather say that than quietly imply it's fixed.

The audit, if you want to run it yourself

Point this at any rule set that ships CVSS numbers and severity strings —
ours included — and it'll tell you exactly where they disagree:

// audit-severity-drift.ts — flags rules where severity doesn't match its own CVSS band
import { readFileSync, globSync } from "node:fs";

const BANDS = [
  [9.0, 10.01, "CRITICAL"],
  [7.0, 9.0, "HIGH"],
  [4.0, 7.0, "MEDIUM"],
  [0.0, 4.0, "LOW"],
] as const;

const bandFor = (cvss: number) =>
  BANDS.find(([lo, hi]) => cvss >= lo && cvss < hi)?.[2] ?? "?";

for (const file of globSync("packages/eslint-plugin-*/src/rules/*/index.ts")) {
  const src = readFileSync(file, "utf-8");
  const cvss = src.match(/cvss:\s*([\d.]+)/)?.[1];
  const severity = src.match(/severity:\s*'([A-Z]+)'/)?.[1];
  if (!cvss || !severity) continue;

  const expected = bandFor(Number(cvss));
  if (expected !== severity) {
    console.log(`${file}: CVSS ${cvss} → ${expected}, shipped ${severity}`);
  }
}

203 rule files scanned, 33 flagged — an 84% clean rate I'd rather publish
honestly than round up. And that rate is itself a snapshot: the rule sources
as they stood when this published (2026-07-06), spread across the ecosystem's
twenty-odd independently-versioned plugins, not one pinned release. The
article's own thesis applies to its own numbers — a printed audit is a cache
too. Re-run the script against today's checkout and trust that count over
mine.

Reading a severity label like a security engineer, not a triage bot

None of this is specific to our plugins — it generalizes to any tool that
prints a severity word next to a CVSS number, including the ones you didn't
write:

The number is the primary source. The word is a cache of it, and caches go stale. If a CRITICAL/HIGH/MEDIUM/LOW string and a CVSS score disagree, trust the score — it's the one with a formula behind it, not a human's word choice.
CWE tells you the shape of the bug, not how loudly to worry. "This is a CWE-918 (SSRF)" is a fact about the code. It says nothing about this instance's severity until a CVSS vector is computed for it specifically.
A low-confidence detector and a low-impact bug produce the same label and mean opposite things. If a tool's docs mention "heuristic" anywhere near a LOW, ask whether LOW is describing the bug or the detector's certainty — they're rarely the same axis, and only one of them tells you whether to actually go read the code.
Don't build a CI gate on the adjective. severity !== 'CRITICAL' as a merge-blocking filter inherits every mislabel in the tool it's reading from. Gate on the CVSS number, or on the CWE list you've decided matters, not on a string a human typed by hand.

The config

npm install --save-dev eslint-plugin-node-security eslint-plugin-jwt

import nodeSecurity from "eslint-plugin-node-security";
import jwt from "eslint-plugin-jwt";

export default [nodeSecurity.configs.recommended, jwt.configs.recommended];

Both plugins print the CWE/CVSS/OWASP line on every finding — that's what
made this audit possible to run against our own output in the first place.
Getting started with eslint-plugin-node-security
and getting started with eslint-plugin-jwt
cover the rest of what each ships.

I'm not writing this to talk our own numbers down — 84% of 203 rules
printing a severity word that matches their own CVSS band is a real bar,
and I'd take it over a tool that doesn't print a CVSS number at all and so
never gets caught disagreeing with itself. I'm writing it because "check
the number, not the adjective" is a rule I only started following rigorously
after writing the script that caught my own tool getting it wrong three
different ways in three different rules.

Your turn: open whatever security linter you already run, pick one
finding, and look up its CVSS vector by hand. If the vector notation is new,
what a CVSS score actually measures
walks the AV/AC/PR/UI fields — and why severity, exploit probability, and
confirmed exploitation are three separate measurements with three separate
owners. Does the severity word next to it still hold up once you've done the
math yourself?

⭐ Star on GitHub if you've ever trusted a severity label more than the score behind it.

I'm Ofri Peretz, a security engineering leader and the author of the
Interlace ESLint ecosystem — domain-specific static analysis for security,
reliability, and performance on the Node.js stack.

ofriperetz.dev · LinkedIn · GitHub

142,076 Weekly Downloads. Zero Releases Since 2021. Is the Niche Defended?

Ofri Peretz — Mon, 20 Jul 2026 04:45:01 +0000

Two reads from the npm registry, same package, same morning (2026-07-19):

downloads, last week:  142,076
last publish:          2021-05-21  (v3.2.0)

serverless-iam-roles-per-function is one of the Serverless Framework ecosystem's most-installed plugins, and it hasn't shipped a release since May 2021. Eleven weeks earlier: 118,761 weekly downloads (measured 2026-05-04). Zero releases in between; 23,315 weekly installs gained.

The package isn't surviving abandonment — its installed base is growing straight through it.

My first read was the common one: "142k a week, the niche is taken, I'm five years late." That's the misconception this article exists to correct. High downloads don't mean a niche is taken. Downloads measure the installed base; maintenance decay measures whether the niche is defended. The gap between those signals is where I decide what to build next, using what I call the incumbent map. Fair warning: I ran the arithmetic in May, shipped against its #1 row on 2026-05-07, and only ran the filter that kills that row while fact-checking this article in July. The method works; I ran half of it. Both halves are below, in the order I should have run them.

How do you build an incumbent map?

Three signals per package, all public, all re-runnable:

Weekly downloads — demand. The installed base a maintained replacement would serve. Downloads are a proxy metric — npm's own account of download counts includes build servers, mirrors, and robots, so the number tracks lockfile momentum, not humans choosing a package. As a relative ranking of where demand pools inside one ecosystem, though, it's the best public signal there is.
Years since last publish — supply-side decay. Read the registry at time[<latest-version>]. Not time.modified — that field moves when npm touches registry metadata: serverless-iam-roles-per-function shows modified: 2022-05-17, a full year after its last release. Trust the version's own timestamp, nothing else.
Open-issue decay — unmet demand. No clean number here, so make the procedure explicit: sort open issues by reactions, list the recurring asks, date the last maintainer reply. For the plugin above: managed-policy attachment and custom role naming — asked for years across multiple issues, no maintainer commitment (the 2018 managed-policy ask drew "open for PRs"; nothing shipped since). And read the replies, not just titles; mine matter later.

The first two signals, as commands:

curl -s https://api.npmjs.org/downloads/point/last-week/<pkg>               # demand
curl -s https://registry.npmjs.org/<pkg> | jq '.time[."dist-tags".latest]'  # decay

Multiply, roughly: demand × decay × unanswered asks, ranked descending. Deliberately not a precise formula — a shortlist generator whose output you still have to judge.

The closest prior art is OpenSSF Scorecard: its Maintained check formalizes the same "is anyone home?" signal for the dependencies you consume — a risk lens. The incumbent map points it the other way: not "should I trust this package?" but "should I compete with it?" I haven't seen that build-side version written down anywhere — a strong claim, and an open invitation to correct me.

What does the serverless incumbent map actually show?

The worked example that produced my roadmap. Two zones, all numbers measured 2026-07-19:

Defended — active maintainers, do not compete on maintenance:

Package	Downloads/wk	Last publish
`serverless-offline`	627,578	2026-06-23
`serverless-esbuild`	388,619	2026-05-21
`serverless-domain-manager`	304,882	2026-07-17
`serverless-step-functions`	208,865	2026-07-13

Abandoned — demand present, nobody home:

Package	Downloads/wk	Last publish	Silence
`serverless-iam-roles-per-function`	142,076	2021-05-21	5.2 yrs
`serverless-plugin-include-dependencies`	52,500	2024-07-22	2.0 yrs
`serverless-plugin-common-excludes`	51,415	2021-07-06	5.0 yrs
`serverless-api-gateway-throttling`	36,528	2023-03-19	3.3 yrs
`serverless-plugin-tracing`	35,709	2017-10-13	8.8 yrs
`serverless-plugin-canary-deployments`	31,562	2022-04-11	4.3 yrs
`serverless-associate-waf`	28,787	2020-04-17	6.3 yrs

The top zone shipped within two months. The bottom zone's median silence: five years — on packages still installed tens of thousands of times a week.

The map moves, in both directions. serverless-associate-waf shows demand finally decaying — down roughly a quarter from my May reading — so abandonment does reach the download number, on a lag of years, not months. And re-pulling before publishing deleted a row: serverless-openapi-documenter, filed in my May notes as a stale documentation-plugin niche, shipped a release on 2026-06-24. Someone re-defended it while I drafted. Row gone. An incumbent map is a snapshot, not a subscription — re-run it the week you commit, not the quarter you planned.

When is a quiet package finished instead of dead?

The honest counter-case, because the map's arithmetic will lie to you twice.

First: stable is not dead. serverless-prune-plugin sits at 414,154 downloads a week (measured 2026-07-19), last publish 2024-10-16. Twenty-one quiet months — and it may simply be done: its job is narrow, its surface barely moves. The separating question: does the surface underneath the package move? IAM — the ground serverless-iam-roles-per-function stands on — moves constantly: new AWS services, new actions, new policy patterns. Five silent years on a moving surface is decay; on a frozen surface, it can be completion. Same silence, opposite meanings — a classic measurement-bias trap.

Second: platform absorption. serverless-plugin-typescript: 151,586 downloads a week (measured 2026-07-19), dormant since 2023-06-05 — by raw arithmetic the juiciest target on the board. It's not on my list: Serverless Framework v4 builds TypeScript natively and won't run build plugins like it unless you explicitly opt out of the native build. That niche isn't undefended; it's dissolving. A replacement would be competing with the platform itself.

An installed base looks like a moat. The publish log says whether anyone still defends it; the platform roadmap, whether the castle is even staying put.

How were the replacement targets actually picked?

The full pipeline, as actually run for the serverless line:

Rank every community plugin by demand × decay (the tables above are the output).
Filter: is the niche still real? — serverless-plugin-typescript fell here first. Then the same filter, run against the current v4 docs instead of my stale notes, reached the top row of my map: v4 ships per-function IAM roles natively — provider-statement inheritance on by default, log/VPC/event-source permissions auto-added, the plugin's own config fields accepted for migration. The official IAM guide credits the plugin for pioneering the feature and says you can now safely remove it. My #1 target wasn't an undefended niche; it was one the platform had absorbed — with honors.
Filter: is the incumbent truly undefended? — re-pull the registry the week you commit; that check removed serverless-openapi-documenter.
Filter: is the unmet demand documented in public — and still unmet? The newest open issue asks "Serverless Framework V4 Support?" — and the thread answers itself: a collaborator's "it works," then a comment pointing at the v4 docs above. The two features its tracker spent years asking for — managed policies, custom role names — shipped in v4's native implementation, not the plugin's. The counter-evidence to my top pick sat in the tracker I was reading for demand. Read the replies.

The worked example deleted two of its own rows — one while I drafted, one while I fact-checked. Here is what the second deletion cost, stated plainly: my May map scored serverless-iam-roles-per-function 23/25 — "the biggest single-plugin opportunity in the portfolio" — and I shipped @interlace/serverless-iam-roles-per-function 1.0.0 off that row on 2026-05-07 (the npm publish date; check it). The filter you just read ran two months later. The package stays published and does what it says; what it is not, anymore, is the line's growth bet — v4 owns that niche now. The ranked ROI analysis lives in the repo, unchanged, dates included — deliberately, as the before-artifact: you can see exactly the map I built from, and exactly what running filter 2 late overturned. Arithmetic generates the shortlist; filters, run against primary sources — before you build — make the decision.

Notice the conflict of interest: the person selling replacements is grading the incumbents — the same conflict as building what I benchmark, with the same mitigation: every number here is a public registry read, stamped with its measurement date, re-runnable in one curl without me in the loop.

The map is step one of entering an ecosystem. Step two keeps the building honest — write down claims and evidence before the marketing exists: the claims registry. Different package types need different metrics once you measure what you built — but the map is how you decide to build at all.

What's the most-downloaded abandoned package in your ecosystem — React components? Terraform providers? Gradle plugins? Two registry reads will tell you. I'm collecting incumbent maps, and I'd genuinely like to see yours.

Quick reference

Signal	What it measures	Where to read it	Failure mode
Weekly downloads	Installed base (demand) — not adoption, not health	`api.npmjs.org/downloads/point/last-week/<pkg>`	Mistaking base for defense; CI/mirror inflation
Years since last publish	Supply-side decay	`registry.npmjs.org/<pkg>` → `time[<latest-version>]` — not `time.modified`	Finished ≠ dead on a frozen surface
Open-issue decay	Unmet demand, stated in public	Issue tracker: sort by reactions, date the last maintainer reply	Reading titles, not replies — the answer may be "the platform shipped it"
Deprecation/archive status + platform roadmap	Whether the niche still exists	Registry `deprecated` field, repo header, framework changelog + docs	High downloads on a dissolving niche
Decision rule	demand × decay × unanswered asks → shortlist → 3 filters	This article	It's a snapshot — re-run the week you commit

References

"numeric precision matters: how npm download counts work" (npm blog, 2014) — npm's own account of what a download includes (build servers, mirrors, robots); the primary source for why downloads can only ever be a demand proxy.
npm registry download-counts documentation — the API the map reads: the endpoints and the data pipeline behind the counts.
Serverless Framework IAM guide — the platform-absorption primary source: v4's native per-function roles, statement inheritance, auto-added permissions, and the credit to the community plugin it replaces.
OpenSSF Scorecard — the closest prior art: its Maintained check operationalizes "is anyone home?" for consumption-side dependency risk; the incumbent map reuses that signal for build-side decisions.

Related deep dives

Proxy Metrics: When the Number Isn't the Thing — why downloads, stars, and every other convenient number measure something adjacent to what you care about
Bias in Measurement — how a real number still produces a wrong inference, including the too-clean-zero forensic
The Claims Registry — step two of the playbook: the evidence discipline that comes before any claim
Different Metrics for Different Package Types — once you build, how the measuring has to change per package type
I Built What I Benchmark. Here's How I Try Not to Cheat. — the conflict-of-interest process this article's mitigation borrows

Star the serverless repo — the map above is its roadmap

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

We Were Wrong About sls remove

Ofri Peretz — Mon, 20 Jul 2026 04:44:23 +0000

The loudest claim in my serverless line was wrong. Early copy for @interlace/serverless-api-gateway-caching implied that the community caching plugin leaves an API Gateway cache cluster running — and billing — after sls remove, and that our cleanup hook fixes it. On 2026-05-04, our own E2E harness ran the community plugin through sls remove on live AWS: 31 seconds, exit 0, no orphans. CloudFormation deleted the stage and took the cache cluster with it. There is no sls remove ghost-billing bug.

The trap itself is real — I had reproduced it on live AWS the day before, 2026-05-03, in about ten minutes for roughly $0.005 of cache-cluster time. It just lives on a different path than the one we were loudest about. This article is the full telling of that re-scoping: the original claim, the two runs that killed and rebuilt it, and why the narrower claim that replaced it does more work than the broad one ever did.

What was the original claim?

The community serverless-api-gateway-caching (v1.11.0) manages the API Gateway cache cluster imperatively — it calls the AWS UpdateStage API during deploy instead of declaring the cluster in the CloudFormation template. CloudFormation never learns the cluster exists. From there, the inference writes itself: if CloudFormation doesn't know about the cluster, teardown won't remove it, so sls remove must leave it behind, billing forever. Our plugin ships a before:remove:remove hook that disables the cluster before teardown — so early copy implied, in effect: the community plugin has a ghost-billing bug on sls remove, and we fix it.

Notice what that claim actually was: an inference from architecture, shipped as if it were a measurement. And it was consistent — repeated the same way across the README, the docs, the comparison table. Consistency is reliability; whether the claim describes the failure that actually happens is validity — and nobody had checked.

What did the measurement actually show?

Two runs on live AWS, one day apart, both against the pinned community release 1.11.0.

2026-05-03 — the trap, reproduced. Deploy with the community plugin (a live stack in 49s), wait for the cache cluster to reach AVAILABLE (4–7 minutes), then do what a real team does when it stops wanting a plugin: delete it from the plugins array and redeploy the still-running service. The redeploy succeeds in ~28 seconds and no plugin code runs — the plugin is gone from the config, so it can't. The cluster afterward: enabled: true, status: AVAILABLE, size: 0.5. Still running, still billing. No CloudFormation event, no log line, no warning. And no exit through the plugin either: sls caching disable and sls caching status both return "command not found" — the community plugin registers no custom commands, so the only escape is dropping to the AWS CLI and patching cacheClusterEnabled to false by hand. Total cost of watching all of this happen: ~10 minutes, ~$0.005 — the recipe is public in docs/ghost-billing-reproduction.md, cheap enough that "it reproduced on my machine" can be your finding rather than my assurance.

2026-05-04 — the loud path, refuted. The community E2E then ran the exact path our copy warned about: sls remove with the plugin still installed. Clean. 31 seconds, exit 0, no orphans — CloudFormation deleted the stage, and the cluster went with it.

The mechanism fits in one sentence: the cache cluster rides on the stage. Delete the stage — which is what sls remove does — and the cluster dies with the stack, no matter who created it. Keep the service deployed and remove only the plugin, and the stage stays up while the one tool that remembered the cluster leaves your toolchain.

I ran that second E2E to collect the evidence for the claim. It came back with the evidence against it — exit 0, no orphans, and a rewrite of my best line of copy. The same discipline that makes you distrust a result that flatters you is the one that makes you run the test that can defeat you, before someone else does.

What is the re-scoped claim?

The old copy didn't get quietly patched; it got re-scoped, on the record. Our claims registry keeps a dedicated table for exactly this case — original copy, what we measured, what's true — and the row for this incident reads, condensed: earlier copy implying "we fix sls remove's ghost-billing bug" was wrong, because there is no sls remove ghost-billing bug. The orphaned-cluster trap lives on the uninstall-while-keeping-service path, where sls caching disable — an explicit offboarding command the community plugin doesn't have — is the actual fix. The before:remove:remove hook stays, verified live (stack deleted in 28s, zero residuals, measured 2026-05-03), but it's defense-in-depth now, not the headline.

That's the whole lifecycle in one incident: claim → reproduction attempt → refutation → re-scope → versioned evidence file. The refuting run is committed as a dated JSON next to the claim row; the trap ships as a recipe anyone can replay.

The stakes are why the claim deserved this much care. The cluster we reproduced was the smallest tier — 0.5 GB at $0.020/hour, $175.20/year. AWS's published cache pricing runs up to 237 GB at $33,288/year, and the charge is per stage, per region, with no flag on orphaned clusters. A number that size deserves a claim pointed at the path where the money actually leaks.

Why is a narrower claim stronger?

Value investors call it margin of safety: you don't pay the price your own estimate says is fair — you pay less, so that being somewhat wrong doesn't ruin you. A claim scoped to exactly what the run files show carries the same margin. The broad claim had none: it rode on an untested inference, and a single ten-minute E2E from anyone could falsify it. Ours did, mercifully before anyone else's.

The narrow claim is also more useful to you. "Run sls caching disable before you remove the plugin from config" is an action. "Beware sls remove" would have taught you to distrust the one command that actually cleans up after itself — misallocating your caution away from the path that costs money and onto the path that doesn't.

Which is the misconception worth naming: "walking back a claim costs credibility." It's backwards. Unmeasured claims cost credibility — on a schedule you don't control. A published re-scope is the cheapest credibility purchase available: it costs one uncomfortable table row and buys back the right to be believed about everything the run files do support. The genre has old prior art — Buffett's shareholder letters have aired his own mistakes for decades, and it's the same model behind how I try not to cheat in my own benchmarks.

I'll admit the first instinct, reading the clean run, was the other genre: quietly delete the sentence and move on. That instinct — fix silently, stay loud — is exactly what a claims registry exists to make impossible, and the reason this article exists at all.

Quick reference

Question	Answer (measured 2026-05-03/04, community plugin v1.11.0)
Does `sls remove` orphan the community plugin's cache cluster?	No — CloudFormation deletes the stage and the cluster with it (31s, exit 0, no orphans)
Where is the actual trap?	Remove the plugin from `plugins` and redeploy the still-running service — no plugin code runs; cluster persists `enabled: true, AVAILABLE`
Is there a plugin-mediated escape (community v1.11.0)?	No — `sls caching disable` / `sls caching status` return "command not found"; manual `aws apigateway update-stage` required
What does an orphaned cluster cost?	$175.20/yr at 0.5 GB up to $33,288/yr at 237 GB — per stage, per region, unflagged by AWS
The safe offboarding path (`@interlace/serverless-api-gateway-caching`)?	`sls caching disable` before uninstalling; `before:remove:remove` covers `sls remove` (verified live: stack deleted in 28s, zero residuals)
Reproduce it yourself	~10 minutes, ~$0.005 — ghost-billing-reproduction.md

The claim that survived is smaller than the one we started with, and it's the only one of the two I'd defend in public — because defending it requires no rhetoric, just the run files. If a claim in our docs matters to you, check whether it has a row in CLAIMS.md; if it doesn't, it isn't allowed to ship. That policy — and the re-scope you just read, kept on the record instead of buried — is the honest answer to "why should I trust a plugin author's comparison table."

⭐ Star the serverless repo — where claims ship with run files, or don't ship

Next in this arc: one step earlier in the story. Before you can re-scope a claim about an incumbent plugin, you have to decide which incumbent is worth challenging at all — the abandoned-incumbent map is how the serverless line made that call.

References

Amazon API Gateway pricing — cache pricing — the per-size price table behind every $/hour and $/year figure in this article.
Cache settings for REST APIs in API Gateway — Amazon API Gateway Developer Guide — the stage-level cache settings (cacheClusterEnabled, cluster size) that make the cluster live and die with its stage.

Bias in Measurement — why a result that flatters you is the one to re-run
Reproducibility vs Replicability — the bar the $0.005 recipe is built to clear
Valid vs Reliable Metrics — how a claim can be perfectly consistent and still measure nothing
The Claims Registry — the table this article's re-scoped row lives in
I Built What I Benchmark. Here's How I Try Not to Cheat. — the same discipline applied to the ESLint benchmarks

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

25% of My Benchmark Verdict Is an Opinion. Here's the Anatomy.

Ofri Peretz — Mon, 20 Jul 2026 04:43:35 +0000

On 2026-05-04, my benchmark harness printed its verdict on two Serverless Framework caching plugins: mine scored 0.88; the community incumbent, 0.3025. Every digit of that gap is arithmetic — seven dimensions, each one measured, normalized, weighted, summed. The raw JSON actually stores my side as 0.8800000000000001, kept as-is: IEEE 754's reminder that a number can be very precise about being an opinion. Because the most consequential input in that file was never measured by any instrument: Lifecycle Correctness is worth 25% of the score because I said so.

A composite score is opinions wearing a number costume — the general argument, and the honesty requirements any composite owes its readers, live in Composite Scores and Weighting, and I won't re-derive them here. What I owe you instead is the costume's pattern, dimension by dimension — including whether the verdict survives when you attack the weights.

What are the seven dimensions — and where does each number come from?

Dimension	Weight	Interlace	Community	Measured from
Lifecycle Correctness	25%	1.0	0.5	local source + live AWS deploy
CLI Surface	15%	1.0	0.0	local source
TypeScript Coverage	15%	1.0	0.0	npm registry*
Maintenance Signal	15%	1.0	0.0	npm registry*
Bundle Weight	10%	0.0	1.0	npm registry*
Hook Coverage	10%	1.0	0.375	local source
Documentation Quality	10%	0.8	0.4	README scan
Composite	Σ=1.0	0.88	0.3025

* Community side only. The Interlace values in these rows came from local source ("source": "local" in the result file) — version 0.0.0 has never been published.

Run stamped 2026-05-04, Node v24.12.0 on darwin/arm64. Two disclosures before anything else. First: my @interlace/serverless-api-gateway-caching was measured from local source at version 0.0.0, while the community's serverless-api-gateway-caching is the published 1.11.0 from the registry. That asymmetry makes one score in the table an artifact, not a measurement: my Maintenance Signal 1.0 rests on daysSincePublish: 0, totalVersions: null, weeklyDownloads: 0 — an unpublished plugin scores perfect maintenance while the community's real 379-day publish gap scores 0. Read that row as unmeasured, not excellent. Second: the lifecycle numbers are not simulations. Both plugins went through a live-AWS end-to-end lifecycle run, and the two E2E result files sit on disk stamped 43 minutes apart from the same night.

The "measured from" column matters as much as the scores. Registry-sourced dimensions are proxy metrics: publish cadence stands in for maintenance, type definitions stand in for TypeScript support. And a README scan is reliable without being automatically valid — it returns the same 0.8 every run; whether 0.8 captures what "good documentation" means to you, the harness cannot answer.

How do kilobytes, hook counts, and pass rates become one scale?

The seven dimensions arrive in incompatible units: bundle size in kilobytes, hook coverage as a count of lifecycle hooks handled, lifecycle correctness as pass/fail steps against a live deploy, documentation as a scan score. A weighted sum over raw units lets the biggest-range unit silently win — the classic mixing-scales trap. So every dimension is normalized to 0–1 before any weight touches it.

The community plugin's Hook Coverage of 0.375 is not a grade someone typed in — it's 3 hooks counted against a ceiling of 8, and that ceiling is my own plugin's full hook-and-command surface, which guarantees my side a 1.0. That denominator is a normalization policy that happens to flatter its author, and it deserves the same suspicion as the bundle curve coming next. On Bundle Weight, my 78 KB unpacked against the community's 47 KB came out 0 against 1 — which tells you the normalization curve is itself a policy decision. A gentler curve would have given my 78 KB partial credit; this one didn't, and I left it that way. Normalization choices change scores without changing a single measurement, so they belong in the published methodology, not in a footnote you have to diff the source to find.

What happens when a dimension can't be measured?

The harness has a standing rule for dimensions it cannot measure — historically the AWS-dependent ones, when there's no live deploy to run against: score them null and never impute a value. An imputed dimension is an invented measurement wearing the same costume as the real ones. But the null path deserves the least flattering disclosure in this article: when nulls are present, compositeScore renormalizes over the measured subset — null dimensions drop out of both numerator and denominator — so a partial run still prints one clean-looking composite. Renormalizing isn't imputation, but it quietly moves weight onto whatever happened to be measurable. The day before this run did exactly that: three of seven dimensions measured, and the harness stored 0.7499999999999999 vs 0.25 — IEEE 754 again, equally precise about a composite built from 40% of the total weight, with nothing in the numbers admitting four dimensions were missing. The n-of-7 label that makes a partial composite honest currently lives one layer up, in the claims registry rather than the harness output — a real gap: "3-of-7, here's what's missing" should be printed by the code, not patched on at the claims layer. In this run the null path went unused — dimensionScores holds numeric values for all seven keys, on both plugins (don't cite the skipped array for this; it was empty in the partial run too) — and I'm telling you anyway, because a policy you only hear about when it's exercised is a policy you can't audit.

Why is Lifecycle Correctness worth 25% — and who decided?

I decided. Here's the reasoning, in tiers.

25% — the damage tier. Lifecycle Correctness is the only dimension where failure means the plugin actively harms your deployment instead of merely annoying you. A caching plugin that mishandles the deploy lifecycle doesn't degrade politely — it leaves your API Gateway stage in a state you didn't ask for.

15% × 3 — the daily-use tier. CLI Surface, TypeScript Coverage, Maintenance Signal: the things you touch or depend on every working day. Individually smaller than correctness, collectively 45% of the score.

10% × 3 — the comfort tier. Bundle Weight, Hook Coverage, Documentation Quality: real costs you can see coming and route around.

Reasonable people could defend a different split — which is the point. The common misconception is that a benchmark score computed from real measurements is objective. The measurements are objective; the weights they run through are editorial, and computed is not neutral. Nor is admitting that unusual: Lighthouse versions its scoring weights and OpenSSF Scorecard publishes its full methodology — the composite-scores canonical walks both. Same page, new Lighthouse version, different score — which means every performance score you've ever pasted into a report was a committee vote you didn't attend. Every index has a committee. The only real question is whether the committee publishes its votes.

Does the winner survive a ±50% attack on the weights?

If a ranking flips when the weights shift, the composite is telling you about the committee, not the plugins. So attack mine as hard as ±50% allows. Halve every weight where my plugin leads: Lifecycle to 12.5%, CLI, TypeScript, and Maintenance to 7.5% each, Hooks and Docs to 5% each. Raise the one dimension where my plugin loses — Bundle Weight — by half, to 15%. Renormalize so the weights sum to 1.0 again.

The composite becomes roughly 0.73 vs 0.42. The order holds, and not by luck: the community plugin leads exactly one dimension out of seven, so no ±50% reweighting exists in which one 10%-class dimension outvotes six others. Push past ±50% and, holding the other six weights in proportion, Bundle Weight needs roughly 43% of the whole composite — against its actual 10% — to flip the winner. The gap is score-broad, not weight-fragile.

That surviving loss deserves its own sentence. My plugin scores 0 on Bundle Weight — 78 KB unpacked against the community's 47 KB — and it's printed in the table above at full weight, in a benchmark I wrote, scoring a plugin I built, under weights I chose. That conflict of interest is real, and the two controls you can audit against it are the published loss and the sensitivity math you can redo on a napkin — the full self-benchmark process is in I Built What I Benchmark. A composite that survives its own worst dimension in public is worth more than one that never shows a loss.

If you'd have weighted it differently, don't argue in the abstract — the whole run is one command: npm run bench:caching from serverless/benchmarks, and this run's output sits at benchmarks/benchmark-results/api-gateway-caching/2026-05-04_v1.0/result.json in the repo. Change the weights, re-run it; if your reweighting flips the order, that's a finding and I want it in an issue. The strongest evidence that this harness follows the data even when it disagrees with me is the next article in this series: the time our loudest serverless claim died under its own E2E run — We Were Wrong About sls remove.

Quick reference

Question	Answer
What is the composite?	Weighted sum over 7 dimensions; weights sum to 1.0
Result (2026-05-04 run)	Interlace 0.88 vs community 0.3025; all 7 dimensions measured
Who sets the weights?	The author — editorial, published, versioned with the results
Weight tiers	Damage 25% · daily-use 15%×3 · comfort 10%×3
Mixed units?	Every dimension normalized to 0–1 before weighting
Unmeasurable dimension?	Null, never imputed; harness renormalizes over measured dims — n-of-7 label lives at the claims layer (this run: 7-of-7)
±50% adversarial reweighting	~0.73 vs ~0.42 — the order holds
The honest loss	Bundle Weight: 78 KB vs 47 KB → scored 0 vs 1, published at full weight
Versions measured	Interlace 0.0.0 (local source, unpublished) vs community 1.11.0 (npm)
Reproduce	`npm run bench:caching` in `serverless/benchmarks` → `benchmark-results/api-gateway-caching/`

External references

OECD/JRC — Handbook on Constructing Composite Indicators: Methodology and User Guide (2008) — the authoritative methodology reference for composite indicators; the normalization and sensitivity sections above are the single-benchmark version of its checklist.
Lighthouse performance scoring documentation — Google's published, versioned scoring weights: the working precedent for treating a weight change as a verdict change.

What's the largest weight you'd assign differently in that table — and which dimension would you demote to pay for it?

Related deep dives

Composite Scores and Weighting — the tier-0 canonical this benchmark instantiates: why composites exist, the honesty requirements, and the Lighthouse / OpenSSF Scorecard case studies
I Built What I Benchmark. Here's How I Try Not to Cheat. — the conflict-of-interest process behind every benchmark on this site
The Claims Registry — the claim→evidence-file contract that this run's result file feeds
Proxy Metrics — why registry-sourced dimensions measure stand-ins, not the thing itself
Valid vs Reliable Metrics — the difference a README scan can't close on its own

Star the serverless repo — you've just read exactly how its scores are built

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Marketing Pages Rot Silently. Mine Print an Expiry Date on Every Claim.

Ofri Peretz — Mon, 20 Jul 2026 04:43:26 +0000

Around 2026-08-01, a "verification pending" banner goes up on eleven claims in my serverless docs. Nothing will have broken. No competitor will have shipped. The banner goes up because every verified row in that repo's CLAIMS.md is stamped 2026-05-03 or 2026-05-04, the file gives evidence a 90-day warranty, and date arithmetic doesn't care that I wrote the rule.

Line 5 of that file:

If a claim doesn't have a row, it can't ship in the docs. If "Last verified" is older than 90 days, the claim is stale and gets a "verification pending" banner in docs until refreshed.

That line exists because marketing pages rot silently: a claim gets written on the day it's true, nobody dates it, and the sentence sits there aging in public. The fix is one file per product line — the claims registry: every public claim holds a row, every row points at a versioned result file, and every date is an expiry date.

This is the methodology piece — the contract, the lifecycle, the staleness rule — and none of it is JavaScript-specific. If your product page makes claims, it transfers in an afternoon.

What does a claim owe you before it ships?

The contract is deliberately boring. A claim backed by static evidence — a benchmark you can run on a laptop — carries four fields: the claim text exactly as it appears in docs or marketing, the suite that measures it, a link to the latest result file, and the date it was last verified. Verbatim matters: paraphrase is where scope creep hides — if the row is politer than the landing page, the landing page is the claim nobody verified.

One worked row from each product line:

Serverless. The claim "Composite score 88% vs community 30% (7 of 7 dimensions)" points at the api-gateway-caching suite and its result file latest.json, last verified 2026-05-04. Open the file: composite 0.88 for my plugin (measured from local source at v0.0.0, not a published release) against 0.3025 for the community serverless-api-gateway-caching@1.11.0, all seven weighted dimensions scored, including the one I lose outright: bundle weight, 78 KB unpacked against their 47 KB. The row carries the loss because the file does.

ESLint. The sister registry's row reads "Top of leaderboard on the ILB-Arena 40-vuln / 38-safe corpus (1st of 18 plugins tested, 17 security-relevant)" and points at the ilb-arena suite's dated result file, where summary.leaderboard[0].rank == 1. That row survived an audit its neighbor didn't: the registry had been asserting "97.6% precision, 100% recall, 98.8% F1" — figures that appear nowhere in the cited JSON, which reports 100% across the board. The 2026-05-13 audit withdrew the claim rather than substitute the perfect score — 100% on a 40-fixture self-authored corpus is the textbook "regression test, not benchmark" failure mode. The ordinal claim kept its row; the flattering one kept only its receipt, in a "Withdrawn claims" section. The suite column enforces validity — measure what the copy asserts; the result file makes the claim reproducible — regenerate the number without trusting me.

Claims only live infrastructure can verify get a second row flavor: claim text, the E2E step that asserts it, the last release verified. "Cache MISS on first request, HIT within TTL" maps to steps 4–5 of a deploy-and-assert run against real AWS, last verified 2026-05-03.

The contract also kills a third category — the adjective. "Battle-tested," "blazing fast," "production-grade": proxies for measurements nobody made. No suite outputs "battle-tested," so no row, so no ship. The registry doesn't argue with weak claims; it has nowhere to put them.

How does a claim live and die?

Draft → evidenced → stale → re-verified or retired — every transition a visible edit to one file.

Draft. The add-a-claim procedure opens with its most important sentence: don't write the marketing copy first. Build or extend the benchmark until it produces a measurable result for your product and at least one competitor, add the row with today's date, and only then write the copy. The rule is that blunt because my instinct runs the other way — when I draft a README, the adjective arrives before the evidence.

Pending. Unbacked copy isn't deleted — it's routed into a "Pending claims" section next to the suite that would back it. As of today my serverless registry holds 11 verified rows (4 static-evidence, 7 live-evidence) and 2 pending, both "Not started." Pending is the honest parking lot: marketing wishes become a benchmark backlog, priced in engineering hours instead of adjectives.

Re-scoped. The state nobody plans for: the measurement comes back and narrows the claim. Early copy in my serverless line implied we fix a ghost-billing bug in sls remove. Then the community-plugin E2E ran on 2026-05-04: sls remove came back clean — 31 seconds, exit 0, no orphans. The real trap is uninstalling the plugin while keeping the service. The original copy went into a re-scoped row — original text, what we measured, what's true — instead of being quietly rewritten (told in full here). The most useful rows in my registry are the ones that say I was wrong, in writing, next to the run file that proved it.

Retired is a re-scope with nothing left: no narrower version survives the measurement. The row stays; the docs copy goes.

When does a claim go stale — and who decides?

The misconception to kill: "claims go stale when the product changes." They don't. A claim goes stale when its evidence ages past its warranty — the product changing is just one way that happens. Your code can be byte-for-byte identical while the competitor you compared against ships three releases and the runtime you benchmarked goes end-of-life. The sentence didn't move; everything it pointed at did.

That's why staleness is declared, not discovered. Discovered staleness means a reader catches the drift, and the correction arrives after the damage. Declared staleness is date arithmetic: "Last verified" older than 90 days, banner up, mechanically, before anyone is misled. Ninety isn't sacred; it's a warranty one maintainer can honor — pick yours and enforce it the same way.

What keeps this from becoming a refresh treadmill is the version stamp. "eslint-plugin-security has 27.5% recall" rots — a standing claim about the present. "eslint-plugin-security v2.1.1 scored 27.5% recall on our 40-fixture corpus, on ESLint 8.57.0" is a dated measurement no future release can falsify — true forever. Financial statements solved this long ago: a balance sheet is always "as of" a date. Stamped measurements carry zero refresh obligation; only standing claims — "fastest," "most complete," anything in the present tense — sit under the 90-day clock.

The opening countdown is this rule running on its author: the banner lands around 2026-08-01 whether or not I get the re-runs done first. The ESLint registry is already there — one row, a synthetic-corpus speedup last verified 2026-01-02, carries the "re-verify recommended" flag today. That's the system working on me, twice.

How do you adopt this in an afternoon?

Nothing above depends on ESLint, serverless, or even software — the suite can be a load test, an accessibility audit, or a lab assay. The afternoon version:

Create CLAIMS.md at the repo root. The whole starting file:

   # Claims Registry

   > No row, no ship. "Last verified" older than 90 days →
   > "verification pending" banner in docs until refreshed.

   | Claim (verbatim from docs/marketing)   | Suite               | Latest result              | Last verified |
   | -------------------------------------- | ------------------- | -------------------------- | ------------- |
   | "Composite score 88% vs community 30%" | api-gateway-caching | benchmarks/.../latest.json | 2026-05-04    |

Inventory every public claim — README, landing page, package description — verbatim, no paraphrase.
For each, name the evidence (suite + latest result + date) or move it to Pending.
For everything new: evidence first, row second, copy last.
On refresh, commit the new result as a dated file (<suite>/<YYYY-MM-DD>_v<version>/) and bump the date — what you claimed, and when it was last true, becomes repo history.

This isn't an island. The closest neighbor is the OpenSSF Scorecard, which treats a repo's security posture as automated, publicly specified checks pointed at repositories from the outside. The claims registry is the manual, product-side sibling, pointed inward at your own marketing copy (corpus design covers the datasheets-for-datasets lineage). And it pairs with the conflict-of-interest discipline in I Built What I Benchmark: I built the products and the registry that polices them — a suspicious arrangement, which is why every row names the command that would catch me.

Next in this arc: We Were Wrong About sls remove — the re-scope told in full, because the lifecycle is easiest to believe when you watch it delete a claim I liked.

Quick reference

Question	Answer
What gets a row	Every claim in docs/marketing, verbatim — no paraphrase
Static-evidence row	Claim text · suite · latest result file · last-verified date
Live-evidence row	Claim text · E2E step · last release verified
Ship rule	No row → the claim cannot ship in docs
Staleness rule	"Last verified" > 90 days → "verification pending" banner until refreshed
Lifecycle	Draft → evidenced → stale → re-verified or retired; re-scoped when measurement narrows it
New-claim order	Evidence first, row second, marketing copy last
Unbacked copy	Routed to Pending with the suite that would back it — never into docs
Refresh mechanics	Re-run suite → commit dated result file (`<suite>/<YYYY-MM-DD>_v<version>/`) → bump date
The immortal form	Version-stamped: "vX scored Y (measured DATE)" — a dated measurement, true forever

The suite that generates the ESLint result files is public — go check:

⭐ Star the benchmark suite — the result files this registry points at

Before the clock catches yours: what's the oldest undated claim on your product page right now — and would it survive a re-run today?

References

OpenSSF Scorecard — the Open Source Security Foundation's automated checks on open-source repos; cited as the closest prior art for treating claims as named, publicly specified checks rather than prose.
Gebru et al., Datasheets for Datasets (arXiv:1803.09010, later published in CACM) — cited for the documentation-contract idea the registry applies to marketing claims instead of datasets: standardized fields, stated provenance, explicit scope.

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Fixtures First, Rules Second: How to Design a Ground-Truth Corpus

Ofri Peretz — Mon, 20 Jul 2026 04:43:02 +0000

'SELECT * FROM users WHERE id = ' + userId is a vulnerable fixture — no debate. Wrap the input in parseInt(userId) — vulnerable or safe? The label depends entirely on your threat model, and whichever way the author calls it, that call is the ground truth. The corpus behind the Interlace security benchmarks took 40 calls like that one; parseInt is the call I've gone back and forth on longest — it's still not in the corpus, which is itself a judgment call. And the cost of getting the discipline wrong isn't hypothetical: this corpus scores its own author's tool at a perfect 100% — a number that reads as evidence only because the fixtures demonstrably came first, and as a mirror otherwise. That discipline — categories, borderline labels, and the sequencing rule — is the rest of this article.

Why do fixtures have to come before rules?

Write the corpus after the tool, with one eye on what it already detects, and the fixtures inherit its blind spots — the benchmark stops measuring and starts reflecting. That's Goodhart's Law operating at design time.

The defense is sequencing: this corpus was designed against published OWASP categories and CWE mappings before any Interlace rule existed to cover it, and its 38 safe patterns represent realistic validated code, not Interlace's allow-listing logic. Sequencing bounds the bias, not removes it — the same person still wrote both sides of the test.

How do you choose categories — and how many of each?

Anchor categories to a public taxonomy, not instinct. The vulnerable corpus is 40 exported fixture functions across 14 categories, each mapped to a CWE ID — from four SQL-injection fixtures (CWE-89) down to a single open redirect (CWE-601), 17 distinct CWE IDs in total. Paired with those are 38 safe fixtures — 78 in all ("the 40-fixture corpus" names the vulnerable half; precision is computed against the safe half).

Prior art: NIST's SARD is the canonical labeled vulnerable-code corpus family — the Juliet suites cover C/C++ and Java, with PHP and C# suites alongside; OWASP's Benchmark Project is the closest scored SAST analog; and SecBench.js (ICSE 2023) labels ~600 real-world server-side JavaScript vulnerabilities with executable exploits. What none of them offered Node.js-native is an OWASP-Benchmark-style scored SAST leaderboard — this corpus is that analog, smaller.

One distortion to own out loud: the corpus is roughly balanced across categories and production code is not — real vulnerabilities are rare, so a balanced corpus says nothing about the false-alarm rate at production base rates. It measures detection per pattern; prevalence needs different evidence.

How do you label the edge cases?

The parseInt label is an author judgment inside the CWE-89 definition, and it earns an annotation carrying its reasoning: borderline; would be vulnerable if the query used string interpolation instead of the typed value. The generalizable rule: record the reasoning, not just the verdict. Six months later, a label without reasoning is an opinion with tenure.

This corpus has one rater, and inter-rater agreement needs at least two — there is no kappa to report, only me agreeing with myself. That's the weakest joint in the design, and why the reasoning must be public: so a second rater can show up later and disagree precisely.

Labels live with the code: a manifest at the fixture file's bottom (EXPECTED_DETECTIONS: function → {cwe, severity}; EXPECTED_NO_DETECTIONS: the safe list) is what the runner imports to score TP/FP/FN/TN — no spreadsheet drifting out of sync.

How do you make a corpus result reproducible?

A corpus result is a claim about specific tool versions — the unicorn false-zero in my own benchmark came from a stale Node version in the shell, not from the plugin. Reproducibility rests on three mechanisms, none of them "trust the manifest": a lockfile pins the install; the runner resolves each plugin's actually-installed version at run time and writes it into the results JSON; and the environment ships with the numbers — per run, inside each results file, because runs happen on different days on different Nodes. The Interlace 3.0.2 run behind the 40 TP / 0 FP / 0 FN line records Node v24.12.0, ESLint 9.39.2, measured 2026-05-30; the sonarjs 3.0.6 run (14/40) records the same environment a day earlier, 2026-05-29. One plugin (eslint-plugin-security 2.1.1) crashes on ESLint 9 and runs in a separate compat package — it declares eslint ^8.57.0 and lets its lockfile hold the exact version — recorded, not smoothed over.

The other half is publishing: corpus public, divergence channel public, and reproduction that needs zero help from me:

git clone https://github.com/ofri-peretz/eslint-benchmark-suite.git
cd eslint-benchmark-suite && npm install
npm run benchmark:fn-fp   # → node benchmarks/fn-fp-comparison/run.js

Run it, get a different number, open an issue — your results JSON carries its own environment block, so the disagreement starts from evidence, not recollection. A corpus nobody can re-run is a claim, not a benchmark.

What can 40 fixtures actually prove?

The 78 fixtures are a constructed set, not a random draw, so results are descriptions, not inferences — no p-value is meaningful at any corpus size, because statistical power assumes a sampling process this design doesn't have. What n=40 is good at is category-level findings: a plugin catching 14 of 40 while missing whole categories is a real, checkable statement about that plugin at that version. Don't decorate results with statistics the design can't carry.

What is the corpus lifecycle?

The lifecycle: design → publish → saturate → refresh.

Saturation is where mine is now: Interlace v3.0.2 scores 40 TP, 0 FP, 0 FN — 100/100/100 precision/recall/F1. The tempting read is "the tool is finished." The correct read: the corpus is finished, for that tool — a saturated corpus can't measure the improvement of the tool that saturates it. Publishing that line felt less like winning and more like watching the instrument go quiet. The corpus still measures competitors (next best: 14 of 40, eslint-plugin-sonarjs 3.0.6), but for Interlace it's now a regression test, not a benchmark.

Then contamination: public fixtures can be tuned against — by any tool, mine included. Public corpora decay the way public trading signals do: once everyone sees the signal, trading on it erases the information it carried. The answer is versioning, not secrecy (secrecy kills reproducibility): treat the published corpus as v1 and plan a harder v2 held to the same fixtures-before-rules discipline — anchored to the taxonomy, not to any tool's known behavior, Interlace's misses included.

Which findings survive a robustness check?

Before publishing, run leave-one-category-out: re-score the leaderboard 14 times, once per removed CWE category, and check whether the ordering survives every drop. A 40/40-versus-14/40 gap survives any single removal by construction; two mid-table plugins a fixture apart may swap the moment a category disappears — that's the ranking to report as fragile, or not as a ranking at all. Cheap, mechanical, and almost nobody does it.

What are Fixture Cards?

The last discipline is documentation, and it deserves a name: Fixture Cards — per-fixture documentation modeled on Datasheets for Datasets (Gebru et al.) and Model Cards (Mitchell et al.), scaled down to the single labeled example. Six fields:

Field	The `parseInt` fixture's card would read
CWE	CWE-89
Label	threat-model-conditional: vulnerable under "any non-parameterized query is unsafe"; safe under "direct injection vector only"
Reasoning	borderline; would be vulnerable if the query used string interpolation instead of the typed value
Author	Ofri Peretz
Date	the day the call is made, pinned to the corpus version it enters
Disputed?	open — threat-model-dependent

Four of the six fields already exist in the suite: CWE and label live in the manifest the runner imports, author and date in the fixture file's git history. The Card names that standard and adds the two fields no runner ever checks — reasoning and dispute status. Packaging all six as a JSON schema is the remaining step, and it doubles as the contribution unit: to submit a fixture is to submit its Card. Labeling debt stops accumulating silently, and the single-rater problem gets its structural fix — every disputed? field is an open seat for a second rater.

The parseInt Card above is the first open seat, and it's yours if you want it: comment with your label — vulnerable or safe — and the threat model that produces it. That's the second rating this corpus doesn't yet have.

Quick reference

Design decision	The rule	If you skip it
Sequencing	Fixtures before rules	The benchmark becomes a mirror
Category selection	Anchor to CWE/OWASP (here: 14 categories, 17 CWE IDs)	Blind spots inherited from instinct
Balance	Balanced corpus — and say what it can't estimate	Base-rate overclaims about production FP rates
Edge-case labels	Record reasoning with the label; mark borderline calls	Labels become opinions with tenure
Machine-readable labels	Manifest lives in the fixture file; runner imports it	Spreadsheet drift
Versions	Lockfile + per-run resolved versions in the results	Irreproducible numbers
Publishing	Public corpus + a divergence channel	A claim, not a benchmark
Sample claims	n=40 describes; it does not infer	Fake statistical rigor
Lifecycle	Version the corpus; plan v2 at saturation	100% scores misread as "done"
Robustness	Leave-one-category-out before publishing	Fragile rankings published as real
Fixture Cards	CWE · label · reasoning · author · date · disputed?	Labeling debt accumulates silently

Corpus design is the unglamorous half of benchmarking — nobody stars a repo for a well-reasoned borderline label. But every downstream precision and recall number is only as honest as these decisions. Next in this arc the corpus does its job: the false-positive/false-negative benchmark scores six plugins against these 78 fixtures, wins and losses included.

If this is the discipline you'd want behind a benchmark you trust, the corpus — fixtures, manifests, runner, results — is public:

⭐ Star the benchmark suite — every fixture and label in this article is in it

References

NIST SARD — Software Assurance Reference Dataset (incl. the Juliet Test Suite) — the canonical labeled vulnerable-code corpora for C/C++/Java; this corpus is the Node.js-native analog, at a fraction of the scale.
OWASP Benchmark Project — the closest prior art overall: a public, scored SAST benchmark on the same TP/FP mechanics.
Gebru et al., "Datasheets for Datasets" (CACM 2021) — the dataset-documentation standard Fixture Cards are modeled on.
Mitchell et al., "Model Cards for Model Reporting" (FAT* 2019) — the same documentation move at model level; Fixture Cards scale it down to the single labeled example.

Related deep dives

Ground Truth in Security Testing: Who Decides What's Vulnerable? — the T1 vocabulary this article builds on, including the full parseInt labeling story
I Built What I Benchmark. Here's How I Try Not to Cheat. — the conflict-of-interest process this corpus discipline lives inside
1.5M Weekly Downloads, 1 False Alarm per Real Bug — the evidence article produced by this corpus

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Valid vs. Reliable Metrics: Consistent Numbers Can Still Be Wrong

Ofri Peretz — Sun, 19 Jul 2026 23:47:32 +0000

A clock that runs ten minutes fast is the friendly kind of broken: it shows the same answer every time you look, so it feels dependable — right up until it makes you miss the train. The clock was never inconsistent. It was consistently wrong, which is the harder problem, because consistency is the part that looks like quality. That gap between steady and correct is the oldest distinction in measurement theory, and it's the one most dashboards quietly skip: a number can be perfectly reliable — the same value on every read — and still be invalid — pointed at the wrong quantity. Precision of the wrong thing is still the wrong thing.

Reliability versus validity applies identically whether you weigh yourself, price a stock, score a diagnostic test, grade a diver, or benchmark software. Reliability is a property of the instrument; validity is a property of the claim you attach to its number. They are independent — you can have either without the other — and telling them apart is the whole job.

What Makes a Metric Reliable?

A metric is reliable when repeated measurement under the same conditions returns the same number. A scale that reads 82.0 kg five times in a row is reliable. A lab assay that returns 5.4 mmol/L on the same blood sample, run three times, is reliable — same sample, same number. Reliability never asks whether 82.0 or 5.4 is correct; only whether the instrument will say it again.

Reliability is a property of the instrument, not of the truth. That makes it the cheap property to check: you don't need to know anything about the world — take the measurement twice and compare. It's also the property people naturally police, because unreliability is loud. A blood-pressure cuff that gives a different reading every squeeze, a poll that swings ten points between identical samples, a stopwatch that disagrees with itself — a wobbling number announces itself, gets questioned, gets fixed.

That loudness is exactly what makes reliability feel like quality. It isn't. It's half of quality, and it's the easy half.

What Makes a Metric Valid?

A metric is valid when it measures the concept you claim it measures. Psychometricians call this construct validity: the "construct" is the real thing you care about — health, aptitude, risk, security — and validity is how faithfully your number tracks it.

Three general examples, in increasing order of how much they cost when they're wrong:

A scale that is 2 kg off reads 82.0 kg every time while you actually weigh 80. Perfectly reliable, invalid for "my actual weight." A calibration error doesn't reduce consistency at all — which is precisely why it survives.
Standardized tests produce famously reliable scores: retake next week, get nearly the same number. The century-old argument is entirely about validity — whether the score measures aptitude or measures practice at taking that kind of test.
A call center reports 95% of calls answered within 20 seconds — cleanly measured, stable month to month. But the customers who gave up before connecting were never in the denominator. The metric is valid for "answered calls, handled fast" and invalid for "customers who could reach us," and the gap between those two is where the complaints live.

This framing isn't mine. Messick's 1989 treatment made construct validity the organizing idea of measurement in the social sciences, and Jacobs & Wallach ported the framework to computational systems in 2021. The practitioner translation: every number you publish is implicitly a validity claim, whether you meant to make one or not.

The hard part is that you cannot check validity by re-running. Reliability is verified from inside the measurement; validity needs an independent route to the construct — a second instrument you trust, a real-world outcome, a ground truth established some other way. You can be certain a number is stable and have no idea whether it is true.

Reliable, Valid, Both, or Neither?

The standard picture is a dartboard. Where the darts land is your metric; the bullseye is the construct.

Tight cluster on the bullseye — reliable and valid. The goal.
Tight cluster in the wrong corner — reliable, invalid. The dangerous quadrant, because consistency masquerades as correctness. Nobody audits a number that never wobbles.
Scattered around the bullseye — valid on average, unreliable. Annoying, but self-announcing: the wobble is visible, so it gets fixed.
Scattered and off-target — neither. Usually caught fast, because nothing about it looks trustworthy.

Here is the misconception this article exists to delete: "a metric that gives consistent numbers is a good metric." Consistency is reliability; whether the number measures the right thing is validity — and they are independent. You can have either without the other, and quadrant 2 is where the expensive failures concentrate.

My clearest picture of quadrant 2 is a silent zero. A fraud filter flags 0 of roughly 50,000 transactions every night for a week — the same 0 on every run, perfectly reproducible — because a data feed upstream had gone null and the model was scoring empty rows. The real fraud rate that week was about 0.8%. Nothing wobbled, so nothing got investigated; the stability was the disguise. A number that never moves is not the same as a number that is right.

One more force pushes metrics toward quadrant 2: optimization. When a number becomes a target, people and processes make it more stable while draining its meaning — the number stays crisp as its connection to the construct erodes. That mechanism has its own article on Goodhart's law.

Is One Valid Metric Enough? Construct Coverage

Validity is a property of a metric. Construct coverage is the property of your metric system: it is valid only if every failure mode that matters has at least one metric watching it.

The expensive failures are usually not a bad metric — they're a missing one. A company can report revenue, gross margin, and burn rate — each audited, each reliable, each valid for its own construct — while no line anywhere tracks customer concentration. Then the single client that was 40% of revenue walks, and every number on the healthy dashboard was true right up to the moment it stopped mattering. A coverage hole is invisible from inside the dashboard, because every number you do have looks fine. The metrics were excellent at answering questions nobody was about to ask.

The audit generalizes to anything you ship or promise:

List the claims. Everything a spec sheet, a contract, a label, or a marketing page asserts — "safe under load," "accurate to ±1%," "no side effects," "holds up outside the sample it was built on."
For each claim, name the check that would falsify it. Not the check that sits near it — the one that fails if the claim is false. A "survives a crash" promise needs a test that actually triggers the crash; a "works after the upgrade" promise needs a run on the upgraded system, not the one you happened to have.
The diff is your exposure. Every claim without a falsifying check is a promise you are making on vibes.

That's the constructive move this whole distinction buys you: reliability tells you your instrument is steady; validity tells you it's pointed at something true; coverage tells you nothing important sits unwatched. Tom DeMarco needed a public recantation in 2009 to walk back decades of "you can't control what you can't measure" — measurement humility from one of the people who taught the industry to measure. The practical version of that humility is small: before you trust a stable number, ask what question it actually answers. Then ask which questions have no number at all.

Quick Reference

Property	Question it answers	How to check	Failure smell
Reliability	Does re-measuring give the same number?	Take it twice under identical conditions	Wobbling numbers — loud, self-announcing
Validity	Does the number measure the concept claimed?	Independent route to the construct (second instrument, real outcome, ground truth)	Stable dashboard, surprised stakeholders
Reliable + invalid	—	The dangerous quadrant: consistency masquerading as correctness	A number nobody audits because it never wobbles
Construct coverage	Does every claim have a metric watching it?	List claims → name the falsifying check for each → diff	A stated guarantee with no check pointed at it

Goodhart's Law, Explained — the mechanism that drives metrics into the reliable-but-invalid quadrant.
Proxy Metrics — every proxy has a validity gap; that gap is this article's subject, measured.
Composite Scores and Weighting — what happens to validity when you collapse many metrics into one.
Different Metrics for Different Package Types — the software worked example this idea was drawn from: a real construct-coverage hole, and the metric system built to close it.

If this is the kind of reference you'll want at your next metrics review, follow me on Dev.to and bookmark it — it's built to be cited, not skimmed once.

References

Messick, S. (1989). "Validity." In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13–103). American Council on Education / Macmillan. The authoritative construct-validity treatment; the source of the modern view that validity is about the interpretation of a score, not the score itself.
Jacobs, A. Z., & Wallach, H. (2021). "Measurement and Fairness." Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT '21). arxiv.org/abs/1912.05511. Measurement theory applied to computational systems — the bridge between psychometrics and the numbers you actually ship.
Trochim, W. M. K. Research Methods Knowledge Base. conjointly.com/kb. Free online reference; the fastest concept-check for reliability/validity vocabulary, including the dartboard figure.
DeMarco, T. (2009). "Software Engineering: An Idea Whose Time Has Come and Gone?" IEEE Software, 26(4). The recantation of "you can't control what you can't measure" — metrics humility from one of the field's original measurement advocates.

Foundations series: ← Inter-Rater Agreement & Cohen's κ · hub · Proxy Metrics →

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Static Analysis vs. SAST vs. Linting: The Taxonomy That Matters for Security Teams

Ofri Peretz — Sun, 19 Jul 2026 23:47:10 +0000

When a security dashboard files a single-file linter under "SAST tools," it's using "SAST" as a marketing umbrella for anything that inspects source code without running it — technically defensible, practically misleading.

A lint rule fires in your editor in under a second. CodeQL runs on a pull request in minutes and can trace a tainted value through a dozen function calls across as many files. SonarQube keeps a persistent model of your codebase and tracks how an issue count moves across releases. Different scopes, different failure modes, different jobs in the pipeline — squeezed into one word that tells you none of it.

The taxonomy

Three levels, broadest to narrowest.

Static analysis is any examination of source code without executing it — abstract interpretation, control-flow analysis, taint analysis, and everything below all qualify. Every tool in this piece is static analysis; the word alone tells you nothing about what a given tool actually catches.

SAST (Static Application Security Testing) narrows that to security vulnerabilities specifically, and adds three defining traits: it runs in CI/CD rather than at edit time, it traces values across function and file boundaries (interprocedural analysis), and it reports a finding with severity, CWE classification, and the data-flow path that proves it. CodeQL, Semgrep, and SonarQube Security are the common names.

Linting narrows further, to syntactic and pattern-level checks meant to run while you're still writing the code. A lint rule works on a single file's AST — the pluggable, per-file architecture Nicholas C. Zakas built ESLint around in 2013 — with no cross-file data flow, and it has to finish in milliseconds to be worth having in the editor. It reports patterns: style violations, correctness bugs, and the security mistakes that are visible from inside one file.

Security linting — the category a dedicated security linter belongs to — is linting aimed at security patterns specifically: same edit-time speed, same single-file scope, but pattern-matching for constructs like string concatenation at a SQL call site, Math.random() generating a token, or a missing JWT verify call. Security linting at edit time. Not a SAST replacement.

What each level detects

The differences get concrete on one vulnerability, shown two ways.

The pattern a security linter catches

// vulnerable.js
app.get('/users/:id', (req, res) => {
  const query = 'SELECT * FROM users WHERE id = ' + req.params.id;
  //                                                 ^^^^^^^^^^^^^^^^^^
  //            ESLint sees: string + user-controlled value at a db.query call
  pool.query(query).then(r => res.json(r.rows));
});

The linter sees the concatenation at the call site — source (req.params.id) and sink (pool.query) both visible in the same file, adjacent lines. A security rule fires on the string-built SQL and attaches CWE-89. The developer sees the warning before they commit.

The multi-hop path that needs taint analysis

// routes/users.js
import { buildQuery } from '../db/query-builder.js';

app.get('/users/:id', (req, res) => {
  const userId = req.params.id;                // source: user-controlled
  const sanitized = userId.trim();              // looks sanitized — only strips whitespace
  const query = buildQuery('users', sanitized); // tainted value crosses file boundary
  pool.query(query).then(r => res.json(r.rows));
});

// db/query-builder.js
export function buildQuery(table, id) {
  return `SELECT * FROM ${table} WHERE id = ${id}`;  // template injection
}

A lint rule scanning routes/users.js sees userId.trim() and a function call — the concatenation itself sits one file away, inside buildQuery(). The call site looks clean; the rule doesn't fire. (And it isn't safe: .trim() removes whitespace, not SQL — 1; DROP TABLE users-- passes through untouched and reaches the template intact. Had the code used parseInt(userId, 10) the payload really would have collapsed to the integer 1; the trap is the sanitizer that only looks like one.) CodeQL traces req.params.id → userId → sanitized → query inside buildQuery → pool.query, and files a CWE-89 finding with the full path as evidence.

Same vulnerability, two different visibility ceilings — not a defect in either tool, just two levels of the taxonomy doing what they're built to do.

Time-in-pipeline analysis

Capability isn't the only axis that matters — timing is. Security linting fires at edit time, under a second. A light SAST pass on a PR takes one to five minutes. A deep nightly scan runs thirty minutes to hours. That gap changes what a developer actually does with a finding: a three-minute CI wait means they've already pushed the branch and moved on to reviewing someone else's PR by the time the result lands. A half-second lint warning arrives while their hands are still on the keyboard, in the same context as the line that triggered it.

Security linting and SAST aren't competing for the same job. They sit at different moments in the workflow, at different depth, for different cost — which is why a mature pipeline runs both instead of picking one.

Why a security linter is not a SAST tool

A security linter carries the constraints of its category on purpose.

Speed and scope. Rules run in milliseconds, on one file's AST, because that's what edit-time feedback requires — any rule needing cross-file data flow would break the guarantee. A linter can't trace userInput through five function calls across three files; it pattern-matches on argument names, import shapes, and option keys instead of building a program-semantics model.

Domain depth over breadth. Where a security linter earns its keep is depth inside specific library surfaces: JWT verification via jsonwebtoken, parameterized queries via pg, SSRF sinks in node-fetch and axios, unsafe deserialization in serialize-javascript. The rule knows the library's dangerous shapes and pattern-matches them within a single call site — narrow, but precise where it looks. That's the honest claim: security linting at edit time, covering what linting can structurally detect — not a replacement for CodeQL, Semgrep, or SonarQube's interprocedural coverage.

A vendor's "SAST" label isn't false — at maximum breadth "SAST" does mean "any source-code security analysis." I've stopped being annoyed that it's technically defensible: a tide pool and the open ocean are both "water," and that stays true right up until you have to decide where it's safe to swim. What the label discards is the one distinction that tells you which tool to reach for — a pattern match at edit time, or a proven data-flow path at CI time.

The precise label is security linter: lint speed, lint scope, right for edit-time and CI, not a substitute for taint analysis. Where that structural ceiling comes from, and how precision and recall diverge under it, is the subject of Taint vs. Heuristic Detection and Precision, Recall, and F1 for Static Analysis.

The recommended stack

A complete Node.js security pipeline uses four positions — not redundant, each catches a different category at a different cost.

1. Edit time — security linting (ESLint plus a security plugin). Catches obvious single-file patterns at developer speed, zero CI wait.

2. Pre-commit / pre-push — CI linting. Same rules, enforced as a hard gate — catches what a skipped editor integration let through.

3. PR check — SAST (CodeQL or Semgrep). Taint analysis across the diff and its call graph; catches multi-hop injection that spans files and imports, before merge.

4. Nightly / weekly — deep SAST or DAST (SonarQube, Checkmarx, OWASP ZAP on staging). Full-codebase analysis, drift tracking across releases, and — for DAST — the runtime behaviors no source read can reach.

Teams that skip positions 1–2 pay for 3–4 to catch what the editor could have blocked for free. Teams that skip 3–4 leave every multi-hop taint path undetected. A complete pipeline runs all four.

Named misconception: "ESLint security plugins are SAST tools." They're security linters — the subset of static analysis that runs at lint speed and lint scope, without interprocedural data flow or taint tracking. The confusion comes from vendors using "SAST" as an umbrella term for any source-code security analysis, which is technically accurate and practically uninformative.

Quick reference

Category	Example tools	When runs	Cross-file	Taint tracking	Speed
Linting	ESLint, Pylint, RuboCop	Edit time	No	No	Under 1 sec
Security linting	eslint-plugin-security, Bandit	Edit time	No	No	Under 1 sec
SAST — light	Semgrep, CodeQL default	PR check	Yes	Partial	1–5 min
SAST — deep	Checkmarx, Veracode, Snyk Code	Weekly	Yes	Yes	30 min+
DAST	OWASP ZAP, Burp Suite	QA / staging	N/A	N/A	Minutes–hours

DAST (Dynamic Application Security Testing) tests a running application, not source — out of scope here, but it catches categories, auth state, session management, server-side rendering injection, that no source-code read can reach.

References

Chess, B., & West, J. (2007). Secure Programming with Static Analysis. Addison-Wesley. The foundational industrial treatment of static analysis for security — data-flow and taint techniques, rule design, and the false-positive economics that separate a linter from a SAST engine.
NIST IR 8397: Guidelines on Minimum Standards for Developer Verification of Software. National Institute of Standards and Technology, 2021. nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8397.pdf. Note that NIST itself uses "static analysis" and "SAST" interchangeably here — even listing ESLint among its example SAST tools — which is itself a data point on how loosely the industry applies the label.
OWASP Source Code Analysis Tools. owasp.org/www-community/Source_Code_Analysis_Tools. OWASP's community-maintained list of source-code (SAST) analysis tools. For the automated-analysis-vs-manual-review distinction specifically, see OWASP's Static Code Analysis control page.
Livshits, B., & Lam, M. S. (2005). Finding security vulnerabilities in Java applications with static analysis. USENIX Security Symposium, 14. The structural basis for what makes taint analysis different from pattern matching.
Zakas, N. C. (2013, July 16). Introducing ESLint. humanwhocodes.com. https://humanwhocodes.com/blog/2013/07/16/introducing-eslint/. The original announcement of ESLint's pluggable, per-file rule architecture — the design decision this whole taxonomy hangs on.

This is the reference for the next time a scan result gets called "SAST" and it's actually a lint rule — bookmark it, and follow me on Dev.to for the rest of the Foundations series.

Foundations series: ← Taint vs. Heuristic Detection · hub · start over: The Confusion Matrix →

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

DEV Community: Ofri Peretz

I Linted My Own Design System. All 8 Breaks Were Pasted In.

Raw color literals: 30 findings, and they cluster

data-slot: 67 findings, the biggest by far

The pattern

The config

Your Token Is Not Safer in a Cookie. It Is Safer From JavaScript.

The axis everyone argues about

The axis that decides it

HttpOnly is the boundary

The corollary people skip

So what counts as sensitive

And the CSRF half?

Where this lands in the standards

Why the argument keeps happening

innerHTML Has Five Doors. Most Reviews Only Watch One.

Sinks are few. Sources are many.

The five doors

The word "ours" is doing all the damage

Why generated code lands here

The taxonomy is the useful artifact

What it means for detection

Two branches, not one chain

The fix is boring, which is the point

Nobody Writes Bad Crypto. They Write Correct Crypto at Four Layers.

Four layers, each hiding the next

Why layers beat a rule list

Detection: the easy layers are the trap

Where this lands in the standards

Why generated code lands on layers 2 and 3

I Audited 203 of Our Own ESLint Security Rules. 16% Mislabel Their Own CVSS Score.

What the three labels actually measure

Case 1: a 9.1 that prints LOW — and the honest reason it might be fine

Case 2: a 9.8 with no heuristic excuse at all

Case 3: the one that runs the other way

Why this happens: severity is the one field nobody enriches

The audit, if you want to run it yourself

Reading a severity label like a security engineer, not a triage bot

The config

142,076 Weekly Downloads. Zero Releases Since 2021. Is the Niche Defended?

How do you build an incumbent map?

What does the serverless incumbent map actually show?

When is a quiet package finished instead of dead?

How were the replacement targets actually picked?

Quick reference

References

Related deep dives

We Were Wrong About sls remove

What was the original claim?

What did the measurement actually show?

What is the re-scoped claim?

Why is a narrower claim stronger?

Quick reference

References

Related

25% of My Benchmark Verdict Is an Opinion. Here's the Anatomy.

What are the seven dimensions — and where does each number come from?

How do kilobytes, hook counts, and pass rates become one scale?

What happens when a dimension can't be measured?

Why is Lifecycle Correctness worth 25% — and who decided?

Does the winner survive a ±50% attack on the weights?

Quick reference

External references

Related deep dives

Marketing Pages Rot Silently. Mine Print an Expiry Date on Every Claim.

What does a claim owe you before it ships?

How does a claim live and die?

When does a claim go stale — and who decides?

How do you adopt this in an afternoon?

Quick reference

References

Fixtures First, Rules Second: How to Design a Ground-Truth Corpus

Why do fixtures have to come before rules?

How do you choose categories — and how many of each?

How do you label the edge cases?

How do you make a corpus result reproducible?

What can 40 fixtures actually prove?

What is the corpus lifecycle?

Which findings survive a robustness check?

What are Fixture Cards?

`data-slot`: 67 findings, the biggest by far

`HttpOnly` is the boundary

Why this happens: `severity` is the one field nobody enriches