DEV Community: Ofri Peretz

I Audited 203 of Our Own ESLint Security Rules. 16% Mislabel Their Own CVSS Score.

Ofri Peretz — Tue, 28 Jul 2026 04:59:46 +0000

Run eslint-plugin-node-security against a Server-Side Request Forgery
pattern and the terminal prints this:

src/fetch-avatar.ts
  4:7  warning  🔒 CWE-918 CVSS:9.1 | HTTP call whose URL argument name
              suggests user input. | LOW

CVSS 9.1. Read the number on the calculator and it's nine tenths of a point
from the maximum possible score a vulnerability can get. Read the word next
to it and it says LOW. Those two facts are printed four characters apart
on the same line, by the same rule, and they disagree with each other.

I write the CWE/CVSS/OWASP metadata that ships in every one of our lint
messages — it's meant to be the thing that tells you, at a glance, how
worried to be. So I went and checked whether it actually does that. I wrote
a 25-line script that reads every rule in the Interlace ESLint ecosystem,
pulls its CVSS score and its shipped severity label, and checks whether the
label matches the official band the score falls in.

33 of 203 rules — 16% — don't. Some of those are defensible once you
know why. Some aren't. All of them are a good excuse to explain what CWE,
CVSS, and OWASP actually measure, because the gap between them is exactly
where lint output stops being trustworthy and starts being decoration.

What the three labels actually measure

A single line of our lint output packs in three unrelated standards. Quick
definitions — there's a full reference for each if you need it:

CWE names what kind of bug it is. CWE-918 is SSRF; CWE-287 is Improper Authentication. A category, not a verdict.
CVSS is a computed 0.0–10.0 score. Its four bands are the ruler this audit uses: 0.1–3.9 Low, 4.0–6.9 Medium, 7.0–8.9 High, 9.0–10.0 Critical. The number is the primary source; the band name is derived from it.
OWASP Top 10 is a category bucket (A01–A10) — an address, not a severity measurement.

Only CVSS is actually a severity measurement. So when a lint rule prints a
severity word, that word has exactly one legitimate source: the CVSS score
next to it. If they disagree, the word is wrong.

Case 1: a 9.1 that prints LOW — and the honest reason it might be fine

no-ssrf flags HTTP calls where the URL argument's name looks
user-supplied (userUrl, req.query.endpoint, targetUri) — a classic
Server-Side Request Forgery
setup, the bug class behind more than one well-known cloud metadata-endpoint
breach:

// flagged — the *name* suggests user input, not a data-flow trace
async function fetchAvatar(userUrl) {
  return fetch(userUrl);
}

The rule's own metadata: cwe: 'CWE-918', cvss: 9.1 — squarely in the
Critical band. The message it prints: severity: 'LOW'.

Here's the part that's genuinely defensible: the rule's own doc comment
calls itself out as "a naming heuristic, not data-flow analysis." It
matches on identifier names, not on whether the value actually reaches an
attacker-influenced source. That's a real limitation — plenty of
userUrl-named parameters are perfectly safe, admin-configured constants.
A low-confidence match that could be a
false positive
does deserve a lower-urgency label than a rule that traced the actual
taint.

The part that isn't defensible: confidence and impact are two different
axes, and severity is only supposed to encode one of them. "How sure are
we this is real" and "how bad is it if it's real" are independent
questions — a low-confidence match on a 9.1-impact bug is still worth
more attention than a high-confidence match on a 3.0. Collapsing both into
one LOW/MEDIUM/HIGH/CRITICAL string means the reader can't tell
which axis they're looking at, and a grep-based triage pass (or an
LLM agent doing the same) will deprioritize a Critical-impact finding on the
strength of a label that was only ever describing detection confidence.

Case 2: a 9.8 with no heuristic excuse at all

require-audience-validation and require-issuer-validation aren't naming
heuristics — they check one deterministic fact: does a jwt.verify() call
pass an audience (or issuer) option? Yes or no, no ambiguity:

// flagged — no audience claim checked
jwt.verify(token, publicKey);

// not flagged
jwt.verify(token, publicKey, { audience: "https://api.example.com" });

Skip audience validation and a JWT minted for any service that trusts the
same signing key gets accepted by this service too — a textbook confused
deputy, CWE-287,
cvss: 9.8. Same story on the issuer side. Both rules print severity: 'MEDIUM'.

There's no false-positive story to hide behind here — the check is exact.
9.8 is a hair under the maximum a CVSS score can reach. MEDIUM sits two
whole bands below where that number lives. If a reader — or a CI gate that
filters on severity string instead of CVSS number — treats MEDIUM findings
as "fix this sprint," a near-maximum-severity auth bypass just got queued
behind a color contrast warning.

Case 3: the one that runs the other way

Not every mismatch under-states. no-unsafe-search-path
— the PostgreSQL search_path hijacking rule I wrote up in full elsewhere —
ships cvss: 7.5 (High band) but prints severity: 'CRITICAL', one band
above its own score. I won't re-run the attack walkthrough here — the
short version, if a label disagreeing with a number bothers you as much as
it bothers me: read that piece for the exploit, come back here for why the
label drifted in the first place.

Why this happens: `severity` is the one field nobody enriches

Every rule funnels its message through one formatter,
formatLLMMessage(), and every formatter call goes through the same
enrichment step first:

// packages/eslint-devkit/src/messaging/formatters.ts
function enrichFromCWE(options) {
  if (!options.cwe) return options;
  const cweData = CWE_MAPPING[options.cwe];
  if (!cweData) return options;

  return {
    ...options,
    owasp: options.owasp ?? cweData.owasp,
    cvss: options.cvss ?? cweData.cvss,
    compliance: options.compliance ?? CWE_COMPLIANCE_MAPPING[options.cwe],
  };
}

There's a canonical lookup table, CWE_MAPPING, and it's correct — its
entry for CWE-918 is { cvss: 9.1, severity: 'CRITICAL' }, internally
consistent, right band. CWE-287 maps to { cvss: 9.8, severity: 'CRITICAL' }. If every rule simply deferred to this table, none of the 33
mismatches would exist.

But look at what enrichFromCWE actually forwards: owasp, cvss,
compliance — and that's the whole list. severity isn't in it. Every
cvss number in our output either comes from this vetted table or from an
explicit override that a human cross-checked against it. Every severity
string is whatever word the rule's author typed into that one call site,
with nothing checking it against the table, the CVSS number, or any other
rule's choice for the same CWE. Two rules can share a CWE, share a CVSS
score, and print two different severity words, and nothing in the pipeline
would ever notice.

That's the actual bug: not "someone mislabeled 33 rules," but "the schema
has a field with no source of truth." Free-text fields drift; that's what
they do when nothing enriches them. The fix is the same one-line pattern
already applied to owasp and cvss — default severity from
CWE_MAPPING[cwe].severity unless a rule explicitly overrides it, the same
options.severity ?? cweData.severity fallback the other three fields
already get. I've filed it; it hasn't shipped as of this writing, and I'd
rather say that than quietly imply it's fixed.

The audit, if you want to run it yourself

Point this at any rule set that ships CVSS numbers and severity strings —
ours included — and it'll tell you exactly where they disagree:

// audit-severity-drift.ts — flags rules where severity doesn't match its own CVSS band
import { readFileSync, globSync } from "node:fs";

const BANDS = [
  [9.0, 10.01, "CRITICAL"],
  [7.0, 9.0, "HIGH"],
  [4.0, 7.0, "MEDIUM"],
  [0.0, 4.0, "LOW"],
] as const;

const bandFor = (cvss: number) =>
  BANDS.find(([lo, hi]) => cvss >= lo && cvss < hi)?.[2] ?? "?";

for (const file of globSync("packages/eslint-plugin-*/src/rules/*/index.ts")) {
  const src = readFileSync(file, "utf-8");
  const cvss = src.match(/cvss:\s*([\d.]+)/)?.[1];
  const severity = src.match(/severity:\s*'([A-Z]+)'/)?.[1];
  if (!cvss || !severity) continue;

  const expected = bandFor(Number(cvss));
  if (expected !== severity) {
    console.log(`${file}: CVSS ${cvss} → ${expected}, shipped ${severity}`);
  }
}

203 rule files scanned, 33 flagged — an 84% clean rate I'd rather publish
honestly than round up. And that rate is itself a snapshot: the rule sources
as they stood when this published (2026-07-06), spread across the ecosystem's
twenty-odd independently-versioned plugins, not one pinned release. The
article's own thesis applies to its own numbers — a printed audit is a cache
too. Re-run the script against today's checkout and trust that count over
mine.

Reading a severity label like a security engineer, not a triage bot

None of this is specific to our plugins — it generalizes to any tool that
prints a severity word next to a CVSS number, including the ones you didn't
write:

The number is the primary source. The word is a cache of it, and caches go stale. If a CRITICAL/HIGH/MEDIUM/LOW string and a CVSS score disagree, trust the score — it's the one with a formula behind it, not a human's word choice.
CWE tells you the shape of the bug, not how loudly to worry. "This is a CWE-918 (SSRF)" is a fact about the code. It says nothing about this instance's severity until a CVSS vector is computed for it specifically.
A low-confidence detector and a low-impact bug produce the same label and mean opposite things. If a tool's docs mention "heuristic" anywhere near a LOW, ask whether LOW is describing the bug or the detector's certainty — they're rarely the same axis, and only one of them tells you whether to actually go read the code.
Don't build a CI gate on the adjective. severity !== 'CRITICAL' as a merge-blocking filter inherits every mislabel in the tool it's reading from. Gate on the CVSS number, or on the CWE list you've decided matters, not on a string a human typed by hand.

The config

npm install --save-dev eslint-plugin-node-security eslint-plugin-jwt

import nodeSecurity from "eslint-plugin-node-security";
import jwt from "eslint-plugin-jwt";

export default [nodeSecurity.configs.recommended, jwt.configs.recommended];

Both plugins print the CWE/CVSS/OWASP line on every finding — that's what
made this audit possible to run against our own output in the first place.
Getting started with eslint-plugin-node-security
and getting started with eslint-plugin-jwt
cover the rest of what each ships.

I'm not writing this to talk our own numbers down — 84% of 203 rules
printing a severity word that matches their own CVSS band is a real bar,
and I'd take it over a tool that doesn't print a CVSS number at all and so
never gets caught disagreeing with itself. I'm writing it because "check
the number, not the adjective" is a rule I only started following rigorously
after writing the script that caught my own tool getting it wrong three
different ways in three different rules.

Your turn: open whatever security linter you already run, pick one
finding, and look up its CVSS vector by hand. If the vector notation is new,
what a CVSS score actually measures
walks the AV/AC/PR/UI fields — and why severity, exploit probability, and
confirmed exploitation are three separate measurements with three separate
owners. Does the severity word next to it still hold up once you've done the
math yourself?

⭐ Star on GitHub if you've ever trusted a severity label more than the score behind it.

I'm Ofri Peretz, a security engineering leader and the author of the
Interlace ESLint ecosystem — domain-specific static analysis for security,
reliability, and performance on the Node.js stack.

ofriperetz.dev · LinkedIn · GitHub

142,076 Weekly Downloads. Zero Releases Since 2021. Is the Niche Defended?

Ofri Peretz — Mon, 20 Jul 2026 04:45:01 +0000

Two reads from the npm registry, same package, same morning (2026-07-19):

downloads, last week:  142,076
last publish:          2021-05-21  (v3.2.0)

serverless-iam-roles-per-function is one of the Serverless Framework ecosystem's most-installed plugins, and it hasn't shipped a release since May 2021. Eleven weeks earlier: 118,761 weekly downloads (measured 2026-05-04). Zero releases in between; 23,315 weekly installs gained.

The package isn't surviving abandonment — its installed base is growing straight through it.

My first read was the common one: "142k a week, the niche is taken, I'm five years late." That's the misconception this article exists to correct. High downloads don't mean a niche is taken. Downloads measure the installed base; maintenance decay measures whether the niche is defended. The gap between those signals is where I decide what to build next, using what I call the incumbent map. Fair warning: I ran the arithmetic in May, shipped against its #1 row on 2026-05-07, and only ran the filter that kills that row while fact-checking this article in July. The method works; I ran half of it. Both halves are below, in the order I should have run them.

How do you build an incumbent map?

Three signals per package, all public, all re-runnable:

Weekly downloads — demand. The installed base a maintained replacement would serve. Downloads are a proxy metric — npm's own account of download counts includes build servers, mirrors, and robots, so the number tracks lockfile momentum, not humans choosing a package. As a relative ranking of where demand pools inside one ecosystem, though, it's the best public signal there is.
Years since last publish — supply-side decay. Read the registry at time[<latest-version>]. Not time.modified — that field moves when npm touches registry metadata: serverless-iam-roles-per-function shows modified: 2022-05-17, a full year after its last release. Trust the version's own timestamp, nothing else.
Open-issue decay — unmet demand. No clean number here, so make the procedure explicit: sort open issues by reactions, list the recurring asks, date the last maintainer reply. For the plugin above: managed-policy attachment and custom role naming — asked for years across multiple issues, no maintainer commitment (the 2018 managed-policy ask drew "open for PRs"; nothing shipped since). And read the replies, not just titles; mine matter later.

The first two signals, as commands:

curl -s https://api.npmjs.org/downloads/point/last-week/<pkg>               # demand
curl -s https://registry.npmjs.org/<pkg> | jq '.time[."dist-tags".latest]'  # decay

Multiply, roughly: demand × decay × unanswered asks, ranked descending. Deliberately not a precise formula — a shortlist generator whose output you still have to judge.

The closest prior art is OpenSSF Scorecard: its Maintained check formalizes the same "is anyone home?" signal for the dependencies you consume — a risk lens. The incumbent map points it the other way: not "should I trust this package?" but "should I compete with it?" I haven't seen that build-side version written down anywhere — a strong claim, and an open invitation to correct me.

What does the serverless incumbent map actually show?

The worked example that produced my roadmap. Two zones, all numbers measured 2026-07-19:

Defended — active maintainers, do not compete on maintenance:

Package	Downloads/wk	Last publish
`serverless-offline`	627,578	2026-06-23
`serverless-esbuild`	388,619	2026-05-21
`serverless-domain-manager`	304,882	2026-07-17
`serverless-step-functions`	208,865	2026-07-13

Abandoned — demand present, nobody home:

Package	Downloads/wk	Last publish	Silence
`serverless-iam-roles-per-function`	142,076	2021-05-21	5.2 yrs
`serverless-plugin-include-dependencies`	52,500	2024-07-22	2.0 yrs
`serverless-plugin-common-excludes`	51,415	2021-07-06	5.0 yrs
`serverless-api-gateway-throttling`	36,528	2023-03-19	3.3 yrs
`serverless-plugin-tracing`	35,709	2017-10-13	8.8 yrs
`serverless-plugin-canary-deployments`	31,562	2022-04-11	4.3 yrs
`serverless-associate-waf`	28,787	2020-04-17	6.3 yrs

The top zone shipped within two months. The bottom zone's median silence: five years — on packages still installed tens of thousands of times a week.

The map moves, in both directions. serverless-associate-waf shows demand finally decaying — down roughly a quarter from my May reading — so abandonment does reach the download number, on a lag of years, not months. And re-pulling before publishing deleted a row: serverless-openapi-documenter, filed in my May notes as a stale documentation-plugin niche, shipped a release on 2026-06-24. Someone re-defended it while I drafted. Row gone. An incumbent map is a snapshot, not a subscription — re-run it the week you commit, not the quarter you planned.

When is a quiet package finished instead of dead?

The honest counter-case, because the map's arithmetic will lie to you twice.

First: stable is not dead. serverless-prune-plugin sits at 414,154 downloads a week (measured 2026-07-19), last publish 2024-10-16. Twenty-one quiet months — and it may simply be done: its job is narrow, its surface barely moves. The separating question: does the surface underneath the package move? IAM — the ground serverless-iam-roles-per-function stands on — moves constantly: new AWS services, new actions, new policy patterns. Five silent years on a moving surface is decay; on a frozen surface, it can be completion. Same silence, opposite meanings — a classic measurement-bias trap.

Second: platform absorption. serverless-plugin-typescript: 151,586 downloads a week (measured 2026-07-19), dormant since 2023-06-05 — by raw arithmetic the juiciest target on the board. It's not on my list: Serverless Framework v4 builds TypeScript natively and won't run build plugins like it unless you explicitly opt out of the native build. That niche isn't undefended; it's dissolving. A replacement would be competing with the platform itself.

An installed base looks like a moat. The publish log says whether anyone still defends it; the platform roadmap, whether the castle is even staying put.

How were the replacement targets actually picked?

The full pipeline, as actually run for the serverless line:

Rank every community plugin by demand × decay (the tables above are the output).
Filter: is the niche still real? — serverless-plugin-typescript fell here first. Then the same filter, run against the current v4 docs instead of my stale notes, reached the top row of my map: v4 ships per-function IAM roles natively — provider-statement inheritance on by default, log/VPC/event-source permissions auto-added, the plugin's own config fields accepted for migration. The official IAM guide credits the plugin for pioneering the feature and says you can now safely remove it. My #1 target wasn't an undefended niche; it was one the platform had absorbed — with honors.
Filter: is the incumbent truly undefended? — re-pull the registry the week you commit; that check removed serverless-openapi-documenter.
Filter: is the unmet demand documented in public — and still unmet? The newest open issue asks "Serverless Framework V4 Support?" — and the thread answers itself: a collaborator's "it works," then a comment pointing at the v4 docs above. The two features its tracker spent years asking for — managed policies, custom role names — shipped in v4's native implementation, not the plugin's. The counter-evidence to my top pick sat in the tracker I was reading for demand. Read the replies.

The worked example deleted two of its own rows — one while I drafted, one while I fact-checked. Here is what the second deletion cost, stated plainly: my May map scored serverless-iam-roles-per-function 23/25 — "the biggest single-plugin opportunity in the portfolio" — and I shipped @interlace/serverless-iam-roles-per-function 1.0.0 off that row on 2026-05-07 (the npm publish date; check it). The filter you just read ran two months later. The package stays published and does what it says; what it is not, anymore, is the line's growth bet — v4 owns that niche now. The ranked ROI analysis lives in the repo, unchanged, dates included — deliberately, as the before-artifact: you can see exactly the map I built from, and exactly what running filter 2 late overturned. Arithmetic generates the shortlist; filters, run against primary sources — before you build — make the decision.

Notice the conflict of interest: the person selling replacements is grading the incumbents — the same conflict as building what I benchmark, with the same mitigation: every number here is a public registry read, stamped with its measurement date, re-runnable in one curl without me in the loop.

The map is step one of entering an ecosystem. Step two keeps the building honest — write down claims and evidence before the marketing exists: the claims registry. Different package types need different metrics once you measure what you built — but the map is how you decide to build at all.

What's the most-downloaded abandoned package in your ecosystem — React components? Terraform providers? Gradle plugins? Two registry reads will tell you. I'm collecting incumbent maps, and I'd genuinely like to see yours.

Quick reference

Signal	What it measures	Where to read it	Failure mode
Weekly downloads	Installed base (demand) — not adoption, not health	`api.npmjs.org/downloads/point/last-week/<pkg>`	Mistaking base for defense; CI/mirror inflation
Years since last publish	Supply-side decay	`registry.npmjs.org/<pkg>` → `time[<latest-version>]` — not `time.modified`	Finished ≠ dead on a frozen surface
Open-issue decay	Unmet demand, stated in public	Issue tracker: sort by reactions, date the last maintainer reply	Reading titles, not replies — the answer may be "the platform shipped it"
Deprecation/archive status + platform roadmap	Whether the niche still exists	Registry `deprecated` field, repo header, framework changelog + docs	High downloads on a dissolving niche
Decision rule	demand × decay × unanswered asks → shortlist → 3 filters	This article	It's a snapshot — re-run the week you commit

References

"numeric precision matters: how npm download counts work" (npm blog, 2014) — npm's own account of what a download includes (build servers, mirrors, robots); the primary source for why downloads can only ever be a demand proxy.
npm registry download-counts documentation — the API the map reads: the endpoints and the data pipeline behind the counts.
Serverless Framework IAM guide — the platform-absorption primary source: v4's native per-function roles, statement inheritance, auto-added permissions, and the credit to the community plugin it replaces.
OpenSSF Scorecard — the closest prior art: its Maintained check operationalizes "is anyone home?" for consumption-side dependency risk; the incumbent map reuses that signal for build-side decisions.

Related deep dives

Proxy Metrics: When the Number Isn't the Thing — why downloads, stars, and every other convenient number measure something adjacent to what you care about
Bias in Measurement — how a real number still produces a wrong inference, including the too-clean-zero forensic
The Claims Registry — step two of the playbook: the evidence discipline that comes before any claim
Different Metrics for Different Package Types — once you build, how the measuring has to change per package type
I Built What I Benchmark. Here's How I Try Not to Cheat. — the conflict-of-interest process this article's mitigation borrows

Star the serverless repo — the map above is its roadmap

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

We Were Wrong About sls remove

Ofri Peretz — Mon, 20 Jul 2026 04:44:23 +0000

The loudest claim in my serverless line was wrong. Early copy for @interlace/serverless-api-gateway-caching implied that the community caching plugin leaves an API Gateway cache cluster running — and billing — after sls remove, and that our cleanup hook fixes it. On 2026-05-04, our own E2E harness ran the community plugin through sls remove on live AWS: 31 seconds, exit 0, no orphans. CloudFormation deleted the stage and took the cache cluster with it. There is no sls remove ghost-billing bug.

The trap itself is real — I had reproduced it on live AWS the day before, 2026-05-03, in about ten minutes for roughly $0.005 of cache-cluster time. It just lives on a different path than the one we were loudest about. This article is the full telling of that re-scoping: the original claim, the two runs that killed and rebuilt it, and why the narrower claim that replaced it does more work than the broad one ever did.

What was the original claim?

The community serverless-api-gateway-caching (v1.11.0) manages the API Gateway cache cluster imperatively — it calls the AWS UpdateStage API during deploy instead of declaring the cluster in the CloudFormation template. CloudFormation never learns the cluster exists. From there, the inference writes itself: if CloudFormation doesn't know about the cluster, teardown won't remove it, so sls remove must leave it behind, billing forever. Our plugin ships a before:remove:remove hook that disables the cluster before teardown — so early copy implied, in effect: the community plugin has a ghost-billing bug on sls remove, and we fix it.

Notice what that claim actually was: an inference from architecture, shipped as if it were a measurement. And it was consistent — repeated the same way across the README, the docs, the comparison table. Consistency is reliability; whether the claim describes the failure that actually happens is validity — and nobody had checked.

What did the measurement actually show?

Two runs on live AWS, one day apart, both against the pinned community release 1.11.0.

2026-05-03 — the trap, reproduced. Deploy with the community plugin (a live stack in 49s), wait for the cache cluster to reach AVAILABLE (4–7 minutes), then do what a real team does when it stops wanting a plugin: delete it from the plugins array and redeploy the still-running service. The redeploy succeeds in ~28 seconds and no plugin code runs — the plugin is gone from the config, so it can't. The cluster afterward: enabled: true, status: AVAILABLE, size: 0.5. Still running, still billing. No CloudFormation event, no log line, no warning. And no exit through the plugin either: sls caching disable and sls caching status both return "command not found" — the community plugin registers no custom commands, so the only escape is dropping to the AWS CLI and patching cacheClusterEnabled to false by hand. Total cost of watching all of this happen: ~10 minutes, ~$0.005 — the recipe is public in docs/ghost-billing-reproduction.md, cheap enough that "it reproduced on my machine" can be your finding rather than my assurance.

2026-05-04 — the loud path, refuted. The community E2E then ran the exact path our copy warned about: sls remove with the plugin still installed. Clean. 31 seconds, exit 0, no orphans — CloudFormation deleted the stage, and the cluster went with it.

The mechanism fits in one sentence: the cache cluster rides on the stage. Delete the stage — which is what sls remove does — and the cluster dies with the stack, no matter who created it. Keep the service deployed and remove only the plugin, and the stage stays up while the one tool that remembered the cluster leaves your toolchain.

I ran that second E2E to collect the evidence for the claim. It came back with the evidence against it — exit 0, no orphans, and a rewrite of my best line of copy. The same discipline that makes you distrust a result that flatters you is the one that makes you run the test that can defeat you, before someone else does.

What is the re-scoped claim?

The old copy didn't get quietly patched; it got re-scoped, on the record. Our claims registry keeps a dedicated table for exactly this case — original copy, what we measured, what's true — and the row for this incident reads, condensed: earlier copy implying "we fix sls remove's ghost-billing bug" was wrong, because there is no sls remove ghost-billing bug. The orphaned-cluster trap lives on the uninstall-while-keeping-service path, where sls caching disable — an explicit offboarding command the community plugin doesn't have — is the actual fix. The before:remove:remove hook stays, verified live (stack deleted in 28s, zero residuals, measured 2026-05-03), but it's defense-in-depth now, not the headline.

That's the whole lifecycle in one incident: claim → reproduction attempt → refutation → re-scope → versioned evidence file. The refuting run is committed as a dated JSON next to the claim row; the trap ships as a recipe anyone can replay.

The stakes are why the claim deserved this much care. The cluster we reproduced was the smallest tier — 0.5 GB at $0.020/hour, $175.20/year. AWS's published cache pricing runs up to 237 GB at $33,288/year, and the charge is per stage, per region, with no flag on orphaned clusters. A number that size deserves a claim pointed at the path where the money actually leaks.

Why is a narrower claim stronger?

Value investors call it margin of safety: you don't pay the price your own estimate says is fair — you pay less, so that being somewhat wrong doesn't ruin you. A claim scoped to exactly what the run files show carries the same margin. The broad claim had none: it rode on an untested inference, and a single ten-minute E2E from anyone could falsify it. Ours did, mercifully before anyone else's.

The narrow claim is also more useful to you. "Run sls caching disable before you remove the plugin from config" is an action. "Beware sls remove" would have taught you to distrust the one command that actually cleans up after itself — misallocating your caution away from the path that costs money and onto the path that doesn't.

Which is the misconception worth naming: "walking back a claim costs credibility." It's backwards. Unmeasured claims cost credibility — on a schedule you don't control. A published re-scope is the cheapest credibility purchase available: it costs one uncomfortable table row and buys back the right to be believed about everything the run files do support. The genre has old prior art — Buffett's shareholder letters have aired his own mistakes for decades, and it's the same model behind how I try not to cheat in my own benchmarks.

I'll admit the first instinct, reading the clean run, was the other genre: quietly delete the sentence and move on. That instinct — fix silently, stay loud — is exactly what a claims registry exists to make impossible, and the reason this article exists at all.

Quick reference

Question	Answer (measured 2026-05-03/04, community plugin v1.11.0)
Does `sls remove` orphan the community plugin's cache cluster?	No — CloudFormation deletes the stage and the cluster with it (31s, exit 0, no orphans)
Where is the actual trap?	Remove the plugin from `plugins` and redeploy the still-running service — no plugin code runs; cluster persists `enabled: true, AVAILABLE`
Is there a plugin-mediated escape (community v1.11.0)?	No — `sls caching disable` / `sls caching status` return "command not found"; manual `aws apigateway update-stage` required
What does an orphaned cluster cost?	$175.20/yr at 0.5 GB up to $33,288/yr at 237 GB — per stage, per region, unflagged by AWS
The safe offboarding path (`@interlace/serverless-api-gateway-caching`)?	`sls caching disable` before uninstalling; `before:remove:remove` covers `sls remove` (verified live: stack deleted in 28s, zero residuals)
Reproduce it yourself	~10 minutes, ~$0.005 — ghost-billing-reproduction.md

The claim that survived is smaller than the one we started with, and it's the only one of the two I'd defend in public — because defending it requires no rhetoric, just the run files. If a claim in our docs matters to you, check whether it has a row in CLAIMS.md; if it doesn't, it isn't allowed to ship. That policy — and the re-scope you just read, kept on the record instead of buried — is the honest answer to "why should I trust a plugin author's comparison table."

⭐ Star the serverless repo — where claims ship with run files, or don't ship

Next in this arc: one step earlier in the story. Before you can re-scope a claim about an incumbent plugin, you have to decide which incumbent is worth challenging at all — the abandoned-incumbent map is how the serverless line made that call.

References

Amazon API Gateway pricing — cache pricing — the per-size price table behind every $/hour and $/year figure in this article.
Cache settings for REST APIs in API Gateway — Amazon API Gateway Developer Guide — the stage-level cache settings (cacheClusterEnabled, cluster size) that make the cluster live and die with its stage.

Bias in Measurement — why a result that flatters you is the one to re-run
Reproducibility vs Replicability — the bar the $0.005 recipe is built to clear
Valid vs Reliable Metrics — how a claim can be perfectly consistent and still measure nothing
The Claims Registry — the table this article's re-scoped row lives in
I Built What I Benchmark. Here's How I Try Not to Cheat. — the same discipline applied to the ESLint benchmarks

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

25% of My Benchmark Verdict Is an Opinion. Here's the Anatomy.

Ofri Peretz — Mon, 20 Jul 2026 04:43:35 +0000

On 2026-05-04, my benchmark harness printed its verdict on two Serverless Framework caching plugins: mine scored 0.88; the community incumbent, 0.3025. Every digit of that gap is arithmetic — seven dimensions, each one measured, normalized, weighted, summed. The raw JSON actually stores my side as 0.8800000000000001, kept as-is: IEEE 754's reminder that a number can be very precise about being an opinion. Because the most consequential input in that file was never measured by any instrument: Lifecycle Correctness is worth 25% of the score because I said so.

A composite score is opinions wearing a number costume — the general argument, and the honesty requirements any composite owes its readers, live in Composite Scores and Weighting, and I won't re-derive them here. What I owe you instead is the costume's pattern, dimension by dimension — including whether the verdict survives when you attack the weights.

What are the seven dimensions — and where does each number come from?

Dimension	Weight	Interlace	Community	Measured from
Lifecycle Correctness	25%	1.0	0.5	local source + live AWS deploy
CLI Surface	15%	1.0	0.0	local source
TypeScript Coverage	15%	1.0	0.0	npm registry*
Maintenance Signal	15%	1.0	0.0	npm registry*
Bundle Weight	10%	0.0	1.0	npm registry*
Hook Coverage	10%	1.0	0.375	local source
Documentation Quality	10%	0.8	0.4	README scan
Composite	Σ=1.0	0.88	0.3025

* Community side only. The Interlace values in these rows came from local source ("source": "local" in the result file) — version 0.0.0 has never been published.

Run stamped 2026-05-04, Node v24.12.0 on darwin/arm64. Two disclosures before anything else. First: my @interlace/serverless-api-gateway-caching was measured from local source at version 0.0.0, while the community's serverless-api-gateway-caching is the published 1.11.0 from the registry. That asymmetry makes one score in the table an artifact, not a measurement: my Maintenance Signal 1.0 rests on daysSincePublish: 0, totalVersions: null, weeklyDownloads: 0 — an unpublished plugin scores perfect maintenance while the community's real 379-day publish gap scores 0. Read that row as unmeasured, not excellent. Second: the lifecycle numbers are not simulations. Both plugins went through a live-AWS end-to-end lifecycle run, and the two E2E result files sit on disk stamped 43 minutes apart from the same night.

The "measured from" column matters as much as the scores. Registry-sourced dimensions are proxy metrics: publish cadence stands in for maintenance, type definitions stand in for TypeScript support. And a README scan is reliable without being automatically valid — it returns the same 0.8 every run; whether 0.8 captures what "good documentation" means to you, the harness cannot answer.

How do kilobytes, hook counts, and pass rates become one scale?

The seven dimensions arrive in incompatible units: bundle size in kilobytes, hook coverage as a count of lifecycle hooks handled, lifecycle correctness as pass/fail steps against a live deploy, documentation as a scan score. A weighted sum over raw units lets the biggest-range unit silently win — the classic mixing-scales trap. So every dimension is normalized to 0–1 before any weight touches it.

The community plugin's Hook Coverage of 0.375 is not a grade someone typed in — it's 3 hooks counted against a ceiling of 8, and that ceiling is my own plugin's full hook-and-command surface, which guarantees my side a 1.0. That denominator is a normalization policy that happens to flatter its author, and it deserves the same suspicion as the bundle curve coming next. On Bundle Weight, my 78 KB unpacked against the community's 47 KB came out 0 against 1 — which tells you the normalization curve is itself a policy decision. A gentler curve would have given my 78 KB partial credit; this one didn't, and I left it that way. Normalization choices change scores without changing a single measurement, so they belong in the published methodology, not in a footnote you have to diff the source to find.

What happens when a dimension can't be measured?

The harness has a standing rule for dimensions it cannot measure — historically the AWS-dependent ones, when there's no live deploy to run against: score them null and never impute a value. An imputed dimension is an invented measurement wearing the same costume as the real ones. But the null path deserves the least flattering disclosure in this article: when nulls are present, compositeScore renormalizes over the measured subset — null dimensions drop out of both numerator and denominator — so a partial run still prints one clean-looking composite. Renormalizing isn't imputation, but it quietly moves weight onto whatever happened to be measurable. The day before this run did exactly that: three of seven dimensions measured, and the harness stored 0.7499999999999999 vs 0.25 — IEEE 754 again, equally precise about a composite built from 40% of the total weight, with nothing in the numbers admitting four dimensions were missing. The n-of-7 label that makes a partial composite honest currently lives one layer up, in the claims registry rather than the harness output — a real gap: "3-of-7, here's what's missing" should be printed by the code, not patched on at the claims layer. In this run the null path went unused — dimensionScores holds numeric values for all seven keys, on both plugins (don't cite the skipped array for this; it was empty in the partial run too) — and I'm telling you anyway, because a policy you only hear about when it's exercised is a policy you can't audit.

Why is Lifecycle Correctness worth 25% — and who decided?

I decided. Here's the reasoning, in tiers.

25% — the damage tier. Lifecycle Correctness is the only dimension where failure means the plugin actively harms your deployment instead of merely annoying you. A caching plugin that mishandles the deploy lifecycle doesn't degrade politely — it leaves your API Gateway stage in a state you didn't ask for.

15% × 3 — the daily-use tier. CLI Surface, TypeScript Coverage, Maintenance Signal: the things you touch or depend on every working day. Individually smaller than correctness, collectively 45% of the score.

10% × 3 — the comfort tier. Bundle Weight, Hook Coverage, Documentation Quality: real costs you can see coming and route around.

Reasonable people could defend a different split — which is the point. The common misconception is that a benchmark score computed from real measurements is objective. The measurements are objective; the weights they run through are editorial, and computed is not neutral. Nor is admitting that unusual: Lighthouse versions its scoring weights and OpenSSF Scorecard publishes its full methodology — the composite-scores canonical walks both. Same page, new Lighthouse version, different score — which means every performance score you've ever pasted into a report was a committee vote you didn't attend. Every index has a committee. The only real question is whether the committee publishes its votes.

Does the winner survive a ±50% attack on the weights?

If a ranking flips when the weights shift, the composite is telling you about the committee, not the plugins. So attack mine as hard as ±50% allows. Halve every weight where my plugin leads: Lifecycle to 12.5%, CLI, TypeScript, and Maintenance to 7.5% each, Hooks and Docs to 5% each. Raise the one dimension where my plugin loses — Bundle Weight — by half, to 15%. Renormalize so the weights sum to 1.0 again.

The composite becomes roughly 0.73 vs 0.42. The order holds, and not by luck: the community plugin leads exactly one dimension out of seven, so no ±50% reweighting exists in which one 10%-class dimension outvotes six others. Push past ±50% and, holding the other six weights in proportion, Bundle Weight needs roughly 43% of the whole composite — against its actual 10% — to flip the winner. The gap is score-broad, not weight-fragile.

That surviving loss deserves its own sentence. My plugin scores 0 on Bundle Weight — 78 KB unpacked against the community's 47 KB — and it's printed in the table above at full weight, in a benchmark I wrote, scoring a plugin I built, under weights I chose. That conflict of interest is real, and the two controls you can audit against it are the published loss and the sensitivity math you can redo on a napkin — the full self-benchmark process is in I Built What I Benchmark. A composite that survives its own worst dimension in public is worth more than one that never shows a loss.

If you'd have weighted it differently, don't argue in the abstract — the whole run is one command: npm run bench:caching from serverless/benchmarks, and this run's output sits at benchmarks/benchmark-results/api-gateway-caching/2026-05-04_v1.0/result.json in the repo. Change the weights, re-run it; if your reweighting flips the order, that's a finding and I want it in an issue. The strongest evidence that this harness follows the data even when it disagrees with me is the next article in this series: the time our loudest serverless claim died under its own E2E run — We Were Wrong About sls remove.

Quick reference

Question	Answer
What is the composite?	Weighted sum over 7 dimensions; weights sum to 1.0
Result (2026-05-04 run)	Interlace 0.88 vs community 0.3025; all 7 dimensions measured
Who sets the weights?	The author — editorial, published, versioned with the results
Weight tiers	Damage 25% · daily-use 15%×3 · comfort 10%×3
Mixed units?	Every dimension normalized to 0–1 before weighting
Unmeasurable dimension?	Null, never imputed; harness renormalizes over measured dims — n-of-7 label lives at the claims layer (this run: 7-of-7)
±50% adversarial reweighting	~0.73 vs ~0.42 — the order holds
The honest loss	Bundle Weight: 78 KB vs 47 KB → scored 0 vs 1, published at full weight
Versions measured	Interlace 0.0.0 (local source, unpublished) vs community 1.11.0 (npm)
Reproduce	`npm run bench:caching` in `serverless/benchmarks` → `benchmark-results/api-gateway-caching/`

External references

OECD/JRC — Handbook on Constructing Composite Indicators: Methodology and User Guide (2008) — the authoritative methodology reference for composite indicators; the normalization and sensitivity sections above are the single-benchmark version of its checklist.
Lighthouse performance scoring documentation — Google's published, versioned scoring weights: the working precedent for treating a weight change as a verdict change.

What's the largest weight you'd assign differently in that table — and which dimension would you demote to pay for it?

Related deep dives

Composite Scores and Weighting — the tier-0 canonical this benchmark instantiates: why composites exist, the honesty requirements, and the Lighthouse / OpenSSF Scorecard case studies
I Built What I Benchmark. Here's How I Try Not to Cheat. — the conflict-of-interest process behind every benchmark on this site
The Claims Registry — the claim→evidence-file contract that this run's result file feeds
Proxy Metrics — why registry-sourced dimensions measure stand-ins, not the thing itself
Valid vs Reliable Metrics — the difference a README scan can't close on its own

Star the serverless repo — you've just read exactly how its scores are built

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Marketing Pages Rot Silently. Mine Print an Expiry Date on Every Claim.

Ofri Peretz — Mon, 20 Jul 2026 04:43:26 +0000

Around 2026-08-01, a "verification pending" banner goes up on eleven claims in my serverless docs. Nothing will have broken. No competitor will have shipped. The banner goes up because every verified row in that repo's CLAIMS.md is stamped 2026-05-03 or 2026-05-04, the file gives evidence a 90-day warranty, and date arithmetic doesn't care that I wrote the rule.

Line 5 of that file:

If a claim doesn't have a row, it can't ship in the docs. If "Last verified" is older than 90 days, the claim is stale and gets a "verification pending" banner in docs until refreshed.

That line exists because marketing pages rot silently: a claim gets written on the day it's true, nobody dates it, and the sentence sits there aging in public. The fix is one file per product line — the claims registry: every public claim holds a row, every row points at a versioned result file, and every date is an expiry date.

This is the methodology piece — the contract, the lifecycle, the staleness rule — and none of it is JavaScript-specific. If your product page makes claims, it transfers in an afternoon.

What does a claim owe you before it ships?

The contract is deliberately boring. A claim backed by static evidence — a benchmark you can run on a laptop — carries four fields: the claim text exactly as it appears in docs or marketing, the suite that measures it, a link to the latest result file, and the date it was last verified. Verbatim matters: paraphrase is where scope creep hides — if the row is politer than the landing page, the landing page is the claim nobody verified.

One worked row from each product line:

Serverless. The claim "Composite score 88% vs community 30% (7 of 7 dimensions)" points at the api-gateway-caching suite and its result file latest.json, last verified 2026-05-04. Open the file: composite 0.88 for my plugin (measured from local source at v0.0.0, not a published release) against 0.3025 for the community serverless-api-gateway-caching@1.11.0, all seven weighted dimensions scored, including the one I lose outright: bundle weight, 78 KB unpacked against their 47 KB. The row carries the loss because the file does.

ESLint. The sister registry's row reads "Top of leaderboard on the ILB-Arena 40-vuln / 38-safe corpus (1st of 18 plugins tested, 17 security-relevant)" and points at the ilb-arena suite's dated result file, where summary.leaderboard[0].rank == 1. That row survived an audit its neighbor didn't: the registry had been asserting "97.6% precision, 100% recall, 98.8% F1" — figures that appear nowhere in the cited JSON, which reports 100% across the board. The 2026-05-13 audit withdrew the claim rather than substitute the perfect score — 100% on a 40-fixture self-authored corpus is the textbook "regression test, not benchmark" failure mode. The ordinal claim kept its row; the flattering one kept only its receipt, in a "Withdrawn claims" section. The suite column enforces validity — measure what the copy asserts; the result file makes the claim reproducible — regenerate the number without trusting me.

Claims only live infrastructure can verify get a second row flavor: claim text, the E2E step that asserts it, the last release verified. "Cache MISS on first request, HIT within TTL" maps to steps 4–5 of a deploy-and-assert run against real AWS, last verified 2026-05-03.

The contract also kills a third category — the adjective. "Battle-tested," "blazing fast," "production-grade": proxies for measurements nobody made. No suite outputs "battle-tested," so no row, so no ship. The registry doesn't argue with weak claims; it has nowhere to put them.

How does a claim live and die?

Draft → evidenced → stale → re-verified or retired — every transition a visible edit to one file.

Draft. The add-a-claim procedure opens with its most important sentence: don't write the marketing copy first. Build or extend the benchmark until it produces a measurable result for your product and at least one competitor, add the row with today's date, and only then write the copy. The rule is that blunt because my instinct runs the other way — when I draft a README, the adjective arrives before the evidence.

Pending. Unbacked copy isn't deleted — it's routed into a "Pending claims" section next to the suite that would back it. As of today my serverless registry holds 11 verified rows (4 static-evidence, 7 live-evidence) and 2 pending, both "Not started." Pending is the honest parking lot: marketing wishes become a benchmark backlog, priced in engineering hours instead of adjectives.

Re-scoped. The state nobody plans for: the measurement comes back and narrows the claim. Early copy in my serverless line implied we fix a ghost-billing bug in sls remove. Then the community-plugin E2E ran on 2026-05-04: sls remove came back clean — 31 seconds, exit 0, no orphans. The real trap is uninstalling the plugin while keeping the service. The original copy went into a re-scoped row — original text, what we measured, what's true — instead of being quietly rewritten (told in full here). The most useful rows in my registry are the ones that say I was wrong, in writing, next to the run file that proved it.

Retired is a re-scope with nothing left: no narrower version survives the measurement. The row stays; the docs copy goes.

When does a claim go stale — and who decides?

The misconception to kill: "claims go stale when the product changes." They don't. A claim goes stale when its evidence ages past its warranty — the product changing is just one way that happens. Your code can be byte-for-byte identical while the competitor you compared against ships three releases and the runtime you benchmarked goes end-of-life. The sentence didn't move; everything it pointed at did.

That's why staleness is declared, not discovered. Discovered staleness means a reader catches the drift, and the correction arrives after the damage. Declared staleness is date arithmetic: "Last verified" older than 90 days, banner up, mechanically, before anyone is misled. Ninety isn't sacred; it's a warranty one maintainer can honor — pick yours and enforce it the same way.

What keeps this from becoming a refresh treadmill is the version stamp. "eslint-plugin-security has 27.5% recall" rots — a standing claim about the present. "eslint-plugin-security v2.1.1 scored 27.5% recall on our 40-fixture corpus, on ESLint 8.57.0" is a dated measurement no future release can falsify — true forever. Financial statements solved this long ago: a balance sheet is always "as of" a date. Stamped measurements carry zero refresh obligation; only standing claims — "fastest," "most complete," anything in the present tense — sit under the 90-day clock.

The opening countdown is this rule running on its author: the banner lands around 2026-08-01 whether or not I get the re-runs done first. The ESLint registry is already there — one row, a synthetic-corpus speedup last verified 2026-01-02, carries the "re-verify recommended" flag today. That's the system working on me, twice.

How do you adopt this in an afternoon?

Nothing above depends on ESLint, serverless, or even software — the suite can be a load test, an accessibility audit, or a lab assay. The afternoon version:

Create CLAIMS.md at the repo root. The whole starting file:

   # Claims Registry

   > No row, no ship. "Last verified" older than 90 days →
   > "verification pending" banner in docs until refreshed.

   | Claim (verbatim from docs/marketing)   | Suite               | Latest result              | Last verified |
   | -------------------------------------- | ------------------- | -------------------------- | ------------- |
   | "Composite score 88% vs community 30%" | api-gateway-caching | benchmarks/.../latest.json | 2026-05-04    |

Inventory every public claim — README, landing page, package description — verbatim, no paraphrase.
For each, name the evidence (suite + latest result + date) or move it to Pending.
For everything new: evidence first, row second, copy last.
On refresh, commit the new result as a dated file (<suite>/<YYYY-MM-DD>_v<version>/) and bump the date — what you claimed, and when it was last true, becomes repo history.

This isn't an island. The closest neighbor is the OpenSSF Scorecard, which treats a repo's security posture as automated, publicly specified checks pointed at repositories from the outside. The claims registry is the manual, product-side sibling, pointed inward at your own marketing copy (corpus design covers the datasheets-for-datasets lineage). And it pairs with the conflict-of-interest discipline in I Built What I Benchmark: I built the products and the registry that polices them — a suspicious arrangement, which is why every row names the command that would catch me.

Next in this arc: We Were Wrong About sls remove — the re-scope told in full, because the lifecycle is easiest to believe when you watch it delete a claim I liked.

Quick reference

Question	Answer
What gets a row	Every claim in docs/marketing, verbatim — no paraphrase
Static-evidence row	Claim text · suite · latest result file · last-verified date
Live-evidence row	Claim text · E2E step · last release verified
Ship rule	No row → the claim cannot ship in docs
Staleness rule	"Last verified" > 90 days → "verification pending" banner until refreshed
Lifecycle	Draft → evidenced → stale → re-verified or retired; re-scoped when measurement narrows it
New-claim order	Evidence first, row second, marketing copy last
Unbacked copy	Routed to Pending with the suite that would back it — never into docs
Refresh mechanics	Re-run suite → commit dated result file (`<suite>/<YYYY-MM-DD>_v<version>/`) → bump date
The immortal form	Version-stamped: "vX scored Y (measured DATE)" — a dated measurement, true forever

The suite that generates the ESLint result files is public — go check:

⭐ Star the benchmark suite — the result files this registry points at

Before the clock catches yours: what's the oldest undated claim on your product page right now — and would it survive a re-run today?

References

OpenSSF Scorecard — the Open Source Security Foundation's automated checks on open-source repos; cited as the closest prior art for treating claims as named, publicly specified checks rather than prose.
Gebru et al., Datasheets for Datasets (arXiv:1803.09010, later published in CACM) — cited for the documentation-contract idea the registry applies to marketing claims instead of datasets: standardized fields, stated provenance, explicit scope.

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Fixtures First, Rules Second: How to Design a Ground-Truth Corpus

Ofri Peretz — Mon, 20 Jul 2026 04:43:02 +0000

'SELECT * FROM users WHERE id = ' + userId is a vulnerable fixture — no debate. Wrap the input in parseInt(userId) — vulnerable or safe? The label depends entirely on your threat model, and whichever way the author calls it, that call is the ground truth. The corpus behind the Interlace security benchmarks took 40 calls like that one; parseInt is the call I've gone back and forth on longest — it's still not in the corpus, which is itself a judgment call. And the cost of getting the discipline wrong isn't hypothetical: this corpus scores its own author's tool at a perfect 100% — a number that reads as evidence only because the fixtures demonstrably came first, and as a mirror otherwise. That discipline — categories, borderline labels, and the sequencing rule — is the rest of this article.

Why do fixtures have to come before rules?

Write the corpus after the tool, with one eye on what it already detects, and the fixtures inherit its blind spots — the benchmark stops measuring and starts reflecting. That's Goodhart's Law operating at design time.

The defense is sequencing: this corpus was designed against published OWASP categories and CWE mappings before any Interlace rule existed to cover it, and its 38 safe patterns represent realistic validated code, not Interlace's allow-listing logic. Sequencing bounds the bias, not removes it — the same person still wrote both sides of the test.

How do you choose categories — and how many of each?

Anchor categories to a public taxonomy, not instinct. The vulnerable corpus is 40 exported fixture functions across 14 categories, each mapped to a CWE ID — from four SQL-injection fixtures (CWE-89) down to a single open redirect (CWE-601), 17 distinct CWE IDs in total. Paired with those are 38 safe fixtures — 78 in all ("the 40-fixture corpus" names the vulnerable half; precision is computed against the safe half).

Prior art: NIST's SARD is the canonical labeled vulnerable-code corpus family — the Juliet suites cover C/C++ and Java, with PHP and C# suites alongside; OWASP's Benchmark Project is the closest scored SAST analog; and SecBench.js (ICSE 2023) labels ~600 real-world server-side JavaScript vulnerabilities with executable exploits. What none of them offered Node.js-native is an OWASP-Benchmark-style scored SAST leaderboard — this corpus is that analog, smaller.

One distortion to own out loud: the corpus is roughly balanced across categories and production code is not — real vulnerabilities are rare, so a balanced corpus says nothing about the false-alarm rate at production base rates. It measures detection per pattern; prevalence needs different evidence.

How do you label the edge cases?

The parseInt label is an author judgment inside the CWE-89 definition, and it earns an annotation carrying its reasoning: borderline; would be vulnerable if the query used string interpolation instead of the typed value. The generalizable rule: record the reasoning, not just the verdict. Six months later, a label without reasoning is an opinion with tenure.

This corpus has one rater, and inter-rater agreement needs at least two — there is no kappa to report, only me agreeing with myself. That's the weakest joint in the design, and why the reasoning must be public: so a second rater can show up later and disagree precisely.

Labels live with the code: a manifest at the fixture file's bottom (EXPECTED_DETECTIONS: function → {cwe, severity}; EXPECTED_NO_DETECTIONS: the safe list) is what the runner imports to score TP/FP/FN/TN — no spreadsheet drifting out of sync.

How do you make a corpus result reproducible?

A corpus result is a claim about specific tool versions — the unicorn false-zero in my own benchmark came from a stale Node version in the shell, not from the plugin. Reproducibility rests on three mechanisms, none of them "trust the manifest": a lockfile pins the install; the runner resolves each plugin's actually-installed version at run time and writes it into the results JSON; and the environment ships with the numbers — per run, inside each results file, because runs happen on different days on different Nodes. The Interlace 3.0.2 run behind the 40 TP / 0 FP / 0 FN line records Node v24.12.0, ESLint 9.39.2, measured 2026-05-30; the sonarjs 3.0.6 run (14/40) records the same environment a day earlier, 2026-05-29. One plugin (eslint-plugin-security 2.1.1) crashes on ESLint 9 and runs in a separate compat package — it declares eslint ^8.57.0 and lets its lockfile hold the exact version — recorded, not smoothed over.

The other half is publishing: corpus public, divergence channel public, and reproduction that needs zero help from me:

git clone https://github.com/ofri-peretz/eslint-benchmark-suite.git
cd eslint-benchmark-suite && npm install
npm run benchmark:fn-fp   # → node benchmarks/fn-fp-comparison/run.js

Run it, get a different number, open an issue — your results JSON carries its own environment block, so the disagreement starts from evidence, not recollection. A corpus nobody can re-run is a claim, not a benchmark.

What can 40 fixtures actually prove?

The 78 fixtures are a constructed set, not a random draw, so results are descriptions, not inferences — no p-value is meaningful at any corpus size, because statistical power assumes a sampling process this design doesn't have. What n=40 is good at is category-level findings: a plugin catching 14 of 40 while missing whole categories is a real, checkable statement about that plugin at that version. Don't decorate results with statistics the design can't carry.

What is the corpus lifecycle?

The lifecycle: design → publish → saturate → refresh.

Saturation is where mine is now: Interlace v3.0.2 scores 40 TP, 0 FP, 0 FN — 100/100/100 precision/recall/F1. The tempting read is "the tool is finished." The correct read: the corpus is finished, for that tool — a saturated corpus can't measure the improvement of the tool that saturates it. Publishing that line felt less like winning and more like watching the instrument go quiet. The corpus still measures competitors (next best: 14 of 40, eslint-plugin-sonarjs 3.0.6), but for Interlace it's now a regression test, not a benchmark.

Then contamination: public fixtures can be tuned against — by any tool, mine included. Public corpora decay the way public trading signals do: once everyone sees the signal, trading on it erases the information it carried. The answer is versioning, not secrecy (secrecy kills reproducibility): treat the published corpus as v1 and plan a harder v2 held to the same fixtures-before-rules discipline — anchored to the taxonomy, not to any tool's known behavior, Interlace's misses included.

Which findings survive a robustness check?

Before publishing, run leave-one-category-out: re-score the leaderboard 14 times, once per removed CWE category, and check whether the ordering survives every drop. A 40/40-versus-14/40 gap survives any single removal by construction; two mid-table plugins a fixture apart may swap the moment a category disappears — that's the ranking to report as fragile, or not as a ranking at all. Cheap, mechanical, and almost nobody does it.

What are Fixture Cards?

The last discipline is documentation, and it deserves a name: Fixture Cards — per-fixture documentation modeled on Datasheets for Datasets (Gebru et al.) and Model Cards (Mitchell et al.), scaled down to the single labeled example. Six fields:

Field	The `parseInt` fixture's card would read
CWE	CWE-89
Label	threat-model-conditional: vulnerable under "any non-parameterized query is unsafe"; safe under "direct injection vector only"
Reasoning	borderline; would be vulnerable if the query used string interpolation instead of the typed value
Author	Ofri Peretz
Date	the day the call is made, pinned to the corpus version it enters
Disputed?	open — threat-model-dependent

Four of the six fields already exist in the suite: CWE and label live in the manifest the runner imports, author and date in the fixture file's git history. The Card names that standard and adds the two fields no runner ever checks — reasoning and dispute status. Packaging all six as a JSON schema is the remaining step, and it doubles as the contribution unit: to submit a fixture is to submit its Card. Labeling debt stops accumulating silently, and the single-rater problem gets its structural fix — every disputed? field is an open seat for a second rater.

The parseInt Card above is the first open seat, and it's yours if you want it: comment with your label — vulnerable or safe — and the threat model that produces it. That's the second rating this corpus doesn't yet have.

Quick reference

Design decision	The rule	If you skip it
Sequencing	Fixtures before rules	The benchmark becomes a mirror
Category selection	Anchor to CWE/OWASP (here: 14 categories, 17 CWE IDs)	Blind spots inherited from instinct
Balance	Balanced corpus — and say what it can't estimate	Base-rate overclaims about production FP rates
Edge-case labels	Record reasoning with the label; mark borderline calls	Labels become opinions with tenure
Machine-readable labels	Manifest lives in the fixture file; runner imports it	Spreadsheet drift
Versions	Lockfile + per-run resolved versions in the results	Irreproducible numbers
Publishing	Public corpus + a divergence channel	A claim, not a benchmark
Sample claims	n=40 describes; it does not infer	Fake statistical rigor
Lifecycle	Version the corpus; plan v2 at saturation	100% scores misread as "done"
Robustness	Leave-one-category-out before publishing	Fragile rankings published as real
Fixture Cards	CWE · label · reasoning · author · date · disputed?	Labeling debt accumulates silently

Corpus design is the unglamorous half of benchmarking — nobody stars a repo for a well-reasoned borderline label. But every downstream precision and recall number is only as honest as these decisions. Next in this arc the corpus does its job: the false-positive/false-negative benchmark scores six plugins against these 78 fixtures, wins and losses included.

If this is the discipline you'd want behind a benchmark you trust, the corpus — fixtures, manifests, runner, results — is public:

⭐ Star the benchmark suite — every fixture and label in this article is in it

References

NIST SARD — Software Assurance Reference Dataset (incl. the Juliet Test Suite) — the canonical labeled vulnerable-code corpora for C/C++/Java; this corpus is the Node.js-native analog, at a fraction of the scale.
OWASP Benchmark Project — the closest prior art overall: a public, scored SAST benchmark on the same TP/FP mechanics.
Gebru et al., "Datasheets for Datasets" (CACM 2021) — the dataset-documentation standard Fixture Cards are modeled on.
Mitchell et al., "Model Cards for Model Reporting" (FAT* 2019) — the same documentation move at model level; Fixture Cards scale it down to the single labeled example.

Related deep dives

Ground Truth in Security Testing: Who Decides What's Vulnerable? — the T1 vocabulary this article builds on, including the full parseInt labeling story
I Built What I Benchmark. Here's How I Try Not to Cheat. — the conflict-of-interest process this corpus discipline lives inside
1.5M Weekly Downloads, 1 False Alarm per Real Bug — the evidence article produced by this corpus

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Valid vs. Reliable Metrics: Consistent Numbers Can Still Be Wrong

Ofri Peretz — Sun, 19 Jul 2026 23:47:32 +0000

A clock that runs ten minutes fast is the friendly kind of broken: it shows the same answer every time you look, so it feels dependable — right up until it makes you miss the train. The clock was never inconsistent. It was consistently wrong, which is the harder problem, because consistency is the part that looks like quality. That gap between steady and correct is the oldest distinction in measurement theory, and it's the one most dashboards quietly skip: a number can be perfectly reliable — the same value on every read — and still be invalid — pointed at the wrong quantity. Precision of the wrong thing is still the wrong thing.

Reliability versus validity applies identically whether you weigh yourself, price a stock, score a diagnostic test, grade a diver, or benchmark software. Reliability is a property of the instrument; validity is a property of the claim you attach to its number. They are independent — you can have either without the other — and telling them apart is the whole job.

What Makes a Metric Reliable?

A metric is reliable when repeated measurement under the same conditions returns the same number. A scale that reads 82.0 kg five times in a row is reliable. A lab assay that returns 5.4 mmol/L on the same blood sample, run three times, is reliable — same sample, same number. Reliability never asks whether 82.0 or 5.4 is correct; only whether the instrument will say it again.

Reliability is a property of the instrument, not of the truth. That makes it the cheap property to check: you don't need to know anything about the world — take the measurement twice and compare. It's also the property people naturally police, because unreliability is loud. A blood-pressure cuff that gives a different reading every squeeze, a poll that swings ten points between identical samples, a stopwatch that disagrees with itself — a wobbling number announces itself, gets questioned, gets fixed.

That loudness is exactly what makes reliability feel like quality. It isn't. It's half of quality, and it's the easy half.

What Makes a Metric Valid?

A metric is valid when it measures the concept you claim it measures. Psychometricians call this construct validity: the "construct" is the real thing you care about — health, aptitude, risk, security — and validity is how faithfully your number tracks it.

Three general examples, in increasing order of how much they cost when they're wrong:

A scale that is 2 kg off reads 82.0 kg every time while you actually weigh 80. Perfectly reliable, invalid for "my actual weight." A calibration error doesn't reduce consistency at all — which is precisely why it survives.
Standardized tests produce famously reliable scores: retake next week, get nearly the same number. The century-old argument is entirely about validity — whether the score measures aptitude or measures practice at taking that kind of test.
A call center reports 95% of calls answered within 20 seconds — cleanly measured, stable month to month. But the customers who gave up before connecting were never in the denominator. The metric is valid for "answered calls, handled fast" and invalid for "customers who could reach us," and the gap between those two is where the complaints live.

This framing isn't mine. Messick's 1989 treatment made construct validity the organizing idea of measurement in the social sciences, and Jacobs & Wallach ported the framework to computational systems in 2021. The practitioner translation: every number you publish is implicitly a validity claim, whether you meant to make one or not.

The hard part is that you cannot check validity by re-running. Reliability is verified from inside the measurement; validity needs an independent route to the construct — a second instrument you trust, a real-world outcome, a ground truth established some other way. You can be certain a number is stable and have no idea whether it is true.

Reliable, Valid, Both, or Neither?

The standard picture is a dartboard. Where the darts land is your metric; the bullseye is the construct.

Tight cluster on the bullseye — reliable and valid. The goal.
Tight cluster in the wrong corner — reliable, invalid. The dangerous quadrant, because consistency masquerades as correctness. Nobody audits a number that never wobbles.
Scattered around the bullseye — valid on average, unreliable. Annoying, but self-announcing: the wobble is visible, so it gets fixed.
Scattered and off-target — neither. Usually caught fast, because nothing about it looks trustworthy.

Here is the misconception this article exists to delete: "a metric that gives consistent numbers is a good metric." Consistency is reliability; whether the number measures the right thing is validity — and they are independent. You can have either without the other, and quadrant 2 is where the expensive failures concentrate.

My clearest picture of quadrant 2 is a silent zero. A fraud filter flags 0 of roughly 50,000 transactions every night for a week — the same 0 on every run, perfectly reproducible — because a data feed upstream had gone null and the model was scoring empty rows. The real fraud rate that week was about 0.8%. Nothing wobbled, so nothing got investigated; the stability was the disguise. A number that never moves is not the same as a number that is right.

One more force pushes metrics toward quadrant 2: optimization. When a number becomes a target, people and processes make it more stable while draining its meaning — the number stays crisp as its connection to the construct erodes. That mechanism has its own article on Goodhart's law.

Is One Valid Metric Enough? Construct Coverage

Validity is a property of a metric. Construct coverage is the property of your metric system: it is valid only if every failure mode that matters has at least one metric watching it.

The expensive failures are usually not a bad metric — they're a missing one. A company can report revenue, gross margin, and burn rate — each audited, each reliable, each valid for its own construct — while no line anywhere tracks customer concentration. Then the single client that was 40% of revenue walks, and every number on the healthy dashboard was true right up to the moment it stopped mattering. A coverage hole is invisible from inside the dashboard, because every number you do have looks fine. The metrics were excellent at answering questions nobody was about to ask.

The audit generalizes to anything you ship or promise:

List the claims. Everything a spec sheet, a contract, a label, or a marketing page asserts — "safe under load," "accurate to ±1%," "no side effects," "holds up outside the sample it was built on."
For each claim, name the check that would falsify it. Not the check that sits near it — the one that fails if the claim is false. A "survives a crash" promise needs a test that actually triggers the crash; a "works after the upgrade" promise needs a run on the upgraded system, not the one you happened to have.
The diff is your exposure. Every claim without a falsifying check is a promise you are making on vibes.

That's the constructive move this whole distinction buys you: reliability tells you your instrument is steady; validity tells you it's pointed at something true; coverage tells you nothing important sits unwatched. Tom DeMarco needed a public recantation in 2009 to walk back decades of "you can't control what you can't measure" — measurement humility from one of the people who taught the industry to measure. The practical version of that humility is small: before you trust a stable number, ask what question it actually answers. Then ask which questions have no number at all.

Quick Reference

Property	Question it answers	How to check	Failure smell
Reliability	Does re-measuring give the same number?	Take it twice under identical conditions	Wobbling numbers — loud, self-announcing
Validity	Does the number measure the concept claimed?	Independent route to the construct (second instrument, real outcome, ground truth)	Stable dashboard, surprised stakeholders
Reliable + invalid	—	The dangerous quadrant: consistency masquerading as correctness	A number nobody audits because it never wobbles
Construct coverage	Does every claim have a metric watching it?	List claims → name the falsifying check for each → diff	A stated guarantee with no check pointed at it

Goodhart's Law, Explained — the mechanism that drives metrics into the reliable-but-invalid quadrant.
Proxy Metrics — every proxy has a validity gap; that gap is this article's subject, measured.
Composite Scores and Weighting — what happens to validity when you collapse many metrics into one.
Different Metrics for Different Package Types — the software worked example this idea was drawn from: a real construct-coverage hole, and the metric system built to close it.

If this is the kind of reference you'll want at your next metrics review, follow me on Dev.to and bookmark it — it's built to be cited, not skimmed once.

References

Messick, S. (1989). "Validity." In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13–103). American Council on Education / Macmillan. The authoritative construct-validity treatment; the source of the modern view that validity is about the interpretation of a score, not the score itself.
Jacobs, A. Z., & Wallach, H. (2021). "Measurement and Fairness." Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT '21). arxiv.org/abs/1912.05511. Measurement theory applied to computational systems — the bridge between psychometrics and the numbers you actually ship.
Trochim, W. M. K. Research Methods Knowledge Base. conjointly.com/kb. Free online reference; the fastest concept-check for reliability/validity vocabulary, including the dartboard figure.
DeMarco, T. (2009). "Software Engineering: An Idea Whose Time Has Come and Gone?" IEEE Software, 26(4). The recantation of "you can't control what you can't measure" — metrics humility from one of the field's original measurement advocates.

Foundations series: ← Inter-Rater Agreement & Cohen's κ · hub · Proxy Metrics →

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Static Analysis vs. SAST vs. Linting: The Taxonomy That Matters for Security Teams

Ofri Peretz — Sun, 19 Jul 2026 23:47:10 +0000

When a security dashboard files a single-file linter under "SAST tools," it's using "SAST" as a marketing umbrella for anything that inspects source code without running it — technically defensible, practically misleading.

A lint rule fires in your editor in under a second. CodeQL runs on a pull request in minutes and can trace a tainted value through a dozen function calls across as many files. SonarQube keeps a persistent model of your codebase and tracks how an issue count moves across releases. Different scopes, different failure modes, different jobs in the pipeline — squeezed into one word that tells you none of it.

The taxonomy

Three levels, broadest to narrowest.

Static analysis is any examination of source code without executing it — abstract interpretation, control-flow analysis, taint analysis, and everything below all qualify. Every tool in this piece is static analysis; the word alone tells you nothing about what a given tool actually catches.

SAST (Static Application Security Testing) narrows that to security vulnerabilities specifically, and adds three defining traits: it runs in CI/CD rather than at edit time, it traces values across function and file boundaries (interprocedural analysis), and it reports a finding with severity, CWE classification, and the data-flow path that proves it. CodeQL, Semgrep, and SonarQube Security are the common names.

Linting narrows further, to syntactic and pattern-level checks meant to run while you're still writing the code. A lint rule works on a single file's AST — the pluggable, per-file architecture Nicholas C. Zakas built ESLint around in 2013 — with no cross-file data flow, and it has to finish in milliseconds to be worth having in the editor. It reports patterns: style violations, correctness bugs, and the security mistakes that are visible from inside one file.

Security linting — the category a dedicated security linter belongs to — is linting aimed at security patterns specifically: same edit-time speed, same single-file scope, but pattern-matching for constructs like string concatenation at a SQL call site, Math.random() generating a token, or a missing JWT verify call. Security linting at edit time. Not a SAST replacement.

What each level detects

The differences get concrete on one vulnerability, shown two ways.

The pattern a security linter catches

// vulnerable.js
app.get('/users/:id', (req, res) => {
  const query = 'SELECT * FROM users WHERE id = ' + req.params.id;
  //                                                 ^^^^^^^^^^^^^^^^^^
  //            ESLint sees: string + user-controlled value at a db.query call
  pool.query(query).then(r => res.json(r.rows));
});

The linter sees the concatenation at the call site — source (req.params.id) and sink (pool.query) both visible in the same file, adjacent lines. A security rule fires on the string-built SQL and attaches CWE-89. The developer sees the warning before they commit.

The multi-hop path that needs taint analysis

// routes/users.js
import { buildQuery } from '../db/query-builder.js';

app.get('/users/:id', (req, res) => {
  const userId = req.params.id;                // source: user-controlled
  const sanitized = userId.trim();              // looks sanitized — only strips whitespace
  const query = buildQuery('users', sanitized); // tainted value crosses file boundary
  pool.query(query).then(r => res.json(r.rows));
});

// db/query-builder.js
export function buildQuery(table, id) {
  return `SELECT * FROM ${table} WHERE id = ${id}`;  // template injection
}

A lint rule scanning routes/users.js sees userId.trim() and a function call — the concatenation itself sits one file away, inside buildQuery(). The call site looks clean; the rule doesn't fire. (And it isn't safe: .trim() removes whitespace, not SQL — 1; DROP TABLE users-- passes through untouched and reaches the template intact. Had the code used parseInt(userId, 10) the payload really would have collapsed to the integer 1; the trap is the sanitizer that only looks like one.) CodeQL traces req.params.id → userId → sanitized → query inside buildQuery → pool.query, and files a CWE-89 finding with the full path as evidence.

Same vulnerability, two different visibility ceilings — not a defect in either tool, just two levels of the taxonomy doing what they're built to do.

Time-in-pipeline analysis

Capability isn't the only axis that matters — timing is. Security linting fires at edit time, under a second. A light SAST pass on a PR takes one to five minutes. A deep nightly scan runs thirty minutes to hours. That gap changes what a developer actually does with a finding: a three-minute CI wait means they've already pushed the branch and moved on to reviewing someone else's PR by the time the result lands. A half-second lint warning arrives while their hands are still on the keyboard, in the same context as the line that triggered it.

Security linting and SAST aren't competing for the same job. They sit at different moments in the workflow, at different depth, for different cost — which is why a mature pipeline runs both instead of picking one.

Why a security linter is not a SAST tool

A security linter carries the constraints of its category on purpose.

Speed and scope. Rules run in milliseconds, on one file's AST, because that's what edit-time feedback requires — any rule needing cross-file data flow would break the guarantee. A linter can't trace userInput through five function calls across three files; it pattern-matches on argument names, import shapes, and option keys instead of building a program-semantics model.

Domain depth over breadth. Where a security linter earns its keep is depth inside specific library surfaces: JWT verification via jsonwebtoken, parameterized queries via pg, SSRF sinks in node-fetch and axios, unsafe deserialization in serialize-javascript. The rule knows the library's dangerous shapes and pattern-matches them within a single call site — narrow, but precise where it looks. That's the honest claim: security linting at edit time, covering what linting can structurally detect — not a replacement for CodeQL, Semgrep, or SonarQube's interprocedural coverage.

A vendor's "SAST" label isn't false — at maximum breadth "SAST" does mean "any source-code security analysis." I've stopped being annoyed that it's technically defensible: a tide pool and the open ocean are both "water," and that stays true right up until you have to decide where it's safe to swim. What the label discards is the one distinction that tells you which tool to reach for — a pattern match at edit time, or a proven data-flow path at CI time.

The precise label is security linter: lint speed, lint scope, right for edit-time and CI, not a substitute for taint analysis. Where that structural ceiling comes from, and how precision and recall diverge under it, is the subject of Taint vs. Heuristic Detection and Precision, Recall, and F1 for Static Analysis.

The recommended stack

A complete Node.js security pipeline uses four positions — not redundant, each catches a different category at a different cost.

1. Edit time — security linting (ESLint plus a security plugin). Catches obvious single-file patterns at developer speed, zero CI wait.

2. Pre-commit / pre-push — CI linting. Same rules, enforced as a hard gate — catches what a skipped editor integration let through.

3. PR check — SAST (CodeQL or Semgrep). Taint analysis across the diff and its call graph; catches multi-hop injection that spans files and imports, before merge.

4. Nightly / weekly — deep SAST or DAST (SonarQube, Checkmarx, OWASP ZAP on staging). Full-codebase analysis, drift tracking across releases, and — for DAST — the runtime behaviors no source read can reach.

Teams that skip positions 1–2 pay for 3–4 to catch what the editor could have blocked for free. Teams that skip 3–4 leave every multi-hop taint path undetected. A complete pipeline runs all four.

Named misconception: "ESLint security plugins are SAST tools." They're security linters — the subset of static analysis that runs at lint speed and lint scope, without interprocedural data flow or taint tracking. The confusion comes from vendors using "SAST" as an umbrella term for any source-code security analysis, which is technically accurate and practically uninformative.

Quick reference

Category	Example tools	When runs	Cross-file	Taint tracking	Speed
Linting	ESLint, Pylint, RuboCop	Edit time	No	No	Under 1 sec
Security linting	eslint-plugin-security, Bandit	Edit time	No	No	Under 1 sec
SAST — light	Semgrep, CodeQL default	PR check	Yes	Partial	1–5 min
SAST — deep	Checkmarx, Veracode, Snyk Code	Weekly	Yes	Yes	30 min+
DAST	OWASP ZAP, Burp Suite	QA / staging	N/A	N/A	Minutes–hours

DAST (Dynamic Application Security Testing) tests a running application, not source — out of scope here, but it catches categories, auth state, session management, server-side rendering injection, that no source-code read can reach.

References

Chess, B., & West, J. (2007). Secure Programming with Static Analysis. Addison-Wesley. The foundational industrial treatment of static analysis for security — data-flow and taint techniques, rule design, and the false-positive economics that separate a linter from a SAST engine.
NIST IR 8397: Guidelines on Minimum Standards for Developer Verification of Software. National Institute of Standards and Technology, 2021. nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8397.pdf. Note that NIST itself uses "static analysis" and "SAST" interchangeably here — even listing ESLint among its example SAST tools — which is itself a data point on how loosely the industry applies the label.
OWASP Source Code Analysis Tools. owasp.org/www-community/Source_Code_Analysis_Tools. OWASP's community-maintained list of source-code (SAST) analysis tools. For the automated-analysis-vs-manual-review distinction specifically, see OWASP's Static Code Analysis control page.
Livshits, B., & Lam, M. S. (2005). Finding security vulnerabilities in Java applications with static analysis. USENIX Security Symposium, 14. The structural basis for what makes taint analysis different from pattern matching.
Zakas, N. C. (2013, July 16). Introducing ESLint. humanwhocodes.com. https://humanwhocodes.com/blog/2013/07/16/introducing-eslint/. The original announcement of ESLint's pluggable, per-file rule architecture — the design decision this whole taxonomy hangs on.

This is the reference for the next time a scan result gets called "SAST" and it's actually a lint rule — bookmark it, and follow me on Dev.to for the rest of the Foundations series.

Foundations series: ← Taint vs. Heuristic Detection · hub · start over: The Confusion Matrix →

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Sample Size and Statistical Power: What a Small Sample Can and Cannot Tell You

Ofri Peretz — Sun, 19 Jul 2026 23:46:47 +0000

A fraud filter that catches zero of the last 200 confirmed-fraudulent charges needs no p-value to convict it. A trading strategy that beat its benchmark by five points last quarter might be genuine skill — or the kind of luck that evaporates by spring. Same data, same afternoon of analysis, two completely different standards of proof.

A test that returns zero detections across every known case of the thing it claims to catch doesn't need a significance test; that zero is a structural fact, and no amount of re-sampling changes it. But ask a subtler question — is a five-point edge real, or is it sampling noise — and the honest answer depends on numbers a summary table never shows: effect size, variance, statistical power. Knowing which of those two questions you're asking is the whole article.

The Two Questions a Small Sample Can Answer

The first question: does this detector catch the failure at all?

A screening test that returns zero hits across every confirmed case of a disease doesn't need a significance test — you just need a sample large enough that one hit was structurally possible, and even a handful of cases clears that floor. Add forty more known-positive cases, still get zero, and the conclusion doesn't move; the same zero just grows more confident.

The second question is different in kind: is the gap between two contenders large enough to be real? (A composite score like an F1 rests on the same true-positive / false-positive bookkeeping covered in the confusion matrix article.) A five-point difference on a small sample could be genuine — or a fluctuation that vanishes if you swap three cases or re-run on a slightly different set. You cannot tell which without knowing the effect size.

The distinction underneath both: not every result carries sampling variability. Whether a deterministic test fires on a fixed, known input is a fact, not a draw from a distribution — no noise in that outcome to test. Statistical tests exist for questions where repeated sampling would produce a spread of answers. Run the same fixed case through the same check twice and you get the same output; a p-value on that has nothing left to measure.

What Statistical Power Actually Means

Statistical power is the probability that a test detects a real effect when one genuinely exists. If the true gap between two contenders is fifteen points and you test it on a sample of forty, how often does the test come back significant? That probability is the power — the field's standard target is 80%, meaning you will still miss one in five real effects. Fine for an exploratory look; too low for a decision you are about to ship.

Three things move it: more observations sharpen the test's sensitivity to smaller effects; larger true effects are easier to detect than small ones; and a stricter significance threshold (α = 0.01 instead of 0.05) cuts false positives but demands a bigger sample to compensate.

Cohen (1988) gives the standard effect-size conventions — small = 0.2, medium = 0.5, large = 0.8 (Cohen's d). At 80% power and α = 0.05, detecting a medium effect needs roughly 64 observations per group; a small effect needs roughly 400.

I learned this at a poker table before I ever met it in a textbook: a player up after forty hands has proven nothing. Variance alone can float a mediocre player for an evening and sink a strong one. The edge — the thing you actually want to measure — only surfaces over thousands of hands. There, sample size isn't a technicality; it's the whole difference between reading skill and reading a good night. n is hands, and forty hands is a story, not a measurement.

The same arithmetic decides which comparisons a report can honestly make. Two diagnostic tests scoring 95% and 45% differ by fifty points — so large any sample of forty detects it. Two tests eight points apart — 63% against 55% — sit close enough that forty cases struggle to separate them from noise without a power calculation first. This is why a careful report leads with the categorical claim ("misses the entire category") and refuses to rank near-neighbours by a handful of points.

When Variability Is Real: The Chi-Squared Case

The opposite situation looks similar on the surface and needs the opposite treatment. Suppose you A/B-test three versions of a checkout page — call their conversion rates 68%, 71%, and 73% over a few hundred sessions each. The eye wants to crown the 73%. Whether the rate truly differs between versions needs a chi-squared test on the counts; run it and it comes back with a small statistic and p ≈ 0.4 — no significant difference, and the three-way ranking your eye drew is noise.

That test earns its p-value precisely because each visitor is a genuine random draw — the sampling story a hand-built checklist never has. Statistical Significance and p-Values owns the full contingency-table walkthrough, including the paired McNemar's-test version for comparing two tools on the same cases.

When Your Test Cases Aren't a Random Sample

This is where most people trip: a confidence interval requires the test set to be a random sample from some population. A hand-built set of cases is not one.

Think of a driving examiner's checklist of manoeuvres, a medical board's curated question bank, or a chess coach's hand-picked tactics puzzles. Each is deliberately constructed to cover chosen categories — which bounds a specific kind of validity (the set covers the categories you chose) without giving you a random draw from the space of situations that show up in the real world. Conflating the two manufactures false precision.

A confidence interval on "score over these forty cases" is computable and practically meaningless: the formula assumes repeated draws from a distribution, and deliberate test cases aren't that. The same problem sinks a p-value on "is performance significantly above 50% here" — it tells you whether you beat a coin flip on these particular cases, not on the next real one you face, because the two don't share a distribution.

What a constructed set is legitimately good for: exact, observational counts — did it catch this case, did it miss an entire category. What it cannot support: a confidence interval or significance claim about performance beyond the cases in it. The table below draws that line for every question type here.

The Practical Rules

Four rules for reading any benchmark, audit, or scorecard:

1. Category absence is evidence without statistics. Zero out of N on a whole category needs no t-test — the tool has no coverage of that class of problem, full stop.

2. Only run significance tests on a genuine sample. A deliberately built set can't support a p-value about real-world generalization; outcomes drawn or generated at random, like the three-variant test above, can. Know which situation you're in before you reach for the test.

3. State your n before your conclusion. "Forty cases, missed the entire category" is complete on its own. "Forty cases, 5% higher score" needs a power calculation before it means anything — the sample size is part of the finding, not a footnote to it.

4. Report null results. "No significant difference" is an honest finding, not a result to bury. Publishing only the comparisons that reached significance is publication bias — the same distortion covered in Bias in Measurement, operating at the analysis stage instead of during data collection.

Named misconception: "A larger sample always gives more accurate results." A larger sample buys more power to detect small effects. It does not turn a deliberately constructed set into a random sample, and no sample size fixes that. Sample size and sampling method are different problems, and conflating them is what manufactures false precision.

Quick Reference

Question type	Needs significance test?	Why
Does the tool detect this failure at all?	No	Presence/absence, not magnitude — zero is zero
Which of two tools scores higher?	Depends on effect size	Run a power calculation before claiming the difference is real
Do three groups have different rates?	Yes (random sample)	Random draws + categorical comparison — chi-squared appropriate
Is our score significantly above 90% on a fixed test set?	No (constructed set)	Not a random sample; a CI would be false precision
Did it miss an entire category?	No	Observational count — report the zeros directly
Is a 5-point improvement real?	Yes	Small effect; small n has low power; significance test or power calc required

If this is the kind of page you'll want open mid-argument, bookmark it — and follow me on Dev.to to catch the next one in the series.

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates. The foundational book on power analysis — defines the small/medium/large effect size conventions (d = 0.2 / 0.5 / 0.8) used throughout statistics and cited in this article.

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. Accessible practical guide to effect size and statistical power — freely available online and the clearest non-textbook treatment of the topic.

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. The paper behind the G*Power tool (freely available at gpower.hhu.de) — the standard software for the power calculations referenced in this article.

Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLOS Biology, 13(3). On what goes wrong when significance tests are applied carelessly — the publication bias and null-result-suppression problem that motivates rule four above.

Builds on — The Confusion Matrix: the four counts every score in this article is ultimately built from.

Read next — Statistical Significance and p-Values: what a p-value means, the full chi-squared walkthrough, and the paired McNemar's test for comparing two tools on one set.

Cited by — The FP-Tax Benchmark and We Ranked 5 AI Models by Security — the evidence articles whose worked numbers (the leaderboard gaps, the null-result chi-squared) live where the product proof belongs. See also Bias in Measurement for the publication-bias angle.

Foundations series: ← Reproducibility vs replicability · hub · p-values & significance →

Part of the Interlace ESLint ecosystem. Source on GitHub · npm · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Proxy Metrics: The Number You Optimize Is Not the Thing You Want

Ofri Peretz — Sun, 19 Jul 2026 23:46:24 +0000

Every number on a dashboard is a proxy. GDP stands in for a nation's prosperity, yet it counts a car crash and the cleanup afterward as growth, while a parent raising three children for free adds nothing to it. A blood-pressure reading stands in for cardiovascular health, yet it climbs the instant a nervous patient sits down in the cuff. A credit score stands in for whether a borrower will repay, yet it cannot see the lifelong renter who simply never borrowed. Each of these is the best single number anyone has, and each is wrong in a way the number itself will never show you.

I reach for numbers like these anyway. Refusing proxies is not discipline; it is refusing to measure, because almost nothing worth wanting can be measured directly. The discipline is knowing, precisely, what each number fails to measure.

I learned the distinction from markets before I ever learned it from a dashboard. A stock's price is a proxy for a business's worth — precise to the cent, quoted all day, and routinely wrong. Mr. Market hands you a fresh number every morning; the discipline is remembering the number is his mood, not the company. I read every metric that way now. GDP is not prosperity, a download is not a decision, and the morning you forget the gap is the morning you start buying the map instead of the territory.

What a Proxy Metric Is

A proxy metric is a measurable stand-in for something you cannot measure directly. The thing you actually want — measurement theory calls it the construct — is almost always abstract: "health," "prosperity," "skill," "quality," "security." What you can observe is a correlate: blood pressure, GDP, a credit score, a customer-satisfaction survey, a scan for known-bad patterns. The correlate is the proxy. The construct is why you bothered.

This is the normal condition of measurement, not a defect. Medicine runs on it: a drug trial may track a surrogate endpoint — lower blood sugar, a shrinking tumor — because the real endpoint, living longer, takes years to observe. Sometimes the surrogate moves and the patient does not. A class of anti-arrhythmic drugs once corrected the irregular heartbeats they targeted while quietly raising deaths: the proxy improved and the construct got worse at the same time. The pattern is identical across domains; only the nouns change.

Value investing has the cleanest statement of the idea. "Price is what you pay; value is what you get," as Warren Buffett put it — the market prints a precise number all day, and that number is a proxy for a value nobody can observe directly. Investors who forget the distinction buy prices. The rest of us optimize dashboards.

A proxy is good exactly to the degree that it tracks its construct — a question of validity, not reliability. A metric can be perfectly reliable, returning the same number on every run, while measuring the wrong thing every time.

The Proxy Gap

The proxy gap is the distance between what a metric actually measures and what you read it as. Two properties make it dangerous.

First, the gap is invisible in the number itself. A statistics office reports 3% GDP growth; nothing in the figure tells you whether it came from rising wages or from rebuilding after a flood. The gap lives entirely outside the data, in the difference between the measurement procedure and the construct — so no amount of staring at the metric reveals it.

Second, the gap widens under optimization pressure. This is Goodhart's Law doing its work: the moment you target a proxy, you improve it by the cheapest available route, and the cheapest route rarely runs through the construct. A school told to raise its average exam score can teach better — or it can quietly counsel its twenty weakest students to stay home on test day. The second route is cheaper.

Named misconception: "If the metric goes up, the thing improved." The proxy moved. Whether the construct moved is a separate empirical question — and under optimization pressure the two actively diverge, because the cheapest way to move a proxy usually runs around the construct, not through it.

Where Proxies Famously Fail

The failure pattern is old and well documented. Lines of code as a measure of programmer productivity is the ancestral software case: reward lines, receive lines — bloated code, duplicated logic, resistance to deletion. Tom DeMarco, who helped canonize measurement-driven software management, spent his 2009 IEEE Software retrospective walking back how much of engineering should be governed by metrics at all.

Jerry Muller's The Tyranny of Metrics (2018) is a book-length catalog of the same pattern outside software. His best-known case: score surgeons on their patients' mortality rates, and some surgeons stop operating on the sickest patients. The proxy improves while the construct — care for the people who most need it — gets worse. Muller finds the identical shape in policing (arrests over safety), in universities (citation counts over scholarship), and across public administration.

Software delivery produced its own famous proxy set with the DORA metrics (Forsgren, Humble & Kim, Accelerate, 2018). Deployment frequency stands in for delivery health; teams game it by slicing one release into ten trivial deploys — which is exactly why DORA pairs its speed metrics against stability metrics like change-failure rate. Forsgren then co-authored the SPACE framework (2021), which builds "no single metric" in as a founding principle rather than a caveat — the closest prior art to the counter-metric rule below, applied one metric at a time instead of at framework scale.

The audit generalizes to any dashboard: write down what a number actually measures next to what you wish it meant, and the gap becomes hard to un-see. Body-mass index measures weight over height squared, yet a muscular athlete and a sedentary desk worker can both post a BMI of 27 while sharing almost nothing about their health — a proxy built to describe populations, routinely misread as a verdict on an individual.

How to Choose a Proxy

Two rules cover most of the discipline.

Rule 1: prefer the proxy closest to the construct that you can afford to measure. Risk-adjusted mortality sits closer to quality of care than raw mortality, because it stops rewarding case-selection. Body-fat percentage sits closer to health than BMI. Closer proxies almost always cost more to collect — more instrumentation, more time, more follow-up — which is exactly why the cheap, distant ones dominate dashboards. Treat that as a cost decision you are making, not a fact of nature.

Rule 2: pair every proxy with a counter-metric that catches its gaming. Deployment frequency pairs with change-failure rate. A sales team's calls made pairs with deals closed, so nobody wins by dialing numbers and hanging up. A model that looks flawless on the data it was tuned on pairs with a test on data it has never seen — because the base rate of what you are hunting is rarely the same in the wild as in the sample you built on. A proxy with no counter-metric is an unfalsifiable claim wearing a number.

Proxies are not the enemy. They are the only measurements you will ever have, and the honest dashboard is not the one with the fewest numbers — it is the one whose owner can state each number's gap out loud. The next problem arrives immediately after: you hold five honest proxies and a decision that needs one number. Merging them means choosing weights, and weights are opinions — that is the subject of composite scores and weighting, next in this series.

If this page earns a bookmark, take it — and follow me on Dev.to to catch the next canonical in the measurement series as it lands.

Quick Reference

Proxy	Construct it stands in for	Known gap	Counter-metric pairing
GDP	National prosperity	Counts disasters and cleanup as growth	Median wage · wellbeing surveys
Stock price	A business's worth	Sentiment, momentum, and mood	Owner earnings · free cash flow
Body-mass index	Health	Muscle reads as fat; built for populations	Body-fat % · metabolic panel
Standardized test score	Learning	Teaching to the test; roster gaming	Transfer tasks · later-life outcomes
Deploy frequency	Delivery health	Trivial-deploy slicing	Change-failure rate (DORA's own pairing)
Lines of code	Productivity	Rewards verbosity, punishes deletion	None — retire it

References

Goodhart, C. A. E. (1975). "Problems of Monetary Management: The UK Experience." Papers in Monetary Economics, Reserve Bank of Australia. The original observation that statistical regularities collapse once used for control — the mechanism behind every widening proxy gap.

Strathern, M. (1997). "'Improving Ratings': Audit in the British University System." European Review, 5(3), 305–321. The source of the familiar measure-becomes-target phrasing commonly misattributed to Goodhart; her university-audit case is itself a proxy-gap study.

Muller, J. Z. (2018). The Tyranny of Metrics. Princeton University Press. The book-length catalog of proxy failures across medicine, education, policing, and business — the surgeon-scorecard case above is his.

Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press. Defines the DORA metrics and, notably, pairs speed metrics with stability metrics precisely to resist single-proxy gaming.

Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021). "The SPACE of Developer Productivity." ACM Queue, 19(1). The framework that made "no single metric" a design principle rather than a disclaimer — prior art for per-metric counter-pairing.

DeMarco, T. (2009). "Software Engineering: An Idea Whose Time Has Come and Gone?" IEEE Software, 26(4). A founder of software-metrics culture revisiting how much control measurement should really carry — the field's most cited note of metrics humility.

Cited by / worked examples: I built what I benchmark — how I try not to cheat runs this exact audit on a detection tool, where a perfect score on a corpus you wrote yourself is partly a statement about the corpus; the FP-tax benchmark is the leaderboard those numbers come from.

Foundations series: ← Valid vs reliable metrics · hub · Composite scores & weighting →

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Precision, Recall, and F1 for Static Analysis: Same Score, Opposite Tools

Ofri Peretz — Sun, 19 Jul 2026 23:45:00 +0000

F1 gives the scanner that screams at everything and the one that stays almost silent the same score — and it is the number every vendor table leads with.

Tool A finds all 50 vulnerabilities in your codebase and buries them under 2,450 false alarms: 100% recall, 2% precision. Tool B raises exactly one flag, which is correct, and misses the other 49: 100% precision, 2% recall. F1 for both: 3.9%.

Same number, opposite products: one drowns every real finding in noise, the other stays silent through almost everything. A single score that maps both to 3.9% cannot tell you which failure mode you are buying. This page is the reference for what sits underneath it.

The Three Counts

Every headline score in a static-analysis benchmark reduces to three counts against a labeled corpus:

TP (true positive) — the tool flagged a real vulnerability.
FP (false positive) — the tool flagged safe code.
FN (false negative) — a real vulnerability the tool stayed silent on.

The fourth cell, the true negative — "every line the tool correctly ignored" — is unbounded, which is why accuracy flatters any tool that mostly stays quiet; the full grid lives in the confusion matrix article. And all three counts assume someone already decided what "a real vulnerability" means — a labeling step with failure modes of its own, covered in ground truth in security testing.

Precision: When It Fires, Is It Right?

Precision = TP / (TP + FP). Of everything the tool flagged, the fraction that was real.

Concrete case: a SAST scanner raises 22 SQL-injection flags on a service; 11 are real, 11 are safe string concatenations it misread. That is 50.0% precision — one false alarm per real bug. Precision is the number your developers feel, because they are the ones who triage every flag.

Recall: Of Everything There, How Much Did It Find?

Recall = TP / (TP + FN). Of all the real vulnerabilities in the corpus, the fraction the tool caught.

Concrete case: a conservative scanner that only fires on high-confidence patterns catches 4 of the 40 real vulnerabilities in a codebase, with 1 false alarm: 80.0% precision, 10.0% recall. When it fires, trust it — but it stayed silent on 36 of 40. Recall is the number your attackers feel, because false negatives are the findings that ship.

F1: One Number, Two Hidden Failure Modes

F1 = 2PR / (P + R) — the harmonic mean of precision and recall. Unlike an arithmetic mean, it punishes imbalance: 100% on one side cannot rescue 2% on the other, which is how both tools in the opening land on 3.9%.

That property is also its blind spot. F1 tells you the two ratios are balanced or not; it never tells you which one collapsed. The misconception worth deleting is the one the opening demonstrated: two tools with the same F1 behave the same. They don't. The value-investing rule applies: price is what you pay, value is what you get. A headline score is the price tag; the three counts are the books. Read the books.

The Worked Numbers

An illustrative benchmark makes the point concrete. Take a corpus of 50 seeded vulnerabilities — SQL injection, SSRF, path traversal — plus safe counterparts, and run three scanners with different temperaments:

Scanner	TP	FP	FN	Precision	Recall	F1
Aggressive matcher	50	50	0	50.0%	100.0%	66.7%
Balanced	35	9	15	79.5%	70.0%	74.5%
Conservative	10	2	40	83.3%	20.0%	32.3%

Three readings no single column supports:

Similar precision, different products. Balanced (79.5%) and Conservative (83.3%) look interchangeable on precision. Balanced has 3.5× the recall (70.0% vs 20.0%), and F1 spreads them to 74.5 vs 32.3. Comparing the two on precision alone compares the wrong thing.

Averages conceal; per-check numbers reveal. The aggressive matcher's 50.0% precision reads like a coin flip until you split it by check. One broad rule — a regex that flags any string concatenation feeding a database query — fired 8 of its false alarms against just 2 real SQL-injection hits: 20% precision hiding inside the 50% average. That 20% check is the one a team mutes first. Pull the per-check breakdown before you quote the tool-level number.

A perfect row is a question, not an answer. A scanner that posts 100/100/100 on a corpus is usually reporting that the corpus is exhausted, not that the tool is flawless. Read a perfect row as a reason to build harder fixtures — be more suspicious, not less.

Cited by: for a real, version-stamped run of this arithmetic across the Node.js linting ecosystem — real tools, real counts, one row per plugin, down to a single rule that fired 8 of a plugin's 11 false positives — see the FP-tax benchmark.

The Security Asymmetry

In most domains, precision and recall errors cost about the same. In security they don't. A false positive costs minutes: a developer reads the flag, recognizes safe code, moves on. A false negative ships a vulnerability. On raw cost per error, recall should dominate.

But the second-order effect inverts the naive conclusion. Sustained low precision trains developers to distrust the tool — first the inline suppression, then the rule lands in the shared config's off list. A muted rule has an effective recall of zero, whatever the benchmark said. This is the false-positive tax: precision failures convert into recall failures through human behavior — which is why optimizing any single number in isolation fails exactly as Goodhart's Law predicts.

The framing that survives both effects: recall-first, precision-floor. Choose tools by what they catch, subject to a hard minimum on precision — below the floor, the catch rate is theoretical because nobody is listening.

The Assumption F1 Hides

F1 weights precision and recall equally. That is a choice, not a law — and for security it is the wrong default. Van Rijsbergen's Fβ generalizes the metric: Fβ = (1+β²)PR / (β²P + R), where β is how many times more you value recall than precision. F2 (recall counts double) is the defensible default when a false negative ships a vulnerability and a false positive costs triage minutes.

Recomputing the two similar-precision scanners above under F2:

Scanner	F1	F2
Balanced (P 79.5% / R 70.0%)	74.5%	71.7%
Conservative (P 83.3% / R 20.0%)	32.3%	23.6%

The order does not change. The gaps do: Conservative, the precision champion, drops from 32.3 to 23.6 as F2 charges it properly for 20% recall, and Balanced's lead over it stretches from 42 to 48 points. That is the lesson — metric choice moves magnitudes before it moves ranks, so check that a conclusion survives the switch before you trust it.

The grown-up version is cost-sensitive evaluation: put a currency cost on each FP (triage time × flag volume) and each FN (expected incident cost), then minimize expected total cost instead of maximizing any F-score. Those costs hinge on how rare real vulnerabilities are in your codebase — the base rate problem, which earns its own page.

How to Read a Vendor's Table

Four questions, in order:

Where are the raw counts? TP/FP/FN recover every ratio; ratios without counts recover nothing.
Is precision per-rule or tool-wide? A 50% average can hide a 20% rule, and the 20% rule is the one your team will mute.
Are the numbers version-stamped and dated? A tool's precision is a property of a specific version, on a specific corpus, on a specific date; last quarter's build, re-run after a rules update, can post a different number. An unstamped number is a claim about nothing in particular.
Points or curves? ML classifiers sweep a threshold, so they report precision-recall curves (Davis & Goadrich is the standard treatment). Rule-based scanners and linters are binary — a rule is on or off, with no threshold to sweep — so an honest benchmark reports operating points. A PR curve on a rule-based benchmark is decoration.

For prior art: the closest public relative is the OWASP Benchmark Project, which scores SAST tools on a large Java corpus using the same TP/FP-derived arithmetic. The same discipline transfers to any ecosystem — pick a labeled corpus, report operating points, and publish the raw counts so every ratio stays recoverable.

None of these metrics decides for you. Used together, they make it impossible for a single flattering number to decide against you. Next in the Foundations arc: why a precision number earned on a balanced corpus quietly falls apart at production base rates — The Base Rate Problem.

If this is the reference page you wished vendor tables came with, bookmark it and follow me on Dev.to — the rest of the Foundations series is written the same way.

Quick Reference

Metric	Formula	Question it answers	Blind spot
Precision	TP / (TP + FP)	When it fires, is it right?	Says nothing about what was missed
Recall	TP / (TP + FN)	Of what's there, how much did it find?	Says nothing about noise
F1	2PR / (P + R)	Are the two in balance?	Symmetric — hides which side failed
F2	5PR / (4P + R)	Balance, valuing recall 2×	Still one number; weights still a choice
Accuracy	(TP+TN) / all	How often was it right overall?	Needs TN — unbounded and near-meaningless for linters

References

Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427–437. The standard taxonomy of classification metrics — which measures are invariant to which changes in the confusion matrix, and therefore which are safe to compare across corpora.

Van Rijsbergen, C. J. (1979). Information Retrieval (2nd ed.). Butterworths. The origin of the effectiveness measure that F1 descends from, including the β parameter — the primary source for the fact that F1's equal weighting was always meant to be adjustable.

Davis, J., & Goadrich, M. (2006). The relationship between Precision-Recall and ROC curves. Proceedings of ICML 2006. Why precision-recall space is the right lens under class imbalance, and the context for when curves (threshold tools) versus points (binary tools) are the honest report.

OWASP Benchmark Project. owasp.org/www-project-benchmark. A public, scored SAST benchmark built on TP/FP-derived metrics over a Java test suite — the closest existing prior art to a hand-labeled, operating-point leaderboard.

Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), e0118432. The companion piece for why ROC-style reporting flatters tools when real positives are rare — which in security codebases they always are.

Foundations series: ← The Confusion Matrix · hub · The Base Rate Problem →

Part of the Interlace ESLint ecosystem. Source on GitHub · npm · Follow: Dev.to/ofri-peretz · ofriperetz.dev

Taint vs. Heuristic Detection: The Difference Between a Proof and a Hunch

Ofri Peretz — Sun, 19 Jul 2026 23:41:25 +0000

await fetch(userUrl);

A linter flags this line in about a millisecond. Here is what it did not do in that millisecond: it did not trace where userUrl came from, and it did not prove that a request-forgery path exists. It flagged the line because a user-ish name reached a network sink. That is the entire analysis.

Whether that is scandalous or sensible depends on a distinction most tool marketing blurs: impact-if-real and confidence-it's-real are different axes. The impact axis is brutal — server-side request forgery is CWE-918, and a serious vector scores CVSS 9.1, a tenth of a point past the line where the scale turns Critical (push every judgment to its worst and it tops out at a perfect 10.0). The confidence axis, for a single name-match flag, is modest: a variable name is circumstantial evidence. Print one axis in the other's vocabulary and you get 9.1 and LOW four characters apart — and that drift is not hypothetical: a rule-metadata audit caught exactly that mislabel in the wild. CVSS scores what a weakness could do, never how sure a detector is that it found one — so this page explains the two detection families that sit on opposite ends of the confidence axis.

What is taint analysis?

Taint analysis answers one question with evidence: can a value from an untrusted source reach a dangerous sink without passing through a sanitizer?

The model has three parts. Sources produce attacker-influenced data: req.query, req.body, CLI arguments, file contents. Sinks are operations where untrusted data causes damage: fetch, child_process.exec, a SQL driver call. Sanitizers are the functions that clear the taint: parameterization, allowlist validation, encoding. The engine builds a data-flow graph of the whole program, propagates "tainted" labels from sources, and reports only when a complete source-to-sink path exists with no sanitizer on it. The lineage runs straight back to Dorothy Denning's 1976 lattice model of secure information flow; Livshits and Lam made it practical for real web vulnerabilities in 2005.

What makes this expensive is that real paths are indirect. A two-hop flow:

const target = req.query.url;   // source
const opts = { url: target };   // taint flows into a property
await fetch(opts.url);          // sink — three statements, one path

Reporting this requires following the value through an assignment, a property write, and a property read — and production paths cross function and file boundaries, which demands a call graph. Aliasing is worse:

const a = { url: DEFAULT };
const b = a;                    // two names, one object
b.url = req.query.url;          // tainting b.url taints a.url
await fetch(a.url);             // sink reached through the alias

Sound alias analysis is where most of the computational cost lives, and in JavaScript — eval, dynamic property access, framework glue — the graph is never complete. The payoff for all that machinery: when a taint engine reports, it hands you a path you can read as a demonstration.

What is heuristic detection?

A heuristic rule answers a much cheaper question: does this code's shape resemble a known-vulnerable pattern? A non-literal argument reaching fetch. A template literal inside exec(). A computed property write. No data-flow proof — a resemblance, computed from a single file's AST in milliseconds.

The failure profile is the mirror image of taint analysis: strong catch-rate on patterns it matches, false positives wherever the pattern's premise fails. Picture the classic offender — a rule that flags every computed property access, obj[key], as possible object injection. It fires because it cannot know whether key is attacker-controlled, so in a codebase where bracket access is mostly over trusted keys it drowns you: a rule like that can sink to 25% precision — three false alarms for every real finding — all on its own. It is not broken. It is a heuristic priced honestly, and the price is only visible once you count TP and FP properly.

The false-positive asymmetry

The two families fail in opposite directions, and the direction is the useful part.

Taint engines make FN-shaped errors. Wherever the model can't follow — dynamic dispatch, reflection, a framework's dependency injection — the path silently disappears, and with it the finding. What a taint engine says is trustworthy; what it doesn't say is unknown.

Heuristic rules make FP-shaped errors. They over-flag whenever a name or shape misleads, and they miss the flows that never look dangerous at the sink line. Look back at the two-hop example: a single-file heuristic staring at fetch(opts.url) sees a property read with nothing user-ish in the name. The flow is invisible without the graph.

Now put the two axes from the hook back together. A taint finding carries high confidence — a demonstrated path. A heuristic finding carries an invitation to look. The impact axis is untouched by any of this: SSRF is CWE-918 at CVSS 9.1 whether CodeQL proved it or a name-match suggested it. Severity belongs to the weakness; confidence belongs to the detection method. Print one axis in the other's vocabulary and you get 9.1 and LOW four characters apart.

Complementary, not competing

Editor-grade security linters are heuristic by design, not by a limitation someone forgot to fix. The design constraints are a single file's AST and a millisecond budget, because the rule runs on every keystroke and every commit. Interprocedural taint tracking under those constraints is not difficult — it is excluded by the specification. Some rules track values within one file's scopes, which is useful, but it is not taint analysis, and the most common misreading in this vocabulary — "ESLint security rules do taint tracking" — is simply false. They pattern-match. Knowing this changes how you read a flag: it is a screen, not a verdict.

The right mental model comes from value investing: a stock screener filtering on price-to-earnings does not tell you a company is undervalued — it tells you which filings are worth reading. Heuristic lint is the screener: cheap, immediate, over-inclusive on purpose, run on everything. Taint analysis is the due diligence: expensive, slower, run where the stakes justify it. In a real pipeline that means heuristic rules in the editor and pre-commit, and an interprocedural engine — CodeQL, Semgrep in taint mode — on the pull-request gate or nightly. Where each tool class sits in the broader taxonomy is its own article; the short version is that they answer different questions, so replacing one with the other means a question goes unanswered.

What this means for benchmarks

One honest disclosure follows. A single-file benchmark corpus can only encode heuristically-detectable patterns — single-file shapes a lint rule could conceivably match — because someone had to decide what counts as vulnerable, and a corpus of one-file fixtures can only hold one-file bugs. So a linter that scores 100% on such a corpus has told you one specific thing: complete coverage of the pattern class the corpus was designed for. It has not told you a taint engine is unnecessary, because the multi-file flows a taint engine exists to find are exactly what a single-file corpus cannot encode.

A perfect score on a corpus you built yourself is a tautology wearing a medal — you wrote the test, you wrote the tool, and the number just confirms they agree. The honest reading is the scope statement hiding underneath it: complete coverage of the one pattern class you chose to encode, and silence about everything you left out.

So the one question to ask any static analysis tool — before the demo, before the leaderboard — is: what evidence does a finding carry? A path means taint. A resemblance means heuristic. Both are worth paying for; neither substitutes for the other; and a tool that names which one it is doing has already told you how to triage its output.

Quick reference

	Taint analysis	Heuristic detection
Core question	Does untrusted data reach a sink unsanitized?	Does this code shape resemble a vulnerable pattern?
Evidence per finding	A demonstrated source→sink path	A pattern match
Scope	Whole program, cross-file	Single file (one AST)
Cost	Minutes; call graph + alias analysis	Milliseconds per file
Characteristic error	False negatives (model gaps: `eval`, dynamic dispatch, framework glue)	False positives (premise fails: names and shapes mislead)
Where it belongs	PR gate, nightly runs	Editor, pre-commit, CI
Examples	CodeQL, Semgrep (taint mode)	ESLint security plugins, single-file lint rules

References

Denning, D. E. (1976). A lattice model of secure information flow. Communications of the ACM, 19(5). The founding citation — every source/sink/label system in use today descends from this model.
Livshits, V. B., & Lam, M. S. (2005). Finding security vulnerabilities in Java applications with static analysis. USENIX Security Symposium. The paper that made taint-based static analysis practical against real web-application vulnerability classes.
Chess, B., & West, J. (2007). Secure Programming with Static Analysis. Addison-Wesley. The standard industrial treatment of both detection families, including why commercial SAST engines mix them.
OWASP Community: Static Code Analysis. Practitioner-level survey of the technique, with an unusually honest limitations list.

This is a vocabulary page — bookmark it for the next time a vendor deck says "taint-aware linting," and follow me on Dev.to for the rest of the foundations series.

Foundations series: ← OWASP Top 10, Explained · hub · Static Analysis vs. SAST vs. Linting →

Part of the Interlace ESLint ecosystem. Source on GitHub · npm: @interlace · Follow: Dev.to/ofri-peretz · ofriperetz.dev

DEV Community: Ofri Peretz

I Audited 203 of Our Own ESLint Security Rules. 16% Mislabel Their Own CVSS Score.

What the three labels actually measure

Case 1: a 9.1 that prints LOW — and the honest reason it might be fine

Case 2: a 9.8 with no heuristic excuse at all

Case 3: the one that runs the other way

Why this happens: severity is the one field nobody enriches

The audit, if you want to run it yourself

Reading a severity label like a security engineer, not a triage bot

The config

142,076 Weekly Downloads. Zero Releases Since 2021. Is the Niche Defended?

How do you build an incumbent map?

What does the serverless incumbent map actually show?

When is a quiet package finished instead of dead?

How were the replacement targets actually picked?

Quick reference

References

Related deep dives

We Were Wrong About sls remove

What was the original claim?

What did the measurement actually show?

What is the re-scoped claim?

Why is a narrower claim stronger?

Quick reference

References

Related

25% of My Benchmark Verdict Is an Opinion. Here's the Anatomy.

What are the seven dimensions — and where does each number come from?

How do kilobytes, hook counts, and pass rates become one scale?

What happens when a dimension can't be measured?

Why is Lifecycle Correctness worth 25% — and who decided?

Does the winner survive a ±50% attack on the weights?

Quick reference

External references

Related deep dives

Marketing Pages Rot Silently. Mine Print an Expiry Date on Every Claim.

What does a claim owe you before it ships?

How does a claim live and die?

When does a claim go stale — and who decides?

How do you adopt this in an afternoon?

Quick reference

References

Fixtures First, Rules Second: How to Design a Ground-Truth Corpus

Why do fixtures have to come before rules?

How do you choose categories — and how many of each?

How do you label the edge cases?

How do you make a corpus result reproducible?

What can 40 fixtures actually prove?

What is the corpus lifecycle?

Which findings survive a robustness check?

What are Fixture Cards?

Quick reference

References

Related deep dives

Valid vs. Reliable Metrics: Consistent Numbers Can Still Be Wrong

What Makes a Metric Reliable?

What Makes a Metric Valid?

Reliable, Valid, Both, or Neither?

Is One Valid Metric Enough? Construct Coverage

Quick Reference

Related

References

Static Analysis vs. SAST vs. Linting: The Taxonomy That Matters for Security Teams

The taxonomy

What each level detects

The pattern a security linter catches

The multi-hop path that needs taint analysis

Time-in-pipeline analysis

Why a security linter is not a SAST tool

The recommended stack

Quick reference

References

Sample Size and Statistical Power: What a Small Sample Can and Cannot Tell You

The Two Questions a Small Sample Can Answer

What Statistical Power Actually Means

When Variability Is Real: The Chi-Squared Case

When Your Test Cases Aren't a Random Sample

The Practical Rules

Quick Reference

References

Why this happens: `severity` is the one field nobody enriches