DEV Community

Jeremy Longshore
Jeremy Longshore

Posted on Edited on Originally published at startaitools.com

Seventeen Spellings of the Same Number

The Tons of Skills marketplace renders skill-count numbers on its public pages. Those counts were bare. A number like "3,068 skills" does not say which population it counted. Different pages could legitimately count different things and a reader could not tell.

The rule went in: every published count must declare its cohort and expose a reproducible resolver command beside it. I built a registry at scripts/published-count-cohorts.json and a CI gate that discovers counts in Astro source and binds each one to a registered cohort.

Then the night began. One feature commit at 20:31, then seventeen fixes landing through 03:43.

Here is the full chain, with the checker's line count and its test file's line count after each commit. The checker grew about two and a half times, from 473 lines to 1,241, while the tests went from 331 to 827. The two biggest single jumps were at 21:52 (binding every expression, plus 157 lines of checker) and 23:17 (closing independent count bypasses, plus 220 lines of checker). Net for the entire chain, from the commit before the feature through the last fix: 30 files, plus 2,986 lines, minus 157. Commit subjects below are abbreviated; the real ones carry a marketplace-site scope.

20:31 1296df609 feat: label published count cohorts              checker= 473L  tests=331L
20:55 33803b51a fix: broaden published count discovery           checker= 536L  tests=418L
21:29 412a615fa fix: reject trailing comment provenance          checker= 536L  tests=450L
21:31 9ed3c33fc fix: cover compact line comments                 checker= 536L  tests=462L
21:52 c9c4015d5 fix: bind every published count expression       checker= 693L  tests=558L
21:59 2443cf54f fix: require rendered count provenance           checker= 716L  tests=613L
22:07 8e976f0d2 fix: enforce rendered count labels               checker= 723L  tests=639L
22:17 b513d7b4c fix: require exact count expressions             checker= 721L  tests=672L
22:24 d538e6b2e fix: reject compound count wrappers              checker= 721L  tests=672L
22:35 0eba12625 fix: parse rendered provenance tags              checker= 784L  tests=690L
22:47 a62f6f946 fix: mask quoted markup attributes               checker= 795L  tests=706L
23:17 6166a7f19 fix: close independent count bypasses            checker=1015L  tests=747L
23:37 ad0dee2bd fix: detect cross-line count expressions         checker=1048L  tests=752L
23:56 79701c643 fix: align count discovery with rendered structure checker=1107L tests=777L
00:11 7ae7f9028 fix: bind counts to heading containers           checker=1132L  tests=779L
00:19 1680d7df6 fix: preserve external heading counts            checker=1169L  tests=781L
00:55 e2f759cc6 fix: harden count discovery exclusions           checker=1193L  tests=783L
03:43 145c8725a fix: govern query-local skill counts             checker=1241L  tests=827L
Enter fullscreen mode Exit fullscreen mode

Each commit closed a spelling the previous checker had missed.

Why Not the Obvious Approaches

A grep for a known variable name fails the moment someone renames it. The failure is silent.

Rendering the site and checking HTML would sidestep most parsing entirely. But it checks output, not the source a reviewer reads in a diff. A reviewer still cannot see the violation at review time.

Manual review is what this replaced. It let unlabeled counts accumulate across five pages.

I chose source parsing with fixtures. The cost is 1,241 lines of checker and 827 lines of tests.

How the Gate Works

The gate has three parts. A registry lists every cohort alongside a command that re-derives its number. A CI check discovers count expressions in Astro source and refuses any that is not bound to a registered cohort. A component renders the label and that command beside the number. It fails closed.

The registry lives at scripts/published-count-cohorts.json. The component is CountProvenance.astro, 48 lines. The check runs as a step inside the existing validate job in .github/workflows/validate-plugins.yml, deliberately, so that it adds no fourth required status context.

Here is the cohort half of the registry, with marketplace-visible spelled out. The real file also carries discovery, surfaces, and deferredGroups sections:

{
  "schemaVersion": 1,
  "cohorts": {
    "marketplace-visible": {
      "label": "marketplace-visible",
      "description": "Tracked plugin skills reachable from canonical marketplace catalog source roots; hidden adapter trees, the root curriculum, and the curated mirror are excluded.",
      "command": "node scripts/corpus-resolver.mjs --cohort marketplace-visible --json",
      "resolver": "scripts/corpus-resolver.mjs"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The other four cohorts are graded, first-party, curated-mirror, and curriculum. A reader looking at any public page can now see which population a count covers and run the command themselves to verify the number.

Calibrate the rest of this against one number before reading on: the enforced surface is five pages (homepage, explore, compare-marketplaces, sponsor, and the skills directory) out of 51 inventoried Astro files. Everything below is what it cost to govern those five honestly. The section near the end on what is enforced and what is deferred says what the other forty-six are and why.

The Spellings the Checker Missed

Each fix was parser hardening: teaching the discovery pass one more legal form it had not seen.

Fake provenance in comments. A trailing comment or compact line comment mentioning a cohort satisfied the naive "is a label nearby" check. Comments now get masked while preserving line geometry:

function stripComments(source) {
  return source
    .replace(/<!--[\s\S]*?-->/g, (comment) => comment.replace(/[^\n]/g, ' '))
    .replace(/\/\*[\s\S]*?\*\//g, (comment) => comment.replace(/[^\n]/g, ' '))
    .replace(/(?<!:)\/\/.*$/gm, maskText);
}

function maskText(value) {
  return value.replace(/[^\r\n]/g, ' ');
}
Enter fullscreen mode Exit fullscreen mode

Every branch replaces characters with spaces rather than deleting them. That is what keeps line and column numbers valid for the passes that run afterward, which matters because the error messages point at a file and a position a human has to open.

The (?<!:) is critical. Without it, https:// gets masked as a comment.

Provenance not actually rendered. A cohort string in Astro frontmatter, in `

Top comments (0)