DEV Community

Daniel Pertu
Daniel Pertu

Posted on

We deleted 2,996 lines of process docs for one 504-line playbook, and the renumbering broke 47 code comments

CogniPrep adds assessment providers in rounds, and every round had produced its
own contract document: the roster, the branch names, the dev ports, the standing
decisions, the merge instructions, the verification steps. After five rounds
there were seven of them in docs/:

file lines
provider-expansion-round-2.md 819
provider-expansion-plan.md 560
provider-expansion-round-3.md 563
provider-expansion-round-4.md 370
us-exams-round-2026-11.md 285
tests-round-2026-11.md 220
game-pages-2026-11.md 179

2,996 lines, most of it history: rosters of providers already shipped, ports
nobody is listening on, open questions long since closed. Every new round
started by reading the most recent one and guessing which parts were still rules.

They are now one file, docs/new-provider-playbook.md, 504 lines. The commit is
118 files, 816 insertions, 3,115 deletions, and 96 of those 118 files changed by
exactly one line, because what they contained was a pointer at a document that
no longer exists.

What survived the merge

The playbook opens with a precedence order, which is the part I would keep if I
could keep only one paragraph:

Precedence: the owner's house rules, then the round contract, then the
provider's dossier, then this file's defaults. A dossier supersedes any brief
or starting lead it was given.

That last sentence inverts the usual direction. The research a worker produces
outranks the brief that commissioned it, because every round so far has found
serious errors in its own brief. A round contract still exists per round, but it
is now a short file: roster, branches, ports, per-provider decisions, and a link
here for everything else.

Then twelve numbered standing decisions, and a line under the heading that
explains the numbering: "Numbered because code comments cite them."

That line is the interesting one, because it is true. Across lib, app and
components there are 99 comments that cite a standing decision by number, in
84 files:

 * our allowance, not POST's: round-wide decision 6 lets a full battery be
 * resumed between sections but never inside one, and the brief is the only
Enter fullscreen mode Exit fullscreen mode
  // source for every item (round 2 decision 8). Reading is a fixed-form bank
Enter fullscreen mode Exit fullscreen mode

81 of the 99 name the round they come from. 18 just say "decision 4".

The thing that broke

The old round 2 contract had 14 numbered decisions. The playbook has 12, and it
is not the same list renumbered. One was dropped into the house rules (item text
uses the spelling the candidate will read, rather than "American English in US
item content"), two became prose elsewhere, and the rest were reordered:

round 2 playbook
1 institutional providers 1, unchanged
2 most-and-least scoring 2, unchanged
3 item language 3, unchanged
4 American English in US items gone, folded into house rules
5 speed weighting 4
6 full-battery resume 5
7 hidden clocks 6
8 fact-knowledge banks 7
9 desktop-only instruments 8
10 audio 12
11 trait games 9
12 titles gone, folded into house rules
13 cross-provider imports gone, became the divergence rule in section 5
14 existing content corrections gone, became a required dossier section

So the citations numbered 1 to 3 still resolve. Of the 81 that name a round, 47
cite a number of 4 or higher, and every one of those now points at a rule with a
different meaning in the only document that still exists. A comment saying
"round 2 decision 6" is about resuming a battery between sections; decision 6 in
the playbook is about hidden clocks. Both are real rules. Both are plausible in
the comment. That is the worst kind of broken reference, because nothing fails
and nothing looks wrong.

One comment cites "round-wide decision 13", which no surviving document numbers
at all. It is about importing another provider's component read-only, which is
now a paragraph with no number, so the reference can only be resolved from
git log --diff-filter=D -- docs/.

Those 18 unqualified citations are now ambiguous in the same quiet way: a bare
"decision 4" meant speed weighting under one document and American English under
the other.

What I checked, and what was fine

Since the file is now the single source for a repeated build, I audited every
reference in it. Stripping fenced code, it has 203 inline code spans: 57 name a
path and 73 name an identifier. Every path resolves, including the nine written
as bare filenames that live once per provider folder. Every identifier appears
in the code. The only token in the file that does not exist anywhere is
pdftotext, in a sentence saying it is not installed and to read PDFs another
way.

One prose reference did rot, and it rotted within a day. The playbook tells
workers not to edit files that derive from the provider list, and the list of
those files includes "landing providers section". That component was deleted on
18 November at 18:43. The playbook was written on 19 November at 14:55. There is
no ProvidersSection anywhere in the repository now.

The lesson I actually take from this is not "write fewer documents". It is that
a numbered list in a document is an interface the moment anything cites it, and
renumbering it is a breaking change even though nothing imports it. If I were
doing it again I would keep the old numbers as stable ids and add new decisions
at the end, which is exactly the advice everyone already follows for database
columns and public API fields.

See it

What the playbook produces is checkable per provider, because it prescribes the
same public surface each time. The four providers from the most recent round:

Each hub has the same furniture in the same order, each has a sibling at
/cheating/<provider>, one guide under /blogs/, and one page per named test.

Two of the rules are visible from outside without an account. The first is that
a per-test page ends with one short section stating our own question count and
clock rather than implying the real test's, which you can read at the bottom of
the Ramsay mechanical page:
"Our free practice test gives you 36 questions in 20 minutes". The second is the
title rule, which is why no title in the family runs past 60 characters with the
brand suffix:

curl -s https://cogniprep.app/games/ramsay | grep -o '<title>[^<]*</title>'
Enter fullscreen mode Exit fullscreen mode

Top comments (0)