The docs/ folder in CogniPrep is 4.9 MB and almost none of it is developer documentation. It is 46 research dossiers, one per assessment vendor, each written before a line of that vendor's code existed.
46 files
4,876,887 bytes
702,067 words
3,750 http(s) citations
11 to 318 citations per file
The rule, stated at the top of 36 of the 46 files, is that every factual claim carries the URL it came from. Where two sources disagree, both are recorded. Where a source is a reseller or a test-prep site rather than the vendor, it is labelled as such and is never the only support for a claim that matters. A dossier opens with a "headline findings" section listing the things the brief got wrong, which is the part I would keep if I could only keep one.
I have written about the format before. What I had never done is audit the output: of 46 research files, what did each one actually become?
Three dossiers did not become a product
43 of the 46 dossier filenames match a live provider hub at /games/<slug>. The three that do not:
cfat.md researched an armed forces aptitude test and found it had been retired in 2024. There is nothing to build. The dossier became one article, The CFAT Was Replaced: What CAF Applicants Sit, whose entire job is to tell someone searching for the old name what they will actually sit.
adf.md researched a defence force aptitude test and could only tie it to another vendor's battery at medium confidence. Shipping a practice product on a medium-confidence match would mean claiming a candidate is practising the right test when we do not know that. It became an employer page instead, ADF Aptitude Test (JOA), which can describe a format without asserting a vendor.
revelian-rcat.md turned out to be one test inside a vendor we were already building, so it folded into the revelian hub. The dossier slug and the product slug simply differ, which is the least interesting of the three and the one most likely to confuse somebody later.
A dossier that produces an article rather than a product is not a wasted dossier, and that was the thing I wanted the audit to confirm. Two of the three findings are "do not build this", which is the result that saves the most time and the only one you cannot get without doing the research.
The citation counts predict nothing
master.md has 318 citations and shipped a four-test product family. aon.md has 11 and shipped one of the bigger ones. cfat.md has 38 and shipped an article.
The correlation is with how scattered a vendor's documentation is, not with how much there is to build. Vendors that publish one clear page need one citation. Vendors that spread the same facts across a help centre, four country domains and a PDF need forty, and 116 of our citations point at one vendor's .se domain while 64 point at its .dk one, for the same tests.
A quarter of the research needed an archive
168 of the 3,750 citations are web.archive.org, spread across 21 of the 46 files. The word "retired" appears in 22 of them.
That is the part of this work nobody warns you about. Assessment vendors rename products, get acquired, and quietly delete the page that described what a candidate sits, while the candidate is still being sent a link to it. Nearly half our dossiers had to cite a snapshot rather than a live vendor page, and nearly half contain the word retired.
It also means the dossier is the only durable copy of some of these facts. If I re-ran the research today, more of those 3,750 URLs would resolve to nothing than did on the day they were read.
The reverse audit was the uncomfortable one
Going the other way, 13 of the 56 providers live on the site have no dossier at all. All 13 appear in the expansion plans, but as merge anchors or as "the existing library", never as a research target: the process started after they shipped. Which means the vendors carrying the most traffic are the ones whose claims were never held to the "every claim carries a URL" rule at the time they were written.
Nothing in the codebase enforces the rule either. A dossier is markdown, so its own conventions drift: the confidence tag appears 33 times in seven different casings, from Confidence: High to confidence: medium. The employer-to-vendor associations elsewhere in the repo are typed data with a confidence enum the compiler checks. The research those associations came from is prose. Backfilling the 13 is the obvious job and the dull one, and writing this post is the first time I have given it a number.
See it
The three outputs are all public, and the contrast is the whole point:
- A dossier that became a product: cogniprep.app/games/master, from the 318-citation file.
- A dossier that became an article because the test no longer exists: cogniprep.app/blogs/cfat-replaced-by-seaf.
- A dossier that became an employer page because the evidence only reached medium confidence: cogniprep.app/employers/australian-defence-force.
Read the last two next to each other and you can see the rule doing its work. Neither page claims a vendor we cannot evidence, both exist because the research said no, and both still answer the question the person typing that test's name into Google actually has.
One loose end you can also check, which I only found while writing this: the three unshipped slugs still resolve.
curl -s -o /dev/null -w '%{http_code}\n' https://cogniprep.app/games/cfat # 200
That is a soft 404 on a route of ours that renders dynamically, which is a separate bug with its own post. Research that correctly decided not to ship a product still left a URL shaped like one, and nobody noticed for a month.
If you keep research files, audit them by outcome once in a while. Mine were in better shape than I expected on the "no" decisions and worse on coverage, and neither of those is visible from inside any single file.
Top comments (0)