Development covered 5 Aug 2026 to 8 Aug 2026 (commit dates).
This stretch was a long automated review over the tie-in novel, and the useful lessons are all about running a checking campaign rather than about the book. Any project that ever points a checker at a large body of existing work will meet the same shapes, so here they are. For a sense of the material, the Nascent Soul act of the story is a typical stretch of what was under review.
Three rules for triage
The first full pass produced around three hundred findings. Nobody reads three hundred findings; they skim twenty and lose faith.
Sampling them against the source gave a stable ratio that has held all the way through: about a third are genuine, about a third are the checker misreading its input, and about a third are real inconsistencies that were put there on purpose and would be wrong to remove.
Three discard rules aimed at the most common false shapes took that run down to fifty eight items, and re-checking a sample of what was dropped found no genuine errors among them. Cutting noise was worth more than any additional check I could have written that week, because the report went from being ignored to being worked through.
Two empty rounds means change the activity
The most useful piece of process here is a stopping rule.
A checking campaign has a yield curve. The first pass finds a lot, the second finds a decent amount, and there is a point after which each pass costs the same and finds nothing. Without a rule you keep going, because each round is cheap and might find something, and you can burn a week that way.
Mine is now: two consecutive rounds with nothing genuine, and the activity changes. Not the checks, the activity. The material has given up what this method can extract, and a different method, or different material, is where the next finding lives.
A metric that never moves is usually missing its denominator
One number in the reporting sat at zero for weeks and I kept meaning to look into it.
It was measuring how many setups planted earlier in the book eventually pay off. Zero, always. The reason was not that nothing pays off, it was that the pool of tracked setups was empty, because the book had been imported into the tooling in a way that never populated it. The metric was dividing by nothing and reporting the result as a failing grade.
The general instinct I want to keep: a metric pinned at exactly zero or exactly one hundred is far more likely to be broken than extreme. Check the denominator before you go looking for the cause.
There is a matching case in the same tooling: a status meaning the checker could not finish a chapter was being read as the chapter having a problem. Eighty four chapters carried it. Sorting the whole set produced no genuine errors at all. A category that means unknown must not be presented next to categories that mean bad.
Parallelise the reading half
One performance note, since it changed the shape of a whole week.
The pipeline held a lock around everything, on the reasonable-sounding grounds that it edits files. But the review half only reads, and only the editing half writes. Splitting them so reviews run in parallel while edits stay serial took a whole-book pass from about sixty five hours to sixteen.
That was sitting there for weeks behind a note in my own documentation that claimed reviews needed the lock. The note was written by me, from an assumption, and never tested. Worth checking which of your constraints were measured and which were merely written down confidently.
Fixing the instance is not fixing the fact
Last one, and the mistake I most want to remember.
When a check names a wrong value in one chapter, the temptation is to fix that chapter. But a recurring fact appears in many places, and the checker only reported where it happened to notice. Changing the one it named leaves every other occurrence contradicting the correction, which is worse than before, because now the work is inconsistent rather than uniformly wrong.
Any change to a fact that recurs has to be applied by searching the whole body of work for that fact, not by editing the location in the report. The report tells you a fact is wrong. It does not tell you where the fact lives. For terms, the nearest thing to a single home for a recurring fact is the site's short dictionary of xianxia words.
Playable today: jump in from the browser without installing anything, with every devlog collected under Myriad Immortal Sect.
Top comments (0)