Summary
In a repository I run on my own for research papers and research operations, I rewrite research prose until the tests pass, the same way I would with code.
I have turned the quality checks into a set of tests written in YAML. The ones a machine can decide are judged by code functions, and the ones that cannot be decided without reading the meaning, such as how an argument is built, are judged by a large language model acting as a reviewer. When I get a point in review, I turn it into a rule on the spot, so that a person does not have to make the same point again.
But the formal checks that mechanically constrain things like banned words and sentence length, far from raising quality, erased the individuality of the writing and made the prose uniform, so partway through I removed the checks I had been adding. When a test fails, it returns a suggested fix, and it tells me whether that failure can be fixed by rewriting.
Rewriting until the tests pass
In software development, there are tests as a way to automatically check whether the code you wrote works as expected. For a while now I have been writing research prose with the approach of rewriting it until those tests pass. Tests for prose might sound strange, but what I am doing is simple. I take the points I get in review and turn them into check rules on the spot, so that a human never has to make the same point twice. That is all it is. For example, there was a time when I turned three points I received at once into rules on the spot and fixed two pieces of writing together.
I split the checks into two layers. The ones a machine can decide, like character counts and syntactic consistency, are judged by code functions, and the ones that cannot be decided without reading the context, like the overall arc of a piece from opening to close, or the flow of cause and effect, are left to a large language model acting as a reviewer. The meaning-reading checks also far outnumber the mechanical ones. At the time of writing this article, there were 131 checks handed to the reviewer and 15 checks seen by machine functions. I keep this whole set of checks as a suite of tests in YAML, and running it against the whole piece returns a pass or a fail for each check.
With this, the basis for rewriting changed too. Now I look at the list of failing checks and decide from it what to fix.
Dropping the checks that were only for form
This setup was not in its current form from the start. When I began building it, I was heading toward turning every point I received into a rule without omission, and I was adding checks a machine can decide, like a list of banned words and an upper limit on sentence length.
But checks that are only for form, like being too strict about character counts, forcing a number of paragraphs, or mechanically banning particular turns of phrase, look at first like they raise quality, while in fact they strip the style out of the writing and make it uniform. On its own, a word may not make clear what it refers to, but if the sentence just before explains it, the meaning comes through. Even usages that hold only because of context, the machine checks reject across the board.
So partway through I changed direction, and I left on the machine side only the checks whose answer is uniquely determined, while I moved the judgments involving meaning and logic toward having the reviewer read them together with their context. To each check I kept, I have tied a source for why that rule exists, one by one, so that whether the basis is a document that sets writing conventions or a point received in review, it can be traced later.
A failing test returns how to fix it
Another part of the design is that when a test fails, the test side returns how it should be fixed. Every failing check comes with a suggested fix, and it also tells me whether that failure can be fixed by rewriting or whether the underlying data is simply not enough. A failure that cannot be fixed is not a problem with the writing but a task of data collection. If the data is not enough, I do not touch the writing and just go back to measuring.
I have also decided how to handle the case where the reviewer's judgment could not be run. A check that did not run goes into a count of its own; it does not stand as a failure. If a failed judgment and a flaw in the writing get mixed together, the number of failures itself stops being usable as a guide to where I should fix things.
What I could confirm goes only as far as stopping recurrence
The checks that turn points into rules accumulate on the side of the system. A person no longer has to hold them in memory. The test results now let me confirm that the checks have stopped the same kind of point from coming back. But that is as far as I can say for certain about the effect right now, and whether the prose written this way has actually become easier for readers to read is not something the passing of a test can tell me.
I also still cannot put a number on how consistently the reviewer returns the same judgment for the same input. I have put in a mechanism that judges two more times only when the first judgment is not a pass and takes a majority vote over the three, but whether that makes the judgment stable is not something I have been able to confirm. If the judgment wavers, trust in the list of failing checks itself is shaken, so I see this stability as the single most important thing to confirm, since it bears on the premise of the setup. Extending this setup to long-form deliverables is a matter still further ahead, and what I hold for certain right now is only this one point: that the checks stop the recurrence of the same point, so a person does not have to say it twice.
(Added 2026-07-11: In this article as published, I had written the mechanism that takes a majority vote over multiple judgments as a plan not yet in place. When I checked the past records during an operational review, I found that a limited mechanism, which judges two more times and takes a majority vote only when the first judgment is not a pass, was already in place on 2026-06-24, before publication. So I have corrected the passage above to read that the mechanism is in place but its stability cannot yet be measured.)
Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.
Top comments (0)