Run a README or a docstring-heavy file through a general-purpose readability checker and you will sometimes get a score that makes no sense relative to how the document actually reads. The culprit is usually the code itself getting swept into the word and sentence count, and formulas built for prose have no concept of what a code block is.
What actually goes wrong
Readability formulas split text into words and sentences using punctuation as the primary signal, periods for sentence boundaries, whitespace for word boundaries. Code does not respect either convention. A single line like const userConfig = await fetchUserConfiguration(userId, { includeDeleted: false }); gets parsed as one enormous "word" by character count, or as multiple short sentence fragments if the formula treats semicolons and periods inside the code as sentence breaks. Either way, the resulting number reflects an artifact of the parsing, not the actual difficulty of the surrounding prose.
Fenced code blocks in markdown are especially prone to this if a readability script naively strips markdown syntax but does not specifically detect and exclude the content inside triple-backtick blocks. The comment text explaining the code might be perfectly clear, grade 8, easy to follow, while the raw code sitting next to it drags a whole-document average toward something meaningless.
Inline code spans cause a subtler version of the same problem
Even outside full code blocks, inline code spans, variableName, functionCall(), --flag-name, get counted as regular words by formulas that are not specifically stripping markdown code syntax first. A sentence like "Set the maxRetries option to control how many times fetchData() will retry before it throws a NetworkTimeoutError" contains three inline code spans that will inflate a naive word-length or syllable count, even though a developer reading that sentence experiences it as perfectly normal technical prose.
How to get a number that actually means something
Before running a readability check on technical content, strip fenced code blocks and inline code spans first, and run the formula only against the remaining prose. Most markdown parsing libraries make this straightforward, since code blocks and inline code spans are already distinctly tagged in the parsed document tree. A reading level analyzer or similar tool applied to raw markdown without this preprocessing step will produce numbers that are not comparable across documents with different amounts of embedded code, which makes tracking a readability trend over time actively misleading.
Comments extracted from source code deserve the same treatment, but from the opposite direction: pull just the comment text out of the source file and run the formula against that, ignoring the code it is attached to. This gives you an honest signal about whether your inline documentation itself is clear, separate from whatever the code surrounding it looks like.
Why this matters beyond getting a clean number
Teams that try to enforce a readability standard on technical writing without handling code exclusion end up with a check that fires unpredictably, sometimes flagging genuinely clear prose because of an unrelated code block sitting nearby. That kind of noise is exactly what causes a team to stop trusting an automated check and eventually disable it, even though the underlying idea, keeping documentation prose readable, was sound. Fixing the code-stripping step first is what makes the rest of a readability-based quality process actually usable for technical content.
A worked example showing the size of the distortion
Take a short README section: three sentences of plain explanatory prose, followed by a five-line code block, followed by two more sentences of prose. Run the whole section through a formula with no code stripping and you might see a reported grade level of 16 or higher, driven entirely by the code block's long variable names and lack of natural sentence boundaries. Strip the code block first and run the same formula against just the five sentences of prose, and the honest number might be grade 8 or 9, a completely different picture of how accessible the actual writing is.
This gap is not a minor rounding difference. It is large enough to make a document look like it needs a rewrite when the writing itself is already fine, or worse, to hide a genuinely dense paragraph of prose behind a code block that happens to offset the average in the other direction. Neither outcome helps a team trying to actually improve their documentation, which is why code stripping has to happen before the formula runs, not as an afterthought applied to the output.
Building this into your own tooling
If you are writing a script to check documentation readability across a repository, budget time for the preprocessing step before you budget time for picking a formula. A regex-based approach that strips triple-backtick fenced blocks and single-backtick inline spans will handle the majority of cases in standard markdown. YAML frontmatter blocks, HTML comments, and embedded configuration snippets deserve the same treatment, since none of them represent the actual prose you are trying to evaluate.
Test your stripping logic against a few real files from your own documentation before trusting the readability numbers it produces. Documentation with heavy use of inline code for configuration keys, environment variable names, or CLI flags will show the distortion most clearly, and confirming your preprocessing handles those cases correctly is worth the extra half hour before you start acting on the scores it generates.
Tables, admonition blocks, and other markdown extensions
Fenced code and inline code spans are the most obvious culprits, but they are not the only markdown constructs that confuse a naive readability check. Tables get rendered as rows of pipe-separated cells in raw markdown, and a formula run against the raw table syntax will count table borders and alignment markers as word-like tokens, producing nonsense. Admonition blocks, the colored "note" or "warning" boxes many documentation frameworks support, often use custom syntax extensions beyond standard markdown, which a generic stripping script will not recognize unless you specifically account for the framework you are using.
The safest approach is to render the markdown to a structured format first, an abstract syntax tree or a parsed HTML document, and then walk that structure extracting only the nodes that represent actual prose paragraphs and list items. This is more setup work than a quick regex, but it scales much better across a documentation site that uses tables, admonitions, and custom components alongside plain paragraphs, since each node type can be explicitly included or excluded rather than guessed at with pattern matching that will eventually miss an edge case.
What this looks like in an actual CI check
Once the stripping and parsing step is solid, wiring it into continuous integration is straightforward: run the readability formula only against the extracted prose nodes for each changed file, compare against a baseline, and flag genuine outliers. The value of doing the preprocessing correctly compounds here, because a CI check that occasionally fires on a false positive from an unstripped code block will train contributors to ignore it within a few weeks, the same failure mode any noisy automated check eventually suffers from regardless of how good the underlying idea was.
For a broader look at why different readability formulas can disagree with each other even on ordinary prose, see EvvyTools' explainer on why readability scores disagree on the same paragraph.
References: the CommonMark specification documents exactly how fenced code blocks and inline code spans are defined in markdown, the Google Developer Documentation Style Guide covers writing clear inline comments and prose separately from code samples, and the Write the Docs community maintains further resources on documentation tooling and quality checks for technical writing teams.
Top comments (0)