DEV Community

Cover image for Add Readability Scoring to Your CI Pipeline (GitHub Actions)
ckmtools
ckmtools

Posted on

Add Readability Scoring to Your CI Pipeline (GitHub Actions)

#node #typescript #javascript #webdev

Your CI pipeline lints code, runs tests, and checks formatting. But what about your documentation? If a pull request makes your README harder to read, nothing catches it. Typos get flagged; readability regressions don't.

Adding automated readability testing to your CI pipeline takes about 10 minutes. This tutorial shows how to set it up with GitHub Actions and textlens.

Why Test Readability in CI?

Documentation drifts. A developer adds a complex explanation. Another one writes a sentence with four nested clauses. Over months, the grade level creeps from 8 to 14, and nobody notices until users start complaining that the docs are hard to follow.

Readability tests catch this the same way type checks catch type errors — automatically, on every PR.

What you can enforce:

  • Maximum grade level (e.g., docs must stay below grade 12)
  • Minimum Flesch Reading Ease score (e.g., above 40)
  • Maximum average sentence length (e.g., under 25 words)
  • Sentiment checks (e.g., changelogs shouldn't sound negative)

Setup

Add textlens to your project:

npm install textlens --save-dev
Enter fullscreen mode Exit fullscreen mode

Option 1: Simple Shell-Based Check

The fastest approach. Add this step to any existing workflow:

# .github/workflows/docs.yml
name: Docs Quality
on: [pull_request]

jobs:
  readability:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Check docs readability
        run: |
          npx textlens docs/README.md --json | node -e "
            const data = JSON.parse(require('fs').readFileSync('/dev/stdin', 'utf8'));
            const grade = data.readability.consensusGrade;
            const ease = data.readability.fleschReadingEase.score;
            console.log('Grade: ' + grade.toFixed(1) + ', Flesch Ease: ' + ease.toFixed(1));
            if (grade > 12) {
              console.error('FAIL: Grade level ' + grade.toFixed(1) + ' exceeds maximum of 12');
              process.exit(1);
            }
            console.log('PASS');
          "
Enter fullscreen mode Exit fullscreen mode

This checks a single file. The workflow fails if the consensus grade exceeds 12.

Option 2: Check All Markdown Files

For projects with multiple docs:

      - name: Check all docs
        run: |
          npm install textlens
          node -e "
            const { readability, statistics } = require('textlens');
            const fs = require('fs');
            const path = require('path');

            const MAX_GRADE = 12;
            const docsDir = 'docs';
            const files = fs.readdirSync(docsDir)
              .filter(f => f.endsWith('.md'))
              .map(f => path.join(docsDir, f));

            let failed = false;

            for (const file of files) {
              const text = fs.readFileSync(file, 'utf8');
              if (text.trim().length < 100) continue; // skip very short files

              const r = readability(text);
              const s = statistics(text);
              const grade = r.consensusGrade;
              const status = grade <= MAX_GRADE ? 'PASS' : 'FAIL';

              console.log(status + ' ' + file + ' (grade ' + grade.toFixed(1) + ', ' + s.words + ' words)');

              if (grade > MAX_GRADE) failed = true;
            }

            if (failed) {
              console.error('\nReadability check failed. Simplify the flagged documents.');
              process.exit(1);
            }
            console.log('\nAll docs pass readability check.');
          "
Enter fullscreen mode Exit fullscreen mode

Option 3: A Dedicated Script with Full Reporting

For teams that want detailed output, create a script in your repo:

// scripts/check-readability.js
const { readability, statistics, sentiment } = require('textlens');
const fs = require('fs');
const path = require('path');

const config = {
  maxGrade: 12,
  minFleschEase: 30,
  maxAvgSentenceLength: 25,
  dirs: ['docs', 'content'],
  extensions: ['.md', '.mdx'],
};

function findFiles(dir) {
  if (!fs.existsSync(dir)) return [];
  return fs.readdirSync(dir, { recursive: true })
    .filter(f => config.extensions.some(ext => f.endsWith(ext)))
    .map(f => path.join(dir, f));
}

const files = config.dirs.flatMap(findFiles);
if (files.length === 0) {
  console.log('No markdown files found. Skipping readability check.');
  process.exit(0);
}

let failures = 0;
const results = [];

for (const file of files) {
  const text = fs.readFileSync(file, 'utf8');
  if (text.trim().length < 100) continue;

  const r = readability(text);
  const s = statistics(text);
  const grade = r.consensusGrade;
  const ease = r.fleschReadingEase.score;
  const avgSentence = s.avgSentenceLength;

  const issues = [];
  if (grade > config.maxGrade) issues.push(`grade ${grade.toFixed(1)} > ${config.maxGrade}`);
  if (ease < config.minFleschEase) issues.push(`Flesch ${ease.toFixed(0)} < ${config.minFleschEase}`);
  if (avgSentence > config.maxAvgSentenceLength) issues.push(`avg sentence ${avgSentence.toFixed(0)}w > ${config.maxAvgSentenceLength}w`);

  const status = issues.length === 0 ? 'PASS' : 'FAIL';
  if (issues.length > 0) failures++;

  results.push({ file, status, grade, ease, avgSentence, words: s.words, issues });
}

// Print results
console.log('\nReadability Report\n' + '='.repeat(60));

for (const r of results) {
  const icon = r.status === 'PASS' ? '✓' : '✗';
  console.log(`${icon} ${r.file}`);
  console.log(`  Grade: ${r.grade.toFixed(1)} | Flesch: ${r.ease.toFixed(0)} | Words: ${r.words} | Avg sentence: ${r.avgSentence.toFixed(0)}w`);
  if (r.issues.length > 0) {
    console.log(`  Issues: ${r.issues.join(', ')}`);
  }
}

console.log('\n' + '='.repeat(60));
console.log(`${results.length} files checked. ${failures} failed.`);

if (failures > 0) {
  console.log('\nTip: shorten sentences and replace complex words to lower grade levels.');
  process.exit(1);
}
Enter fullscreen mode Exit fullscreen mode

Reference it in your workflow:

      - name: Check readability
        run: node scripts/check-readability.js
Enter fullscreen mode Exit fullscreen mode

Sample output:

Readability Report
============================================================
✓ docs/getting-started.md
  Grade: 7.2 | Flesch: 62 | Words: 842 | Avg sentence: 14w
✓ docs/api-reference.md
  Grade: 10.8 | Flesch: 44 | Words: 1203 | Avg sentence: 18w
✗ docs/advanced-configuration.md
  Grade: 14.1 | Flesch: 28 | Words: 567 | Avg sentence: 27w
  Issues: grade 14.1 > 12, avg sentence 27w > 25w

============================================================
3 files checked. 1 failed.

Tip: shorten sentences and replace complex words to lower grade levels.
Enter fullscreen mode Exit fullscreen mode

Option 4: PR Comment with Results

Post readability results as a PR comment so reviewers see the scores:

      - name: Readability report
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const { readability, statistics } = require('./node_modules/textlens');
            const fs = require('fs');

            const files = fs.readdirSync('docs').filter(f => f.endsWith('.md'));
            let body = '## Readability Report\n\n| File | Grade | Flesch | Words |\n|---|---|---|---|\n';

            for (const file of files) {
              const text = fs.readFileSync('docs/' + file, 'utf8');
              if (text.trim().length < 100) continue;
              const r = readability(text);
              const s = statistics(text);
              const icon = r.consensusGrade <= 12 ? '✅' : '❌';
              body += `| ${icon} ${file} | ${r.consensusGrade.toFixed(1)} | ${r.fleschReadingEase.score.toFixed(0)} | ${s.words} |\n`;
            }

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });
Enter fullscreen mode Exit fullscreen mode

This creates a comment on the PR with a table showing each file's readability metrics.

Configuring Thresholds

Different projects need different standards:

Project type Max grade Min Flesch Max avg sentence
User-facing docs 8 60 20
API reference 12 40 25
Internal docs 14 30 30
Blog posts 9 55 22

Start with grade 12 and tighten as your docs improve. Jumping straight to grade 8 on an existing codebase creates too much noise.

What textlens Measures

The consensus grade combines 7 readability formulas:

  • Flesch-Kincaid Grade — sentence length + syllables per word
  • Gunning Fog Index — penalizes 3+ syllable words
  • Coleman-Liau Index — uses character counts (deterministic)
  • SMOG Index — focuses on polysyllabic words
  • Automated Readability Index — character-based
  • Dale-Chall — compares against a familiar-word list
  • Linsear Write — weights easy vs hard words

One number from seven formulas. More reliable than picking any single formula.

Tips for Lowering Grade Levels

When CI flags a document, here's how to fix it:

  1. Split long sentences. If a sentence has more than 20 words, break it at the conjunction.
  2. Replace complex words. "utilize" → "use", "implement" → "build", "approximately" → "about".
  3. Cut unnecessary clauses. "The function, which was originally designed for internal use, processes the input" → "The function processes the input."
  4. Use active voice. "The config is loaded by the parser" → "The parser loads the config."
  5. Avoid stacking qualifiers. "extremely important critical security update" → "critical security update."

Links

This is part of the textlens series — tutorials on text analysis in JavaScript and TypeScript.

Top comments (2)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.