How we measure the real CEFR level of a book, and where it still fails

NewGen Books — Tue, 03 Nov 2026 21:40:28 +0000

Most graded readers have a level printed on the cover. Almost nobody can tell you how it was decided.

We publish free books at books.dev, each in the original English and many in simpler editions, and we wanted a number on every edition that we could defend. Here is how our meter works today, how its first version failed, and what it still can't do.

Why a readability formula wasn't enough

Classic formulas such as Flesch-Kincaid look at the length of words and sentences. We compared them with teachers' difficulty judgments on a public set of passages, and they underrate fiction: a novel can read as much easier than teachers find it. Short words can hide a lot. For a library of novels, that bias sits exactly where we need accuracy.

What the meter combines

The score blends three signals:

A CEFR text classifier (weight 0.70)
Vocabulary (0.25): average word level, the level at which 95% of the text is covered, and rare lemmas per 1,000 words
Syntax (0.05): 90th percentile sentence length, parse tree depth, subordinate clauses

The sum then passes through isotonic regression, a monotonic mapping to the final score. We show it with tenths, such as B1.5.

Version 1 was calibrated against itself

The first version had hand-set weights, and its vocabulary and syntax parts were fitted to the classifier's own answers. Only two reference points had a known level. It looked fine until we tested it on exam texts with official levels: both C1 and C2 came out around B2.8. The scale was compressed at the top.

Version 1.1 was fitted to texts of known level instead. The important decision was how to choose the weights. Cross-validating open corpora against each other picked almost pure classifier (0.95), which looks great on those corpora and generalizes worse. So we held out one exam board at a time, trained on everything else, and tested on the board the model had never seen. The mean absolute error on 200 unseen exam texts fell from 0.70 to 0.51 of a level.

The calibration texts belong to their owners, so we keep them private and publish only our own results.

Exams are not novels

Version 1.1 still had an open question: does a scale fitted on exam papers carry over to fiction? Against those teacher judgments it reached a Spearman correlation of 0.70, and it placed fiction 0.134 lower than non-fiction of equal difficulty. Version 1.2 adds that offset for fiction. It's a small correction, but it comes from a measurement rather than a hunch.

What it looks like on a real book

Alice's Adventures in Wonderland measures C1.2 in the original. Our adapted editions measure B2.1 and B1.5. A reader sees all three before choosing one.

What it still can't do

It can't separate C1 from C2. No version does, so C2 gets a bare label with no tenths rather than a false precision.
Short texts are noisy. The error on a single short passage is about half a level.
It measures the text, not the reader. Background knowledge and interest change how hard a book feels, and no number on a page captures that.

If you've worked on readability, I'd like to know what you would test next. The library is free at books.dev.