DEV Community

fei mai
fei mai

Posted on

Measuring 37,426 Generated Answers to Find Out What the Prompt Actually Controls

Our flashcard generator produces answers that all come out around sixty words. I assumed the prompt had a word target. It doesn't — it has a sentence target, and the word count is downstream of how dense the source book happens to be.

Here's how that got measured, and why the first explanation I wrote down was wrong.

The data is already in the HTML

Every deck page ships its whole deck inline:

window.BC_CARDS = [{ "id": "a--01", "ch": "c01", "q": "...", "a": "...", "ref": "Chapter 1" }, ...];
Enter fullscreen mode Exit fullscreen mode

So there's no API to write, no headless browser, no pagination. Fetch 217 sitemap URLs, keep the 174 that are decks, pull the array with a regex, json.loads it. 67 seconds on four threads. 37,426 cards.

One deck broke the parser: Unterminated string starting at char 95,083. A quote escaping bug in our own build, not in the content. It costs one deck out of 174 and it's on the list.

The distribution

Words Cards Share
<30 15 0.04%
30–39 268 0.7%
40–49 2,611 7.0%
50–59 13,027 34.8%
60–69 16,261 43.4%
70–79 4,687 12.5%
80–89 499 1.3%
90–99 48 0.13%
100+ 10 0.03%

Mean 60.93, σ 8.25, median 61, min 25, max 109.

Three results

No truncation anywhere. Zero of 37,426 answers end without terminal punctuation. And there's no pile-up at a ceiling — the max of 109 occurs exactly once, 58 cards clear 90 words, 10 clear 100, none clear 120. A length clamp produces a spike at the clamp value plus a pile of unfinished sentences. Neither is present.

Sentence count is what's pinned, not word count. 86.7% of the library is two or three sentences (14,722 and 17,731 respectively). Answer length is then sentences × words per sentence, and the second term is the one with range in it.

Deck Cards Words Sentences Words/sentence
Meditations 25 40.88 2.68 16.19
The Art of War 24 43.29 2.79 16.27
The Prince 32 46.41 2.62 18.49
Tao Te Ching 270 50.12 2.51 21.05
Discourses on Livy 531 59.09 2.51 25.06

Site average is 24.50 words per sentence. The five shortest decks are the five aphoristic classics, in the exact order of their sentence density. Not one narrative history among them.

Nothing else predicts it. Correlation between deck card count and mean answer length: 0.04 over 174 decks. Between question length and answer length: 0.05. Chapter structure is unchanged from the previous run — 7,963 of 10,050 chapters hold exactly four cards (79.2%), and only 4 answer pairs out of 37,426 share their first 120 characters.

The bit I can't measure

The payload has no chapter text. I can't tell whether Meditations produces 16-word sentences because Marcus Aurelius wrote short lines or because that deck was generated with a different setting. Identical output either way. I also didn't validate any answer against its book.

Retraction

Six days ago I wrote that the answers "sit in a 60-word band, which suggests a 60-word target." Wrong cause, right arithmetic. If the prompt constrains anything, it's the sentence count; the word band is a side effect of the source's own density. Which means the useful knob for making answers longer is not "add a sentence." It's picking a book that writes longer ones.

Disclosure: WakeCards is ours. Everything above came from crawling our own pages on 19 November 2026 — 37,426 cards, all public, all reproducible with curl and forty lines of Python.

If you've generated Q&A pairs from text before: does your output length track the source, or the prompt? I've stopped trusting my own guess about which.

Top comments (0)