Why LLM-Generated Flashcards Are Usually Bad, and How to Screen Them

Alex Chen — Sat, 25 Jul 2026 08:11:06 +0000

I've watched a lot of teams bolt an LLM onto a document and call the output flashcards. It demos beautifully and it teaches badly. The generated cards look plausible — grammatical, on-topic, correctly formatted — and they fall apart the moment a real learner tries to review them six weeks later.

The core problem is that "summarize this into Q&A pairs" optimizes for a different objective than "produce items that are cheap to recall and hard to fake." Those two things diverge fast. Here's what I've learned building generation pipelines that survive contact with a review queue.

The Failure Modes Are Predictable

Naive generation produces four bad card shapes, over and over.

Compound questions. "What is the TCP three-way handshake and why does it exist?" That's two cards wearing a trench coat. The learner half-recalls, grades it "Good," and the scheduler now believes both facts are known. You've corrupted the scheduling signal, which is worse than not having the card at all.

Ambiguous prompts. "What year did it happen?" — extracted from a paragraph where "it" was obvious. On the card, out of context, there's no unique answer. The learner fails, marks it Again, and it returns tomorrow to fail identically. This is the single most common defect I see, because the model has the source context in its window and doesn't notice that the card doesn't.

Cloze deletions with too little residual signal. Take "The mitochondrion is the powerhouse of the cell." Blank the subject and you get a reasonable card. Blank two spans and the second one is often recoverable from grammar alone. A cloze that can be solved by syntax trains nothing.

Pattern-matchable answers. If every card in a German-nouns deck has an answer starting with "der," the learner learns the deck, not the material. This only shows up at deck level, so per-card evaluation never catches it.

Atomicity Is a Structural Constraint, Not a Prompt Instruction

Telling the model "make cards atomic" gets you partial compliance and no guarantees. The reliable move is to make the pipeline structurally incapable of emitting a compound card: split extraction from card-writing.

Stage one extracts claims — self-contained propositions with entities already resolved. Stage two turns one claim into one card. A card can't be compound if its input was a single claim.

CLAIM_SCHEMA = {
    "subject": str,        # resolved entity, no pronouns
    "predicate": str,      # exactly one assertion
    "qualifiers": list,    # conditions / scope
    "source_span": str,    # provenance, for verification
}

def extract_claims(chunk) -> list[Claim]:
    # one call, constrained decoding into the schema
    ...

def claim_to_card(claim: Claim) -> Card:
    # template chosen deterministically from claim shape;
    # the model only writes the natural-language surface
    ...

The second win is that pronoun resolution happens at claim time, with the source paragraph still in view. By the time you're writing the card, the subject is already a proper noun, and the "what year did it happen" class of bug becomes unrepresentable rather than merely discouraged.

For cloze specifically, I stopped letting the model choose the deletion span freely. Instead: extract the claim, identify which token carries the claim's information payload (usually the object of the predicate, or a numeric qualifier), and delete that. If a sentence has no high-information span, it doesn't become a cloze — it becomes a Q&A card, or nothing.

Screening Card Quality Without Humans

You can catch most defects with cheap deterministic checks before anything reaches a learner. These are the ones that earned their keep:

def screen(card, deck):
    issues = []

    # compound: coordination joining two verb phrases in the prompt
    if re.search(r"\b(and|as well as)\b", card.front, re.I) and has_two_verb_phrases(card.front):
        issues.append("compound")

    # unresolved reference
    if re.search(r"\b(it|this|they|these|the former|above)\b", card.front, re.I):
        issues.append("ambiguous_reference")

    # answer leakage: content words shared between prompt and answer
    back_words = content_words(card.back)
    overlap = content_words(card.front) & back_words
    if back_words and len(overlap) / len(back_words) > 0.5:
        issues.append("answer_in_prompt")

    # trivial cloze: deleted span is a function word
    if card.type == "cloze" and is_closed_class(card.deleted_span):
        issues.append("trivial_cloze")

    # verbosity: long answers are usually compound in disguise
    if len(card.back.split()) > 20:
        issues.append("answer_too_long")

    return issues

Then two model-based checks worth the tokens:

Answerability without source. Give a fresh model only the card front — no document — and ask it to answer. If it can't produce the expected answer, the prompt is under-specified. If it produces the answer trivially for material the learner shouldn't already know, the prompt is leaking. Batch this per generation job; it's one call for many cards.

Deck-level distinctness. Embed every front and flag pairs above a cosine-similarity threshold. Near-duplicate fronts with different backs are the nastiest failure in spaced repetition: the learner can't tell which card they're looking at and starts guessing from queue position. Anything very close in a decent sentence embedding gets merged or rewritten.

The screening layer rejects a meaningful share of raw generations, and the rejection rate turns out to be the most useful signal we have about source material. A chapter with a high reject rate is usually narrative prose with few extractable claims — which means the fix is the chunking strategy, not the card prompt. That whole extract-screen-regenerate loop is what sits behind card generation in SmartRecall, and it's the part that took longest to get right; the initial "prompt an LLM for flashcards" version took an afternoon.

What I'd Skip Next Time

I spent too long on a learned quality classifier. The deterministic screens plus one answerability check catch nearly everything it would, and they're debuggable — when a card is rejected I can point at the rule that rejected it.

I'd also ship with cloze generation off by default. Q&A cards fail loudly: the learner can't answer. Bad clozes fail quietly: the learner answers from syntax and feels productive. Ship the loud failure mode first.

SM-2 Is Not Enough: Where Classic Spaced Repetition Breaks Down

Alex Chen — Fri, 24 Jul 2026 19:21:26 +0000

SM-2 — the algorithm behind Anki and most spaced repetition tools — was published in 1987 and is remarkably durable. It is also wrong in specific, predictable ways that matter once your deck grows.

What SM-2 does

Each card carries an ease factor, starting at 2.5. Grade a review 0-5; on success the interval multiplies by ease, on failure it resets:

if quality >= 3:
    interval = 1 if reps == 0 else 6 if reps == 1 else round(interval * ease)
    reps += 1
else:
    reps, interval = 0, 1

ease = max(1.3, ease + (0.1 - (5-quality) * (0.08 + (5-quality) * 0.02)))

Simple, no training data, works offline. Genuinely good engineering for 1987.

Failure 1: ease hell

Every lapse decrements ease and it recovers slowly. A card you failed a few times early gets pinned near the 1.3 floor — permanently reviewed at short intervals even after you have learned it perfectly.

The algorithm has no way to say "this card was hard then, it is easy now." Ease is a one-way ratchet in practice.

Failure 2: binary-ish grading carries too much weight

The 0-5 scale collapses in practice to "got it / did not." But how you recalled it matters — instant recall and dredging it up after eight seconds are very different memory states that produce identical grades.

Response latency is a strong signal and SM-2 ignores it entirely.

Failure 3: cards are treated as independent

They are not. Learning "kanji A" helps with "compound containing A." Confusable pairs interfere with each other. SM-2 schedules every card in isolation, so interference-prone cards get scheduled adjacently by chance, which is the worst possible arrangement.

Failure 4: no forgetting curve per user

SM-2 assumes one exponential decay shape. Real retention varies by person, by material type, and by time of day. A fixed curve over-schedules some users and under-schedules others, and neither group can tell.

What modern approaches change

FSRS models memory with three variables — difficulty, stability, retrievability — and fits parameters to your review history. Concretely that means:

Difficulty is not a one-way ratchet
You can target an explicit retention rate (say 0.9) rather than accepting whatever falls out
Latency and grade both inform the update

The cost is that it needs review history to fit, so cold-start is worse. Reasonable compromise: SM-2 defaults until you have a few hundred reviews, then refit.

I work on this at SmartRecall — AI-generated cards with scheduling that does not pin everything to the ease floor.

Caveat

For small decks with few lapses, SM-2 is fine and the added complexity buys nothing. These failures compound with deck size and time — they are invisible at 200 cards and painful at 5000.

DEV Community: Alex Chen