Why LLM-Generated Flashcards Are Usually Bad, and How to Screen Them

I've watched a lot of teams bolt an LLM onto a document and call the output flashcards. It demos beautifully and it teaches badly. The generated cards look plausible — grammatical, on-topic, correctly formatted — and they fall apart the moment a real learner tries to review them six weeks later.

The core problem is that "summarize this into Q&A pairs" optimizes for a different objective than "produce items that are cheap to recall and hard to fake." Those two things diverge fast. Here's what I've learned building generation pipelines that survive contact with a review queue.

The Failure Modes Are Predictable

Naive generation produces four bad card shapes, over and over.

Compound questions. "What is the TCP three-way handshake and why does it exist?" That's two cards wearing a trench coat. The learner half-recalls, grades it "Good," and the scheduler now believes both facts are known. You've corrupted the scheduling signal, which is worse than not having the card at all.

Ambiguous prompts. "What year did it happen?" — extracted from a paragraph where "it" was obvious. On the card, out of context, there's no unique answer. The learner fails, marks it Again, and it returns tomorrow to fail identically. This is the single most common defect I see, because the model has the source context in its window and doesn't notice that the card doesn't.

Cloze deletions with too little residual signal. Take "The mitochondrion is the powerhouse of the cell." Blank the subject and you get a reasonable card. Blank two spans and the second one is often recoverable from grammar alone. A cloze that can be solved by syntax trains nothing.

Pattern-matchable answers. If every card in a German-nouns deck has an answer starting with "der," the learner learns the deck, not the material. This only shows up at deck level, so per-card evaluation never catches it.

Atomicity Is a Structural Constraint, Not a Prompt Instruction

Telling the model "make cards atomic" gets you partial compliance and no guarantees. The reliable move is to make the pipeline structurally incapable of emitting a compound card: split extraction from card-writing.

Stage one extracts claims — self-contained propositions with entities already resolved. Stage two turns one claim into one card. A card can't be compound if its input was a single claim.

CLAIM_SCHEMA = {
    "subject": str,        # resolved entity, no pronouns
    "predicate": str,      # exactly one assertion
    "qualifiers": list,    # conditions / scope
    "source_span": str,    # provenance, for verification
}

def extract_claims(chunk) -> list[Claim]:
    # one call, constrained decoding into the schema
    ...

def claim_to_card(claim: Claim) -> Card:
    # template chosen deterministically from claim shape;
    # the model only writes the natural-language surface
    ...

The second win is that pronoun resolution happens at claim time, with the source paragraph still in view. By the time you're writing the card, the subject is already a proper noun, and the "what year did it happen" class of bug becomes unrepresentable rather than merely discouraged.

For cloze specifically, I stopped letting the model choose the deletion span freely. Instead: extract the claim, identify which token carries the claim's information payload (usually the object of the predicate, or a numeric qualifier), and delete that. If a sentence has no high-information span, it doesn't become a cloze — it becomes a Q&A card, or nothing.

Screening Card Quality Without Humans

You can catch most defects with cheap deterministic checks before anything reaches a learner. These are the ones that earned their keep:

def screen(card, deck):
    issues = []

    # compound: coordination joining two verb phrases in the prompt
    if re.search(r"\b(and|as well as)\b", card.front, re.I) and has_two_verb_phrases(card.front):
        issues.append("compound")

    # unresolved reference
    if re.search(r"\b(it|this|they|these|the former|above)\b", card.front, re.I):
        issues.append("ambiguous_reference")

    # answer leakage: content words shared between prompt and answer
    back_words = content_words(card.back)
    overlap = content_words(card.front) & back_words
    if back_words and len(overlap) / len(back_words) > 0.5:
        issues.append("answer_in_prompt")

    # trivial cloze: deleted span is a function word
    if card.type == "cloze" and is_closed_class(card.deleted_span):
        issues.append("trivial_cloze")

    # verbosity: long answers are usually compound in disguise
    if len(card.back.split()) > 20:
        issues.append("answer_too_long")

    return issues

Then two model-based checks worth the tokens:

Answerability without source. Give a fresh model only the card front — no document — and ask it to answer. If it can't produce the expected answer, the prompt is under-specified. If it produces the answer trivially for material the learner shouldn't already know, the prompt is leaking. Batch this per generation job; it's one call for many cards.

Deck-level distinctness. Embed every front and flag pairs above a cosine-similarity threshold. Near-duplicate fronts with different backs are the nastiest failure in spaced repetition: the learner can't tell which card they're looking at and starts guessing from queue position. Anything very close in a decent sentence embedding gets merged or rewritten.

The screening layer rejects a meaningful share of raw generations, and the rejection rate turns out to be the most useful signal we have about source material. A chapter with a high reject rate is usually narrative prose with few extractable claims — which means the fix is the chunking strategy, not the card prompt. That whole extract-screen-regenerate loop is what sits behind card generation in SmartRecall, and it's the part that took longest to get right; the initial "prompt an LLM for flashcards" version took an afternoon.

What I'd Skip Next Time

I spent too long on a learned quality classifier. The deterministic screens plus one answerability check catch nearly everything it would, and they're debuggable — when a card is rejected I can point at the rule that rejected it.

I'd also ship with cloze generation off by default. Q&A cards fail loudly: the learner can't answer. Bad clozes fail quietly: the learner answers from syntax and feels productive. Ship the loud failure mode first.