Paper Q&A tools aren't remembering the literature. They're doing RAG.

A lot of paper-reading and literature Q&A tools now feel like chatting with an assistant that has already read the field. You ask a question, it answers in complete sentences, and it sometimes even points to sources. It's easy to walk away thinking the model has the whole library in its head.

Usually, that isn't what's happening. The important piece is a pattern called retrieval-augmented generation, or RAG, introduced by Lewis and colleagues in 2020. The core idea is simple: first find the passages that might matter, then let the model write an answer from those passages.

Think of it as an open-book exam, not a closed-book recitation. In the closed-book case, the model can only continue text from patterns it saw in training. Once the details get specific, or the paper is newer than the training cutoff, it starts inventing — including citations that look official and aren't. In the open-book case, the system pulls a few pages from the stacks, then lets the model write while looking at those pages. The model is still generating sentences. The raw material, though, comes from retrieval rather than from memory alone.

Literature tools need this especially badly. The corpus is huge, new papers appear every day, and no training cutoff can keep up. Stuffing every full text into one context window isn't realistic either.

A typical pipeline

A common literature RAG path looks like this:

User question
    ↓
Turn the question into a search request (keywords, vectors, or both)
    ↓
Retrieve a handful of passages from the paper collection
    ↓
Optionally rerank them and drop the obvious misses
    ↓
Send the passages and the question to the model
    ↓
Generate an answer with sources

In practice, papers are first split into chunks. A full article is too long to embed as a single unit, and too long to drop into the prompt every time. Teams usually cut by paragraph or by a fixed length, with a little overlap so a sentence isn't chopped in half. Scientific papers also have a structure of their own: introduction, methods, results, and discussion live in different sections. If you slice only by character count, a methods setup and the numbers that belong with it can end up in different chunks. Retrieval then finds half the meaning and misses the rest.

After chunking, an embedding model turns each passage into a vector and stores it in a vector database. You can also keep a keyword index, such as BM25. The two methods catch different things. Keyword search is good at proper names, gene symbols, and method acronyms. Vector search is better when the same idea is phrased differently. Someone may ask "how to stabilize fine-tuning," while the paper says "training with a smaller learning rate." The wording doesn't match; the meaning might. Many production systems blend both, then optionally rerank with a heavier model. "Semantically similar" is not the same thing as "this is the evidence you need."

When a user asks a question, that question is also embedded, and the system fetches the nearest chunks. Those chunks are then packed into a prompt along the lines of: answer from the passages below, cite your sources, and don't invent anything that isn't there.

The snippet below is the retrieve-then-generate path — not "ask the model to answer from memory":

# Pseudocode: the retrieve-then-generate path in a literature Q&A tool
q_vec = embed(question)
hits = index.search(q_vec, k=5)          # fetch the nearest passages first
context = "\n\n".join(
    f"{h.text}\n(Source: {h.paper_id})" for h in hits
)
prompt = (
    "Answer the question using only the passages below. "
    "If the passages do not contain enough evidence, say you don't know.\n\n"
    f"{context}\n\nQuestion: {question}"
)
answer = llm.generate(prompt)

What RAG does not guarantee

Once you see this path, the limits are easier to spot. RAG reduces the chance that the model will fabricate from memory. It does not guarantee a correct answer. If the chunks are bad, retrieval may return only a fragment of the idea. If the system pulls a loosely related commentary passage and misses the actual experimental details, the model can still write a fluent answer on the wrong material. Even when the passages are right, the model can skip a sentence, mix two papers together, or misstate a number. Figures, equations, and supplementary files are often harder to index than body text. If the index is stale, the tool cannot "know" work that was published after the last update. And only a limited number of passages fit in the prompt; evidence that ranks just outside that window might as well not exist.

So when you evaluate a tool like this, don't start with "has it read every paper?" Ask more engineering questions: did it retrieve the passages you would actually want to read? Can each claim in the answer be traced to a source? When you open that source, does it really say that? If you're building something similar, split the evaluation too. Whether retrieval found the right passages, and whether the final answer sounds polished, are two different questions. If you only score fluency, a failed retrieval can look like a successful generation.

Ask those three questions and you're checking the system the way you'd check an open-book exam — not treating it as a literature engine that can't be wrong.

More research explainers: https://www.crushsci.com