Turn a folder of PDFs into a question-answering API in an afternoon

Someone hands you 200 PDFs and asks for a "chatbot that knows this stuff." You could read all 200. Or you could build a small API that reads them for you and answers questions with citations. This post is the second option, start to finish, with the parts that usually break called out.

I'll show the mechanics first so you understand what's happening, then a shorter path if you don't want to run the plumbing yourself.

What you're actually building

A question-answering API over your own documents is four moving parts:

1. Ingest. Read each PDF, pull out the text.
2. Chunk. Split that text into pieces small enough to search.
3. Embed and store. Turn each chunk into a vector, keep it in a store you can query.
4. Answer. At question time, find the closest chunks and hand them to a model with the question.

People call the whole thing RAG (retrieval-augmented generation). The name makes it sound heavier than it is. It's search, then a prompt.

Step 1: get the text out

PDFs are a mess. Some are clean text, some are scanned images, some have two columns that extract in the wrong order. Start simple with pypdf:

from pypdf import PdfReader

def read_pdf(path: str) -> str:
    reader = PdfReader(path)
    return "\n".join(page.extract_text() or "" for page in reader.pages)

Where it breaks: scanned PDFs return empty strings because there's no text layer, only pixels. If extract_text() gives you nothing, those files need OCR (Tesseract, or a hosted vision model). Check for empty output early so you don't silently index blank pages.

Step 2: chunk the text

You can't embed a whole document as one vector and expect good answers. Retrieval works on pieces. The size of those pieces matters more than most tutorials admit.

Too small (a sentence) and each chunk loses its context. Too big (a whole page) and the model gets a wall of mostly-irrelevant text and the answer drifts. A reasonable starting point is 500 to 800 tokens per chunk with a small overlap so a sentence split across a boundary isn't lost.

def chunk(text: str, size: int = 600, overlap: int = 80) -> list[str]:
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

This splits on whitespace, which is crude but fine to start. If your documents have clear structure (headings, sections), splitting on those boundaries beats a fixed window because a chunk that matches a section is a chunk that makes sense on its own.

Step 3: embed and store

An embedding turns a chunk into a list of numbers that captures its meaning. Similar text lands near similar text. You query by embedding the question the same way and finding the nearest chunks.

For a folder of 200 PDFs you do not need a managed vector database. A local store is plenty:

import chromadb
from chromadb.utils import embedding_functions

client = chromadb.PersistentClient(path="./store")
ef = embedding_functions.DefaultEmbeddingFunction()
col = client.get_or_create_collection("docs", embedding_function=ef)

for doc_id, text in enumerate(all_texts):
    chunks = chunk(text)
    col.add(
        ids=[f"{doc_id}-{i}" for i in range(len(chunks))],
        documents=chunks,
        metadatas=[{"source": sources[doc_id]} for _ in chunks],
    )

Keep the source filename in the metadata. That's what lets you cite where an answer came from, which is the difference between a demo and something people trust.

Step 4: answer with citations

At question time: embed the question, pull the top handful of chunks, put them in the prompt, ask the model to answer only from what it was given.

def answer(question: str) -> str:
    hits = col.query(query_texts=[question], n_results=5)
    context = "\n\n".join(hits["documents"][0])
    sources = {m["source"] for m in hits["metadatas"][0]}

    prompt = (
        "Answer the question using only the context below. "
        "If the context does not contain the answer, say so.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    reply = call_your_model(prompt)  # Claude, GPT, a local model, your choice
    return f"{reply}\n\nSources: {', '.join(sources)}"

Two things that matter here:

• n_results. Five is a good default. More context is not better. Past a point you're paying for tokens and diluting the signal.
• The "say so" instruction. Without it the model fills gaps with plausible fiction. With it you get "the documents don't cover this," which is the honest answer and the one that keeps people trusting the system.

Wrap answer() in any web framework and you have an API:

from fastapi import FastAPI

app = FastAPI()

@app.post("/ask")
def ask(question: str):
    return {"answer": answer(question)}

Where this gets hard in production

The afternoon version above works. The things that turn it into a real project:

• Re-indexing. Documents change. You need a way to update or drop chunks without rebuilding everything.
• Bad retrieval. When answers are wrong it's usually retrieval, not the model. The right chunk never made it into the context. Logging what got retrieved for each question is the first debugging tool you'll want.
• Access control. If different people should see different documents, retrieval has to filter by permission before it ranks. Bolting this on later is painful.
• Cost. Embeddings are cheap. The generation call is not, and it scales with traffic. Caching common questions helps.

The shorter path

If you'd rather not run the ingest, chunking, vector store, and prompt loop yourself, this is the exact shape of thing LLMGraph builds. You point it at your documents, it handles the chunking and retrieval, and you get a REST API and an embeddable chat widget out the other side. The four steps above still happen. You just don't maintain them.

Either way, the mental model is the same: read, chunk, embed, retrieve, answer. Once that clicks, "chatbot that knows our stuff" stops sounding like a research project and starts sounding like an afternoon.