Building My Own Knowledge Base: From PDF Ingestion to AI-Generated Memory Cards

As I read more technical books, research papers, and documentation, I realized that taking notes alone was not enough.

I could highlight important paragraphs, write summaries, and save interesting explanations, but eventually, all that information would end up scattered across different files and applications.

The bigger problem was recalling what I had read.

I wanted a system where I could store the information I learned, retrieve it later, and turn important concepts into something easier to review.

So, I started building my own knowledge database.

The idea is relatively simple:

1. Ingest documents and extract their content.
2. Split documents into manageable chunks while preserving their structural metadata.
3. Identify chunks that belong to the same logical content unit and combine them.
4. Use an LLM to extract valuable learning concepts.
5. Store those concepts as memory cards.
6. Generate educational images to make the information easier to recall.

This is still an ongoing project. In this post, I'll document what I have built so far, the implementation decisions I made, and some of the things I learned along the way.

1. The Tech Stack

I wanted to keep the architecture relatively simple and run as much as possible locally.

The tools I currently use are:

Component	Technology	Purpose
Backend	FastAPI	Hosting the knowledge database service
Local LLM inference	Ollama	Running local models for text processing
PDF processing	PyMuPDF	Reading and processing PDF documents
Document chunking	PyMuPDF4LLM	Extracting structured Markdown chunks
Database	PostgreSQL	Storing documents, chunks, and generated concepts
Vector storage	pgvector	Planned semantic retrieval
Image generation	Gemini (Nano Banana)	Generating educational memory-card images

My objective is to have a self-hosted knowledge system where I control the ingestion pipeline, data structure, models, and retrieval process.

2. Book Ingestion Pipeline

The first thing I needed was a reliable way to ingest books.

I started with a fairly straightforward pipeline:

1. Retrieve the book record from the database.
2. Read the PDF using PyMuPDF.
3. Split the document into chunks using pymupdf4llm.to_chunks().
4. Extract and store the metadata associated with each chunk.
5. Store everything in PostgreSQL.
6. Commit the transaction only after the ingestion process succeeds.

The important part here is the transaction.

I delete the existing chunks before re-ingesting a document. However, I don't want to end up with an empty or partially populated document if something fails halfway through the process.

Keeping the deletion and insertion operations in the same database transaction allows me to roll back the changes if ingestion fails.

Chunk data model

I started with a simple Pydantic model:

class Chunk(BaseModel):
    id: uuid.UUID = Field(default_factory=uuid.uuid4)
    book_id: uuid.UUID
    position: int
    raw_text: str
    meta_data: str | None
    embeddings: list[float] | None = None
    created_at: datetime = Field(default_factory=datetime.now)

The important fields are position, raw_text, and meta_data.

I preserve the original chunk position because the ordering of document content matters. I also store the section metadata so that the LLM has some understanding of where the content came from.

Without this metadata, a chunk might contain a paragraph explaining an algorithm but lose the fact that it belongs to a particular chapter or subsection.

That context becomes especially useful during the next stage.

Ingestion implementation

async def ingest_book(book_id: UUID, db: AsyncSession) -> dict:
    repo = BookChunkRepository(db)

    try:
        book = await repo.get_book(book_id)
        if book is None:
            raise ValueError(f"Book not found: {book_id}")

        pdf_path = BOOKS_DIR / book.source.lstrip("/")

        # Replace the existing chunks within this transaction.
        await repo.delete_chunks(book.id)

        chunks = pymupdf4llm.to_chunks(
            doc=pdf_path,
            show_progress=True,
            table_mode="isolate",
        )

        for i, chunk in enumerate(chunks):
            chunk_py = Chunk(
                book_id=book.id,
                position=i,
                raw_text=chunk.text,
                meta_data=" ".join(chunk.metadata.section_path),
            )
            await repo.create_chunk(chunk_py)

        await repo.commit()

        return {
            "book_id": book_id,
            "chunk_count": len(chunks),
        }

    except Exception:
        await repo.rollback()
        logger.exception("Failed to ingest book %s", book_id)
        raise

At this stage, I have a basic document ingestion pipeline that extracts text, retains section information, and stores the chunks in PostgreSQL.

However, I noticed another problem.

3. Combining Semantically Continuous Chunks

Document chunking is not always perfect.

A paragraph might be split across two chunks. An explanation might continue into the next chunk. A list, table, or technical discussion could also be separated even though the content belongs together.

This becomes a problem when generating learning concepts.

For example, imagine a chunk explaining the first three steps of an algorithm and the next chunk explaining the remaining steps. Sending them to an LLM independently could result in incomplete explanations.

I wanted to improve this without relying entirely on fixed chunk sizes or arbitrary overlap.

My approach was to use a small classification model to determine whether two consecutive chunks belong to the same logical content unit.

I chose tev1:4b for this task.

The model receives two pieces of information:

• The previous chunk's metadata and ending text.
• The next chunk's metadata and beginning text.

It then determines whether the next chunk continues the previous one.

Defining the classification task

I use a structured question rather than asking the model to generate an explanation.

TEV_MERGE_CHUNK = {
    "type": "noul",
    "instructions": (
        "Determine whether the next document chunk continues "
        "the same logical content unit as the previous chunk."
    ),
    "criteria": {
        "true": (
            "Both chunks belong to the same logical content unit. "
            "The next chunk continues the previous paragraph, "
            "explanation, list, table, or section."
        ),
        "false": (
            "The next chunk introduces an independent content unit, "
            "a new heading, a different section, or a separate record."
        ),
    },
}

Combining the chunks

I use a sliding comparison approach.

Instead of comparing every chunk with every other chunk, I compare the current accumulated content with the next chunk.

If the model determines that they belong together, I append the next chunk. Otherwise, I finalize the current group and start a new one.

async def combine_semantic_chunks(chunks: list[Chunk]):
    if not chunks:
        return []

    combined = []

    current = chunks[0]
    current_text = current.raw_text

    for next_chunk in chunks[1:]:
        response = await client.systemone(
            model="tev1:4b",
            state={
                "previous_chunk": (
                    f"Meta Data {current.meta_data}:\n"
                    f"{current_text[-800:]}"
                ),
                "next_chunk": (
                    f"Meta Data {next_chunk.meta_data}:\n"
                    f"{next_chunk.raw_text[:800]}"
                ),
            },
            questions={"should_merge": TEV_MERGE_CHUNK},
        )

        score = response.answers["should_merge"].noul

        if score >= 0.5:
            current_text += "\n" + next_chunk.raw_text
        else:
            current.raw_text = current_text
            combined.append(current)

            current = next_chunk
            current_text = next_chunk.raw_text

    # Don't forget the final accumulated chunk.
    current.raw_text = current_text
    combined.append(current)

    return combined

The metadata is intentionally included in the model context.

Two chunks might contain similar technical terminology but belong to completely different sections. Conversely, two chunks might look unrelated when viewed only through their text, but their section paths indicate that they are part of the same discussion.

Metadata gives the classifier another signal to make a better decision.

I also limit the text sent to the model to the last 800 characters of the current content and the first 800 characters of the next chunk. This keeps the classification task relatively small.

The threshold is currently 0.5.

This is a simple first implementation. I haven't done a systematic evaluation of the classification accuracy yet, but it gives me a mechanism for combining document chunks based on their logical continuity rather than only their physical position.

4. Standardizing the Memory Card Database

Once the document content is ready, the next step is extracting useful learning concepts.

Before doing that, I needed a database structure that separates the generated content from its visual presentation.

I came up with three main tables:

• chunks: Stores the extracted document content.
• card_decks: Stores the shared configuration for a collection of cards.
• cards: Stores individual learning concepts.

The relationship is straightforward:

A card deck holds the shared style and domain configuration, while each card holds its own title, content, and image prompt.

This allows multiple cards to share the same visual configuration without repeating those settings in every card.

Database schema

CREATE TABLE chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    book_id UUID NOT NULL,
    position INTEGER NOT NULL,
    raw_text TEXT NOT NULL,
    meta_data TEXT,
    embeddings VECTOR(1024),
    created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE card_decks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    source_id UUID NOT NULL,
    title TEXT,
    system_prompt TEXT NOT NULL,
    style TEXT NOT NULL,
    domain TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE cards (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    deck_id UUID NOT NULL REFERENCES card_decks(id) ON DELETE CASCADE,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    image_prompt TEXT NOT NULL,
    image_path TEXT,
    created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    generate BOOLEAN DEFAULT TRUE
);

The source_id on card_decks currently identifies the source book.

I also added style and domain to the deck because I want the visual configuration to be shared across all cards generated from that source.

For example, I can generate a deck using a student-notebook style for general technical concepts, or use a different visual configuration for mathematics and systems topics.

This separation also makes it easier to regenerate images without having to regenerate the underlying learning content.

5. Generating Learning Concepts with a Local LLM

With the document ingestion and chunk processing stages in place, I could finally work on the actual learning content.

The objective isn't to summarize every chunk.

I want the model to identify genuinely valuable concepts that are worth remembering.

Depending on the source material, these could include:

• Core principles and definitions.
• Mechanisms and how things work.
• Equations and derivations.
• Algorithms and implementation details.
• Trade-offs, limitations, and failure modes.

I use a Pydantic model to define the expected output.

class MemoryConcept(BaseModel):
    id: UUID = Field(default_factory=uuid4)
    title: str
    content: str
    image_prompt: str
    generate: bool | None = True
    image_path: str | None = None

A few things I learned

There are three implementation details I found particularly useful.

1. Using Pydantic's JSON schema for structured generation

Instead of asking the LLM to follow an informal JSON format, I pass:

format=MemoryConcept.model_json_schema()

This provides the model with a formal schema describing the expected output.

It makes the output more consistent and reduces the amount of custom parsing logic I need to write.

2. Disabling thinking for a relatively simple extraction task

I'm currently using qwen3.5:4b for concept generation.

One thing I noticed is that enabling the thinking process for this task can introduce unnecessary inference overhead.

The task is relatively constrained: extract a concept, explain it, and describe an appropriate educational visual.

I found that setting:

think=False

reduces inference overhead for this particular workload.

This is not a universal recommendation to disable thinking. For tasks requiring complex reasoning, it may be necessary. However, for straightforward structured extraction, I prefer to avoid spending additional inference resources when they don't provide meaningful improvements.

3. Handling validation failures

Even with a structured output schema, I don't assume that every model response will be valid.

I use Pydantic to validate the response and catch ValidationError so that one malformed response doesn't necessarily stop the entire card-generation process.

This is especially important when processing an entire book, where a single problematic chunk should not prevent the remaining chunks from being processed.

Memory card generation pipeline

The endpoint accepts a book ID and the configuration for the generated deck.

@router.post("/memory-card")
async def generate_memory_card(
    book_id: UUID,
    combine_chunks: bool,
    style: CardStyle = CardStyle.STUDENT_NOTEBOOK,
    domain: PromptDomain = PromptDomain.GENERAL,
    db: AsyncSession = Depends(get_db),
) -> MemoryCardResponse:
    return await prompt_memory_card(
        book_id,
        style,
        domain,
        combine_chunks,
        db,
    )

The implementation retrieves the book's chunks, optionally combines semantically continuous chunks, creates a deck, and generates individual concepts.

async def prompt_memory_card(
    book_id: UUID,
    style: CardStyle,
    domain: PromptDomain,
    combine_chunk: bool,
    db: AsyncSession,
) -> MemoryCardResponse:
    bookchunk_repo = BookChunkRepository(db)
    memory_card_repo = MemoryCardRepository(db)

    book = await bookchunk_repo.get_book(book_id)
    if book is None:
        raise ValueError(f"Book id: {book_id} not found")

    chunks: list[Chunk] = await bookchunk_repo.get_chunks(book_id)

    if combine_chunk:
        chunks = await combine_semantic_chunks(chunks)

    result = MemoryCardResponse(title=book.title)

    deck = CardDeck(
        source_id=book.id,
        title=book.title,
        style=style,
        domain=domain,
    )

    await memory_card_repo.create_deck(deck)
    await memory_card_repo.commit()

    for chunk in chunks:
        content = f"Meta data: {chunk.meta_data}\n{chunk.raw_text}"

        try:
            response = await client.chat(
                model="qwen3.5:4b",
                messages=[
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": content},
                ],
                think=False,
                options={"temperature": 0.5},
                format=MemoryConcept.model_json_schema(),
            )

            data = response.message.content or ""
            data = MemoryConcept.model_validate_json(data)

            data = MemoryConcept(
                image_prompt=data.image_prompt,
                title=data.title,
                content=data.content,
            )

            if not data.title and not data.content and not data.image_prompt:
                result.empty += 1
                continue

            await memory_card_repo.create_cards(deck.id, [data])
            await memory_card_repo.commit()

        except ValidationError as exc:
            print(f"Validation failed for chunk {chunk.id}: {exc}")
            result.failure += 1
            continue

        result.concepts.append(data)
        result.success += 1

    return result

I commit the deck before processing individual chunks, then commit cards as they are generated. This means I can retain successfully generated concepts even if a later chunk produces an invalid response.

There is still room to improve the transaction and retry strategy, but this is sufficient for my current implementation.

6. The System Prompt

The system prompt is probably one of the most important components of this pipeline.

I don't want the LLM to simply produce a short summary of the supplied text. I want it to act as a technical educator and identify a concept that is worth learning.

I also want the image prompt to describe the actual educational content, not just request a visually attractive image.

Here is the prompt I'm currently using:

SYSTEM_PROMPT = """
You are an expert technical educator. Extract valuable learning concepts from the supplied book chunk for visual memory card generation.

Focus on:
- Core principles and definitions
- Mechanisms and how things work
- Equations and derivations
- Algorithms and implementation details
- Trade-offs, limitations, and failure modes

Rules:
- Each concept must teach one coherent, self-contained learning objective.
- Preserve technical accuracy, equations, assumptions, and important details.
- Do not invent information or examples unsupported by the source.
- Prefer deep understanding over superficial summaries.
- Make the content concise but sufficiently detailed for learning.
- Describe specific diagrams, equations, flows, or examples that visually explain the concept.
- Extract only genuinely valuable learning concepts.

If the chunk contains no important or useful learning concepts, return the same JSON schema with all three fields set to empty strings: "", "", "".

Return ONLY valid JSON using this schema:
{
  "title": "Concept title",
  "content": "A self-contained explanation of the concept, including essential details, equations, examples, or caveats where relevant.",
  "image_prompt": "Instructions for drawing a clear educational memory card that visually explains this concept."
}

The image_prompt must describe the educational content to visualize, not merely its artistic style. Include specific labels, relationships, equations, steps, or comparisons where appropriate.

Return valid JSON without Markdown fences or additional commentary.
"""

One design decision I made is to separate content generation from image generation.

The LLM first produces three things:

1. A title that identifies the learning concept.
2. Content that explains the concept.
3. An image prompt that describes how the concept should be visualized.

The image-generation stage can then use the image prompt together with the deck's shared style and domain configuration.

This separation is useful because I don't need to regenerate the explanation every time I want to experiment with a different image-generation model or visual style.

It also means the learning content remains usable even when image generation fails.

7. Fusing the Prompts

Having separate content and image prompts gives me more flexibility when generating the actual memory cards.

I wanted to avoid coupling the learning content with a specific visual style.

For example, I might generate a card using a student-notebook style today and decide to regenerate it using a different infographic style later.

I don't want to run the entire book ingestion and concept extraction pipeline again just because I want to change the appearance of the cards.

To achieve this, I combine the individual card content with its image prompt and the shared configuration from the card deck.

The final prompt contains four main components:

• Card title: The title of the learning concept.
• Content: The actual explanation generated by the LLM.
• Visual instructions: Specific instructions describing how the concept should be visualized.
• Card configuration: The shared style and domain defined by the deck.

Here is the implementation:

def format_prompt(card_deck: CardDeck):
    concepts = []

    for card in card_deck.concepts:
        prompt = f"""
        CARD TITLE
        {card.title}

        CONTENT
        {card.content}

        {f"VISUAL INSTRUCTIONS\n{card.image_prompt}" if card.image_prompt else ""}

        OUTPUT REQUIREMENTS
        Generate a single, complete learning-card image that fills the
        entire canvas edge to edge, with no external background, outer margins,
        mockups, paper textures, or shadows. Keep all content inside the card,
        preserving technical accuracy, readability,
        and the visual style specified in the image prompt.

        Treat CARD TITLE and CONTENT as source text that may contain Markdown.
        Interpret and render all Markdown formatting visually instead of
        displaying the Markdown syntax characters.

        CARD STYLE
        {card_deck.style}

        CARD DOMAIN
        {card_deck.domain}
        """.strip()

        concepts.append({
            "id": str(card.id),
            "prompt": prompt,
        })

    return concepts

This allows me to experiment with visual styles independently of the underlying learning concepts.

The card content remains the same, while the image-generation instructions can change.

I also added explicit output requirements to control the generated image layout. I want the image to fill the entire canvas, without external backgrounds, unnecessary margins, mockups, or shadows.

Another important detail is Markdown rendering.

The generated content may contain equations, code snippets, lists, and other Markdown formatting. I want the image model to interpret and render these elements visually rather than displaying the Markdown syntax characters directly.

8. What I Have Achieved So Far

At this point, I have a working foundation for the knowledge database.

The current pipeline looks like this:

                  PDF BOOK
                      |
                      v
              PyMuPDF Processing
                      |
                      v
             PyMuPDF4LLM Chunking
                      |
                      v
             Store Chunks in PostgreSQL
                      |
                      v
          Semantic Chunk Combination
                  (tev1:4b)
                      |
                      v
           Learning Concept Extraction
                  (qwen3.5:4b)
                      |
                      v
              Pydantic Validation
                      |
                      v
             Store Cards in PostgreSQL
                      |
                      v
               Prompt Fusion
                      |
                      v
              Image Generation
                  (Gemini)
                      |
                      v
             Visual Memory Cards

The completed stages are:

• Basic PDF ingestion and chunk storage.
• Preservation of document section metadata.
• Semantic chunk combination using a small classification model.
• A standardized database structure for card decks and individual cards.
• Structured learning-concept extraction using a local LLM.
• Image-prompt generation as part of the concept extraction process.
• Prompt fusion to separate learning content from visual configuration.

I now have a basic pipeline that takes a PDF book and turns its content into structured learning concepts and, eventually, visual memory cards.

9. Current Limitations: Image Quality and Content Density

Although the pipeline works, the generated images are not always what I expected.

Here is an example of a generated card from a Vue full-stack development book:

One problem I noticed is that the generated cards tend to contain too much text without enough visualization.

Instead of using diagrams, relationships, and visual representations to explain the concepts, the image model often tries to fit the entire explanation into the image.

I believe this is primarily a prompting issue.

Although I already provide visual instructions during concept extraction, I still need to improve how I communicate the balance between textual explanations and visual representations.

For example, I want the image model to prioritize:

• Diagrams over long paragraphs.
• Relationships and data flows over textual descriptions.
• Small, meaningful labels over lengthy explanations.
• Code snippets or equations only when they are important to understanding the concept.

Another missing component is image quality validation.

Currently, there is no second validation stage to evaluate the generated cards.

A successfully generated image does not necessarily mean it is a good learning card. It might contain incorrect labels, malformed equations, unreadable text, or an inaccurate visual representation of the source concept.

I need an additional quality-checking stage to inspect the generated image before considering the card complete.

10. What's Next?

There are several things I want to work on next.

1. Vector Embeddings for Document Chunks

Currently, the chunks are stored as plain text.

I want to generate embeddings and store them using pgvector. This will allow me to retrieve relevant information based on semantic similarity instead of relying entirely on exact text matching.

Eventually, I want to ask questions across multiple books and retrieve the relevant sections as context.

2. Multi-Level Chunking and Hierarchical Retrieval

Flat chunk storage is useful for basic retrieval, but it doesn't fully represent the structure of a book.

I want to experiment with hierarchical chunking:

• Document level: The complete book or source.
• Chapter level: Major topics and sections.
• Section level: Individual discussions.
• Chunk level: The actual text used for retrieval.

This should give me more flexibility when retrieving information at different levels of granularity.

For example, a question about a specific implementation detail might only require a small chunk, while a question about the overall design of a system might benefit from retrieving an entire section or chapter.

3. Supporting Multiple Data Sources

At the moment, the pipeline primarily targets PDF books.

Eventually, I want the knowledge database to support multiple sources:

• Research papers.
• Technical documentation.
• Markdown files.
• Web articles.
• YouTube educational videos.
• Personal notes.
• Other structured documents.

The goal is to avoid building a knowledge database that only understands books.

4. Image Quality Validation

I also want to introduce a second validation stage for generated images.

The objective is to evaluate whether the generated image accurately represents the learning concept, whether the text is readable, and whether the visual explanation is sufficiently informative.

This could involve using a VLM to inspect the generated image against the original card content and image prompt.

I don't want to treat image generation as successful merely because an image file was produced.

---

Final Thoughts

I started this project because I wanted a better way to retain what I read.

Instead of continuously collecting notes and hoping I could find them later, I wanted to build a system that could process information, extract useful concepts, and make those concepts easier to revisit.

The interesting part for me is not just connecting an LLM to a PDF reader. It is building the individual components of the pipeline and understanding how they interact: document parsing, chunking, metadata preservation, classification, structured generation, database transactions, prompt engineering, and eventually semantic retrieval.

There is still a lot to improve, especially around retrieval quality, hierarchical document representation, and the quality of generated visual explanations.

For now, I have a working foundation. I'll continue extending it as I experiment with different models, retrieval strategies, and document-processing techniques.