The LLM Didn’t Need to See the Diagram. It Just Needed a Seat Number

AI note apps love doing one very modern thing: taking a clean lecture PDF, feeding it to an LLM, and proudly returning notes with all the diagrams missing.

Amazing. We automated disappointment.

This project started with a simple goal: turn lecture PDFs and slides into useful study notes, without losing diagrams. Not “diagram summaries.” Not “imagine a flowchart here.” The actual source diagrams, in the right places.

PROJECT REPOSITORY: Open

Project Brief

I built Smart Notes Generator, a local-first app that takes PDFs and PowerPoint files, extracts text and diagrams, sends only the useful text structure to an LLM, and then rebuilds the final notes with the original diagrams inserted back.

The stack is simple:

• React + TypeScript for the frontend.
• FastAPI for the backend.
• PyMuPDF for PDF text/image/vector extraction.
• python-pptx for PowerPoint parsing.
• Pillow for image scoring.
• SQLite for local saved notes.
• LLM API or manual copy-paste mode for generation.

The Normal Approach: Throw Everything at the Model and Pray

The naive approach is:

The problem is that most document extraction pipelines treat images as either:

1. Noise.
2. Base64 blobs.
3. Something to send to a vision model.
4. Someone else’s problem.

Sending every diagram to a vision-capable model sounds fancy, but it creates new problems:

• It increases input size.
• It adds cost.
• It adds latency.
• It still may not place the diagram correctly.
• It can describe the diagram, but that is not the same as preserving it.

The key realization was:

The LLM does not need to understand every pixel of the diagram. It mostly needs to know where the diagram belongs.

That one sentence changed the architecture.

The Idea: Give Every Diagram a Seat Number

Instead of sending images to the model, I extract diagrams locally and replace them with stable placeholders.

Example:

A finite automaton can be represented using states and transitions.

{{IMG_001}}

The transition function defines how the automaton moves between states.

The LLM sees the placeholder as part of the source context. It can move it into the right location in the generated notes.

After generation, the backend replaces:

{{IMG_001}}

with the original local image.

So the model handles reasoning and structure. The system handles files, images, and reliability.

How It Works

The flow basically looks like:

A simplified version of the placeholder registry looks like this:

registry["IMG_001"] = {
    "placeholder": "{{IMG_001}}",
    "path": "/local/session/imgs/automata_p2_f0.png",
    "source_file": "lecture_automata.pdf",
    "page": 2,
    "score": 0.82,
    "context": "DFA transition diagram",
    "included": True,
}

Then the prompt contains text, not image bytes:

Use the following source material to create clear study notes.
Preserve image placeholders exactly where they belong.

Source:
A DFA consists of states, alphabet, transition function...

{{IMG_001}}

The accepting state is shown with a double circle.

The LLM returns something like:

## Deterministic Finite Automata

A **DFA** is a finite-state machine where each input symbol leads to exactly one next state.

{{IMG_001}}

The double-circled state represents an accepting state.

Then the postprocessor turns the token into a real figure:

def inject_images(markdown: str, registry: dict) -> str:
    for img_id, info in registry.items():
        figure_html = f"""
<figure>
  <img src="{info["path"]}" alt="{info["alt_text"]}" />
  <figcaption>{info["source_file"]}, page {info["page"] + 1}</figcaption>
</figure>
"""
        markdown = markdown.replace(info["placeholder"], figure_html)
    return markdown

But Not Every Image Deserves a VIP Pass

Lecture PDFs contain useful diagrams, yes. They also contain logos, headers, footers, slide backgrounds, decorative lines, and other visual confetti.

So I added image filtering.

Each extracted image gets a quality score based on:

• Pixel dimensions
• Aspect ratio
• File size
• Color entropy
• Non-blankness
• Duplicate hash

A simplified scoring idea:

score = (
    size_score    * 0.25 +
    aspect_score  * 0.20 +
    file_score    * 0.20 +
    entropy_score * 0.20 +
    blank_score   * 0.15
)

This removes tiny logos, blank slide backgrounds, repeated assets, and weird banner strips.

The user can still review the gallery and manually include or exclude images. Because yes, sometimes the “low quality” image is actually the one diagram the professor will ask for in the exam. Naturally.

Handling Failure: Because LLMs Have Vibes, Not Contracts

The LLM is told to preserve placeholders.

Does it always do that?

Of course not. It is an LLM, not a legally binding agreement.

Sometimes it drops {{IMG_003}}. So the system needs a fallback.

For every image, I store nearby text context and extract keywords:

"fallback_keywords": ["transition", "state", "dfa", "accepting", "alphabet"]

If a placeholder is missing after generation, the backend scans the generated notes and inserts the image near the paragraph with the strongest keyword match.

Simplified version:

def place_dropped_image(paragraphs, image):
    keywords = set(image["fallback_keywords"])

    best_index = 0
    best_score = 0

    for i, paragraph in enumerate(paragraphs):
        words = set(paragraph.lower().split())
        score = len(words & keywords)

        if score > best_score:
            best_score = score
            best_index = i

    paragraphs.insert(best_index + 1, image["placeholder"])
    return paragraphs

The LLM gets freedom to structure the notes. The system keeps guardrails around the parts that must not break.

Other Benefits I Got Almost for Free

1. Lower Cost

Images are not sent as model input. A placeholder like {{IMG_001}} is tiny. A base64 image is a suitcase full of nonsense as far as the prompt is concerned.

2. Better Privacy

Source files and extracted diagrams stay local. The model only sees text and placeholder IDs. For student notes, academic content, or internal training material, this matters.

3. Exact Diagrams

The final document uses the original image. No re-generated diagram. No “close enough.” No AI slop.

4. Easier Debugging

Placeholders make the pipeline inspectable.

If a diagram is missing, I can ask:

• Was it extracted?
• Was it filtered out?
• Was the placeholder assigned?
• Did the model drop it?
• Did postprocessing fail?

That beats staring at a final blob of generated Markdown wondering where everything went.

The Actual Design Decision

The big decision was separating responsibilities:

Document parser: extract facts and files
Image filter: decide what is useful
Prompt builder: create clean LLM input
LLM: rewrite and organize
Postprocessor: restore local diagrams
Evaluator/RAG: help review and reuse notes

The LLM is not the system. It is one component inside the system.

That distinction matters.

A lot of AI apps are just:

This project is closer to:

That is the difference between a demo and an actual product.

Takeaway

The best AI engineering trick in this project was not a giant prompt. It was knowing what not to send to the model.

The diagram did not need to be seen. It needed to be tracked.

The LLM did not need image pixels. It needed a placeholder, nearby context, and a clear instruction.

The system did the rest.

And honestly, that is the lesson I keep coming back to:

Good AI systems are not built by asking the model to do everything. They are built by giving the model the right job, then surrounding it with boring, reliable software.