Big Howdy,
I'm so tired of hearing all the hype about AI.
Don't get me wrong, I think AI is pretty cool and I use it all the time for various tasks throughout the day, but I hate AI marketing.
What do I mean by that? Whenever there is any hype, advertising, or corporate shilling about AI, it's always vague. Talks about agents, multi-agent systems, RAG pipelines, etc. are everywhere, and yet the specifics are always just out of reach.
Even when I try to dig into deeper technical articles, the handwaving usually begins almost immediately. Several architecture diagrams and acronyms later, I still do not have a much better idea of what is actually happening (although, to be fair, I wasn't exactly looking that hard).
I’m going to fix that.
For myself, at least.
First Stop: RAG
My first target is RAG. It's been out for quite a while now and I've been hearing about it for forever. It also sounds like enough progress has been made that the problem has effectively been solved. Best practices have been established and it's pretty commonplace now, so it's a great place to start.
But what is RAG?
RAG stands for Retrieval-Augmented Generation, which is... not a helpful name if you are just learning about this for the first time like I am. However, if you use AI at all, you've already been using a form of it without even knowing it.
RAG is simply a pattern:
- Retrieve relevant external information.
- Add that information to the AI model’s working context.
- Generate an answer grounded in it.
That's it.
When you use an AI assistant and it goes and searches the internet for relevant information to give you an answer, that's a form of RAG. That doesn't mean your AI assistant is a RAG system, but it can perform RAG functions.
But this STILL sounds a bit hand-wavy to me. How does this work in real life?
The Part I Kept Missing
Context windows are massive these days, but I have seen technical documents that are literally more than 3,000 pages long and full of dense technical information. Even if one technically fits inside a model’s context window, sending the entire document with every question would be slow, wasteful, and probably not very good at finding the exact paragraph I need.
But an LLM would be perfect for working with documents like that. If I am reading a specification for something like High Bandwidth Memory, it would be fantastic to ask:
What does each bit in this register represent? If a bit maps to a different register, show me that too.
That would be infinitely more convenient than manually jumping around a massive PDF trying to find every related table, footnote, and register definition.
That's where the "retrieval" part of a RAG system comes in. Somehow, I need to structure and index the data, search it based on the user's question, and then give the relevant results to the LLM.
That's the part I'm going to figure out.
Enter Librarian
My project idea for learning this is to create an assistant that specifically answers questions about my personal ebook library. I call it Librarian (original, I know). Librarian will ingest my ebooks, many of which are several megabytes in size (although the size comes from images embedded in the file, not the text itself), structure the data, then search across it when I prompt it.
The goal is to support questions like:
- "What is this book about?"
- "What does this author have to say about this topic?"
- "Compare how these books discuss suffering."
- "Recommend a fantasy book with war themes."
- "Where does this book explain this specific idea?"
The important part is that Librarian should not simply produce a plausible answer. It should retrieve evidence from my books and cite the passages it used.
Of course, describing the finished product is the easy part. I still need to figure out what happens between dropping an ebook into a folder and getting a useful answer back.
How do I extract and structure the text? How do I make thousands of pages searchable? How does the system decide which passages are relevant? How do I preserve enough information to cite the book, chapter, and section an answer came from?
And most importantly, how do I know the answer is actually based on my books instead of something the model made up? The engineer in me can't stand relying on nondeterministic output without something I can measure, so I need a way to tell whether the system actually works and quantify how well it works.
These are the parts I want to understand.
I have already started building Librarian, and the next few posts will follow the project as I work through them. The next post will cover the high-level design: what components I think the system needs, what each one is responsible for, and how information will move from an EPUB file to an answer.
After that, I’ll start digging into the individual pieces.
You can follow the project here:
Top comments (0)