In my recent blog post, I was pretty happy about setting up a simple RAG system. Well, it turns out it wasn't that big of a deal. After reading more on the topic, I realized my approach was just scratching the surface. That is why I added a quick P.S. to the original post with a reality check.
Reflecting on my "RAG in a box", I realized the following issues:
- My AI factory pattern is essentially a hardcoded if/else block, lacking dynamic registration.
- My Docker configuration is perfectly functional but very basic; there is nothing highly app-specific about it.
- My approach to chunking documents relied purely on LlamaIndex's built-in functionality, which I now know is a black box that can lead to retrieval failures and hallucinations when data gets messy.
- My response payload is blind: The API currently just spits out a text string. Because I am not returning any source nodes, citations, or similarity scores, I have no way to prove if the system actually retrieved the answer from my PDF, or if the LLM is just confidently hallucinating.
- In general, I orchestrated the tools well, but I really only hit the tip of the iceberg.
- My Python skills could also use some polishing.
What is next? I think I will stay with this project for now so I can do it right. I am not quite there yet on a theoretical level, and I need to play around with the different parts of the problem scope. However, for now, my priority is getting ready for my AI-900 exam this coming Thursday. I am using NotebookLM heavily to study properly. I added some videos, documentation, and the MS Learn path into the system, and I am getting a lot of fantastic learning tools out of it.
Well... that’s my update. Not much, but something honest to share.
Cheers
Pat
Top comments (0)