The public UAJY Academic Document RAG Chatbot repository packages a Streamlit assistant for the 2025/2026 academic handbook of Universitas Atma Jaya Yogyakarta’s Faculty of Industrial Technology. For developers, it provides an inspectable implementation of document ingestion, local vector search, cited answer generation, refusal controls and a 20-question evaluation harness in one codebase.
From handbook PDF to cited answer
The offline ingestion pipeline extracts text and tables with pdfplumber, divides the material into semantic or paragraph-based chunks, creates 3,072-dimensional embeddings with gemini-embedding-001, and persists the vectors in a FAISS index alongside page and heading metadata. The repository says the resulting index contains 350 chunks drawn from 112 handbook pages.
At runtime, a Streamlit query triggers top-K similarity retrieval from FAISS. The application builds a prompt from the retrieved context and conversation history, sends it to Gemini 2.5 Flash, and formats the answer with page numbers and chapter or section titles, according to the documented architecture.
The project describes two controls for unsupported questions: a similarity threshold filters retrieval results, while the system prompt instructs Gemini to refuse requests that the retrieved document context cannot answer. The README characterizes these controls as an anti-hallucination defense and says generated answers must rely exclusively on retrieved PDF chunks.
What the published evaluation covers
The repository includes an evaluation runner and a test file with 20 questions. Its benchmark summary reports Retrieval Recall@4 of 100% for 15 in-scope questions, refusal correctness of 100% for five out-of-scope questions, average retrieval latency of 0.42 seconds and total response latency of approximately 1.85 seconds. These figures describe the project’s published test suite.
Analysis: an inspectable design with bounded evidence
Analysis: The repository is best read as a concrete RAG implementation whose grounding mechanisms can be inspected, rather than conclusive evidence for its broader “production-grade” and “hallucination-free” labels. The published summary establishes reported behavior for 20 prompts; it does not, by itself, establish behavior beyond that suite. The project’s own disclaimer draws a similar boundary by describing the chatbot as an educational and information-search assistant while reserving official authority for the deanery and academic administration office. Project README
The unresolved deployment trade-off is the boundary between local and hosted components. The FAISS index and CPU-based similarity search run locally, while embedding creation and answer generation use Google Gemini services through an API; setup therefore requires a Gemini API key. Repository architecture and setup Analysis: This gives adopters a local retrieval layer, but not a fully local RAG runtime. The relevant engineering decision is whether that split fits the intended deployment environment—not whether local vector search alone is sufficient.
Top comments (0)