Most RAG demos begin with PDFs in English.. The SVSU Knowledge Assistant started with a messy stack of paper circulars in Hindi and English. These were photocopied scanned at an angle full of ruled tables and needed by university staff who had to find answers. This is what I learned while building it in the order I learned it.
OCR quality decides everything
I spent the first few weeks focused on retrieval and models. That was the place to start. If the text coming out of OCR is broken or wrong nothing that follows can be good. Three things changed everything:
Detect the script per page before running Tesseract and use the right language model. Some pages mix Hindi and English. We run both. Take the one with higher confidence.
Rebuild tables cell by cell. Circulars are mostly tables. Fee schedules, academic calendars, code lists. Raw OCR output turns a table into a jumble. By detecting the ruled lines and reading each cell "Course code 302" stays next to "Fee ₹1,200" of being scattered across three rows.
Flag low-confidence words. Keep them visible to reviewers. Don’t guess silently. Let people see what’s uncertain.
The pipeline runs in the background. A staff member uploads a scan. Moves on with their day.
Store the location of every passage, not just the text
Each paragraph. Every table row gets embedded and stored in PostgreSQL with pgvector. We also store the page number and the bounding box from which it came. That might seem like work until you see the benefit: an answer can cite "Circular 42, page 3 row 7". Clicking that citation opens the scan, scrolled and highlighted to that exact spot. For people who need to act on the answer the citation is the product.
Vectors alone miss the things that matter most
Semantic search works well for questions like "What's the policy on late fees?". It fails completely on things like "What does code EX-2024-17 say?" Circulars are full of codes, dates and exact amounts. Those are what people actually search for. The fix was retrieval:
Rewrite the question into a query and a Hindi query. Users speak in Hinglish and documents can be in either language.
Run vector search to find similar passages.
Run matching for codes, dates and rare tokens.
Merge the results and re-rank them using a cross-encoder model.
The re-ranker was the biggest improvement in the whole project.
Answer from the passages then check the answer
The language model gets the retrieved passages and is asked to answer only from them with numbered citations. Then a second faster model checks the answer against the sources. It flags any claim that isn’t supported. These errors are rare. They are the ones that would have broken trust.
Answers stream in as they are written. That makes the whole thing feel fast even when retrieval takes a second.
The boring parts are the product
A draft-to-active approval workflow so a document can’t be used until it’s reviewed.
Roles and per-document visibility. Some circulars are for staff.
Bulk. Automatic expiry. Circulars with end dates are automatically retired.
Two-step. Lockouts. This is university data. Security matters.
An automatic answer-quality test. A set of questions with known answers runs in CI on every push. If retrieval regresses the deploy fails.
CI/CD runs from GitHub Actions to an AWS server using Docker and Caddy. Nothing fancy. That’s the point.
One API, two clients
The web app is built with Next.js. The Android app is Flutter. They share the API and the same accounts. The mobile app can take photos of paper letters. Show in-app update notices. Widget tests open every screen on a phone- viewport in CI. Building the app second on an API that was already solid took a fraction of the time the web app did.
What I would do differently
Start with the OCR evaluation set, not the chat UI.
Keep the chunking tied to document structure. Rows, sections. From day one. Not fixed windows.
Budget time for the approval workflow early. It shaped the data model more than the AI did.
If you have a pile of scanned documents and need something your team can trust this is the kind of work I do. The AI assistants service page has the version. The project page has screenshots.
Originally published at abhishekdev.online. I am a web developer, in Meerut, India building websites Laravel apps and AI tools.
Top comments (0)