Building an Enterprise RAG Knowledge Assistant: Lessons from Production

Kiran Krishna P — Thu, 23 Jul 2026 17:56:24 +0000

When I joined Strokx Technologies as an AI Engineer, my first real project was one that a lot of teams are tackling right now: how do you let employees ask natural-language questions against a large, messy pile of internal documents — and get answers that are actually correct, not just plausible-sounding?

That's the core challenge of Retrieval-Augmented Generation (RAG), and it's what I spent four months building: an enterprise-grade knowledge assistant that ingests internal documentation and answers questions against it, with specific engineering choices aimed at reducing hallucination — the single biggest reason RAG systems fail to earn user trust in production.

Here's how it was built, what broke along the way, and what I'd do differently next time.

Why naive RAG isn't enough

The basic RAG recipe is well known at this point: embed your documents, store the vectors, retrieve the top-k most similar chunks for a given query, stuff them into a prompt, and let the LLM generate an answer.

It works — right up until it doesn't. In practice, naive RAG breaks in a few predictable ways:

Bad chunking splits a table or a procedure mid-thought, so retrieval returns a fragment that's technically relevant but semantically incomplete.
Over-retrieval dumps too much loosely-related context into the prompt, and the model starts blending facts from different documents.
Silent failure — when nothing in the knowledge base actually answers the question, a naive system will still confidently generate something, because nothing in the pipeline is checking whether the retrieved context actually supports the answer.

That last one was the problem I spent the most time on.

The ingestion pipeline

The first real design decision was chunking strategy. Fixed-length chunking (e.g., every 500 tokens) is simple but ignores document structure — it'll happily cut a numbered procedure in half. Instead, I built a chunking step that respects semantic boundaries: splitting primarily on headings and paragraph structure, with a [~400–600 token] target size and overlap between adjacent chunks so that context isn't lost at the boundary.

Each chunk was embedded using [sentence-transformers / OpenAI embeddings — replace with actual model] and stored in [vector database — e.g. ChromaDB / FAISS / Pinecone] alongside metadata: source document, section heading, and ingestion timestamp. That metadata turned out to matter more than I initially expected — it's what let the system cite where an answer came from, which became one of the most-requested features once early users started testing it.

Reducing hallucination

This was the core engineering problem, not an afterthought bolted on at the end. A few things helped meaningfully:

Grounding checks before generation. Before the LLM generates a final answer, the pipeline checks whether the retrieved chunks actually contain content relevant to the query — not just embedding-similarity-relevant, but substantively relevant. If retrieval confidence falls below a threshold, the system returns an explicit "I don't have enough information on this" response instead of letting the model fill the gap with a fluent-sounding guess.

Constrained prompting. The generation prompt explicitly instructs the model to answer only from the provided context and to say so when the context is insufficient — a small thing, but it measurably reduced confident-but-wrong answers in testing.

Source attribution as a forcing function. Requiring every answer to cite the source chunk it came from turned out to be a good hallucination deterrent in its own right — it's harder for a model to fabricate a fact and simultaneously fabricate a plausible-looking citation for it.

None of these eliminate hallucination completely — nothing does, at least not yet — but together they turned "occasionally confidently wrong" into "usually honest about its own uncertainty," which is the bar that actually matters for a tool people are expected to trust at work.

What I'd do differently

If I were starting this again, I'd invest earlier in retrieval evaluation — building a small labeled set of questions with known-correct source chunks, and measuring retrieval precision/recall before ever touching the generation side. I built this mostly by iterating on end-to-end answer quality, which made it hard to tell whether a bad answer was a retrieval problem or a generation problem. A retrieval-specific eval set would have cut debugging time significantly.

I'd also want to expand the ingestion pipeline to handle non-text sources — a lot of internal knowledge lives in slide decks and spreadsheets, not clean prose, and the current pipeline assumes fairly well-structured text documents.

Where this goes from here

RAG is one of those areas where the gap between a weekend demo and a production-trustworthy system is almost entirely in the details covered above — chunking discipline, retrieval evaluation, and treating "I don't know" as a valid and important output. That's the part I found most interesting about this project, and it's the direction I want to keep building in.

The code for this project is on GitHub: Enterprise-GenAI-Knowledge-Assistant

If you're working on something similar or have thoughts on RAG evaluation strategies, I'd like to hear them — find me on LinkedIn or check out more of my work at kirankrishna2024.github.io.

DEV Community: Kiran Krishna P

Building an Enterprise RAG Knowledge Assistant: Lessons from Production