This is a submission for the MLH x DEV Writing Challenge
Most docs bots lie. I built one that admits when it doesn't know.
Ask a docs chatbot about a feature that doesn't exist and watch it invent one. Parameter names, a config block, the whole thing. You paste it in, it breaks, and now you trust the docs a little less than you did before you asked.
I got tired of that, so I built the opposite for OLake, the open-source tool that moves data from databases and Kafka into Apache Iceberg. Its sibling, OLake Fusion, looks after the Iceberg tables. The bot is called Ask OLake and it plays by two rules. Every answer cites where it came from. And when it can't find the answer, it tells you.
I built it for the OLake x Hacktoberfest hackathon, and it won first prize. This is how it works, and the things I got wrong on the way there.
It's live here: https://ask-olake-nh7j.onrender.com. The code is at https://github.com/KunalSiyag/ask-olake.
Read the rendered site, not the repo
My first decision was to scrape the rendered docs site instead of the docs repo. The docs are Docusaurus MDX, and about a hundred of those files are shared snippets that only get inlined at build time. If you read the raw files, you miss text that users actually see on the page. The rendered pages have everything, plus stable anchors I can link to.
On top of the docs it reads the blog and comparison pages, four OLake GitHub repos (READMEs, Helm values, connector config specs, licenses) and a couple hundred closed issues that had real discussion in them. I gave them a trust order: docs first, then repo content and issues, then blog posts. When two sources disagree, the docs win.
No vector database, and it was fine
I didn't use a vector DB and I didn't use LangChain. The whole index is a few thousand chunks, and it fits in memory with room to spare. Fewer moving parts meant faster cold starts and a lot less to debug.
Retrieval is hybrid. People type exact identifiers like pgoutput or max_threads, and they also type things like "merge tiny files" when the docs say "compaction". BM25 handles the first kind well. Embeddings help with the second. The key I had for the event only allowed chat models, so I ran embeddings locally with model2vec, a tiny static embedder that answers in under a millisecond.
Here's the part that surprised me. I assumed adding embeddings would just make things better. On my 36 answerable test questions, BM25 alone found the right page in the top eight 94% of the time. Adding embeddings with equal weight dropped that to 86%. The embedder was simply weaker than BM25 on this corpus. Turning its weight down to half got me to 100%. If I hadn't had an eval set, I'd have shipped the worse version and felt good about it.
Honesty can't live in the prompt
Putting "don't make things up" in the prompt is a request. I wanted guarantees.
If retrieval comes back weak, the app never calls the model at all. It returns a plain "I couldn't find this in the OLake documentation", and that costs nothing. The sources list is built from the [1], [2] markers that actually appear in the answer, mapped back to the pages that were retrieved. The model can't cite a URL it was never given.
The eval caught another case. Someone asks "does OLake support Snowflake as a source?" and the bot said yes, because it found a GitHub issue where somebody requested that feature. So now there's a rule that issues are discussions, never proof that something exists. That one line fixed a whole class of wrong answers.
I let a benchmark pick the model
The organizers suggested GLM-4.7-Flash first, and they said teams were free to reorder the three allowed models. So I ran my full 42-question eval with each one as the primary.
GLM cited the right page 97% of the time, which is fine. But on a prompt-injection question it leaked its own system prompt, and its slowest answers took over 17 seconds. DeepSeek V4.1 Flash cited the expected page every time, handled all six unanswerable and injection questions correctly, and had a median of 2.3 seconds with a 95th percentile of 4.0.
So DeepSeek goes first, Gemini second and GLM third. If the first model hasn't answered after six seconds, the same request goes to Gemini and whichever answers first wins. I also added a check in code that refuses any answer containing system prompt text, because a prompt rule alone clearly wasn't enough.
The idea I was sure about (it wasn't)
I was sure query rewriting would help. The plan was to have a fast model turn "how do I install this locally" into the docs' own vocabulary before searching. I built it and measured it. On my set, accuracy fell from 97% to 89%. The rewrites kept echoing words from their own prompt instead of the user's intent.
The code is still there behind a flag that's switched off. I'd rather leave a dead feature in the repo than pretend I never tried it.
What did work was boring: a short synonym list that adds the docs' words next to the user's words ("write data to" also searches "destinations"), and feeding the model 20 chunks instead of 10, because a lot of correct answers were sitting at ranks 11 to 20.
Fresh docs for less than a tenth of a cent
Docs change, so a snapshot that's baked in once goes stale. A background job checks every hour whether the docs repo or any code repo has moved. If something changed, it re-fetches only that part, re-embeds only the chunks that are different, and swaps the new index in atomically. The old index keeps serving until the new one is ready. A nightly GitHub Action rebuilds everything from scratch and runs the retrieval eval as a gate before it commits.
A typical answer costs about $0.0006. I had a $3 credit for the event, and all my development and evaluation, around a thousand model calls, used about 55 cents of it. Between the "I don't know" shortcut, an answer cache and a daily cap on model calls, I wasn't worried about running out.
Where it still falls over
The known miss is a question that needs two facts from two different pages, like Kafka upsert behavior combined with a specific Databricks setup. Fixing it means breaking the question into parts, which means a second model call, and I decided one call per question was worth more than that case.
The embedder is also small. With an embedding API, a stronger model could carry more of the ranking. And the rate limiter and cache live in process memory, so scaling past one instance would need Redis.
Go break it
Ask it something real, or something it shouldn't know, and see what it does. The widget is a single script tag you can drop into any docs site:
<script src="https://ask-olake-nh7j.onrender.com/widget.js" defer></script>
If you find a question it answers badly, open an issue on the repo. That's how the eval set got better in the first place.
I'm Kunal. I build with AI and open source, and I post what I learn along the way. You can find me on X at @kunal_twts and on GitHub at KunalSiyag.
Top comments (0)