TL;DR
I built a Slack bot that answers "how does X work?" and "where does Y live?" questions about our private codebase, using Claude with tool-calling retrieval instead of embedding-based RAG. It took about two weekends, it answers most questions in under 30 seconds, and the biggest engineering problems had nothing to do with AI β they were about trust, staleness, and permissions. Here's the full build log, including the parts that went wrong. π
The Problem
Every team has that one person who knows where everything is. On my projects, that person is me β and I was tired of being a human grep.
The questions were always the same shape:
- "Where do we validate webhook signatures?"
- "How does the retry logic work in the billing worker?"
- "Is there already a helper for parsing these date formats?"
Each one costs the asker 20 minutes of digging or costs me a context switch. Multiply that by a few questions a day and you're losing real engineering time to archaeology, not building.
The obvious answer in 2026 is "point an LLM at the codebase." But the obvious implementation β chunk the repo, embed it, cosine-similarity your way to an answer β is where I started, and it's the first thing I threw away.
How I Solved It
The architecture that didn't work: embedding RAG
My first version was textbook RAG: split every file into 500-token chunks, embed them, stuff the top-k matches into a prompt.
It failed in a very specific way. Code questions are usually structural, not topical. When someone asks "how does retry logic work in the billing worker," the answer isn't in any single chunk. It's spread across a config file, a decorator definition, and the call site β three files that don't share much vocabulary. Embedding similarity finds you five chunks that all mention the word "retry" (including two from tests and one from a changelog) and none of the ones that matter.
I spent a week tuning chunk sizes and rerankers before admitting the approach was wrong for this job.
The architecture that worked: let the model drive the tools
The fix was to stop pre-deciding what context the model needs and instead give it the same tools I would use: search and read. Claude decides what to look for, reads what it finds, and follows the trail β the same loop a human does, just faster.
flowchart LR
A[Slack mention] --> B[Bot server]
B --> C{Claude agent loop}
C -->|search_code| D[ripgrep over repo]
C -->|read_file| E[File reader]
D --> C
E --> C
C -->|final answer| F[Slack thread reply]The whole thing is a Slack Bolt app (Python 3.13, slack-bolt 1.21) plus the Anthropic SDK. Two tools, that's it:
TOOLS = [
{
"name": "search_code",
"description": "Search the codebase with a regex. Returns matching "
"lines with file paths and line numbers.",
"input_schema": {
"type": "object",
"properties": {
"pattern": {"type": "string"},
"glob": {"type": "string", "description": "Optional file filter, e.g. '*.py'"},
},
"required": ["pattern"],
},
},
{
"name": "read_file",
"description": "Read a file (or a line range) from the codebase.",
"input_schema": {
"type": "object",
"properties": {
"path": {"type": "string"},
"start_line": {"type": "integer"},
"end_line": {"type": "integer"},
},
"required": ["path"],
},
},
]
search_code is just ripgrep behind a subprocess call, with output capped so a careless .* doesn't blow up the context window:
def search_code(pattern: str, glob: str | None = None) -> str:
cmd = ["rg", "--line-number", "--max-count", "8", "--no-heading", pattern]
if glob:
cmd += ["--glob", glob]
out = subprocess.run(cmd, cwd=REPO_ROOT, capture_output=True, text=True, timeout=10)
return out.stdout[:6000] or "No matches."
The agent loop is the standard tool-use pattern: call the model, execute any tool calls, feed results back, repeat until it produces a text answer. I run it on claude-sonnet-5 β fast enough that a 6-hop investigation still comes back inside Slack-attention-span, and cheap enough that I don't think about the bill.
A typical answer takes 3β6 tool calls. Watching the transcripts is genuinely fun: it greps for the obvious keyword, reads the hit, notices an import, follows the import, then answers with file paths and line numbers.
For the numbers people: median answer latency is around 20 seconds end-to-end, the slowest multi-hop investigations land near 45, and a month of real usage cost less than a single takeout lunch in API fees. Compare that to the 20 minutes of human digging each question used to burn and the math stops being interesting β it's just obviously worth it. π‘
The three problems that actually mattered
1. Hallucinated file paths destroy trust instantly. β οΈ
Early on, the bot answered a question with a confident reference to utils/validation.py β a file that did not exist. It had seen similar codebases in training and invented a plausible path. One wrong answer like that and your teammates stop trusting every answer.
The fix was mechanical, not prompt-based: before sending the reply, I extract every path-looking string from the answer and check it against the actual file tree. Any path that doesn't exist gets the whole answer flagged:
def verify_paths(answer: str, repo_files: set[str]) -> list[str]:
candidates = re.findall(r"[\w./-]+\.(?:py|ts|tsx|go|sql|yaml|toml)\b", answer)
return [p for p in candidates if p.strip("`") not in repo_files]
If verify_paths returns anything, the bot appends a visible warning and links the search query it ran, so the human can check. Since adding this, "confidently wrong path" incidents went from a few per week to zero reaching users.
2. Stale checkouts give stale answers.
The bot answers from a local clone. A clone that's three days old will cheerfully describe code that was refactored away on Tuesday. My first "fix" was pulling on a timer; the real fix was pulling on demand: every question triggers a git fetch + fast-forward before the agent loop starts. It adds ~2 seconds and eliminated the entire class of "that function doesn't exist anymore" answers. The answer also includes the commit SHA it read from, which turned out to be the single most trust-building detail in the whole project.
3. The bot must not see more than the asker can.
This is the one people skip. A codebase Q&A bot is effectively a read amplifier: anyone who can ask it questions can extract anything in its clone. Two hard rules came out of this:
- The bot's clone excludes anything secret-shaped β
.envfiles never land in git anyway (right? π ), but I also deny-list config directories and run the same secret scanner we use in CI against every tool result before it enters the model context. - The bot only joins channels where everyone already has repo read access. No DMs. This one is organizational, not technical, and it's non-negotiable.
Lessons Learned
Tool-calling retrieval beats embedding RAG for code Q&A. Code questions are graph traversals, not similarity lookups. Give the model grep and read, and let it walk the graph. My answer quality jumped more from this one architectural change than from everything else combined.
Verify outputs mechanically, not with prompts. "Do not invent file paths" in the system prompt reduced hallucinations. Checking paths against the file tree eliminated them. Anything you can verify with code, verify with code.
Citations are a feature, not decoration. Answers that end with
billing/worker.py:141(at commita3f9c21) get trusted and clicked. Answers without them get re-asked to a human. If your bot can't cite, it's a liability.Freshness is a product requirement. Nobody forgives "that code was deleted last week." Sync on demand, stamp the SHA, and staleness stops being a conversation.
Scope the bot's eyes to the asker's eyes. Decide what the bot can read and who can ask it before the first deploy, not after the first awkward answer. Retrofitting access control onto a chatbot is miserable.
What's Next
Two things are on the list. First, cross-repo questions β "which services call the notification API?" needs the bot to search several clones and merge results, which mostly means smarter tool routing. Second, letting it answer with diagrams: it already understands structure well enough that generating a Mermaid diagram of a subsystem on request feels within reach.
I'm deliberately not giving it write access. An answer bot that occasionally opens "helpful" PRs is a different product with a very different blast radius β that's a post for another day.
Wrap-up
Total damage: ~600 lines of Python, two weekends, and one discarded RAG pipeline. The bot now handles the majority of "where is / how does" questions in our Slack, and the ones it can't handle come to me with context attached instead of from zero.
If you've been thinking about pointing Claude at your own codebase: skip the embedding pipeline, start with two tools and an agent loop, and spend your saved week on verification and access control. That's where the real product is. β
If this was useful, follow me here on Dev.to β I write weekly about building with AI coding agents, including the failures. And if you build your own version, I'd genuinely love to hear what broke first: drop it in the comments. π¬
Top comments (0)