Two Ways of Knowing: Building an Agentic RAG Explorer for LLM Zoomcamp Capstones.

A technical deep-dive into an agentic RAG assistant built for the LLM Zoomcamp 2026 capstone — the architecture, five real production bugs, and what a race-condition stress test taught me about spend caps.

The problem with many GitHub repos

The LLM Zoomcamp 2026 cohort produced over 300 capstone submissions — real RAG projects, each with its own GitHub repo, README, dependency manifest, and a whole course leaderboard score. Browsing that by hand doesn’t scale. So I built a chatbot over the whole corpus to obbtain the all themes, used technology, and library.

_But the questions people actually ask about a corpus like this split cleanly into two kinds, and they need fundamentally different retrieval:

1. ”What does project X do? How does it implement reranking?” — a content question. Answer it by finding and grounding on the right README excerpt.

2. ”Which libraries are most common? What’s the top-scored project? Who’s the leaderboard’s top scorer?” — an aggregation question over the corpus’s structured metadata: score, votes, pass/fail, theme, declared dependencies, each author’s total course score._

The second category is the interesting one, because it’s exactly where naive RAG quietly breaks. Ask an LLM to “count” or “rank” over a pile of retrieved text chunks, and it will confidently produce a plausible-sounding wrong answer — not because the model is bad, but because counting isn’t a retrieval problem. It’s a database problem wearing a chat interface.

And there’s a subtler trap hiding in that second category: “top-scored project” and “top of the leaderboard” sound like the same question. They aren’t. A project’s own score and its author’s total course score (homeworks + project + learning-in-public) are genuinely different numbers, and conflating them is a real, plausible failure mode — one I actually hit and had to fix (more on that below).

Solving this well means routing each question to the right retrieval mechanism, not picking one strategy and forcing every question through it.

Architecture: one router, two ways of knowing

The ingestion side is what you’d expect from a RAG pipeline — scrape the listing pages, crawl every submitted repo’s markdown via the GitHub API, chunk by heading, cache everything to disk. Theme and tech-stack classification run as OpenAI Batch API jobs (bulk, offline, ~50% cheaper than synchronous calls) rather than one API call per project in the hot path.

The interesting part is the runtime query path. Instead of one retrieval strategy, there’s an agentic router sitting in front of two completely different tools: search_docs (hybrid keyword+vector search, RRF-fused, cross-encoder reranked, LLM-query-rewritten) and eleven parameterized, hand-written SQL functions-never LLM-generated SQL.

A “which libraries are most common” question never touches the model’s judgment about counting. It runs COUNT(*) … GROUP BY.

Why an agentic router, not “just RAG plus a calculator tool”?

The tool-calling loop itself is unremarkable — it’s the same shape as any OpenAI function-calling agent: feed the model a question and a set of tool schemas, let it pick zero or more tools, feed the results back, repeat until it produces a final answer instead of another tool call.

for _turn in range(self.max_turns):
    response = self.llm_client.responses.create(
        model=self.model,
        input=input_items,
        tools=self._tool_schemas(),
    )

    function_calls = [item for item in response.output if item.type == "function_call"]

    if not function_calls:
        return {"answer": response.output_text, "sources": all_sources, "tool_calls": tool_calls_log}

    # ... execute each function_call, append results, loop again

What actually matters is what’s in those eleven tool schemas — specifically, how much implicit intent you have to make explicit before the model reliably picks it up. Here’s a real example from this project. most_common_libraries takes an optional filter_top_scored_n parameter that restricts the aggregation to the top-N scored projects instead of the whole corpus. The first version of its description just documented what the parameter did. Ask “what are the most common libraries among the top-scored projects?” and the model would call the tool — but leave filter_top_scored_n unset, silently answering for the entire corpus instead of the subset the question actually asked about. The count was real, correctly computed, exactly matching the SQL — and still the wrong answer, because it answered a different question than the one asked.

The fix wasn’t code. It was making the qualifier explicit in the tool description itself:

IMPORTANT: if the question says ‘top-scored’, ‘top projects’, ‘best projects’, or similar, you MUST set filter_top_scored_n — leaving it null answers for ALL projects instead, which is a different (wrong) answer to that question.

Tool descriptions are prompts. Implicit qualifiers a human reader would catch instantly — ‘top-scored’, clearly means ‘restrict the scope’, — need to be spelled out just as explicitly as any other instruction, or the model will happily run the wrong exact query with complete confidence.

Five bugs that actually happened:

Everything above is the design that looks right on a whiteboard. Here’s what broke in practice.

1. CUDA wheels on a code path that never touches a GPU

Every embedding call in this project forces device=”cpu” explicitly, in code, everywhere — sentence-transformers on CPU, CrossEncoder reranking on CPU. And yet uv sync was still resolving torch’s full CUDA build by default: cuda-bindings, cuda-toolkit, half a dozen nvidia-* packages — several gigabytes of GPU tooling for a code path that never runs on a GPU, on any platform, including ARM (the cuda-toolkit extras aren’t gated to exclude aarch64).

The fix is a standard uv pattern most people don’t reach for until it bites them:

[tool.uv.sources]
torch = [{ index = "pytorch-cpu" }]

[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

One subtlety: this silently did nothing until torch was also listed as a direct dependency in pyproject.toml, not just a transitive one pulled in via sentence-transformers. Once it was explicit, re-locking dropped 18 CUDA/nvidia/triton packages from uv.lock, and uv sync actually uninstalled them from the local environment. Smaller image, faster builds, no wasted disk on a code path that was never going to touch a GPU.

2. A killed connection took the whole app down until restart

The router keeps one Postgres connection open for the process’s entire lifetime — avoiding a fresh-connection-per-query cost for a chat app that might field hundreds of questions. That’s a reasonable optimization, and it worked fine until a killed backend connection (psycopg.errors.AdminShutdown) took down every subsequent query_projects call until the whole process was manually restarted. One dead connection, and the assistant stopped answering any structured question for anyone.

The fix was reconnect-and-retry-once logic wrapped around every SQL tool call. But there was a second, quieter bug hiding underneath it: every one of those SQL tools is a read-only SELECT that never calls .commit(), and Postgres’s default autocommit=False means each one silently opens a transaction that never closes. Left alone, the shared connection accumulates “idle in transaction” state — which then blocks unrelated DROP TABLE statements running elsewhere in the same database. It happened twice before the pattern was obvious. The real fix was one line: roll back (never commit — nothing was written) after every single tool call, closing the implicit transaction immediately instead of leaving it open indefinitely.

3. Three judge false-negatives, one root cause

The live chat has an LLM-as-judge step that scores every answer’s relevance against the context it was actually grounded in. Three separate times, it flagged a correct answer as unsupported, and all three traced back to the same underlying mistake: the judge was being shown a derived view of the grounding data, not the real thing.

Round one: aggregation answers (no retrieved document chunks) were judged against a hardcoded empty context string, so every purely-SQL-grounded answer looked ungrounded by construction. Round two: after fixing that by building context from the sources list, a score-only field left the judge unable to verify an author_total_score field the answer legitimately cited, because that field simply wasn’t in the sources view. Round three, the sharpest one: aggregation-only tool results (a library count, say) have no github_url to become a sources entry — correctly, there’s nothing to link to — but that left the judge with nothing at all to check a fully correct count against, so it flagged genuinely accurate answers as hallucinated.

All three converged on the same fix: stop trying to derive judge context from a UI-facing view of the answer (the sources list), and instead hand the judge the actual tool-call results the answering model itself saw — the real data, not a lossy projection of it built for a different purpose.

4. The budget-cap race condition

Write on Medium
This is the one I’m proudest of catching before it mattered in production, because the naive version looked completely reasonable.

A public demo needs a hard spend cap — otherwise it’s a free, unlimited ChatGPT proxy to anyone with the URL. The first version did the obvious thing: read the total spent so far from Postgres, compare it to the cap in Python, and only call the LLM if there was room left.

That’s a classic time-of-check-to-time-of-use race. Under concurrent traffic, every simultaneous request reads “total is under the cap” before any of them commits their own cost. With enough concurrent visitors, the real total can blow straight past the cap — a $0.05 limit could, in the worst case, cost many times that if enough requests land in the same window.

The fix replaces “read, compare, spend” with one atomic Postgres operation — a conservative reservation made before the LLM call, not a check made after:

def try_reserve_budget(cap: float, amount: float = RESERVATION_USD) -> bool:
    conn = get_connection()
    try:
        with conn.cursor() as cur:
            cur.execute(
                """
                UPDATE budget_ledger
                SET total_spent = total_spent + %s
                WHERE id = 1 AND total_spent + %s <= %s
                RETURNING total_spent
                """,
                (amount, amount, cap),
            )
            row = cur.fetchone()
        conn.commit()
        return row is not None
    finally:
        conn.close()

Because a turn’s real cost isn’t known until after the LLM responds, this reserves a conservative fixed upper bound up front and trues it up to the real cost afterward (or fully refunds it if the call fails) — but the cap enforcement itself is a single atomic UPDATE … WHERE … RETURNING, which Postgres serializes across concurrent transactions the same way it serializes any other concurrent row update. There’s no gap between “check” and “spend” for a race to land in.

I didn’t just reason about this — I stress-tested it against the real database. Fifty concurrent reservation attempts against a $0.05 cap, with $0.01 reservations:

concurrent attempts: 50
succeeded reservations: 5
final ledger total: $0.0500 (cap was $0.05)
PASS: ledger never exceeded the cap under concurrency

Exactly five succeeded. The ledger landed at exactly the cap, not a cent over. The naive read-then-check version would have let far more than five through under the same load.

5. The disk-exhaustion saga that took two fixes to actually fix

Deploying to a small (2GB RAM, 20GB disk) EC2 instance, docker compose up — build failed with “no space left on device” mid-build. The instance had 8GB free at the time — that should have been plenty for one ~1.6GB Python image.

The first diagnosis looked right: docker-compose.yaml built five separate images from the identical Dockerfile (one per service — init-db, load-projects, build-index, streamlit, dashboard), each independently unpacking a multi-GB layer in parallel. Giving all five services a shared image: tag seemed like the obvious fix — one name instead of five.

It wasn’t enough. The retry failed with the exact same error. Compose was still building and exporting the image once per service definition, just landing on the same final tag — five redundant multi-gigabyte export operations racing for disk, instead of five distinct named images. The tag was shared; the work wasn’t.

The actual fix: only one service keeps a build: block. The other four reference the same image: with no build: of their own. Compose’s build phase — triggered by — build — runs to completion for every service that owns a build: block before it starts any containers at all; depends_on only orders container start, not the build phase. So by the time the other four services start, the image the first one built already exists on disk, and they just use it — no rebuild attempted, no redundant export.

The lesson, generalized past Docker: when a fix makes the symptom go away sometimes but the same failure mode returns under slightly different conditions, look for whether you actually eliminated the redundant work, or just gave the redundant work a nicer name.

Evaluating an agentic router

Retrieval-quality evaluation for a fixed single-strategy RAG pipeline is well-trodden: hit-rate, MRR, precision@k against a ground-truth question set. An agentic router adds a layer on top of that — it’s not just “did retrieval find the right chunk,” it’s “did the router pick the right tool.”

This project runs two evaluations at different depths. The deep one — comparing keyword/vector/hybrid/hybrid+rerank retrieval strategies and two answer-generation prompts against an LLM-generated ground-truth set, via the Batch API — is fully scaffolded but wasn’t run for this write-up; it needs a real API key budget I hadn’t allocated yet. No numbers invented to fill that gap.

The one that did run is faster and, in practice, more useful day to day: a 30-question smoke eval, running every question from the app’s own example-questions sidebar through the live assistant — real retrieval, real SQL, real LLM calls — judged the same way a live chat turn is judged. It’s a regression check on the router’s tool-routing, not a substitute for the deeper eval, and it re-runs in under three minutes:

30 questions — RELEVANT: 30 PARTLY_RELEVANT: 0 NON_RELEVANT: 0 missing: 0

relevance_score: 1.0 | total cost: $0.0248 | avg response time: 5.64s

This eval set earned its keep twice during development — it’s what caught the filter_top_scored_n bug above, and separately caught a one-shot Docker Compose loader container silently reverting an author_total_score field to NULL after a stale image was reused. A fast, always-current smoke eval that’s genuinely wired into your deploy process catches real regressions a slower, less-frequently-run ground-truth suite won’t.

Hardening for a public demo

Three small features made the difference between “a demo I can run locally” and “a demo I can actually hand someone a URL for”:

A production flag that hides the ingestion trigger from public visitors — nobody browsing a public demo should be able to kick off a live re-crawl of 278 GitHub repos.
The atomic budget cap described above.
Answer-from-history caching: an exact-match repeat of a previously-asked question (unsurprisingly common when people click the same example question twice) answers instantly from the logged prior turn — no new LLM call, no new cost, and no budget reservation at all, since nothing is actually being spent.
None of these are exotic. All three are the kind of thing that’s easy to skip when a project is “just a demo” — right up until it’s a public URL.

What generalizes

A few things from this build feel true beyond this specific project:

• Tool descriptions are prompts, not documentation. An implicit qualifier a human reader catches instantly — “top-scored” clearly means “restrict the scope” — has to be spelled out as explicitly as any other instruction, or the model will run the wrong exact query with complete confidence and no visible error.
• A judge needs the real grounding data, not a UI-shaped view of it. Every judge false-negative in this project traced back to feeding the judge something derived from the actual answer-generation context, built for a different purpose (usually, rendering source links), rather than the real tool output the answering model itself saw.
• Anything with a spend cap under concurrent traffic needs an atomic reservation, not a read-then-check. This is true well beyond LLM cost caps — any shared, capped resource checked by “read the total, compare, then act” has the same race, and it’s worth stress-testing with real concurrent load, not just reasoning about it on paper.

The code is on GitHub:

Built as a capstone project for [LLM Zoomcamp] 2026 cohort