A story about building a workforce intelligence tool and learning that retrieval isn't reasoning.
I recently built an internal AI tool that lets people search and analyze our company's workforce data — think "who has experience with Kubernetes?" or "compare these two candidates for a backend role."
The first version used RAG. It worked. Sort of.
The second version used an agentic approach. It actually answered questions.
Here's what I learned.
The Problem
We had a growing repository of employee profiles — skills, experience, education, certifications. The data existed, but finding anything useful meant manually digging through documents or asking around.
I wanted to build something where you could just ask: "Find me someone with Python and AWS experience who's worked on data pipelines" — and get a real answer.
Classic RAG territory, right?
First Attempt: RAG
I did what the tutorials said:
- Chunk the documents — Split profiles into smaller pieces with metadata
- Generate embeddings — Used OpenAI's text-embedding-3-large
- Store in a vector database — Supabase with pgvector
- Retrieve and generate — Semantic search → feed results to LLM → get response
It took about a week to build the ingestion pipeline, set up the vector store, and wire up a basic chat interface.
And then I tried it.
Me: "Find Python developers with cloud experience"
System: "No matching data found."
Okay. Maybe my query was too vague.
Me: "Who knows Python?"
System: Returns three chunks of resume text, two of which mention Python in passing.
Not great. Let me try something specific.
Me: "Tell me about John Smith's experience"
System: Returns a chunk about John Smith plus a chunk about a different John. Generates a confused summary.
The Pain
Here's what I realized after a few days of testing:
RAG gave me chunks. Not answers.
The retrieval part worked fine. Semantically similar text came back. But the system had no intelligence about what to do with a query. Every question got the same treatment: embed → search → summarize chunks.
- Ask for a comparison? You get chunks.
- Ask for a recommendation? You get chunks.
- Ask for statistics? You get chunks.
I started adding logic to handle different query types. If the query mentions "compare," do this. If it mentions a name, do that. If it asks for "how many," do something else.
I was building a switch-case router on top of RAG. And it was getting ugly.
The Aha Moment
I was reading about tool use in LLMs — the pattern where you give the model a set of functions it can call, and it decides which ones to use based on the query.
And it clicked.
Instead of me deciding what to do with each query, the LLM could decide.
I didn't need a switch-case router. I needed to give the model tools and let it reason about which ones to use.
That's the difference between RAG and an agentic approach:
| RAG | Agentic |
|---|---|
| You define the pipeline | The LLM defines the pipeline |
| Same flow for every query | Dynamic flow based on intent |
| Retrieval-focused | Reasoning-focused |
| You handle edge cases | The model handles edge cases |
The Rebuild
I rebuilt the system using LangGraph with a simple agent loop using practices from Anthropic:
- Think — The LLM looks at the query and decides what tools to call
- Act — Execute the tools (search, get details, analyze, compare)
- Observe — Feed results back to the LLM
- Repeat — Until it has enough information to respond
I created about 10 tools:
-
semantic_search— Natural language search with vector similarity -
search_candidates— Exact-match filtering by skills/experience -
get_candidate_details— Full profile lookup -
compare_candidates— Side-by-side comparison -
analyze_skill_distribution— Workforce analytics -
recommend_candidates— Ranked recommendations for a role
The key insight: each tool does one thing well. The LLM figures out which combination to use.
The Difference
Same queries, different system:
Me: "Find Python developers with cloud experience"
Agent: Calls semantic_search with the query. Gets 8 candidates. Summarizes their backgrounds, highlights relevant experience, mentions that 3 have AWS certifications.
Me: "Compare the top two for a backend role"
Agent: Calls get_candidate_details for both. Calls compare_candidates with criteria. Returns a structured comparison with strengths and weaknesses.
Me: "What skills are we missing for a machine learning team?"
Agent: Calls analyze_skill_distribution. Calls identify_skill_gaps with ML-related skills. Returns a gap analysis with recommendations.
Same data. Same vector store. But now the system reasons about what to do instead of blindly retrieving.
When to Use Which
I'm not saying RAG is bad. It's great for what it's designed for.
Use RAG when:
- You need to ground LLM responses in source documents
- Questions are mostly retrieval-based ("what does document X say about Y?")
- You want simplicity and predictability
- The query type is consistent (always the same kind of question)
Consider an agentic approach when:
- Questions require reasoning, not just retrieval
- You need multiple operations to answer a query (search + analyze + compare)
- Query types vary widely (some need search, some need calculation, some need comparison)
- You're building switch-case logic on top of RAG (that's a smell)
You can also combine them — I kept the vector search as a tool the agent can use. RAG becomes one capability among many.
The Lesson
Building the RAG version taught me how retrieval works.
Building the agentic version taught me that retrieval isn't reasoning.
If your RAG system feels dumb — if it returns chunks when you want answers — consider whether you're asking it to do something it wasn't designed for. Maybe you don't need better retrieval. Maybe you need an agent.
I'm building AI tools at work and writing about what I learn. If this was useful, I'll share more about the architecture — the LangGraph setup, tool design, and the guardrails I added for PII protection.
About the author: I'm an engineer working on agentic AI systems. Currently building internal tools that use modern agentic workflows while keeping humans in the loop for ethical decisions.
Top comments (0)