From RAG Pipeline to RAG Tool: Building an Agentic Sales Knowledge Assistant

Why I stopped bolting retrieval onto the prompt and started handing it to the agent as a tool.

The problem every sales team has

Every services company has the same quiet problem. Over the years it writes hundreds of proposals, RFP responses, capability decks and case studies. Each one holds hard-won knowledge: how a similar deal was priced, which team was proposed, which accelerator was used, what the client actually asked for.

Almost none of it can be found when it's needed.

It lives in SharePoint folders, Teams channels, email attachments and people's laptops. The person who wrote the winning proposal three years ago has moved on. A new bid starts from a blank page.

The fix is a governed sales knowledge store. Every document is linked to a client, and to an opportunity or project when one is known. Every extracted fact is cited to the slide or page it came from. On top of that sits a chat assistant that salespeople can simply ask:

"What did we propose for discovery on our last three banking RFPs, and where do those deals stand now?"

They get a short answer with numbered sources that open the original document at the exact slide.

This post covers the most important design decision behind that chat: don't build RAG as a pipeline. Build it as a tool.

Classic RAG: retrieval as a pipeline

Most RAG systems in tutorials look like this:

user question
   └─> embed question
        └─> vector search (top k chunks)
             └─> paste chunks into the prompt
                  └─> LLM writes the answer

Retrieval happens once, before the model thinks, and always the same way. That works well for demos. In a real sales knowledge base it breaks quickly:

• One search is rarely enough. "Compare our proposals for Client A and Client B" needs at least two searches with different filters.
• The question is not the best query. "Where are we with them?" is a terrible search string. The model knows what it means. The pipeline doesn't.
• Some answers aren't in documents at all. The current stage, value and close date of a deal live in the CRM, not in a PDF.
• The pipeline can't recover. If the first search returns nothing useful, the model is stuck answering from bad context, or worse, from its own imagination.

Agentic RAG: retrieval as a tool

So turn the retrieval step inside out. Instead of the application deciding what to retrieve and when, give the agent a function, search_sales_knowledge, and let it decide.

user question
   └─> Agent (LLM on an agent runtime)
         ├─ thinks: "I need the proposals for this client"
         ├─ calls search_sales_knowledge("client X discovery phase", brand_id=…)
         │      └─> API runs hybrid search, returns numbered sources [1]…[8]
         ├─ thinks: "one proposal is linked to an opportunity, check where it stands"
         ├─ calls crm_opportunity_details(opportunity_id)
         │      └─> API reads the CRM (read only), returns source [9]
         ├─ thinks: "results are thin for the second client, rephrase"
         ├─ calls search_sales_knowledge("client Y cloud migration proposal")
         └─ writes the answer, citing [n] after every fact

The tool definition is a plain OpenAI-style function schema:

export const chatSearchToolDefinition = {
  type: 'function',
  function: {
    name: 'search_sales_knowledge',
    description: "Search the sales knowledge store (document sections, extracted facts and account knowledge pages). Returns numbered sources; cite them as [ref].",
    parameters: {
      type: 'object',
      properties: {
        query:       { type: 'string',  description: 'What to look for, in plain words.' },
        top:         { type: 'integer', description: 'How many sources to return (1-15, default 8).' },
        item_type:   { type: 'string',  description: 'Only items of one type (e.g. case studies).' },
        brand_id:    { type: 'string',  description: 'Only sources linked to one CRM account.' },
        accelerator: { type: 'string',  description: 'Only items that name this accelerator.' },
        offering:    { type: 'string',  description: 'Only items that name this offering.' },
      },
      required: ['query'],
    },
  },
};

Next to it, the agent gets a small set of read-only CRM tools: search accounts, account details, search opportunities, opportunity details, recent activities, pipeline totals, and wins and losses. That lets the agent combine two kinds of truth in one answer:

• What was said (proposals, decks, case studies, sales updates) comes from the knowledge store.
• Where the deal stands now (stage, value, owner, close date) comes from the live CRM.

The agent is told to use at most six tool calls per question, to search again with different words when the first results don't answer the question, and to say plainly when the store doesn't cover something.

A detail that matters: the tool runs in your code

Foundry Agents can attach Azure AI Search directly as a built-in tool. I deliberately didn't do that. The search tool is a client-side function. The agent run streams back a requires_action event, the API executes the search itself, and it submits the results back into the same stream.

That one choice gives the application, not the model, control over every retrieval:

Guard	What the tool handler enforces, regardless of what the model asks for
Active categories only	Retired categories never come back, even if they're still in the index.
Discontinued / deleted docs	Excluded by ID on every search.
Internal-only values	Facts found only in speaker notes stay hidden unless the category allows them.
Scoped questions	If the user scoped the chat to a client or to specific documents, that filter is forced in and the model can't widen it. CRM tools are switched off for document-scoped questions.
Related accounts	A client is often several CRM accounts. A filtered search is interleaved one-for-one with the same search unfiltered, so sibling accounts' sources are never missed.
Budget	Arguments are validated and clamped (top between 1 and 15).

The model chooses what to look for. The application decides what it's allowed to see.

Retrieval itself: hybrid, filtered, semantic

Behind the tool is Azure AI Search with a single index for the whole store. Every search is hybrid:

• Keyword search (BM25) on the chunk title and text. This is good for client names, product names and acronyms.
• Vector search (HNSW) with the question embedded by the same model used for the documents (text-embedding-3-large). This is good for meaning, so "cost reduction" finds "lower TCO".
• Semantic re-ranking on top, when available. If the semantic ranker refuses (for example, because the free quota is used up), the search silently falls back to plain hybrid for ten minutes instead of failing the user.

All of it runs under an OData filter built from the guards above. Every chunk also carries filterable facets: account IDs, opportunity IDs, project IDs, category, item type, and tags for industry, client, technology, service line, offering, accelerator, partner and named people.

Every hit records where it came from: the document ID, the slide or page and its number, and a deep link into the in-app viewer. That's what makes a citation clickable.

Citations you can trust

An agent that cites is only useful if the citations are real. This is enforced in two places.

At answer time. Every search result gets a reference number in a per-answer source book. The same chunk keeps the same number if a later search finds it again, so the agent can cite across several tool calls. When the answer is finished, a final pass does four things:

• It removes any [n] that points at nothing the tools returned, so there is never a citation without a source.
• It renumbers the rest 1..k in order of first use.
• It merges chunks of the same slide into one source for the reader.
• It stores the answer with its citation list.

At ingestion time. This is the bigger story.

The other half: a governed ingestion pipeline

Agentic retrieval is only as good as what it retrieves. Garbage in means confidently cited garbage out. So the write side is the opposite of agentic. It's a strict, deterministic pipeline where the model is only ever a helper and code makes every decision that matters.

Upload / SharePoint folder / Teams channel / API feed
  └─> store original privately, queue a job (Azure Service Bus)
       └─> convert to PDF (LibreOffice), read layout per page (Document Intelligence)
            └─> CLASSIFY  - does this document belong to its category? (refuse if confident it doesn't)
            └─> MAP       - which sections feed which fields?
            └─> EXTRACT   - strict JSON {field: {value, citations[]}} per field group
            └─> CHECK     - code verifies every citation: the snippet must exist on that slide/page
            └─> LINK      - client, opportunity, project scored against CRM masters (never guessed)
            └─> DEDUP     - same identity → new version of the same row; same bytes → blocked
            └─> WRITE     - row, fields, citations, Markdown file, history: one transaction
            └─> INDEX     - sections, facts and items chunked, embedded, pushed to AI Search

A few rules make this trustworthy:

• Every extracted value has a citation that code has checked. The model must return a snippet and a page number. Code then looks for that snippet on that page. If it isn't there (or is only on a different page, which gets corrected), the value isn't stored and the field is marked needs review. The model can't invent a fact into the store.
• Never guess an ID. Links to clients and opportunities are scored on stated evidence only: names, dates, values, people, and distinctive words from the CRM description. Only a clear winner, with a high score and a wide margin over the runner-up, links automatically, and the evidence is recorded. Otherwise a human decides.
• One document, one row. A document's identity is its category, client, opportunity and normalized file name, with "v4", "final", "draft" and "(1)" stripped. Upload a new version and the same row is updated in place, with the previous version archived. Upload the exact same bytes again and nothing is written.
• Final versions only. Drafts and working copies are refused with a reason.
• No personal data. Personal phone numbers and private emails never appear in any stored value, snippet or Markdown file.
• An append-only history. Every create, overwrite, block, retag, delete and index event is recorded, and the database itself denies updates to that history.

The result is that when the agent cites "[3] Proposal, slide 14", there is a verified chain behind it. The slide exists, the text is on it, the document is the final version, and it's linked to the right client.

Adding a category is data, not code

The store started with a single category, RFP proposals. It grew to hold RFP packs, supporting documents (meeting transcripts, clarifications), sales leadership updates fed from an API, capability decks, case studies and executive playbooks.

None of these added a new code path. A category is rows in three tables:

• category holds its name, its rules (client required? opportunity required?), and whether it has items.
• category_field holds what its fields mean, their group, description and type. The extraction prompt is built from this list.
• category_prompt holds versioned classify, map and extract prompts, with one active per stage.

The chat agent's instructions work the same way. The list of what the store holds is injected from the active categories at sync time, so the agent definition never names a category.

The stack, briefly

• Language and runtime: TypeScript (strict) everywhere, Node 24 and ESM, in one monorepo.
• packages/core: the only place business rules live. The API, worker and CLI all call it.
• apps/api: a Fastify REST API that never does long work inside a request.
• apps/worker: queue workers for the pipeline, media, feeds and schedules.
• apps/web: Angular with Entra sign-in, usable on a phone.
• Azure: Microsoft Foundry (an LLM for extraction and chat, text-embedding-3-large for vectors, Foundry Agents for the chat runtime), Azure AI Search, Document Intelligence, Service Bus and Blob Storage. Everything is reached with managed identity via DefaultAzureCredential, with no keys in code.
• Database: SQL Server for the governed store, with numbered forward-only migrations.

What I learned

1. Make retrieval a tool, but keep the tool in your code. The model is much better than a fixed pipeline at deciding what to search and whether to search again. The application is much better at deciding what may be returned. A client-side function tool gives you both.

2. Give the agent more than one source of truth. Documents tell you what was said. The CRM tells you what's true now. The questions salespeople actually ask need both, and an agent with two kinds of tools combines them naturally.

3. Citations are a contract, not a prompt instruction. Asking the model to cite isn't enough. Verify citations against the source at ingestion, and drop unknown references at answer time.

4. Go agentic on the read side and deterministic on the write side. Let the model explore freely when answering, because nothing it does there changes data. Let it do nothing on its own when storing, because everything stored is trusted later.

5. Put the variety in data. Six categories in, the pipeline, index and chat are still the same code. That's what keeps a system like this small enough to stay governed.

In one sentence

A governed, cited knowledge store turns years of scattered proposals into something searchable. An agent that treats retrieval as a tool can then choose its own queries, combine documents with live CRM data, and never state a fact it can't point to.

---

Questions and feedback welcome in the comments.