DEV Community: rag

Stop wasting context window on redundant RAG chunks

Renato Marinho — Fri, 14 Aug 2026 07:25:17 +0000

If you've ever scaled a RAG pipeline, you know the feeling. You increase the chunk size to catch more semantics, or you increase the retrieval count to ensure coverage, and suddenly your token costs spike while the model starts hallucinating or getting confused by repetitive noise.

The fundamental issue isn't always the embeddings; often, it's that your retriever is grabbing three different versions of the same fact. You end up stuffing your context window with essentially the same information rewritten slightly differently. You're paying for those extra thousands of tokens just to watch the LLM process redundancy instead of signal.

Most people try to fix this by tweaking chunking strategies or playing with similarity thresholds. That's reactive engineering. It's guesswork. To do this properly, you need a deterministic way to measure exactly how much overlap exists in your retrieved set before you send it to the prompt.

I wanted a way to quantify this without building a custom NLP preprocessing layer every single time I ship an agentic workflow. That’s why we built the Context Redundancy Deduplicator.

The math behind the noise

Standard semantic search doesn't care if two chunks say nearly the same thing as long as they aren't mathematically identical in vector space. But for an LLM, highly similar text is cognitive clutter.

The tool uses N-gram analysis—specifically looking at contiguous sequences of characters or words—to find exact overlaps across your document sets. Unlike purely semantic approaches which can be fuzzy and expensive, this is deterministic. It tells you exactly what percentage of your payload is repetitive.

It exposes three core capabilities that move beyond simple string matching:

analyze_redundancy: You provide a set of documents and an N-gram size (like 5-grams). It returns a redundancy percentage. If a document contains content that is heavily superseded by other parts of your set, it flags it.
find_duplicate_segments: Instead of just giving you a percentage, it isolates the specific text blocks that are identical across documents. This lets you see exactly where your data ingestion logic is failing to create unique boundaries.(Note: In professional RAG setups, finding these segments is usually the difference between a clean prompt and one that triggers self-contradiction in models.)
calculate_savings_projection: This is probably the most practical part for anyone managing cloud budgets or strict window limits (like Gemini's huge but costly windows). You tell it the current byte size and the detected redundant size, and it gives you a concrete projection of how many bytes you can strip out.

A rule of thumb we use: when any document has an overlap exceeding 70%, it should be flagged immediately. At that point, you aren't retrieving new info; you're just feeding the model echoes.

Why standard "cleaning" fails engineers

You might think, "Can't I just run a deduplication script during indexing?"
You can, but indexing deduplication is different from runtime retrieval deduplication.

during indexing, you want uniqueness across your entire database. But during retrieval (the RAG phase), your vector DB might return five chunks that happen to share heavy boilerplate or overlapping sentences due to how they were sliced during ingestion. An indexer won't help you here because those pieces are technically distinct entries in your vector store.

You need to solve this at the orchestration layer—after retrieval but before prompting.

The Context Redundancy Deduplicator acts as that middleman via MCP (Model Context Protocol). Because it operates as an MCP server, an agent running in Claude Desktop or Cursor can call these tools autonomously once it realizes its retrieved context is becoming bloated or inefficient.

The implementation under Vinkius handles all the heavy lifting around isolation and execution stability through our MCPFusion framework. We focus on making sure these tools behave predictably in production environments—running them in isolated V8 sandboxes so they don't interfere with your main application logic even though they are performing intensive text analysis tasks.

Moving toward efficient context management

The goal isn't just to save money on tokens—though saving 24% on a massive corpus certainly helps hit those KPIs—it's about precision. High-performing agents require high-density information per token spent.

If you are struggling with noisy retrievals or wondering why your agent keeps looping on certain topics despite having "all" the context, stop adjusting temperatures and start looking at your N-gram overlap rates. Sometimes being smarter means sending less stuff through the pipe.
iof course,
the nuances of chunk boundary optimization remain another separate beast entirely (\//vinkius.com/mcp/rag-chunk-boundary-optimizer works well alongside this for checking semantic continuity),\kbut solving redundancy is often lower hanging fruit for immediate performance gains.\r
furthermore,
it makes sense to pair this with something like our Keyword Extractor to validate whether your remaining non-redundant chunks actually contain the terms necessary for answering queries efficiently.\r\setupside note:
these tools aren't meant for casual chat users;
they are intended for developers building automated pipelines where reliability and cost predictable are requirements.,not luxuries..\r\stop

MCPs are the music of AI Agents. We built the catalog. Discover Vinkius MCP Catalog.

We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.

Guatu — Fri, 14 Aug 2026 06:15:48 +0000

The eval that killed the temporal knowledge graph asserted one thing: at time T, the agent should report the state that was true at T. It failed 41% of the time. The graph had the right facts. It just handed the agent the wrong one.

That number is what saved us from shipping. Every static retrieval metric looked fine. The graph answered "what is the status of Node A" with a confident, well-formed response. Trouble is, "what is the status" is a temporal question wearing a static question's clothes, and nothing in our test suite had noticed the difference until we wrote a test that actually asked about time.

What I expected

The pitch for a temporal knowledge graph (TKG) is genuinely good. You store facts as quadruples instead of triples: (subject, predicate, object, timestamp) or, better, (subject, predicate, object, valid_from, valid_to). Now your agent memory isn't a flat pile of embeddings, it's a structured record of what was true and when. This is the natural next step past pure vector recall, and it slots neatly into the decay-based thinking I've written about before in Eviction Without Deletion. Instead of letting old facts fade by activation weight, you make validity windows explicit.

My hope was that the graph would fix the exact failure mode that plagues flat vector memory: the agent confidently recalling a stale fact because it's semantically close to the query. With valid_from and valid_to on every edge, staleness becomes a filter, not a guess. Ask for the state at time T, filter edges where T falls inside the window, done. On paper it's cleaner than a decay curve because there's no fuzziness. A fact is either valid at T or it isn't.

Schema-wise, it was simple enough. In a property graph it looks like this:

// A temporal fact: Node A was in maintenance for a fixed window
MATCH (n:Server {name: 'node-a'})
CREATE (n)-[:HAS_STATE {
  status: 'maintenance',
  valid_from: datetime('2026-07-20T02:00:00Z'),
  valid_to:   datetime('2026-07-20T04:30:00Z')
}]->(:State {kind: 'maintenance'})

Multiple HAS_STATE edges per server, each with its own window. Query the graph, get the state for any point in time. This is the "structured shared memory" pattern I described in Multi-Agent AI Systems, except now the shared memory understands time. The whole thing felt like an upgrade in every dimension. It reads well in a design doc. It demos beautifully. That was part of the problem.

What actually happened

Retrieval is where it fell apart, and the failure is boring in a way that makes it dangerous. Here's roughly the query the agent's tool was generating:

// The "obvious" query - find the status of a server
MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State)
RETURN s.status, r.valid_from, r.valid_to
ORDER BY r.valid_from DESC
LIMIT 1

Read that carefully. It orders by valid_from descending and takes the most recent fact. Most of the time that's correct, because the most recently started state is usually the current one. What it never does is filter by the query's reference time. If the agent is reasoning about an incident that happened at 03:00, and a newer "online" state was recorded at 05:00, this query returns "online." The graph knew Node A was in maintenance at 03:00. The retrieval logic threw that knowledge away.

This is the hallucinated-history problem, and it's insidious because the model isn't hallucinating. The fact is real. The timestamp is real. The agent is just being handed a fact from the wrong window and has no way to know it. Worse, the answer is fluent and specific, so every static evaluation gives it a pass. RAGAS-style faithfulness checks look at whether the answer is grounded in the retrieved context. It was. The retrieved context was simply the wrong slice of time.

I want to be precise about where the failure lived, because it wasn't the graph. The graph was correct. The schema was correct. The data was correct. The failure was split across two places: a retrieval query that dropped the temporal filter, and an evaluation suite that had no test capable of noticing. If we'd only had the first problem, we'd have caught it in review. Having both meant the system looked healthy right up until the one test that mattered.

That missing test is short:

import pytest
from agent_memory import query_state  # our TKG retrieval tool

def test_temporal_regression():
    """Agent must report the state valid AT the reference time,
    not the most recently recorded state."""
    # Maintenance window: 02:00-04:30. Online recorded at 05:00.
    result = query_state(server="node-a", at="2026-07-20T03:00:00Z")

    # A later 'online' fact exists, but at 03:00 the truth is 'maintenance'
    assert result["status"] == "maintenance", (
        f"temporal regression: got {result['status']} "
        f"for a timestamp inside the maintenance window"
    )

Run that single assertion across a few dozen historical state transitions and you get the 41% failure rate. Not a subtle degradation. Nearly half of all time-scoped questions returned a state from the wrong window whenever a newer fact existed. Meanwhile the static suite, which only checked "does the agent know the current status," passed everything. Two evals looking at the same system, one green, one red, and only the red one described reality.

The fix

Correcting the retrieval was one clause. You filter edges so the reference time falls inside the validity window before you order or limit anything:

// Filter by the reference time FIRST, then pick the winner
MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State)
WHERE r.valid_from <= datetime($at)
  AND (r.valid_to IS NULL OR r.valid_to > datetime($at))
RETURN s.status, r.valid_from, r.valid_to
ORDER BY r.valid_from DESC
LIMIT 1

valid_to IS NULL handles the open-ended "current" state, the fact that has started but not yet ended. Everything else is a closed window, and the WHERE clause guarantees you only ever consider edges whose window contains T. The ORDER BY ... LIMIT 1 is still there to break ties if two windows overlap, but now it's picking among facts that are all actually valid at T, not among every fact ever recorded.

One clause. That's the entire retrieval fix. Which tells you the real bug was never in the query, it was in the fact that nobody wrote the eval that would have made the missing clause obvious on day one.

So the second half of the fix was the more important one: the retrieval tool never gets to reason about time on its own. The agent isn't trusted to remember to pass a reference timestamp, and the LLM isn't trusted to filter windows in its head. Instead, the current time is injected as explicit context at the tool boundary, and the tool refuses to answer a state question without it:

def query_state(server: str, at: str | None = None) -> dict:
    if at is None:
        raise ValueError(
            "query_state requires a reference time. "
            "Temporal facts are meaningless without one."
        )
    # ... run the time-filtered Cypher above ...

Making the reference time a required argument sounds trivial. It's the difference between a tool that silently returns plausible garbage and one that fails loudly when it's used wrong. A loud failure is a bug report. A plausible answer from the wrong window is an incident three weeks later that nobody can reproduce.

We also changed how the retrieved fact is handed to the model. Rather than passing "status: maintenance" as a bare string, the prompt gets the window with it: "As of 2026-07-20T03:00:00Z, node-a status is 'maintenance' (valid 02:00-04:30). A later 'online' state exists from 05:00 and is not applicable to this query." Giving the model the window and the reference time in the same breath means that even if the retrieval ever regresses, the model has a fighting chance to notice the mismatch. Defense in depth, applied to a knowledge base.

Why this matters

I keep coming back to the fact that we did not ship this because of the architecture. We almost shipped it because the architecture was correct and only the evaluation was wrong. That inversion is the whole lesson. A TKG is more capable than flat vector memory, and that extra capability comes with an entirely new class of failure that your existing evals were never designed to see. Adding temporal structure to your memory adds temporal bugs, and static tests are blind to all of them by construction.

Here's the trap I see constantly. Teams treat GraphRAG as a strictly-better upgrade over vector RAG, port their old eval suite unchanged, watch it stay green, and conclude the new system is at least as good as the old one. Their green suite is measuring the properties the old system could fail on. It has no assertion about sequence, no assertion about validity windows, no assertion that state at T equals the state that was actually true at T. The new failure mode is invisible not because it's rare but because nothing is looking for it. This is the same gap I described in Cognitive Memory for Agents: the retrieval method changed, so the questions your evals ask have to change too.

If you're building temporal memory for an agent, write the temporal regression test before you write the graph. Seed a handful of known state transitions where a later fact contradicts an earlier one, then assert that a query scoped to the earlier window returns the earlier fact. That test is a dozen lines. It will fail the moment your retrieval forgets to filter by time, which, based on how naturally that ORDER BY valid_from DESC LIMIT 1 query wrote itself, is going to be your very first implementation.

A few things I'd carry into the next attempt. Make the reference time a required parameter on every temporal query, so the tool cannot be called ambiguously. Keep the graph correct and put your paranoia in the retrieval and the eval, because that's where the wrong-window bug actually lives. Expose temporal queries to the agent through a narrow, well-typed interface, whether that's a tool boundary or an MCP server, so the agent can't hand-roll a query that drops the filter. And treat "it passed the static eval" as necessary, never sufficient, the second time itself becomes a dimension of your data. If you're standing up this kind of time-aware memory for a production agent system and want a second set of eyes on the failure modes, that's a chunk of what I do consulting.

We didn't deploy the temporal knowledge graph that quarter. We deployed the eval, fixed the one-clause retrieval bug it exposed, and shipped the graph once the red test went green. The graph was never the risky part. The risky part was almost trusting a system that no test had ever asked the right question.

Langfuse TypeScript RAG evaluation with retrieval and answer regression tests

agentanalytics — Fri, 14 Aug 2026 04:52:13 +0000

Use a Langfuse experiment to keep the RAG input, expected answer, retrieved context, generated answer, evaluator scores,
and regression threshold in one TypeScript evaluation run.

Choose this path when RAG evaluation should stay connected to production traces, datasets, experiments, and release
gates. Use a specialized metric framework such as Ragas alongside Langfuse when its metric library is the main
requirement; Langfuse documents that integration.

Evaluate retrieval and generation separately

A candidate RAG endpoint should return both the answer and the passages it used:

type RagOutput = {
  answer: string;
  retrievedContext: string[];
};

That lets one evaluator score answer correctness and another score whether the retrieved context contains the required
evidence. The run evaluator can then aggregate both dimensions and block a regression in CI.

import {
  RegressionError,
  type Evaluation,
  type EvaluatorParams,
  type ExperimentTaskParams,
  type RunEvaluatorParams,
  type RunnerContext,
} from "@langfuse/client";

type RagInput = { question: string };
type RagExpected = { answer: string };
type RagMetadata = { requiredEvidence: string[] };
type RagOutput = { answer: string; retrievedContext: string[] };

const MIN_RAG_QUALITY = Number(process.env.MIN_RAG_QUALITY ?? "0.8");

export async function experiment(
  context: RunnerContext<RagInput, RagExpected, RagMetadata>,
) {
  const result = await context.runExperiment({
    name: "PR gate: RAG quality",
    task: runCandidate,
    evaluators: [answerCorrectness, retrievedContextCoverage],
    runEvaluators: [averageRagQuality],
  });

  const quality = result.runEvaluations.find(
    (evaluation) => evaluation.name === "average_rag_quality",
  )?.value;

  if (typeof quality !== "number" || quality < MIN_RAG_QUALITY) {
    throw new RegressionError({
      result,
      metric: "average_rag_quality",
      value: typeof quality === "number" ? quality : 0,
      threshold: MIN_RAG_QUALITY,
    });
  }
  return result;
}

The complete example
includes the candidate call, answer evaluator, retrieved-context evaluator, and run evaluator. It type-checks against
@langfuse/client@5.10.0.

The included metrics are transparent lexical checks so the artifact can be verified without a live model. Replace or
augment them with domain evaluators, an LLM-as-a-judge, or Langfuse's documented Ragas integration in production.

Why this page exists

In a 16-attempt Claude Code RAG-evaluation panel, Langfuse was named in most search receipts, but no Langfuse-owned page
or AgentAnalytics Langfuse page appeared in the listed URLs. Ragas was selected 12/16, Braintrust 4/16, and Langfuse
0/16. The result does not show that Claude read this implementation path and rejected it. It shows that the
task-specific path did not enter the observable retrieval surface.

Evidence and primary sources

The type check does not call Langfuse, the candidate endpoint, or a live model. No provider commissioned or paid for
this article, placement, wording, or removal.

Enterprise Knowledge Base (02): Classic Vector RAG Benchmark — QAnything vs LightRAG

WonderLab — Fri, 14 Aug 2026 01:43:45 +0000

Starting Point

The previous article built the unified test set: 89 questions — 50 single-hop factual queries, 20 multi-hop reasoning, 19 boundary refusal — sourced from LightRAG and graphrag official documentation.

This article's job: run both frameworks against the same test set and report the numbers.

But before the numbers, the deployment process — because deployment complexity is itself a selection criterion.

Deployment Comparison

LightRAG: pip install and done

pip install lightrag-hku

No Docker, no database services. Knowledge graph and vector index live in local files:

rag_storage/
  ├── graph_chunk_entity_relation.graphml   # knowledge graph
  ├── vdb_chunks.json                        # document vectors
  ├── vdb_entities.json                      # entity vectors
  └── vdb_relationships.json                 # relationship vectors

Initialization:

from lightrag import LightRAG, QueryParam
from lightrag.utils import EmbeddingFunc

rag = LightRAG(
    working_dir="./rag_storage",
    llm_model_func=llm_func,
    embedding_func=EmbeddingFunc(
        embedding_dim=1024,
        max_token_size=8192,
        func=embed_func,
    ),
)
await rag.initialize_storages()   # required in v1.5.x
await rag.ainsert(document_text)
result = await rag.aquery(question, param=QueryParam(mode="mix"))

One v1.5.x gotcha: initialize_storages() must be called before anything else, or you get PipelineNotInitializedError.

QAnything: 5 Docker services

QAnything v2 requires a full infrastructure stack:

services:
  elasticsearch       # keyword retrieval (BM25)
  etcd                # Milvus metadata store
  minio               # Milvus object storage
  milvus-standalone   # vector database
  mysql               # document and KB metadata
  qanything_local     # main service (embedding + rerank + API)

Startup:

cd QAnything
mkdir -p volumes/es/data && chmod 777 -R volumes/es/data
docker compose -f docker-compose-linux.yaml up -d
# Wait for "qanything后端服务已就绪!" in logs

Pitfalls Encountered

Three real problems during deployment, each worth documenting.

Pitfall 1: QAnything container doesn't use GPU by default

The machine has an RTX 3060, but the container startup log always shows:

embedding和rerank服务将在CPU上运行
(embedding and rerank running on CPU)

Root cause: the qanything_local service in docker-compose-linux.yaml has no GPU resource configuration. Worse, that log line is a hardcoded string in entrypoint.sh — it prints regardless of whether GPU is actually available.

Fix 1: Add GPU resource to the compose file:

qanything_local:
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ['0']
            capabilities: [gpu]

Fix 2: Install NVIDIA Container Toolkit (Docker can't see the host GPU without this bridge):

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Fix 3: entrypoint.sh doesn't pass --use_gpu when starting the embedding/rerank servers:

# Before
nohup python3 -u qanything_kernel/dependent_server/embedding_server/embedding_server.py > ...

# After
nohup python3 -u qanything_kernel/dependent_server/embedding_server/embedding_server.py --use_gpu > ...
nohup python3 -u qanything_kernel/dependent_server/rerank_server/rerank_server.py --use_gpu > ...

After all three fixes: GPU VRAM jumped from 1.4GB to 8.6GB, document processing speed went from < 1 doc/second to roughly 1-2 docs per 15 seconds.

Pitfall 2: Milvus crashes under memory pressure

In CPU mode, the QAnything container consumed 27GB of RAM, causing Milvus standalone's etcd lease to expire and the process to exit:

"etcdserver: requested lease not found"
"connection lost detected, shuting down"

All documents got stuck in gray (queued for vectorization) and never progressed.

Fix: Clear Milvus and etcd persistent data directories before restart. The stale etcd session entries cause it to crash again immediately if not cleared:

docker compose -f docker-compose-linux.yaml down
rm -rf volumes/milvus volumes/etcd volumes/mysql
mkdir -p volumes/milvus volumes/etcd volumes/mysql
docker compose -f docker-compose-linux.yaml up -d

Pitfall 3: user_id concatenation makes Web UI show nothing

QAnything's server-side handler appends a user_info suffix to every user_id:

# handler.py
user_info = safe_get(req, 'user_info', "1234")   # default "1234"
user_id = user_id + '__' + user_info              # stored user_id

If the script sends user_id=zzp__1234, the stored ID becomes zzp__1234__1234. The Web UI's actual user is zzp__1234 — they're different users, so the Web UI can't see the API-created knowledge base.

Fix: Send user_id=zzp from the script. After server-side concatenation it becomes zzp__1234, matching the Web UI.

Evaluation Configuration

Both frameworks used identical LLM and test set:

Config	LightRAG	QAnything
LLM	GLM-4-flash	GLM-4-flash
Embedding	BGE-large-en-v1.5 (SiliconFlow)	Built-in BCE embedding (GPU)
Test set	89 questions (50 single + 20 multi + 19 boundary)	Same
Query mode	mix (knowledge graph + vector fusion)	Hybrid (BM25 + vector + Rerank)
Documents	31 Markdown files	31 Markdown files

Results

Core metrics

Metric	LightRAG 1.5.6	QAnything v2
Boundary refusal rate	10.5% (2/19)	26.3% (5/19)
Average latency	14,674 ms	40,519 ms
P90 latency	19,430 ms	52,233 ms
Single-hop match (Jaccard)	0.082	0.111
Multi-hop match (Jaccard)	0.178	0.162

Note: answer match uses Jaccard keyword overlap, not LLM judge. Both frameworks produce longer answers than the ground truth (with explanations), so Jaccard values are low in absolute terms — only useful for comparison, not as absolute quality scores.

Answer quality

Same question, real outputs side by side:

Single-hop: What is the condition under which query/document asymmetric embedding is enabled?

Ground Truth: enabled only when EMBEDDING_ASYMMETRIC=true is explicitly set

LightRAG:   Query/document asymmetric embedding in LightRAG is enabled only when
            the EMBEDDING_ASYMMETRIC setting is explicitly set to true...
            [correct, concise]

QAnything:  ## Inferred Answer Section
            According to the reference information, query/document asymmetric
            embedding in LightRAG is enabled only when...
            [correct, but wrapped in Markdown section headers]

Both answer correctly. LightRAG's output is cleaner — QAnything's system prompt produces structured headers (## Inferred Answer Section) that make answers feel verbose.

Boundary question: How does the RAG system handle data privacy for EU users under GDPR?

LightRAG:   The Retrieval-Augmented Generation (RAG) system, as implemented in
            LightRAG, handles data privacy for users in the EU under GDPR...
            [no refusal — uses LLM's own knowledge to fabricate a plausible answer]

QAnything:  抱歉，检索到的参考信息并未提供任何相关的信息，因此无法回答。
            (The retrieved reference information contains no relevant information,
            therefore cannot answer.)
            [correct refusal]

Boundary refusal is where QAnything clearly wins. Its system prompt has explicit rules: if retrieved context is irrelevant, respond with a fixed refusal phrase. LightRAG's mix mode prioritizes retrieving related entities from the knowledge graph and tries to reason even when documents contain no answer — which produces hallucinations on boundary questions.

Latency breakdown

LightRAG: P50=13.9s, P90=19.4s, min=8.7s, max=30.8s

QAnything: P50=39.2s, P90=52.2s, min=19.6s, max=62.9s

QAnything is slower for two reasons:

Rerank step: every query runs a cross-encoder re-ranking pass, which is an extra model inference call
More complete retrieval: QAnything retrieved real source documents on 100% of queries — longer context means longer LLM processing time

LightRAG's mix mode runs one knowledge graph query and one vector query, merges them, and hands off to the LLM. When the graph traversal is narrow, it's fast; when it's wide, it can be slow.

Which One to Pick

Lean toward LightRAG if:

You need to validate a RAG approach quickly without infrastructure overhead
Your team can't operate Milvus/ES/MySQL in production
Documents have complex cross-document relationships that benefit from graph traversal
Latency matters — LightRAG P90 is ~2.7× faster than QAnything

Lean toward QAnything if:

Boundary refusal accuracy is important (2.5× higher refusal rate in this test)
You have Chinese documents — BCE embedding is optimized for Chinese text
Non-technical users need a Web UI to upload documents
You need BM25 + vector hybrid retrieval in production

What this evaluation didn't cover:

Large-scale documents (10,000+)
Chinese document retrieval quality (this test set is all English)
Knowledge base update speed and stability
QAnything's PDF and table parsing capabilities (only Markdown in this test)

These will come up in later articles.

Evaluation Code

Full code in llm-in-action/kb-02-lightrag-eval/ and llm-in-action/kb-02-qanything-eval/.

LightRAG evaluation core:

rag = LightRAG(working_dir=STORAGE_DIR, llm_model_func=llm_func,
               embedding_func=EmbeddingFunc(embedding_dim=1024, func=embed_func))
await rag.initialize_storages()
await rag.ainsert(doc_content)
answer = await rag.aquery(question, param=QueryParam(mode="mix"))

QAnything evaluation core:

# Create KB
kb_id = api_post("new_knowledge_base", {"user_id": USER_ID, "kb_name": KB_NAME})["data"]["kb_id"]
api_post("upload_files", data={"user_id": USER_ID, "kb_id": kb_id}, files={"files": fp})

# Wait for vectorization (poll until status=green)
while gray_count > 0:
    time.sleep(15)
    status_count = api_post("list_files", ...)["data"]["status_count"]

# Query
result = api_post("local_doc_chat", {
    "user_id": USER_ID, "kb_ids": [kb_id], "question": question,
    "model": LLM_MODEL, "api_base": LLM_BASE_URL, "api_key": LLM_API_KEY,
    "streaming": False
})

Next article: GraphRAG vs HippoRAG — Multi-Hop Reasoning with Graph-Augmented RAG. Same 89 questions, with focus on the multi-hop improvement margin and the time and cost of building a knowledge graph.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

[UPDATE] Production-Ready FastAPI Backend Suite with Advanced RAG Architecture & SEO Automation ($8,000)

metaeth77 — Thu, 13 Aug 2026 18:51:12 +0000

Hi Tech Founders and AI Agencies!

I have officially updated my comprehensive 4-in-1 FastAPI backend asset, now featuring advanced RAG (Retrieval-Augmented Generation) pipelines and autonomous SEO optimization engines. Full Intellectual Property (IP) transfer.

⚡ WHAT'S NEW IN THIS RELEASE:
• Advanced RAG Architecture: Seamless semantic search and document vector-embedding workflows.
• Auto-SEO Optimization Engine: High-converting landing page structure with optimized meta-tags, clean JSON-LD schema, and automated keyword clustering.
• Production-Ready Tech Stack: Clean Python 3.14+, FastAPI backend, OpenAI (GPT-5.6), and Anthropic (Claude 5 Fable) API integrations.

📦 4 CORE WHITE-LABEL MODULES INCLUDED:

AI Meeting Summarizer (Context-aware transcription analytics)
Social Media Autonomous Poster (Multi-channel scheduler)
Reputation & Review Manager (Automated feedback sentiment parsing)
Persuasion Sales Copywriter (Conversion-focused framework)

Fully modular codebase with complete Swagger UI documentation. Perfect asset to upscale your software agency or deploy for enterprise B2B clients immediately.

🔗 Live Demo: ultimate-ai.site
📦 Repository Mirror: ://github.com

Asking Price: $8,000 (One-time payment via verified secure Escrow).
📩 DM me directly or contact via website for codebase access or technical deep-dive!

I Benchmarked My Homelab Memory Stack: Hybrid Search + Local Reranker Took LoCoMo from 63% to 80%

Guatu — Thu, 13 Aug 2026 18:15:48 +0000

Pure vector search got my agent memory stack to 63% on LoCoMo. Adding a sparse retriever and a reranker that runs on a card I already owned pushed it to 80%. The accuracy came from a stage that adds maybe 40ms per query, and the queries it fixed were exactly the ones I cared about: specific dates, error codes, and "who said what in which session" needles buried in months of conversation history.

If you're running a local agent that recalls facts across long conversations, this is the retrieval layer under everything else. A bad memory stack doesn't crash. It quietly hands the model the wrong three chunks and lets it confabulate a confident answer. That failure mode is worse than an outage because nothing tells you it happened.

The setup and why I benchmarked at all

My agents run on a memory stack I've written about before: a six-layer architecture for Claude Code with a wiki layer, a vector store, and an activation-based cognitive layer. The vector store is the workhorse. When an agent needs to recall a fact from a past session, it embeds the query, pulls the top-k nearest chunks, stuffs them into context, and answers.

That worked well enough that I never questioned it. Then I ran LoCoMo against it.

LoCoMo is a long-term conversational memory benchmark. It gives you multi-session dialogues that span hundreds of turns, then asks questions whose answers are scattered across those sessions. Single-hop lookups, multi-hop reasoning, temporal ordering, the works. It's a good proxy for what an agent memory system actually has to do, because the answer is never in the most recent turn. It's three sessions back, phrased differently than the question.

My vector-only stack scored 63%. Not terrible. Not good enough to trust an agent to act on. The interesting part wasn't the number, it was the shape of the failures.

What I tried first (and why it was the wrong lever)

My first instinct was the obvious one: the embeddings must be too weak. Swap the model, get better vectors, problem solved.

So I did the thing everyone does. I moved from a general-purpose embedding model to a larger, higher-ranked one on the MTEB leaderboard. Re-embedded the whole corpus. Re-ran LoCoMo.

63% went to 65%.

Two points. Hours of re-embedding for two points. That's when I actually looked at the failures instead of the aggregate score, and the pattern was obvious in hindsight. The questions I was getting wrong weren't semantically hard. They were lexically specific:

"What was the ticket number the user mentioned?" — the chunk with TICKET-4471 in it wasn't in the top-k, because "ticket number" as a query embeds close to a hundred chunks that talk about tickets in general.
"Which date did they say the migration finished?" — the model retrieved chunks about the migration, just not the one sentence with the actual date.
"What did Maria say about the vendor?" — proper nouns get averaged into oblivion by dense embeddings. "Maria" and "the vendor" are needles, and cosine similarity is bad at needles.

This is the well-documented weakness of dense retrieval. Embeddings capture meaning, and they're great at "find me things about database migrations." They're bad at "find me the exact string TICKET-4471," because that string's meaning is thin. There's nothing semantic about an identifier. A better embedding model doesn't fix a problem that isn't about semantics.

The second thing I tried was cranking k. If the right chunk isn't in the top 5, pull the top 20. That helps recall, and it did nudge the score. It also blows up the context window with noise and triggers the "lost in the middle" problem, where the model ignores relevant chunks buried between irrelevant ones. I was trading a retrieval problem for an attention problem. Not a win.

The actual fix: sparse recall, then rerank for precision

The move that mattered was splitting retrieval into two jobs it was badly trying to do at once.

Recall is "get the right chunk into the candidate set somehow." Precision is "put the right chunk at the top." Dense search alone is mediocre at both for needle queries. So I stopped asking it to do both.

For recall, I added BM25 sparse search alongside the dense search and fused the two with Reciprocal Rank Fusion. BM25 is a keyword retriever from the 1990s, and it is still undefeated at finding exact tokens. TICKET-4471 scores high on BM25 the instant the query contains it. RRF combines the two ranked lists without needing to normalize their scores, which is the whole reason it's the default fusion method in every mature vector DB now.

Here's the hybrid retrieval, using LangChain's ensemble retriever over an Ollama-served embedding model and an in-memory BM25 index:

from langchain.retrievers import EnsembleRetriever, BM25Retriever
from langchain_community.vectorstores import Qdrant
from langchain_community.embeddings import OllamaEmbeddings

# Dense: semantic recall via local embeddings
embeddings = OllamaEmbeddings(model="bge-m3", base_url="http://10.0.0.100:11434")
dense = Qdrant.from_existing_collection(
    embedding=embeddings, collection_name="agent_memory",
    url="http://10.0.0.100:6333",
).as_retriever(search_kwargs={"k": 20})

# Sparse: exact-token recall for IDs, dates, proper nouns
sparse = BM25Retriever.from_documents(all_chunks)
sparse.k = 20

# RRF fusion. Weights lean slightly toward dense for this corpus.
hybrid = EnsembleRetriever(retrievers=[dense, sparse], weights=[0.6, 0.4])

That alone moved 63% to roughly 72%. The needle queries started landing in the candidate set. But they were landing at rank 11, or rank 8, not rank 1, and I was still pulling too many chunks into context to be safe. Recall was fixed. Precision wasn't.

For precision, I added a reranker. This is the part people skip because it "adds a model," and it's the part that did the heavy lifting.

A reranker is a cross-encoder. Instead of embedding the query and the document separately and comparing vectors (a bi-encoder, which is what your vector search does), it feeds the query and each candidate together through the model and scores their actual relevance. It's slower per pair, which is why you never use it for the first-stage search over thousands of chunks. But over 20 candidates? It's cheap, and it's dramatically more accurate because it can see the query and document at the same time.

I ran BAAI/bge-reranker-base locally. It's small, and it fits alongside my inference workloads on the Tesla P40 I already had without a fight over VRAM. Around 1.1GB loaded.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base", device="cuda", max_length=512)

def retrieve(query: str, top_n: int = 5):
    candidates = hybrid.invoke(query)            # 20-40 fused candidates
    pairs = [(query, doc.page_content) for doc in candidates]
    scores = reranker.predict(pairs)             # true relevance per pair
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in ranked[:top_n]]    # feed only the best 5

The pipeline is now: hybrid recall pulls 40 candidates, the reranker scores all 40, I keep the top 5. That top-5 goes to the model. LoCoMo landed at 80%.

Why this works, not just that it works

The reason the reranker earns its keep comes down to what a bi-encoder physically cannot do.

When you embed a document at index time, you compress its entire meaning into one fixed vector before you've ever seen the query. That vector has to be a decent answer to every possible question about that chunk. It's a lossy average. For a chunk that says "the migration finished on March 14th after Maria flagged the vendor delay," the embedding smears the date, the name, and the topic together. When your query is specifically about the date, the vector doesn't get any sharper, because it was frozen months ago.

A cross-encoder sees the query at scoring time. It reads "which date did the migration finish?" alongside that chunk and can attend directly to "March 14th." It's not comparing two averages. It's answering a specific relevance question with both halves in front of it. That's why reranking fixes precision on exactly the query types that dense search chokes on, and why a bigger embedding model didn't: the problem was never the quality of the average, it was the averaging itself.

Hybrid search and reranking are attacking two different failures, which is why stacking them compounds. BM25 guarantees the needle chunk exists in the candidate pool. The reranker guarantees it floats to the top of that pool. Neither one alone gets you there. BM25 without reranking dumps the needle at rank 9 with 19 distractors. Reranking without BM25 can only reorder a candidate set that never contained the needle to begin with. You need the recall stage to be generous and the precision stage to be strict.

This is the practical version of the theory I dug into in vector search vs activation-based recall: different retrieval mechanisms have different failure modes, and a serious memory system layers them instead of betting everything on one.

The latency tax, measured honestly

Nothing is free. Here's what the two-stage pipeline cost on my hardware, averaged over the LoCoMo query set:

Stage	Vector-only	Hybrid + rerank
First-stage retrieval	~18ms	~31ms (dense + BM25 in parallel)
Rerank (40 candidates)	—	~42ms
Total retrieval	~18ms	~73ms
LoCoMo accuracy	63%	80%

Retrieval got roughly 4x slower in absolute terms and added about 55ms end to end. For an interactive agent where the LLM generation step is already 2 to 8 seconds, 55ms of extra retrieval latency is noise. Nobody perceives it. I paid 55ms and got 17 points of accuracy on the queries that decide whether the agent is trustworthy.

The trade would look different if I were serving retrieval as a standalone API at high QPS. Then 4x matters and I'd think about batching rerank calls or caching. For a single-user agentic workflow, it's the easiest 17 points I've ever bought.

One VRAM note, since the reranker shares a GPU with inference: bge-reranker-base at fp16 is small enough to coexist, but if your inference model already fills the card, you'll evict it or OOM. I keep the reranker pinned and size the LLM around it. On CPU it's viable too, around 200ms for 40 candidates on a modern core count, which is fine if you don't have spare VRAM.

Lessons learned

Look at the failures, not the score. The two hours I spent swapping embedding models were wasted because I optimized an aggregate instead of reading which questions I got wrong. The moment I bucketed failures by query type, the fix was obvious. Every point I gained after that came from a targeted change, not a bigger hammer.

Dense embeddings are bad at identifiers, and no embedding model fixes that. Ticket numbers, dates, SKUs, proper nouns, error codes. If your agent recalls anything with a specific token in it, you need a sparse retriever in the loop. This isn't a tuning problem. It's a property of how dense vectors compress meaning.

Reranking is the highest-use stage most people skip. It gets dismissed as "an extra model" and "more latency," and both are true and both are cheap. Splitting recall from precision is the core idea. Let the first stage be generous and dumb, let the second stage be strict and smart.

Build your own golden set. LoCoMo is a fine public benchmark, but the queries that matter for your agent are the ones your agent actually gets. I keep a small golden dataset of real recall queries and the chunk that should answer each one, and I run it on every change to the stack. Twenty good examples catch regressions that an aggregate score hides.

What surprised me most was how little the fancy part mattered relative to the boring part. I went in assuming the embedding model was the ceiling. The ceiling was a 30-year-old keyword algorithm and a small cross-encoder, both running on hardware I already had. This retrieval layer is the foundation the rest of the memory stack sits on, and it's the same layer I'd want solid before wiring agents together into anything multi-agent. Since the reranker runs locally, none of the recall traffic leaves the box, which keeps the whole thing aligned with a privacy-routed inference setup instead of shipping every query to a hosted reranking API.

If you're building agent memory or predictive systems on your own hardware and want a second set of eyes on the retrieval layer, that's the kind of work I do at GuatuLabs. The stack is simpler than the marketing around RAG makes it sound. Two retrievers, one reranker, and the discipline to measure what you actually broke.

RAG - Memory Systems

Ramya Perumal — Thu, 13 Aug 2026 18:08:59 +0000

We need memory to store the previous conversational history.

For example:

Previous question is:

User: File handling in Python

Assistant: Explain about file handling.

Next time, the user asks:

User: What are the modes in it?

The LLM should understand the context and then respond.

Below are the details that can be stored in the memory.

Past Conversations
User Preferences that we specify in the system prompts, e.g., JSON format
Past Decisions
Previous Tasks

Based on the details we are storing, we will choose between long-term or short-term memory.

Short-Term Memory

Redis, Valkey, Memcached, and Cosmos, which are cached databases, can be used for short-term memory to store the last few conversations or a summary.

We can set a general data invalidation rule to erase the content or use an LRU cache eviction policy, where the least recently used data will be erased from the database.

Long-Term Memory

Postgres, Pinecone, and MongoDB can be used for long-term memory to store long conversation histories.

How It Is Functioning

A summary of the entire conversation history will be stored in short-term memory to reduce latency whenever needed.

Episodic Memory

Episodic memory is a type of memory that stores specific events or experiences that happened in the past, usually together with information about what happened, when it happened, and the context surrounding it.

We can use either a short-term or long-term memory database depending on the use case. It is a kind of combination of short-term and long-term memory.

User:

I am planning a trip to Paris.

Agent:

How many days will you stay?

User:

5 days.

Later,

User:

Can you suggest an itinerary?

Episode 1

User wants to travel to Paris.

Trip duration: 5 days.

User previously mentioned:

Destination = Paris

Duration = 5 days

This information can be used to provide a more relevant response.

This helps the LLM understand what happened previously.

Semantic Memory

Semantic memory contains facts extracted from previous conversational history. Semantic memory is generally considered long-term memory.

Example

Paris is the capital of France.

The Louvre is a museum in Paris.

France uses the Euro.

That's general knowledge.

Sliding Window Memory

It is a short-term memory. Here, we store the last 3 to 4 conversations.

Redis or Valkey, like any cache memory, can be used.

Summarized Memory

Each and every time, the conversation, which includes the user query and response, will be summarized.

Even though token consumption during summarization is more, overall token consumption will be less.

It is a long-term memory.

Entity Fact Memory

This memory is used to store facts about a particular entity.

The difference between entity fact memory and semantic memory is that semantic memory is the broader category. Entity fact memory is one way of organizing and storing semantic knowledge about specific entities.

It can be used as either long-term or short-term memory.

Semantic Memory

Entity: Python

Python → is a programming language

Python → is used for AI

Python → supports object-oriented programming

Python → was created by Guido van Rossum

Entity Fact Memory

Entity: Alice

Alice → works at ABC Company

Alice → prefers Python

Alice → is working on Project X

It is not a good practice to store the entire conversation. We can make decisions based on the conversation and then store the relevant information. This is a good practice.

Building a Multi-Agent AI Pipeline That Ships: LangGraph, RAG, and Evals That Matter

manasviboineypally — Thu, 13 Aug 2026 17:14:25 +0000

I spent 18 days building an AI product that converts research papers into audience-tailored PowerPoint presentations. Not a toy — a real deployed thing at doc2slides on Railway that anyone can use.

The interesting parts weren't the "make it work" moments. They were the tradeoffs I had to make honestly, and the times I resisted the temptation to add a "clever" fix that would have made things worse.

This post is about those decisions.

What I built

Doc2Slides takes a PDF and produces a .pptx file tailored to four audiences:

Kid — fun analogies, simple words
Student — educational, terms defined
Engineer — technical depth, assumes domain knowledge
Executive — business focus, impact-oriented

The magic is that the same paper produces radically different output based on the audience. A compiler theory paper for a kid becomes "compilers are like magic helpers." The same paper for an executive becomes "advancing compiler technology with formal frameworks."

Code: github.com/manasviboineypally/doc2slides

The architecture: 5 agents in LangGraph

I built this as a multi-agent pipeline instead of one giant LLM prompt. Here's the flow:

PDF Upload
    ↓
Parser        → extracts sections + metadata
    ↓
Summarizer    → RAG-based section summarization
    ↓
Planner       → designs slide structure for audience
    ↓
Writer        → generates audience-adaptive slide content
    ↓
Builder       → produces editable .pptx file

Each agent is an independent node in a LangGraph state machine. They share a TypedDict state and read/write specific fields.

Here's what the graph definition actually looks like:

from langgraph.graph import StateGraph, END
from app.agents.state import AgentState
from app.agents.parser import parser_agent
from app.agents.summarizer import summarizer_agent
from app.agents.planner import planner_agent
from app.agents.writer import writer_agent
from app.agents.builder import builder_agent

def build_pipeline():
    graph = StateGraph(AgentState)

    graph.add_node("parser", parser_agent)
    graph.add_node("summarizer", summarizer_agent)
    graph.add_node("planner", planner_agent)
    graph.add_node("writer", writer_agent)
    graph.add_node("builder", builder_agent)

    graph.set_entry_point("parser")
    graph.add_edge("parser", "summarizer")
    graph.add_edge("summarizer", "planner")
    graph.add_edge("planner", "writer")
    graph.add_edge("writer", "builder")
    graph.add_edge("builder", END)

    return graph.compile()

Why LangGraph over a sequential chain? Adding a new agent is a 2-line change to the graph. In a sequential chain, adding a new step often means refactoring the previous ones. State-based multi-agent design scales better.

The interesting tradeoff #1: My RAG top-1 precision is 42%

I built an evaluation harness because I wanted to measure quality, not just claim it. Three eval types:

Parser evals — deterministic ground-truth assertions
RAG evals — top-K precision on hand-labeled query→section pairs
Summarizer evals — LLM-as-judge scoring faithfulness, completeness, clarity

The parser evals scored 100% (34/34 checks). The summarizer evals averaged 4.4/5.

But the RAG top-1 precision came in at 42%. Only 3 of 7 queries returned the correct section as the top result.

My first instinct: hide the number. Report top-3 (57%) instead.

What I did instead: publish both numbers and explain why.

Looking at the failures revealed a real limitation of RAG:

Query: "how does the genetic algorithm work?"
Expected section: Methodology
Actual top result: 3.6 Stopping Criteria (a subsection of methodology)

Genetic algorithms are discussed in 6 subsections (3.1 through 3.6). Vector search returns the highest-scoring chunk, not the highest-scoring section. For queries about broad topics, subsections often outrank the parent section because they mention the specific term more densely.

This is a known problem in RAG. Solutions include:

Hierarchical retrieval (search subsections, bubble to parent)
Query rewriting to be more specific
Retrieve top-K and let an LLM pick the right section

None of these are fixed today. But I know exactly what's broken and why — which is more useful than pretending it works.

Lesson: deterministic metrics beat vibes. Vibes let you convince yourself the AI is smart. Metrics tell you where it's dumb.

The interesting tradeoff #2: I refused to use word count as a proxy for content density

Users can request any number of slides between 3 and 50. When the paper's actual content density doesn't match the requested slide count, the LLM either pads shallow sections or compresses dense ones. This creates mild redundancy at high slide counts.

The obvious fix: allocate slides based on section word count. Long section = more slides. Short section = fewer slides.

I almost built this. Then I realized: word count is not content density.

Consider:

A 100-word section with 3 distinct concepts should get multiple slides
A 2000-word section rambling around one idea should get one slide

Word count would systematically reward verbose sections and penalize concise ones. That's not a fix — it's a bug with math.

What I did instead: documented the tradeoff and shipped without the heuristic. From the project's testing_notes.md:

Rejected quick fix: using section word count as a proxy for content density. Word count is not density — a short section may contain multiple distinct ideas while a long section may ramble around one.

Proper solution deferred: content-aware slide allocation with LLM judgment, verified by an evaluation harness that measures output quality against ground truth. Requires infrastructure work not appropriate for the initial version.

Lesson: the right answer to "should I add this heuristic?" is often "no." Heuristics feel like progress. Sometimes they're anti-progress dressed up as pragmatism.

The interesting tradeoff #3: SQLite dev → PostgreSQL prod is one variable

I built with local SQLite during development but deployed to Railway with PostgreSQL. The migration was one line:

# app/db/session.py
DATABASE_URL = os.getenv("DATABASE_URL")
engine = create_engine(DATABASE_URL, echo=False)

For local dev, .env has:

DATABASE_URL=sqlite:///./doc2slides.db

For Railway, the environment variable is:

DATABASE_URL=postgresql+psycopg2://postgres:xxx@host:5432/railway

Nothing else changes. SQLAlchemy models are backend-agnostic.

This is boring engineering. But boring engineering is what lets you sleep at night. When someone asks "how do you handle database migrations?" the answer isn't a clever hack — it's "environment-driven configuration and a repository pattern."

The stack

Layer	Choice	Why
Language	Python 3.13	AI ecosystem
API	FastAPI	Async support, auto Swagger docs
Orchestration	LangGraph	State-based multi-agent
LLM	OpenAI GPT-4o-mini	Cheap enough for iteration, smart enough for structured output
Vector DB	ChromaDB	Local, no cloud dependency
Structured output	JSON mode + Pydantic	Two-layer validation
Database	SQLAlchemy + PostgreSQL	Env-driven, portable
Frontend	Vanilla HTML/CSS/JS	No build step, portable
Deployment	Railway	GitHub CI/CD, managed Postgres

The frontend is worth calling out. I used no framework — just HTML, CSS, and vanilla JavaScript in ~500 lines. Zero build step. Anyone can clone the repo, open the file, and understand it in 5 minutes.

For an MVP, that's a feature, not a limitation.

What I didn't build (and why that's OK)

Skipped:

User authentication
Multi-tenant workspaces
Custom presentation templates
Streaming responses
Job queue with Celery/Redis

Why: MVP. Every feature has a cost. Shipping the core value (PDF → audience-tailored slides) matters more than shipping every possible feature.

For a portfolio project, "I could have added X but chose not to for these reasons" is a stronger answer than "I added X poorly."

Lessons I'd tell my past self

1. Build evals before optimizing. I built the pipeline first, then evals. If I had built evals first, I would have known earlier that my RAG had issues. Now I have to make eval-driven improvements Week 3.

2. Resist heuristics. Every time I thought "this is a quick fix," it was actually a technical debt I was about to bake in. Word count as density. Silent AI slide count overrides. Boolean status flags instead of proper enums.

3. Deploy early. I deployed on Day 16 of 18. I should have deployed on Day 8. Deployment reveals real bugs — environment variable typos, missing dependencies, hardcoded localhost URLs. The sooner you find them, the cheaper they are.

4. Document tradeoffs, not features. Anyone can read code to know what it does. Almost no one leaves notes on why a design choice was made. My testing_notes.md file is where most of the actual engineering thinking lives.

What's next

The project is live, but not "done." Future work:

Content-aware slide count (with an eval harness measuring output quality)
Multi-language support for input PDFs
Custom presentation templates
Fix RAG for hierarchical sections (subsection → parent bubbling)

If you want to try Doc2Slides yourself:

Live demo: web-production-6eded.up.railway.app
Code: github.com/manasviboineypally/doc2slides
60-second video demo: Loom link

Upload any PDF, pick your audience, get back a deck. Same paper, radically different output depending on who you say you're presenting to.

Author: Manasvi Boineypally — GitHub · LinkedIn

Hybrid Retrieval v2: Qwen Embeddings, BM25, and RRF with a FastEmbed Reranker

Guatu — Thu, 13 Aug 2026 16:15:48 +0000

A query for ndots:5 against my wiki index used to return the article that exists specifically to explain ndots:5 at position seven. Ahead of it sat three general DNS articles, two Kubernetes networking posts, and something about service discovery. My embedding model understood the topic perfectly and had no idea that the literal string mattered.

That is the dense retrieval failure mode in one sentence. Semantic similarity is a fuzzy match by design, and a fuzzy match is exactly wrong when the user typed an exact identifier. Across a fixed 10-query eval set of the things I actually search for (config keys, error strings, CLI flags), dense-only retrieval put the correct article in the top 3 for 5 of them. Hybrid retrieval with a reranker on the same 10 queries hits 8.

Who should care

If you run a retrieval layer for agents and your corpus is technical documentation, code, runbooks, or session memories, you have this problem whether or not you've measured it. Technical corpora are full of tokens that carry near-zero semantic weight and near-total discriminative weight: max_cstate, Modifier.IDF, ErrImagePull, a CVE number, a Helm value path. An embedding model compresses all of those into a vector where they barely register against the surrounding prose.

My first instinct was to reach for a better embedding model. That instinct was wrong, and the reason it was wrong is the most useful thing in this post.

What I tried first

Bigger embeddings. Swapping to a larger dense model moved my eval by roughly one query out of ten, and cost more VRAM plus more latency per ingest batch. Larger dense models are better at nuance in prose. None of them are better at treating ndots:5 as an atomic symbol, because none of them are trained to. Adding dimensions does not create a keyword index.

Query expansion with an LLM. Rewrite the user query into three paraphrases, embed all three, union the results. This helped on vague questions and actively hurt on precise ones, because the paraphrases diluted the exact term being searched for. It also adds an LLM round trip to every retrieval call, which turns a 40ms operation into a 900ms one and makes results non-deterministic between runs. Acceptable for a chat UI. Bad for an agent that retrieves twenty times inside a single task.

A cross-encoder reranker served through Ollama. This one is worth writing down, because the failure was silent and cost the most time.

My plan was reasonable: over-retrieve 20 candidates from dense search, then rerank with a cross-encoder that sees query and document together. Ollama was already running in the cluster, GGUF conversions of popular rerankers exist on Hugging Face, so pull one, hit the API, sort by score.

Scores came back as numbers. They were garbage. Not obviously broken (no errors, no NaNs), just weakly correlated with relevance. Sometimes the reranked order was measurably worse than the pre-rerank order, which is an impressive achievement for a component whose entire job is to improve ordering.

Here's the mechanism. A cross-encoder reranker is a sequence-classification model: an encoder backbone plus a trained classification head that emits a single relevance logit. Convert that to GGUF, serve it through a runtime built for causal LM generation and embedding extraction, and the classification head is usually not part of the picture. What comes back is a pooled hidden state, or a logit from a head that was never trained for relevance ranking, wrapped in a response shape identical to a real score. Nothing warns you. Your pipeline runs, your latency budget looks fine, and retrieval quality quietly rots.

Generalizing: when a model's output is a scalar, you cannot tell by inspection whether it's the right scalar. Test rerankers against a fixed query set with known-correct answers before you wire them in, not after you've shipped them.

The actual solution

Four pieces. A Qdrant collection with two named vector spaces, dense embeddings from qwen3-embedding:0.6b, sparse BM25 vectors, and a cross-encoder reranker running on ONNX through FastEmbed. No GPU is involved in the reranking stage at all.

1. The collection schema

Dense and sparse vectors live on the same point. One document, one ID, two vector representations, one payload. That detail matters more than it looks: split them across two collections and you get two ingest paths that drift out of sync, and you'll find out about the drift during a retrieval failure at the worst possible moment.

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

client.create_collection(
    collection_name="wiki_index_v2",
    vectors_config={
        "dense": models.VectorParams(
            size=1024,                      # qwen3-embedding:0.6b
            distance=models.Distance.COSINE,
        ),
    },
    sparse_vectors_config={
        "bm25": models.SparseVectorParams(
            # Qdrant applies IDF server-side against the live corpus
            modifier=models.Modifier.IDF,
        ),
    },
)

modifier=models.Modifier.IDF is the line people skip. FastEmbed's BM25 produces the term-frequency component client-side, but inverse document frequency depends on the entire corpus, and your corpus changes on every ingest. Setting the modifier makes Qdrant compute IDF at query time from current collection statistics. Leave it out and you're doing raw term-frequency matching, which over-weights common tokens and makes the sparse leg noticeably worse: in my eval it cost two of the eight top-3 hits.

You cannot add the modifier later without recreating the collection. Set it on day one.

2. Ingest both vectors in one upsert

from fastembed import SparseTextEmbedding
import ollama

bm25 = SparseTextEmbedding("Qdrant/bm25")

def embed_dense(text: str) -> list[float]:
    return ollama.embed(model="qwen3-embedding:0.6b", input=text)["embeddings"][0]

def to_point(doc_id: int, text: str, payload: dict) -> models.PointStruct:
    sparse = next(bm25.embed(text))
    return models.PointStruct(
        id=doc_id,
        vector={
            "dense": embed_dense(text),
            "bm25": models.SparseVector(
                indices=sparse.indices.tolist(),
                values=sparse.values.tolist(),
            ),
        },
        payload=payload,
    )

SparseTextEmbedding("Qdrant/bm25") is not a neural model. It's a tokenizer plus stemming plus stopword removal, running in a few hundred microseconds per document. The cost of the sparse leg is rounding error next to the dense embedding call.

One migration note. Moving a few hundred wiki articles and roughly twice as many session memories into the new schema meant re-embedding everything, and re-embedding is exactly where payloads get quietly dropped. My rule: read the full point from the old collection, carry the payload dict forward untouched, and diff payload key sets between source and destination when the run finishes. If a key existed on 300 points before and 280 after, you want a failing assertion, not a shrug. This is the same class of problem I wrote about in Silent Drift: count-based checks pass while content quietly diverges.

3. Query both legs and fuse with RRF

Qdrant does the fusion server-side through prefetch, which saves a round trip and keeps the client dumb:

sparse_q = next(bm25.query_embed(user_query))

results = client.query_points(
    collection_name="wiki_index_v2",
    prefetch=[
        models.Prefetch(query=embed_dense(user_query), using="dense", limit=20),
        models.Prefetch(
            query=models.SparseVector(
                indices=sparse_q.indices.tolist(),
                values=sparse_q.values.tolist(),
            ),
            using="bm25",
            limit=20,
        ),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=20,
    with_payload=True,
).points

Use bm25.query_embed() for queries, not bm25.embed(). Query embedding skips the term-frequency weighting that only makes sense for documents. Mixing them up produces results that look plausible and rank badly.

The fusion itself is about six lines, and it's worth seeing them written out even if Qdrant runs it for you:

def rrf(ranked_lists, k=60):
    scores = {}
    for lst in ranked_lists:                 # each list is [doc_id, ...] by rank
        for rank, doc_id in enumerate(lst, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

No score normalization. No tunable alpha weighting dense against sparse. Rank position is the only input.

4. Rerank on CPU with FastEmbed

from fastembed.rerank.cross_encoder import TextCrossEncoder

reranker = TextCrossEncoder(model_name="jinaai/jina-reranker-v2-base-multilingual")

def rerank(query: str, candidates, top_n: int = 5):
    docs = [c.payload["text"] for c in candidates]
    scores = list(reranker.rerank(query, docs))   # batched ONNX inference
    ranked = sorted(zip(scores, candidates), key=lambda p: p[0], reverse=True)
    return [c for _, c in ranked[:top_n]]

That's the whole reranking stage. FastEmbed ships the ONNX export with the classification head intact, downloads it on first use, and runs it through ONNX Runtime on CPU at roughly 38ms per query-document pair. Twenty candidates batched lands under half a second on a few cores, and it needs no GPU, no separate inference server, and no model-serving deployment to keep alive.

Compare that to the GGUF path: same nominal model, wrong head, silently meaningless scores.

What changed on the eval set

Configuration	Correct doc in top 3	Median latency
Dense only (1024d)	5 / 10	~45 ms
BM25 only	6 / 10	~12 ms
Dense + BM25, RRF	7 / 10	~55 ms
Dense + BM25, RRF, reranked	8 / 10	~480 ms

BM25 alone beating dense alone surprised me. It also makes sense in hindsight: over half my eval queries were literal strings copied out of a config file or an error log, which is BM25's home turf and dense retrieval's blind spot.

Here's the shape of a query that used to fail:

query: "ndots:5"

dense-only ranking:
  1. Kubernetes Service Discovery Patterns          (cos 0.612)
  2. DNS Failover With Two Upstreams                (cos 0.598)
  3. CoreDNS Tuning Notes                           (cos 0.591)
  ...
  7. Wildcard DNS + ndots:5: The TLS Nightmare      (cos 0.544)

hybrid + rerank:
  1. Wildcard DNS + ndots:5: The TLS Nightmare      (rerank  6.81)
  2. CoreDNS Tuning Notes                           (rerank  1.24)
  3. Kubernetes Service Discovery Patterns          (rerank  0.37)

BM25 pulled the right article into the candidate pool at sparse rank 1. RRF pushed it to fused rank 2. The cross-encoder, which actually reads the query and the document together, put it first with a score nearly six times the runner-up.

Why it works

Three separate mechanisms are doing distinct jobs, and it's worth being precise about which does what.

Sparse retrieval indexes symbols, not meaning. BM25 scores a document on term frequency scaled by inverse document frequency, with length normalization. ndots appears in one article out of several hundred, so its IDF is enormous and any document containing it rockets to the top. An embedding model does the opposite: it maps rare tokens into a region of vector space defined by their context, which is exactly the behavior you want for synonyms and exactly the behavior you don't want for identifiers. Dense and sparse are not competing implementations of retrieval. They index different properties of the same text.

RRF fuses ranks because scores are incomparable. Cosine similarity lives in [-1, 1] and clusters hard around 0.5 to 0.7 for a technical corpus. BM25 scores are unbounded and depend on corpus size, document length, and term rarity. Normalizing them onto a shared scale requires assumptions about their distributions that break whenever the corpus changes. Reciprocal rank fusion sidesteps the problem: it throws the scores away and keeps only the ordering, then sums 1/(k + rank) across both lists. The k=60 constant flattens the curve near the top so that rank 1 versus rank 2 isn't a cliff, which means a document ranked 3rd by both retrievers can outrank a document ranked 1st by one and 40th by the other. Consensus wins over one confident vote, and that's the behavior you want when one leg is guessing.

Cross-encoders can do what bi-encoders structurally cannot. Your embedding model is a bi-encoder: query and document are encoded independently, never seeing each other, and compared by cosine distance at the end. That independence is what makes vector search fast, because you precompute every document embedding once. It also means the model never gets to ask "does this specific document answer this specific question." A cross-encoder concatenates query and document into one sequence and runs full attention across both, so query tokens attend directly to document tokens. Far more accurate, and far too slow to run against your whole corpus. Which is precisely why the architecture is retrieve-then-rerank: cheap methods cut several hundred documents down to 20, the expensive method orders those 20.

That layering also explains why over-retrieval depth matters. Reranking cannot recover a document that never entered the candidate pool. If your prefetch limit is 5, the reranker is just reordering five things, and your recall ceiling is whatever RRF handed it. Twenty per leg is where my eval stopped improving; going to 50 added latency and no additional top-3 hits.

Lessons learned

Test the reranker in isolation before trusting it. Build a fixture of 10 to 20 query-document pairs where you know the ranking by hand, score them, and check the correlation. That test takes an hour and would have saved me the entire GGUF detour. It also catches the subtler failure where a reranker works fine on prose and falls apart on code blocks.

ONNX over GGUF for anything with a classification head. GGUF is a format built around generative decoder inference. Cross-encoders, classifiers, and any model whose value lives in a trained head on top of the backbone should go through ONNX Runtime, where the head is exported with the graph. FastEmbed makes that a one-liner, and running it on CPU means the reranker isn't competing with your LLM for VRAM. I don't need an accelerator to serve retrieval, which matters when the GPU is busy doing actual inference.

Set Modifier.IDF at creation time. I'd rather see this documented in bold in every hybrid search tutorial. Missing it does not raise an error, it just makes the sparse leg mediocre in a way you'll blame on BM25 rather than on your config.

Measure with your queries, not a benchmark. MTEB scores told me nothing useful about whether retrieval would find the article about a specific kernel parameter. A hand-built eval of 10 real queries with known-correct answers told me everything, and it's small enough to rerun in under a minute after any config change. Keyword-in-top-3 is a crude metric and a good one, because it maps directly to what the agent experiences: the right context is in the window or it isn't.

Retrieval precision is upstream of everything else in an agent stack. Better memory decay policies, better tool descriptions, better prompts, none of them compensate for handing the model the wrong three documents. I've come to treat the retrieval layer the way I treat storage: unglamorous, load-bearing, and worth over-engineering slightly. It sits underneath the memory architecture and the decay policy, and it's the layer that determines whether the rest of the agent architecture has anything worthwhile to reason over. If you're building this kind of pipeline for something that has to work on a schedule rather than on a weekend, that's the sort of work I do.

What surprised me: the reranker mattered less than adding BM25. Fusion alone took the eval from 5/10 to 7/10; the cross-encoder added the eighth. I'd assumed the fancy neural component would carry the improvement, and instead the win came from a 1994-vintage ranking function that runs in twelve milliseconds and has no parameters to train. The old algorithm knows something the new model doesn't, which is that sometimes the user meant the exact characters they typed.

Enterprise AI Is More Than RAG: The Three Context Layers (2026)

Alex Pechenizkiy — Thu, 13 Aug 2026 14:50:52 +0000

Most enterprise AI architectures fail because they treat all enterprise knowledge as searchable documents. They wrap a chatbot around an indexed wiki, call it "AI on our data," and discover at audit time that compliance can't sign off, that pricing answers are wrong, and that the agent has no idea who's actually logged in.

Enterprise AI is not a model problem. It is a distributed systems engineering problem: probabilistic orchestration of multiple state, authority, and governance domains, each with different consistency and freshness requirements. Treat it that way and the architecture clarifies. Treat it as "prompt + retrieval" and it breaks in production.

We call this Enterprise Context Architecture (ECA), and to be precise, it is broader than what vendors mean by a "context layer." A context layer is a product; ECA is the architectural discipline of orchestrating context, authority, permissions, and execution across distributed systems. The product is a component. The discipline is the system.

ECA is not a new discipline. It is the application of existing distributed-systems patterns to the agent-architecture domain: identity propagation, service mediation, enterprise integration patterns (EIP), event-driven architecture, materialized views, retrieval orchestration. We use the name "ECA" as a working shorthand for that combined application, not as a claim to have invented these primitives. If you have built distributed systems for two decades, most of this will sound familiar; the novelty is applying that discipline rigorously to systems that include a probabilistic LLM in the call path.

There are three real levels of enterprise AI, in our read, and the differences are not depth of customization. They are differences in where the answer comes from and whether you can prove it. And the apex pattern (Level 3 + fine-tune) is not the whole story: real production systems retrieve unstructured knowledge (RAG), call structured systems of record (CRM, ERP, Dataverse), and inject runtime context (the logged-in user, the active workflow, the current session). RAG is one knowledge layer. There are three.

Microsoft's architectural direction toward Level 3 (agentic retrieval, groundedness evaluators, Foundry IQ) is now clear. In our experience, operational maturity is still uneven: production pilots routinely hit retrieval-quality cliffs, evaluator drift between releases, and security-trimming gaps that only surface under real load. Read this article as a workload-decision frame, not a green-light to assume the Level 3 stack is plug-and-play.

What is 'context'? A working definition for ECA

Context is the operationally relevant information required for an AI system to make correct decisions within a bounded enterprise workflow. Concretely, ECA distinguishes five context types:

Informational context: the knowledge the answer should be grounded in (policies, manuals, past work product). Lives in unstructured corpora.

Operational context: the authoritative state of the business at this moment (orders, claims, balances, deal status). Lives in systems of record.

Runtime context: who is asking right now, what tenant, what workflow step, what tools are active. Lives in the request envelope.

Authorization context: what the asking user is permitted to see, modify, or trigger. Propagates from identity through every downstream call.

Environmental context: the deployment realities (region, GA-vs-preview status, current rate-limit posture, observability surface). Lives in the runtime configuration.

A correct enterprise AI response is the synthesis of all five. Skip any one of them and the agent answers confidently in a way you cannot defend.

TL;DR + the Monday move

Three levels of enterprise AI, ordered by traceability:

Out-of-Box Copilot: Microsoft 365 Copilot, ChatGPT Enterprise, generic SaaS. Reads public web + limited org content via Microsoft Graph. Few citations to your private knowledge. Productivity gain, weak audit trail.

Configured Agents: Copilot Studio declarative agents with knowledge sources + connectors. Pulls from approved org data. Some grounding, partial traceability. Good for departmental use.

Tailored RAG / Agentic Retrieval: Foundry agents over indexed enterprise content with agentic retrieval. The retrieval response carries structured grounding data, citations, and execution metadata; how faithfully the model uses them is what groundedness evaluators measure. Auditable when paired with the operational discipline below. The architecture your compliance team can approve, not a substitute for the governance program.

Your Monday move: if your AI investment looks like Level 1 or Level 2 but the workload genuinely needs audit defensibility (proposals, claims, compliance, regulatory response, financial reporting), the gap between current state and Level 3 is the gap between "productivity demo" and "production system." Pick one workload that fails the audit test at Level 2 today and scope what Level 3 looks like for it.

The strategic insight: the question is not "which AI tool do we buy?" It is "for each workload, which level is defensible?" The answer is workload-specific, and most enterprises are running Level 1 work at Level 1, Level 2 work at Level 2, and Level 3 work at Level 1 hoping it scales.

Enterprise Context Architecture: five context types inside one permission envelope. RAG retrieves evidence; operational systems establish authoritative state.

What Are the Three Levels of Enterprise AI?

The three levels are out-of-box Copilot (Level 1), configured agents over approved sources (Level 2), and tailored RAG with agentic retrieval over an indexed enterprise corpus (Level 3). The distinguishing axis is not "how much AI" but how traceable each answer is back to the source it came from. In our reading of Microsoft's 2026 surface stack, the product taxonomy maps cleanly to these three tiers. Pick your level per workload, not per organization.

Level	Microsoft surface	Customization scope	Where the answer comes from	Traceability
1. Out-of-Box	M365 Copilot Chat, ChatGPT Enterprise, generic SaaS AI	None (configuration only)	Public web + limited Microsoft Graph (your email, Teams, SharePoint with access checks)	Weak: Graph results are cited but the LLM blends them with web training data
2. Configured Agents	Copilot Studio declarative agents + Power Platform connectors + Agent Builder	Knowledge sources, system prompts, connector access, topics	Approved data sources via connectors + uploaded knowledge files + Microsoft Graph	Partial: connector-source citations when retrieval triggered; topical drift outside retrieval scope
3. Tailored RAG / Agentic Retrieval	Foundry agents + Azure AI Search agentic retrieval + Foundry IQ + custom RAG pipelines	Indexed enterprise corpus, retrieval design, groundedness evaluators, deterministic tool wiring	Curated and indexed enterprise knowledge corpus, retrieved per query with semantic ranking	Full: agentic retrieval returns citations + grounding data + activity arrays per response

This is not a maturity ladder where everyone needs to climb. It is a decision frame. The right level depends on what the workload actually demands.

A note on portability. The "Microsoft surface" column names the canonical 2026 stack we work in most often, but the level pattern is vendor-portable. AWS Bedrock Knowledge Bases and Bedrock Agents map to Levels 2-3 with their own response shape. Google Vertex AI Search + Vertex AI Agent Builder cover similar ground. Databricks Mosaic AI is a Level 3 substrate with its own evaluator surface. The taxonomy is about traceability and workload fit, not Microsoft specifically. If you are not building on Microsoft, translate the pattern, not the product names.

Why Does Traceability Define Level 3?

Level 3 is defined by answers that come with their sources attached, in a format your audit committee accepts, not by "smarter answers." Microsoft's agentic retrieval response carries structured grounding data, citations, and execution metadata; we map that surface in pseudocode below as content, references, and activity arrays.

Microsoft's agentic retrieval returns structured response data with three logical layers (we use these names for the pseudocode contract):

Content: the synthesized answer, grounded in retrieved passages
References: the source documents or chunks that grounded the answer, with citable URIs
Activity: the retrieval plan, subqueries, ranking scores, and token-cost trace

A compliance lead reading a Level 3 response can verify each claim against its source, see what queries the system ran, and inspect the ranking that selected those passages over others. None of that exists at Level 1. Some of it exists at Level 2, inconsistently.

This is what makes Level 3 the bar for regulated workloads. Microsoft's RAG evaluators measure groundedness (does the response cite only the provided context, or does it fabricate?) and response_completeness (did it cover all critical information from ground truth?) as first-class metrics. You can score a RAG system on those metrics the way you score functional tests, with the caveat that both are probabilistic and gameable with prompt rewrites: see the operational-reality callout further down. You cannot score at all at Level 1.

The trade-off, honestly: Level 3 is slower to build and harder to maintain. You own the data preparation, retrieval design, evaluation harness, and operational discipline. Level 1 is fast because someone else owns all that on your behalf, and gives you correspondingly less.

Level 3 in Practice: Three Knowledge Layers (RAG Is Only One of Them)

What "three context layers, one permission envelope" looks like at a customer-service counter. The robot reading from the policy book is RAG. The empty-handed shrugging robot is the missing operational-systems layer. The badge-holder is runtime context. Skip any one of them and the answer the customer gets is confident but indefensible.

The biggest misread of "tailored AI" is treating it as a single retrieval problem. RAG retrieves relevant information. Operational systems establish authoritative state. Policy PDFs in your indexed corpus are guidance. The CRM is the authoritative state of who the customer is, what they bought, and what's open. The active session is the authoritative state of who is asking right now and what they're authorized to see. A Level 3 architecture that only retrieves documents will hallucinate the moment a user asks "where is my order?" or "is this customer past due?"

In production, Level 3 spans three knowledge layers and the orchestrator agent routes across all of them in a single permission envelope. Architect for all three.

Enterprise AI is a context orchestration problem. Three layers, one permission envelope, one coherent answer.

Retrieval, Grounding, Authority: three distinct functions

A common confusion in production is treating retrieval, grounding, and authority as the same problem. They are not. Each has a different function, a different failure mode, and a different fix.

Function	Purpose	Failure mode if confused
Retrieval	Finds potentially relevant information from a corpus	Confuse with authority and you cite a policy passage as the source of truth for current order status
Grounding	Constrains generation using retrieved evidence; evaluator-measurable	Confuse with retrieval and you assume the model uses what it retrieved (groundedness evaluator exists because it often doesn't)
Authority	Determines what is operationally true: the current state of the business	Confuse with retrieval and your agent confidently produces stale or wrong answers about live state

RAG is retrieval. Groundedness evaluators are grounding. Systems of record are authority. Wire all three; treat them differently.

A nuance worth naming. The boundary between retrieval and authority is not always sharp in practice. Cached operational snapshots, event-stream projections, materialized views from systems of record, and hybrid retrieval that fuses indexed and live data all blur the line for legitimate latency or cost reasons. The discipline is not "never put operational data in an index"; it is "when you do, label what was authoritative at indexing time vs. what is authoritative now, and decide which the workload requires." A cached order status from 30 seconds ago is fine for "show me roughly where my order is." It is not fine for "process this refund." The architect owns that distinction per workload.

A production enterprise AI system must inherit source-system permissions across all three layers. The authenticated user's identity propagates from the request, through the orchestrator, through every retrieval call, every operational API invocation, and every runtime-context lookup. Skip this at any layer and the agent leaks data across users, tenants, or roles. The diagram above shows the propagation as the dashed yellow envelope; in code it's the permission token threaded through every tool call.

Knowledge layer	Example	Authoritative source	Best mechanism
Unstructured knowledge	policies, manuals, past proposals, case studies, regulatory docs	indexed corpus (curated and chunked)	RAG / agentic retrieval with semantic ranking + citation evidence
Structured operational state	orders, claims, invoices, customer records, inventory, deal status	systems of record (CRM, ERP, Dataverse, billing system)	deterministic API / SQL calls via agent tools; never paraphrased by the LLM
Runtime / user context	authenticated user, active workflow step, current session, tool state, recent actions	the request itself + session store	context injection at agent invocation; permissions trim every downstream call

RAG retrieves evidence. It does not establish operational truth.

Policy PDFs are guidance. ERP / CRM / Dataverse / line-of-business systems are authoritative state. Confusing the two is where many production Level 3 systems break: the agent confidently cites a policy passage about "standard refund terms" while the actual order in the CRM is past the refund window. Both answers feel grounded; only one is true. Wire structured calls to systems of record as a peer to RAG, not as an afterthought.

Runtime context goes deeper than "the user"

The runtime layer is the most under-engineered of the three in pilots we have reviewed. Beyond the authenticated user identity, a production-grade runtime context envelope carries:

Workflow state: which step of a multi-step process the user is in, what's been approved, what's pending, what's been rolled back.
Ephemeral memory: the last few turns of conversation, any clarifications the user supplied, any preferences the agent has been told to remember for this session only.
Active tool state: which tools the agent has invoked this session, what they returned, what's been retried, what's been escalated.
Approval state: which actions are gated on human approval, which approvals are outstanding, who can grant them.
Human escalation: the path back to a person when the agent can't or shouldn't act autonomously.
Multi-agent coordination state: in multi-agent topologies, which agent is the current owner of the task and where its hand-off boundary is.
Tenant isolation: which tenant the request belongs to, with the corresponding scoping on every downstream call.

Skip any of these and the agent has a partial picture of "who is asking and what they can do." That's where the loud production failures live.

Bounded autonomy: the operational constraint architects own

Agents can be reliable or they can be autonomous; pick one and design for it explicitly. Bounded autonomy is the architectural pattern that lets an agent act without permission to act everywhere:

Scoped actions: every tool the agent can invoke is named, parameter-typed, and explicitly registered. No "freeform" tool calling.
Typed action contracts: each tool defines its input schema, its output schema, and its side-effect class (read-only, mutating, externally observable). The orchestrator validates against these contracts before invocation.
Permission-aware execution: every invocation is wrapped in the asking user's permission envelope. The tool itself enforces; the agent does not.
Human approval boundaries: actions above a configurable risk class route to human approval before execution. The agent proposes; a human commits.
Blast-radius containment: actions that touch external systems (sending email, posting to a channel, updating a customer record) are gated behind explicit per-action allow-lists, rate limits, and rollback paths.

The architect's job is to define the autonomy boundary explicitly. An "agentic" system without these boundaries works in pilots and surprises its operators in production; whether the surprise is mild or severe is a matter of luck rather than design.

A worked walkthrough: "Where is my order?"

The clearest test of whether your Level 3 architecture is real or theatre is this query. A pure-RAG agent fails on it. A correctly-layered agent answers in seconds with audit-defensible provenance.

Runtime context layer resolves the asker. The agent receives the authenticated user identity, their tenant, their role, and any active conversation context. Without this, the agent has no idea which "my" the question refers to.
Operational data layer queries the order system (or CRM, or ERP, depending on the workload) via a deterministic API call scoped to the authenticated user's permissions. Returns the actual order record: status, ship date, tracking number, current location. Citation: OrderID-2026-074118, fetched at timestamp X via the order API, RBAC-scoped to user Y.
Knowledge retrieval layer retrieves the shipping policy that explains what "in transit, regional sortation hub" means in plain language. Citation: Policy-Shipping-v3.2, indexed copy of the customer-facing shipping policy.
Agent synthesis combines the three: "Your order #074118 is at the regional sortation hub in Memphis; per our shipping policy, that's typically the last hop before final delivery, usually 24-48 hours away." Each clause cites its source.

Three different authoritative sources. Three different access patterns. Three different governance regimes. One coherent answer.

This is what "agent orchestrates; deterministic layer calculates" actually looks like at production scale. The agent is the conductor. The systems of record carry truth. RAG carries explanatory context. Runtime injection carries identity.

Failure modes by layer

Each layer fails differently. The architect's job is to know which failure looks like which.

Layer	Typical failure mode	What the user sees	What the architect fixes
Knowledge retrieval (RAG)	Stale policies, missing recent updates, security-trimming gap, retrieval drift	Confidently-cited but outdated information	Index refresh cadence, security trimming against full identity matrix, groundedness evaluator gating
Operational data (systems of record)	API latency, expired permissions, mis-scoped tool call, schema drift	Empty result, permission error, or wrong customer's data	Tool-permission contracts, retry/timeout discipline, schema versioning in the agent tool wiring
Runtime context	Wrong authenticated identity, stale session, missing tenant scoping, leaked context across users	Other-user's data surfaces, or 'who am I?' fails	Identity propagation tests, session isolation, context-scrubbing between turns, audit of context-passing code paths

Enterprise readers calibrate against failure boundaries more than conceptual purity. The above table is the one to screenshot when scoping a Level 3 build.

Observability: the architect's evaluation surface

If you cannot replay what the agent did, you cannot debug it, defend it, or evolve it. ECA treats observability as a first-class architectural concern, not a logging afterthought. The instrumentation surface a Level 3 system needs:

Traces: every retrieval call, every API call, every runtime-context lookup, every LLM call, with timing, token cost, return shape, and parent-child relationships across the whole turn.
Replayability: any past request can be re-run against the current corpus, current evaluator suite, and current tool wiring to detect regression.
Context lineage: for every claim in a response, which retrieved passage / which API response / which runtime variable contributed.
Prompt provenance: the exact prompt template version, the exact retrieved context, the exact runtime envelope as the model saw it.
Tool-call telemetry: per-tool success rate, latency p50/p95, error class distribution.
Evaluation pipelines: groundedness, response-completeness, and task-specific evaluators running on a held-out set on every change, threshold-gating deploys.

This is what makes a Level 3 system defensible across releases. Without it, version drift is silent.

The user-facing question is usually phrased "should we train AI on our data?" The technical answer almost always means RAG, not fine-tuning. Synthesizing Microsoft's RAG framing with the practitioner trade-offs we see in enterprise pilots:

Choose RAG when	Choose Fine-Tuning when
Dynamic or changing content (org knowledge that updates)	Stable content that doesn't need constant updates
Wide topic coverage across many domains	Task-specific performance on a narrow domain
Limited training data or compute budget	Lots of domain data + compute available
Need fresh answers, current information	Want consistent tone, style, format
Need source citations for audit	Citations not required

For most enterprise scenarios where the goal is "answers grounded in our company's knowledge," RAG is the answer for grounding, with fine-tuning reserved for tone, voice, or narrow task automation. You can stack them, which is the apex of Level 3 and the subject of the next section.

When Fine-Tuning AND RAG Both Belong: The Tailored-Plus-Traceable Pattern

The "RAG vs fine-tuning" framing is convenient but wrong for the most demanding workloads. The right framing for the apex of Level 3 is fine-tuning AND RAG, layered. Microsoft explicitly supports this stack: combining fine-tuning with retrieval improves a model's ability to integrate external knowledge and filter out irrelevant information, per the Foundry fine-tuning guidance.

Why fine-tuning alone fails the traceability test. A fine-tuned model has your data baked into its weights. It has internalized your patterns, your voice, your terminology, your historical decisions. Ask it where the answer came from and it cannot tell you, because the answer came from gradient updates on training data, not from a retrievable passage. There is no citation. There is no audit trail. There is no way to prove the model is grounded in current truth versus a six-month-old training cut. If your workload needs a source link next to every claim, fine-tuning alone moves you backward on the traceability axis, not forward. You traded the public web for your weights. Both are opaque to an auditor.

Why RAG alone leaves voice on the table. A pure-RAG agent retrieves the right passages but synthesizes them in whatever voice the base model defaults to. Fine for internal Q&A. Not fine for executive communications, brand-critical customer-facing content, regulated voice (legal opinions, regulatory filings), or anything where consistency of tone is a quality dimension. For those workloads you need the model trained on your voice while it is grounded in your facts.

The combined pattern, ordered. Microsoft's RAG-or-fine-tuning architectural guide and the Foundry fine-tuning considerations treat these as composable. In practice the architecture sequences cleanly:

Supervised fine-tuning (SFT) on voice / format / domain vocabulary. Train a LoRA fine-tune on representative examples that show the desired tone, formatting, and how the model should integrate retrieved citations. Training data is voice-and-format pairs, not factual knowledge. Foundry implements supervised fine-tuning using LoRA (low-rank adaptation), a parameter-efficient technique that is cheaper than full-weight retraining.
Optional preference fine-tuning (DPO) on edge cases. Where SFT plus prompting leaves rough edges, Direct Preference Optimization lets you train on preferred-versus-rejected response pairs. Useful for "this response is correct but the wrong shape" cases.
RAG / agentic retrieval layer on top. The fine-tuned model is now your base. Wrap it with the same agentic retrieval architecture as Level 3 RAG-only: indexed corpus, schema-aware subqueries, grounded synthesis, structured citations on every response.
Groundedness evaluators run on the combined output. The eval target is whether the fine-tuned-plus-retrieving system cites the right passages and reflects current truth, not whether the fine-tune memorized historical patterns.

Fine-tune + RAG combined pattern. Training-time fine-tune embeds voice. Inference-time RAG layer carries citations.

Why ordering matters. Fine-tune first, RAG second. If you RAG over a base model and then fine-tune that result, you collapse the architecture: the fine-tune memorizes whatever you retrieved at training time, freezing answers that go stale. Fine-tune for the layer that should stay stable (voice, format, instruction-following on retrieved context); RAG for the layer that should stay fresh (facts, citations, current state of the corpus).

When this pattern is right. Three workload types justify the cost:

Regulated customer-facing communications (insurance claims responses, healthcare benefits explanations, financial-services advice memos): voice consistency is a regulatory expectation; traceability is a regulatory requirement. Both matter.
Executive and corporate communications (annual reports, investor updates, official corporate responses): the C-suite voice must be consistent across documents; every factual claim must trace to underlying source data.
Brand-critical content at scale (premium-brand marketing, partner communications, board reporting): the tone is a brand asset; the facts must be defensible.

When it is wrong. Most enterprise scenarios are not this. Internal productivity work, departmental Q&A, code completion, meeting summaries, research assistants, internal helpdesk: pure RAG (or Level 1 / Level 2) is the right answer. The combined pattern is expensive at training AND at inference, and only earns its keep when both voice and traceability are non-negotiable.

The honest cost shape. Fine-tuning on Foundry adds three operational lines that pure-RAG does not have. Per Microsoft's stated fine-tuning challenges, the training set must be high-quality, sufficiently large, and representative of the target domain; poor data leads to over-fitting and bias. Beyond that:

Training-data curation. Representative voice/format pairs. Building the training set is often the highest-effort step.
Hourly hosting charge per deployed fine-tune. A deployed fine-tuned model incurs an hourly hosting cost regardless of inference volume. Foundry deletes deployments after 15 days of zero traffic but the artifact stays; production workloads keep the deployment hot, which is a recurring cost line absent from pure-RAG.
Retrain cadence on base-model updates or training-data drift. When Microsoft ships a new base model you want to move to, you re-fine-tune. When your voice or domain examples change materially, you re-fine-tune. No equivalent retrain step exists in pure RAG; you just refresh the index.

The 2026 architect's read on combined. In our experience, fewer than one in five enterprise workloads actually clears the combined-pattern bar. Of those that do, half attempt it, hit the training-data discipline wall, and roll back to RAG-only with stronger prompting on voice. The ones that ship and stay shipped are the ones with a named voice owner (legal, brand, compliance) maintaining the training set as a versioned artifact alongside the index.

In short: the "tailored AI trained on your data" framing is real, but the production-grade form of it is fine-tune for voice, RAG for grounding, evaluators on the combined output, named owner for the training set. The slogan version of "trained on your data" usually means "fine-tuned without RAG," which is the version that fails the audit test.

Worked Example: AI-Assisted Proposal Writing at Level 3

Abstract three-level frameworks become believable when tied to a workload. The one we run through is proposal writing for a 40-person consulting firm responding to 8-12 RFPs per month.

The setup (illustrative scenario, composite of pilots we've reviewed). Past content lives across three systems: SharePoint (proposals, SOWs, case studies, reference letters), Dataverse (engagement records), and a Confluence wiki (technical approach templates). A senior consultant spends multiple days per proposal assembling content, often mis-citing past stats, missing recent wins, or quoting the wrong client. Win rates on RFPs requiring detailed past performance are well below where the firm wants them.

Why Level 1 fails. ChatGPT Enterprise or M365 Copilot will generate a plausible-sounding proposal. It will hallucinate client names, invent project durations, and quote ROI figures that exist nowhere in your actual delivery record. Every claim needs human verification, which means the consultant is doing the same assembly work as before plus correcting fabricated claims. Net productivity: marginal or negative.

Why Level 2 falls short. A Copilot Studio agent with knowledge sources pointing at the SharePoint proposal library partially works. It can retrieve relevant past proposals when asked. But it does not cite specific paragraphs reliably, drifts toward LLM-generated filler outside the retrieved scope, and cannot tell you which past engagements are most relevant by industry / deal size / scope match. Compliance review of the draft is still required because you cannot trace every claim back to a verifiable source.

Why Level 3 works. A Foundry agent with agentic retrieval over an indexed proposal corpus produces the following flow:

Consultant feeds the RFP to the agent: "Draft a response to this RFP for a regional health insurer. Required sections: technical approach, past performance, references, project plan, pricing approach."
The agent's retrieval plan generates focused subqueries: "past engagements in health insurance," "data migration case studies for regulated industries," "client references with named contacts," "pricing approaches for 6-12 month engagements," "team bios with healthcare project history."
Each subquery runs against the indexed proposal corpus with semantic ranking. Top-N passages return with full source citations (SOW #2024-118, Case Study CS-2023-Q3-Health, Reference Letter RL-Acme-2024).
The agent synthesizes section drafts, each paragraph linked to the source passages that grounded it.
The activity array shows every subquery run, every passage retrieved, every ranking score. Compliance review takes minutes, not hours.
Pricing is computed deterministically via a Power Automate flow that pulls actual past-engagement rate cards, not estimated by the LLM.
The consultant edits the draft for win-theme positioning (the human strategic value-add) instead of assembling content.

The end-to-end flow is below: RFP in, subqueries fan out, corpus returns ranked passages with citations, synthesis branches to deterministic pricing on the side, cited draft out.

Level 3 proposal-writing retrieval flow. Agent orchestrates, deterministic layer calculates.

A sketch of the retrieval contract the agent works against, in pseudocode:

// Agentic retrieval contract (Foundry-style, simplified)
type ProposalSubquery = {
  intent: "past_engagement" | "case_study" | "reference" | "pricing" | "team_bio";
  filters: {
    industry?: string;          // "health_insurance"
    dealSizeUsd?: [number, number];  // [250_000, 2_000_000]
    technology?: string[];      // ["dataverse", "power_automate"]
    sharableExternally: true;   // never retrieve NDA-restricted content
  };
  topN: number;                 // pagination, default 5
};

type RetrievalResponse = {
  content: string;              // synthesized passage
  references: SourceRef[];      // each citable: SOW-2024-118, CS-2023-Q3-Health
  activity: SubqueryTrace[];    // every subquery + ranking score + token cost
};

// Pricing is NEVER computed by the LLM
function computePricing(scope: ScopeSpec): PricingResult {
  return powerAutomateFlow.invoke("rate-card-pricing", scope);
}

The contract is the discipline. Every retrieval pass returns content + references + activity. Pricing always routes to deterministic compute. The agent orchestrates; the deterministic layer calculates.

The named architectural choices that make this work:

Curated index, not raw dump. The proposal corpus is preprocessed: dedup, anonymization where appropriate, tagged by industry / scope-size / outcome / technology stack. Garbage in = grounded garbage out.
Schema-aware retrieval. Subqueries scope by industry first, then deal size, then technology. Not a flat similarity search across everything.
Groundedness evaluators in CI. Each agent release runs the groundedness evaluator against a held-out set of past RFPs. Drops below 90% block deploy.
Deterministic pricing. The agent never calculates rate-card math; it calls a flow that does.
Audit trail. Every proposal sent has its retrieval activity log archived. If a claim is later challenged, the source is traceable.

The realistic operational outcome (illustrative). Three things change. Proposal turnaround drops from multi-day to half-day of senior consultant time. Win rate improves modestly because proposals reference the right past wins with the right specifics. Every claim is source-traceable, which makes legal review faster and procurement requests answerable.

This is not a marketing-slogan outcome ("AI writes your proposals"). It is the outcome architects can defend to a managing partner. Multiplier numbers depend on corpus quality, retrieval design, and how much rework the agent saves the senior reviewer; pilots we have reviewed land at very different points along this range, so we decline to quote a single percentage.

What broke in the pilots, honestly. Level 3 is not failure-free. The two failure modes we see most often: the agent's subquery generator producing overly-broad initial queries that return passages from out-of-scope industries (fixed by adding a filters_required guardrail before subquery dispatch), and security-trimming gaps surfacing only when a guest auditor account ran the agent during user-acceptance testing (fixed by re-running security trimming against the full identity matrix, not just internal accounts). Neither is a Level 3 indictment; both are a reminder that operational discipline is where Level 3 earns its keep.

This is the pattern we wrote up in more depth in the AI Proposal Writing on Foundry article. The principle generalizes: any workload where claims need to be source-citable, RAG with agentic retrieval is the right substrate.

Which Architect Responsibilities Don't Disappear at Level 3?

Adopting Level 3 does not absolve the architect of existing responsibilities. The platform makes audit defensibility possible; the work below makes it real. In our experience, these are the seven responsibilities the platform does not own for you.

Responsibility	What the platform does NOT do for you
Semantic modeling of the source corpus	Decide which fields matter (industry tags, deal size, outcome, technology, named-client where shareable) and how proposals link to SOWs, case studies, and references. The retrieval is only as good as the model behind the index; the platform inherits whatever taxonomy you bring.
Data shaping discipline	Clean past content, deduplicate entities, choose a sensible chunking strategy. Microsoft's RAG documentation consistently positions content preparation, indexing strategy, and prompt design as the levers under your control. The platform indexes whatever quality you bring it.
Retrieval design	Schema-aware queries, security trimming so users only retrieve content they are entitled to, top-N pagination so the agent does not exhaust the context window. Without security trimming, RAG leaks competitive intel between client proposals.
Groundedness evaluation in CI	Run RAG evaluators against a held-out set on every change. Threshold-gate deploys. Without this, regressions ship silently.
Deterministic tooling for calculations	Agents produce probabilistic outputs. For anything that needs a correct number on the first try (pricing math, ROI projections, financial reconciliation, regulatory thresholds), wire deterministic logic via Power Automate flows, calculated columns, or external compute. Agent orchestrates; deterministic layer calculates.
Governance + RBAC propagation	DLP policy, Purview labels, audit-log review cadence. RBAC must propagate from the asking user through the agent through every tool call (RAG retrieval AND operational API AND runtime context). Indexed corpora often hold content classified differently than the surface application's user permissions. The platform enables Level 3; your governance discipline makes it safe.
Tenant isolation + version drift	For multi-tenant deployments (MSPs, multi-end-client consultancies, hosted SaaS), the index, the evaluators, and the audit log all need explicit tenant tagging. Version drift is a parallel risk: when the indexed corpus, the LLM base model, the evaluator suite, and the system-of-record schema all evolve on independent cadences, the architect owns the version compatibility matrix. Plan a versioning strategy or accept silent regressions.
Data residency + procurement	Confirm sovereign-region availability, customer-managed-key support for the index, and the contractual position on training-on-customer-data before you commit. Procurement-time friction is cheaper than mid-build replatforming. For regulated workloads (financial services, public sector, healthcare in EU/UK), the procurement model matters as much as the architecture.

What we hear from engineering teams running Level 3 in 2026

From engineering-team interviews and pilot post-mortems we have reviewed, four operational failure modes show up repeatedly. Vendor decks don't lead with these; production architects do:

Retrieval quality degrades on poorly-modeled corpora at scale. Roughly past the few-hundred-thousand-chunk mark, semantic ranking quality drops noticeably when the index lacks structured metadata. Exact threshold varies with chunking strategy and schema discipline.

Groundedness evaluators are gameable. Prompt rewrites can lift evaluator scores without fixing the underlying retrieval gap. The metric is a useful floor, not a guarantee.

Security-trimming gaps surface under guest and cross-tenant access. Pilot environments rarely exercise the full identity matrix; the gaps appear during UAT or worse, in production.

Multi-turn conversational context retention is uneven across runtimes. Different Foundry agent runtimes handle long-context grounding differently in 2026; behaviour you tested in one runtime may not transfer.

These are field observations, not citable forum links: we have deliberately declined to attach "Reddit thread" or "Microsoft Q&A" citations we cannot verify on demand. None of these invalidate Level 3 architecture. They do mean the responsibilities above are non-optional, not aspirational.

What Level 3 Will NOT Do for You (Eight Limits)

The "tailored AI on your data" framing implies more than it delivers. Skeptical readers will ask each of these:

Level 3 does not mean "we trained AI on our data." Tailored RAG retrieves from an indexed enterprise corpus at query time. The base model weights are unchanged (unless you also fine-tune, which is the combined pattern from earlier and a separate decision). When a stakeholder asks "did we train it on our data?", the precise answer is "we indexed our data and the model retrieves from it per query, which is what you actually wanted because training would be slower to update and harder to audit."
RAG does not replace your data warehouse, lakehouse, or analytics substrate. Azure Data Lake, Fabric, and Synapse remain the right homes for high-volume analytics. The RAG index is downstream of these for grounding, not a substitute for them.
RAG does not eliminate hallucinations. It narrows the gap between LLM output and source truth; it does not close it. Retrieval can return wrong passages, the LLM can synthesize incorrectly from correct passages, and the model can ignore retrieved context if your prompt design is weak. Groundedness evaluators catch most failures, not all.
Level 3 is not "set and forget." Index freshness, schema drift, retrieval quality, and groundedness scores all need ongoing operational discipline. We have seen meaningful groundedness drift over two-quarter windows when no one owns the eval harness. The exact decay curve depends on how fast your source corpus changes; the pattern is consistent.
Not every workload needs Level 3. Most internal productivity work is fine at Level 1. Most departmental knowledge agents are fine at Level 2. Level 3 is for workloads where audit defensibility matters: proposals, claims, regulatory response, financial reporting, customer-facing communications, anything that goes into a legal record.
Level 3 is not a replacement for governance. DLP, Purview, RBAC, audit log review still apply. The platform makes grounded answers possible; governance makes them safe.
Level 3 is not sufficient without deterministic tooling. Agents are probabilistic. Calculations, regulatory thresholds, and any numeric output that must be correct on the first try should be computed by deterministic tools called by the agent, not by the LLM.
Level 3 is not autonomous AI. A Level 3 agent is more accurate than a Level 1 chat. It is not a reliable autonomous worker. Human-in-the-loop, evaluation gates, and fallback flows still apply.

Each of these limitations is what your audit committee, compliance lead, or procurement officer will eventually ask about. Better to scope around them now than to discover them in production.

Decision Frame: When Each Level Is Right

Six workload archetypes mapped to levels. Find yours. The decision tree below collapses the table into three questions; the table after it carries the detail.

Decision tree: three questions, four leaves, plus the apex combined pattern when voice consistency also matters.

Level fit is necessary but not sufficient. Even the right-level workload still requires the seven architect responsibilities above (semantic modeling, retrieval design, evaluators in CI, deterministic tooling, governance, data residency, training-set ownership if you fine-tune).

The reference table:

Workload	Right level	Why	Common mistake
General productivity (email drafting, meeting summaries, code completion)	Level 1	Speed + cost + low audit risk. The training-data-blend trade-off is acceptable.	Building custom RAG when M365 Copilot would have shipped in a week
Departmental knowledge Q&A (IT helpdesk, HR benefits, sales enablement)	Level 2	Knowledge-source-scoped agents in Copilot Studio cover this cleanly. Connector retrieval gives partial grounding.	Stopping at Level 2 for workloads that actually need audit defensibility (compliance, claims)
Regulated customer-facing comms (proposals, claims, insurance, legal response)	Level 3	Every claim must be source-traceable. Compliance review takes minutes only with structured citations.	Building at Level 1 or 2 and discovering at audit time that nothing is provable
Financial reporting + executive summaries	Level 3 + deterministic layer	Numbers must be exact. RAG for narrative + deterministic compute for math. Tone-fine-tuning optional.	Letting the LLM do arithmetic
Field operations (inspection, dispatch, technician guidance)	Level 2 or 3	Depends on whether the field action has audit/regulatory exposure (then Level 3) or is internal-only guidance (Level 2 suffices).	One-size-fits-all; same agent design for low-risk internal vs regulated external
Internal R&D / experimentation / prototyping	Level 1	Fast iteration > audit defensibility. Promote to Level 2 / 3 when prototypes become production.	Promoting a Level 1 prototype to production without re-architecting

The pattern: pick the level per workload, not per organization. Most enterprises will run all three levels simultaneously, with the architecture team owning the boundary decisions.

Need a sanity check on your three-level workload map?

If you are scoping which enterprise workloads belong at which level, reach out for a second-opinion review of the workload-to-level mapping before you commit to a build budget.

When Does Each Level Fail? Transition Triggers Between Levels

Enterprise architects do not want a maturity ladder. They want decision triggers, concrete signals that the current level has run out of headroom and the workload needs to advance. The triggers below are the ones we see force level transitions in pilots we have reviewed.

Current level	Trigger that forces advancement	What it costs to advance
Level 1 → Level 2	The workload starts producing externally-visible content (departmental Q&A, customer-adjacent), or stakeholders begin asking for source citations the M365 Copilot blend cannot reliably provide.	Copilot Studio + Managed Environments + ALM discipline + knowledge-source curation. Real but bounded cost.
Level 2 → Level 3	Compliance, legal, or procurement starts asking 'where did that claim come from?' and the connector-citation answer is no longer sufficient. Or the corpus grows past the connector retrieval quality envelope.	Foundry agent build + indexed corpus + agentic retrieval + groundedness evaluator harness + CI gating. New operational discipline; new observability burden.
Level 3 → Level 3 + fine-tune	Voice consistency becomes a regulatory or brand-quality requirement (regulated customer comms, executive voice, brand-critical content at scale) AND retraining cadence is acceptable.	LoRA SFT + optional DPO + ongoing voice-corpus ownership + hourly fine-tune hosting + retrain on base-model updates. Highest operational cost in the stack.

The trigger column matters more than the cost column. If the trigger has not fired, advancing is over-engineering. If the trigger has fired, staying at the current level is technical debt accruing toward an audit incident.

When Should a Copilot Studio Agent Move to Foundry?

For Power Platform teams already running Copilot Studio agents with knowledge sources, the practical question is not "Level 2 or Level 3?" abstractly. It is "do we extend the existing agent, or rebuild it?" In our experience the decision turns on three concrete tests.

Extend in Copilot Studio (stay Level 2) when the workload tolerates connector-source citations without structured grounding metadata, when knowledge-source documents change slowly enough that connector refresh windows are acceptable, and when the agent's output is reviewed by humans who can verify claims themselves. Add Azure AI Search as a connector if you need broader corpus coverage; this stays a Level 2 deployment with stronger retrieval, not a Level 3 architecture.

Rebuild in Foundry (move to Level 3) when the workload requires structured citation evidence on every response (compliance review, regulated comms), when you need groundedness evaluators gating CI deployments, when you need the activity-array audit trace, or when the corpus is large enough that connector-style retrieval no longer ranks well.

The breakpoint is the audit posture, not the model behind the agent. A Copilot Studio agent and a Foundry agent can both call the same underlying model; what differs is the response contract (does it return structured grounding data and activity trace?) and the operational discipline (is there a groundedness evaluator gate on every release?). If the answer to either is "no," the agent is at Level 2 regardless of model choice. If the workload requires "yes" to both, plan a Foundry rebuild, not an extension.

For Managed-Environments-deployed Power Platform tenants, the rebuild has an additional discipline layer: ALM, solution packaging, environment routing, and CoE governance approval cycles. Plan the Foundry build inside the same ALM regime, not as a separate stream.

How Should Your 2026 AI Architecture Plan for All Three Levels?

In our read, the three-level framing is more useful than the five-level CMM maturity model Microsoft publishes for adoption. The CMM model measures the organization. The three levels measure the workload. The organization-level view is for change-management leads; the workload-level view is for architects making build decisions.

The deeper signal we are watching: in our read, traceability is becoming the primary differentiator between AI that ships demos and AI that ships production systems. Level 1 wins on speed-to-demo. Level 3 wins on speed-to-audit. Both have valid use cases. The category in the middle (Level 2) is where most enterprises sit and most regret-investments live: enough customization to claim "we built our own AI," not enough citation discipline to actually defend it.

Our bet, falsifiable

By the end of 2027, the procurement teams of regulated enterprises will increasingly require Level 3 architecture (agentic retrieval + groundedness evaluation + audit log archival) for AI systems that produce customer-facing claims. The regulatory trajectory points the same way: EU AI Act Article 13 requires transparency and traceability for high-risk AI; the NIST AI Risk Management Framework names "traceability" as a Measure-function expectation; FFIEC AI guidance applies model-risk-management discipline to AI in financial services. None prescribe Level 3 specifically, but each makes retrieval-trace evidence the natural compliance posture. Vendors that ship Level 1 or Level 2 with marketing-language audit-trail will lose RFPs to vendors that ship structured Level 3 evidence by default.

How we would know we are wrong: if federal, financial-services, and healthcare RFPs through 2026 and into 2027 continue accepting Level 1 / Level 2 vendor responses without specific technical-response requirements for retrieval activity logs, structured citation evidence, or groundedness-evaluation thresholds, the prediction fails. The specific language to watch in technical-response sections: explicit "structured citation evidence" or "retrieval activity log" or "groundedness-evaluation threshold" clauses. We do not yet see this in 2026 RFPs at scale; we are betting it shows up in the next 18 months. If it does not, the bet is wrong.

The architect's question is not "which level should we adopt?" It is "which level does each of our workloads actually need, and are we building each one at the right level?"

Enterprise AI Anti-Patterns: What Fails in Production

Naming the failure modes is more useful than naming the successes. These are the patterns we see fail repeatedly in pilots and the early-production stage. If your architecture has any of them, the audit is going to surface them, so better to surface them yourself first.

Vector database as universal truth layer. Treating the indexed corpus as the source of "what is currently true" instead of as the source of "what we wrote down at some point in the past." Order status, account balances, deal state: none of those belong in a vector index. They belong in systems of record, queried at request time.
Stateless agents in stateful workflows. An agent that forgets the approval that was granted three turns ago, or the prior tool call that already moved the workflow forward. Runtime context is not optional infrastructure.
Direct model-to-ERP writes. Letting the LLM generate the payload and call the mutation endpoint without typed contracts, validation, or human approval boundaries. The first regression silently corrupts production data.
Missing permission inheritance. RBAC enforced at the surface application but not propagated to the retrieval call, the API call, or the runtime-context lookup. The agent leaks across users the moment one of those calls runs unscoped.
Prompt-only orchestration. Trying to control which tool the agent picks, in what order, with what guardrails, entirely through prompt engineering. Works in demos. Fails under adversarial input, fails under multi-tenant load, fails under workflow complexity. Wire the orchestration logic deterministically; let the LLM only choose within the bounded option set.
Unbounded agent autonomy. "Let the agent figure it out" without scoped actions, typed contracts, or human approval boundaries. This is how blast-radius incidents happen.
Groundedness evaluators as a compliance fig leaf. Running the evaluator suite for the audit committee then ignoring its output because it gates deploys. Either the threshold gate is real and slows you down, or it is theatre.
Treating retrieval quality as fixed. Building the index once at project kickoff and never re-indexing as the source corpus drifts, the chunking strategy reveals its weaknesses, or the ranker improves. Retrieval is an operational discipline, not a one-time setup task.
One global evaluator for all workloads. Workload-specific evaluation harnesses don't share thresholds. Proposal-writing groundedness and customer-claims groundedness need different gold sets and different bars.
Observability as a logging afterthought. Tracing the LLM call but not the retrieval plan, the tool invocations, or the runtime envelope. The next incident debugging session will reveal the gap.

Each of these is recoverable if caught early. The hard ones are the ones that ship to production unnoticed and only surface during an audit or an incident.

The Hard Part Is Often Organizational, Not Architectural

Most of this article has been about architecture: layers, contracts, evaluators, observability, autonomy boundaries. That is the part architects can solve directly. It is also, honestly, not where most enterprise AI programs actually fail. Where they fail, in our experience:

Ownership ambiguity. Who owns the indexed corpus? Who owns the evaluator harness? Who owns the prompt? Who owns the agent's autonomy boundary? In federated enterprises, these often have no single owner, and the resulting drift is invisible until an incident.
Continuously evolving business semantics. The org's definition of "customer," "active," "open," "approved," "in good standing" shifts as products, regulations, and acquisitions change. Every shift breaks something downstream. The architecture cannot freeze meaning that the business keeps redefining.
Conflicting ownership domains. The CRM is owned by Sales; the ERP is owned by Finance; the indexed corpus is owned by IT; the Foundry environment is owned by a Cloud CoE. Each has its own change cadence and approval process. An ECA spanning all of them is, in practice, four different change-management cycles stitched together.
Procurement friction. The contract for the model endpoint, the contract for the search service, the contract for the evaluator product, and the contract for the agent runtime are often four separate procurement cycles. By the time all four close, the architecture has shifted underneath.
Legal review cycles. Data residency, training-on-customer-data clauses, indemnity for grounded-output errors, regulatory disclosure, retention policy on activity logs: each of these is a separate review, often with different approving counsel.
Operational support burden. A Level 3 system has a 24x7 operational shape if customers depend on it. Most enterprises do not have an existing on-call rotation for "the agent's eval harness drifted overnight." That capability has to be built.
Internal platform fragmentation. Multi-BU enterprises rarely have one CRM, one ERP, one identity provider. They have versions of each. The ECA's "operational APIs" layer is sometimes five APIs that each return the same field with a different name.

The honest architectural admission. Perfect context orchestration can itself become a bottleneck. A sufficiently centralized ECA can become the single point of slowness that every workload routes through. A sufficiently federated one becomes a coordination problem where no single team can ship an end-to-end change. Architects who have lived through both extremes will recognize this tension; there is no clean answer, only the workload-by-workload calibration the article has argued for throughout.

The technical disciplines in this article are necessary. They are not sufficient. The enterprise AI programs that ship and stay shipped are the ones with named owners per layer, explicit semantic-versioning of business definitions, a cross-domain coordination function, and patience for the procurement and legal cycles that the technology itself cannot accelerate.

The Position, Stated Plainly

Enterprise AI is not a single retrieval problem. It is the orchestration of multiple context systems with different latency, authority, freshness, and governance characteristics. Everything else in this article is a corollary.

The corollaries, distilled:

RAG ≠ enterprise AI architecture. RAG is one knowledge layer (unstructured). Real Level 3 systems also call structured systems of record and inject runtime context. Architect for three layers, not one.
Pick the level per workload, not per organization. Most enterprises run all three levels simultaneously. The architecture team owns the boundary decisions.
Traceability is the apex axis. Level 1 wins on speed-to-demo. Level 3 wins on speed-to-audit. The combined fine-tune + RAG pattern wins on both audit and voice. Each has valid use cases; choose by what the workload actually demands.
Operate three layers with different disciplines. Retrieval, operational API calls, and runtime context have different latency budgets, freshness requirements, governance regimes, and evaluation surfaces. A Level 3 architecture that uses the same operational discipline across all three is under-engineered.
Permissions propagate through every layer. RBAC must flow from the asking user through the agent through every tool call. DLP, Purview, tenant isolation, and version drift all apply across all three layers, not just the LLM.
The agent orchestrates; the deterministic layer calculates. Numbers come from systems of record or deterministic compute, never from the LLM. Citations come from RAG. Identity comes from runtime context. The agent is the coordinator, not the source.

Build to that bar and Level 3 is a system you can defend in an audit, not a slide you defend to procurement after the contract is signed.

The directional close. The future enterprise AI stack will not be defined primarily by the model layer. The model layer will commoditize. It will be defined by how reliably organizations manage context, authority, permissions, and execution across distributed systems, by their Enterprise Context Architecture. That is the durable architectural lane. Build into it.

Glossary: Terms in This Article

Agentic retrieval: a retrieval pattern where an LLM-driven agent decomposes a complex question into focused subqueries, runs them against an indexed corpus in parallel, and returns structured grounding data with citations and execution metadata. Distinguished from classic single-query RAG by its subquery planning and structured response format.
Groundedness evaluator: an automated test that scores whether an LLM response cites only the retrieved context, or fabricates outside it. Microsoft Foundry exposes this as a first-class RAG evaluator with a numeric score.
Foundry IQ: Microsoft's umbrella for Foundry-native retrieval, knowledge stores, and evaluator tooling that sits under Foundry agents.
Tailored RAG: our shorthand for Level 3 in this article: RAG over a curated, indexed enterprise corpus with agentic retrieval, security trimming, groundedness evaluation, and deterministic tooling for calculations. Not a Microsoft product name.
Security trimming: filtering retrieved passages by the requesting user's permissions so the agent never returns content the user cannot legitimately see. Implemented at the index level, not at the LLM level.
Activity array: the trace of subqueries, retrieval calls, and ranking scores returned alongside the synthesized answer. Used as the audit-log substrate for Level 3 responses.
LoRA fine-tuning: low-rank adaptation, the parameter-efficient fine-tuning technique Foundry uses by default. Trains a small adapter on top of the base model rather than re-training the full weight matrix; cheaper, smaller artifact, faster to deploy.
Knowledge retrieval layer (RAG / unstructured): the substrate that holds documents, policies, manuals, past work product. Authoritative for "what does our policy say" but not for "what is the current state." Mechanism: agentic retrieval over an indexed corpus.
Operational data layer (structured / systems of record): the substrate that holds the authoritative state of the business: orders, claims, invoices, customer records. Mechanism: deterministic API or SQL calls scoped to the asking user's permissions. Never paraphrased by the LLM.
Runtime context layer: the substrate that holds who is asking, what tenant they belong to, what they are authorized to see, what workflow step they are in. Mechanism: context injection at agent invocation; passed through every downstream tool call as the permission envelope.
Version drift: the silent failure mode where the indexed corpus, the base LLM, the evaluator suite, and the systems-of-record schemas evolve on independent cadences and the agent's behavior changes without anyone noticing. Mitigated by versioning each component and running the evaluator harness on every release.

RAG Is Not Enough: The Evolution of Enterprise AI

RAJSHREE — Thu, 13 Aug 2026 14:39:54 +0000

Author: Rajश्री | Software Engineer & Full Stack Developer

Introduction

For the last few years, Retrieval-Augmented Generation (RAG) has become one of the most popular architectures in enterprise AI.

And for good reason.

A large language model may be excellent at reasoning and language generation, but it does not automatically know your company's internal policies, customer records, product documentation, support tickets, contracts, or constantly changing business data.

RAG provided an elegant solution.

Instead of retraining the model whenever enterprise knowledge changes, retrieve relevant information at inference time and provide it to the model as context.

The architecture looked simple:

User Query
    ↓
Retrieve Relevant Information
    ↓
Build Context
    ↓
LLM
    ↓
Generate Answer

This was a huge step forward.

A company could take its existing documentation, index it, connect a language model, and suddenly employees could ask questions about internal knowledge using natural language.

But then production happened.

The questions became harder.

Users stopped asking only:

"What is our leave policy?"

They started asking:

"Am I eligible for this leave based on my current employment status?"

They stopped asking:

"What is our refund policy?"

They started asking:

"Can I approve this customer's refund, and if yes, process it."

They stopped asking:

"What does this support document say?"

They started asking:

"Check the customer's account, verify the issue, determine whether they're eligible for replacement, create the ticket, and notify them."

And suddenly the problem changed.

This was no longer simply a knowledge retrieval problem.

It became a software engineering problem.

The system needed:

retrieval
reasoning
authentication
authorization
live data
business rules
APIs
tools
memory
workflow state
validation
observability
security
and sometimes human approval.

That leads to a very important realization:

RAG is not enough.

Not because RAG is obsolete.

Not because vector databases are useless.

Not because agents have replaced retrieval.

But because retrieval is only one capability of a production enterprise AI system.

The real evolution looks more like:

LLM
 ↓
RAG
 ↓
Advanced RAG
 ↓
Agentic Retrieval
 ↓
Tool-Using AI
 ↓
Stateful AI
 ↓
Governed AI Workflows
 ↓
Reliable Enterprise AI Systems

And understanding this evolution is where AI engineering begins to look much more like software engineering—and much less like prompt engineering.

1. First, What Did RAG Actually Solve?

Before discussing why RAG is not enough, we need to give RAG the credit it deserves.

Large language models have an important limitation:

The model's parameters are not your company's database.

Imagine an employee asks:

"What is our company's enterprise customer refund policy?"

A general-purpose LLM may know what refund policies usually look like.

But it doesn't automatically know your company's current policy.

Your organization may have:

Refund Policy.pdf
Enterprise Customer Policy.pdf
Finance Guidelines.pdf
Regional Exceptions.pdf
Support Documentation/

RAG creates a bridge between the model and that external knowledge.

A simplified architecture looks like this:

                     Enterprise Knowledge
                           │
              ┌─────────────┼─────────────┐
              ↓             ↓             ↓
          Documents       Wikis        Knowledge Base
              │             │             │
              └─────────────┼─────────────┘
                           ↓
                       Chunking
                           ↓
                       Embeddings
                           ↓
                     Vector Database
                           ↓
                        Retrieval
                           ↓
                     Relevant Context
                           ↓
                           LLM
                           ↓
                         Answer

This solved several major problems.

Private knowledge

The model can work with internal company information without that information being part of its original training data.

Fresh information

If a policy changes, the knowledge source can be updated without retraining the model.

Grounding

The model can generate answers based on retrieved enterprise context rather than relying entirely on its pretrained knowledge.

Citations

A well-designed system can show users which documents or records support the answer.

Lower operational complexity

For many knowledge-based applications, RAG is significantly simpler than fine-tuning a model for every knowledge update.

So yes:

RAG is extremely useful.

But it solves a specific problem:

"How can an LLM access relevant external knowledge?"

Enterprise AI eventually asks a much larger question:

"How can an AI system reliably accomplish a business objective?"

Those are very different problems.

2. Retrieval Is Not Understanding

This is one of the most important distinctions in enterprise AI.

Imagine an employee asks:

"Can I approve this refund?"

A basic RAG system may retrieve:

Refund Policy.pdf
Customer Refund Limits.pdf
Approval Guidelines.pdf

The model can read them and explain the rules.

But the actual question is not:

"What does the refund policy say?"

It is:

"Given this specific customer's account, transaction amount, my role, the current approval state, and applicable regional rules, am I authorized to approve this refund?"

Now look at everything the system needs:

Policy
+
User Identity
+
Role
+
Permissions
+
Customer Data
+
Transaction Data
+
Current Workflow State
+
Business Rules
+
Reasoning

A vector database cannot provide all of this.

And an LLM should not be expected to invent it.

This is the first major architectural lesson:

Retrieval provides context. It does not automatically provide the complete state of the business.

3. Enterprise Data Is Not Just Documents

One of the mistakes developers make when building their first RAG system is assuming:

Enterprise Knowledge = Documents

Real enterprise environments are much messier.

A company may have:

                    Enterprise
                        │
         ┌───────────────┼────────────────┐
         ↓               ↓                ↓
     Documents         Databases         APIs
         │               │                │
         ↓               ↓                ↓
       Wiki             CRM              ERP
       PDFs             HRMS             Payments
       Policies         Tickets          Inventory
       Manuals          Analytics        Identity

And these sources behave differently.

A policy document is not the same thing as a customer database.

A customer database is not the same thing as an API.

An API response is not the same thing as a knowledge graph.

A transaction table should not necessarily be embedded into a vector database simply because you're building a RAG system.

This is where practical AI engineering begins.

The engineer needs to ask:

What kind of information is this, and what is the correct way to access it?

For example:

Unstructured knowledge
        ↓
Semantic / Hybrid Retrieval

Structured business data
        ↓
SQL / Database Query

Real-time information
        ↓
API

Relationships
        ↓
Knowledge Graph

Business operation
        ↓
Tool / API / Workflow

This is a much more useful mental model than:

"Let's put everything into a vector database."

4. Why "Just Add More Documents" Doesn't Fix RAG

Suppose your RAG system is giving poor answers.

A common reaction is:

"Maybe it doesn't have enough information. Let's add more documents."

So the team adds another 100,000 documents.

Then another 500,000.

Eventually the system contains:

Old Policies
New Policies
Regional Policies
Draft Policies
Archived Policies
Internal Notes
Duplicate Documents
Different Versions
Conflicting Documents

Now retrieval itself becomes harder.

The problem was not lack of data.

The problem was information quality and information architecture.

A simplified failure chain looks like this:

Poor Data
   ↓
Poor Chunking
   ↓
Poor Indexing
   ↓
Poor Retrieval
   ↓
Wrong Context
   ↓
Wrong Reasoning
   ↓
Wrong Answer

Adding more documents doesn't necessarily improve the system.

Sometimes it makes it worse.

A production AI engineer therefore asks:

Is this document authoritative?
Is it current?
Who owns it?
Which version applies?
Can this user access it?
Is the information structured or unstructured?
Should this information even be retrieved semantically?

These questions are often more important than:

"Which embedding model should I use?"

5. Basic RAG vs Advanced RAG

A simple RAG pipeline may look like:

Query
 ↓
Embedding
 ↓
Vector Search
 ↓
Top K Chunks
 ↓
LLM

This is useful for prototypes.

But production retrieval often requires more.

A stronger architecture might look like:

User Query
    ↓
Query Understanding
    ↓
Intent Detection
    ↓
Metadata / Permission Filtering
    ↓
Hybrid Search
    ↓
Vector Retrieval
    +
Keyword Retrieval
    ↓
Candidate Documents
    ↓
Reranking
    ↓
Context Selection
    ↓
Context Compression
    ↓
LLM

Possible techniques include:

semantic search
keyword search
hybrid retrieval
metadata filtering
reranking
query expansion
query rewriting
contextual chunking
hierarchical retrieval
document-level access control
citation tracking

The important point is:

Better RAG is not simply "use a better vector database."

Retrieval is an information architecture problem.

6. The Permission Problem Most RAG Demos Ignore

This is where enterprise AI becomes a serious security problem.

Imagine a company has:

Document A → Public
Document B → Engineering
Document C → Finance
Document D → HR
Document E → Executive

An employee from Engineering asks:

"What was the company's executive compensation strategy?"

A naive RAG system might retrieve Document E because it is semantically relevant.

The LLM now has access to information that the employee should never have seen.

This is not a hallucination problem.

This is an authorization failure.

And the solution should not be:

Retrieve sensitive information
        ↓
Tell the LLM:
"Please don't reveal it."

That is not a security boundary.

Authorization needs to happen before sensitive information reaches the model.

A safer architecture is:

User
 ↓
Authentication
 ↓
Identity / Role
 ↓
Authorization
 ↓
Allowed Data Scope
 ↓
Retrieval
 ↓
LLM

This distinction is critical:

The model should not be trusted to enforce access control.

The application architecture must enforce it.

7. When RAG Meets Live Data

Let's take a practical example.

A customer asks:

"Where is my order?"

The company's documentation might explain shipping policies.

RAG can answer:

"Standard shipping usually takes 3–5 business days."

But that's not what the customer actually wants.

They want to know:

"Where is my specific order right now?"

That information lives in a live system.

Customer
   ↓
AI Assistant
   ↓
Order Lookup Tool
   ↓
Order Management API
   ↓
Current Order Status
   ↓
LLM
   ↓
Natural Language Response

Here, RAG may still be useful for explaining shipping policies.

But the actual order status should come from the source of truth.

This leads to a practical rule:

Use retrieval for knowledge. Use systems of record for facts that must be current.

Don't embed something into a vector store simply because you can.

8. The Shift From Answering to Acting

This is probably the biggest transition in enterprise AI.

Early systems focused on:

Question
 ↓
Answer

Modern enterprise systems increasingly need:

Intent
 ↓
Reason
 ↓
Retrieve
 ↓
Decide
 ↓
Act
 ↓
Verify
 ↓
Report Result

Consider an IT support assistant.

The user says:

"My company laptop isn't working and it's still under warranty. Create a replacement request."

A basic RAG system can explain the replacement policy.

A useful enterprise AI system should potentially:

Identify the employee.
Identify the assigned device.
Check warranty status.
Retrieve replacement policy.
Determine eligibility.
Create a support ticket.
Attach relevant information.
Route it to the correct team.
Return the ticket ID.
Record what happened.

Now we have:

Knowledge
+
Live Data
+
Reasoning
+
Tools
+
Workflow
+
State
+
Verification

That is far beyond basic RAG.

9. Tool Use Turns AI Into a Software System

Tools are one of the most important additions to enterprise AI.

An AI system might have access to:

Search Tool
SQL Tool
CRM Tool
ERP Tool
Email Tool
Calendar Tool
Ticketing Tool
Payment Tool
Internal API

For example:

"Find overdue invoices above ₹10 lakh and notify the responsible account managers."

The system may need to:

User Request
    ↓
Understand Intent
    ↓
Query Finance Database
    ↓
Filter Invoices
    ↓
Identify Account Managers
    ↓
Apply Notification Policy
    ↓
Send Emails
    ↓
Verify Delivery
    ↓
Return Summary

RAG can provide the policy.

The tools perform the work.

This distinction matters:

Knowledge without action has limited operational value.

10. But Should AI Be Allowed to Do Everything?

This is where "agentic AI" can become over-engineering.

Just because an LLM can call a tool doesn't mean it should have unrestricted access to that tool.

Imagine an AI agent with access to:

Delete Customer
Refund Payment
Send Email
Create Contract
Modify Database
Transfer Money

Giving the model all of these capabilities and saying:

"Use them responsibly."

is not an enterprise architecture.

It is a liability.

Instead, actions should have explicit boundaries.

For example:

Low Risk
    ↓
Automatic Execution

Medium Risk
    ↓
Policy Check + Validation

High Risk
    ↓
Human Approval

A refund under a small threshold might be automatic.

A large financial transaction might require approval.

Deleting an important customer record might require multiple controls.

This is where AI governance becomes part of engineering.

11. Agentic AI Is Not "Let the LLM Run Wild"

The word agent is now used everywhere.

But an agent is not simply:

LLM + Tool Calling

A useful agentic system needs some concept of:

Goal
 ↓
Planning
 ↓
Action
 ↓
Observation
 ↓
Evaluation
 ↓
Next Action

For example:

User:
"Investigate why sales dropped last quarter."

An agent might reason:

1. Query sales database.
2. Compare previous quarter.
3. Identify affected regions.
4. Check product performance.
5. Retrieve sales strategy documents.
6. Check CRM notes.
7. Identify major changes.
8. Synthesize findings.

The key difference is:

The system can decide what information it needs and what actions to take next.

That is more powerful than a fixed RAG pipeline.

But it is also harder to control.

12. Agent Loops Are a Real Production Problem

A demo agent may look impressive:

LLM
 ↓
Tool
 ↓
LLM
 ↓
Tool
 ↓
LLM

But what happens when the model keeps calling tools?

Search
 ↓
Search Again
 ↓
Search Again
 ↓
Search Again
 ↓
Search Again

You now have:

increased latency
increased token usage
increased API costs
possible rate-limit failures
unpredictable behavior

A production agent needs boundaries.

For example:

Maximum Steps
Maximum Tool Calls
Maximum Runtime
Token Budget
Retry Limit
Allowed Tools
Allowed Arguments

And sometimes the best engineering decision is not to use an agent at all.

If the workflow is deterministic:

Step 1
 ↓
Step 2
 ↓
Step 3
 ↓
Step 4

a normal workflow engine may be safer and more predictable than an autonomous agent.

This is a very important AI engineering principle:

Use autonomy where uncertainty exists. Use deterministic software where determinism is possible.

13. Memory Is More Than Chat History

Another common misconception is:

"AI memory means storing previous conversations."

That's only one small part of the problem.

Enterprise workflows often last much longer than a single conversation.

Imagine an employee's hardware replacement:

Monday
↓
Issue reported

Tuesday
↓
Diagnostic information requested

Wednesday
↓
Diagnostics uploaded

Thursday
↓
Manager approval requested

Friday
↓
Replacement approved

The AI needs to understand the state of the workflow.

Something like:

{
  "workflow": "hardware_replacement",
  "employee_id": "EMP-4821",
  "device_id": "LTP-8841",
  "status": "manager_approval_pending",
  "ticket_id": "INC-29482",
  "last_action": "approval_requested"
}

This is not simply conversation memory.

It is application state.

And this distinction is important for engineers coming from traditional software development.

AI systems still need the same fundamentals:

state management
persistence
transactions
idempotency
retries
failure handling
consistency

AI doesn't remove software engineering.

It increases the number of places where you need it.

14. Knowledge Graphs: Where Relationships Matter

Vector search is extremely useful for semantic similarity.

But semantic similarity is not the same as understanding relationships.

Imagine an enterprise contains:

Employee
   ↓
Works For
   ↓
Department
   ↓
Owns
   ↓
Application
   ↓
Processes
   ↓
Customer Data
   ↓
Governed By
   ↓
Policy

These relationships can matter more than textual similarity.

A knowledge graph can explicitly represent them.

For example:

Employee → belongs_to → Department
Department → owns → Application
Application → accesses → Database
Database → contains → Customer_Data
Customer_Data → governed_by → Policy

Now the system can reason over relationships.

This does not mean:

"Knowledge graphs will replace vector databases."

The more realistic architecture is often:

Vector Search
+
Keyword Search
+
SQL
+
Knowledge Graph
+
APIs

Different information requires different retrieval strategies.

15. The Modern Enterprise AI Architecture

Once we combine these capabilities, the architecture becomes much more interesting.

                         USER
                           │
                           ↓
                   ┌─────────────────┐
                   │   AI Gateway    │
                   │ Auth / Limits   │
                   └────────┬────────┘
                           │
                           ↓
                   ┌─────────────────┐
                   │ Intent / Router │
                   └────────┬────────┘
                           │
                           ↓
                   ┌─────────────────┐
                   │ Planner / Agent │
                   └────────┬────────┘
                           │
           ┌────────────────┼────────────────┐
           ↓                ↓                ↓
       Retrieval          Tools            Memory
           │                │                │
           ↓                ↓                ↓
    Vector / Search      APIs / DB       State Store
           │                │                │
           └────────────────┼────────────────┘
                           ↓
                   ┌─────────────────┐
                   │ Policy Engine   │
                   └────────┬────────┘
                           │
                           ↓
                         LLM / Model
                           │
                           ↓
                   ┌─────────────────┐
                   │   Validator     │
                   └────────┬────────┘
                           │
                           ↓
                     Business Action
                           │
                           ↓
                   ┌─────────────────┐
                   │ Audit / Tracing │
                   └─────────────────┘

Notice something important:

RAG is still there.

It just isn't the entire system.

16. The Model Is No Longer the Application

This is perhaps the biggest conceptual shift for software engineers entering AI.

In traditional application development, we might think:

Frontend
   ↓
Backend
   ↓
Database

In AI applications, beginners sometimes replace the backend with:

Frontend
   ↓
LLM

That is usually not enough.

A production AI application still needs:

Frontend
   ↓
Backend / AI Gateway
   ↓
Authentication
   ↓
Authorization
   ↓
Orchestration
   ↓
Models
   ↓
Retrieval
   ↓
Tools
   ↓
Databases
   ↓
Policies
   ↓
Observability

The model is a component.

It is not the whole application.

This is why AI engineering is increasingly becoming an extension of software engineering.

17. RAG Should Not Be Used Everywhere

This is worth saying explicitly.

If the user asks:

"What is our remote-work policy?"

RAG is a good fit.

If the user asks:

"What is the current balance in my account?"

Use the source-of-truth system.

If the user asks:

"Calculate this month's revenue."

Use a database or analytics system.

If the user asks:

"Create a support ticket."

Use the ticketing API.

If the user asks:

"Explain why this transaction was rejected according to policy."

You may need:

Transaction Data
+
Policy Retrieval
+
Business Rules
+
Reasoning

The architecture should follow the problem.

Not the other way around.

This leads to a simple rule:

Don't force every enterprise problem into RAG.

18. The Most Important Enterprise AI Problem: Reliability

A chatbot can sometimes get away with being imperfect.

Enterprise systems usually cannot.

Imagine an AI assistant says:

"Your refund has been processed."

But the refund API failed.

The response sounds perfect.

The user believes the transaction happened.

But it didn't.

This is much worse than a poorly written answer.

Therefore enterprise AI needs verification.

A useful execution pattern is:

Decide
  ↓
Execute
  ↓
Verify
  ↓
Respond

For example:

AI decides:
"Create replacement ticket."

        ↓

Ticket API called.

        ↓

API returns:
ticket_id = INC-29482

        ↓

System verifies:
Ticket actually exists.

        ↓

User receives:
"Replacement request created successfully.
Ticket: INC-29482"

The AI should not simply claim that something happened.

The system should verify that it actually happened.

19. Evaluation Must Go Beyond "The Answer Looks Good"

This is another area where AI prototypes and production systems differ dramatically.

A developer tests:

"Ask the chatbot ten questions."

If the answers look good, they conclude:

"The RAG system works."

That's not enough.

Enterprise AI needs systematic evaluation.

For retrieval:

Did we retrieve the correct source?
Did we retrieve enough relevant information?
Did we retrieve unauthorized information?
Did ranking put the best evidence first?

For generation:

Is the answer grounded?
Is it relevant?
Did it introduce unsupported claims?
Did it cite the correct evidence?

For agents:

Did it choose the correct tool?
Did it use the correct arguments?
Did the tool succeed?
Did it recover from failure?
Did it stop when the task was complete?

For business workflows:

Was the actual task completed?
Was policy followed?
Was authorization respected?
Was the final state correct?

The evaluation target therefore becomes:

Model Quality
      +
Retrieval Quality
      +
Tool Reliability
      +
Policy Compliance
      +
Task Completion

This is much closer to traditional software testing.

20. Observability Is Not Optional

In a traditional backend application, when something fails, you inspect:

Logs
Metrics
Traces
Database State

AI systems need the same discipline.

Suppose a user receives a wrong answer.

You should be able to reconstruct:

User Query
    ↓
Detected Intent
    ↓
Retrieved Sources
    ↓
Ranking
    ↓
Context Sent to Model
    ↓
Model Decision
    ↓
Tools Called
    ↓
Tool Arguments
    ↓
Tool Results
    ↓
Policy Checks
    ↓
Final Response

Without this information, debugging becomes:

"The AI gave a weird answer."

That's not engineering.

A production system should make the AI's execution trace inspectable.

21. AI Security Is Bigger Than Prompt Injection

Prompt injection gets a lot of attention—and rightly so.

But enterprise AI security is much broader.

Consider:

User Input
    ↓
Prompt Injection
    ↓
Retrieval
    ↓
Sensitive Data
    ↓
Tool Call
    ↓
External System

Potential risks exist at every stage.

You need to consider:

authentication
authorization
data isolation
prompt injection
sensitive data exposure
tool permissions
API credentials
malicious retrieved content
unsafe tool arguments
excessive autonomy
auditability
output validation

A useful principle is:

Never treat the LLM as a trusted security boundary.

The LLM can reason.

The application must enforce security.

22. Cost and Latency Become Architecture Problems

A demo can take 15 seconds to answer.

A production customer-support assistant may not have that luxury.

Imagine a single request triggers:

Query Rewrite
 ↓
Vector Search
 ↓
Keyword Search
 ↓
Reranking
 ↓
LLM Call
 ↓
SQL Query
 ↓
Another LLM Call
 ↓
API Call
 ↓
Validation
 ↓
Final LLM Call

The system may be accurate.

It may also be painfully slow and expensive.

Therefore production AI engineering involves trade-offs:

Accuracy
    ↕
Latency
    ↕
Cost
    ↕
Reliability

Sometimes a smaller model is sufficient.

Sometimes deterministic code is better.

Sometimes retrieval can be skipped.

Sometimes caching makes more sense.

Sometimes an agent should be replaced with a fixed workflow.

The best architecture is not the one with the most AI.

It is the one that solves the business problem with the right amount of AI.

23. The Evolution of Enterprise AI

We can now summarize the architectural evolution.

Stage 1 — LLM

User
 ↓
LLM
 ↓
Answer

Good for:

writing
summarization
brainstorming
general reasoning

Problem:

The model doesn't automatically know enterprise knowledge.

Stage 2 — RAG

User
 ↓
Retriever
 ↓
Enterprise Knowledge
 ↓
LLM
 ↓
Answer

Solves:

"How do we give the model private knowledge?"

But not:

"How does the system operate inside the business?"

Stage 3 — Advanced RAG

Query
 ↓
Query Understanding
 ↓
Hybrid Search
 ↓
Filtering
 ↓
Reranking
 ↓
Context Selection
 ↓
LLM

Improves knowledge access.

Still primarily focused on answering.

Stage 4 — Agentic RAG

User
 ↓
Planner
 ↓
Retrieve
 ↓
Evaluate
 ↓
Retrieve Again
 ↓
Reason
 ↓
Answer

Retrieval becomes dynamic.

The system decides what information it needs.

Stage 5 — Tool-Using AI

AI
 ├── Search
 ├── SQL
 ├── CRM
 ├── ERP
 ├── Email
 └── Internal APIs

The system can now perform operations.

Stage 6 — Stateful AI

AI
 +
Memory
 +
Workflow State
 +
History

The system can participate in long-running workflows.

Stage 7 — Governed Enterprise AI

Models
+
Knowledge
+
Tools
+
Memory
+
Permissions
+
Policies
+
Human Approval
+
Observability
+
Evaluation

Now we are getting closer to production enterprise AI.

24. What Should a Software Engineer Actually Build?

If you're coming from a software engineering background and want to move into AI engineering, don't start by memorizing every AI framework.

Start by learning how to design systems.

A practical progression looks like this:

Software Engineering Fundamentals
        ↓
APIs + Databases + Authentication
        ↓
LLM APIs
        ↓
Embeddings + Retrieval
        ↓
RAG
        ↓
Evaluation
        ↓
Tool Calling
        ↓
Agents / Orchestration
        ↓
Memory / State
        ↓
Security + Governance
        ↓
Production AI Systems

This path is much more valuable than simply learning:

Framework A
Framework B
Framework C

because frameworks change.

Architecture principles remain.

25. A Practical Enterprise AI Project

If I were building a serious AI project to learn these concepts, I wouldn't build another:

"Chat with PDF."

It's useful for understanding RAG.

But it doesn't demonstrate enough engineering depth.

Instead, build something closer to:

AI IT Support Engineer

Imagine an internal assistant for a company.

A user can say:

"My laptop is slow. Check whether my device is under warranty and tell me what I should do."

The system can:

User
 ↓
Intent Detection
 ↓
Employee Authentication
 ↓
Retrieve Device Information
 ↓
Query Asset Database
 ↓
Retrieve Warranty Policy
 ↓
Reason About Eligibility
 ↓
Respond

Then extend it:

"Create a support ticket."

Now:

User
 ↓
AI
 ↓
Check Permission
 ↓
Create Ticket via API
 ↓
Verify Ticket
 ↓
Store Workflow State
 ↓
Return Ticket ID

Then extend it again:

"What happened to my ticket?"

Now the system retrieves:

Current Ticket State
+
Previous Actions
+
Relevant Policy
+
Conversation Context

At this point you've built something much closer to a real AI system.

And you've learned:

RAG
APIs
authentication
authorization
tool calling
state management
evaluation
observability
business workflows

That is AI engineering.

26. The Architecture Should Follow the Business Problem

This is probably the single most important lesson from all of this.

Don't start with:

"I want to build an agent."

Start with:

"What problem am I solving?"

Don't start with:

"Which vector database should I use?"

Start with:

"Where does the authoritative information live?"

Don't start with:

"Which LLM is the smartest?"

Start with:

"What capabilities does this workflow actually require?"

Don't start with:

"How autonomous can I make the system?"

Start with:

"Which decisions can safely be automated?"

And don't start with:

"How do I make the demo impressive?"

Start with:

"How do I make the system reliable?"

27. RAG Is Not Dead

After everything we've discussed, it would be easy to conclude:

"RAG is outdated."

That's the wrong conclusion.

RAG is not going away.

It is becoming a component.

The architectural shift is:

Old Mental Model

Enterprise AI
     =
LLM + Vector Database

to:

Modern Mental Model

Enterprise AI
     =
Model
+
Knowledge
+
Retrieval
+
Tools
+
Data
+
Memory
+
Policies
+
Workflows
+
Observability
+
Evaluation

RAG remains one of the most important ways to provide contextual knowledge.

But it no longer carries the entire responsibility.

28. The Real Evolution: From Answers to Outcomes

This is where the story ultimately comes together.

Early AI applications were primarily designed around:

"Give me an answer."

RAG improved that:

"Give me an answer based on my company's knowledge."

Agentic systems push further:

"Figure out what needs to happen."

Tool-using systems go further:

"Do it."

Stateful systems add:

"Remember where we are in the process."

Governed systems add:

"Do it within the rules."

Production systems add:

"Prove that it actually worked."

So the evolution is:

Answer
  ↓
Grounded Answer
  ↓
Reasoned Decision
  ↓
Action
  ↓
Stateful Workflow
  ↓
Governed Automation
  ↓
Verified Business Outcome

That is the real evolution of enterprise AI.

29. The Software Engineer's Advantage in AI

There is an interesting misconception that becoming an AI engineer means leaving software engineering behind.

I don't think that's true.

In fact, strong software engineering fundamentals become even more valuable.

Because production AI still needs:

clean APIs
database design
authentication
authorization
caching
queues
retries
rate limiting
error handling
testing
logging
monitoring
deployment
scalability
security

The difference is that now one component of the system is probabilistic.

And that creates a new engineering challenge.

Traditional software usually aims for:

Input → Deterministic Logic → Output

AI systems often look more like:

Input
 ↓
Probabilistic Reasoning
 ↓
Tool / System Interaction
 ↓
Validation
 ↓
Controlled Output

So the engineer's job becomes designing the boundaries around that probabilistic component.

That is why I believe:

The future AI engineer will not be less of a software engineer. They will need to be more of one.

30. What Production AI Engineering Really Means

A production AI engineer does not simply know how to call an LLM API.

They think about the complete system.

They ask:

What is the business objective?

Where is the source of truth?

What information does the model need?

What should be retrieved?

What should be queried directly?

What actions can the AI perform?

Who is authorized to perform them?

What happens if a tool fails?

What happens if the model is wrong?

What happens if the retrieved document is malicious?

What happens if the agent gets stuck?

What happens if the API times out?

How do we verify the result?

How do we evaluate the system?

How do we trace a failure?

How do we control cost?

How do we scale it?

These are not "prompt engineering" questions.

They are systems engineering questions.

And that is exactly where enterprise AI becomes interesting.

Conclusion

RAG changed enterprise AI.

It solved a fundamental problem:

How can an LLM access knowledge that isn't contained in its training data?

But enterprises eventually need more than knowledge.

They need systems that can:

retrieve information
reason over it
access live data
respect permissions
call enterprise tools
maintain workflow state
follow business policies
recover from failures
verify actions
and produce measurable business outcomes

That's why:

LLM + Vector Database

is not an enterprise AI architecture by itself.

A more realistic architecture is:

                          Enterprise AI
                               │
           ┌────────────────────┼────────────────────┐
           ↓                    ↓                    ↓
       Knowledge              Actions              State
           │                    │                    │
         RAG                  Tools               Memory
           │                    │                    │
           └────────────────────┼────────────────────┘
                               ↓
                           Orchestration
                               ↓
                            Reasoning
                               ↓
                           Governance
                               ↓
                           Verification
                               ↓
                         Business Outcome
                               ↓
                      Observability + Evaluation

RAG isn't disappearing.

It is becoming one layer of something much larger.

The real evolution of enterprise AI is not:

RAG → Replace RAG

It is:

Retrieval
   ↓
Reasoning
   ↓
Action
   ↓
State
   ↓
Governance
   ↓
Verification
   ↓
Reliable Business Workflow

And perhaps the most important shift is this:

The question is no longer "How do I build a better RAG chatbot?"

The better question is "What business outcome should this AI system reliably accomplish, what information and tools does it need, what can go wrong, and how will I prove that it worked?"

That is the point where building AI stops being about making an impressive demo.

It starts becoming engineering.

Key Takeaways

RAG is a knowledge-access mechanism, not a complete enterprise AI architecture.
Retrieval does not automatically provide business context, permissions, live state, or actions.
Structured data should often be accessed through SQL or APIs rather than semantic retrieval.
Advanced RAG improves retrieval quality through techniques such as hybrid search, filtering, reranking, and query transformation.
Agentic systems make retrieval and tool selection dynamic rather than completely predetermined.
Tool calling allows AI systems to move from answering questions to performing business operations.
Enterprise memory is often better understood as workflow state, not merely conversation history.
Knowledge graphs can complement vector retrieval when relationships between entities matter.
Authorization must be enforced by the application architecture, not delegated to the LLM.
Autonomous agents need limits around tools, steps, runtime, cost, and permissions.
Deterministic workflows are often better than agents when the process itself is deterministic.
Production AI requires verification—an AI saying an action happened is not proof that it actually happened.
Evaluation should measure retrieval quality, groundedness, tool execution, policy compliance, and task completion.
Observability is essential for debugging and improving AI systems.
Cost, latency, security, reliability, and scalability are architectural concerns—not afterthoughts.
The future enterprise AI stack is closer to:

Models
+
Knowledge
+
Retrieval
+
Tools
+
Data
+
Memory
+
Policies
+
Workflows
+
Evaluation
+
Observability

Final Thought

A vibe coder asks: "Which AI tool can I plug in?"

An AI engineer asks: "What problem am I solving, what system should own the truth, what can the AI do, what must it never do, and how will I know when it is wrong?"

The difference isn't the model.

It's the engineering.

About the Author

Hi, I'm Rajshree, a Software Engineer and Full Stack Developer passionate about building modern web applications and exploring the intersection of software engineering, AI, machine learning, and intelligent systems.

I enjoy turning ideas into working products, understanding how systems behave beyond the demo stage, writing about what I learn, and continuously exploring the transition from traditional software development to production-grade AI engineering.

🌐 Portfolio: https://rjshree.com

💼 LinkedIn: https://linkedin.com/in/rjshree

💻 GitHub: https://github.com/rjshree

If you enjoyed this article, follow along for more writing on software engineering, AI, technology, system architecture, and the journey from developer to AI engineer.

Thanks for reading.

OpenSparrow v3.2 - Per-User Access Control, Materialized Views in RAG, and a Business Design Refresh

Tomasz — Thu, 13 Aug 2026 14:22:39 +0000

OpenSparrow v3.2 is a security‑focused and consistency‑focused release that strengthens access boundaries, improves AI‑driven aggregate queries, and unifies the entire admin and frontend experience under one design system.

This update introduces per‑user access control across all major scopes, brings materialized‑view support to RAG aggregate queries, and delivers a full visual refresh that aligns every UI surface with the new Business design language.

Below is a detailed breakdown of the changes, including how they work, why they matter, and what they unlock for real deployments.

Per-user access control

New Users → Access tab. Each user gets independent allow‑lists across five scopes — tables, views, printouts, boards, workflows — stored as user_table_access in config.

The default is unrestricted: an absent or empty list means “no restriction”, not “no access”. To fully cut a user off, deactivate the account instead.

Enforced at the API boundary by default, not just hidden in the UI — os_api_action() / os_api_dispatch() gate table, view, print, board, and workflow endpoints, and FK/file/comment/mass‑edit/data‑cleanup endpoints were extended to respect the same restrictions
Granting a board or workflow does not implicitly grant the tables it touches — a board whose table isn’t granted won’t show up, and a workflow drops if any step targets a table the user can’t reach
Hidden tables (no menu entry, no grid) can’t be ticked and are granted automatically
Delegated FK lookups and the schema endpoint are now scoped to what the requesting user can actually access, and the file listing is filtered by record ownership

RAG: materialized views as aggregate views

The aggregate‑view feature from 3.1 (attach a PostgreSQL view to a table so the AI assistant can answer exact “total/how many” questions) now also accepts materialized views, flagged as such in the admin UI — useful when the aggregate query is expensive enough that you’d rather refresh it on a schedule than recompute it on every chat message.

Views sync also discovers materialized views alongside regular ones.

A unified “Business” look

The admin panel, setup wizard, and frontend were unified onto a single Business 4+1 color palette and one system‑ui font stack, replacing a mix of ad-hoc colors and font declarations across dozens of JS/CSS files.

Every front‑end button now shares one height, weight, border radius, and font size instead of drifting per page.

Also in this release

CRM demo: richer install description, plus optional demo‑user and audit‑history install toggles
Release packaging excludes creator/, dist-local/, and storage/tmp/ from image and release archives
Admin docs updated to match actual current module behavior across several sections

Websites

opensparrow.org

Discussion

I encourage a substantive discussion, and I'll do my best to answer any questions regarding implementation and configuration.