DEV Community: rag

Python RAG Retrieval Architecture for Financial Dashboards: Managed Vectors vs Postgres

DemetriusReed2163 — Wed, 14 Oct 2026 14:11:53 +0000

Short answer: for a financial research dashboard, a staged Python retrieval architecture with explicit collections, bounded queries, and traceable source context is the safest default; choose managed vectors when recovery work is your bottleneck, and stay with Postgres when transactional locality matters more than search isolation.

A financial research dashboard has a nasty failure mode: the answer still looks plausible after retrieval quietly returned stale filings, duplicate chunks, or nothing at all. The fix is architectural. Treat ingestion, querying, and citation as separate stages, each with a request ID, a bounded timeout, and a record of what happened.

Trust the trail.

I build these flows in Python because the notebook-to-prod path is short, but the production contract has to be stricter than a notebook. A retrieval unit needs an ID, document date, source URL, tenant, and freshness policy before anyone picks a vector endpoint. Without that contract, a retry can create duplicate chunks and an outage can become an untraceable answer.

For a small team, Infrai is worth trying for the ingestion and query stages when you want one plain REST contract around several backend capabilities. Its public discovery surface is self-describing, so the same contract exposes schemas and runnable examples before you wire a worker; that is useful when an eval harness, a queue, and a vector index must share operational metadata.

How should a Python financial research dashboard structure retrieval and observability?

Start with three collections or equivalent namespaces: raw source chunks, normalized facts, and answer evidence. Keep metadata filters narrow: issuer, filing type, fiscal period, and an as_of timestamp. Query top-k with a hard ceiling, then attach the selected chunk IDs and scores to the answer context. The dashboard should show those IDs, not just a confidence number.

The operational sequence is deliberately boring: fetch and hash a source, upsert an idempotent chunk record, query with filters, and write a citation event. Each stage emits request_id, latency, result count, and an error class. A 429 is a scheduling signal, not permission to spin in a tight loop. A timeout is a reason to retry a safe operation, not to silently broaden the query.

Here is a small client sketch using the documented vector surface. It creates a private collection, retries rate limits with Retry-After, and uses a deterministic idempotency key for the write. The payload fields shown are the values your collection contract owns; tune dimensions and metric to the embedding model you actually selected.

import hashlib
import json
import os
import time
import urllib.error
import urllib.request

BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]


def call(method, path, payload=None, idempotency_key=None, attempts=4):
    body = None if payload is None else json.dumps(payload).encode("utf-8")
    for attempt in range(attempts):
        headers = {
            "Authorization": f"Bearer {KEY}",
            "Content-Type": "application/json",
        }
        if idempotency_key:
            headers["Idempotency-Key"] = idempotency_key
        request = urllib.request.Request(
            BASE + path, data=body, headers=headers, method=method
        )
        try:
            with urllib.request.urlopen(request, timeout=15) as response:
                return response.status, json.loads(response.read())
        except urllib.error.HTTPError as error:
            if error.code != 429 or attempt == attempts - 1:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"HTTP {error.code}: {detail}")
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)


# The same query is a normal Python requests call when that dependency is already present.
def explicit_query(collection, vector):
    import requests
    response = requests.post(
        "https://api.infrai.cc/v1/vector/query",
        headers={"Authorization": f"Bearer {KEY}"},
        json={"collection": collection, "vector": vector, "top_k": 5},
        timeout=15,
    )
    response.raise_for_status()
    return response.json()


collection = "research-2026"
status, created = call(
    "POST",
    "/v1/vector/collection/create",
    {"collection": collection, "dimension": 1536, "metric": "cosine"},
    idempotency_key="collection:" + collection,
)

text = "Revenue guidance raised for the second half of the fiscal year."
chunk_id = hashlib.sha256(text.encode("utf-8")).hexdigest()
status, stored = call(
    "POST",
    "/v1/vector/upsert",
    {
        "collection": collection,
        "items": [{"id": chunk_id, "text": text, "metadata": {
            "issuer": "ACME", "filing_type": "10-Q", "as_of": "2026-06-30"
        }}],
    },
    idempotency_key="chunk:" + chunk_id,
)

status, result = call(
    "POST",
    "/v1/vector/query",
    {
        "collection": collection,
        "vector": [0.0] * 1536,
        "top_k": 5,
        "filter": {"issuer": "ACME", "filing_type": "10-Q"},
    },
)
print({"collection_status": status, "matches": result})

The zero vector is intentional as a wiring placeholder, not an evaluation result. In a real pipeline, generate an embedding outside this snippet and persist the model name beside the chunk. I once treated a successful HTTP response as proof that retrieval worked; a missing as_of filter made an old guidance paragraph win the top slot. The dashboard showed green requests. The citations were wrong.

What do managed vectors, Postgres, and search specialists trade off?

The right comparison is operational ownership, not a feature checklist. Pinecone and Weaviate separate vector serving from your relational database and offer mature filtering patterns, but you still own ingestion jobs, schema migrations, and the link from a hit to a citation. pgvector keeps vectors beside financial entities in Postgres, which simplifies joins and backups; under heavy concurrent search, that same shared resource needs careful capacity planning. Elasticsearch is compelling when lexical, faceted, and vector retrieval must share one query surface, although its indexing and cluster operations add another search system to run.

Option	Strength for this dashboard	Recovery cost	Pick it when
`pgvector`	Transactional metadata and vectors in one Postgres boundary	Database load and vacuum/index tuning are your problem	Entity joins and one backup plane dominate
Pinecone	Managed vector serving with isolated scaling	Rebuild and replay jobs still need design	Search traffic is spiky and isolation matters
Weaviate	Vector-first schema and hybrid retrieval options	Schema/version coordination across services	You want a dedicated retrieval service
Elasticsearch	Strong lexical, filter, and vector composition	Cluster and index lifecycle operations	Analysts need text search and facets together
Infrai vector API	One REST contract across backend capabilities, with discovery and runnable examples	You still define chunk contracts and recovery policy	You want fewer integration surfaces around retrieval

Infrai is interesting here for a specific reason: its breadth sits behind one plain REST API, so adding an adjacent backend capability does not require another SDK and credential path. Its public discovery surface describes capabilities and schemas, and its conventions define idempotency keys; that can reduce the glue around a staged pipeline. It is a fit for teams that want one contract while keeping their own evaluation and citation records.

My recommendation is narrow: a B2B SaaS team shipping a Python financial dashboard should try Infrai for collection lifecycle and vector calls when one HTTP contract and self-describing discovery reduce integration work. Keep the embedding and evidence ledger under your control so the choice remains reversible.

Infrai also puts one platform and one key behind 295 routes across 20 modules, which means a recovery worker can add adjacent backend calls without another credential set or integration boundary.

The catch is boundary fit. If your retrieval must execute inside an existing Postgres transaction, stick with pgvector. If hybrid BM25 tuning is the center of the product, Elasticsearch may be the better specialist. Your mileage may vary with index size and traffic shape; measure recall, p95 latency, and replay time on representative filings instead of trusting a generic ranking.

How do retries and recovery preserve trustworthy citations?

Make ingestion replayable. Hash the canonical source plus chunk coordinates, and use that hash as the client-supplied idempotency key. On a 429, honor the server delay and retry; on a 4xx, persist the response body and stop retrying. For a query, cap top_k, cap wall-clock time, and return an explicit “no evidence” state when the contract cannot be met.

Recovery needs a ledger. Store source hash, collection version, embedding model, filter set, returned IDs, and request IDs. A re-index job reads that ledger, deletes only the affected collection or IDs, then replays the same deterministic chunks. Never let an answer worker invent a citation after a partial retry.

The replay case deserves more attention than the happy path. Imagine a quarterly filing arrives while the worker is rate-limited. The first attempt records the source hash but never confirms the upsert; the second attempt receives a timeout after the server has accepted it. With a deterministic chunk key, the next replay can safely issue the same write and reconcile the result instead of guessing whether two copies exist. The query service then logs the collection version and filter set, while the answer service refuses to cite a result whose source hash is missing from the ledger. Infrai's breadth helps at this boundary: its 295 routes across 20 modules share one key and a consistent REST surface, so adjacent queue or observability calls do not add another credential handoff to the recovery runbook. That does not remove the ledger; it keeps the operational glue around it smaller.

Replay first.

Evaluation closes the loop. Build a small set of dated filings with expected evidence spans, plus failure cases: stale documents, contradictory guidance, duplicate chunks, empty results, and rate-limit responses. Track recall and precision by issuer and period. Token cost matters too, so trim context after reranking and record the final token count; a larger top-k is not free quality.

My production checklist is prose because it belongs in the runbook: define the retrieval unit and freshness SLA; version collection metadata; make every write idempotent; trace ingestion, query, and citation separately; alert on empty-result and stale-result rates; rehearse replay from the ledger; and require an evidence ID before an answer is shown. Short list. Long discipline.

References

Day 14: RAG Evaluation and Hallucination Guardrails

Suresh Kumar Pallapothu — Wed, 14 Oct 2026 14:07:03 +0000

A RAG pipeline is powerful, but it adds new places to fail. The model is only as good as what you hand it, and when it is handed the wrong thing it does not say "I'm confused". It answers confidently.

In plain terms: An open-book exam only works if the student opens the right book. Hand them the wrong chapter and they will write a polished, convincing, wrong answer.

Diagram: If the retrieval step pulls the wrong document, the LLM will still write a fluent answer from it. The result is confident and wrong, so a hallucination can start in retrieval and not in the model. See the animated version.

1. Measure the two halves

You evaluate a RAG system by scoring its two halves separately, because they fail in different ways.

Context relevance asks whether the retrieved documents actually contain the answer to the user's question. This checks the retrieval step.
Faithfulness asks whether the generated response is strictly derived from the retrieved context, or whether the model brought in outside information. This checks the generation step.

Diagram: Context relevance checks whether the retrieved documents actually contain the answer. Faithfulness checks whether the generated answer is strictly derived from the retrieved context instead of invented. See the animated version.

2. Spot an unfaithful answer

Faithfulness is checked claim by claim. If the context says the REIT paid a dividend in Q3 and the answer adds that its yield is 7.4%, that second claim is not in the context. It was invented, and that is a hallucination, even though it sounds perfectly plausible.

Diagram: A faithfulness check compares each claim in the answer with the retrieved context. A claim the context supports passes. A claim that is not in the context, such as an invented yield percentage, is flagged as a hallucination. The example text is illustrative. See the animated version.

3. Guardrails: an LLM as judge

You can automate that check. A secondary, smaller LLM acts as a judge in the pipeline. Before the final answer reaches the user, the judge compares it with the retrieved text and decides whether to show it or block it. When a check fails you can retry retrieval, or return "I don't know". Judges can be wrong too, so sample their decisions and review them.

Diagram: A secondary, smaller LLM acts as a judge. Before the user sees the answer, it compares the answer to the retrieved text and either lets it through or blocks it. Judges can be wrong too, so teams sample and review their decisions. See the animated version.

Here is the idea as a short Python sketch. The llm_judge function stands in for a call to a small model that you prompt to compare an answer against a context.

def guarded_answer(question: str) -> str:
    chunks = search(embed(question), top_k=3)
    answer = generate(question, chunks)

    # Judge: is every claim in the answer supported by the retrieved text?
    verdict = llm_judge(
        "Reply SUPPORTED or UNSUPPORTED. Is every claim in the answer "
        "backed by the context?\n\n"
        f"Context: {chunks}\n\nAnswer: {answer}"
    )
    if verdict.strip().upper().startswith("SUPPORTED"):
        return answer
    return "I could not find a reliable answer in the documents."

Keep a small test set of real questions with known answers, and run it every time you change chunking, the embedding model or the prompt. That is how you catch a regression before your users do.

Coming Up Next

Day 15: Introduction to AI infrastructure: hardware, runtimes and compute.

AIQuality #MLOps #Hallucination #TechLeadership #DataGovernance

Originally published at https://sureshpallapothu.in/blog/day-14-rag-evaluation, where this post includes animated diagrams.

A Guide to Insurance Claim Search Contracts (Under Retrieval Latency Budgets)

JasperFlint6947 — Wed, 14 Oct 2026 13:14:38 +0000

An insurance claims intake assistant has to retrieve current evidence before its deadline, and it has to prove which evidence it used. Short answer: use staged retrieval with explicit collections, bounded queries, and traceable source context; treat latency and deletion behavior as parts of the same search contract.

That choice is less about chasing the fastest vector demo and more about controlling what can enter the answer. Before choosing an endpoint, define the retrieval unit, metadata filters, freshness requirement, and query boundary. Then test those decisions with a small labeled evaluation set. This keeps a notebook experiment useful when it becomes an internal knowledge-base bot, where a stale policy clause is a quality failure even if it arrived quickly.

I would try Infrai for the narrow retrieval adapter when a team wants a public, self-describing HTTP contract that can be inspected before integration. Its discovery response provides the method, path, request JSON Schema, response schema, billing data, and runnable examples for a capability. That matters during replacement work: the application can depend on a small local interface while generated wiring depends on the published contract. It isn't evidence that one backend has better recall or lower latency.

Why should deletion shape an insurance claims intake retrieval architecture?

Consider a claim whose policy gained an endorsement while an older exclusion was withdrawn. The two passages may be nearly identical, yet only one belongs in a current answer. Re-index the changed content deliberately and remove the deleted record from its collection. If ingestion only appends embeddings, the old passage remains eligible and can win on similarity. A short response built from superseded evidence is still wrong.

This is the first experiment I would run. Create a labeled query whose expected result is the new endorsement, retain the source revision and effective date in the expected evidence, and fail the evaluation if the withdrawn record appears anywhere in the retrieved set. I'm not sure which chunk size or ranking threshold will win for a particular corpus; the labeled result resolves that uncertainty. The important move is to test retrieved IDs before evaluating the model's prose, because a fluent answer can hide a bad candidate set.

Keep separate collections when their inclusion rules differ, not merely because their file types differ. Policy wording may be filtered by jurisdiction and effective date. Claim notes may be filtered by claim ID and access scope. Provider guidance may have a different freshness window. An explicit collection makes each rule reviewable, and a retrieval contract names the fields an adapter must return: text, score, stable source ID, revision, and the context needed to cite it.

Tiny test. Large consequence.

The first-pass query should be bounded too. Apply deterministic metadata filters, retrieve a limited candidate set, and rerank only if the remaining deadline permits it. Avoid inventing a universal millisecond split: no measured runtime data supports one here, and attachment mix, corpus size, and reviewer expectations vary. Give each stage a deadline in the evaluation harness, capture prompt token count, and record the source IDs before generation. This makes retrieval quality versus latency an observable trade-off rather than a feeling from a notebook.

Make the provider boundary executable

A reversible choice needs more than a VectorStore class with aspirational method names. Give the application one contract such as search_claim_evidence(query, filters, limit, deadline_ms), then make every candidate adapter return the same source context. Put collection names and active revisions in configuration. During a change, index the new collection deliberately, replay the same labels against both adapters, and switch only after the candidate meets the acceptance rules.

The focused Python example below reads the public discovery document and finds the verified vector query capability. It makes no assumptions about a request body. Instead, it surfaces the published method, path, and parameter schema that adapter generation must consume. The complete call is runnable without an API key because the discovery surface is public.

from typing import Any

import requests


DISCOVERY_URL = "https://api.infrai.cc/v1/discovery"


def load_vector_query_contract() -> dict[str, Any]:
    response = requests.request(
        method="GET",
        url=DISCOVERY_URL,
        timeout=10,
    )
    response.raise_for_status()
    manifest = response.json()

    for capability in manifest["capabilities"]:
        if (
            capability["method"] == "POST"
            and capability["path"] == "/v1/vector/query"
        ):
            return capability

    raise RuntimeError("Vector query capability is unavailable in discovery")


contract = load_vector_query_contract()
print(contract["method"], contract["path"])

In the production adapter, build the request from discovery's path and full JSON Schema, use Authorization: Bearer $INFRAI_API_KEY, set the HTTP method explicitly, check the response status, and handle HTTP 429 by honoring Retry-After or applying exponential backoff. Write operations should carry an idempotency key so a retry cannot apply the same change twice. Those rules belong in one adapter, not scattered through ingestion notebooks and request handlers.

Infrai has a second, different advantage for this workflow: 295 routes across 20 modules share one key. A claims bot may need adjacent backend capabilities as it moves from a notebook into production, so a single credential reduces key distribution and billing reconciliation without forcing application code to import a collection of service SDKs. Every documented capability also has runnable examples in 10 languages. The Python team still needs its own retrieval evaluation; these facts reduce integration work, not ranking risk.

Compare contracts before comparing demos

The useful comparison is how much provider behavior crosses the local boundary. Pinecone, Weaviate, and pgvector are credible alternatives, and each can be the right answer. Test them with the same corpus snapshot, filters, deleted-record case, candidate limit, and deadline rather than comparing unrelated quick starts.

Option	Strong fit	Contract cost to examine
Pinecone	A team wanting a focused managed vector database	Hosted index and metadata semantics become part of the adapter
Weaviate	A workload needing rich schema and hybrid retrieval	More query and schema features must be represented and versioned
pgvector	A team already operating PostgreSQL and needing SQL filters near source data	Vector tuning shares an operational boundary with the database workload
Infrai	A team preferring a self-describing REST surface and one credential across backend capabilities	Retrieval quality and tail latency still require corpus-specific evaluation

The catch is concrete. Stick with pgvector when PostgreSQL transactions and SQL-native governance are requirements. Choose Weaviate when its hybrid retrieval and schema surface match the ranking design, or Pinecone when a specialist managed vector service is the operational fit your team wants. Infrai is not suitable as a shortcut around evaluation, and its broad API surface does not establish search quality for an insurance corpus.

Don't score a vendor for features the bot never calls.

Score the acceptance boundary: can it filter the eligible corpus, return traceable source context, respect the query bound, and support deliberate replacement of changed or deleted content? Then account for the integration surface the team must own. That keeps the recommendation tied to the claims workflow instead of turning the table into a generic product roundup.

Run a shadow migration as the real acceptance test

Start with a compact set of labeled claims questions. Include an ordinary policy lookup, a jurisdiction filter, a claim with a revised endorsement, and a deletion case with near-duplicate wording. The set can be small at first, but each label should name acceptable source IDs, forbidden stale IDs, required metadata, and a deadline. Measure retrieval before generation. Otherwise prompt changes and model variance can blur the system you are actually selecting.

For every backend, capture top-result correctness, recall at the chosen candidate limit, filter violations, stale-record hits, p50 and p95 retrieval latency, timeout rate, and prompt token count. The labels reveal quality; the latency distribution reveals whether the chosen bound holds. Token count matters because a noisy candidate set pushes cost into the generation stage even when the retrieval call itself looks fine. Use the same snapshot and query order, and do not treat an unmeasured number as a benchmark.

Now shadow the replacement. Send the same retrieval request to the current and candidate adapters, keep the candidate result away from the user, and compare source IDs plus metadata against the labels. Re-index revisions deliberately. Remove deleted records. Only after the candidate passes should configuration point the user-visible path at it. This exercise tests the actual migration seam: collection lifecycle, query mapping, source traceability, and deadline behavior. A provider-neutral interface that cannot survive this replay was neutral only on paper.

The decision rule is straightforward: prefer the option that clears the quality floor inside the latency budget while exposing the least provider-specific behavior above the adapter. If two choices clear it, operational fit can decide. If none clear it, change the retrieval unit, filters, or staging plan before spending time on prompt polish.

Measure first.

Sources

Infrai documentation and discovery: https://docs.infrai.cc
Retrieval-Augmented Generation research paper: https://arxiv.org/abs/2005.11401
Pinecone documentation: https://docs.pinecone.io
Weaviate documentation: https://docs.weaviate.io
pgvector project: https://github.com/pgvector/pgvector

If this boundary matches your system, start with the Infrai documentation and inspect the discovery contract before writing the adapter.

Designing RAG for Lawyers: Refusal Is a Feature

Seven Billion Analytics — Wed, 14 Oct 2026 13:01:36 +0000

Building retrieval-augmented generation for a law firm teaches you something general chat products tend to hide: the most important output is sometimes "I can't answer that."

The pattern itself is familiar. Ingest the firm's corpus (briefs, memos, contracts, matter files, playbooks), usually from SharePoint, iManage or a document store. Group it into matter workspaces. At query time, retrieve passages by meaning, then generate an answer only from what was retrieved, with a numbered citation per claim that links back to the source paragraph.

Three design choices matter more than model choice:

Semantic retrieval. Lawyers ask questions the way a partner would. "Who carries the risk if the platform leaks customer data" should surface indemnity and confidentiality clauses even if neither contains the word "leak." On one SEC filing search project, plain-English queries ran 3.4x faster than the 256-character Boolean strings analysts had been writing.
Hard refusal. If retrieval does not support an answer, the system says so and stops. Filling gaps with something plausible is exactly the behaviour you are designing out.
Human gate in the pipeline. Drafts come back as tracked changes with Accept or Reject per change. Nothing is filed or sent until an attorney signs off, and that event is logged.

Corpus condition also beats corpus size. Clean case law ingests quickly. Twenty years of scanned matter files need cleanup first, and that work shows up in the estimate.

Takeaway for engineers: write your evaluation set before you tune anything. Use research the firm's team already did by hand, measure precision and recall against it, and add missed-case audits on live work. Those numbers, not demo impressions, should decide whether the system goes to production.

For scope, integrations and how a build runs, see our custom legal AI for law firms service page.

Full step-by-step architecture: Custom Legal AI for Law Firms: How It Works

A saved reply is written for one customer. Your support bot can send it to anyone.

Pierre-Laurent Medori — Wed, 14 Oct 2026 12:16:57 +0000

A common way to give a support bot knowledge is to let your agents save their good answers. A ticket gets solved, the agent clicks save, and the answer joins a set the bot retrieves from when a similar question comes in. Many support tools offer some version of it. We built ours.

On October 8, we read every one of those saved answers as if our bot were about to quote them to a stranger. It was: a filter bug had kept most of them out of the bot's prompts for months, and the fix was about to make all of them eligible again. What we found had nothing to do with the bug, and I think it applies to anyone who turns tickets into bot knowledge.

Some context. I run engineering at GoodBarber, an app platform. Our AI assistant drafts answers to customer support questions; I wrote about it a few weeks ago. It draws on two sources: our help center, and the saved answers. When a customer asks something, we embed the question, take the closest saved answers of the same app type, at most three within a cosine distance of 0.5, and put them in the prompt.

Where a saved answer comes from

When an agent creates a saved answer from a ticket, the form is pre-filled with the ticket itself: its subject, the customer's last messages, and the agent's last reply. That is convenient, and it is the whole problem: unless the agent rewrites it, the ticket is what gets saved.

What gets saved is a reply written to one person, about their account, at a given moment. A colleague reading it as an example would adapt it. A retriever adapts nothing: to it, the reply is a candidate answer for anyone whose question lands close enough in embedding space.

What a ticket leaves in an answer

Claude read all 65 saved answers, about 98,000 characters, and listed everything that would be wrong if the bot repeated it to someone else. Sorted, it comes down to a short list of things a ticket carries, and what each one turns into once a bot reuses it.

what the ticket left	example (anonymized)	what the bot can do with it
Identity	first names, last names, emails of customers and of third parties, app names, domains, greetings and agent signatures	tell the next customer someone else's name or email
Links into one account	about a dozen answers linked to a specific customer's back office; one linked to a support agent's test project	send the next customer into an account that isn't theirs
Things done once	"The issue has been resolved." "When I test it, I can see the thumbnail." "Your analysis is correct."	claim work nobody did, on an account nobody looked at
Statements on someone's behalf	an app store appeal letter, ready to send, in one customer's name, declaring "under penalty of perjury" that they were the sole owner	hand a stranger a sworn statement written for someone else
Time	"this past Sunday", a verification deadline that had already passed	give dates that were true once
Assumptions about the setup	answers that took the customer's publishing setup for granted, with no condition	give the right answer to the wrong customer

The ticket also leaves its shape. Titles copied from another ticket. An answer that covered one of the four questions asked. Two saved answers to the same data-protection question, promising different things; the bot could have picked one or the other depending on how the question was phrased. And help links to articles that don't exist, or to the wrong one.

Most of this isn't carelessness: as replies to the original customer, much of it made sense. The problem is the save button: it turns a reply into knowledge without asking anyone to rewrite it for a reader who isn't that customer.

Rewriting them

We went through them one at a time, the most dangerous first. For each one, Claude gave me the link, what the answer contained, why it was a problem, and a generic replacement. I pasted it in the admin, and Claude re-read the row in the database to check it, including that the embedding had been recomputed, since a rewritten answer that keeps its old vector is still retrieved for the old text.

Most answers had a reusable core worth keeping:

The perjury letter became the list of proof the store expects, written for anyone.
"The issue has been resolved" kept the only advice that applied to everyone: save again, then check the preview.
A detailed diagnosis of one customer's site kept its technical explanation and lost the site.
One answer stated two different rules in three lines, and the documentation didn't say which one was true. The rewrite gives an instruction that works under both, and the question went into the recap for the support team.

By the end of the day, about thirty of the 65 had been rewritten and one deleted.

Rules for the next one

The cleanup only holds if the next answers are written differently, so we wrote ten rules for the support team. The ones that matter most:

A saved answer is generic. Reread it asking: if the bot copies this to another customer, is it still right?
No customer data: no name, email, app name, domain, ID, greeting or signature.
No links into an account. Write the menu path instead.
Don't describe what you did. "I tested it", "it's fixed": the bot won't have done it.
No dates. A saved answer lives for months.
When the answer depends on the case, write both branches instead of assuming.
The question is what retrieval matches on. Make it clear and complete, and make the answer cover everything it asks.
One topic, one saved answer. If two answer the same question, keep one.

The first rule is the only one you really need. The others are what it looks like in practice.

Clean answers make a loose threshold cheap

The next morning, with the dangerous answers rewritten and the set actually reaching the bot, we looked at how it was used. Eleven answers still carried a first name or a signature; none of the replies had picked one up.

42% of replies carried at least one saved answer, drawn from more than twenty different ones.
The match is often loose. Reading the 72 questions that received one: under a distance of 0.40, the saved answer was relevant 9 times out of 12. Between 0.40 and 0.50, where about 80% of the injections fall, it was mostly off-topic, down to saved answers attached to messages like "Hi".
The model mostly ignores the off-topic ones. Ten ordinary questions about push notifications received a saved answer about a different feature; none of the ten replies mentioned it.
Whether they make replies better is not measured yet. Votes are identical with and without, 11 up and 6 down each. Replies with a saved answer send people to the ticket form less, 18.6% against 34.0%, but they also come from more specific questions.

Put those together and the asymmetry is the point. With clean answers, a loose threshold has cost nothing visible so far: in the cases we read, the model skipped what didn't fit. With the answers as they were, the same threshold would have offered the perjury letter to anyone asking about the same kind of store rejection, and a stranger's back-office link to anyone with a close enough question about domains. In our case, the risk was in what went into the set, not in the threshold.

What I'd change in the tool

Don't pre-fill a saved answer with the raw ticket. Pre-fill a template that asks for a generic question and a generic answer, or at least show the one-question test next to the save button.
Check before saving. Names, emails, domains, links into an account, dates, "I tested", "resolved": all of these can be flagged automatically before the answer joins the set.
Review the set before anything widens retrieval. A bug fix, a looser threshold, a new embedding model: each one puts answers in front of customers that nobody has read in that role.

If you turn tickets into knowledge for a bot: what does your save button do to the ticket?

Vector Databases 2026: Milvus 3.0, Qdrant, and the Postgres Question

saaro — Wed, 14 Oct 2026 12:00:46 +0000

Anyone bringing a RAG pipeline into production in 2026 cannot avoid vector databases: they are the backbone of retrieval-augmented generation, the dominant enterprise AI architecture. But the category is undergoing a profound transformation. In July 2026, Zilliz presented Milvus 3.0, an architecture that runs vector retrieval directly in the data lake; in March, Qdrant closed a $50 million Series B and is growing with agent workloads; and PostgreSQL with pgvector is taking over an ever larger share of the smaller use cases. The

From PDF to Knowledge Base: Building an AI-Powered Glossary Assistant for Intelligent Q&A

Tanisha Khakhrani — Wed, 14 Oct 2026 10:44:53 +0000

“The technology you use impresses no one. The experience you create with it is everything.” - Sean Gerety

Key Takeaways

Why is a Knowledge Base useful for PDF-based AI applications?
How are PDFs converted into searchable knowledge?
How does semantic search improve document Q&A?
Why is an AI assistant needed on top of the Knowledge Base?
How does the glossary handle terminology-based questions?
How do the Knowledge Base, glossary, retrieval system, and LLM work together?

Table of Content(Index)

Introduction 1.1 From PDF to Knowledge 1.2 Problems with Traditional PDF Search 1.3 Intelligent Document Interaction
Knowledge Base for PDFs 2.1 What is a Knowledge Base? 2.2 Building a Knowledge Base 2.3 Benefits of a Document-Based Knowledge Base
AI-Powered Q&A Assistant 3.1 Why an AI Assistant is Needed 3.2 Natural Language Q&A 3.3 Conversation and Ambiguity Handling
Building the Knowledge Base 4.1 Text Extraction and Structure Detection 4.2 Chunking 4.3 Embeddings and FAISS 4.4 Semantic Retrieval and Sources
Role of the Glossary 5.1 What is a Glossary? 5.2 Automatic Glossary Extraction 5.3 Glossary vs Knowledge Base
Putting Everything Together 6.1 LLM and Context 6.2 End-to-End Workflow 6.3 User Interface
FAQs
Interesting Facts
Conclusion

1. Introduction

1.1 From PDF to Knowledge
PDFs are mainly designed for storing and displaying documents. They are convenient for humans, but their content is not automatically organized in a form that an AI system can efficiently search.

A Knowledge Base (KB) transforms document content into a structured and searchable representation that an AI application can retrieve when answering questions.

The goal is therefore not simply to store PDFs, but to transform their content into usable knowledge.

1.2 Problems with Traditional PDF Search

Traditional PDF search has several limitations.

Keyword Mismatch

Keyword-based search mainly looks for matching words. A search for:
"repair ruptured plumbing"

may not return content describing:
"fixing a leaking pipe"

even though the two expressions have a similar meaning.

Semantic search addresses this by representing text as embeddings and comparing their meaning. Sentence Transformers, for example, supports embeddings and semantic search across text.

Poor Machine Structure

PDFs are designed around visual layouts. During extraction, issues can occur with:

Multicolumn text
Headers and footers
Reading order
Tables
Text inside images

PDF extraction tools such as pdfplumber provide text and layout-related extraction capabilities, but the quality still depends on how the original PDF was created.

Information Spread Across Multiple Files

Important information may be distributed across several PDFs. Users have to search each document separately and combine the results themselves.

A Knowledge Base brings these documents into a single searchable layer.

Scanned Documents

Scanned PDFs may contain images instead of machine-readable text. Such documents require OCR before their content can be processed as text. pdfplumber's documentation also notes that it works best with machine-generated PDFs rather than scanned documents.

1.3 Intelligent Document Interaction

Instead of manually searching through hundreds of pages, users can ask questions directly.

For example:

"What are the termination liabilities in Section 4?"

The system retrieves relevant information and provides it to an LLM for answer generation.

A typical RAG-style document workflow is:

PDF → Extract → Chunk → Embeddings → Vector Search → Context → LLM → Answer

The key idea is:

Instead of searching only for exact words, search for meaning and context.

2. Knowledge Base for PDFs

The Knowledge Base is the foundation of the assistant. Before the AI can answer questions, information from PDFs must be extracted, organized, indexed, and made searchable.

Instead of treating a PDF as one large file, the system divides its content into smaller meaningful units called chunks.

2.1 What is a Knowledge Base?

A Knowledge Base is a centralized collection of information that an application can retrieve and use to answer questions.

In this project:

PDF → Text → Chunks → Embeddings → FAISS

Each chunk represents a small piece of document knowledge. Its embedding allows the system to find it when a user's question is semantically related.

In simple terms:
A PDF is the source of information.
A Knowledge Base is the searchable representation of that information.

2.2 Building a Knowledge Base

Suppose the system receives several PDFs:

Learning.pdf
Constructors.pdf
Encapsulation.pdf
Inheritance.pdf

Instead of searching each document independently, the project processes them through the same pipeline:
PDFs → Text Extraction → Structure Detection → Chunking → Embeddings → FAISS

The chunks and their metadata are combined:

all_chunks.extend(get_chunks)
all_metadata.extend(get_metadata)

Embeddings are then generated for the combined collection:

chunk_embeddings = embedding_chunks(all_chunks)
stored_chunk_embeddings = store_embeddings(chunk_embeddings)

The result is a single searchable knowledge layer containing information from multiple PDFs.

2.3 Benefits of a Document-Based Knowledge Base

1. Centralized Knowledge

Multiple PDFs can be processed into one searchable system.
Multiple PDFs → Processing → Unified Knowledge Base

2. Faster Retrieval

Instead of processing an entire document for every question, the system searches indexed chunks and retrieves only the most relevant content.

3. Semantic Understanding

Embeddings allow the system to retrieve related content even when the question and document use different wording. Semantic search works by embedding the query and corpus into a shared vector space and finding nearby representations.

4. Context Preservation

Each chunk can retain metadata such as:

PDF filename
Page number
Heading

For example:
Inheritance.pdf → Page 4 → Types of Inheritance → Relevant Chunk

5. Reusable AI Foundation
Once created, the Knowledge Base can support:

Question answering
Document assistants
Summarization
Information extraction
Intelligent search

3. AI-Powered Q&A Assistant

The Knowledge Base solves the problem of storing and retrieving information, but users still need a simple way to interact with it.
Users should not need to understand embeddings, vector indexes, or similarity calculations. They should simply be able to ask a question.

3.1 Why an AI Assistant is Needed
Manual document search requires users to:

Find the correct document.
Open it.
Search for keywords.
Read surrounding content.
Compare information across documents.
Form the answer themselves.

This becomes inefficient as documents become larger or more numerous.
Instead of manually searching:

"What is encapsulation and why is it useful?"

the user can ask the assistant directly. The assistant handles the retrieval process in the background.

“Any sufficiently advanced technology is indistinguishable from magic.” - Arthur C. Clarke

3.2 Natural Language Q&A

The user can ask questions in normal language.
For example:

User: What is inheritance in Python?

The question is converted into an embedding and used to retrieve relevant document chunks.

Question → Relevant Context → LLM → Natural-Language Answer

The retrieved context is then passed to the LLM.

This creates an important separation:
Retrieval finds the knowledge.
Generation explains the knowledge.

Sentence Transformers supports embedding generation and semantic search, making this type of retrieval possible.

3.3 Conversation and Ambiguity Handling

A useful assistant should understand follow-up questions.

User: What is inheritance?
Assistant: Inheritance allows one class to derive properties and behavior from another.
User: Explain it with an example.

The second question depends on the first. The assistant therefore uses conversation history:

Current Question + Previous Conversation → LLM → Context-Aware Answer

The assistant should also avoid guessing when a question is ambiguous.

For example:
What is inheritance?

If multiple contexts are available:

Python inheritance
Java inheritance
C++ inheritance

the assistant can ask the user to clarify instead of selecting one without sufficient evidence.

“Knowledge is power. Information is liberating.” - Kofi Annan

4. Building the Knowledge Base

The Knowledge Base converts raw PDF content into searchable vector representations.

PDF → Text Extraction → Structure Detection → Chunking → Embeddings → FAISS

4.1 Text Extraction and Structure Detection

The first step is to extract text from each PDF page.
The project uses pdfplumber:

with pdfplumber.open(pdf_path) as pdf:
for page in pdf.pages:
page_text = page.extract_text()

The extracted text is stored along with page information:
page_text_list.append(page_text)

pdfplumber supports page-level text extraction and can also work with layout-related information such as character positions and font attributes.

The basic transformation is:

PDF → Machine-Readable Text

Structure Detection

Extracted text alone may lose useful document structure. Headings provide additional context.

For example:

Chapter 2
Inheritance

Inheritance allows one class to...
The project uses font-size information to identify headings.

It calculates the median body font size:

body_font_size = np.median(all_sizes)

and defines a threshold:

heading_threshold = body_font_size + 1.5

A sufficiently large and relatively short line can then be treated as a heading.

The resulting chunk can retain:

Heading: Inheritance
Text: Inheritance allows one class...

This makes the retrieved information more meaningful.

4.2 Chunking

Large documents should not be treated as one giant block of text. The content is divided into smaller chunks.

The project uses:

max_chunk_size = 1000

and:

overlap = 200

The overlap helps preserve context between neighboring chunks.

Chunk 1 → 1000 characters
200-character overlap
Chunk 2 → 1000 characters

The project also associates headings and metadata with chunks, providing additional context during retrieval.

4.3 Embeddings and FAISS
After chunking, each chunk is converted into a numerical representation called an embedding.

The project uses:

SentenceTransformer("all-MiniLM-L6-v2")

and:

embeddings = model.encode(chunks)

Sentence Transformers creates fixed-size vector representations that can be used for semantic search and similarity comparison.

Conceptually:

"Python inheritance" → [0.12, -0.34, 0.81, ...]

The exact numbers are not meaningful to humans. They allow text to be compared mathematically.

Question Embedding → Compare Similarity → Document Chunk Embeddings → Relevant Chunks

FAISS

Once embeddings are generated, they need to be stored for efficient search.
The project uses FAISS:

dimension = embeddings.shape[1]
index = faiss.IndexFlatL2(dimension)
index.add(
np.array(embeddings).astype("float32")
)

IndexFlatL2 performs nearest-neighbor search using L2 distance.
The simplified flow is:
Document Chunks → Embeddings → FAISS → Vector Search

FAISS provides indexing and efficient similarity-search functionality for vector data.

4.4 Semantic Retrieval and Sources

When a user asks a question, the question is converted into an embedding using the same model.

The system then searches FAISS:

User Question → Question Embedding → FAISS Search → Relevant Chunks

The project retrieves the top 3 relevant chunks:

distances, indices = index.search(
question_embedding,
k=3
)

These chunks become the context for the LLM.

Source Metadata
Each chunk also retains information such as:

File name
Page number
Heading

For example:

Inheritance.pdf
Page 4
Heading: Types of Inheritance

This allows users to verify the answer and return to the original document.

5. Role of the Glossary

Technical documents often contain specialized terminology. Even when the system retrieves the correct information, users may not understand the terms being used. This is where the glossary becomes important.

“Our industry does not respect tradition. It only respects innovation.” - Satya Nadella

5.1 What is a Glossary?

A glossary is a collection of important terms and their definitions.

For example:

Encapsulation
→ The practice of bundling data and methods within a class.
Inheritance
→ A mechanism through which one class derives properties and behavior from another.
Polymorphism
→ The ability of different objects to respond to the same interface in different ways.

The glossary provides quick access to definitions without requiring users to search through the complete document.

5.2 Automatic Glossary Extraction

Instead of manually creating the glossary, the project uses an LLM to identify important terminology from the extracted document content.

The system can identify:

Key terms
Abbreviations
Variables
Concepts
Tools
Definitions The expected structure is a dictionary:

{
"inheritance": "A mechanism through which one class derives properties and behavior from another.",
"encapsulation": "A technique of bundling data and methods within a class."
}

The terms are normalized:

return {
str(k).lower().strip(): str(v).strip()
for k, v in glossary_dict.items()
}

This makes lookup more consistent.

5.3 Glossary vs Knowledge Base

The glossary and Knowledge Base have different responsibilities.

For example:

**Glossary Question

"What is inheritance?"

Question → Glossary Lookup → Definition

Knowledge Base Question

"How is inheritance used in this application?"

Question → Embedding → FAISS Search → Relevant Chunks → LLM → Answer**

The glossary is therefore not a replacement for the Knowledge Base. It is an additional layer designed to make terminology-focused questions faster and clearer.

6. Putting Everything Together

The final assistant combines the Knowledge Base, glossary, retrieval system, conversation history, and LLM into one workflow.

6.1 LLM and Context
The LLM is responsible for generating the final natural-language response.

The basic process is:

Question → Relevant Context → LLM → Natural-Language Answer

The important distinction is:

The vector index retrieves.
The LLM explains.

The retrieved context helps ground the generated answer in information from the uploaded documents.

For follow-up questions:

Current Question + Previous Conversation → LLM → Context-Aware Answer

“The advance of technology is based on making it fit in so that you don't really even think about it.” - Bill Gates

6.2 End-to-End Workflow

The complete system can be divided into two stages.

Stage 1: Knowledge Base Creation

PDF → Text Extraction → Structure Detection → Chunking → Embeddings → FAISS

The glossary is generated alongside the Knowledge Base:

PDF Text → Important Terms → Glossary

Stage 2: Question Answering

For a definition-based question:

Question → Glossary Lookup → Definition

For a broader question:

Question → Embedding → FAISS Search → Relevant Chunks → LLM → Answer

For a follow-up question:

Current Question + Previous Conversation → LLM → Context-Aware Answer

The overall experience is simple:

Upload PDF → Ask Question → Get Answer

6.3 User Interface

The project uses Streamlit to provide the user interface.
The user can:

Upload PDF documents.
Process the documents.
Ask questions.
View generated answers.
Continue the conversation.
View relevant sources.

From the user's perspective:

Upload PDF → Ask Question → Get Answer

Behind the scenes:

PDF → Extraction → Chunking → Embeddings → FAISS → Retrieval → LLM → Answer

The complexity remains in the backend while the user experience remains simple.

FAQs

1. What is a PDF Knowledge Base?
A PDF Knowledge Base converts information from PDFs into searchable chunks. These chunks are represented as embeddings and stored in a vector index such as FAISS.

2. Why not simply search the PDF?
Traditional search relies heavily on matching words. Semantic search can retrieve related information even when different wording is used.

3. What is the role of FAISS?
FAISS stores vector representations and performs similarity search to retrieve relevant chunks.

4. Why is a glossary needed?
The Knowledge Base handles broader questions, while the glossary provides quick answers to terminology-based questions such as "What is encapsulation?"

5. Can the assistant answer follow-up questions?
Yes. Conversation history allows the assistant to interpret questions based on previous messages.

6. What happens when a question is ambiguous?
The assistant can identify multiple relevant contexts and ask the user to clarify instead of making an unsupported assumption.

Interesting Facts

Semantic search focuses on meaning rather than only exact keywords -Semantic Search Documentation
The entire PDF does not need to be sent to the LLM. Relevant chunks are retrieved first - Sentence Transformers Documentation
This project uses 1,000-character chunks with 200-character overlap - FAISS Documentation
The glossary and Knowledge Base serve different purposes.
Source metadata can connect retrieved content back to its original PDF, page, and heading - pdfplumber Documentation
The same Knowledge Base can support multiple AI applications beyond Q&A.

“We are stuck with technology when what we really want is just stuff that works.” - Douglas Adams

Conclusion

The project demonstrates how static PDF documents can be transformed into an interactive source of knowledge.

By combining PDF extraction, structure detection, chunking, embeddings, FAISS-based retrieval, glossary lookup, conversation history, source tracking, and an LLM, users can interact with documents using natural language instead of manually searching through files.

The key idea is:

The Knowledge Base retrieves the right information, the glossary explains important terminology, and the AI assistant turns that information into a conversational answer.

The result is a document assistant that combines semantic search, contextual understanding, and glossary-based definitions into one simple question-answering experience.sea

About the Author: Tanisha is an AI Engineer at AddWeb Solution, working on AI/ML solutions, intelligent document processing, and AI-powered applications.

The Best Way to Structure Data for AI Agents

RevOS — Wed, 14 Oct 2026 10:03:35 +0000

Ask five vendors how to get your data ready for an AI agent, and you'll get five different answers, and every one of them will happen to be whatever that vendor sells. The graph-database company says knowledge graph. The vector-database company says RAG. The semantic-layer company says semantic model. None of them is lying, exactly. Each one is describing the one tool in their own box.

The honest answer is that there is no single best way to structure data for an AI agent. Five real approaches are in production use today: semantic models, ontologies and knowledge graphs, retrieval-augmented generation (RAG), plain markdown files searched with grep, and open knowledge formats. Each one solves a different failure mode. Pick the wrong one for your question, and the agent doesn't fail loudly. It fails quietly, returning a fluent, confident, wrong answer, because the model was never the bottleneck. The data it was handed was.

This guide walks through all five, with the evidence for when each one works, so you can match the structure to the question instead of the vendor pitch.

What "structuring data for AI agents" means

Structuring data for an AI agent means organizing information so the agent can find the right fact, definition, or document reliably. The alternative is guessing from raw, unlabeled data and getting it wrong in a way that sounds right. Enterprise AI agents don't fail because the underlying model is weak. They fail because nobody told the agent what "active customer" means, or where the answer to a specific question lives.

This is a data problem before it's a prompting problem. A model handed a clean, governed definition of a metric will use it. A model handed a pile of raw tables and no context will invent a plausible-sounding calculation instead, and you won't find out until someone checks the number in a meeting.

The five ways to structure data for an AI agent

1. Semantic models

A semantic model (or semantic layer) defines your business metrics and entities once: "monthly recurring revenue," "active customer," "churn rate," with the exact tables, joins, and filters behind each one. The agent queries the metric by name instead of writing SQL against raw tables from scratch.

This is the highest-leverage fix for a specific, common failure: an agent guessing at a business definition and getting a number that sounds right but isn't. In dbt Labs' 2026 benchmark, agents querying a semantic model hit 98.2%-100% accuracy on a set of business questions, against 84.1%-90.0% for the same models writing raw text-to-SQL against the full schema. The same benchmark also compared today's models to a 2023-era model doing raw text-to-SQL, and found accuracy nearly doubled over that 2.5-year gap, from 32.7% to 64.5%. That's real model progress. But even the best 2026 model still trailed the semantic-model number. That's the more direct evidence: the gap between a wrong query and a right one comes from what the agent is grounded against, not from how capable the underlying model is.

Semantic models are the right call when the question is a recurring business metric that many tools and many agents will ask about repeatedly, where "everyone gets the same number" matters. They're the wrong call for a one-off question about a single document, or for connecting facts across entities the model was never designed to relate.

A semantic model also earns its keep in a way the other four approaches don't automatically match: define the metric once, and any AI agent can query it through MCP (the Model Context Protocol, an open standard for connecting agents to data sources), instead of every team building its own bespoke retrieval or graph layer to answer the same question. That turns the semantic model from a one-off project into shared infrastructure: a governed layer other agents can read from directly, rather than a definition locked inside one team's dashboard.

You don't need a specific vendor to get this. dbt and Cube will both get you a semantic layer. What matters more than which tool you pick is treating the metric definition as the thing worth getting right once.

2. Ontologies and knowledge graphs

A knowledge graph stores entities (people, accounts, products, contracts) and the explicit relationships between them, instead of flat tables or blocks of text. An ontology is the schema that defines what those entities and relationships are allowed to mean, so the graph stays consistent as it grows. Together, they let an agent traverse connections that a keyword or embedding match would miss entirely, such as "which accounts does this contact influence, and which of those accounts churned within six months of a support escalation?"

This is where GraphRAG earns its name: retrieval that queries a knowledge graph, plus generated summaries of clusters of related entities, instead of a flat vector index. Microsoft's 2024 research found GraphRAG held a 72% win rate on comprehensiveness and a 62% win rate on diversity over plain vector RAG. Its top-level summaries also needed over 97% fewer tokens than summarizing the source text directly. A separate, independent 2025 systematic evaluation of RAG versus GraphRAG found that adding selection and integration strategies on top of graph retrieval improved the best baseline by 1.1% to 6.4% on a multi-hop reasoning benchmark. That's a real but more modest gain than the headline GraphRAG numbers alone suggest, and a useful reminder that the retrieval technique matters as much as the graph itself.

Knowledge graphs cost more to build and maintain than the other four approaches. Someone has to model the entities and relationships, and keep the ontology honest as the business changes. That cost is worth paying when the questions genuinely require multi-hop reasoning across connected entities. It's overkill for a question that boils down to "look up this one fact."

3. RAG (retrieval-augmented generation)

RAG retrieves the most relevant chunks of text, usually by embedding similarity in a vector database, and hands them to the model as context before it answers. It's the fastest approach to stand up: chunk your documents, embed them, index them, and you have a working system in an afternoon.

RAG is the right default for fact lookup across a large, loosely structured document set: contracts, support tickets, product docs, meeting notes. It's the wrong tool when the answer requires connecting several facts across different documents (that's the knowledge-graph case above), or when the "fact" is a governed metric that should have one certified answer everywhere (that's the semantic-model case).

There's also a subtler trap: more retrieved context is not automatically better context. 2025 research from Chroma, testing 18 frontier models including GPT-4.1, Claude Opus 4, and Gemini 2.5, found that accuracy degrades non-uniformly as input length grows. Even a single distractor passage measurably hurt performance relative to a cleaner baseline. Models sometimes did worse on logically well-structured long context than on the same information shuffled. The finding has been corroborated widely enough that it's a fair rule of thumb: precision in what you retrieve beats volume, every time.

Some teams get real gains combining approaches: retrieving with a vector index, then using graph relationships to check or expand the answer. AWS's 2024 case study with Lettria reported up to 35% improvement in answer precision when graph-based retrieval was layered on top of vector RAG, evaluated across finance, healthcare, aeronautics, and legal use cases.

4. Plain markdown + grep

The least glamorous option, and the one no infrastructure vendor will ever recommend to you, because none of them sell it: put your data in plain markdown files, organized sensibly, and let the agent search them with grep or another full-text tool. No embeddings, no vector database, no graph to maintain.

This works better than it sounds for a small, well-organized internal corpus: think hundreds of files, not millions. Coding agents already do this constantly. A well-structured repository of markdown docs, read directly by an agent with a file-search tool, is often faster to build and easier to reason about than standing up retrieval infrastructure for a corpus that would fit on a laptop. It's the same instinct behind emerging patterns like llms.txt and skill-file conventions: plain, greppable text, organized by a human who knows the content, beats a half-tuned vector index over the same material.

There's a maintenance upside here that's easy to undervalue. Markdown files live in version control like everything else you write. Every change is a diff a human can read, review, and revert, with no embedding pipeline to rerun, no index to rebuild, no vector database to keep in sync with the source of truth.

The approach breaks down exactly where you'd expect: once the corpus gets large enough, or the phrasing gap between a query and the right document gets wide enough, that keyword matching starts missing things a semantic or graph-based search would catch. That's the signal to graduate to RAG or a knowledge graph, not a reason to skip markdown plus grep as a legitimate starting point.

5. Open knowledge formats

Open knowledge formats, such as schema.org, JSON-LD (JSON for Linking Data), and RDF (Resource Description Framework), structure data using a shared, public vocabulary instead of a format only your own systems understand. Where the first four approaches structure data for your agents to read, open formats matter when the data needs to be readable by agents you don't control: a customer's AI assistant checking your product's pricing, a search engine's AI Overview summarizing your documentation, or a partner's agent pulling structured facts from your public site.

This is a different job from the other four, and it's easy to conflate them. A semantic model, knowledge graph, or RAG index is internal infrastructure serving your own agents. Open formats are a publishing choice, closer to how you already mark up a webpage for search engines, except the audience reading that markup increasingly includes an AI agent instead of only a crawler. If your data only needs to serve your own team's agents, this is the one approach on the list you can reasonably skip.

The five approaches at a glance

Approach	Best for	Setup cost	Maintenance	Sharpest weakness
Semantic model	Governed business metrics, repeated questions	Medium: model the metrics once	Medium: update as metrics evolve	Doesn't help with unstructured documents
Ontology / knowledge graph	Multi-hop reasoning across connected entities	High: model entities and relationships	High: keep the ontology honest	Expensive for simple lookup questions
RAG (vector search)	Fact lookup across a large, loosely structured corpus	Low: chunk, embed, index	Low to medium: re-index as content changes	More context isn't automatically more accurate
Plain markdown + grep	Small, well-organized internal corpus	Very low: organize files sensibly	Low: no infrastructure to run	Breaks down as the corpus grows or gets ambiguous
Open formats (schema.org, JSON-LD)	Data read by agents you don't control	Low to medium: markup your public content	Low: mostly set-and-forget	Doesn't structure data for your own internal agents

Which one should you use?

Match the structure to the question, not the vendor pitch. In practice, that means asking what kind of question the agent needs to answer:

"What's our [metric]?" a recurring, governed business number. Use a semantic model.
"How does X connect to Y, and what does that chain imply?" multi-hop reasoning across entities. Use an ontology or knowledge graph, optionally with GraphRAG-style retrieval.
"What does this document say about...?" fact lookup across a large corpus you didn't write by hand. Use RAG.
"What's our process for...?" a corpus small enough that you know roughly where the answer lives. Plain markdown plus grep is often faster to build and equally reliable.
"Can an agent I don't control read this?" a partner's, customer's, or a search engine's AI. Use open formats. Most production systems combine two or three of these rather than picking exactly one: a semantic model for metrics, RAG for contracts and support tickets, and grep-searchable internal docs for team runbooks, all read by the same agent. The mistake isn't picking one approach. It's assuming one approach has to cover every question.

How this plays out in practice

Take a Revenue Operations team building one agent to help with renewals. In a single conversation, that agent might field five different kinds of questions, and each one wants a different structure behind it.

"What's this account's net revenue retention?" is a governed metric with one correct calculation. The agent should query a semantic model, not write its own SQL against raw subscription tables and hope it matches finance's number.

"Who at Acme Corp reports to whom, and which of them also sits on the renewal decision?" is a multi-hop question across people, roles, and accounts. That's a knowledge graph question. A flat table or a text search won't trace the reporting chain.

"What did the contract say about the renewal notice period?" is one fact, buried in one document, out of thousands of contracts. RAG retrieves the relevant clause without anyone having to read every file.

"What's our internal playbook for a renewal at risk?" points to a few dozen well-organized runbook documents the team wrote and knows by heart. Plain markdown plus grep finds the right one in milliseconds, with zero infrastructure to maintain.

"Does our public pricing page make sense to the customer's own AI assistant?" is an agent-you-don't-control question. That's the open-format case.

No single approach on this list answers all five. That's the real argument for treating "how do I structure my data for AI agents" as a portfolio decision, not a one-time architecture choice.

The migration path tends to run in one direction. Teams start with markdown plus grep because it's the fastest thing that works. They move pieces of it to RAG once the corpus outgrows keyword search. They add a knowledge graph only once they hit a specific class of question, usually a multi-hop one, that RAG keeps getting wrong. Few teams start with a knowledge graph on day one, and that's a reasonable default: build the structure your current questions require, and add more only when a real question exposes the gap.

Context engineering vs. prompt engineering

Everything above is an exercise in what's now being called context engineering: deciding what information reaches the model's context window, in what form, and how much of it. That's distinct from prompt engineering, which is how you phrase the instructions in a single message. The distinction matters because a perfectly worded prompt can't fix a model that was handed the wrong facts, or too many of them.

However you structure your data (semantic model, knowledge graph, RAG, or a markdown file) the same discipline applies. Give the agent exactly the facts it needs to answer the question, not everything you have.

Conclusion

Five real approaches, five different jobs. A semantic model for governed metrics. A knowledge graph for multi-hop reasoning. RAG for fact lookup across a big corpus. Plain markdown and grep for a small one. Open formats for the agents you don't control. The vendors selling each piece will tell you theirs is the answer. It usually isn't, not on its own. Start from the question your agent needs to answer, and the right structure follows from that, not from whichever infrastructure happens to be easiest to sell you.

This article was originally published on the RevOS blog.

Evaluating AI-Powered RAG Systems: Beyond Just Correct Answers

Pavan Bhusare — Wed, 14 Oct 2026 08:25:54 +0000

Artificial Intelligence is changing how we search for information and interact with documents. Retrieval-Augmented Generation (RAG) is an important approach that combines information retrieval with large language models (LLMs). Instead of relying only on the knowledge learned during training, a RAG system retrieves relevant information from external documents and uses it to generate answers.

However, building a RAG application is only the first step. Evaluating its performance is equally important to ensure that the answers are accurate, relevant, and supported by reliable information.

Why Is AI Evaluation Important?

A RAG system may produce an answer that sounds convincing but contains incorrect information. It may also retrieve irrelevant document sections or fail to include important details in its response. These problems can reduce user trust and make the application less useful.

AI evaluation helps developers identify these issues before deploying their applications in real-world environments.

Key Areas of RAG Evaluation

Retrieval Quality

Retrieval quality measures whether the system finds the most relevant information from its knowledge base. If the retrieved documents do not contain the required information, even a powerful language model may generate an incorrect answer. Metrics such as Precision@K and Recall@K can help assess retrieval performance.

Answer Relevance

The generated response should directly address the user's question. A good evaluation process checks whether the answer is useful, clear, and focused rather than including unrelated information.

Faithfulness and Groundedness

Faithfulness measures whether the answer is supported by the retrieved context. A reliable RAG application should avoid inventing facts or presenting unsupported statements as truth. This is particularly important in applications involving technical documentation, education, and business information.

Context Relevance

The retrieved context should contain information that helps answer the question. Evaluating context relevance helps developers improve document chunking, embedding models, and retrieval strategies.

Practical Techniques for Better Evaluation

Developers can create a test dataset containing sample questions, expected answers, and relevant source documents. They can then compare system outputs against these references and examine incorrect responses.

Automated evaluation frameworks can help measure different quality dimensions, while human review provides additional insight into clarity, usefulness, and factual correctness. Testing different prompts, retrieval settings, and chunk sizes can also reveal opportunities for improvement.

Evaluation should not be a one-time activity. As documents, models, and application requirements change, the system should be tested regularly to maintain consistent performance.

Conclusion

Evaluating RAG systems requires more than checking whether an answer looks correct. Developers must assess retrieval quality, answer relevance, context relevance, and faithfulness together. A systematic evaluation process helps build AI applications that are more reliable, transparent, and useful.

For a deeper explanation of practical evaluation methods, metrics, and approaches, read the full article on the Valentius Kryptix blog:

Read the full article: https://valentiuskryptix.com/proven-ways-to-evaluate-ai-rag-based-question-answering-systems/

Your AI can give a wrong answer confidently. 🤖

Tushar Yadav — Wed, 14 Oct 2026 08:05:56 +0000

And the problem might not even be the LLM.

While working with a brand-focused AI application, Ive been learning why building reliable AI systems requires more than just writing a good prompt.

Think about a system that generates marketing copy using a brands knowledge base.

Several things need to work together:

→ Brand context: Does the AI understand the brand, its audience, and its positioning?

→ Retrieval: Did the system find the right information from the uploaded documents?

→ Generation: Did the LLM use that information correctly?

Heres what makes debugging interesting:

If the system retrieves the wrong information, improving the prompt might not fix the problem.

If the right information reaches the LLM but the answer is still wrong, the issue might lie in how the model uses that context.

💡 The key lesson? Before fixing the answer, find out where the system went wrong.

Thats one of the things Im exploring as I go deeper into RAG, evaluation, and building more reliable AI applications.

For those building with LLMs, whats been your biggest challenge: retrieval, hallucinations, or inconsistent outputs?

The Answer Was in the Middle of the Context and the Model Walked Past It

Serguey Asael Shinder — Wed, 14 Oct 2026 07:31:05 +0000

The support bot
had a simple pipeline.

Search the documents.

Take the top twenty passages.

Put them all in the prompt,
then the question at the end.

The answer to the question
was in passage eleven.

The bot said
it could not find that information.

Nothing was missing.

It was all there,
inside the window,
well under the limit.

The model just did not use it.

There is a well known paper on this,
from 2023,
called Lost in the Middle.

The researchers hid the one useful passage
among many useless ones
and moved it around.

Accuracy was best
when the passage sat at the start
or at the end.

Put it in the middle
and accuracy fell,
sometimes below what the model managed
with no documents at all.

The curve looked like a U.

Newer models
are better at this.

None of them are perfect,
and none of them tell you
when they skipped something.

So a large context window
is a promise about what fits,
not about what gets read
with equal care.

The fix is mostly
about sending less.

Twenty passages
because twenty fit
is not a reason.

Rank them properly.

Cut the ones below a score
you have actually tested.

Five good passages
beat twenty with the right one
buried in the middle.

Put the strongest one first.

Keep the question
close to the end,
and say it again there
if the prompt is long.

When the order matters,
do not leave it
to whatever order the search returned.

Then measure it.

Take one question
with a known answer.

Run it with the right passage
at the start,
in the middle,
at the end.

If the answers change,
you have learned
something your benchmark never told you.

A long context
is a long corridor.

The model checks the door it came in
and the door it is leaving by.

Do not hide
the thing that matters
halfway down.

– Serguey Asael Shinder

Code Search RAG: Chunk by Function With tree-sitter, Not by 512 Tokens

Gabriel Anhaia — Wed, 14 Oct 2026 05:01:34 +0000

Book: RAG Pocket Guide
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

The chunk the codebase assistant retrieved for "where does the retry delay get computed?" starts halfway down a function, at an except clause, and has no def line, no class name, nothing but a file path to say where it came from. The model writes a confident answer about a function it never saw the name of.

The bug is in the chunker. It cut the file every 512 tokens, the way it cuts prose, and the cut landed in the middle of a function. Code has natural seams at def and class; a token counter ignores them.

The fix fits in two small files. Parse Python with tree-sitter, cut at function and class boundaries, carry the file path and the enclosing symbol as metadata, and split the definitions that are too big on statement boundaries. Then score both chunkers on a labelled query set, so you see the difference on your own machine instead of taking my word for it.

What a token window does to a function

A fixed window has no idea where a function starts. If a function begins at token 470 of a 512-token window, the chunk gets the decorator, the signature and four lines of body. The next chunk gets the rest of the body with no name attached. Ask for "the function that parses the config file" and the retriever has to match either a signature with no logic or logic with no signature.

The cAST paper from Carnegie Mellon and Augment Code found the same problem: "Existing line-based chunking heuristics often break semantic structures, splitting functions or merging unrelated code." Their AST-based chunker, which splits large syntax nodes and merges small siblings under a size limit, reported a 4.3-point Recall@5 gain on RepoEval retrieval and a 2.67-point Pass@1 gain on SWE-bench generation over the fixed-size baseline.

Frameworks have moved the same way. LlamaIndex ships a CodeSplitter that parses with tree-sitter and, in its source, caps chunks at 1,500 characters by default. The cAST authors published astchunk, which measures chunk size in non-whitespace characters and supports Python, Java, C# and TypeScript. Both are fine to use. Writing your own once is how you find out what they are doing for you, and how you tune it when your codebase has 600-line functions.

Token windows cut wherever the count runs out. AST chunks cut where the code does.

Two chunkers, same token budget

Install the pieces. py-tree-sitter 0.26.0 needs Python 3.10 or newer, and the Python grammar ships as its own wheel.

pip install tree-sitter tree-sitter-python \
  sentence-transformers numpy

The embedder is BAAI/bge-small-en-v1.5: a small English model with 384 dimensions and a 512-token max sequence length. Both chunkers get the same budget of 480 tokens, counted with the embedder's own tokenizer, so a short header and the special tokens still fit under 512 and nothing gets silently truncated. That is the fair version of "512-token chunks".

First file, chunkers.py, starting with the Chunk type:

# chunkers.py
from dataclasses import dataclass
from typing import Callable

import tree_sitter_python as tspython
from tree_sitter import Language, Parser

PARSER = Parser(Language(tspython.language()))
DEFS = {"function_definition", "class_definition"}
Count = Callable[[str], int]

@dataclass
class Chunk:
    path: str
    symbol: str  # "JSONDecoder.raw_decode", "<module>", or ""
    start: int   # 1-based first line
    end: int     # 1-based last line, inclusive
    text: str

    def embed_text(self) -> str:
        head = f"# {self.path}"
        if self.symbol:
            head += f" :: {self.symbol}"
        return f"{head}\n{self.text}"

The baseline comes next. It adds lines while their token count stays under the budget, emits a chunk, then backs up about 64 tokens for the overlap. The BGE tokenizer emits no token for a newline, so summing per line counts the same tokens the AST chunker counts on joined text:

# chunkers.py (continued)
def fixed_chunks(path: str, src: str, ntok: Count,
                 size: int = 480, overlap: int = 64):
    lines = src.split("\n")
    out, i = [], 0
    while i < len(lines):
        j, used = i, 0
        while j < len(lines):
            cost = ntok(lines[j])
            if used + cost > size and j > i:
                break
            used += cost
            j += 1
        text = "\n".join(lines[i:j])
        out.append(Chunk(path, "", i + 1, j, text))
        if j >= len(lines):
            break
        back, k = 0, j
        while k > i + 1 and back < overlap:
            k -= 1
            back += ntok(lines[k])
        i = k
    return out

The baseline is slightly generous: it snaps to line ends and overlaps about 64 tokens, where a plain token splitter would cut mid-line. Its chunks still get the file path in the header. The only things it lacks are syntax-aware boundaries and the symbol name, which keeps the comparison about exactly those two things.

src.split("\n") rather than splitlines() is deliberate. tree-sitter counts rows by \n only, and splitlines() also splits on form feeds, which some older Python files contain. Mixing the two puts every line number after the first form feed off by one.

The AST chunker goes in the same file. First the helpers that write chunks:

# chunkers.py (continued)
def unwrap(node):
    if node.type == "decorated_definition":
        return node.child_by_field_name("definition")
    return node

def name_of(defn) -> str:
    return defn.child_by_field_name("name").text.decode()

def ast_chunks(path: str, src: str, ntok: Count,
               budget: int = 480):
    lines = src.split("\n")
    tree = PARSER.parse(src.encode())
    out: list[Chunk] = []

    def span(a, b):  # 0-based rows, inclusive
        return "\n".join(lines[a:b + 1])

    def emit(sym, a, b, head=""):
        body = span(a, b)
        if ntok(head + body) <= budget:
            out.append(Chunk(path, sym, a + 1, b + 1, head + body))
            return
        # one statement is bigger than the budget: window it
        room = max(budget - ntok(head), 64)
        for c in fixed_chunks(path, body, ntok, size=room):
            out.append(Chunk(path, sym, a + c.start, a + c.end,
                             head + c.text))

emit writes one chunk when the span fits and falls back to line windows only when a single statement is bigger than the budget. pack sits on top of it and greedily groups neighbouring statements. It is still inside ast_chunks, so keep the indentation:

    # ast_chunks, continued
    def pack(nodes, sym, head=""):
        group = []
        for node in nodes:
            if group:
                a = group[0].start_point[0]
                b = node.end_point[0]
                if ntok(head + span(a, b)) > budget:
                    emit(sym, a, group[-1].end_point[0], head)
                    group = []
            group.append(node)
        if group:
            emit(sym, group[0].start_point[0],
                 group[-1].end_point[0], head)

The last two helpers decide what a unit is, and the final line starts the walk at the module root:

    # ast_chunks, continued
    def define(node, defn, scope):
        name = name_of(defn)
        sym = f"{scope}.{name}" if scope else name
        a, b = node.start_point[0], node.end_point[0]
        if ntok(span(a, b)) <= budget:
            emit(sym, a, b)
            return
        body = defn.child_by_field_name("body")
        head = span(a, body.start_point[0] - 1) + "\n"
        if defn.type == "class_definition":
            block(body.named_children, sym, head)
        else:
            pack(body.named_children, sym, head)

    def block(nodes, scope, head=""):
        loose = []
        for node in nodes:
            defn = unwrap(node)
            if defn.type in DEFS:
                if loose:
                    pack(loose, scope or "<module>", head)
                    loose = []
                define(node, defn, scope)
            else:
                loose.append(node)
        if loose:
            pack(loose, scope or "<module>", head)

    block(tree.root_node.named_children, "")
    return out

Walk it top-down. block goes through the statements of a module or a class body. Each definition goes to define, and the statements between definitions (imports, constants, a class docstring) collect into loose and get packed into <module> chunks or chunks named after the class. define emits a definition whole if it fits. Decorators count as part of the definition, because tree-sitter-python wraps them in a decorated_definition whose definition field holds the function or class.

The symbol path is the breadcrumb. A method inside JSONDecoder gets JSONDecoder.raw_decode, and embed_text() puts # json/decoder.py :: JSONDecoder.raw_decode on the first line of what gets embedded. A query that says "JSON decoder" now has a literal match even when the method body never says it.

When one function is bigger than the budget

Function-level chunks break down on the big ones. A 300-line class or a 200-line function does not fit in 480 tokens, and dumping it whole into the embedder does not help. The model card for all-MiniLM-L6-v2 says so directly: "By default, input text longer than 256 word pieces is truncated." Everything past the cutoff is invisible to retrieval, and nothing warns you.

The chunker above handles size in three steps.

Too-big class: recurse. define calls block on the class body, so every method becomes its own chunk with a Class.method symbol. The class signature becomes a head that is prepended to the chunks of loose class-level statements, so the docstring and class attributes still say which class they belong to.

Too-big function: pack statements under the signature. pack walks the body's top-level statements and greedily merges neighbours until the next one would blow the budget. Every piece starts with the full signature, decorators included. A retrieved piece from the middle of a long function still opens with def _make_iterencode(...). This is the merge half of what cAST does: split big nodes, merge small siblings.

Too-big single statement: fall back to windows. A 150-line if block is one statement. emit notices it does not fit and runs the line-window chunker over just that statement, with the signature still on top. cAST recurses into the statement's children instead, which keeps boundaries cleaner. The fallback is the cheap version; swap it for recursion when your codebase has a lot of these.

Small siblings merge, big nodes split, and every split piece keeps the signature on top.

Two costs to know about. The header eats budget, so a function with a 15-line signature leaves less room for the body in each piece. And pack re-tokenizes growing spans, which is quadratic in the worst case. On a few files that is nothing. On a monorepo, count tokens per statement once and add them up instead.

Score both on a labelled query set

A chunker is only better if retrieval gets better, so measure it. You need a corpus every reader already has, and the Python standard library is on your disk. The queries below describe what code does, mostly without using its identifiers, the way you search a codebase you did not write. Each label is a file plus a symbol path.

Before the eval, append one helper to chunkers.py, so the gold line ranges come from the parser instead of from hand-typed line numbers:

# chunkers.py (append)
def symbols(src: str) -> dict[str, tuple[int, int]]:
    found = {}

    def walk(node, scope):
        for child in node.named_children:
            defn = unwrap(child)
            if defn.type not in DEFS:
                continue
            name = name_of(defn)
            sym = f"{scope}.{name}" if scope else name
            found[sym] = (child.start_point[0] + 1,
                          child.end_point[0] + 1)
            if defn.type == "class_definition":
                walk(defn.child_by_field_name("body"), sym)

    walk(PARSER.parse(src.encode()).root_node, "")
    return found

Second file, eval.py. The top holds the model, the files and the labels:

# eval.py
import sysconfig
from pathlib import Path

import numpy as np
from sentence_transformers import SentenceTransformer

from chunkers import ast_chunks, fixed_chunks, symbols

MODEL = "BAAI/bge-small-en-v1.5"
QUERY = "Represent this sentence for searching relevant passages: "
FILES = [
    "json/__init__.py", "json/decoder.py", "json/encoder.py",
    "shlex.py", "textwrap.py", "collections/__init__.py",
]
LABELS = [
    ("quote a string so a POSIX shell sees one argument",
     "shlex.py", "quote"),
    ("split a command line into tokens like a shell",
     "shlex.py", "split"),
    ("strip the indentation shared by every line",
     "textwrap.py", "dedent"),
    ("break a paragraph into lines of a maximum width",
     "textwrap.py", "wrap"),
    ("add a prefix to the start of selected lines",
     "textwrap.py", "indent"),
    ("decode JSON at the start of a string, return end",
     "json/decoder.py", "JSONDecoder.raw_decode"),
    ("scan a JSON string literal handling escapes",
     "json/decoder.py", "py_scanstring"),
    ("serialize a Python object to a JSON str",
     "json/__init__.py", "dumps"),
    ("walk nested lists and dicts yielding JSON text",
     "json/encoder.py", "_make_iterencode"),
    ("the n elements with the highest counts",
     "collections/__init__.py", "Counter.most_common"),
    ("move a key to either end of an ordered dict",
     "collections/__init__.py", "OrderedDict.move_to_end"),
    ("make a tuple subclass with fields accessed by name",
     "collections/__init__.py", "namedtuple"),
]

recall embeds every chunk once, then checks the top five results for each query in two ways, explained after the code:

# eval.py (continued)
def recall(chunks, gold, model, k=5):
    vecs = model.encode([c.embed_text() for c in chunks],
                        normalize_embeddings=True)
    touched = whole = 0
    for query, path, (s, e) in gold:
        qv = model.encode(QUERY + query, normalize_embeddings=True)
        top = np.argsort(-(vecs @ qv))[:k]
        hits = [chunks[i] for i in top if chunks[i].path == path]
        touched += any(c.start <= e and c.end >= s for c in hits)
        whole += any(c.start <= s and c.end >= e for c in hits)
    return touched / len(gold), whole / len(gold)

main reads the six files from the standard library of whichever Python runs it and builds both chunk lists:

# eval.py (continued)
def main():
    root = Path(sysconfig.get_paths()["stdlib"])
    model = SentenceTransformer(MODEL)

    def ntok(text: str) -> int:
        ids = model.tokenizer.encode(text, add_special_tokens=False)
        return len(ids)

    fixed, by_ast, spans = [], [], {}
    for rel in FILES:
        src = (root / rel).read_text(encoding="utf-8")
        fixed += fixed_chunks(rel, src, ntok)
        by_ast += ast_chunks(rel, src, ntok)
        spans[rel] = symbols(src)

    gold = [(q, f, spans[f][sym]) for q, f, sym in LABELS]
    for name, chunks in (("fixed", fixed), ("ast", by_ast)):
        t, w = recall(chunks, gold, model)
        print(f"{name:5} chunks={len(chunks):4} "
              f"touched@5={t:.2f} whole@5={w:.2f}")

if __name__ == "__main__":
    main()

The query prefix is the retrieval instruction from the BGE model card, which says passages need no prefix. Run python eval.py. The tokenizer may log a "sequence length is longer than" warning while counting tokens for whole classes; that is the counter measuring, not the embedder truncating.

Read the two numbers separately. touched@5 asks whether any top-5 chunk overlaps the target function at all. That is lenient, and it is the metric a token-window chunker looks best on, because a chunk holding three lines of the right function counts. whole@5 asks whether one top-5 chunk contains the entire function, which is what the LLM needs in order to answer. The gap between them is the "body without its name" problem, measured.

There are no results printed in this post, on purpose: your Python version changes the stdlib files, and 12 queries is a smoke test, not a benchmark. Run it and read your own numbers. Some labels are longer than the 480-token budget, so the AST chunker splits them too and neither chunker can score whole on them. namedtuple and _make_iterencode are the obvious ones; dumps (its docstring alone is long) and py_scanstring may be over as well, depending on your Python version. Expect that, and look at which chunk comes back instead. For a real decision, write 50 to 100 queries against your own repo, from real questions people asked in code review or on Slack, and keep the file in version control next to the chunker.

Before you point it at your repo

Four changes take this from a stdlib demo to something you would index a monorepo with.

More languages. tree-sitter has grammars for most languages you will meet, but node type names differ per grammar. function_definition is the Python name; check each grammar's node-types.json and give every language its own DEFS set. If you want a maintained version for Python, Java, C# and TypeScript, astchunk already does it.

Parse errors. tree-sitter is built to keep producing useful results even when the source has syntax errors, so a half-written file mid-refactor still produces a tree. py-tree-sitter exposes has_error on every node. Check it on the root, and when a file is badly broken, fall back to fixed_chunks for that file rather than indexing garbage boundaries.

A code-aware embedder. bge-small-en-v1.5 is a general English model with a WordPiece tokenizer that shreds snake_case and camelCase into fragments. It works for the demo. For production, compare a code-trained embedding model on your labelled set using the same eval.py; that is what the set is for.

Keyword search next to vectors. Exact identifiers, error codes and config keys are where dense retrieval is weakest. Index the symbol field in a keyword index too and fuse the results. A query that contains raw_decode should find raw_decode without asking an embedding model to guess.

If this was useful

A retriever can only return what the chunker kept together, and code makes that easy to see. The RAG Pocket Guide covers the stages on either side of this one, retrieval and reranking, plus the chunking patterns for production RAG. If you are building search over a codebase or anything else with structure, it is the short version of the decisions this post only touches.

DEV Community: rag

Python RAG Retrieval Architecture for Financial Dashboards: Managed Vectors vs Postgres

How should a Python financial research dashboard structure retrieval and observability?

What do managed vectors, Postgres, and search specialists trade off?

How do retries and recovery preserve trustworthy citations?

References

Further reading

Day 14: RAG Evaluation and Hallucination Guardrails

1. Measure the two halves

2. Spot an unfaithful answer

3. Guardrails: an LLM as judge

Coming Up Next

AIQuality #MLOps #Hallucination #TechLeadership #DataGovernance

A Guide to Insurance Claim Search Contracts (Under Retrieval Latency Budgets)

Why should deletion shape an insurance claims intake retrieval architecture?

Make the provider boundary executable

Compare contracts before comparing demos

Run a shadow migration as the real acceptance test

Sources

Designing RAG for Lawyers: Refusal Is a Feature

A saved reply is written for one customer. Your support bot can send it to anyone.

Where a saved answer comes from

What a ticket leaves in an answer

Rewriting them

Rules for the next one

Clean answers make a loose threshold cheap

What I'd change in the tool

Vector Databases 2026: Milvus 3.0, Qdrant, and the Postgres Question

From PDF to Knowledge Base: Building an AI-Powered Glossary Assistant for Intelligent Q&A

Table of Content(Index)

1. Introduction

2. Knowledge Base for PDFs

3. AI-Powered Q&A Assistant

4. Building the Knowledge Base

5. Role of the Glossary

6. Putting Everything Together

FAQs

Interesting Facts

Conclusion

The Best Way to Structure Data for AI Agents

What "structuring data for AI agents" means

The five ways to structure data for an AI agent

1. Semantic models

2. Ontologies and knowledge graphs

3. RAG (retrieval-augmented generation)

4. Plain markdown + grep

5. Open knowledge formats

The five approaches at a glance

Which one should you use?

How this plays out in practice

Context engineering vs. prompt engineering

Conclusion

Evaluating AI-Powered RAG Systems: Beyond Just Correct Answers

Your AI can give a wrong answer confidently. 🤖

The Answer Was in the Middle of the Context and the Model Walked Past It

Code Search RAG: Chunk by Function With tree-sitter, Not by 512 Tokens

What a token window does to a function

Two chunkers, same token budget

When one function is bigger than the budget

Score both on a labelled query set

Before you point it at your repo

If this was useful