DEV Community: AlaiKrm

The Retrieval Problem Nobody Talks About: When Your AI Knows Too Much Old Information

AlaiKrm — Fri, 10 Jul 2026 11:34:16 +0000

There is a specific failure mode in enterprise RAG deployments that is distinct from hallucination and distinct from poor retrieval quality, but that gets misdiagnosed as both. I want to describe it precisely because the fix is specific and different from the fixes for those other problems.

The failure mode is temporal confusion. The AI retrieves accurate information from your documents, generates a response that is faithful to those documents, and the response is wrong because the documents are out of date.

This sounds like a data quality problem, and at one level it is. But the reason it is more insidious than a straightforward data quality problem is that the AI has no native sense of time. When it retrieves a document from three years ago and a document from last month on the same topic, it does not know that the older document may be superseded by the newer one. It synthesizes them based on semantic relevance to the query, not based on temporal priority.

The result is responses that blend current and outdated information in ways that are factually coherent but substantively wrong. They are wrong in a way that is hard to detect because the response reads well and cites real documents.

In practice this shows up in specific patterns. Policy questions return answers that reflect the policy as it was written rather than as it has since been amended. Product documentation queries return information about features that have been deprecated. Process questions return steps from a workflow that has been restructured. In each case, the AI is accurately summarizing something real that is no longer true.

The fix requires changes at two levels.

At the data level, every document in your index needs a status field that distinguishes current from superseded from archival. Documents that have been replaced by newer versions need to be explicitly marked as superseded rather than just left in the index. This requires human judgment, and it requires ongoing maintenance as documents continue to evolve. There is no automated solution that reliably identifies when one document supersedes another without organizational context that the system does not have.

At the retrieval level, you need to build in freshness weighting that gives more recent documents higher retrieval priority when the query is about a topic where currency matters. The tricky part is that not all queries are sensitive to document age. A query about company history should return old documents. A query about current policy should prefer recent ones. The retrieval system needs to distinguish between these cases, which requires either query classification or explicit user signals about whether they want current versus historical information.

def freshness_weighted_search(query, vectorstore, query_type):
    results = vectorstore.similarity_search_with_score(query, k=20)

    if query_type in ("policy", "process", "current_state"):
        # Apply freshness penalty to older documents
        reranked = []
        for doc, score in results:
            days_old = (datetime.now() - doc.metadata["last_modified"]).days
            freshness_penalty = min(0.3, days_old / 365 * 0.1)
            adjusted_score = score - freshness_penalty
            reranked.append((doc, adjusted_score))
        return sorted(reranked, key=lambda x: x[1], reverse=True)[:5]

    return results[:5]  # no freshness adjustment for historical queries

The supersession problem is harder to solve with code than with process. The technical solution can deprioritize old documents, but only the people who maintain the knowledge base can correctly mark which documents supersede which others. Building that into your document management workflow, making it someone's explicit responsibility to update document status when policies change, is the organizational fix that the technical fix depends on.

The organizations that handle this well treat their AI knowledge base with the same editorial discipline as a publication. Content has a status, a review cycle, and an owner responsible for keeping it current. The ones that struggle treat the knowledge base as a document dump and expect the AI to figure out what is current. It cannot.

Evaluation-Driven Development for RAG Systems: Building the Feedback Loop That Makes AI Actually Improve

AlaiKrm — Thu, 09 Jul 2026 14:04:35 +0000

Most RAG systems get built in one direction: design the architecture, implement it, test it manually, deploy it. The evaluation, if it exists, happens before deployment as a gate. After deployment, quality assessment happens informally through user complaints and anecdotal feedback.

This approach produces systems that get worse over time, not better. The document corpus changes. User query patterns shift. The system drifts without any systematic mechanism for detecting or correcting that drift.

Evaluation-driven development is a different approach that treats systematic quality measurement not as a pre-deployment gate but as a continuous operational practice that drives ongoing improvement. The eval suite is not the last step before you ship. It is the mechanism through which you learn what needs to change and verify that changes worked.

Here is how I build and operate evaluation suites for production RAG systems, including the specific components that most implementations skip.

Building the evaluation dataset

The evaluation dataset is the foundation of everything else. It needs to be representative of real production queries, not queries you invented to test hypothetical capabilities.

The process I use to build the initial dataset:

First two weeks of any new RAG deployment, log every query submitted to the system along with the retrieved documents and generated response. This gives you the actual distribution of what users ask rather than what you thought they would ask. The actual distribution is almost always different from the anticipated distribution in ways that matter.

From this log, sample 200 to 300 queries that represent the spread of query types. For each query, establish a ground truth answer. For factual questions with specific correct answers, this is straightforward. For synthesis questions, it requires human judgment about what constitutes a good response. This human annotation is time-consuming and cannot be automated away. Budget 30 to 45 minutes per query for a careful annotator.

Organize the dataset by query type and difficulty. I typically use four categories: direct factual lookup, policy or procedural questions, synthesis across multiple documents, and edge cases where the answer requires acknowledging uncertainty or where the correct answer is that the information is not in the knowledge base.

from dataclasses import dataclass
from typing import Optional, List
from enum import Enum

class QueryType(Enum):
    FACTUAL_LOOKUP = "factual_lookup"
    POLICY_PROCEDURAL = "policy_procedural"
    SYNTHESIS = "synthesis"
    EDGE_CASE = "edge_case"

class DifficultyLevel(Enum):
    EASY = "easy"      # single document, clear answer
    MEDIUM = "medium"  # multiple documents, some synthesis
    HARD = "hard"      # sparse evidence, requires reasoning, or should decline

@dataclass
class EvalQuery:
    query_id: str
    query_text: str
    query_type: QueryType
    difficulty: DifficultyLevel
    ground_truth_answer: str
    relevant_document_ids: List[str]    # documents that should be retrieved
    acceptable_answer_variants: List[str]  # alternative correct phrasings
    failure_indicators: List[str]          # strings that indicate a bad answer
    should_decline: bool                   # True if AI should say "I don't know"
    annotator_notes: str
    created_at: str
    last_verified: str  # when ground truth was last confirmed as current

The last_verified field is critical and often missing. Ground truth answers go stale as the knowledge base evolves. An evaluation dataset that was accurate when built becomes misleading over time if it is not maintained. Schedule quarterly reviews of the evaluation dataset to confirm that ground truth answers still reflect current organizational truth.

The metrics that matter

Retrieval metrics and generation metrics are both necessary and measure different things.

For retrieval:

def compute_retrieval_metrics(
    retrieved_doc_ids: List[str],
    relevant_doc_ids: List[str],
    k_values: List[int] = [1, 3, 5]
) -> dict:
    metrics = {}

    for k in k_values:
        retrieved_at_k = set(retrieved_doc_ids[:k])
        relevant = set(relevant_doc_ids)

        precision_at_k = len(retrieved_at_k & relevant) / k if k > 0 else 0
        recall_at_k = len(retrieved_at_k & relevant) / len(relevant) if relevant else 0

        metrics[f"precision@{k}"] = precision_at_k
        metrics[f"recall@{k}"] = recall_at_k

    # Mean Reciprocal Rank: where did the first relevant doc appear?
    mrr = 0
    for rank, doc_id in enumerate(retrieved_doc_ids, 1):
        if doc_id in set(relevant_doc_ids):
            mrr = 1 / rank
            break
    metrics["mrr"] = mrr

    return metrics

For generation, automated metrics alone are insufficient. ROUGE and BLEU scores measure lexical overlap, not factual accuracy or usefulness. I use a combination:

def compute_generation_metrics(
    generated_response: str,
    ground_truth_answer: str,
    retrieved_docs: List[str],
    query: EvalQuery,
    llm_judge  # small LLM used for automated quality judgment
) -> dict:
    metrics = {}

    # Groundedness: is the answer supported by retrieved documents?
    groundedness_prompt = f"""
    Query: {query.query_text}
    Retrieved context: {' '.join(retrieved_docs)}
    Generated answer: {generated_response}

    Rate from 0 to 1: Is every factual claim in the generated answer
    directly supported by the retrieved context?
    Respond with only a number between 0 and 1.
    """
    metrics["groundedness"] = float(llm_judge.complete(groundedness_prompt))

    # Correctness: does it match the ground truth?
    correctness_prompt = f"""
    Question: {query.query_text}
    Correct answer: {ground_truth_answer}
    Generated answer: {generated_response}

    Rate from 0 to 1: How factually correct and complete is the generated answer
    compared to the correct answer? Minor wording differences are fine.
    Respond with only a number between 0 and 1.
    """
    metrics["correctness"] = float(llm_judge.complete(correctness_prompt))

    # Failure indicator check: did the answer contain known bad patterns?
    metrics["contains_failure_indicator"] = any(
        indicator.lower() in generated_response.lower()
        for indicator in query.failure_indicators
    )

    # Appropriate decline: did it decline when it should have?
    if query.should_decline:
        decline_indicators = ["don't have", "no information", "not in", "cannot find"]
        metrics["appropriate_decline"] = any(
            indicator in generated_response.lower()
            for indicator in decline_indicators
        )
    else:
        metrics["appropriate_decline"] = True  # N/A for queries that should be answered

    return metrics

The LLM judge approach for groundedness and correctness is an approximation, not a ground truth. It is faster and cheaper than human evaluation and good enough for continuous monitoring, but significant changes in scores should be verified with human review before drawing strong conclusions.

Running evals as a deployment gate

Every change to the retrieval configuration, the prompts, the chunking strategy, or the embedding model should be evaluated against the full eval suite before deployment to production.

class EvalGate:
    def __init__(
        self,
        eval_dataset: List[EvalQuery],
        rag_pipeline,
        thresholds: dict
    ):
        self.eval_dataset = eval_dataset
        self.pipeline = rag_pipeline
        self.thresholds = thresholds  # e.g., {"recall@5": 0.80, "correctness": 0.85}

    def run_evaluation(self) -> dict:
        all_metrics = []

        for query in self.eval_dataset:
            retrieved_docs, response = self.pipeline.run(query.query_text)

            retrieval_metrics = compute_retrieval_metrics(
                retrieved_doc_ids=[doc.id for doc in retrieved_docs],
                relevant_doc_ids=query.relevant_document_ids
            )
            generation_metrics = compute_generation_metrics(
                generated_response=response,
                ground_truth_answer=query.ground_truth_answer,
                retrieved_docs=[doc.content for doc in retrieved_docs],
                query=query,
                llm_judge=self.llm_judge
            )

            all_metrics.append({**retrieval_metrics, **generation_metrics})

        return self.aggregate_metrics(all_metrics)

    def passes_gate(self) -> tuple[bool, dict]:
        results = self.run_evaluation()
        failures = {
            metric: (results[metric], threshold)
            for metric, threshold in self.thresholds.items()
            if results.get(metric, 0) < threshold
        }
        return len(failures) == 0, failures

This gate prevents regressions from reaching production. Before deploying any change, run the gate. If it fails, the failing metrics tell you specifically what degraded and by how much. You can then either fix the issue or make an explicit decision to accept the regression with full awareness of what you are accepting.

The monitoring layer that connects to the eval suite

Running the eval suite before deployment catches regressions from changes you made intentionally. It does not catch regressions from changes that happen without your involvement: document corpus changes, query distribution shifts, or model behavior drift.

For continuous monitoring, run a subset of the eval suite (50 to 100 queries) on a daily schedule against the production system. Track the metrics as time series. Alert when any metric drops more than a defined threshold from its rolling average.

import schedule
import time

def daily_quality_check(eval_subset, rag_pipeline, alert_threshold=0.05):
    results = run_evaluation_subset(eval_subset, rag_pipeline)
    baseline = load_rolling_baseline(window_days=30)

    regressions = {
        metric: {
            "current": results[metric],
            "baseline": baseline[metric],
            "drop": baseline[metric] - results[metric]
        }
        for metric in results
        if baseline[metric] - results[metric] > alert_threshold
    }

    if regressions:
        send_alert(
            subject="RAG Quality Regression Detected",
            body=format_regression_report(regressions)
        )

    store_daily_results(results)

schedule.every().day.at("06:00").do(daily_quality_check, eval_subset, rag_pipeline)

while True:
    schedule.run_pending()
    time.sleep(60)

The combination of pre-deployment gates and continuous monitoring gives you coverage against both types of quality change: the ones you cause and the ones that happen to you.

The part that requires human judgment

I want to be direct about the limits of automated evaluation. Automated metrics measure proxies for quality. They are useful proxies and they are far better than no measurement, but they are not the same as direct human judgment about whether the system is useful.

The evaluation-driven development loop needs a human component. Monthly, someone should manually review a random sample of production queries: what was retrieved, what was generated, and whether the output was genuinely useful. This review catches things the automated metrics do not: outputs that are technically accurate but misleadingly incomplete, responses that are polite and hedged in a way that makes them useless, or answer quality that is systematically different for specific user groups or query types.

The automated metrics tell you when something changed. The human review tells you whether the change matters. Both are necessary. The organizations that rely entirely on automated metrics will miss qualitative degradation that the metrics cannot capture. The organizations that rely entirely on human review will miss systematic patterns that only become visible at scale.

Build both. They are not in tension. They are complementary layers of the same feedback loop.

The Vector Dumpster Fire: Why Your RAG Pipeline is Hallucinating

AlaiKrm — Wed, 08 Jul 2026 16:28:59 +0000

There is a dirty secret in the enterprise AI space right now, and it is living inside your vector database.

Every week, I talk to engineering teams who are completely baffled by their Retrieval-Augmented Generation (RAG) applications. They followed the standard playbook. They spun up a vector database. They connected it to their company’s Google Drive, Confluence, and Jira. They ran a standard chunking script, generated embeddings, and hooked it all up to GPT-4 or Claude.

The demo looked great. Then they pushed it to production, and the system immediately started hallucinating, contradicting itself, and pulling completely irrelevant information. The engineers usually blame the LLM. They spend weeks tweaking system prompts and adjusting temperature settings, hoping to magically fix the outputs.

But the LLM is not the problem. Your language model is doing exactly what it was designed to do: reasoning over the context you provide. The problem is that the context you are providing is absolute garbage.

You haven't built an enterprise knowledge base. You have built a vector dumpster fire. Here is exactly why it is failing, and how to actually fix it.

The Myth of "Naive Chunking"

If you are using the default text splitters from LangChain or LlamaIndex—cutting your documents into arbitrary 512 or 1024 token chunks with a 50-token overlap—you are actively destroying your own data.

Documents are not continuous streams of homogenous text. They have structure. They have headers, bullet points, code blocks, and tables. When a naive chunker indiscriminately slices a document at exactly token number 512, it does not care if it just cut a crucial financial table in half. It does not care if it orphaned a paragraph from the header that gives it context.

Imagine a chunk that simply reads: "Approval is granted for the $4.2M budget." Approval for what? By whom? For which department? Because the title of the document and the project name were left behind in chunk number 42, and this is chunk number 43, the LLM has absolutely no idea. When the user asks about the marketing budget, the vector search might pull this chunk simply because the word "budget" is in it, leading to a massive hallucination.

The fix: Move to Semantic Chunking. You have to write parsers that respect document boundaries. Chunk by markdown headers. Chunk by paragraphs. Keep tables intact as single semantic units. If a section is too long, summarize it and embed the summary alongside the raw text. Never blindly slice enterprise data.

PDFs Are Where RAG Goes to Die

Most enterprise knowledge is locked inside PDFs. And most teams are using basic OCR libraries to extract text from them before embedding. This is a catastrophic architectural error.

Standard PDF extraction reads left-to-right, top-to-bottom. It completely ignores multi-column layouts, sidebars, headers, and footers. If you have a two-column research paper, a basic parser will read the first line of the left column, jump across the gap, and read the first line of the right column, mashing them together into a nonsensical string of words.

When you embed that nonsensical string, its location in the vector space is meaningless. It will never be retrieved when it should be, and it will be retrieved when it shouldn't.

The fix: You need Layout-Aware Parsing. Before embedding anything, you must use vision models or specialized document intelligence APIs (like Azure Document Intelligence or unstructured.io) to identify the bounding boxes of tables, images, and text columns. If you cannot parse the visual layout of the PDF accurately, do not put it in your vector database.

The Absence of Metadata Pre-Filtering

Vector search relies on cosine similarity. It calculates the mathematical distance between the user's query and the chunks in your database. But cosine similarity has no concept of time, authority, or access control.

If a user asks, "What is our current WFH policy?", a pure vector search will happily retrieve the 2019 WFH policy, the 2021 WFH policy, and a draft of a 2024 WFH policy that was never approved. To a semantic search engine, all of these chunks are highly relevant to the query. To the user, it is a disaster.

You cannot rely on the LLM to sort out which document is the "right" one based on text alone.

The fix: Metadata is not optional in enterprise RAG; it is the primary filter. Every single chunk in your database must be tagged with metadata: creation_date, last_modified, author, document_status (draft/approved), and department.

When a user asks a question, you do not just embed the query. You extract intent. You use a lightweight routing model to determine that they want the current policy. You apply a hard SQL-style pre-filter to your database (e.g., WHERE document_status = 'approved' AND creation_date > '2023-01-01') BEFORE you run the vector search. This drastically reduces the search space and entirely eliminates chronological hallucinations.

The "Lost in the Middle" Phenomenon

Because vector search is cheap, teams get greedy. They configure their retriever to pull the top 20 or 30 chunks and stuff them all into the LLM's context window, assuming the model will figure it out.

Extensive research has shown that LLMs suffer from a "lost in the middle" effect. They pay heavy attention to the first few chunks in the prompt, and the last few chunks. Everything in the middle gets heavily degraded or ignored. If the actual answer to the user's question is buried in chunk #12 out of 20, there is a high probability the model will miss it and hallucinate an answer instead.

The fix: Implement a Re-ranking layer. Your initial vector search can pull 30 chunks. That is fine. But before those chunks ever touch your expensive LLM, they must pass through a Cross-Encoder model. The Cross-Encoder evaluates the actual logical relevance of each chunk against the specific user query and scores them. You then take only the top 3 to 5 absolute best chunks and pass those to the LLM.

Fewer, higher-quality chunks will always beat a massive context window filled with noise.

Architecture is Not Magic

We need to stop treating AI as magic. A vector database is just a database. An LLM is just a reasoning engine. If your data pipeline is lazy, your output will be unreliable.

Stop blaming the models. Go back to your ingestion pipeline, look at the actual raw text you are embedding, and ask yourself: "If I handed this isolated, broken paragraph to a human, could they answer the question?"

If the answer is no, your RAG pipeline is broken by design. Fix your data tier, and the hallucinations will fix themselves.

Stop Making Your AI Agent Smarter. Make Its Context Smaller.

AlaiKrm — Tue, 07 Jul 2026 15:32:43 +0000

One of the most common questions I hear is:

"Which LLM should we use?"

It's an understandable question.

Model capability matters.

But after spending time reviewing how AI agents behave in production systems, I've become convinced that context quality influences reliability far more often than model choice.

A smarter model can't consistently compensate for poor context.

The misconception: more context equals better answers

Many teams assume an AI agent should receive as much information as possible.

Context has a cost

Every document added to an AI agent creates another potential source of conflict.

Imagine an HR policy.

Version 2 says employees receive 18 days of annual leave.

Version 3 updates that to 20 days.

Both documents remain searchable.

A retrieval system might surface either one depending on metadata, ranking, or document similarity.

The language model isn't "hallucinating."

It's answering from conflicting evidence.

The real problem started before inference ever happened.

Retrieval is not understanding

Retrieval-Augmented Generation (RAG) has become the default architecture for many enterprise AI applications.

Its purpose is straightforward:

Retrieve relevant information first.

Generate an answer second.

That approach works well.

But retrieval alone doesn't guarantee correctness.

A retrieval system can successfully find five documents.

The harder question is:

Which of those five should the model trust?

Without document governance, freshness indicators, ownership metadata, or clear source hierarchy, retrieval quality eventually reaches a ceiling.

The model can only reason with the evidence it's given.

Small knowledge bases often outperform large ones

This sounds counterintuitive.

Many organizations celebrate connecting every repository they own.

In practice, narrower knowledge domains often produce more reliable answers.

Why?

Because the signal-to-noise ratio improves.

Instead of searching across every department, imagine an AI assistant dedicated only to Finance.

Its searchable information becomes:

Accounting procedures
Budget guidelines
Approved financial policies
Finance documentation

Nothing else.

The agent now has fewer conflicting documents and fewer irrelevant retrieval results.

Accuracy improves without changing the model.

Context boundaries are an architectural decision

One lesson I've taken from reviewing enterprise AI systems is that context should be designed, not accumulated.

Ask questions like:

Which documents should this agent never read?
Who owns this knowledge source?
How frequently is this information updated?
Which documents are authoritative?
What happens when two sources disagree?

Those questions belong in architecture discussions.

Not after deployment.

Before it.

The real optimization target

Teams often optimize for model intelligence.

I think they should optimize for decision confidence.

A model that answers 95% of questions correctly because it searches a carefully curated knowledge base is usually more valuable than a larger model searching an uncontrolled information warehouse.

That's because trust is cumulative.

Users don't remember the hundred correct answers.

They remember the one answer that confidently quoted an outdated policy.

Takeaway

Before upgrading your model, examine your context.

Ask yourself:

Are duplicate documents still searchable?
Can the AI distinguish current policies from archived ones?
Is every knowledge source actively maintained?
Would a smaller, more focused knowledge base produce better answers?

Enterprise AI isn't only about building smarter models.

More often, it's about building systems that give those models better evidence to reason with.

Reranking in Enterprise RAG: Why It Matters More Than Your Embedding Model Choice

AlaiKrm — Mon, 06 Jul 2026 10:06:05 +0000

There is a point in the maturity arc of most enterprise RAG systems where the team has optimized the embedding model, tuned the chunking strategy, and is still seeing retrieval quality that is good but not as precise as the use case demands. The next lever, and often the highest-leverage one remaining in the retrieval stack, is reranking.

Reranking is a second-pass scoring step that takes the candidates returned by the initial vector search and applies a more computationally expensive but more accurate relevance model to reorder them before passing the top results to the LLM. The initial vector search is fast and operates at scale. The reranker operates on a smaller candidate set (typically 20 to 50 documents) and can apply significantly more sophisticated relevance judgments than embedding similarity alone can produce.

I want to explain why this distinction matters, how to implement reranking correctly, and specifically where it adds the most value in enterprise settings.

Why embedding similarity is an imperfect proxy for relevance

The initial retrieval step in a RAG system converts the query and all indexed documents into dense vector representations, then finds the documents whose vectors are closest to the query vector in that embedding space. The assumption is that documents with similar vector representations are semantically similar to the query.

This assumption holds well for many query types and breaks down for others in ways that are predictable once you understand the mechanism.

Embedding models are trained to capture general semantic similarity. They are good at matching the conceptual domain of a query to the conceptual domain of a document. They are less good at:

Relevance versus tangential similarity. A document about "data security in cloud environments" is semantically similar to a query about "GDPR compliance requirements." But if the user is asking about GDPR, a document specifically about GDPR is more relevant than a document about data security that mentions regulatory compliance in passing. Embedding similarity cannot reliably distinguish centrality from tangential mention.

Query intent versus document content. A query asking "what went wrong with the Q3 forecast" is seeking a specific type of content, a post-mortem or analysis of a missed target. Documents that mention the Q3 forecast without being analytical about it will have similar semantic distance from the query as documents that actually perform the analysis the user needs. Embedding similarity treats these as equivalent.

Exact terminology when approximate matches exist. In domains with precise technical or legal language, a document that uses the exact term in the query should be ranked above a document that uses approximate synonyms. Embedding models often fail to preserve this distinction because they are trained to recognize semantic equivalence across terminological variation.

A reranker trained specifically for relevance judgment, as opposed to semantic similarity, addresses these failure modes because it is making a different kind of prediction: not "are these semantically similar" but "is this document the right answer to this specific question."

Cross-encoder rerankers and how they work

The dominant architecture for reranking in RAG systems is the cross-encoder, which processes the query and each candidate document together as a single input and produces a relevance score directly, rather than encoding them separately and measuring vector distance.

This joint processing allows the model to attend to the relationship between the query and the document in a way that bi-encoder embedding models cannot. A cross-encoder can recognize that a document answers the specific question asked, not just that it covers the same general topic.

The tradeoff is computational cost. A cross-encoder must process each query-document pair separately, making it O(n) in the number of candidate documents at query time. This is why reranking is applied to a small candidate set from the initial retrieval rather than to the full index: reranking every document in a large corpus would be prohibitively slow.

from sentence_transformers import CrossEncoder
from typing import List, Tuple
import numpy as np

class CrossEncoderReranker:
    def __init__(self, model_name: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"):
        self.model = CrossEncoder(model_name)

    def rerank(
        self,
        query: str,
        candidates: List[Tuple[object, float]],  # (document, initial_score)
        top_k: int = 5,
        initial_score_weight: float = 0.1
    ) -> List[Tuple[object, float]]:

        if not candidates:
            return []

        docs = [doc for doc, _ in candidates]
        initial_scores = [score for _, score in candidates]

        # Build (query, document_text) pairs for the cross-encoder
        pairs = [(query, doc.page_content) for doc in docs]

        # Score all pairs
        rerank_scores = self.model.predict(pairs)

        # Normalize rerank scores to [0, 1]
        rerank_scores = np.array(rerank_scores)
        if rerank_scores.max() != rerank_scores.min():
            rerank_scores = (rerank_scores - rerank_scores.min()) / (rerank_scores.max() - rerank_scores.min())

        # Blend with initial retrieval scores (small weight to initial)
        initial_scores_normalized = np.array(initial_scores)
        if initial_scores_normalized.max() != initial_scores_normalized.min():
            initial_scores_normalized = (
                (initial_scores_normalized - initial_scores_normalized.min()) /
                (initial_scores_normalized.max() - initial_scores_normalized.min())
            )

        blended_scores = (
            (1 - initial_score_weight) * rerank_scores +
            initial_score_weight * initial_scores_normalized
        )

        # Sort by blended score and return top_k
        scored_docs = list(zip(docs, blended_scores))
        scored_docs.sort(key=lambda x: x[1], reverse=True)

        return scored_docs[:top_k]

The small weight on the initial retrieval score (10% in this example) is a hedge against cases where the cross-encoder produces scores that do not strongly differentiate between candidates. The initial retrieval score provides a signal about broad semantic relevance that complements the cross-encoder's more specific relevance judgment.

Selecting the right reranker model

The default choice for most teams is one of the MS-MARCO-trained cross-encoders from Sentence Transformers. These are trained on a large dataset of query-passage relevance judgments from web search and perform well on general-purpose retrieval tasks.

For enterprise use cases, the limitations of general-purpose rerankers become relevant in specific scenarios.

Domain-specific terminology. In legal, medical, financial, or highly technical domains, general-purpose rerankers may not accurately judge the relevance of documents that use specialized terminology. A reranker trained on web search data has limited exposure to the specific vocabulary and relevance judgments that matter in your domain.

Organizational context. A reranker has no access to organizational context. It cannot know that in your specific organization, a document tagged as "draft" is less authoritative than one tagged as "approved" even if the content is similar. Metadata signals like document status, authorship authority, or recency need to be incorporated outside the reranker through score blending rather than expected from the model itself.

Query intent categorization. For organizations with clearly distinct query types, fine-tuning a reranker on domain-specific relevance data or training a query intent classifier that routes to different retrieval and reranking configurations is worth the investment.

class QueryIntentRouter:
    def __init__(self):
        self.intent_classifier = self.load_classifier()
        self.rerankers = {
            "factual_lookup": CrossEncoderReranker("cross-encoder/ms-marco-MiniLM-L-6-v2"),
            "policy_question": CrossEncoderReranker("your-fine-tuned-policy-reranker"),
            "synthesis": CrossEncoderReranker("cross-encoder/ms-marco-MiniLM-L-12-v2"),  # larger model for complex queries
            "procedural": CrossEncoderReranker("cross-encoder/ms-marco-MiniLM-L-6-v2"),
        }

    def get_reranker(self, query: str) -> CrossEncoderReranker:
        intent = self.intent_classifier.predict(query)
        return self.rerankers.get(intent, self.rerankers["factual_lookup"])

    def rerank(self, query: str, candidates: list, top_k: int = 5) -> list:
        reranker = self.get_reranker(query)
        return reranker.rerank(query, candidates, top_k=top_k)

Where reranking adds the most value in enterprise settings

Not all query types benefit equally from reranking. Understanding where the value is concentrated helps prioritize the implementation.

Policy and compliance queries benefit most. When an employee is looking for the authoritative answer on a specific policy question, the cost of returning the second-most-relevant document is high. These are exactly the high-precision queries where cross-encoder reranking demonstrates the clearest improvement over embedding-only retrieval.

Queries with domain-specific terminology benefit significantly. Legal definitions, technical specifications, financial terms, and medical terminology are all cases where exact relevance matters and where general-purpose embedding models often underperform.

Synthesis queries benefit less than lookup queries. When the goal is to retrieve multiple relevant documents for the LLM to synthesize, reranking's primary value of identifying the single most relevant document matters less. In these cases, retrieval diversity, getting a range of relevant perspectives rather than the top-ranked single document, is sometimes more valuable than precision.

High-volume queries where reranking latency is a concern may need a lighter reranking model or a latency budget allocation that limits reranking to specific query types. The computational cost of reranking should be evaluated against the latency budget for your specific use case.

The systems where I have seen reranking produce the most dramatic improvements are the ones with high-precision retrieval requirements and specialized domain vocabulary. In those cases, moving from embedding-only retrieval to a retrieval-plus-reranking pipeline has consistently improved recall at k=3 by 20 to 35 percentage points over the baseline. That improvement translates directly into answers that users can trust rather than answers that are plausible but imprecise.

What a Production RAG System Actually Looks Like After 18 Months

AlaiKrm — Fri, 03 Jul 2026 09:08:25 +0000

I want to write the post I wish existed when I started building enterprise RAG systems. Not a tutorial on how to build one. There are enough of those. This is about what the system looks like after eighteen months of real usage has shaped it into something that actually works reliably.

The gap between a RAG system that works in a demo and one that works reliably in production is significant and specific. I am going to describe that gap concretely because "production-ready" gets used as a vague quality signal when it should describe a specific set of components and properties.

The ingestion pipeline has become a system of record

In a demo RAG system, documents go in and they stay in. You index your documents, you query them, done. In a production system that has been running for eighteen months, the ingestion pipeline has evolved into something that looks more like a data management system than a simple indexing operation.

Every document in our index has a canonical identifier that links it to its source, a version history, a status field (current, superseded, archived), an access control tier, and a freshness score based on when it was last verified against the source system. When a new version of a document is ingested, the previous version is marked as superseded rather than deleted, giving us rollback capability if the new version has quality problems. When a document is deleted from the source system, it is flagged in the index rather than removed immediately, to allow for investigation of why it was deleted before the vectors disappear from retrieval.

The code that implements this is not complicated. It is approximately 300 lines of Python that runs every four hours against our primary document sources. What makes it work is the discipline of treating document lifecycle as a first-class concern rather than an afterthought.

class DocumentLifecycleManager:
    def ingest(self, doc_path: str, source_system: str, metadata: dict):
        canonical_id = self.compute_canonical_id(doc_path, source_system)
        existing = self.index.get_by_canonical_id(canonical_id)

        if existing and existing.content_hash == self.compute_hash(doc_path):
            self.index.update_last_verified(canonical_id)
            return  # content unchanged, just update freshness timestamp

        if existing:
            self.index.mark_superseded(canonical_id)  # version the old one

        new_doc = self.prepare_document(doc_path, canonical_id, metadata)
        self.index.insert(new_doc)
        self.log_ingestion_event(canonical_id, "new_version" if existing else "first_ingestion")

    def run_freshness_audit(self, max_age_days: int = 30):
        stale_docs = self.index.get_docs_not_verified_since(days=max_age_days)
        for doc in stale_docs:
            source_exists = self.check_source_system(doc.source_path)
            if not source_exists:
                self.index.mark_for_review(doc.canonical_id, reason="source_deleted")
            else:
                self.re_ingest(doc.source_path, doc.source_system)

This is not the code you write in a tutorial. It is the code you write after you have been bitten by stale documents surfacing in retrieval for the third time.

The retrieval layer has access control baked in, not bolted on

The single most important architectural lesson from eighteen months of running a multi-user enterprise RAG system is that retrieval-layer access control cannot be an afterthought. If access control is applied after retrieval, as a filter on the results before they are shown to the user, you have a system that has already retrieved restricted content and is now choosing not to show it. That distinction matters in environments where the fact that restricted content exists is itself sensitive information.

The current architecture applies access control as a pre-retrieval filter, using metadata on the vector embeddings to ensure that the retrieval query itself only considers vectors that the requesting user is authorized to access.

def retrieve_with_access_control(
    query: str,
    user_context: UserContext,
    k: int = 8
) -> list:
    # Build filter from user's actual permissions, not a static role
    access_filter = {
        "$or": [
            {"access_tier": "public"},
            {"access_tier": "internal", "departments": {"$in": user_context.departments}},
            {"access_tier": "restricted", "authorized_users": {"$in": [user_context.user_id]}},
        ]
    }

    results = vectorstore.similarity_search_with_score(
        query=query,
        k=k * 2,  # retrieve more, will filter and re-rank
        filter=access_filter
    )

    # Re-rank by combining similarity score with document freshness
    reranked = self.freshness_weighted_rerank(results)
    return reranked[:k]

def freshness_weighted_rerank(self, results: list, freshness_weight: float = 0.15) -> list:
    scored = []
    for doc, similarity_score in results:
        days_since_verified = (datetime.now() - doc.metadata["last_verified"]).days
        freshness = max(0, 1 - (days_since_verified / 180))  # decay over 6 months
        combined_score = (1 - freshness_weight) * similarity_score + freshness_weight * freshness
        scored.append((doc, combined_score))
    return [doc for doc, _ in sorted(scored, key=lambda x: x[1], reverse=True)]

The freshness-weighted reranking came from a specific incident where a user was getting outdated answers from old documents that scored well on semantic similarity but poorly on factual currency. The weight is small (15%) but meaningful, and it has reduced complaints about outdated answers by roughly half.

Observability is not optional and it is not metrics dashboards

The observability layer in our current system is built around a concept I think of as "explainability by default." For every query-response cycle, we log enough information that a human investigator can reconstruct exactly what happened, starting from the user's query and ending with the generated response.

The log record for each interaction includes: the original query, the access control context applied to the retrieval, the top-k results with their similarity scores and freshness scores, the assembled prompt (hashed, not stored as raw text), the model and prompt version used, the response, the response latency by component, and any errors or fallbacks that occurred.

This logging architecture has two specific properties that I want to call out because they were not in the original design and had to be added after incidents made their absence painful.

The component-level latency tracking was added after we spent three days debugging a performance regression that turned out to be a specific document type causing embedding failures that triggered silent retries. The overall latency looked slightly elevated but not alarming. Component-level latency showed the embedding step spiking on a specific document category.

The model and prompt version logging was added after a prompt update introduced a subtle behavior change that we did not catch in evaluation because our evaluation set did not cover the edge case well. When user reports started coming in about changed response formatting, we could not initially determine whether the issue was the prompt change, a model update the provider had deployed, or a data quality issue. Version logging lets us correlate behavior changes with specific deployments.

The evaluation suite has become a regression prevention system

The evaluation suite we run against every change to the retrieval configuration or prompts has grown from twelve queries in the initial version to 340 query-response pairs representing the actual distribution of queries we see in production.

The queries are categorized by type (factual lookup, synthesis across documents, policy question, procedural guide) and by sensitivity of the underlying content. We measure recall at k for retrieval, groundedness of the response against retrieved content, and instruction following for the behavioral constraints in the system prompt.

The most important addition was the adversarial query set. These are queries specifically designed to probe failure modes: queries where we know the answer has changed recently, queries that reference content the user should not have access to, queries that contain instruction-like text designed to test prompt injection resistance, and queries where the AI's honest answer should be "I don't know" rather than a generated response.

This adversarial set has caught three significant issues that the standard evaluation set missed, in all three cases because the issue only manifested on query types that normal users do not typically ask but adversarial users or edge cases produce. Finding those in the evaluation suite is considerably better than finding them in production.

What the system cannot do well, and why that is okay

After eighteen months I have a clear-eyed view of where this system fails. It struggles with queries that require reasoning across more than four or five documents simultaneously. It degrades significantly when the knowledge base has not been maintained and documents are stale. It cannot handle procedural tasks that require maintaining state across multiple turns. It produces inconsistent results for queries that are ambiguous enough that the same query phrased differently would retrieve different document sets.

None of these failures are surprising and most of them are inherent to the current state of the technology rather than to specific implementation choices. What matters is that the failure modes are known, the system communicates uncertainty honestly when it encounters them, and the monitoring infrastructure surfaces new failure patterns quickly enough that they can be addressed before they erode user trust at scale.

The system that works reliably in production is not the one that never fails. It is the one where failure is understood, observable, and recoverable.

Your AI Agent Doesn't Need More Tools. It Needs Fewer Decisions.

AlaiKrm — Thu, 02 Jul 2026 17:20:50 +0000

Every AI agent demo seems to end the same way.

"We can connect it to Slack."

"And Jira."

"And HubSpot."

"And Google Drive."

"And Salesforce."

"And..."

Somewhere along the way, the conversation shifts from solving a business problem to collecting integrations.

I think that's backwards.

The most reliable AI agents I've seen aren't the ones connected to everything.

They're the ones forced to make fewer decisions.

The architecture mistake nobody notices

When people talk about AI agents, they usually focus on capabilities.

Can it search documents?

Can it send emails?

Can it update CRM records?

Can it create tickets?

Those are reasonable questions.

But architecture isn't just about what an agent can do.

It's about defining what an agent should never be allowed to decide.

Every additional permission expands the decision space.

Every new integration introduces another failure mode.

That's why complexity grows much faster than feature count.

Decision boundaries matter more than model quality

Imagine an agent responsible for handling support tickets.

There are two possible designs.

The first gives the agent permission to:

read customer records
update ticket status
issue refunds
send follow-up emails
escalate issues

The second allows the agent to:

summarize the ticket
recommend the next action

A human approves everything else.

Both systems use the same language model.

The difference is architectural.

The second system has a much smaller blast radius when something goes wrong.

That's usually a better trade-off for production environments.

Every integration creates operational debt

Adding another connector looks easy during implementation.

Maintaining it is a different story.

Now you have to think about:

authentication
permission mapping
API version changes
retries
monitoring
audit logging
failure recovery

None of those appear in product demos.

All of them appear six months later.

That's why I don't measure integrations by how quickly they can be added.

I measure them by how much operational complexity they introduce.

Context should stay close to the workflow

One challenge I keep seeing is context fragmentation.

Customer conversations live in one platform.

Files live somewhere else.

Project discussions are in another tool.

The AI spends more time gathering context than solving the problem itself.

Moving data across multiple systems also increases governance complexity.

Permissions become harder to reason about.

Audit trails become harder to reconstruct.

This is one reason unified AI workspaces are becoming more interesting—not because they replace every SaaS tool, but because they reduce unnecessary context switching.

A better question to ask during architecture reviews

Instead of asking:

"How many tools can this AI agent connect to?"

I prefer asking:

"If this integration disappeared tomorrow, would the business still function?"

If the answer is yes, it probably isn't a core dependency.

If the answer is no, then it deserves significantly more attention from an architectural perspective.

Not every integration should have the same level of trust.

Final thought

The strongest AI architectures rarely win because they're the most sophisticated.

They win because they're predictable.

Small decision boundaries.

Clear permissions.

Observable actions.

Controlled automation.

The more decisions an AI system is allowed to make, the more carefully those decisions need to be designed.

Architecture isn't about giving AI unlimited capability.

It's about knowing exactly where capability should stop.

The bug that took me four hours to find had nothing to do with the model

AlaiKrm — Wed, 01 Jul 2026 12:27:28 +0000

It was 11pm. The AI assistant had been returning slightly wrong answers for three days and nobody could figure out why. Not wrong enough to obviously break anything. Wrong enough that two engineers had opened tickets saying "I think the AI is getting worse?"

I started where I always start: what actually got retrieved.

Added five lines of logging to dump the retrieval results before they hit the LLM. Ran the same query that had been producing bad answers.

The top result was from a document dated 14 months ago.

The current document, the one with the right information, was ranked fourth.

Similarity scores: 0.89 for the old document, 0.81 for the new one. The old document won because it was written more cleanly and the semantic match was slightly stronger. The model did exactly what it was supposed to do. It used the best matching document. The best matching document was outdated.

Not a model problem. Not a prompt problem. A data problem that looked like a model problem for three days.

The fix was two parts. Metadata filtering so that documents tagged as superseded never enter the retrieval pool. And a freshness signal in the ranking so that when two documents match similarly, the newer one gets a small boost.

# What we added to the retrieval call
results = vectorstore.similarity_search(
    query=query,
    k=10,
    filter={"status": {"$ne": "superseded"}}
)

# Re-rank by blending similarity score with freshness
def freshness_score(doc, max_age_days=365):
    age = (datetime.now() - doc.metadata["last_modified"]).days
    return max(0, 1 - (age / max_age_days))

def rerank(results):
    return sorted(results, key=lambda r: (
        0.8 * r[1] +  # similarity
        0.2 * freshness_score(r[0])  # freshness
    ), reverse=True)

The fix took forty minutes once I understood the actual problem.

The lesson I keep relearning: when an AI system gives bad answers, the instinct is to look at the model or the prompt. Start with the retrieval. Most of the time the model is doing exactly what you told it to do. The question is whether what you told it to do was right.

Why Your RAG System Needs Hybrid Search (And How to Actually Implement It)

AlaiKrm — Tue, 30 Jun 2026 05:33:48 +0000

Vector similarity search is powerful but it has a well-known weakness: exact term matching. If a user searches for "SOC 2 Type II report" and your documents contain that exact phrase, a well-tuned vector search will find them. But if the query is "security certification audit document" and the document says "SOC 2 Type II," the semantic match might miss it depending on how the embedding model handles that specific terminology.

The solution is hybrid search: combining vector similarity search with traditional keyword search and merging the results. Most production RAG systems I have reviewed that are performing below expectations are doing vector-only search. Adding hybrid search is one of the highest-leverage improvements available.

Here is how to implement it properly.

The two search types and what each catches

Dense retrieval (vector search) is good at: semantic similarity, paraphrase matching, concept-level queries, finding relevant content even when exact terms differ. It struggles with: rare terms, product names, codes, identifiers, and precise technical terminology where exact matching matters.

Sparse retrieval (keyword search) is good at: exact term matching, rare words, codes, identifiers, and queries where the user knows the specific terminology used in the document. It struggles with: synonyms, paraphrases, and concept-level queries where the words differ from the document.

Hybrid search combines both. You retrieve candidates from each system separately and then merge and re-rank.

Implementation with Reciprocal Rank Fusion

The simplest and most effective merging strategy is Reciprocal Rank Fusion. It does not require knowing the score scale of either system, just the rank positions.

from typing import List, Dict, Tuple

def reciprocal_rank_fusion(
    dense_results: List[Tuple[str, float]],
    sparse_results: List[Tuple[str, float]],
    k: int = 60,
    dense_weight: float = 0.5,
    sparse_weight: float = 0.5
) -> List[str]:
    """
    dense_results: list of (doc_id, score) from vector search
    sparse_results: list of (doc_id, score) from keyword search
    k: RRF constant (60 is standard default)
    Returns: list of doc_ids ranked by fused score
    """
    scores: Dict[str, float] = {}

    for rank, (doc_id, _) in enumerate(dense_results):
        rrf_score = dense_weight * (1 / (k + rank + 1))
        scores[doc_id] = scores.get(doc_id, 0) + rrf_score

    for rank, (doc_id, _) in enumerate(sparse_results):
        rrf_score = sparse_weight * (1 / (k + rank + 1))
        scores[doc_id] = scores.get(doc_id, 0) + rrf_score

    return sorted(scores.keys(), key=lambda x: scores[x], reverse=True)

Wiring it up with Elasticsearch for the sparse side

Most enterprise environments already have Elasticsearch or OpenSearch running. Use it for your sparse retrieval.

from elasticsearch import Elasticsearch

es = Elasticsearch(["http://localhost:9200"])

def sparse_search(query: str, index: str, top_k: int = 20) -> List[Tuple[str, float]]:
    response = es.search(
        index=index,
        body={
            "query": {
                "multi_match": {
                    "query": query,
                    "fields": ["content^2", "title^3", "metadata.section"],
                    "type": "best_fields"
                }
            },
            "size": top_k
        }
    )
    return [
        (hit["_id"], hit["_score"])
        for hit in response["hits"]["hits"]
    ]

def dense_search(query: str, vectorstore, top_k: int = 20) -> List[Tuple[str, float]]:
    results = vectorstore.similarity_search_with_score(query, k=top_k)
    return [(doc.metadata["doc_id"], score) for doc, score in results]

def hybrid_search(query: str, vectorstore, es_index: str, top_k: int = 10) -> List[str]:
    dense = dense_search(query, vectorstore, top_k=20)
    sparse = sparse_search(query, es_index, top_k=20)
    fused = reciprocal_rank_fusion(dense, sparse)
    return fused[:top_k]

Tuning the weights

The default 50/50 weight split is a reasonable starting point. For query types where exact terminology matters heavily (compliance documents, technical specifications, product names), skew toward sparse. For conceptual queries where paraphrasing is common, skew toward dense.

You can measure this empirically with your evaluation set. Run 50/50, 70/30 dense-heavy, and 30/70 sparse-heavy on the same query set and compare recall at k. The results will tell you where to set the production weights.

In my experience, most enterprise knowledge base deployments benefit from a slight sparse-heavy weighting around 40/60 dense/sparse because enterprise documents tend to use precise technical terminology that benefits from exact matching. Tune to your actual content.

One gotcha

Document IDs need to be consistent between your vector store and your Elasticsearch index. If you use different identifiers in the two systems, the RRF merge will not find overlapping results correctly. Use the source document path or a stable UUID as the canonical identifier and store it in both systems at ingestion time.

Hybrid search adds meaningful complexity to your retrieval pipeline. In most enterprise deployments where I have added it to a previously vector-only system, recall at k=5 improved by 15 to 25 percentage points on the evaluation set. For a knowledge base that employees rely on for accurate answers, that improvement is worth the implementation effort.

Stop Using Fixed-Size Chunking for Everything. Here Is What to Use Instead.

AlaiKrm — Fri, 26 Jun 2026 15:40:21 +0000

Fixed-size chunking is the default in almost every RAG tutorial. Split your documents into 512-token chunks with 50-token overlap, embed them, call it done. It works well enough that most people never question it, and then they wonder why retrieval quality plateaus.

The problem is that fixed-size chunking is a compromise that optimizes for simplicity, not for retrieval quality. It ignores document structure entirely. A 512-token chunk might cut a reasoning chain in half. It might merge two unrelated policy points that happen to appear near each other. It might split a table between chunks in a way that makes both chunks uninterpretable.

Here are the strategies I actually use depending on document type.

For structured documents with clear sections: hierarchical chunking

Technical documentation, legal contracts, policy documents, anything with explicit heading structure benefits from chunking that follows the document's own hierarchy.

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#", "section"),
    ("##", "subsection"),
    ("###", "subsubsection"),
]

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on,
    strip_headers=False
)

chunks = splitter.split_text(document_text)
# Each chunk now inherits the header hierarchy as metadata
# section="Data Handling" subsection="Retention Policy"

The metadata inheritance is the key value here. When you retrieve a chunk about retention policy, you know it came from the Data Handling section of the document, not just that it mentioned retention somewhere.

For dense technical or scientific content: semantic chunking

When document sections do not map cleanly to headings, use sentence-level semantic similarity to find natural break points.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

# Or replace with your self-hosted embedding model
semantic_splitter = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=85
)

chunks = semantic_splitter.split_text(dense_technical_doc)

This is slower than fixed-size splitting because it requires embedding intermediate sentences to find break points. For documents where semantic coherence matters significantly, like research summaries or detailed technical analyses, the retrieval quality improvement is worth the indexing cost.

For tables and structured data: row-level chunking with header injection

Tables chunked mid-row are useless. Every row needs its column headers.

import pandas as pd

def chunk_table(df: pd.DataFrame, metadata: dict) -> list:
    chunks = []
    header = " | ".join(df.columns.tolist())

    for idx, row in df.iterrows():
        row_text = " | ".join([f"{col}: {val}" for col, val in row.items()])
        chunk_text = f"Columns: {header}\nRow {idx}: {row_text}"

        chunks.append({
            "text": chunk_text,
            "metadata": {**metadata, "row_index": idx, "chunk_type": "table_row"}
        })
    return chunks

Every row chunk contains the full column context. The AI can answer "what is the retention period for category B data" because the column header "retention period" is in every chunk, not just in the header row.

For long-form prose where nothing else fits: sliding window with parent retrieval

When you genuinely have documents that do not have structure you can exploit, sliding window chunking with parent document retrieval gives you the best of both worlds. Small chunks for precise retrieval, larger parent chunks sent to the LLM for actual generation.

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain.text_splitter import RecursiveCharacterTextSplitter

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=40)

store = InMemoryStore()  # use persistent store in production

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

The child chunks are what gets matched during similarity search. The parent chunk is what gets sent to the LLM. Precise matching, rich context for generation.

The right chunking strategy is the one that preserves the semantic units that actually matter for your document type. Fixed-size chunking ignores document structure because it does not know what that structure is. Using document structure when you have it is almost always worth the extra implementation effort.

Securing AI Access to HR Systems: The Architecture That Actually Works

AlaiKrm — Thu, 18 Jun 2026 10:10:43 +0000

I get asked this more than almost any other architecture question right now. How do you give an AI assistant access to HR data without creating a security and compliance disaster? Here is the architecture I have landed on after working through it on several deployments.

The short version: you do not give the AI access to HR data directly. You give specific AI agents access to specific HR data subsets, under specific access policies, with full audit logging. These are four separate design decisions and most teams collapse them into one.

The threat model first

Before designing anything, you need to be clear about what you are protecting against.

The external threat is straightforward: you do not want HR data accessible to anyone outside the authorized user set, which means your inference infrastructure cannot call out to external APIs with HR context in the payload.

The internal threat is less obvious but more common in practice: you do not want an employee to be able to query HR data they would not have access to through normal channels. An SDR should not be able to ask your AI what their manager's performance review said. A new hire should not be able to ask what the company's compensation bands are if that data is restricted.

Most RAG deployments handle the external threat reasonably well by using enterprise agreements with LLM providers. Almost none of them handle the internal threat adequately without deliberate architectural design.

The architecture

The pattern I use separates the knowledge base into access-controlled partitions that map to your existing permission structure.

User Query
    |
    v
Query Router (authenticated, knows user role/permissions)
    |
    +-- If user has HR_GENERAL access --> HR General partition
    |   (org chart, public policies, benefits info)
    |
    +-- If user has HR_MANAGER access --> HR Manager partition
    |   (team performance data, review summaries)
    |
    +-- If user has HR_ADMIN access  --> HR Admin partition
    |   (compensation data, disciplinary records)
    |
    v
Retrieval runs ONLY against authorized partitions
    |
    v
LLM receives context only from authorized partitions

The retrieval layer enforces access before the LLM sees anything. The LLM cannot reason about data it was never given. This is the key property that most architectures miss: filtering after retrieval is not equivalent to never retrieving in the first place.

Implementation with metadata filtering

In practice this looks like tagging every document at ingestion time with its access tier:

def ingest_hr_document(doc_path, access_tier, department=None):
    metadata = {
        "access_tier": access_tier,          # "hr_general", "hr_manager", "hr_admin"
        "doc_category": "hr",
        "department": department or "all",
        "ingested_at": datetime.now().isoformat(),
        "status": "current"
    }
    chunks = chunk_document(doc_path)
    for chunk in chunks:
        chunk.metadata.update(metadata)
    vectorstore.add_documents(chunks)

And filtering at query time based on the authenticated user's permissions:

def retrieve_with_access_control(query, user_permissions):
    allowed_tiers = []
    if "hr_general" in user_permissions:
        allowed_tiers.append("hr_general")
    if "hr_manager" in user_permissions:
        allowed_tiers.append("hr_manager")
    if "hr_admin" in user_permissions:
        allowed_tiers.append("hr_admin")

    results = vectorstore.similarity_search(
        query=query,
        filter={"access_tier": {"$in": allowed_tiers}},
        k=5
    )
    return results

This is the minimum viable implementation. It assumes your permission tiers are stable enough to hardcode. If your permission structure is more dynamic, you need the filter to call out to your IAM system at query time rather than using a static list.

The audit logging requirement

Every HR-related AI query needs an audit trail that captures: who asked, what they asked, which documents were retrieved, what access tier those documents were in, and what response was generated. This is not optional if you are in a regulated industry or if you have employees in jurisdictions with strong data rights.

def log_hr_query(user_id, query, retrieved_docs, response, session_id):
    audit_record = {
        "timestamp": datetime.now().isoformat(),
        "session_id": session_id,
        "user_id": user_id,
        "query_hash": hash(query),    # hash to avoid storing PII in the log
        "retrieved_doc_ids": [doc.metadata["doc_id"] for doc in retrieved_docs],
        "access_tiers_accessed": list(set([doc.metadata["access_tier"] for doc in retrieved_docs])),
        "response_length": len(response)
    }
    audit_store.insert(audit_record)

Store the query hash rather than the raw query if the query itself might contain sensitive information. Store doc IDs rather than doc content. You want the audit log to be auditable without itself being a vector for data exposure.

On self-hosted vs cloud for this use case

I want to be direct about one thing. The access control architecture above can be implemented on top of external LLM APIs with enterprise agreements. But there is a fundamental limitation: even with perfect metadata filtering, the assembled prompt still contains HR data that travels to an external inference endpoint.

For most organizations this is an acceptable risk given enterprise agreements and "zero training" commitments. For organizations in healthcare, financial services, or jurisdictions with strict data residency requirements, it is not acceptable regardless of the contract terms.

The clean solution for those cases is inference on premises, where the assembled prompt containing HR context never leaves your network. A few platforms now package this as a deployable product rather than a DIY infrastructure project. PrivOS (https://privos.ai/) is one of the ones I have evaluated that handles the room-scoped isolation model natively, meaning the access control is built into the data model rather than implemented as a filter layer on top of a general-purpose vector store. Worth evaluating if your threat model requires true data residency.

The architecture I described above is the right architecture for this problem. The implementation details vary based on your stack and your threat model, but the shape of the solution is consistent.

When RAG Gives Wrong Answers: A Debugging Walkthrough

AlaiKrm — Wed, 17 Jun 2026 11:27:44 +0000

Last week a client pinged me. Their internal AI assistant was confidently telling employees the wrong vacation policy. Not hallucinating from nothing. Retrieving an outdated document and presenting it as current. Classic RAG failure. Here is exactly how I debugged it and what we fixed.

The symptom: assistant returns 15 days PTO for new hires. Correct answer is 20 days (policy changed 8 months ago).

First thing I always do is check what actually got retrieved.

# Add this temporarily to your retrieval pipeline
results = vectorstore.similarity_search_with_score(
    query="vacation policy new hire PTO days",
    k=5
)

for doc, score in results:
    print(f"Score: {score:.4f} | Source: {doc.metadata.get('source')} | Date: {doc.metadata.get('last_modified')}")
    print(doc.page_content[:200])
    print("---")

Output showed the problem immediately:

Score: 0.8821 | Source: hr_policy_2021.pdf | Date: 2021-03-15
"New employees are entitled to 15 days PTO in their first year..."

Score: 0.8134 | Source: hr_policy_2023.pdf | Date: 2023-11-02  
"Effective Q4 2023, all new hires receive 20 days PTO..."

The 2021 document was scoring higher than the 2023 document. Why? The 2021 version had clearer, more keyword-dense language. The 2023 update was buried in a longer policy revision document with more surrounding text, which diluted the semantic match.

Two problems here. Retrieval is not filtering by document freshness at all. And the index still contains the outdated document.

For the freshness problem, the fix depends on your stack. In LangChain with a Chroma backend:

from langchain.retrievers import TimeWeightedVectorStoreRetriever

retriever = TimeWeightedVectorStoreRetriever(
    vectorstore=vectorstore,
    decay_rate=0.01,  # adjust based on how fast your docs go stale
    k=5
)

This is a soft fix. It penalizes older documents but does not exclude them. For policy documents where the old version is actively wrong, you want a harder approach: metadata filtering.

results = vectorstore.similarity_search(
    query=query,
    k=5,
    filter={"doc_type": "policy", "status": "current"}
)

This only works if you are tagging documents with status at ingestion time. We were not. So step two was fixing the ingestion pipeline:

def ingest_document(filepath, metadata={}):
    # When ingesting a new version of an existing document,
    # mark all previous versions as superseded
    existing = vectorstore.get(
        where={"source_canonical": metadata.get("source_canonical")}
    )
    if existing:
        for doc_id in existing["ids"]:
            vectorstore.update(doc_id, metadata={"status": "superseded"})

    # Then ingest the new version as current
    metadata["status"] = "current"
    metadata["ingested_at"] = datetime.now().isoformat()
    # ... rest of ingestion

Third problem: even after fixing future ingestion, the old chunks were still in the index. Delete them:

# Get all chunks from the outdated document
old_chunks = vectorstore.get(
    where={"source": "hr_policy_2021.pdf"}
)

# Delete them
vectorstore.delete(ids=old_chunks["ids"])

print(f"Deleted {len(old_chunks['ids'])} chunks from outdated policy document")

After these three fixes, the retrieval order flipped. Current policy document now scores first. Assistant returns the correct 20 days.

The actual bug took about 90 minutes to find and fix. The underlying issue took two days to properly address because it was not just this one document. We found seven other policy documents in the same situation: outdated versions living in the index alongside updated ones, with no mechanism to prefer the current version.

The lesson I keep relearning is that document lifecycle management is not a nice-to-have for enterprise RAG. It is load-bearing. You cannot build a trustworthy knowledge assistant on an index that does not know which version of a document is authoritative.