I Rebuilt My RAG Pipeline Without LangChain — What Got Better and What Got Worse

The first time I seriously doubted the framework was not because the model hallucinated. It was because the answer looked plausible, contained a citation, and was still wrong.

The assistant had retrieved a chunk from a deprecated help page because one part of the pipeline applied a metadata filter, another part did not, and the final prompt assembly made the whole thing look coherent. Debugging it meant stepping through wrappers, runnable compositions, and framework-specific assumptions instead of asking the real question: why did retrieval favor the wrong document?

That was the point where I stopped treating LangChain as the core of the RAG system and started treating it as an optional integration layer.

This is not an anti-framework article. LangChain solved a real problem: it gave developers a fast way to compose LLM applications when the ecosystem was young and everyone was still figuring out the basics. But once RAG moved from demo to production, the problems changed. The hard parts stopped being “call the model” and became:

• permission-aware retrieval
• stable chunking
• hybrid search
• reranking
• evaluation
• document ingestion failures
• embedding migrations
• traceability when an answer goes wrong

Rebuilding the pipeline without LangChain made some things dramatically better. It also made some things more annoying, more expensive, and more time-consuming than I expected.

This article is about both.

---

TL;DR

If you are deciding whether to keep, adopt, or remove LangChain from a production RAG system:

• Removing LangChain improved debugging, retrieval control, evaluation, observability, and cost discipline.
• Removing LangChain made harder document loading, integration maintenance, and the long tail of “small” pipeline decisions.
• The biggest win was not performance. It was that the pipeline became explicit.
• The biggest downside was that I became responsible for a lot of boring glue code that frameworks usually hide.
• My current rule: prototype with high-level tools, but own the retrieval core when the product depends on answer quality.

---

📋 Table of Contents

• 1. The abstraction stopped being a shortcut and became a boundary
• 2. Chunking stopped being “split by 800 characters”
• 3. Retrieval became a small query planner
• 4. Hybrid search was the unglamorous fix for exact identifiers
• 5. Reranking became the highest-leverage quality gate
• 6. Embedding generation became a data-engineering job
• 7. Evaluation got easier once the pipeline had seams
• 8. Observability changed from “the answer looks weird” to “chunk 7f2a was dropped”
• 9. What got worse: the long tail of boring integration work
• 10. Where I draw the line now

---

1. The abstraction stopped being a shortcut and became a boundary

Scenario:

A user asks, “What changed in webhook authentication?” The system retrieves something that mentions authentication, but not the correct product version. The final answer sounds confident. The problem is not the model. The problem is that the retrieval request did not carry the right filters, and the abstraction made that hard to see.

Why it matters:

In early RAG projects, abstractions help you move quickly. You connect a loader, a splitter, an embedding model, a vector store, and a prompt template. But in production, the interesting failures happen in the spaces between those components.

When those spaces are hidden behind generic chain-like composition, you end up debugging the composition layer instead of the retrieval behavior.

Solution:

I rebuilt the pipeline around explicit stages with small interfaces. Not a huge framework. Just enough structure to make each stage testable.

from dataclasses import dataclass
from typing import Protocol


@dataclass(frozen=True)
class RetrievedChunk:
    chunk_id: str
    doc_id: str
    text: str
    score: float
    metadata: dict


class Retriever(Protocol):
    def retrieve(
        self,
        query: str,
        *,
        filters: dict | None = None,
        limit: int = 20,
    ) -> list[RetrievedChunk]:
        ...

The pipeline then became a sequence of ordinary functions:

def answer_question(user_query: str, user_context: UserContext) -> FinalAnswer:
    plan = plan_query(user_query, user_context)
    candidates = retriever.retrieve(
        plan.retrieval_query,
        filters=plan.filters,
        limit=40,
    )
    evidence = select_evidence(plan.raw_query, candidates)
    prompt = build_prompt(plan.raw_query, evidence)
    return generate(prompt, request_id=plan.request_id)

Why this works:

The important part is not that this code is “framework-free.” The important part is that the seams are visible.

If retrieval is bad, I look at plan_query and retriever.retrieve.

If the prompt is bad, I look at build_prompt.

If the answer is unfaithful, I inspect evidence.

There is no chain abstraction sitting between me and the failure.

💡 Practical note:

If your LangChain usage already has clear boundaries around retrieval, parsing, and prompt construction, removing the framework may not help much. The problem is not the library itself. It is whether the library hides the decisions you now need to debug.

---

2. Chunking stopped being “split by 800 characters”

Scenario:

A support article contains a table of error codes. The user asks about one specific code. The retriever returns a chunk that includes the correct code, but not the header row explaining what the columns mean. The model guesses. Sometimes it guesses wrong.

Why it matters:

A lot of early RAG advice treated chunking as a text-length problem:

Pick a chunk size, add overlap, repeat.

That works for simple prose. It falls apart for real documents:

• tables
• code blocks
• numbered steps
• headings with nested context
• FAQs
• legal clauses
• API reference docs
• product changelogs

In production, chunking is not a text problem. It is a document-structure problem.

Solution:

I stopped thinking of chunks as “pieces of text” and started treating them as evidence units.

An evidence unit should carry enough context to be interpreted without its surrounding document.

For Markdown-like documents, that usually means:

• preserve heading hierarchy
• keep tables intact when possible
• attach column/header context to table rows
• keep code blocks with their immediately preceding explanation
• avoid splitting a numbered step away from its introductory sentence

A simplified version of the chunk model:

@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    doc_id: str
    heading_path: tuple[str, ...]
    text: str
    block_type: str  # paragraph, table, code, list
    token_estimate: int
    source_url: str
    updated_at: str

For tables, I do not only store the raw row. I store enough surrounding structure to make the row meaningful:

row_text = (
    "Error Code: E1042\n"
    "Meaning: Webhook signature expired\n"
    "Resolution: Regenerate signing key and replay event\n"
    "From table: Error reference / Webhooks / Common failures"
)

Why this works:

The model does not only need the right passage. It needs the right passage in a form where the meaning is self-contained.

A chunk like:

E1042 | Webhook signature expired | Regenerate signing key

is much weaker than:

Error Code: E1042
Meaning: Webhook signature expired
Resolution: Regenerate signing key and replay event
From table: Error reference / Webhooks / Common failures

The second one is easier to retrieve, easier to rerank, and easier for the model to use faithfully.

⚠️ Gotcha:

Overlap is not a substitute for context. Overlap helps at sentence boundaries, but it does not recover lost table headers, section titles, or document-level metadata.

---

3. Retrieval became a small query planner

Scenario:

A customer asks, “How do I rotate API keys?” The system retrieves a generic security page instead of the tenant-specific admin guide. Another user asks, “What changed in v2?” The retriever returns v1 and v3 documentation because the query does not carry version intent.

Why it matters:

A raw user query is rarely the best retrieval query.

Users are terse. They use pronouns. They assume context. They mix product names, abbreviations, and incomplete descriptions. If you pass the raw query straight into a vector store, you are asking semantic search to solve problems that often belong to query planning.

Solution:

I introduced a lightweight planning stage before retrieval.

It does three things:

1. rewrite the query for retrieval
2. extract filters
3. choose the retrieval mode

@dataclass(frozen=True)
class QueryPlan:
    request_id: str
    raw_query: str
    retrieval_query: str
    filters: dict
    mode: str  # "hybrid", "keyword", "semantic"

A simplified planner:

def plan_query(raw_query: str, user_context: UserContext) -> QueryPlan:
    filters = {
        "tenant_id": user_context.tenant_id,
        "allowed_doc_types": user_context.allowed_doc_types,
    }

    if user_context.product_version:
        filters["product_version"] = user_context.product_version

    retrieval_query = rewrite_for_search(raw_query)
    mode = choose_retrieval_mode(raw_query)

    return QueryPlan(
        request_id=user_context.request_id,
        raw_query=raw_query,
        retrieval_query=retrieval_query,
        filters=filters,
        mode=mode,
    )

The important part is not the exact implementation. The important part is that retrieval now receives structured intent.

For example:

{
    "tenant_id": "acme",
    "product_version": "v2",
    "doc_type": ["admin_guide", "api_reference"]
}

That is much better than hoping the embedding model figures it out.

Why this works:

Most production RAG failures are not “the model is dumb.” They are retrieval-context failures.

A query planner lets you separate:

• what the user asked
• what should be searched
• what documents are allowed
• whether exact matching matters more than semantic matching

That separation becomes critical once you add multi-tenancy, permissions, or versioned documentation.

🔍 Why this matters:

Permission filtering should be part of retrieval, not a post-processing idea. If you retrieve first and filter later, you often lose the best candidates to inaccessible documents.

---

4. Hybrid search was the unglamorous fix for exact identifiers

Scenario:

A user searches for SKU-8842 or error E1042 or endpoint /v2/webhooks/signatures. Vector search returns conceptually related content, but not the exact item. Semantic search is good at meaning, but it can be surprisingly bad at identifiers.

Why it matters:

A lot of RAG demos focus on natural language questions:

• “How do refunds work?”
• “What is your billing policy?”
• “Summarize this document.”

But production systems also get queries like:

• ORD-55213 stuck in pending
• ERR_TIMEOUT during sync
• Stripe webhook 500 after upgrade
• INV-2049 tax calculation mismatch

These are not purely semantic queries. They often contain exact tokens that matter more than conceptual similarity.

Solution:

I stopped pretending vector search was enough by itself and moved to a hybrid retrieval model:

• vector search for semantic recall
• keyword search for exact terms, IDs, and rare tokens
• fusion to combine the result sets

A simple reciprocal rank fusion implementation looks like this:

from collections import defaultdict


def reciprocal_rank_fusion(
    ranked_lists: list[list[str]],
    k: int = 60,
) -> list[str]:
    scores: dict[str, float] = defaultdict(float)

    for ranked_list in ranked_lists:
        for rank, chunk_id in enumerate(ranked_list):
            scores[chunk_id] += 1.0 / (k + rank + 1)

    return sorted(scores, key=scores.get, reverse=True)

The idea is simple:

• semantic search contributes candidates that match meaning
• keyword search contributes candidates that match exact strings
• RRF gives extra credit to chunks that appear in multiple result lists

Why this works:

Hybrid search improves recall in a way that feels almost unfair when you first see it.

Vector search is great at:

• paraphrasing
• conceptual questions
• intent-level matching

Keyword search is great at:

• product names
• error codes
• function names
• filenames
• SKUs
• API routes
• rare domain terms

Together, they cover failure modes the other one misses.

Practical note:

The biggest hidden cost of hybrid search is not the code. It is maintaining the lexical side properly:

• analyzers
• stemming rules
• stopwords
• synonyms
• field weights
• tokenization of identifiers

If you treat keyword search as an afterthought, it will disappoint you.

---

5. Reranking became the highest-leverage quality gate

Scenario:

The retriever returns 30 chunks. The correct one is in the set, but it is buried below several plausible-looking alternatives. The prompt builder takes the top 6 by raw retrieval score. The model uses the wrong one because it appeared first.

Why it matters:

Retrieval score is not the same as answer usefulness.

A chunk can be semantically close and still be the wrong evidence because:

• it is from the wrong version
• it is too generic
• it is outdated
• it lacks the exact detail required
• it is a sibling section, not the target section

This is where reranking earns its place.

Solution:

The pipeline changed from:

retrieve top-k → put into prompt

to:

retrieve broad → rerank narrow → select evidence

A reranker can be any model or function that scores query-document relevance more carefully than the first-stage retriever.

A simple interface:

class Reranker(Protocol):
    def score(self, query: str, text: str) -> float:
        ...


def select_evidence(
    query: str,
    candidates: list[RetrievedChunk],
    reranker: Reranker,
    *,
    top_n: int = 6,
    min_score: float | None = None,
) -> list[RetrievedChunk]:
    scored = [(chunk, reranker.score(query, chunk.text)) for chunk in candidates]
    scored.sort(key=lambda item: item[1], reverse=True)

    selected: list[RetrievedChunk] = []

    for chunk, score in scored[:top_n]:
        if min_score is not None and score < min_score:
            break
        selected.append(chunk)

    return selected

Why this works:

The first retrieval stage should optimize for recall.

The reranking stage should optimize for precision.

The prompt should receive only the strongest evidence.

That separation gives you much more control than trying to make one retrieval pass do everything.

In practice, reranking often improves answer quality more than:

• prompt wording changes
• minor chunk size tweaks
• changing temperature
• adding more verbose instructions

It is not glamorous, but it is one of the highest-leverage upgrades you can make.

🧠 The important part:

Do not rerank hundreds of chunks per request. Retrieve a broader candidate pool, then rerank a manageable subset. Reranking is powerful, but it should still be treated as a scarce resource.

---

6. Embedding generation became a data-engineering job

Scenario:

A document source changes. A few hundred pages are updated. Later, you switch embedding models or change chunking. Suddenly you need to re-embed a large corpus without duplicating content, losing traceability, or overwhelming rate limits.

Why it matters:

In demos, embedding is a function call.

In production, embedding is a data pipeline.

You need to know:

• which document version produced this chunk
• which embedding model produced this vector
• whether the chunk changed since last index
• whether reindexing is incremental or full
• whether two documents are accidentally producing duplicate chunks
• whether a failed batch can be retried safely

Solution:

I started treating chunks and embeddings as derived artifacts with deterministic identifiers.

The most important change was using stable chunk IDs based on content and position, not random IDs.

import hashlib


def chunk_identity(
    doc_id: str,
    chunk_position: str,
    text: str,
) -> str:
    canonical = f"{doc_id}|{chunk_position}|{text.strip()}".encode("utf-8")
    return hashlib.sha256(canonical).hexdigest()

Then the database row becomes something you can upsert safely:

INSERT INTO chunks (
    chunk_id,
    doc_id,
    heading_path,
    content,
    embedding,
    embedding_model_version,
    updated_at
)
VALUES (
    $1, $2, $3, $4, $5, $6, now()
)
ON CONFLICT (chunk_id) DO UPDATE SET
    content = EXCLUDED.content,
    embedding = EXCLUDED.embedding,
    embedding_model_version = EXCLUDED.embedding_model_version,
    updated_at = EXCLUDED.updated_at;

Why this works:

When chunk identity is stable, reindexing becomes manageable.

You can:

• re-embed only changed documents
• keep model version in metadata
• compare old and new embeddings during migrations
• avoid duplicate chunks from repeated ingestion runs
• roll back to a previous embedding model more easily

This sounds boring until you have to migrate an index in production. Then it becomes one of the best decisions you made.

Production warning:

If your chunk IDs are random, every ingestion failure eventually becomes a data integrity problem. You will either accumulate duplicates or lose traceability between chunks and source documents.

---

7. Evaluation got easier once the pipeline had seams

Scenario:

You change the prompt. The answers look better on a few examples. But you do not know whether retrieval improved, whether the model is just writing more confidently, or whether you quietly broke a previously working case.

Why it matters:

RAG systems fail in multiple places:

1. The retriever did not find the right chunk.
2. The reranker buried the right chunk.
3. The prompt included too much noise.
4. The model ignored the evidence.
5. The evidence was wrong because the source data was stale.

If you only evaluate final answers, you cannot tell which of those happened.

Solution:

Once the pipeline had explicit stages, I started evaluating them separately.

For retrieval, the most useful metrics were simple:

• Hit@k
• MRR
• recall over known golden chunks
• failure analysis by document type

A small MRR helper:

def mean_reciprocal_rank(
    golden: list[tuple[str, set[str]]],
    retriever: Retriever,
    k: int = 10,
) -> float:
    if not golden:
        return 0.0

    total = 0.0

    for query, relevant_chunk_ids in golden:
        results = retriever.retrieve(query, limit=k)
        result_ids = [chunk.chunk_id for chunk in results]

        for rank, chunk_id in enumerate(result_ids, start=1):
            if chunk_id in relevant_chunk_ids:
                total += 1.0 / rank
                break

    return total / len(golden)

That alone changed the debugging conversation.

Instead of saying:

“The answers feel worse.”

we could say:

“Retrieval MRR dropped from 0.82 to 0.61 on API reference questions.”

That is a real engineering signal.

Why this works:

Separating retrieval evaluation from generation evaluation prevents you from optimizing the wrong thing.

If retrieval is bad, prompt engineering only hides the problem temporarily.

If retrieval is good but generation is bad, then prompt and model choices matter more.

Once those boundaries were clear, improvements became much less emotional.

💡 Practical note:

Keep a small golden set that is adversarial: exact IDs, version-specific questions, permission-sensitive documents, ambiguous phrasing, and multi-step questions. A golden set full of easy questions will flatter your pipeline and teach you nothing.

---

8. Observability changed from “the answer looks weird” to “chunk 7f2a was dropped”

Scenario:

A user reports a bad answer. You try the same question later and it works. Or you try it again and it fails, but you do not know why. The final response gives you almost nothing to debug with.

Why it matters:

RAG failures are often stateful and data-dependent.

They can depend on:

• which retriever was used
• which filters were applied
• which candidates were retrieved
• which chunks survived reranking
• which prompt template was selected
• which model version answered
• whether the source document had changed that morning

If you only log the final prompt and final answer, you are missing most of the story.

Solution:

I started logging a structured trace for every request.

The fields that proved most useful:

trace = {
    "request_id": plan.request_id,
    "raw_query": plan.raw_query,
    "retrieval_query": plan.retrieval_query,
    "filters": plan.filters,
    "mode": plan.mode,
    "candidate_chunk_ids": [c.chunk_id for c in candidates],
    "selected_chunk_ids": [c.chunk_id for c in evidence],
    "rerank_scores": rerank_scores,
    "prompt_template_id": prompt_template.id,
    "model_name": model.name,
    "latency_ms": latency_ms,
}

The important detail is that I log chunk IDs, not only text.

That lets me answer questions like:

• Was the right chunk retrieved but dropped by reranking?
• Was it retrieved but filtered out by permissions?
• Was it retrieved but excluded because of prompt budget?
• Did two queries hit different document versions?
• Did the source document change between incidents?

Why this works:

Observability in RAG is not just about LLM calls. It is about tracing evidence flow.

A good trace tells you where the evidence was lost:

query → filters → retrieval → fusion → rerank → prompt → generation

Without that, you end up guessing.

With it, you can often reduce a “bad answer” report to one of a few concrete causes in minutes.

---

9. What got worse: the long tail of boring integration work

This is the part people often skip in “I quit the framework” articles.

Removing LangChain did improve the core pipeline. But it also transferred a lot of maintenance burden back to me.

Document loaders are deceptively hard

The easy part of a loader is reading a file.

The hard part is handling:

• PDFs with multi-column layouts
• scanned documents
• tables that break across pages
• headers and footers polluting content
• HTML navigation menus mixed into article text
• Markdown with embedded components
• PowerPoint slides with text boxes in random order
• Confluence pages with macros
• Notion blocks with nested toggles
• tickets with rich text and attachments
• permission metadata that lives outside the document itself

LangChain and the broader ecosystem do not solve all of this perfectly, but they give you a starting point. When you remove that layer, you either:

• build your own parsers
• maintain wrappers around third-party extraction tools
• accept lower quality ingestion

None of those are free.

You inherit more operational glue

Once the framework is gone, you also own more of the surrounding plumbing:

• retry logic for embedding APIs
• batching and rate-limit handling
• ingestion workers
• schema migrations for vector indexes
• versioned prompt templates
• feature flags for retrieval modes
• cache invalidation
• multi-tenant filtering
• source synchronization
• failed document quarantine

This is not glamorous work. But it is the work that keeps a RAG system alive after the demo.

The ecosystem advantage is real

One underrated benefit of popular frameworks is that common patterns are already visible:

• loaders
• retrievers
• splitters
• record managers
• evaluators
• tracers
• example recipes

When you build everything yourself, you lose some of that shared vocabulary. That can slow down onboarding and make it easier to reinvent mediocre solutions.

So no, the rebuild was not a pure win.

It was a trade.

I gained control, but I also accepted responsibility for parts of the system I would have preferred to outsource.

---

10. Where I draw the line now

After going through this, I no longer ask:

“Should we use LangChain or not?”

That question is too broad.

The better question is:

“Which parts of the RAG pipeline need to be explicit, and which parts can remain high-level?”

For me, the answer depends on what the product actually needs.

Use a high-level framework when:

• you are still prototyping
• your main goal is speed
• retrieval quality is not the core product differentiator
• you need many quick integrations
• your documents are relatively simple
• you are still exploring whether RAG is even the right approach
• you need agent/tool orchestration more than retrieval precision

In those cases, LangChain or a similar framework can still be a very reasonable choice.

Build the core yourself when:

• answer quality is directly tied to product value
• retrieval must respect permissions, tenancy, or versioning
• your corpus contains messy documents
• exact identifiers matter as much as semantic meaning
• you need rigorous evaluation
• you need fine-grained observability
• you are optimizing cost and latency carefully
• you want stable interfaces across models and vendors

That is where owning the pipeline starts to pay off.

The comparison I keep coming back to

Dimension	LangChain / high-level framework	Custom pipeline
Speed to first demo	Very strong	Slower
Debugging retrieval behavior	Can be indirect	Usually clearer
Integration breadth	Strong	Requires more work
Control over chunking/retrieval	Medium	High
Evaluation discipline	Possible, but not automatic	Easier to enforce
Operational ownership	Partially shared	Mostly yours
Best fit	Exploration, broad tooling, agent-heavy systems	Production systems where retrieval quality is critical

A practical rule

If I am building a system where the user mostly needs an assistant over many tools, and retrieval is just one supporting feature, I am still open to high-level orchestration.

If I am building a system where the core promise is:

• accurate answers
• correct citations
• permission-aware search
• version-specific behavior
• reliable evidence selection

then I want the retrieval core to be explicit, boring, and fully owned.

That is the real lesson.

The benefit of removing LangChain was not that the pipeline became smaller. It became legible.

And once the pipeline was legible, every hard problem became easier to locate:

• bad chunking
• weak query planning
• missing hybrid recall
• noisy reranking
• stale documents
• broken permissions
• prompt overflow

At that point, improving RAG stopped feeling like prompt roulette and started feeling like engineering.

That is the state you want to reach, whether you use a framework or not.

Comments

[Thomas Hansen]

In Magic Cloud, I don't do chunking at all - Instead, I use gpt-5.6-luna to "break up longer documents into multiple facts".

[Hossein Hezami]

That's an interesting alternative to traditional chunking.

I think the key difference is that you're changing the retrieval unit from a document fragment into a semantic fact. That can be a much better fit for certain knowledge bases, especially when the original document structure isn't the best retrieval boundary.

The trade-off I'd be curious about is provenance and context preservation: once a document is decomposed into facts, how do you retain enough source structure to reconstruct the surrounding context when several facts need to be combined?

That's actually where I see this approach fitting nicely into the discussion in the article: chunking isn't necessarily the goal — producing retrieval units that preserve enough meaning and provenance is.

I'll have to look more closely at how you implemented the fact extraction in Magic.