The problem in context
The audit that led me to look hard at langchain alternatives started with a 2am page. A transitive dependency of a LangChain integration shipped a breaking change, our pinned versions had drifted, and the on-call engineer spent an hour bisecting a dependency tree to restore a feature that, at its core, does exactly one thing: retrieve relevant chunks and answer a question over them.
The deeper problem wasn't the outage. It was that nobody on the team could fully explain our own retrieval path anymore. We had reached for LangChain reflexively on day one — everyone does; it is the default — and accumulated so many abstractions between the query and the answer that the system had become opaque to the people who owned it. Opaque systems fail at 2am, and they fail slowly, because you cannot point at the layer that broke.
This is worth stating plainly because it is not a hit piece. LangChain genuinely solves a real problem: it orchestrates tools, prompts, memory, and model calls into coherent multi-step flows, and for a true agentic assistant that abstraction earns its keep. The failure here was not the framework. It was reaching for a general-purpose orchestrator to run a workload that was never agentic in the first place.
The principle
The principle here is that you choose a RAG framework by workload, not by momentum. A framework's abstractions are a tax you pay in latency, dependency surface, and debuggability. That tax is worth paying when your problem shape matches what the framework abstracts over, and it is dead weight when it does not.
Applied to our product, the principle exposed something the branding had hidden: we did not have one workload, we had two, wearing one framework. An enterprise semantic search over our docs, and a fast document-QA feature over uploaded PDFs. Neither was an agent. We were paying the full orchestration cost twice for two problems that each had a sharper, dedicated tool.
There is a useful decision guide in a third-party comparison that matches RAG tools to the jobs they are actually good at, and it reads like it was written against our backlog:
- RAG at production scale: Haystack.
- Fast document indexing and QA: LlamaIndex.
- Intent-based multi-turn dialog: Rasa.
- Low-code customer bots: Botpress or Dialogflow.
- Direct model access and fine-tuning: Hugging Face Transformers.
- Multi-agent orchestration: CrewAI, AutoGen, or DSPy.
We evaluated the agent-first options honestly. CrewAI orchestrates role-playing agents into crews and event-driven flows; AutoGen leans on multi-agent conversation while CrewAI mirrors organizational workflows; DSPy trades manual prompt engineering for declarative, self-optimizing programs (ZenML's side-by-side is a fair read). All genuinely interesting — and all built for multi-agent orchestration, which is not the axis our problem lived on. Two retrieval workloads meant Haystack and LlamaIndex won on the only axis that mattered.
Trade-offs
Mapping our two workloads onto the guide produced a clean split. The table is the whole argument:
| Workload | Needs | Tool | What you give up |
|---|---|---|---|
| Enterprise semantic search | Explainable, testable, scalable, self-hostable | Haystack | More moving parts, a vector DB to run |
| Uploaded-PDF doc-QA | Fast setup, low latency, low compute | LlamaIndex | Deliberately narrow; not an orchestrator |
| True multi-tool agent | Tool calling, memory, prompt templating | LangChain / CrewAI | Latency + dependency tax on the hot path |
Enterprise search → Haystack. This workload has to be explainable, testable, and scalable — exactly the profile Haystack targets. It is an open-source orchestration framework built around modular pipelines with explicit control over retrieval, routing, and generation (Haystack docs), it works with real vector backends (Elasticsearch, OpenSearch, Weaviate), and it is self-hostable — which our data-privacy requirements demanded. The pipeline shape is legible; you can see every stage:
from haystack.document_stores import InMemoryDocumentStore
from haystack.nodes import DensePassageRetriever, FARMReader
from haystack.pipelines import ExtractiveQAPipeline
document_store = InMemoryDocumentStore()
retriever = DensePassageRetriever(document_store=document_store)
reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2")
pipeline = ExtractiveQAPipeline(reader=reader, retriever=retriever)
response = pipeline.run(query="What is Haystack used for?")
Retriever, reader, pipeline — the retriever scores documents and hands the top candidates to the reader (how retrievers work). When something is slow or wrong, you know which node to inspect. In production we swapped InMemoryDocumentStore for OpenSearch and left everything else intact.
Doc-QA → LlamaIndex. This workload needs fast, relevant QA where compute budget and latency matter. LlamaIndex is a data framework for connecting LLMs to your own data — ingest, index, query (LlamaIndex) — and it is deliberately narrower than LangChain. The narrowness is the feature:
from llama_index import SimpleDirectoryReader, GPTTreeIndex
documents = SimpleDirectoryReader("<directory_path>").load_data()
index = GPTTreeIndex(documents)
response = index.query("What is the purpose of this document?")
Three lines from a folder of PDFs to a queryable index. For a feature where users expect an answer in under a couple of seconds, shedding orchestration overhead is a direct latency win.
The outcome of the split, measured against the monolith:
| Metric | LangChain monolith | Haystack + LlamaIndex |
|---|---|---|
| p95 query latency | baseline | down ~43% |
| Deps in the RAG path | heavy | roughly halved |
| "Which layer failed?" | opaque | node-level clarity |
| Framework-churn incidents / quarter | 3-4 | ~0 |
| Self-host / data-privacy fit | workable | first-class |
| Onboarding to the pipeline | days | hours |
The latency drop came from removing indirection on the hot path; the reliability drop from a smaller, more purposeful dependency surface; the debugging clarity from being able to point at a retriever or a reader node.
How to adopt
The adoption mistake is the big-bang rewrite. Stage it, and gate every step on measurement.
- Shadow mode. Run the candidate alongside the incumbent, send the same queries to both, and diff results and latency for a couple of weeks. No user impact.
- Feature-flag cutover. Once the candidate's answer quality matches or beats the old path on your eval set, flip traffic over a percentage at a time.
- Carve out independent workloads separately. We migrated the PDF feature to LlamaIndex on its own because it shared no state with search.
- Delete last. Only after both paths ran green for a full release did we remove the LangChain dependency.
The non-negotiable running through all four steps is an eval harness — a fixed set of questions with known-good answers — so "is the new thing actually as good?" is a number, not a vibe. Ours earned its keep on day one of shadow mode: Haystack's DensePassageRetriever pulled better passages on about a fifth of queries, but the extractive reader truncated a few long answers the generative path had handled. Without the eval set we would have shipped a silent regression; with it, the fix was obvious — route long-form questions to a generative reader, keep the extractive one for precise lookups. A framework change you cannot measure is a bet, not an engineering decision.
Two guardrails on the principle. Do not cargo-cult this specific split: if you are building an intent-driven support bot, the answer is Rasa or a low-code tool, not Haystack; if you need fine-tuning and raw model control, it is Hugging Face Transformers. And do not ban the incumbent — we kept LangChain on the table for the day we build a genuine multi-tool agent, because that is the workload it is best at.
Where this goes next
The near-term work is incremental and, by design, all measured against the same eval harness: a reranking node in the Haystack pipeline, a generative reader for longer-form answers, and a Rasa intent layer for a support-bot experiment — tool matched to workload each time.
The larger trajectory is what makes the workload-first principle durable rather than a one-off win. The framework landscape is churning fast — the 2026 shortlist already folds in agent-first options like CrewAI, AutoGen, and DSPy and low-code RAG builders like RAGFlow and Flowise, and it will churn again. Teams that pick by hype re-litigate their entire stack every time the default shifts. Teams that decompose their product into named workloads and choose per workload only revisit the piece that actually changed. As agentic patterns and retrieval patterns keep diverging into specialized tools, that decomposition is the thing that ages well. If LangChain feels like it is fighting you in production, the fix is probably not more LangChain — it is the right framework for your actual workload.
Sources & further reading
- Haystack — Introduction — the modular-pipeline model for production RAG.
- Haystack — Retrievers — how retrieval nodes score and return documents.
- LlamaIndex (run-llama) — the data framework for indexing and querying your own documents.
- CrewAI — Introduction — role-based multi-agent orchestration, for when your problem really is agents.
- ZenML — CrewAI vs AutoGen — a fair comparison of the agent-framework options.
- A workload-first comparison of LangChain alternatives — the runnable per-option reference this migration drew on.
Top comments (0)