Menlo Ventures surveyed nearly 500 US enterprise decision-makers for its 2025 State of Generative AI report: 51% of production LLM deployments use retrieval-augmented generation, against 9% for fine-tuning. Long context windows did not kill RAG. What died is the naive chunk-and-retrieve pipeline; retrieval in 2026 is increasingly a behavior an agent invokes on demand. And with 76% of enterprise AI now bought rather than built, managed RAG services compete directly with the open-source frameworks.
I published a fuller version of this guide on DevToolLab: Best RAG Platforms & Tools in 2026. This condensed version covers the orchestration layer only - ingestion, chunking, embedding, retrieval, reranking, generation - not the vector database underneath, which is its own decision.
The pipeline every tool implements
All RAG systems run the same six steps: ingest documents, parse and chunk them, embed the chunks into vectors, retrieve the relevant ones for a query (usually hybrid dense + BM25), rerank the candidates, and generate a cited answer. Frameworks give you composable pieces and full control. Managed platforms hide the steps behind an upload-and-ask API. That is the whole build-vs-buy decision.
The contenders at a glance
| Tool | Type | License | Best for |
|---|---|---|---|
| LlamaIndex | OSS framework | MIT | Ingestion and messy documents |
| LangChain / LangGraph | OSS framework | MIT | Integrations, agentic RAG |
| Haystack | OSS framework | Apache 2.0 | Typed production pipelines |
| DSPy | OSS framework | MIT | Optimizing instead of prompt-tuning |
| txtai / R2R | OSS, lightweight | Apache 2.0 / MIT | Self-hosted all-in-one / RAG server |
| Bedrock Knowledge Bases | Managed (AWS) | Proprietary | Managed RAG on AWS |
| Vertex AI RAG Engine | Managed (GCP) | Proprietary | Managed RAG with Gemini |
| Azure AI Search | Managed (Azure) | Proprietary | Enterprise hybrid retrieval |
| Pinecone Assistant | Managed API | Proprietary | Fastest documents-to-answers |
| Contextual AI | Managed platform | Proprietary | Regulated, accuracy-critical work |
The open-source frameworks
LlamaIndex (MIT, ~51k stars, $19M Series A in June 2026) wins when your data is the hard part. LlamaHub connectors, the LlamaParse parser and LlamaCloud make it the strongest ingestion story here, and Workflows handle agentic orchestration. The cost: a big, fast-moving API surface and some confusion about where the open framework ends and LlamaCloud begins.
LangChain and LangGraph (MIT, ~142k stars, both 1.0 since October 2025, $125M raised at ~$1.25B) bring the largest integration ecosystem, and LangGraph is the natural home for agentic RAG where the model decides whether and what to retrieve, loops, and reformulates. LangSmith covers tracing. The long-standing criticism holds for simple cases: a plain retrieve-then-answer pipeline needs less machinery than this.
Haystack from deepset (Apache 2.0, ~26k stars) models RAG as typed Components in explicit Pipelines, with an async runtime, YAML serialization and built-in evaluation. If LangChain feels like notebook glue, Haystack feels like deployable software. Smaller community, fewer tutorials.
DSPy from Stanford NLP (MIT, ~36k stars) is the genuinely different one: declare modules with input/output signatures, then let optimizers like MIPROv2 and GEPA compile the prompts and few-shot examples. You stop hand-tuning prompts and start optimizing programs. Real learning curve; overkill for a demo.
txtai (Apache 2.0, ~12.7k stars) bundles vector, graph and relational indexes with RAG and agents in one library - no separate vector DB to run. R2R (MIT, ~7.9k stars) is a deployable RAG server exposing ingestion, hybrid search, knowledge graphs and agentic retrieval over REST. Both trade community size for setup simplicity.
The managed platforms
Bedrock Knowledge Bases is the AWS default: GraphRAG on Neptune Analytics went GA in March 2025, and re:Invent 2025 added multimodal retrieval (text, image, audio, video), S3 Vectors storage and text-to-SQL structured retrieval. Vertex AI RAG Engine is the GCP equivalent, tightly coupled to Gemini, with a serverless mode and pluggable vector backends including Pinecone and Weaviate. Azure AI Search has best-in-class hybrid search plus semantic ranking, and its agentic retrieval (LLM-decomposed parallel subqueries) reached GA in the 2026-04-01 REST API - but it is a retrieval engine, not an end-to-end app.
Pinecone Assistant went GA in February 2026: upload documents, get grounded cited answers from a Chat API with evaluation built in. Contextual AI, founded by RAG-paper co-author Douwe Kiela with ~$100M raised, jointly optimizes retriever and generator ("RAG 2.0") for regulated knowledge work.
One warning from the startup end: Ragie, a developer-focused managed RAG service, announced its shutdown effective July 19, 2026. A managed platform removes your infrastructure work, and a small vendor can also remove itself.
The bolt-on tools that actually move quality
Three categories reliably lift RAG accuracy regardless of platform. A reranker (Cohere Rerank, Voyage AI - now inside MongoDB) is often the cheapest accuracy win in the pipeline. Document parsing decides everything downstream: IBM's Docling (~63k stars, Linux Foundation) is free and local, Unstructured covers the most formats, LlamaParse and Reducto handle complex tables and scans. Evaluation closes the loop: Ragas gives you faithfulness and context precision/recall without labeled data, and TruLens (Snowflake) adds tracing.
Chunking: the step nobody does for you well
Retrieval quality is decided at the chunking step, before any model runs. Fixed-size chunks with overlap are the sane default, and the overlap is what keeps a sentence at a chunk boundary attached to its context. The core logic fits in a page of dependency-free Python:
def chunk_text(text, chunk_size=200, overlap=40):
"""Split text into overlapping word chunks for embedding."""
if overlap >= chunk_size:
raise ValueError("overlap must be smaller than chunk_size")
words = text.split()
step = chunk_size - overlap
chunks = []
for start in range(0, len(words), step):
chunk = words[start:start + chunk_size]
if chunk:
chunks.append(" ".join(chunk))
if start + chunk_size >= len(words):
break
return chunks
Production systems split on sentences or semantic boundaries and size by tokens rather than words, but this is the shape of what every pipeline does first. While tuning chunk sizes, DevToolLab's AI Token Counter checks how much retrieved context actually fits in a model's window, and the Cosine Similarity Calculator lets you sanity-check the similarity math your vector store runs at scale.
2026 trends worth building around
Agentic RAG is now the dominant pattern - first-class in Azure AI Search, Vertex RAG Engine, R2R and any LangGraph agent. Multimodal RAG went mainstream in the managed platforms this year. GraphRAG, popularized by Microsoft's open-source project, is productized in Bedrock. And the buy-over-build tide means the honest starting point for many teams is a managed platform, dropping to a framework only when they hit a wall.
How to choose
- No infrastructure appetite: managed platform on your existing cloud (Bedrock, Vertex, Azure AI Search, or Pinecone Assistant).
- Messy PDFs and many sources: LlamaIndex.
- Production rigor and testable pipelines: Haystack.
- Agentic retrieval with loops and human-in-the-loop: LangGraph.
- Systematic accuracy optimization: DSPy.
- Lightweight and self-hosted: txtai, or R2R for a REST API.
Whatever you pick, the parts no platform owns - parsing quality, chunking, a reranker, an eval loop - decide whether the system is trustworthy. The original guide goes deeper on each tool, with funding details, feature timelines and the full decision framework.
Top comments (0)