DEV Community: Parth Sarthi Sharma

Reflection vs Reflexion Agents: The Next Leap in Agentic AI

Parth Sarthi Sharma — Sun, 22 Mar 2026 03:04:03 +0000

As generative AI systems evolve from simple prompt-response tools into autonomous agents, one capability is becoming increasingly critical:

The ability for AI systems to improve themselves during execution.

This is where two powerful concepts come into play:

Reflection
Reflexion

They sound similar. They are often confused.

But architecturally — and practically — they are very different.

Let’s break them down.

🚀 Why This Matters

If you're building:

AI copilots
Autonomous workflows
Multi-step reasoning systems
Or agentic architectures

Then how your system learns from mistakes will define:

Accuracy
Reliability
Cost efficiency
User trust

🧠 What is Reflection?

Reflection is when an AI system:

Reviews its own output and improves it within the same execution loop.

🔁 How it works

Generate response
Evaluate response (self-critique or evaluator model)
Refine response
Repeat until acceptable

🧩 Architecture Pattern

User Input
↓
LLM → Output
↓
Self-Evaluation (LLM or rule-based)
↓
Refinement Loop
↓
Final Output

✅ Key Characteristics

Happens within a single session
No memory across runs
Iterative improvement
Often uses:
- Self-critique prompts
- Evaluation models
- Chain-of-thought refinement

💡 Example

User asks:

"Summarize this legal document."

Reflection agent:

Generates summary
Checks:
- Missing clauses?
- Ambiguity?
Refines output

👍 Pros

Improves output quality instantly
No infrastructure complexity
Easy to implement

👎 Cons

No long-term learning
Repeats same mistakes across sessions
Increased latency (multiple LLM calls)

🔁 What is Reflexion?

Reflexion goes a step further.

It enables an AI system to learn from past mistakes and improve future performance.

This concept was popularized by research on self-improving agents with memory.

🔄 How it works

Perform task
Evaluate outcome
Store feedback in memory
Use memory to improve future decisions

🧩 Architecture Pattern

User Input
↓
Agent Execution
↓
Outcome Evaluation
↓
Memory Store (success/failure insights)
↓
Future Runs Use Memory

🧠 Key Difference

Reflection	Reflexion
Session-based	Cross-session
No memory	Persistent memory
Improves current output	Improves future outputs
Stateless	Stateful

💡 Example

AI agent writing grant applications:

Attempt 1: Rejected ❌
Stores feedback:
- "Too generic"
- "Lacks domain-specific references"

Next attempt:

Uses stored insights
Produces better output ✅

🔥 Why Reflexion is a Big Deal

Reflexion introduces something critical:

Learning without retraining the model

Instead of fine-tuning:

You store experiences
You adapt behavior dynamically

🏗️ Real-World Implementation

Reflection (simple)

Prompt chaining
Self-critique prompts
ReAct-style loops

Reflexion (advanced)

Requires:

Memory layer:
- Vector DB (e.g., embeddings)
- Key-value store
Feedback signals:
- Human feedback
- Automated scoring
Retrieval mechanism:
- Inject past learnings into prompts

⚙️ Example Stack

LLM: Claude / GPT / Nova
Memory: Vector DB (FAISS, OpenSearch)
Orchestration: LangChain / custom agents
Evaluation: Rule-based or LLM-as-judge

⚖️ When to Use What?

Use Reflection when:

You need better answers now
No need for memory
Simpler workflows

Use Reflexion when:

Tasks are repetitive and evolving
Feedback is available
Long-term improvement matters

🧠 Combining Both (Best Practice)

The most powerful systems use both:

Reflexion (long-term learning)
+
Reflection (short-term refinement)

👉 This creates:

Immediate quality improvement
Continuous learning over time

🧪 Real-World Use Cases

AI coding assistants
Customer support agents
Financial advisory copilots
Healthcare decision support
Autonomous research assistants

⚠️ Challenges

Reflection

Cost (multiple LLM calls)
Latency

Reflexion

Memory design complexity
Signal quality (bad feedback = bad learning)
Retrieval accuracy

🧭 Final Thoughts

We are moving from:

Prompt → Response

to:

Prompt → Reason → Reflect → Learn → Improve

🔥 Key Insight

Reflection makes AI smarter in the moment

Reflexion makes AI smarter over time

✍️ Closing

If you're building next-gen AI systems,

understanding this difference is not optional — it's foundational.

The future of AI is not just about better models.

It’s about better systems around those models.

💬 Curious how to implement Reflexion in production?

Happy to share a deep dive in the next post.

Prompt Engineering Is Not Enough: Enter Flow Engineering for Production LLM Systems

Parth Sarthi Sharma — Sat, 07 Mar 2026 03:20:46 +0000

Large Language Models have unlocked a new generation of applications — copilots, assistants, RAG systems, autonomous agents, and internal AI tools.

But many teams building with LLMs hit the same wall.

Their application works in demos… but becomes unreliable in production.

Why?

Because prompt engineering alone is not enough.

To build reliable AI systems, we need something more powerful:

Flow Engineering.

In this article, we'll explore:

Why prompt engineering alone fails in production
What Flow Engineering actually means
The architecture of real-world LLM systems
Practical examples engineers can implement today

The Era of Prompt Engineering

When GPT-style models first became popular, the focus was on prompt engineering.

Prompt engineering is the art of crafting instructions to guide the LLM to produce better responses.

Example:

You are a helpful assistant. 
Summarise the following meeting transcript in bullet points.
Focus only on action items.

Developers quickly discovered techniques like:

Few-shot prompting
Chain-of-thought prompts
Role prompting
Structured output prompts

These techniques improve individual LLM calls.

But they only solve part of the problem.

Prompt engineering optimises one interaction.

Real applications involve many interactions and system components.

The Problem with Prompt-Only Systems

Let's imagine we are building a simple customer support AI assistant.
A naive architecture might look like this:

User Question
      ↓
     LLM
      ↓
   Response

This works in simple demos.

But real systems quickly require more complexity.

For example:

Retrieve relevant documents
Use tools (APIs, databases)
Validate outputs
Retry on errors
Maintain conversation context
Apply guardrails
Log reasoning steps

Suddenly, our architecture looks more like this:

User Question
      ↓
Context Retrieval (RAG)
      ↓
Tool Selection
      ↓
LLM Reasoning
      ↓
Output Validation
      ↓
Response Generation

This multi-step pipeline is where Flow Engineering comes in.

What Is Flow Engineering?

Flow Engineering is the design of structured execution flows around LLMs.

Instead of focusing on a single prompt, engineers design end-to-end reasoning pipelines.

Think of it as:

Prompt Engineering = How the LLM thinks

Flow Engineering = How the system operates

Flow engineering involves designing:

Execution pipelines
Tool orchestration
State management
Error handling
Validation
Feedback loops

In other words:

Flow engineering treats LLM applications as distributed systems, not chatbots.

A Real Production Flow

Let's look at a simplified production AI flow.

User Question
   ↓
Input Guardrails
   ↓
Context Retrieval (Vector DB)
   ↓
Tool Routing
   ↓
LLM Reasoning
   ↓
Tool Execution
   ↓
Response Validation
   ↓
Final Answer

Each step solves a real engineering problem.

Guardrails

Prevent prompt injection or malicious input.

Context Retrieval

Fetch relevant documents using vector search.

Tool Routing

Determine which tools the AI should use.

Validation

Ensure output matches schema or safety rules.

Without this flow, AI systems become unpredictable.

Example: Prompt vs Flow

Let's compare two implementations.

Prompt Engineering Only

response = llm.invoke(
    "Summarise this transcript and extract action items."
)

This may work sometimes.

But what if:

transcript is too long
model hallucinate action items
output format changes
context is missing

Now let's see a flow-based approach.

Example: Flow Engineered System

def generate_meeting_summary(transcript):

    chunks = split_transcript(transcript)

    summaries = []

    for chunk in chunks:
        summary = llm.invoke(
            f"Summarise this transcript section:\n{chunk}"
        )
        summaries.append(summary)

    combined_summary = llm.invoke(
        f"Combine these summaries and extract action items:\n{summaries}"
    )

    validated_output = validate_schema(combined_summary)

    return validated_output

Now we have:

chunking
intermediate reasoning
structured validation

This dramatically improves reliability.

Key Components of Flow Engineering

Most production LLM flows include these components.

1. State Management
Flows maintain state across steps.

Example:

Conversation History
Retrieved Documents
Tool Results

Frameworks like LangGraph model this using state machines.

2. Tool Orchestration

LLMs often interact with tools.

Examples:

databases
APIs
search engines
internal systems

Flow engineering controls:

which tool to use
when to call it
how to merge results

3. Retry & Error Handling

LLMs are probabilistic.

Sometimes outputs are invalid.

A flow can automatically:

retry generation
correct formatting
request clarification

4. Guardrails & Validation

Before returning outputs, systems often validate:

JSON schema
safety policies
hallucinations

This prevents unreliable responses.

Flow Engineering Frameworks

Several frameworks help engineers implement LLM flows.

LangGraph

Models AI workflows as state machines.

Great for:

complex agent workflows
branching logic
memory management

Semantic Kernel

Popular in enterprise environments.

Supports:

planners
function calling
workflow orchestration

Custom Orchestration

Many teams implement flows directly using:

Python
Node.js
serverless pipelines

Because flows are essentially application logic.

Why Flow Engineering Matters

Companies deploying production AI systems quickly discover:

The challenge is not the model.

The challenge is system design around the model.

Flow engineering provides:

✔ reliability
✔ reproducibility
✔ observability
✔ safety
✔ scalability

Without it, LLM applications behave unpredictably.

The Shift AI Engineers Must Make

Early LLM development focused on prompts.

But the industry is moving toward AI systems engineering.

That means thinking in terms of:

pipelines
workflows
orchestration
tool ecosystems

In short:

AI applications are evolving from prompt-driven apps to flow-driven systems.

Final Thoughts

Prompt engineering is still important.

But in production systems, prompts are only one component.

The real power of modern AI systems comes from well-designed execution flows.

If you want reliable AI applications, start thinking like a systems engineer, not just a prompt writer.

What’s Next

In upcoming articles, we'll dive deeper into:

Reflection vs Reflexion agents
LangGraph state machines
Semantic Kernel orchestration
Model Context Protocol (MCP)

These concepts build on flow engineering to create more capable AI systems.

Secrets Management for LLM Tools: Don’t Let Your OpenAI Keys End Up on GitHub 🚨

Parth Sarthi Sharma — Sat, 14 Feb 2026 04:04:56 +0000

"A practical guide to securing LLM API keys, embeddings, vector

TL;DR: If you're building with LLMs and you're not treating secrets as first-class infrastructure, you're already at risk.

Every week, we see:

OpenAI keys pushed to GitHub
API keys logged in CloudWatch
Secrets hardcoded in Streamlit demos that later go to production

LLM systems multiply secrets quickly. If you don’t design for this early, things get messy fast.

This is a production-ready blueprint for securing LLM systems properly.

The Problem: LLM Secrets Multiply Fast 🐰

One LLM integration turns into dozens of credentials:

1 LLM API key (OpenAI / Anthropic)
→ 3 embedding endpoints
→ 5 vector store connections (Pinecone / Weaviate)
→ 2 RAG databases
→ 10 external tools (SerpAPI, Wolfram, etc.)
→ 50 microservices
= 70+ secrets

The bigger your AI system gets, the larger your attack surface becomes.

1️⃣ Never Hardcode Secrets

❌ Wrong (guaranteed leak eventually)

# NEVER DO THIS
from openai import OpenAI

client = OpenAI(api_key="sk-123...")

Hardcoded secrets:

End up in git history
Get copied into logs
Leak via screenshots or stack traces

✅ Right: Runtime Environment Injection

# config.py

import os
from openai import OpenAI

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

client = OpenAI(api_key=OPENAI_API_KEY)

Principle:
Secrets should be injected at runtime, never committed to source code.

2️⃣ Use Cloud-Native Secrets Managers

If you're in production, use a managed secrets service.

AWS Secrets Manager + Lambda Example

# lambda_function.py
import json
import boto3
from openai import OpenAI

def get_secrets():
    client = boto3.client("secretsmanager")
    secret = client.get_secret_value(SecretId="llm-prod/openai")
    return json.loads(secret["SecretString"])

def lambda_handler(event, context):
    secrets = get_secrets()
    client = OpenAI(api_key=secrets["OPENAI_API_KEY"])
    # LLM logic here

Benefits:

Centralized storage
IAM-based access control
Audit logs
Automatic rotation support

Terraform for Secret Infrastructure

resource "aws_secretsmanager_secret" "llm_keys" {
  name = "llm-prod/openai"
  tags = {
    Environment = "Production"
    Team        = "AI"
  }
}

resource "aws_secretsmanager_secret_version" "llm_keys_version" {
  secret_id     = aws_secretsmanager_secret.llm_keys.id
  secret_string = jsonencode({
    OPENAI_API_KEY    = "sk-..."
    ANTHROPIC_API_KEY = "sk-ant-..."
    PINECONE_API_KEY  = "pxl-..."
  })
}

Infrastructure-as-Code ensures:

Repeatability
Auditability
No manual copy-paste secret management

3️⃣ Prefer Dynamic Credentials Over Static API Keys ⚡

Static API keys are long-lived and high risk.

Dynamic credentials reduce blast radius.

IAM Roles for Service Accounts (Kubernetes + AWS IRSA)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: llm-worker
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/llm-worker-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-worker
spec:
  template:
    spec:
      serviceAccountName: llm-worker
      containers:
        - name: llm-worker
          env:
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: llm-secrets
                  key: openai-key

Even better: eliminate API keys entirely where possible and use workload identity federation.

4️⃣ Secure CI/CD with OIDC (No Long-Lived AWS Keys)

Never store AWS credentials in GitHub secrets if you can avoid it.

Use OIDC federation instead:

name: Deploy LLM Pipeline

on: [push]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-llm-deploy
          aws-region: us-east-1

      - run: python deploy.py

This avoids:

Static AWS access keys
Manual credential rotation
CI secret sprawl

5️⃣ Agentic LLM Systems Need Scoped Secrets 🧠

When building multi-agent systems:

Each agent should have scoped credentials
Short-lived tokens preferred
No shared global API key across agents

Example pattern:

class LLMAgentSecrets:
    def __init__(self, sm_client):
        self.sm_client = sm_client

    def get_agent_secret(self, agent_id: str):
        secret_name = f"llm-agent-{agent_id}"
        secret = self.sm_client.get_secret_value(SecretId=secret_name)
        return secret["SecretString"]

Design for:

Isolation
Least privilege
Auditable access

✅ Production Security Checklist

☐ No hardcoded secrets (git grep -i "sk-")
☐ Cloud secrets manager in use
☐ IAM roles preferred over static keys
☐ OIDC for CI/CD
☐ Secrets scanning enabled (TruffleHog, GitGuardian)
☐ Log sanitization in place
☐ Rotation policy defined (≤ 90 days)
☐ Audit logging enabled
☐ Least privilege enforced

Common Leak Vectors 🚫

Leak Vector	Detection	Prevention
Git commits	`git log -p	grep sk-`	Pre-commit hooks
Logs	CloudWatch Insights	Log scrubbing
Docker images	Inspect image layers	Multi-stage builds
Memory dumps	`/proc/[pid]/environ`	Container hardening

Cost vs Risk 💰

Typical monthly cost for secure secrets management:

AWS Secrets Manager: ~$0.40 per secret
Secret scanning tools: modest monthly fee
OIDC: no additional cost
Compare that to:
Revoking leaked keys
Service outages
Customer trust damage

Security is cheaper than cleanup.

Key Takeaways 🎯

Dynamic > Static
Inject at runtime, never commit
Audit secret access
Rotate regularly
Scan continuously
Apply least privilege everywhere

LLMs are powerful.

But API keys are still just credentials — treat them like production infrastructure.

Have you ever dealt with an exposed LLM API key in production? What happened?
Let’s discuss 👇

Observability in AI Systems

Parth Sarthi Sharma — Tue, 27 Jan 2026 13:17:25 +0000

Why RAG Pipelines Fail Silently (and How to See It)

Traditional software taught us a hard lesson:

If you can’t observe it, you can’t operate it.

AI systems — especially RAG pipelines — are repeating the same mistakes we made with distributed systems a decade ago.

They look fine.
They respond fast.
They return answers.

And yet — they are quietly wrong.

This article explains:

Why observability is fundamentally harder in AI systems
What observability actually means for RAG pipelines
What signals matter (and which ones don’t)
How mature teams design observable AI systems

No dashboards for the sake of dashboards — only what helps you debug reality.

Why AI Observability Is Different From Traditional Observability

In classic systems, we observe:

CPU
memory
latency
error rates

In AI systems, the hardest failures are:

Semantic
Probabilistic
Contextual

A RAG pipeline can:

return HTTP 200
respond in 300ms
use the correct model

…and still give a wrong answer.

That’s why AI observability must go deeper.

The Core Problem With RAG Pipelines

A basic RAG flow looks like this:

User Query
↓
Embedding
↓
Vector Search
↓
Top-K Chunks
↓
Prompt Assembly
↓
LLM Generation

When the output is wrong, where did it fail?

Bad query?
Wrong chunks?
Missing chunks?
Prompt formatting?
Model hallucination?

Without observability, you’re guessing.

What “Observability” Means in the AI World

AI observability is the ability to answer:

Why did the system produce this answer for this input at this time?

That requires traceability, not just metrics.

The Four Pillars of RAG Observability

1️⃣ Query Observability

You must log:

Original user query
Rewritten / normalized query (if any)
Detected intent or routing decision

Why it matters

Many failures start with:

ambiguous questions
underspecified intent
bad query rewriting

If you can’t see the effective query, you can’t debug retrieval.

2️⃣ Retrieval Observability (Most Important)

This is where most RAG systems fail.

You should observe:

Retrieved chunk IDs
Source documents
Similarity scores
Chunk rank
Retrieval strategy used (vector, keyword, hybrid)

Example questions observability should answer:

Which chunks were retrieved?
Which chunk influenced the answer most?
Was relevant information missing?

If you don’t log retrieved chunks, you don’t have RAG observability.

3️⃣ Prompt Observability

Your prompt is your runtime program.

You must capture:

Final prompt sent to the LLM
Context size and token count
Chunk ordering
System instructions

Why?
Because subtle changes in:

ordering
truncation
formatting

can completely change answers.

4️⃣ Generation & Answer Observability

Beyond the final answer, log:

Model name & version
Temperature / decoding params
Token usage
Latency
Safety or refusal triggers

Advanced systems also track:

Answer confidence
Self-evaluation scores (Self-RAG)
Groundedness signals

The Most Common RAG Failure Modes (Seen in Production)

❌ “The model hallucinated”

Usually false.

More often:

Wrong chunk retrieved
Right chunk ranked too low
Context truncated
Outdated document used

Observability makes this visible.

❌ “Vector search is bad”

Often:

Chunking is wrong
Embedding mismatch
Query rewriting failed

Again — visible with the right signals.

Tracing a Single RAG Request (What Good Looks Like)

A single request trace should show:

Request ID: 9f23...

Query:
"Can I carry forward unused leave?"

Rewritten Query:
"Leave carry forward policy Australia"

Retrieved Chunks:

handbook.md#leave-carry-forward (score: 0.89)

policy.md#exceptions (score: 0.81)

Prompt Tokens:
3,214

Model:
gpt-4.1-mini

Answer:
"Yes, up to 10 days can be carried forward..."

Confidence:
High

If you can’t reconstruct this — you can’t debug.

Why Traditional Metrics Are Not Enough

Latency and cost are necessary — but insufficient.

AI systems need semantic metrics:

Groundedness
Faithfulness
Retrieval coverage
Answer stability over time

These are harder — but essential.

Observability Enables Advanced RAG Patterns

You cannot safely implement:

Adaptive RAG
Corrective RAG
Self-RAG

without observability.

Why?
Because all of them rely on feedback signals:

Was retrieval good?
Was the answer grounded?
Should we retry?

No signals → no control loop.

A Simple Observability Checklist

If you’re building RAG in production, you should be able to answer:

Which document influenced this answer?
Why was this chunk chosen over others?
What changed compared to yesterday?
Would a different retrieval strategy help?
Can I replay this request?

If the answer is “no” — observability is missing.

Final Thought

RAG pipelines don’t usually fail loudly.

They fail:

quietly
confidently
at scale

The future of AI systems isn’t just better models.

It’s systems that can explain themselves.

And observability is how that starts.

If you’ve debugged a RAG issue that turned out to be “invisible” at first, I’d love to hear what signal finally revealed it.

Self-RAG vs Adaptive RAG vs Corrective RAG

Parth Sarthi Sharma — Thu, 22 Jan 2026 11:24:30 +0000

How Retrieval Systems Are Learning to Fix Themselves

Retrieval-Augmented Generation (RAG) started simple:

Retrieve documents → add them to the prompt → generate an answer.

That worked… until it didn’t.

As RAG systems moved into production, teams began to see the same failures again and again:

Hallucinations despite having “good” data
Irrelevant chunks polluting the prompt
Silent failures that were hard to debug
High token costs with low answer quality

The response wasn’t just better embeddings.

It was smarter control loops.

That’s how Self-RAG, Adaptive RAG, and Corrective RAG emerged.

They all share one idea:

RAG shouldn’t be static.

It should reason about its own failure.

But they solve different layers of the problem.

The Core Problem With Traditional RAG

Classic RAG makes three assumptions:

The user query is well-formed
Retrieved chunks are relevant
More context leads to better answers

In reality:

Queries are vague or underspecified
Vector search returns plausible but wrong chunks
LLMs answer confidently even when context is poor

Traditional RAG has no self-awareness.

Modern RAG patterns add it.

Self-RAG: “Should I Even Answer This?”

What it is

Self-RAG teaches the model to evaluate its own generation using explicit self-reflection.

Instead of blindly answering, the model asks:

Did I actually use the retrieved context?
Is this answer supported by evidence?
Should I revise, regenerate, or refuse?

How it works (conceptually)

Retrieve documents
Generate a draft answer
Run self-critique prompts such as:
- Is this answer grounded in the retrieved text?
- Is there missing or contradictory information?
Regenerate or abstain if confidence is low

What it’s good at

Reducing hallucinations
Citation-aware answers
Knowledge-intensive question answering

Limitations

Still depends on retrieval quality
Adds latency
Reflection quality depends heavily on prompt design

Mental model

Self-RAG adds a judge after generation.

Adaptive RAG: “Do I Even Need Retrieval?”

What it is

Adaptive RAG dynamically changes the pipeline itself based on the query.

Instead of:

Always retrieve → always generate

It asks:

Is retrieval needed at all?
How much context is enough?
Should the query be rewritten?

Typical adaptations

Skip retrieval for simple or well-known facts
Increase retrieval depth for complex queries
Rewrite ambiguous questions
Route between different tools (search, DB, memory)

Why this matters

Many RAG systems are:

Over-fetching
Overstuffing prompts
Burning tokens unnecessarily

Adaptive RAG optimizes for cost and accuracy.

Mental model

Adaptive RAG adds a router before retrieval.

Corrective RAG: “Something Went Wrong — Fix It”

What it is

Corrective RAG focuses on detecting and repairing retrieval failures.

It assumes failure is inevitable and designs for recovery.

Common corrective strategies

Detect low-quality or irrelevant chunks
Drop contradictory context
Trigger re-retrieval with a refined query
Switch retrieval strategies (BM25 ↔ vector search)

Key difference from Self-RAG

Self-RAG critiques the answer
Corrective RAG critiques the context

Why this matters

In production, most RAG failures come from:

Wrong chunks
Missing chunks
Outdated information

Corrective RAG attacks the root cause.

Mental model

Corrective RAG adds a repair loop around retrieval.

Putting It All Together

These approaches are not competing ideas.

They are layers.

A mature RAG system often looks like this:

User Query
↓
Adaptive Router (Do we retrieve? How?)
↓
Retrieval
↓
Corrective Check (Are these chunks good?)
↓
Generation
↓
Self-RAG Evaluation (Is this answer grounded?)
↓
Final Response (or retry / refuse)

Each layer addresses a different failure mode.

Why This Matters in Real Systems

If you’re building:

Enterprise search
Customer support assistants
Internal knowledge bots
Agentic workflows

Static RAG will fail — often quietly.

The future of RAG is not:

Bigger models or longer prompts

It is:

Systems that know when they are wrong.

Final Thought

RAG is evolving from a simple pipeline into a control system.

The teams that succeed won’t be the ones with the largest models —

but the ones with the tightest feedback loops.

If you’re experimenting with Self-RAG, Adaptive RAG, or Corrective RAG in production,

I’d love to hear what worked (or broke) for you.

LangChain vs LangGraph vs Semantic Kernel vs Google AI ADK vs CrewAI

Parth Sarthi Sharma — Tue, 20 Jan 2026 12:57:54 +0000

Choosing the Right LLM Framework Without the Hype

The LLM ecosystem is moving fast. Every few weeks, a new framework promises to “simplify AI agents,” “orchestrate reasoning,” or “make production-ready AI easy.”

But if you’re building real systems, you’ve probably asked:

Why do I need so many frameworks for what feels like the same thing?

I’ve worked with multiple LLM stacks and this article is my attempt to cut through the noise and explain:

What problem each framework actually solves
Where they shine
Where they become liabilities
Which one you should choose depending on your use case

This is not a feature checklist. It’s a mental model.

The Big Picture: What Problem Are We Solving?

All these frameworks exist because LLMs are not applications.
They are components.

Real-world LLM systems need:

Prompt orchestration
Tool calling
Memory
Retrieval (RAG)
Control flow
Observability
Failure handling

Each framework makes different trade-offs around these problems.

LangChain: The Swiss Army Knife (and its curse)

What it is:
LangChain is a high-level abstraction layer for building LLM-powered apps quickly.

What it does well:

Rapid prototyping
Huge ecosystem of integrations
Easy chaining of prompts, tools, retrievers
Strong community momentum

Where it struggles:

Hidden control flow
Debugging is painful at scale
Abstractions leak under complex logic
Performance tuning is hard

When to use LangChain

MVPs
Hackathons
POCs
Teams new to LLMs

When to avoid

Complex, stateful workflows
Systems needing precise control or observability

LangChain is optimized for speed of development, not clarity of execution.

LangGraph: When You Realize LLMs Are State Machines

What it is:
LangGraph is LangChain’s answer to the criticism: “LLM workflows aren’t linear.”

It models AI systems as graphs instead of chains.

What it does well:

Explicit state transitions
Cycles, retries, branching
Long-running agents
Better reasoning visibility

Trade-offs:

More complex mental model
Still tied to LangChain ecosystem
Steeper learning curve

When LangGraph shines

Multi-step agents
Tool-heavy workflows
Systems with retries and loops
Human-in-the-loop scenarios

LangGraph is what you reach for when LangChain starts to feel “magical.”

Semantic Kernel: Engineering-first, AI-second

What it is:
Microsoft’s take on LLM orchestration, designed for software engineers, not prompt hackers.

Key strengths:

Strong typing
Explicit planners
Native support for C# and Python
Enterprise-friendly architecture

Weaknesses:

Smaller ecosystem
Less “plug-and-play”
Slower iteration for experiments

Best fit

Enterprise teams
Strong engineering discipline
Systems that need maintainability over speed

Semantic Kernel feels like it was designed by people who maintain systems at 3am.

Google AI ADK: Opinionated and Cloud-native

What it is:
Google’s Agent Development Kit focuses on structured agent workflows, tightly integrated with Google Cloud and Gemini.

Strengths:

Clear agent lifecycle
Strong observability hooks
Cloud-native design
Production-aligned abstractions

Limitations:

Less flexible outside Google’s ecosystem
Smaller open-source community (for now)
More opinionated architecture

Best fit

Teams already on GCP
Production-first AI systems
Regulated or large-scale environments

ADK assumes you care about deployment and monitoring from day one.

CrewAI: The “Multi-Agent” Narrative

What it is:
CrewAI focuses on orchestrating multiple agents with roles, mimicking human teams.

What it’s good at:

Role-based agent design
Easy mental model
Content generation pipelines

Where it falls short:

Limited control
Less suitable for complex state handling
Not ideal for deeply engineered systems

Use CrewAI if

You’re building collaborative agent demos
Content or research workflows
Experimenting with agent behavior

CrewAI is great for storytelling, not systems engineering.

A Practical Decision Framework

Instead of asking “Which framework is best?”, ask:

1. Do I need speed or control?

Speed → LangChain
Control → Semantic Kernel / LangGraph

2. Is this production-critical?

Yes → Semantic Kernel / Google ADK
No → LangChain / CrewAI

3. Is the workflow stateful and complex?

Yes → LangGraph
No → LangChain

4. Enterprise or startup?

Enterprise → Semantic Kernel / ADK
Startup → LangChain

The Uncomfortable Truth

Most mature AI teams eventually:

Start with LangChain
Outgrow it
Move to custom orchestration or graph-based systems

Frameworks should accelerate learning, not lock you in.

Final Thought

LLM frameworks are evolving because we still don’t fully understand how to engineer AI systems.

Choose tools that:

Make failure visible
Encourage explicit design
Don’t hide complexity forever

Because eventually, complexity always shows up.

If this helped you think more clearly about the LLM ecosystem, feel free to share or comment with your experience. I’d love to learn how others are navigating this space.

Local RAG vs Cloud RAG: What Changes When You Leave the Demo

Parth Sarthi Sharma — Mon, 12 Jan 2026 11:25:38 +0000

Local RAG feels free.
Until your first production incident.

If you’ve built any RAG system recently, chances are it started locally.

A small dataset.
A local vector store.
Fast queries.
Clean answers.

Everything feels under control.

And for a while — it is.

This article is about what quietly changes when RAG systems move from demos to real usage, and why the Local vs Cloud RAG decision is less about tools and more about operational guarantees.

Why Local RAG Feels Like the Right Choice (At First)

Local RAG optimises for exactly what you want early on:

Zero infra friction
Near-zero cost
Tight iteration loops
Full control over data and logic

You can:

Restart the process
Rebuild the index
Tune chunk sizes
Experiment freely

For:

Prototypes
POCs
Internal tools
Early-stage features

Local RAG is not just acceptable — it’s ideal.

So where does it go wrong?

The Problem: Local RAG Doesn’t Fail Loudly

Local RAG rarely explodes.

It degrades.

Slowly.

Subtly.

In ways that are hard to reproduce.

At first:

One user
Sequential queries
Index fits comfortably in memory

Then usage grows:

Concurrent requests increase
Memory pressure rises
Index rebuilds take longer
Latency becomes inconsistent

Nothing is “broken”.

But the system becomes unpredictable.

And unpredictability is the worst failure mode in production.

What Actually Breaks First (And Surprises Teams)

1. Concurrency

Most local vector stores are optimised for:

Single-process access
Limited parallelism

Under load:

Queries queue
Writes block reads
Latency spikes

2. Memory & Resource Contention

Local RAG competes with:

The app runtime
The LLM client
Other background processes

A single spike can:

Trigger OOM
Kill the process
Lose in-memory state

3. Index Lifecycle Management

Rebuilding indexes locally often means:

Blocking reads
Restarting services
Manual intervention

This is fine once. It’s painful at scale.

Why Teams Jump to Cloud RAG Too Early

On the flip side, many teams move to cloud RAG before they need to.

Common reasons:

Fear of future scale
“Production readiness” anxiety
Over-indexing on best practices

The result:

Paying for capacity you don’t use
Higher baseline latency
Vendor lock-in decisions too early

Cloud RAG is not “better RAG”. It’s RAG with guarantees.

And guarantees come with cost.

What Cloud RAG Actually Buys You

Cloud-managed RAG systems exist to solve operational problems, not retrieval quality.

They give you:

Concurrency handling
Persistence and durability
Observability hooks
Backups and recovery
Predictable performance envelopes

What they don’t magically fix:

Poor chunking
Bad retrieval logic
Overstuffed prompts
Weak context engineering

If ingestion is broken locally, it will be broken in the cloud — just more expensively.

The Real Decision Axis (This Is the Key)

The Local vs Cloud RAG decision is not about:

Chroma vs Pinecone
FAISS vs Weaviate

It’s about answering four questions honestly:

How many concurrent users do I expect?
How painful is downtime or degraded answers?
Do I need observability and auditability?
How often will my index change?

Local RAG optimises for:

Speed
Control
Learning

Cloud RAG optimises for:

Reliability
Predictability
Scale

Neither is “correct” in isolation.

A Practical Migration Pattern That Works

Mature teams rarely jump straight from local to fully managed cloud RAG.

Instead, they:

Start local
Learn their retrieval patterns
Stabilise chunking and routing
Introduce cloud RAG only when operational pain appears

This keeps:

Cost low early
Architecture flexible
Decisions reversible

Final Takeaway

Local RAG fails quietly.
Cloud RAG fails expensively.

The right choice depends on when you’re willing to pay:

With engineering effort
Or with infrastructure cost

The worst choice is deciding too early — in either direction.

What’s Next

In the next article, we’ll dive into one of the most under-discussed problems in RAG systems:

Observability in RAG Pipelines: Knowing Which Chunk Failed (and Why)

We’ll explore:

Why “LLM hallucinated” is usually a monitoring failure
What should be traced in a RAG request (retrieval, ranking, prompt, tokens)
How to identify:

Wrong chunk retrieval
Empty or partial context
Latency bottlenecks
Silent failures in agents and tools

How tools like OpenTelemetry, LangSmith, and custom tracing fit together

Because you can’t fix what you can’t see — and most RAG systems today are completely blind.

Prompt Routing & Context Engineering: Letting the System Decide What It Needs

Parth Sarthi Sharma — Fri, 09 Jan 2026 21:59:50 +0000

Most LLM systems fail not because the model is weak
but because we shove everything into the prompt and hope for magic.

If you’ve ever built a RAG or agentic system, you’ve probably tried this at least once:

Retrieve more documents
Increase chunk count
Add system instructions
Extend the prompt
Increase context window

And yet… the answer still feels off.

That’s because context is not information. Context is relevance + timing + placement.

This article is about how mature LLM systems stop stuffing prompts
and start deciding what context they actually need.

The Core Problem: Static Prompts in a Dynamic World

Most early-stage LLM systems look like this:

User Query
  → Retrieve top K chunks
  → Stuff everything into a single prompt
  → Generate response

This works… until it doesn’t.

Why?

Because:

Not all questions need the same context
Not all tasks need the same instructions
Not all users need the same depth

Yet we treat every request identically. That’s where prompt routing enters.

What Is Prompt Routing (Really)?

Prompt routing is decision-making before generation.

Instead of asking:

“How do I write the perfect prompt?”

You ask:

“Which prompt, context, and tools does this request require?”

Think of it as a traffic controller for LLM calls.

A routing layer decides:

Which system prompt to use
Which context sources to include
Whether retrieval is even required
Whether the model should reason, summarise, or act

A Mental Model: LLMs Don’t Need More Context — They Need the Right Context

Consider these two queries:

“Summarise the payment terms in this contract”
“Can we safely terminate this contract early and what are the risks?”

Same document. Very different needs.

Query 1 needs:

A small, focused chunk
No reasoning
No tools

Query 2 needs:

Multiple clauses
Cross-referencing
Risk interpretation
Possibly external policy context

If both go through the same prompt pipeline, one of them will fail.

Prompt Routing in Practice (Without Buzzwords)

A practical routing layer usually classifies queries into intent buckets, such as:

❓ Factual lookup
📄 Summarisation
🧠 Reasoning / decision-making
🛠 Tool execution
🔁 Multi-step workflows

This classification can be:

Rule-based (early stage)
LLM-based (later stage)
Hybrid (best in production)

Once intent is known, everything else follows.

Context Engineering: The Part Most People Miss

Prompt routing decides what path to take.
Context engineering decides what to inject and where.

Bad context engineering looks like:

Dumping raw chunks
No ordering
No metadata
No separation between instructions and data

Good context engineering is deliberate.

Proven patterns that actually work:
1. Instruction / Data Separation

Never mix:

System rules
Retrieved content
User instructions

LLMs treat early tokens as authority.

2. Query-Aware Retrieval

Retrieve based on intent, not keywords.

A “why” question should retrieve:

Explanations
Rationale
Trade-offs

A “what” question should retrieve:

Definitions
Tables
Direct facts

3. Context Placement Matters

Important facts belong:

At the start (primacy bias)
Or at the end (recency bias)

Middle content is often ignored (hello, Lost in the Middle).

Why This Is the Bridge Between RAG and Agentic Systems

Prompt routing is the missing layer between:

Simple RAG
Agentic RAG

Without routing:

Agents overthink
Simple RAG underperforms

With routing:

Simple RAG stays simple
Agents are invoked only when needed

This is how mature systems stay:

Faster
Cheaper
Easier to debug

A Simple Rule of Thumb

If retrieval answers the question → don’t use an agent
If decisions must be made → route to reasoning
If actions are needed → allow tools
If uncertainty exists → slow the system down

That’s not prompt engineering.

That’s system design.

What’s Next

In the next article, we’ll explore:

Local RAG vs Cloud RAG: What Changes When You Leave the Demo

We’ll look at:

Why local RAG feels perfect during development
Where it quietly breaks under concurrency and scale
What cloud RAG actually buys you (and what it doesn’t)
How routing and context strategies behave differently in local vs managed setups

Because once your system can decide what context it needs,
the next challenge is making sure that decision is reliable, observable, and repeatable in production.

Simple RAG vs Agentic RAG: What Problem Are You Actually Solving?

Parth Sarthi Sharma — Thu, 08 Jan 2026 12:50:08 +0000

Let’s start with a real problem.

“Can I terminate this contract early, and what penalties apply?”

You have:

A set of contracts (PDFs)
A user asking a natural-language question
An LLM-powered application

The question is not:

“Should I use RAG or agents?”

The real question is:

How much reasoning does this problem actually require?

Step 1: The Simple RAG Approach (And Why It Often Works)

What Simple RAG Looks Like

A typical Simple RAG pipeline:

User asks a question
Embed the query
Retrieve top-K chunks
Inject them into the prompt
Generate an answer

In code terms (conceptually):

query → retriever → context → prompt → LLM → answer

What Happens in Practice

For many questions, this works surprisingly well:

“What is the notice period?”
“When does the contract expire?”
“Is early termination allowed?”

Why?
Because the answer exists verbatim in the documents.

No planning.
No tool chaining.
No decision-making.

Step 2: Where Simple RAG Starts to Break

Now try this question:

“If I terminate early due to breach, does the penalty still apply?”

Suddenly:

The answer spans multiple clauses
Conditions matter
Exceptions override defaults

What Simple RAG does:

Retrieves multiple chunks
Dumps them into context
Hopes the LLM figures it out

Sometimes it does.
Sometimes it hallucinates confidently.

The failure mode isn’t retrieval — it’s implicit reasoning.

Step 3: Enter Agentic RAG (And Why People Overuse It)

Agentic RAG introduces explicit reasoning steps.

Instead of:

“Answer directly”

The system does:

Identify sub-questions
Decide which tools to call
Retrieve information iteratively
Synthesize an answer

Conceptually:

plan → retrieve → evaluate → retrieve → decide → answer

This shines when:

Questions are multi-hop
Dependencies exist
Decisions affect next steps

For example:

“Check termination clause”
“Check breach exceptions”
“Check penalty override”
“Combine results”

This is real reasoning, not just recall.

Step 4: Where Agentic RAG Becomes a Liability

Now consider this question:

“What is the termination notice period?”

An agent might:

Plan unnecessarily
Call tools repeatedly
Increase latency
Increase cost
Introduce new failure modes

You traded:

A 1-step pipeline for
A 5-step reasoning loop To answer a lookup question.

This is overengineering.

The Core Insight Most Teams Miss

Agentic RAG is not “better RAG.”
It’s a different tool for a different problem.

The decision is not:

Simple vs Agentic
It’s:
Recall vs Reasoning

A Practical Decision Rule (Use This)

Use Simple RAG when:

The answer exists verbatim
Questions are independent
Latency and cost matter
Determinism is important

Use Agentic RAG when:

Answers span multiple sources
Decisions affect next retrieval
You need traceable reasoning
You accept higher cost for correctness

Why Many Systems Fail in Production

Most teams:

Jump to Agentic RAG too early
Before fixing ingestion
Before fixing chunking
Before understanding attention limits

Agents amplify:

Bad context
Poor retrieval
Weak observability

They don’t fix fundamentals.

Final Takeaway

Simple RAG fails when reasoning is required.
Agentic RAG fails when reasoning is unnecessary.

The best systems:

Route questions intentionally
Use agents selectively
Treat reasoning as a cost, not a default

What’s Next

Next, we’ll go one level deeper:

Prompt Routing & Context Engineering: Letting the System Decide What It Needs

That’s where real production intelligence starts.

Chunking, Batching & Indexing: The Hidden Costs of RAG Systems

Parth Sarthi Sharma — Tue, 06 Jan 2026 21:40:49 +0000

Most RAG discussions focus on retrieval quality.

Which embeddings to use.
Which vector database is faster.
Which similarity metric performs better.

But in production, RAG systems rarely fail because of retrieval alone.

They fail because of how content is chunked, batched, and indexed — quietly, expensively, and at scale.

Why Chunking Is a Cost Decision, Not Just a Text Decision

Chunking is often treated as a preprocessing step:

“Split documents into 500-token chunks and move on.”

That decision impacts:

Retrieval accuracy
Context window usage
Latency
Token cost
Index size
Re-ranking complexity

Bad chunking doesn’t just reduce answer quality — it multiplies operational cost.

The Real Trade-Offs in Chunk Size

Small Chunks
Pros

More precise retrieval
Better semantic focus
Lower “Lost in the Middle” risk

Cons

More chunks per document
Larger vector index
Higher retrieval fan-out
More context assembly overhead

Large Chunks

Pros

Fewer vectors
Smaller index
Faster ingestion

Cons

Lower relevance density
More noise per chunk
Higher chance of ignored context
Worse attention utilisation

👉 There is no “perfect” chunk size.
There is only context-aware chunking.

Why Batching Quietly Becomes Your Biggest Cost Lever

Most teams underestimate batching.

Batching affects:

Ingestion throughput
Embedding API cost
Failure recovery
Observability
Reprocessing overhead

Common Anti-Pattern

Ingesting documents one by one
Embedding synchronously
No retry or visibility

This works for demos. It collapses at scale.

What Good Batching Looks Like

Production-grade ingestion pipelines:

Batch documents intentionally
Track batch IDs
Log failures per batch
Allow partial retries
Emit metrics per stage

Batching isn’t just optimisation — it’s operational control.

Indexing: The Forgotten Scaling Problem

Indexing is often treated as “fire and forget”.

But indexing decisions affect:

Query latency
Memory footprint
Rebuild cost
Migration complexity

Questions teams forget to ask:

Can we re-index incrementally?
Can we support multiple indexes per domain?
Can we rebuild without downtime?
Can we version indexes safely?

RAG systems age badly without good indexing strategy.

Why These Costs Compound in Production

Here’s the uncomfortable truth:

Every extra chunk
→ increases retrieval cost
→ increases prompt size
→ increases token spend
→ increases latency
→ reduces answer quality

Poor chunking and batching don’t fail loudly.
They fail financially.

Practical Guidelines (That Actually Work)

Some battle-tested principles:

Prefer smaller, semantically complete chunks
Avoid “just in case” retrieval
Batch ingestion with observability
Track cost per document, not per query
Treat indexes as versioned assets
Re-evaluate chunking as usage evolves

RAG is not a static pipeline — it’s a living system.

Final Takeaway

RAG systems don’t get expensive overnight.

They get expensive through:

Over-chunking
Over-retrieval
Under-observability
Poor batching discipline

If you don’t design ingestion for scale, your costs will scale for you.

What’s Next

In the next article, we’ll step back and ask:

Simple RAG vs Agentic RAG: What Problem Are You Actually Solving?

Because adding agents before fixing ingestion is usually a mistake.

Discussion

How are you currently handling chunking and batching in your RAG pipelines?
What trade-offs have surprised you the most?

Why “Lost in the Middle” Breaks Most RAG Systems

Parth Sarthi Sharma — Sun, 04 Jan 2026 12:37:17 +0000

If you’ve built a RAG system, you’ve probably seen this:

The retriever finds the right document
The chunk clearly contains the answer
Yet the LLM responds as if it never saw it

This isn’t a vector problem.
It’s not an embedding issue.
And it’s usually not your prompt.

It’s a context window problem — commonly called “Lost in the Middle.”

What “Lost in the Middle” Actually Means

Large Language Models do not treat all tokens equally.

When processing long prompts, models tend to:

Pay more attention to tokens at the beginning
Pay more attention to tokens at the end
Pay less attention to tokens in the middle

This behaviour emerges from how transformer attention works at scale — especially when prompts approach the context window limit.

So even if the correct chunk is retrieved, placing it in the middle of a long prompt makes it statistically easier for the model to ignore.

Why This Hits RAG Systems Especially Hard

RAG pipelines usually look like this:

User asks a question
Retriever fetches top-K chunks
Chunks are concatenated into context
Prompt is sent to the LLM

The problem?

Most systems:

Append retrieved chunks after instructions
Stack chunks in relevance order
Push critical information into the middle of the prompt

Typical RAG prompt layout:

[System Instructions]
[User Question]
[Retrieved Chunk 1]
[Retrieved Chunk 2]
[Retrieved Chunk 3]
[Retrieved Chunk 4]
[Answer Instruction]

Where do most chunks land?

👉 Right in the middle

Result:

Relevance ≠ Visibility

Even though:

They are semantically correct
They are retrieved via similarity

The retriever did its job — but the model never fully used the information.

Why “Better Embeddings” Don’t Fix This

This is the trap many teams fall into:

Switching from OpenAI → Cohere → BGE
Tweaking vector dimensions
Changing similarity metrics

But embeddings only decide what gets retrieved.
They don’t control what gets attended to.

You can have perfect embeddings and still get poor answers if:

Context is too long
Chunks are poorly ordered
Important facts sit in the middle

How “Lost in the Middle” Shows Up in Production

Common symptoms:

Model answers partially correct
Hallucinations despite relevant context
Correct answers during testing, failures at scale
“It works for short queries, not long ones”

These are not random failures — they’re structural.

Practical Ways to Mitigate It

You don’t eliminate “Lost in the Middle” — you design around it.

Effective strategies include:

Putting critical chunks at the beginning or end
Query-aware chunk re-ordering
Context compression / summarisation
Smaller, intent-focused context windows
Multi-step prompting instead of one giant prompt

The goal isn’t more context — it’s better-positioned context

Final Takeaway

RAG doesn’t fail because retrieval is wrong.
It fails because attention is finite.

If you don’t design for how models actually consume context, they’ll ignore the very information you worked hard to retrieve.

What’s Next

In the next article, we’ll go deeper into:

Chunking, Batching & Indexing — the Hidden Costs of RAG Systems

Because once attention is understood, scale, latency, and cost become the real problems.

Loaders, Splitters & Embeddings — How Bad Chunking Breaks Even Perfect RAG Systems

Parth Sarthi Sharma — Sat, 03 Jan 2026 12:34:11 +0000

When people debug poor RAG results, they usually blame:

the vector database
the embedding model
the prompt

But in real systems, the most common root cause sits much earlier:

👉 The document ingestion pipeline

If ingestion is wrong, retrieval will be wrong — no matter how good your embeddings or LLM are.

In this article, we’ll break down:

What a document ingestion pipeline actually is
The role of loaders, splitters, and embeddings
Why chunking is the most underestimated design decision
How ingestion mistakes silently ruin RAG systems

This is concept-first thinking, with tooling examples only where helpful.

1. What Is a Document Ingestion Pipeline?

At a high level, an ingestion pipeline converts raw data into retrievable semantic units.

Raw Source
  → Documents
    → Chunks
      → Embeddings
        → Vector Store

Each stage:

Loses information if done poorly
Constrains everything downstream
Is extremely hard to “fix later”

A bad ingestion pipeline doesn’t fail loudly — it fails by returning plausible but wrong answers.

2. Document Loaders: The Foundation (Often Ignored)

What loaders do

Document loaders are responsible for:

Reading raw sources (PDFs, HTML, Markdown, APIs, DBs)
Extracting text
Attaching metadata

Examples of sources:

PDFs (policies, contracts)
Websites / wikis
Git repositories
Knowledge bases (Confluence, Notion)
Databases or APIs

Common loader failures

PDFs with broken text order
Headers/footers mixed into content
Navigation menus treated as text
Missing or inconsistent metadata

If metadata is lost at this stage, you cannot recover it later.

Design rule #1

Treat metadata as first-class data.

At minimum, preserve:

source (URL / file / system)
page or section
document type
timestamp
access scope

Frameworks (e.g. LangChain loaders) help — but you must inspect loader output manually, at least once.

3. Text Splitters: Where Most RAG Systems Break

LLMs do not retrieve documents.
They retrieve chunks.

This makes text splitting one of the most important — and misunderstood — steps in RAG.

Why splitting matters

Bad splitting causes:

Partial facts
Broken reasoning
“Lost in the middle” effects
Irrelevant or misleading retrievals

Once text is chunked incorrectly, embeddings faithfully encode the wrong thing.

4. Chunk Size Is a Trade-Off, Not a Constant

There is no universally “correct” chunk size.

Small chunks

Pros

Higher recall
Precise matching

Cons

Loss of local context
Fragmented meaning

Large chunks

Pros

Better semantic completeness
More context per chunk

Cons

Fewer chunks fit in the context window
Irrelevant content dilutes relevance

The right chunk size depends on document structure and query intent — not a blog default.

5. Why Bad Chunking Ruins Even Perfect Embeddings

This is the key misconception:

“If embeddings are good, retrieval will be good.”

Not true.

Embeddings encode what you give them.

If a chunk:

mixes multiple topics
cuts sentences mid-thought
spans unrelated sections

Then the embedding becomes a semantic average — and retrieval quality collapses.

This is why:

identical embedding models can perform wildly differently
RAG quality varies more by ingestion than by model choice

6. Overlap: A Necessary Evil

Chunk overlap exists to:

preserve continuity
avoid cutting critical information

But overlap has costs:

More chunks
Higher storage cost
Higher retrieval noise

Overlap should be:

intentional
minimal
justified by document structure

Blindly adding overlap is not a fix — it’s a tax.

7. Embeddings Come Last (For a Reason)

Embeddings are often treated as the “magic step”.

In reality:

They are deterministic
They faithfully reflect upstream decisions
They cannot repair bad ingestion

By the time text reaches the embedding stage:

Most architectural decisions are already locked in

This is why changing the embedding model rarely fixes poor RAG results.

8. Ingestion Is an Architectural Decision

In production systems:

Ingestion pipelines evolve over time
Different document types need different strategies
Re-ingestion is expensive and risky

That makes ingestion:

a platform concern
not a one-off script
not a junior task

If you design ingestion casually, you pay for it forever.

9. Key Takeaways

RAG failures usually start at ingestion
Loaders must preserve clean text and metadata
Chunking decisions dominate retrieval quality
Embeddings encode mistakes faithfully
Ingestion pipelines are architecture, not plumbing

What’s Next

In the next article, we’ll explore:

Why “Lost in the Middle” Breaks Most RAG Systems
— and why retrieving the right chunks doesn’t guarantee the model will use them.