DEV Community: jacobjerryarackal

I Built a RAG Pipeline. Then I Realized Retrieval Is the Real Model

jacobjerryarackal — Wed, 08 Apr 2026 03:03:07 +0000

Everyone talks about the LLM. GPT‑4, Claude, Gemini – that’s the celebrity. But after building my first real RAG pipeline, I learned something humbling: the LLM is the interchangeable part. The retrieval system is the actual worker.

Let me show you what I mean.

The 4‑Step Pipeline We All Copy

You’ve seen the tutorial code a hundred times:

Ingest – chunk your documents
Embed – turn chunks into vectors
Retrieve – find top‑k similar chunks
Generate – LLM answers with that context

It works. My bot could answer company policy questions with citations. I felt smart.

Then I asked: “Can I get a refund for a digital product?”

The LLM gave a beautiful, confident answer which was completely wrong. Because my retrieval returned a chunk about physical returns (30 days, original packaging) and completely missed the digital product exception sitting two paragraphs away.

The LLM did its job perfectly. The retrieval failed.

Why Retrieval Is the Real Model

Here’s what I learned the hard way:

What you think matters	What actually matters
Which LLM you use	How you chunk documents
Prompt engineering	Embedding quality
System prompts	Re‑ranking after retrieval

The LLM just formats the answer. Retrieval decides whether the answer is true.

The Code That Fixed My Pipeline

Semantic search alone misses exact phrases like “non‑refundable after download”. Keyword search alone misses meaning. Hybrid search combines both. Here’s the core (using FAISS + BM25):

from sentence_transformers import SentenceTransformer
import faiss, numpy as np
from rank_bm25 import BM25Okapi

# 1. Load documents and embed
docs = ["Refund within 30 days, physical items only.",
        "Digital products: non-refundable after download.",
        "Contact support for defective digital items."]
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(docs)
index = faiss.IndexFlatL2(embeddings.shape[1])
index.add(np.array(embeddings, dtype='float32'))

# 2. BM25 keyword index (tokenized)
tokenized_docs = [doc.lower().split() for doc in docs]
bm25 = BM25Okapi(tokenized_docs)

# 3. Hybrid search function
def hybrid_search(query, top_k=2, alpha=0.5):
    # Semantic score (distance -> similarity)
    query_vec = model.encode([query])
    distances, indices = index.search(query_vec, top_k)
    semantic_scores = 1 / (1 + distances[0])  

    # Keyword score
    query_tokens = query.lower().split()
    bm25_scores = bm25.get_scores(query_tokens)
    top_bm25_idx = np.argsort(bm25_scores)[-top_k:][::-1]
    keyword_scores = [bm25_scores[i] for i in top_bm25_idx]

    # Combine (normalized)
    combined = {}
    for i, idx in enumerate(indices[0]):
        combined[idx] = alpha * semantic_scores[i]
    for i, idx in enumerate(top_bm25_idx):
        combined[idx] = combined.get(idx, 0) + (1-alpha) * (keyword_scores[i] / max(keyword_scores))

    return sorted(combined.items(), key=lambda x: x[1], reverse=True)[:top_k]

# 4. Test
query = "Can I get my money back for a digital product?"
results = hybrid_search(query)
for idx, score in results:
    print(f"Score: {score:.2f} | {docs[idx]}")
# Output: Score: 0.92 | Digital products: non-refundable after download.

That alpha=0.5 balances meaning and exact wording. Without hybrid search, the digital product chunk was #3 (ignored). With it, #1.

Three Changes That 10x’ed My Pipeline

Chunk size is not a default – Moved to overlapping chunks (200 tokens with 50 overlap).
Semantic search alone lies – Added BM25 hybrid search (see code above).
Re‑ranking changes everything – A small cross‑encoder re‑scored top‑10 chunks, lifting accuracy from 72% to 91%.

The Mistake Most People Make

We treat RAG as an LLM problem. So we tweak prompts, swap models, add system instructions.

But the LLM is forced to use whatever context you give it. If you feed it the wrong chunk, it will hallucinate confidently. If you feed it the right chunk, even a small model answers correctly.

The bottleneck is almost never the LLM. It’s the retriever.

What I Do Differently Now

Before I write a single line of agent code, I ask three questions:

“If I searched my vector database by hand, would I find the exact sentence that answers this?”
“Does my retrieval work for synonyms AND exact keywords?” → if no, hybrid search.
“Is the top‑1 retrieved chunk actually the best?” → if no, add a re‑ranker.

The Bottom Line

The AI industry sells you on the model. But in production RAG systems, the model is the cheapest, most replaceable component. The hard part – the part that separates working bots from demoware – is getting the right information into the context window.

The LLM is the pen. Retrieval is the memory. And memory is what makes a system useful.

So next time your RAG bot fails, don’t blame GPT. Look at what you retrieved. I promise that’s where the real problem lives.

We Let an LLM Control a File System and Run Commands – Here’s What Actually Broke First

jacobjerryarackal — Sat, 04 Apr 2026 08:02:02 +0000

I wanted to push an LLM beyond simple chat and see if it could actually build real code.

So I gave it direct access to the file system and the ability to run terminal commands. The task was straightforward: “Create a clean React login page with email, password, remember-me checkbox, and form validation.”

It started confidently. Within minutes everything broke.
The System We Built
We connected two tools to the LLM:

file_system (list, read, write, delete files)
run_command (execute npm, start dev server, etc.)

We used MCP (the “USB-C for AI” protocol) so the model could call tools cleanly. The goal was to let the LLM act like a real developer — explore the folder, create files, install packages, and test the app.
It sounded simple. It was not.

Failure #1: It Assumed the Project Already Existed
What broke:
The model immediately started writing Login.jsx in an empty folder. No package.json, no React setup, no dependencies.
Why it broke:
The LLM had no understanding of project bootstrapping. It assumed a full React app was already there.
What we learned:
We had to explicitly tell it “first create the project structure” in every new session. This became our first mandatory step.

Failure #2: It Ran Commands at the Wrong Time
What broke:
After creating a few files, it ran npm start and npm run build before any dependencies were installed. The terminal exploded with 47 errors.
Why it broke:
The model treated commands like a checklist instead of understanding dependencies. It didn’t realise you can’t run the app before npm install.
What we learned:
We added a rule: never run npm start or npm run build until package.json exists and all dependencies are installed. This single rule saved us from multiple crashes.

Failure #3: It Mixed Concerns and Created Messy Code
What broke:
It put all the Tailwind CSS and form logic inside a single Login.jsx file. The component became 180 lines long, impossible to read, and had styling mixed with business logic.
Why it broke:
The model was optimising for “one file = done” instead of proper component structure.
What we learned:
We had to force it to create separate files (Login.jsx, Login.css, utils/validation.js). Once we added this constraint, the code quality jumped dramatically.

Failure #4: It Had No Memory of Previous Mistakes
What broke:
Even after we fixed the directory issue, in the next loop it tried to create the same wrong file again in the wrong location.
Why it broke:
The model had no persistent memory of what it had already tried and failed.
What we learned:
We started saving a small agent-log.md file after every loop so the model could read its own history before making the next decision. This simple trick reduced repeated mistakes by almost 70%.
After 8 loops and 14 minutes, we finally had a clean, working React login page with proper validation and structure.

The Real Lesson
The LLM wasn’t the problem. The problem was that we treated it like a magician instead of a junior developer with superpowers.
Once we gave it real tools (file system + terminal) and forced it to work inside real constraints, it went from completely broken to actually useful.

In 2026, the biggest unlock isn’t a smarter model.
It’s giving the model the right tools and the right guardrails.
I no longer ask LLMs to “write me some code.”
I give them a file system, terminal access, and clear rules.
That single change is what turns toys into tools you can actually ship.

“Prompt Engineering Is Enough” Is Wrong – Here’s What I Had to Add

jacobjerryarackal — Thu, 02 Apr 2026 12:10:36 +0000

I used to believe the hype.

I thought if I just wrote better prompts clearer instructions, few‑shot examples, chain‑of‑thought so that I could make any LLM do whatever I wanted. I spent weeks refining prompts, tweaking wording, adding “think step by step” like it was magic.

Then I tried to build something useful.

I asked the model to check the weather and tell me if I needed an umbrella. The response was confident and completely wrong. It hallucinated the forecast based on its training cut‑off. No real data, just made‑up facts wrapped in perfect English.

The prompt was excellent. The model was powerful. The result was useless.

That’s when I realised the uncomfortable truth: prompt engineering is not enough.

I went back and added one thing that actually fixed it tools.

I gave the model a simple weather tool and forced it to use a basic OBSERVE → THINK → ACT loop. Nothing fancy. Just three steps every single time.

Here’s what happened.

First loop – OBSERVE

The model receives my request: “Check the weather in Kochi and tell me if I need an umbrella today.” It sees it has access to a tool called get_weather(location).

Second loop – THINK

Instead of guessing, it reasons out loud: “I don’t have current weather data. I should use the tool to get real information.”

Third loop – ACT

It calls the tool in clean JSON:

{
  "tool": "get_weather",
  "arguments": {
    "location": "Kochi, Kerala"
  }
}

The tool returns actual data. The model goes back to THINK mode, sees the result (“light rain expected”), and only then generates the final answer: “Yes, take an umbrella.”

No hallucination. No made‑up weather. Just one tool + one loop.

I tested it on something harder, building a small React login page from scratch. With pure prompting, the model produced broken, outdated code and confidently told me it was correct. After adding file_system and run_command tools plus the same OBSERVE‑THINK‑ACT loop, it actually listed the directory, read the existing package.json, wrote proper components, fixed its own bugs, and shipped working code across eight loops.

The model was the same. The prompts were similar. The only difference was that I stopped treating the LLM as a magic oracle and started treating it as a brain that needs hands.

I also added MCP, the simple protocol everyone now calls “USB‑C for AI.” It made plugging in new tools ridiculously easy. No custom glue code. Just declare the tool once, and the agent knows exactly how to call it.

The change was night and day.

I stopped wasting time writing longer and longer prompts. I started adding tools and a reliable loop instead. The results went from “impressively wrong” to “actually useful.”

The lesson for 2026 is brutally simple: prompt engineering is table stakes, not the complete solution. If your LLM keeps hallucinating, forgetting tasks, or failing at real work, stop tweaking the prompt. Give it proper tools and a structured loop to use them.

The model is the brain. Tools are the hands. Without hands, even the smartest brain is stuck guessing.

I no longer believe “prompt engineering is enough.”

I now know exactly what I have to add.