DEV Community: Vishnu K

Your RAG Pipeline Is Bleeding Tokens. We Cut 86% Without Losing Accuracy.

Vishnu K — Wed, 03 Jun 2026 10:29:35 +0000

We were sending ~10,000 tokens per query to the LLM. We got it down to ~1,500 — and got more accurate doing it. Here's what we changed, and why flat vector search is the wrong tool for relational data.

We mapped a 158M-token synthetic CRM into TigerGraph, ran 3 pipelines head-to-head on 90 questions, and GraphRAG answered 87/90 (96.7%) while a well-resourced BasicRAG managed 64/90 (71.1%) — at 86% fewer tokens and 17.5% lower latency.

The problem with flat vector search on CRM data

Ask "How many customers were impacted by OUTAGE-001 through their shared vendor and region?" — there's no single chunk that answers it. The answer spans multiple hops:

Outage → Region → Vendor → Customers

BasicRAG embeds the question and returns chunks ranked by surface similarity. It has no concept of "customers sharing this vendor and this region." That's structural, not a tuning gap — flat search treats relational data like a document store. It isn't one.

The honest test: we gave BasicRAG the relevant documents for every question. It still capped at 71.1% — the failures aren't about coverage, they're about reasoning across relationships.

What we built

Three pipelines on a TigerGraph knowledge graph:

Pipeline	Retrieval	Avg Tokens	Accuracy
LLM-Only	None	~14	3.3%
BasicRAG	Cosine similarity	~10,867	71.1%
GraphRAG	HNSW seed + GSQL traversal	~1,483	96.7%

Same LLM (Gemini 2.5 Flash). Same 90 questions. Same data. Only retrieval changes.

The graph

158M tokens of CRM-native data across 100,820 documents — customers, vendors, outages, regions, employees, tickets, compliance cases, projects, all interlinked. Re-chunked and embedded into 577,175 vector chunks with TigerGraph's native HNSW index — 1.58× the hackathon's 100M-token minimum (measured via Gemini count_tokens).

How retrieval works

GraphRAG embeds the question, finds seed chunks via TigerGraph's native HNSW vector search, traverses typed relationships to pull only the connected context, then reranks to the most relevant. The LLM gets a focused ~1,483-token brief, not a 10K-token wall.

The evaluation (we were paranoid about grading our own homework)

Generator: Gemini 2.5 Flash — every answer, same model for all 3 pipelines
Judge: Groq Llama 3.1 8B — independent, scores PASS/FAIL blind
Semantic eval: canonical HuggingFace bert_score (roberta-large, rescale_with_baseline=True) → F1 raw 0.932 / rescaled 0.599 — clears both bonus bars (≥0.88 raw, ≥0.55 rescaled)

The numbers that mattered

96.7% vs 71.1% — a 25-point gap on identical data + LLM, purely from retrieval method
86.4% fewer tokens (1,483 vs 10,867) — the cost story; at scale, the difference between a product and a runaway bill
17.5% faster (7.5s vs 9.1s) — traversal is a targeted lookup; BasicRAG pays to embed and rank a large context every query

The honest version of the TigerGraph setup

Community Edition via Docker. Two real learning curves: infrastructure — an unclean shutdown mid-embedding corrupted the graph store once, so we learned to snapshot gstore immediately after embedding and before evaluation; and GSQL — multi-hop queries and accumulators took time to click, but then ran in milliseconds locally.

What we'd do differently

Tune the hybrid HNSW + graph pipeline (hop depth per query type)
200 eval questions instead of 90 to tighten confidence intervals
TigerGraph Savanna (cloud) over local Docker to skip infra debugging

Stack

Layer	Technology
Graph DB	TigerGraph Community Edition (Docker)
Queries	GSQL multi-hop + native HNSW + REST++
Embeddings	Google `gemini-embedding-001` (768-dim)
LLM (all pipelines)	Gemini 2.5 Flash
Judge	Groq Llama 3.1 8B
Semantic eval	HuggingFace `bert_score` (roberta-large)
API	Fastify + Node 20 + TypeScript
Dashboard	crm-nexus-team-brocode.vercel.app

GitHub → github.com/vishnu-k-dev/crm-nexus

Built for the TigerGraph GraphRAG Inference Hackathon 2026 — Team BroCode (Vishnu K & Revanth M)

TigerGraph #GraphRAG #GraphDatabase #LLM #Hackathon #RAG

Your RAG Pipeline Is Bleeding Money. Here's How We Fixed It.

Vishnu K — Thu, 14 May 2026 18:54:12 +0000

We were paying for 2,124 tokens per query. We got it down to 584. Here's what we changed — and why BasicRAG is the wrong tool for relational data.

We mapped a 2.69M-token synthetic CRM dataset into TigerGraph, ran 3 pipelines head-to-head on 36 questions, and GraphRAG answered 35/36 at 97.2% accuracy while BasicRAG managed 14/36. Here's how we built it and what we learned.

The Problem with Flat Vector Search on CRM Data

When someone asks "What is Pinnacle Enterprises' renewal risk?" — there's no single chunk of text that answers that. The answer lives across three graph hops:

Customer → Deals → Deal Owner → Territory

BasicRAG embeds the question, runs cosine similarity against 2,124 tokens of raw text, and returns chunks that don't mention "Pinnacle" at all. It answered 14 out of 36 questions correctly.

This is the core insight we built CRM Nexus around: CRM data is inherently relational. Flat vector search treats it like a document store. It isn't one.

What We Built

CRM Nexus is a three-pipeline inference system sitting on top of a TigerGraph knowledge graph:

Pipeline	Retrieval	Avg Tokens	Accuracy
LLM-Only	None	~50	8.3%
BasicRAG	Cosine similarity	~2,124	38.9%
GraphRAG	GSQL 3-hop traversal	~584	97.2%

Same LLM (Llama 3.3 70B via Groq). Same 36 questions. Only retrieval changes.

The Graph

21,318 vertices. 48,201 edges. 2.69M tokens of synthetic CRM data:

7,500 Deals (stage, value, owner, close date)
6,000 Customers (health score, ARR, NPS, renewal date)
4,318 Employees (role, department, skills)
5 Products with competitors and roadmap
5 Departments with Q4 goals

Every record is CRM-native — no Wikipedia articles, no generic text. Designed to expose flat vector search failures on relational queries.

The GSQL Query That Powers It

CREATE QUERY getRelevantContext(STRING entity_name) FOR GRAPH CRM {
  Start = {Customer.*};

  Matched = SELECT c FROM Start:c
            WHERE c.name LIKE "%" + entity_name + "%";

  Deals = SELECT d FROM Matched:c -(OWNS)-> Deal:d;

  Owners = SELECT e FROM Deals:d -(MANAGED_BY)-> Employee:e;

  PRINT Matched, Deals, Owners;
}

3 hops. 584 tokens returned. The LLM gets exactly what it needs — not a wall of text.

The TigerGraph Setup (Honest Version)

We ran TigerGraph Community Edition via Docker. First attempt — REST++ calls timed out because port 9000 wasn't exposed correctly in our docker-compose config. Spent a few hours on that before it was obvious it was a port mapping issue, not TigerGraph itself.

The bigger learning curve was GSQL. Different enough from SQL that our first multi-hop queries threw compile errors we didn't understand. Once we grasped how accumulators work — thread-safe variables that aggregate across parallel traversals — everything clicked. The 3-hop query took about a day to write correctly, but once it ran it was consistently under 200ms on a 48K-edge local graph.

The Evaluation Setup

We were paranoid about grading our own homework. So:

Generator: Llama 3.3 70B (Groq) — produces the answer
Judge: Llama 4 Scout 17B (meta-llama/llama-4-scout-17b-16e-instruct) — independent, scores PASS/FAIL per question
Semantic eval: BERTScore F1 with rescale_with_baseline=True → 0.94 (target ≥ 0.55 ✓)

The judge never knows which pipeline produced which answer. Outputs evaluated blind.

The Numbers That Surprised Us

BasicRAG failed on entity-specific questions — not because of bad embeddings, but because CRM entities like "Acme Corp" or "LoneStar" don't appear in enough text chunks to surface via cosine similarity. They live in structured records. BasicRAG was flying blind on 14 of 36 questions.

72.5% token reduction meant cost per query dropped from ₹0.125 (BasicRAG) to ₹0.049 (GraphRAG) — 61% cheaper, while being more accurate.

87.4% faster — 9.9s average vs 78.5s. BasicRAG's latency came from embedding and ranking 15 chunks. Graph traversal is a targeted lookup.

The Live Dashboard

Static site (vanilla HTML/CSS/JS, zero build step) deployed on Vercel. Shows graph traversal hop-by-hop in real time — watch the query walk Customer → Deal → Employee as it resolves.

Live: crm-nexus-team-brocode.vercel.app

GitHub: github.com/vishnu-k-dev/crm-nexus

What We'd Do Differently

Tune the hybrid HNSW + graph pipeline — vector seed first, then traversal. We implemented it but didn't have time to optimize.
More eval questions — 36 proves the pattern, 100 would tighten confidence intervals.
TigerGraph Cloud over local Docker — eliminates infra debugging entirely.

Stack

Layer	Technology
Graph DB	TigerGraph Community Edition (Docker)
Queries	GSQL multi-hop + REST++ API
Embeddings	Jina AI jina-embeddings-v2-base-en (768-dim)
LLM	Llama 3.3 70B via Groq
Judge	Llama 4 Scout 17B via Groq
API	Fastify + Node 20 + TypeScript
Dashboard	Vanilla HTML/CSS/JS on Vercel

Built for the TigerGraph GraphRAG Inference Hackathon 2026 — Team BroCode (Vishnu K & Revanth M)

Tags: #TigerGraph #GraphRAG #GraphDatabase #LLM #Hackathon #RAG