Ishwar

Posted on Jul 11

I Rebuilt My AI Legal Assistant After Learning Why Vector-Only RAG Wasn't Enough

#devchallenge #weekendchallenge #ai #sqlite

DEV Weekend Challenge: Passion Edition Submission

This is a submission for Weekend Challenge: Passion Edition

What I Built

LawDecoder is an AI-powered legal assistant that explains Indian laws in plain language while showing the exact legal provisions used to generate each answer.

Unlike many dense-only retrieval prototypes, LawDecoder focuses on retrieval quality. It was born out of a real-world failure. One day I asked my original prototype:

"Someone forged my signature."

The first law it retrieved wasn't about forgery. It was about counterfeit coins.

That was the moment I realized the problem wasn't the LLM—it was my retrieval pipeline.

The surprising part: I didn't change the LLM. Nearly all of the massive retrieval accuracy gains came from redesigning the search architecture to combine semantic search, keyword search, Reciprocal Rank Fusion (RRF), and deterministic domain reranking.

Demo

Here is a visual walkthrough of the production UI, showing how the hybrid search handles citations and developer metrics:

1️⃣ User Chat Interface

Clean, legal explanation interface for end users:

2️⃣ Structured Offence & Citation Details

Deduplicated citations with developer metrics visible in Developer Mode:

3️⃣ System Evaluation Dashboard

Performance comparisons and technical architecture story:

Code

The complete implementation—including SQLite ingestion, FTS5 indexing, Reciprocal Rank Fusion, evaluation queries, and benchmark samples—is available on GitHub:

ishwar170695 / LawDecoder

⚖️ LawDecoder: A Case Study in Hybrid Retrieval for Legal AI

LawDecoder is an AI-powered legal assistant that explains Indian laws in plain language while showing the exact legal provisions used to generate each answer.

Unlike many dense-only retrieval demos, LawDecoder focuses on retrieval quality. It combines semantic search, keyword search, Reciprocal Rank Fusion (RRF), and deterministic reranking to improve legal citation accuracy.

🚀 Features

🔍 Hybrid Retrieval: Fuses SQLite FTS5 (sparse BM25 keyword matching) and local dense vector embeddings.
🔀 Reciprocal Rank Fusion (RRF): Fuses sparse and dense search rankings to prioritize matches returned by both.
🎯 Domain Reranker: Deterministically demotes irrelevant matches (like counterfeit stamp/coin sections) and boosts direct offences (like document forgery acts) for signature queries.
🧾 Citation Transparency: Shows the exact acts, sections, and selection details for every explanation.
🔧 Developer Mode: Toggle view to inspect RRF ranks and retrieval selection reasons.
🤖 Empathetic AI: Structured…

View on GitHub

How I Built It

The Failure: Tracing the Cause

The original vector-only (v1) search failed on a simple forgery query:

Query:
"Someone forged my signature"

Top result (v1):
❌ BNS Section 180 — Possession of counterfeit coin

Expected:
✅ BNS Section 336 — Forgery
✅ BNS Section 340 — Using forged document

The embedding model wasn't "wrong"—it placed semantically close concepts close together in vector space. But in legal search, semantically similar isn't the same as legally relevant.

Because "forgery" and "counterfeit" ended up close in embedding space, the retriever ranked counterfeit coin and government stamp provisions above the actual definition of document forgery. The retriever generalized too aggressively, missing the direct definition of forgery (BNS Section 336) because the word "signature" was semantically distant from generic statutory descriptions of the offence.

Additionally, caching large raw text strings (text, titles, act names) in a JavaScript array caused the Node.js process to consume over 320 MB of RAM at startup.

The Redesigned Retrieval Pipeline

I redesigned the retrieval pipeline to combine sparse and dense search methods.

The LLM remained almost unchanged. Nearly all improvements came from redesigning retrieval.

1. SQLite + FTS5 Sparse Indexing

I moved all 4,892 legal sections out of JSON files and persisted them in a local SQLite database. A virtual FTS5 index handles exact-match keyword indexing (BM25 ranking), ensuring precise matches for terms like "Section 65", "forgery", or "signature".

2. Lightweight Vector Memory Cache

I stripped all text metadata from Node.js memory. The startup script now loads only the id (string) and the coordinate list—pre-processed into a compact Float32Array object—into RAM. The actual text content remains on disk in SQLite and is hydrated for the top 5 matched sections on-demand, reducing memory usage by 85% (from 320 MB to 48 MB).

3. Reciprocal Rank Fusion (RRF)

Fuses the top 50 semantic matches (dense) and top 50 keyword matches (sparse) into a single unified list using the reciprocal ranks of both retrievers. Here is the core JS implementation of the RRF merge:

const rrfScores = new Map();
const k = 60; // Standard constant for RRF

// Process vector ranking positions (dense)
vectorRankings.forEach((item, index) => {
  rrfScores.set(item.id, 1 / (k + index + 1));
});

// Process FTS5 keyword ranking positions (sparse) and add to scores
ftsRankings.forEach((item, index) => {
  const existingScore = rrfScores.get(item.id) || 0;
  rrfScores.set(item.id, existingScore + (1 / (k + index + 1)));
});

// Sort matched IDs based on fused RRF score
const mergedRanking = Array.from(rrfScores.entries())
  .sort((a, b) => b[1] - a[1])
  .slice(0, 20); // Top 20 candidates for reranking

4. Domain Reranker (Deterministic Guardrail)

It evaluates the top 20 candidates returned by the RRF step. This is a deterministic rule-based reranker tailored to the legal domain—not a learned neural cross-encoder:

If the query is document/signature forgery-related, it checks if a retrieved document is a coin or banknote counterfeit section. If yes, it penalizes the score by 99% (* 0.01).
It boosts direct document forgery offences (containing "forgery" or "forged") by 300% (* 3.0).
It filters out duplicate sections using content snippet prefixes.

// Heuristic Domain Reranking
if (isDocumentForgeryRelated) {
  const isCoinOrStampOrCurrency = 
    titleLower.includes('coin') || titleLower.includes('stamp') || 
    titleLower.includes('currency') || titleLower.includes('bank-note') ||
    contentLower.includes('coin') || contentLower.includes('stamp') || 
    contentLower.includes('currency-note');

  if (isCoinOrStampOrCurrency) {
    adjustedScore *= 0.01; // heavily penalize counterfeit coin/stamps (reduce by 99%)
  } else if (titleLower.includes('forgery') || titleLower.includes('forged')) {
    adjustedScore *= 3.0; // strong boost for direct forgery definitions/offences
  }
}

Performance & Evaluation Benchmarks

To evaluate the redesign, I assembled a benchmark of 100 manually verified legal queries spanning criminal law, cybercrime, family law, consumer protection, and procedural law.

Metric	v1 (Naive Vector RAG)	v2.1 (Hybrid Search - Current)	Change
Search Engine	Dense Vector (Linear JSON scan)	Hybrid (SQLite FTS5 + Dense Vector + RRF + Reranker)	Major retrieval precision upgrade
Avg. Query Latency	`466 ms`	`12 ms`	97.4% speedup
Memory Cache Footprint	`~320 MB`	`~48 MB`	85.0% RAM savings
Duplicate Citations	Present (up to 40% overlaps)	Deduplicated (0% overlaps)	Verified
Top-5 Relevant Retrieval Rate	~68%	~91%	+23% accuracy gain

Latency is based on 100 benchmark queries. Memory is process-level heap size at startup. Accuracy is evaluated on top-5 target matches using a manually verified benchmark dataset of 100 queries.

None of these improvements required changing the language model. The gains came almost entirely from retrieval engineering. For this benchmark query, the top retrieved references aligned with the expected legal provisions:

BNS 340: Forged document and using it as genuine
BNS 336: Forgery definition and penalty
BNS 339: Possession of forged document
BNS 335: Making a false document
Evidence Act Section 65: Proof of signature and handwriting

Lessons Learned

Retrieval quality sets the upper bound for RAG quality: The LLM can only reason over what you retrieve. Improving retrieval turned out to be far more impactful than switching models.
Dense embeddings alone are rarely enough for domain-specific search: Hybrid retrieval is often a better default because it combines exact keyword matching with semantic understanding.
SQLite + FTS5 is a powerhouse: For corpora under 100,000 documents, SQLite FTS5 and typed arrays in Node.js deliver sub-15ms latency on CPU with zero operational complexity.
Simple deterministic rerankers work: They can eliminate domain-specific retrieval errors without requiring another neural model.

What's Next

If I continue evolving this project, the next improvements I'd explore are:

Cross-encoder reranking: Integrate lightweight cross-encoders (e.g. BGE reranker) for advanced ranking.
Metadata-aware retrieval: Allow users to filter queries by Act or category before searching.
Legal Case Retrieval: Expand indexing to cover legal precedents and court cases.
Multilingual support: Support query translation for regional languages.

Prize Categories

Best Use of Google AI

LawDecoder integrates the Google Gemini API (gemini-3.5-flash or custom models) for generating context-grounded, empathetic, and structured legal advice derived from the hybrid SQLite-FTS5 retrieval output.

DEV Community