DEV Community: Shekhar maheshwari

Your LLM Is Reading Too Much: How RAG Can Lower AI Costs

Shekhar maheshwari — Wed, 21 Oct 2026 03:07:01 +0000

When teams talk about Retrieval-Augmented Generation, they usually focus on accuracy. RAG helps a model answer with current, relevant information and reduces hallucinations.

That is true, but it misses another practical benefit:

RAG can make an AI application much cheaper to run.

The reason is simple. Every time you call an LLM, you pay for what the model reads as well as what it writes. If you repeatedly send a long policy manual, a large product catalog, or an entire conversation history, the model processes all of it—even when only a small part is useful.

RAG gives us a better option. Search first, retrieve only the relevant information, and let the model read that smaller set of evidence.

The Short Version
LLMs charge for input tokens as well as output tokens.
Sending more context than necessary increases cost and latency.
RAG retrieves a focused evidence set before calling the model.
Deterministic parsing and structured filters can reduce the work even further.
The goal is not to retrieve everything that might be relevant. It is to retrieve enough trustworthy information to answer the question correctly.

Most of the Waste Is in the Prompt

A simplified LLM bill looks like this:

request cost = input tokens × input rate
             + output tokens × output rate

The input is not just the user's question. It may include the system prompt, chat history, reference documents, product records, examples, formatting rules, and tool definitions.

Imagine a support assistant with a 40,000-token policy manual. A customer asks one question about refunds, but the application sends the complete manual on every request. The model may only need two paragraphs, yet you still pay for it to process everything else.

RAG changes the flow:

Question
   ↓
Search and filtering
   ↓
A small set of relevant evidence
   ↓
Grounded LLM response

If retrieval cuts the reference material from 40,000 tokens to 2,000, the model avoids reading 38,000 unnecessary tokens each time. That saving becomes significant once the application handles thousands of requests.

Search Should Do Search Work
Without retrieval, we often ask the LLM to do several jobs at once: find the relevant information, ignore unrelated content, apply business rules, resolve ambiguity, and write the answer.

An LLM can do this, but it is an expensive way to search.

A search engine, vector database, or hybrid retrieval system can narrow the candidate set before generation. Metadata filters can remove products from the wrong category, incompatible devices, expired policies, or records the user is not allowed to access.

The model then receives a cleaner prompt and has a smaller job: explain the evidence.

A Practical Multi-Intent Example
Consider this request to a telecom shopping assistant:

Find a case for my iPhone 15 and a screen protector for my Galaxy S24, both under $50.
This looks like one sentence, but it contains two separate product searches. Each accessory must remain connected to the correct phone, and both searches must respect the budget.

One approach is to send the sentence and a large product catalog to an LLM and ask it to work everything out. A more efficient pipeline looks like this:

User query
   ↓
Deterministic telecom parsing
   ↓
Multi-intent routing
   ↓
Typesense retrieval with compatibility and price filters
   ↓
A small set of matching products
   ↓
LLM writes the final

The parser can recognize the devices, accessory categories, and budget using phrase matching and regular expressions. That does not require an LLM call.

Conceptually, the result looks like this:

{
  "requests": [
    {
      "category": "case",
      "device": "iPhone 15",
      "maximum_price": 50
    },
    {
      "category": "screen protector",
      "device": "Galaxy S24",
      "maximum_price": 50
    }
  ]
}

The routing layer handles the two intents separately. Typesense searches the correct categories and applies compatibility and price filters. Only the matching products reach the final prompt.

This matters for both cost and quality. The LLM does not have to inspect a broad catalog, guess which accessory belongs to which device, or remove incompatible products. It receives focused evidence and turns it into a useful answer.

Sometimes You Do Not Need Generation
Not every message in a chat interface needs a generated response.

Requests such as these are often structured searches:

Show me cases compatible with an iPhone 15.
Find plans under $80.
Look up order 12345. If the application can answer with a product list, account record, or database result, it may be able to skip generation completely.

That gives us three useful execution paths:

Exact lookup          → search or database response
Grounded explanation → retrieval + smaller LLM
Complex

The cheapest model call is still the one you do not make.

RAG Can Also Become Expensive

Adding a vector database does not automatically reduce cost. A careless RAG implementation can make the system more expensive.

One common mistake is retrieving too much. Increasing top_k may improve recall, but every extra result adds tokens to the prompt. Ten similar passages are not necessarily better than three useful and diverse ones.

Another mistake is putting an LLM at every stage: one model rewrites the query, another classifies it, another reranks the results, and a final model writes the answer. That may be justified for difficult use cases, but it can also turn one model call into four.

Conversation history is another source of hidden cost. Even if document retrieval is efficient, an unlimited chat history can make the prompt grow on every turn. Keep the history relevant to the active task and store durable state in a structured form where possible.

Finally, do not ignore output length. A product lookup usually does not need an essay. Short, task-appropriate answers reduce cost and are often better for users.

What a Cost-Aware Pipeline Looks Like
In practice, I would design the flow around a few questions:

Can the request be normalized or parsed without an LLM?
Are there explicit filters such as category, compatibility, price, date, or permissions?
Can search answer the request directly?
If generation is required, what is the smallest useful evidence set?
Does the task need a large model, or can a smaller model handle the grounded answer?

The resulting architecture might normalize the request, apply structured filters, run lexical and semantic retrieval, deduplicate the results, and pass only a bounded evidence set to the appropriate model.

Measure Cost per Successful Task
Token usage is important, but it should not be measured in isolation. An answer that is cheap and wrong simply creates another request or sends the customer to support.

Useful metrics include:

Input and output tokens per request
Number of retrieved tokens that are actually used
Retrieval and generation latency
Requests answered without an LLM
Small-model and large-model routing rates
Retry rate
Cache-hit rate
Cost per successfully completed task

These metrics reveal whether the pipeline is genuinely becoming more efficient or merely moving the cost somewhere else.

The Principle I Keep Coming Back To
The purpose of RAG is not to send the model as much potentially useful information as possible.

It is to find the

smallest trustworthy set of evidence that can answer the question.

That principle leads to better engineering decisions. Parse predictable inputs without an LLM. Apply business rules before retrieval. Use search for search problems. Keep the context focused. Call a larger model only when the task truly needs one.

RAG reduces cost when it changes what the model is responsible for. Search retrieves. Filters remove irrelevant options. Caches reuse stable work. The LLM focuses on the part it does best: turning the right evidence into a clear answer.

Your model does not need to read everything you know. It only needs what helps it answer the question in front of it.

From Click Logs to Search Ranking: Building an Offline Popularity Trainer with Python and Typesense

Shekhar maheshwari — Mon, 31 Aug 2026 03:58:52 +0000

Most learning-to-rank examples end when the model has been trained. A production search system still has several harder questions to answer:

Which user events are safe to treat as training examples?
How should result position affect a click?
How do we build features across catalogs with different schemas?
How do we prevent document leakage between training and evaluation?
What happens when there is too little usable data?
Does the model need to run inside every search request?

I encountered these questions while building the popularity-training pipeline for an Intelligent Search Platform backed by Python, PostgreSQL, scikit-learn, and Typesense.

The resulting design trains a small model offline, converts its predictions into bounded popularity scores, and writes those scores back to the search index. The search path reads a number; it never invokes scikit-learn.

This article explains the architecture, the decisions behind it, and several edge cases that only became obvious after testing the full training path.

Why replace a fixed popularity formula?

A simple popularity calculation might assign every event a contribution such as:

event contribution = position weight × event value

A click could be worth 1.0, an impression could be worth 0.1, and results farther down the list could receive a logarithmic discount.

This approach is understandable, cheap, and often a reasonable starting point. Its limitation is that every catalog receives the same assumptions. A fixed formula cannot learn that ratings matter in one catalog, recency matters in another, or a particular categorical attribute is associated with user engagement in a third.

The trainer changes the role of the formula. Rather than directly adding position-discounted events into the final score, it uses position weighting while fitting a model from document features and click labels.

That distinction matters:

Fixed formula:
events → hand-written aggregation → popularity

Offline trainer:
events + document features → fitted model → predictions → popularity

This is not a claim that a small logistic-regression model automatically improves relevance. It is an implementation that makes the ranking signal learnable and independently testable.

The complete offline architecture

The pipeline looks like this:

Browse impressions and clicks in PostgreSQL
                   │
                   ▼
       Join events to current documents
                   │
                   ▼
       Build schema-independent features
                   │
                   ▼
      DictVectorizer → sparse matrix
                   │
                   ▼
   Position-discounted logistic regression
                   │
                   ▼
        Predict a score for every document
                   │
                   ▼
      Max-normalize scores to 0–1,000,000
                   │
                   ▼
        Partial updates sent to Typesense

Training is exposed through an administrative endpoint:

POST /v1/training/popularity?index_id=my-catalog&days=30

The endpoint is deliberately on demand. A deployment can invoke it manually or from cron or a Kubernetes CronJob, but scheduling is not hidden inside the search service.

Step 1: restrict the first model to browse behavior

The first version uses events where the query is empty:

SELECT doc_id, rank, clicked
FROM search_events
WHERE index_id = $1
  AND query = ''
  AND created_at >= $2

This models browse popularity rather than query-specific relevance.

That narrower scope avoids mixing two different questions:

Which documents are generally attractive during browsing?
Which documents are relevant to a particular query?

Query-conditional ranking requires query features, query-document interactions, and a different evaluation design. Treating all traffic as interchangeable would make the model simpler on paper but less precise about what it actually learns.

There is an important mismatch in the current implementation: this browse-only boundary is enforced during training, but not during serving. The resulting popularity field is also added to explicit-query results. The later section, How the learned score enters ranking today, quantifies why that unscoped use is not yet a relevance-safe deployment policy.

Step 2: build features without catalog-specific branches

The platform can index unrelated domains. Hardcoding fields such as brand, plant_type, or screen_size into the trainer would make every new catalog an engineering task.

Instead, the trainer consumes affinity dimensions already discovered for each index and converts the current document into a flat feature dictionary:

def document_features(document, affinity_dimensions):
    features = {}

    for dimension in affinity_dimensions:
        if not dimension.get("enabled", True):
            continue

        field = dimension["field"]
        derived_from = dimension.get("derived_from")

        if derived_from:
            raw_value = document.get("metadata", {}).get(derived_from)
            if raw_value is not None:
                features[f"num__{derived_from}"] = float(raw_value)
            continue

        value = document.get(field) or document.get("metadata", {}).get(field)
        if isinstance(value, str) and value.strip():
            features[f"cat__{field}"] = value.lower().strip()

    return features

The current implementation also includes these optional signals when available:

is_featured
a rating signal: averageStarRating × log(1 + ratingsCount)
bounded recency: 1 / (1 + days_since_indexing)
the raw numeric source behind discovered tiers such as price_tier

is_featured deserves special caution. Featured placement can itself cause additional exposure and clicks, and ISP applies a separate serving-time featured boost as well. Including it in the learned model can encode an editorial intervention into popularity and then apply that intervention again during serving. I retained it in this description because it is present in the current implementation, but I would exclude it from a subsequent version unless an experiment explicitly estimates its effect.

Missing fields are omitted. There is no requirement for every catalog to expose an identical schema.

DictVectorizer is a good fit for this representation:

from sklearn.feature_extraction import DictVectorizer

vectorizer = DictVectorizer(sparse=True)
X = vectorizer.fit_transform(feature_dicts)

It one-hot encodes categorical strings, retains numeric values, and produces a sparse matrix without building a DataFrame or maintaining a manual vocabulary.

Step 3: account for result position—carefully

Clicks are not direct relevance labels. A result near the top is more likely to be examined than the same result near the bottom.

The current trainer uses a logarithmic weight:

import math

def position_weight(rank: int) -> float:
    return 1.0 / math.log2(rank + 2)

Each event becomes:

X = features of the displayed document
y = 1 for click, 0 for impression without click
w = 1 / log2(rank + 2)

The model receives w through sample_weight:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)
model.fit(X, labels, sample_weight=weights)

This logarithmic weight is an exposure-confidence discount inherited from the formula-based pipeline. It reduces the influence of events at deeper positions, including deep clicks. It therefore does not correct position bias and may suppress informative clicks on poorly ranked documents.

Proper inverse-propensity scoring requires estimated examination probabilities, usually obtained through randomized exposure or a validated click model. Click contributions are then weighted by inverse propensity, often with clipping to control variance. Non-clicks also require careful treatment because absence of a click does not establish irrelevance. The counterfactual basis for this approach is described in Unbiased Learning-to-Rank with Biased Feedback.

Calling this version “position-discounted” is accurate. Calling it “position-debiased” would not be. Propensity estimation, IPS or SNIPS objectives, randomized exposure data, and counterfactual evaluation remain separate follow-up work.

Step 4: join against the current index

Interaction logs can outlive documents. Products may be removed between an impression and a later training run.

The trainer therefore fetches the current documents from Typesense and joins each event by doc_id. Events for documents that no longer exist are skipped.

This creates an easily missed edge case. Suppose the database returns six events, clearing a five-event minimum, but four events reference deleted documents. Only two usable examples remain.

The minimum must be checked twice:

if len(raw_events) < MIN_EVENTS:
    return empty_result()

usable_examples = join_events_to_current_documents(raw_events, documents)

if len(usable_examples) < MIN_EVENTS:
    return empty_result()

Checking only the SQL result count can send an unexpectedly tiny matrix into model fitting.

The five-event threshold is only a mechanical guard against invalid model fitting. It is not evidence that five events are statistically sufficient for a useful ranking model. A production promotion policy should require a separately validated traffic threshold and ranking-oriented evaluation.

Step 5: handle data that cannot train a classifier

Production data is often incomplete in uninteresting ways. The trainer treats these situations as valid empty runs instead of server failures:

fewer than the minimum number of browse events
Typesense unavailable or the collection missing
too few events remaining after the document join
all labels belonging to one class
every document producing an empty feature dictionary

The last case exposed a real failure mode. DictVectorizer can produce a matrix with rows but zero columns when every feature dictionary is empty:

X = vectorizer.fit_transform(feature_dicts)

if X.shape[1] == 0:
    return empty_result()

Without the guard, the downstream estimator receives no features and raises instead of reporting that there is nothing to learn.

An empty training run is still recorded in the training_runs table. “No model was produced” is operational information, not an event that should disappear from history.

Step 6: prevent document leakage during evaluation

The same document can generate many events, and every one of those events shares the same document feature vector.

A random per-event split can therefore place impressions for one document in training and clicks for the same document in testing. The model has effectively already seen the test document's features, producing an overly optimistic metric.

The implementation groups by document ID:

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(
    n_splits=1,
    test_size=0.2,
    random_state=0,
)

train_indices, test_indices = next(
    splitter.split(X, labels, groups=example_document_ids)
)

All events for a document remain on one side of the split.

Small grouped datasets introduce another edge case: one side may contain only clicks or only non-clicks. In that situation, evaluation AUC is unavailable. The trainer returns auc: null but still fits the production model on the complete usable dataset.

Evaluation failure and training failure are not necessarily the same event.

Step 7: score the entire catalog

The model is trained from documents associated with observed events, but it scores every current document using its features:

all_document_matrix = vectorizer.transform(all_document_features)
probabilities = model.predict_proba(all_document_matrix)[:, 1]

This is useful for documents that have features but little or no direct interaction history. It does not solve cold start completely—the model still depends on relationships learned from other documents—but it avoids limiting scores only to documents that already received clicks.

The predictions are max-normalized into the existing popularity range. This is a serving-scale transformation, not statistical probability calibration:

maximum = max(probabilities, default=0.0)

if maximum > 0:
    popularity_scores = [
        round(probability / maximum * 1_000_000)
        for probability in probabilities
    ]
else:
    popularity_scores = []

The trainer then performs a partial update:

typesense.update_document_fields(
    collection_name,
    document_id,
    {"popularity": popularity},
)

Structurally, the existing search and merchandising paths can continue reading the same field, so replacing the score generator does not require model inference inside every query. That is an operational compatibility property—not a claim that the current serving-time blend is relevance-safe. The next section describes the unresolved ranking risk.

How the learned score enters ranking today

The training and serving boundaries are decoupled, but the scale used to combine their outputs still matters.

ISP currently retrieves lexical and semantic candidates separately and merges their ranks using:

rrf_score =
    0.7 / (60 + lexical_rank + 1)
  + 0.3 / (60 + semantic_rank + 1)

It then adds the stored popularity value to every candidate's score:

popularity_boost = min(popularity, 1_000_000) / 2_000_000
final_score = rrf_score + popularity_boost + other_configured_boosts

A document ranked first in both retrieval channels receives an RRF score of approximately 0.0164, while the largest popularity boost is 0.5. Popularity can therefore dominate the relative order of the retrieved candidates rather than act as a small tie-breaker.

That behavior requires particular caution because this model is trained only on empty-query browse events, while the stored field is currently consumed during both browsing and explicit search. The safer target design is:

during browsing, allow popularity to be a primary signal;
during explicit search, disable it, bound it relative to relevance, or use it only within relevance groups;
measure how often popularity displaces a more relevant result before promoting a new blending policy.

Typesense documents a related relevance-bucketing approach in which a custom popularity score reorders results only within text-relevance groups. That is not identical to ISP's application-side RRF, but it illustrates the serving constraint. See Ranking Based on Relevance and Popularity.

This serving refinement is not implemented yet. The present article documents the current pipeline rather than claiming that its blending coefficient is already validated.

Why keep inference outside the request path?

Offline scoring has several practical advantages:

no scikit-learn model needs to be loaded by every API worker
no feature vector is constructed for every candidate during search
the search latency budget remains controlled by the search engine
rollback can restore or recompute one numeric field
the existing formula-based pipeline remains available as an independent fallback

The trade-off is freshness. Scores change only when the training job runs. That is acceptable for a browse-popularity signal whose update interval can be measured in hours or days, but it would be inappropriate for a feature requiring immediate adaptation.

Testing the path that matters

Router tests can confirm authentication, index resolution, and graceful API responses while still missing the core ML path.

The domain-level suite therefore drives the complete trainer with a real test database and mocked Typesense reads and writes. Its regression cases include:

Zero-feature matrix: mixed click labels but no usable document features must return zero scored documents rather than crash.
Post-join minimum: raw events may clear the threshold while usable events do not.
Grouped evaluation: events for the same document must never cross the evaluation boundary.
Single-class grouped split: AUC can become unavailable without preventing the final full-data model from scoring documents.

These tests caught assumptions that ordinary endpoint tests could not exercise.

What this system does not prove

An operational model pipeline and an improved ranking policy are not the same achievement.

This implementation does not yet provide:

unbiased propensity correction
query-specific learning-to-rank
counterfactual policy evaluation
nDCG or MRR validation against relevance judgments
automated retraining schedules
pooled learning across low-traffic indexes
evidence of conversion or revenue improvement
a validated guarantee that popularity cannot overpower explicit-query relevance
freedom from editorial-exposure confounding or double-counting through is_featured
statistically calibrated probabilities across separate training runs

A held-out AUC is a sanity check for the classifier, not proof that users receive better rankings. Demonstrating ranking improvement requires a stronger offline evaluation protocol with temporal splits and ranking metrics such as nDCG or MRR and, when appropriate and authorized, a controlled online experiment.

Lessons learned

The model was the shortest part of the feature. The important engineering decisions were around it:

Define exactly which behavior the model represents.
Reuse schema-derived features instead of hardcoding a vertical.
Describe position discounting accurately; do not present it as debiasing.
Recheck data sufficiency after every destructive transformation.
Group repeated entities when splitting interaction data.
Record empty and failed-to-evaluate runs explicitly.
Keep offline learning separate from query-time serving when the signal allows it.
Test vectorization, fitting, scoring, normalization, and write-back—not just the endpoint.
Keep learned browse signals from silently overpowering explicit-query relevance.

The result is intentionally modest: a small, auditable model that learns a browse-popularity signal and hands the search engine a number it already knows how to use.

That modest boundary is also what makes the system practical.