Two Patterns for Reducing LLM Costs in Data-Heavy RAG Apps

#llm #python #webdev #ai

How we cut token usage significantly in an F1 telemetry analyzer by rethinking what goes into the context window — and when.

When building RAG applications on top of structured data (databases, APIs, telemetry), the naive approach is to dump everything into the context and let the LLM figure it out. It works, but it's expensive and slow. After building F1 Analyst Pro — a chat interface for Formula 1 race analysis backed by FastF1 + Supabase + Claude — we refined two patterns that significantly reduced both token usage and API costs without sacrificing response quality.

This post covers both patterns with real code from the project.

The Problem: Context Bloat

A typical race weekend in F1 Analyst Pro has data across multiple sessions: FP1, FP2, FP3, Qualifying, and Race. Each session has lap-by-lap data for 22 drivers — compound, lap time, position, stint, sector times, tyre life. On top of that: qualifying results, race results, stint summaries, pit stop analysis, key moments, race incidents, and journalist notes.

If you send all of that to the LLM on every query, two things happen:

Cost explodes. A full context dump for a race weekend easily reaches 8,000-12,000 tokens per query. At $3/MTok for Claude Sonnet input, that's $0.024-$0.036 per query — fine for one user, painful for many.
Quality degrades. LLMs perform worse when the context contains irrelevant information. A question about qualifying doesn't need race stint summaries — and including them adds noise that can distract the model from what matters.

The solution is two complementary patterns: conditional context and pre-generation.

Pattern 1: Conditional Context via Intent Detection

Instead of always building the full context, detect what the user is actually asking about and fetch only the relevant data.

Intent Detection

import unicodedata

def _detect_intents(prompt: str) -> dict:
    # Normalize accents so "clasificación" matches "clasificacion"
    p = unicodedata.normalize("NFD", prompt.lower())
    p = "".join(c for c in p if unicodedata.category(c) != "Mn")

    wants_qualy = any(w in p for w in [
        "clasificacion", "qualifying", "q1", "q2", "q3", "pole", "sector"
    ])
    wants_race = any(w in p for w in [
        "carrera", "race", "resultado", "vuelta rapida", "stint", "degradacion"
    ])
    wants_telemetry = any(w in p for w in [
        "telemetria", "aceleracion", "frenada", "velocidad", "throttle", "brake"
    ])
    wants_undercut = any(w in p for w in [
        "undercut", "overcut", "parada", "pit stop", "estrategia de pit"
    ])
    wants_practice = any(w in p for w in [
        "entrenamiento", "practica", "fp1", "fp2", "fp3", "long run", "evolucion"
    ])
    wants_race_sim = any(w in p for w in [
        "race simulation", "simulacion de carrera", "ritmo de fp2"
    ])

    # load_all is the fallback: fires only when no specific intent is detected
    load_all = not any([
        wants_qualy, wants_race, wants_telemetry,
        wants_undercut, wants_practice, wants_race_sim
    ])

    return {
        "wants_qualy": wants_qualy,
        "wants_race": wants_race,
        "wants_telemetry": wants_telemetry,
        "wants_undercut": wants_undercut,
        "wants_practice": wants_practice,
        "wants_race_sim": wants_race_sim,
        "load_all": load_all,
    }

Conditional Context Building

Each intent triggers specific SQL queries. Critically, expensive analyses (undercut/overcut, key moments, race incidents) are only fetched when they're actually relevant:

def build_context(self, prompt: str, gp_name: str, year: int) -> str:
    intents = _detect_intents(prompt)
    context = ""

    # --- QUALIFYING ---
    if intents["wants_qualy"] or intents["load_all"]:
        q_id = self.db.get_session_id(year, gp_name, "Q")
        if q_id:
            q_results = self.db.get_qualifying_results(q_id)
            context += "--- QUALIFYING (Q) ---\n" + q_results.to_string(index=False) + "\n\n"

    # --- RACE ---
    if intents["wants_race"] or intents["wants_undercut"] or intents["load_all"]:
        r_id = self.db.get_session_id(year, gp_name, "R")
        if r_id:
            # Always include stint summary for race queries
            stints = self.db.get_stint_summary(r_id)
            context += "--- RACE STINT SUMMARY ---\n" + stints.to_string(index=False) + "\n\n"

            # Undercut analysis: only when explicitly asked OR general race query
            if intents["wants_undercut"] or intents["wants_race"] or intents["load_all"]:
                pit_df = self.db.get_pit_stop_analysis(r_id)
                if not pit_df.empty:
                    context += "--- UNDERCUT/OVERCUT ANALYSIS ---\n" + pit_df.to_string(index=False) + "\n\n"

            # Key moments: only for race/general queries, not undercut-specific ones
            if intents["wants_race"] or intents["load_all"]:
                km_df = self.db.get_key_moments(r_id)
                if not km_df.empty:
                    context += "--- KEY MOMENTS ---\n" + km_df.to_string(index=False) + "\n\n"

            # Race incidents: always when race data is present
            inc_df = self.db.get_race_incidents(r_id)
            if not inc_df.empty:
                context += "--- RACE INCIDENTS ---\n" + inc_df.to_string(index=False) + "\n\n"

    # --- PRACTICE (FP1/FP2/FP3) ---
    if intents["wants_practice"] or intents["load_all"]:
        for fp_code, fp_label in [("FP1", "PRÁCTICA 1"), ("FP2", "PRÁCTICA 2"), ("FP3", "PRÁCTICA 3")]:
            fp_id = self.db.get_session_id(year, gp_name, fp_code)
            if fp_id:
                best = self.db.get_best_lap_per_driver(fp_id)
                if not best.empty:
                    context += f"--- {fp_label} ({fp_code}) ---\n" + best.to_string(index=False) + "\n\n"

    return context

The Impact

A query like "¿Cuál fue la pole?" used to pull in race stints, pit stop analysis, key moments, and FP laps — all irrelevant. With intent detection, it only fetches qualifying results: ~400 tokens vs ~6,000 tokens for the same question. That's a 15x reduction in input tokens for qualifying-specific queries.

For general race queries (load_all=False, wants_race=True), the context is still substantial but focused — stints, undercut analysis, key moments, incidents — all directly relevant to any race question.

Pattern 2: Pre-Generation for Charts

The standard approach when a user asks for a chart is to let the LLM generate the plotting code, return it to the client, execute it, and display the result. This sounds reasonable but has three problems in practice:

It costs tokens. A matplotlib or Plotly snippet is 50-150 lines of code — all in the output window.
It's unreliable. The LLM may hallucinate API calls, use deprecated methods, or generate code that fails silently.
It's slow. The user waits for the full code generation before seeing anything.

The alternative: generate the chart from real database data before the API call, then tell the LLM the chart exists.

Implementation

def send_message(self, prompt: str, gp_name: str, year: int) -> dict:
    intents = _detect_intents(prompt)

    # Step 1: pre-generate chart if telemetry is requested
    pre_chart = None
    chart_notification = ""

    if intents["wants_telemetry"]:
        drivers = self._extract_drivers(prompt, gp_name, year)
        session_type = self._detect_session_type(prompt)
        qualifying_segment = self._detect_qualifying_segment(prompt)  # Q1/Q2/Q3

        if drivers:
            pre_chart = plot_telemetry_trace(
                laps_data=None,         # fetches directly from FastF1
                gp_name=gp_name,
                year=year,
                drivers=drivers,
                session_type=session_type,
                qualifying_segment=qualifying_segment,
            )
            if pre_chart is not None:
                chart_notification = (
                    f"\n\n[SYSTEM: A telemetry chart comparing "
                    f"{', '.join(drivers)} in {session_type} has been generated. "
                    f"Reference it in your analysis without generating any code.]"
                )

    # Step 2: build context (conditional, as shown above)
    context = self.build_context(prompt, gp_name, year)

    # Step 3: inject chart notification into the user message
    user_content = context + prompt + chart_notification

    # Step 4: call the LLM
    response = self.anthropic.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1000,
        system=self._system_prompt(),
        messages=[{"role": "user", "content": user_content}],
    )

    return {
        "text": response.content[0].text,
        "chart": pre_chart,   # passed to Streamlit for rendering
    }

The Qualifying Segment Detail

One refinement worth highlighting: qualifying telemetry in F1 has three segments (Q1, Q2, Q3). A driver eliminated in Q1 never ran in Q3 — comparing their "best lap in Q" against a Q3 driver's best lap is meaningless.

The pre-generation step handles this by detecting the segment from the prompt:

def _detect_qualifying_segment(self, prompt: str) -> str | None:
    match = re.search(r'\b(q[123])\b', prompt.lower())
    return match.group(1).upper() if match else None

And in plot_telemetry_trace:

if qualifying_segment:
    stint_map = {"Q1": 1, "Q2": 2, "Q3": 3}
    seg_laps = drv_laps[drv_laps["Stint"] == stint_map[qualifying_segment]]
    drv_laps = seg_laps if not seg_laps.empty else drv_laps  # fallback if driver wasn't in that segment

fastest = drv_laps.loc[drv_laps["LapTime"].idxmin()]

The Impact

The pre-generation pattern eliminates chart code from the LLM output entirely. A telemetry response that previously cost ~800 output tokens (code + explanation) now costs ~200 tokens (explanation only). For charts, that's a 75% reduction in output tokens.

More importantly, the chart is guaranteed to use real data — no hallucinated driver codes, no deprecated FastF1 API calls, no silent failures.

Combining Both Patterns

The two patterns compose naturally. Here's what happens for a query like "Show me the telemetry of COL vs GAS in Q2":

Intent detection → wants_telemetry=True, wants_qualy=True
Context building → fetches qualifying results only (~400 tokens)
Pre-generation → generates Q2-filtered telemetry chart for COL and GAS from FastF1 directly
API call → sends qualifying context + chart notification (~600 tokens total input)
Response → LLM references the chart and analyzes the data (~200 tokens output)

Total: ~800 tokens. Without these patterns, the same query would use ~7,000 tokens (full context dump + chart code generation).

What This Doesn't Cover

These patterns work well for structured, database-backed RAG. For unstructured document RAG (PDFs, articles, knowledge bases), vector search with embedding-based retrieval is still the right tool. The key insight is matching the retrieval mechanism to the data structure — keyword/intent-based for tabular data, semantic for text.