How We Slashed Real-Time LLM Token Costs by 65% in an Always-Listening Meeting Copilot

When building real-time AI agents, the hardest challenge isn't just getting accurate answers—it's keeping API bills from exploding.

While developing LiveAssist—a silent, real-time meeting copilot designed to help hosts answer customer questions on live calls by querying corporate RAG vaults—we immediately ran into a massive architectural roadblock: Token Over-Consumption in Continuous Audio Streams.

Here is the engineering breakdown of how we solved it, cut unnecessary token costs by over 65%, and kept latency under 2 seconds.

The Problem: The "Always-Listening" Trap

In a standard 60-minute business meeting or sales call:

• ~70% of the conversation consists of greetings, small talk, pleasantries ("Can you hear me?", "Nice weather today"), and non-actionable chatter.
• Only ~30% consists of critical technical, commercial, or compliance questions requiring precise answers.

If your system pipes the entire real-time transcription stream directly into an LLM and vector database:

1. Insane Token Costs: You burn tens of thousands of tokens per call on useless conversational filler.
2. Noise & High Latency: The RAG pipeline gets triggered repeatedly, spamming the meeting host with irrelevant document chunks and slowing down response speed.

Why Simple Fixes Failed

• Naïve Full-Stream Ingestion: Pushing everything to the LLM bankrupts the unit economics of a SaaS product.
• Large Filtering System Prompts: Instructing a high-end model (e.g., GPT-4o or Claude 3.5) with a long system prompt to "ignore chatter" still charges you input tokens for every single word uttered during the hour.

The Multi-Stage Filtering Architecture

To solve this, we decoupled speech ingestion from the core RAG generation pipeline into a three-tier architecture:

[ Real-Time Audio Stream ]
│
▼
[ Stage 1: Lightweight Intent Classifier / Fast Semantic Router ]
├── Casual Small Talk? ──────► [ Dropped / Local State Update ]
└── High-Intent Query? ──────► [ Proceed to Stage 2 ]
│
▼
[ Stage 2: Context Condenser & RAG Vault Retrieval ]
├── Queries AES-256 Encrypted Corporate Vector DB
└── Strips Redundant Context Chunks
│
▼
[ Stage 3: Real-Time Host UI Display ]
└── Low-Latency Answer Generation on Screen

1. Stage 1: Fast Intent Gating Before any text reaches the primary LLM or vector search, it passes through a sub-50ms lightweight intent router. This layer analyzes speech chunks to classify intent: Is this a direct inquiry/question directed at the host? Is it casual banter or an off-topic interjection? If it’s casual filler, the pipeline terminates immediately with zero RAG or foundation LLM token spend.
2. Stage 2: Query Normalization & Semantic Retrieval When a legitimate query is detected (e.g., "What is your SLA on enterprise escrow?"), the engine cleans conversational artifacts, structures the query vector, and retrieves only the top relevant chunks from the company's uploaded docs (PDFs, pricing sheets).
3. Stage 3: Zero-Data Leak Streaming The answer is synthesized concisely and surfaced to the host's cockpit in real time—delivering the technical answer without latency spikes. The Results By shifting the heavy lifting away from continuous LLM inference to upstream gating: Token Cost Reduction: Slashed input/output token usage by over 65% on 45–60 minute calls. Response Latency: Reduced answer turnaround to near real-time by eliminating false-positive RAG lookups. Host Focus: Completely removed UI noise, ensuring hosts only see answers when a client actually asks a question. Key Takeaway for AI Engineers Real-world AI application development is rarely about just connecting an API endpoint. True production-ready AI requires cost engineering, intelligent gating, and strict latency optimization. We rolled this architecture directly into LiveAssist. If you're managing complex client calls or technical sales, you can test it out with a 3-hour free trial.

How are you handling the tokens?
Let's discuss in the comments!