I’ve been building AI systems for autonomous vehicles long enough to develop a love-hate relationship with retrieval-augmented generation (RAG). It’s a great concept — bring relevant context into your LLM prompt at runtime — but the second you move beyond text-heavy enterprise use cases into robotics or real-time perception, things get weird fast.
Let’s talk about what happens when you try to apply RAG to high-dimensional, multimodal data, and why your choice of vector database can quietly make or break your pipeline.
Not All Embeddings Are Created Equal
Most RAG tutorials use sentence-transformer or OpenAI embeddings on small textual corpora. But when you’re fusing LiDAR, radar, and camera inputs — or even running multimodal embeddings from perception models like Perceiver or CLIP — you’re suddenly dealing with:
- 2,048 to 4,096 dimensions per vector
- tens of millions of vectors per sensor window
- updates on the scale of milliseconds, not hours
The vector DBs that look great on standard SIFT1M or Wikipedia benchmarks often collapse here. I’ve seen Milvus handle this scale better than most (especially with its tiered IVFPQ indexing), while something like Pinecone starts to choke unless you heavily batch and precompute everything.
Querying in the Chaos: Real-Time Constraints
In AV systems, RAG isn’t just about semantic search — it’s about making the right decision right now. Think:
- “What similar trajectories did I see in prior encounters with a jaywalking pedestrian?”
- “Are there any annotated LiDAR clusters from edge cases similar to this object’s motion?”
That means your vector DB needs sub-50ms recall with high accuracy — and most importantly, low tail latency. An index that gives you 95% recall at P50 but spikes to 800ms at P99 is a nonstarter. For me, that ruled out FAISS-on-disk solutions and pushed us toward in-memory hybrid setups, sometimes backed by Milvus or even Redis-AI when latency spikes were unacceptable.
Hybrid Search Isn’t Optional
Another trap: pure ANN (approximate nearest neighbor) isn’t enough. We need hybrid search — combining structured filters (e.g. location, object class, time window) with vector similarity — to avoid surfacing irrelevant results that are semantically close but contextually useless.
The systems I’ve liked best so far:
- Milvus: Flexible filtering + multi-modal vector support + GPU acceleration
- Weaviate: Graph-aware queries and filters, good for chaining across knowledge
- Qdrant: Surprisingly solid for real-time hybrid search, nice JSON filter DSL
On the other hand, Chroma and Lancedb are great for lightweight prototyping but start to wobble under serious ingestion or query pressure.
What I’d Do Differently (And What I’d Keep)
If I were rebuilding a RAG stack for AV today, here’s where I’d land:
Keep:
- HNSW-based indexes tuned for short queries
- Streaming ingestion pipelines with nightly reindexing
- Embedding normalization (even small vector scale issues cascade fast)
Change:
- Use separate DBs for long-term recall vs short-term context
- Bake in observability for query latency distribution — not just mean/median
- Use hybrid pipelines: Redis or Vespa for immediate low-latency + Milvus for batch-heavy recall
Final Thought
RAG in robotics isn’t just a language problem — it’s a systems problem. The tech that works for enterprise chatbots often breaks under the weight of real-time perception and control loops. But with the right infra — and a vector DB that understands filters, scale, and latency — it’s not just possible. It’s damn useful.
If you’re working on similar problems (or have war stories from trying RAG with non-text data), I’d love to swap notes.
Top comments (0)