I Tried Using an LLM as My Robot's Memory. The Latency Alone Was 3x My Control Loop.

Last March I wired a small rover to GPT-4 for scene understanding. The setup was simple: camera feeds went to the model, responses went into a vector store, and the navigation planner queried that store before each move. It worked on my desk. It even worked in the hallway.

Then I took it outside.

The first thing I noticed was the lag. My control loop runs at 20Hz (50ms budget per cycle). A single GPT-4 vision call took 140-180ms. Add vector search round-trips and I was spending 200ms per cycle just on memory lookups. The rover was effectively blind for 3 out of every 4 control cycles.

The second thing I noticed was worse: the memory was stale. By the time the LLM finished describing a scene, the rover had moved 2 meters. It was reacting to where it had been, not where it was.

I spent two weeks trying to fix this with caching, prefetching, and async pipelines. None of it worked because the fundamental problem was architectural: I was using a remote, probabilistic system as a deterministic memory substrate.

This is the story of how I ended up building an embedded database instead. Not because I wanted to, but because every alternative I tried hit the same wall.

The Three Walls

Wall 1: Network latency. Even with a local LLM (Llama 3 8B on a Jetson Orin), inference took 60-90ms per frame. That's already over my 50ms budget. With a remote model, I was at 140-200ms. There's no amount of caching that fixes a 4x latency overrun on a hard real-time loop.

Wall 2: Probabilistic storage. LLM-based memory systems distill conversations into embeddings and summaries. The summarization step is lossy in ways that matter for robotics. "The obstacle was 2.3 meters ahead" becomes "there was an obstacle nearby." For a planner that needs sub-meter precision, that's useless.

Wall 3: Cost at scale. A single robot writing 500 memories/day through an LLM ingestion pipeline costs $7-15/day in API calls. Scale to a fleet of 50 robots and you're burning $350-750/day on memory alone. That's more than the hardware depreciation.

What I Actually Needed

I needed four things:

1. Sub-10ms writes and reads for the control loop
2. Deterministic queries (same input, same output, every time)
3. Vector + time-series + key-value in one store (robots produce all three)
4. Zero network dependency (works offline, works in a basement, works on a Mars rover)

None of the existing solutions checked all four boxes. SQLite is fast and deterministic but has no native vector search. Qdrant and Milvus have great vectors but are network-first and too heavy for edge. DuckDB is analytical, not designed for real-time writes. Postgres with pgvector is powerful but requires a server process.

So I started writing moteDB. 100% Rust, embedded, multimodal. Vectors, time-series, and key-value in one engine, with sub-millisecond reads on a Raspberry Pi 4.

The Hard Part Wasn't Speed

Getting to sub-millisecond reads was actually straightforward. A well-tuned LSM tree with memory-mapped files gets you there on any modern ARM chip. The hard part was temporal consistency.

When a robot writes sensor data, there are two timestamps that matter:

• observed_at: when the sensor actually fired
• ingested_at: when the database received the record

On a Raspberry Pi with DMA-backed SPI reads, that gap is 40-80ms. If your temporal queries use ingested_at, your dead-reckoning estimate drifts. At 1.5 m/s (walking speed), 80ms of timestamp error equals 12cm of position error. Add that up over a 60-second navigation run and you're 7 meters off.

The fix sounds simple: store both timestamps, query on observed_at. But that means every consumer of the data needs to know which timestamp to use, and the write path needs to preserve the original sensor timestamp through the entire pipeline. That's easy to get wrong, and the bugs are silent. Everything works until you try to close the loop and the robot is in the wrong place.

I found this bug three months in. The rover kept drifting right in long corridors. I checked the IMU calibration, the wheel encoders, the motor controller. All fine. The problem was that my query layer was using ingested_at for temporal ordering, and the SPI DMA buffer was adding 60ms of latency between observation and ingestion. The dead-reckoning code was integrating positions in the wrong order, and the error accumulated.

What I'd Tell Someone Starting This Today

Don't use an LLM for real-time memory. Use it for reasoning over memory, not for storing it. The distinction matters more than it sounds. An LLM is great at "given these observations, what should I do next?" It's terrible at "store this sensor reading at exactly this timestamp and retrieve it in under 5ms."

Measure your timestamp latency. If you don't know the gap between observed_at and ingested_at on your hardware, you have a bug you haven't found yet. This is true even if you're not doing dead reckoning. Any temporal query over sensor data assumes the timestamps mean what you think they mean.

Keep your memory local. The offline argument isn't just about privacy. It's about latency, determinism, and cost. Three problems that don't get better with a bigger model. A 400B parameter model running in a data center will always be slower than a 50MB embedded database running on your device. That's physics, not engineering.

moteDB is at cargo add motedb if you want to try it. It's not perfect, but it runs at 20Hz on a Pi 4 and doesn't call any LLMs.