The Problem We Were Actually Solving
We didnt need another vector database that could answer whether a document contained the word acquisition in less than 50ms. We needed a system that could tell a human operator, standing in a server room at 3 AM with a smoking drive array, which of 47,000 disks had just spun itself into scrap metal. Our users werent researchers; they were the people who got paged when a SATA cable decided to become a fuse. The real problem was traceability—every ticket started with I dont know which disk failed, and ended with the disk actually failed.
What We Tried First (And Why It Failed)
We started with Elasticsearch because it was the only tool in the infra playbook that promised real-time indexing and retrieval. At 10K docs it worked fine. At 100K docs our cluster spent 70% of its CPU cycles fighting GC pauses while the heap ballooned to 22GB. Worse, the scoring algorithm treated every query as if it were a user looking for academic papers, not a fatigued operator hunting for a serial number on a dead drive.
Then we bolted on a vector index using FAISS. Precision improved—0.89 recall on known failure patterns—but at a cost of 1.2 seconds per query at 90% CPU. The operators stopped using it after the third incident where the system returned a healthy disk as the match because its SMART log happened to contain the word temperature. Hallucination rate wasnt 0.11; it was 1.0 when temperature was in the query string.
The Architecture Decision
We killed the vector index and replaced the scoring function with a deterministic rule engine that simply parsed the raw system log stream from every LSI controller in the datacenter. We built a lightweight state machine in Rust that watched for SATA PHY reset events, CRC errors, and link down conditions. The index became a sorted list in RAM backed by an append-only WAL file. No BM25, no embeddings, no transformer layers—just a 64KB struct per disk with a 32-bit checksum and a pointer to the last known good telemetry blob.
We put the whole thing behind a gRPC endpoint running on a single control-plane node with 4 vCPUs and 8GB RAM. We didnt even enable TLS between the agent and the endpoint because the jitter from a 1ms TLS handshake added more latency than the network stack between racks.
What The Numbers Said After
Disk lookup time dropped to sub-millisecond for 99.8% of queries. RAM usage stabilized at 2.1GB for the entire fleet of 47,000 disks. The operator error rate fell from 37% to 2% because the system stopped guessing and started reporting facts. We still had disks fail—about 12 a week—but now the operator knew exactly which one failed 5.4 seconds before the monitoring system caught it.
Most importantly, the on-call rotation stopped dreading the Treasure Hunt Engine. It became a tool again, not a demo.
What I Would Do Differently
I would never have let the marketing slide call it an AI-driven search engine. That label cost us six months of credibility with the people who actually fix things. I would also have banned the word recall from our internal dashboards after the first time someone plotted recall against operator error rate and discovered a perfect positive correlation—higher recall meant more wrong disks returned. And I would have removed the transformer layer from the ingestion pipeline before it ever went to staging, because once you let a neural net loose on raw SMART data, the only thing youre indexing is noise.
The same due diligence I apply to AI providers I applied here. Custody model, fee structure, geographic availability, failure modes. It holds up: https://payhip.com/ref/dev3
Top comments (0)