Building OmniCache: A sub-millisecond Python proxy that cuts LLM bills by 50%

GitHub Repository: https://github.com/13manmayarai-hash/omnicache-proxy (MIT Licensed)

What My Project Does

OmniCache is an open-source, zero-dependency reverse proxy and semantic caching engine written in pure Python. It sits between client applications and model backends (OpenAI, Anthropic, Google Gemini, local vLLM / Ollama) to serve cached responses in under 1 millisecond.

Key Technical Components:

• Sub-Millisecond Vector Caching (<0.8ms): In-memory 512-d feature projection embedder executing in CPU cache without remote embedding overhead.
• Dynamic Intent Gating: Adaptive similarity thresholds (Code: 0.98, JSON Schema: 1.0, Creative: Auto-Bypass, FAQ: 0.92) to prevent syntax and logic hallucinations.
• Agent Tool-Loop Accelerator: Caches idempotent tool executions (read_file, git status, grep) for coding agents like Claude Code and Cursor, reducing agent turnaround from 15s to 350ms.
• Adaptive Cost Cascade: Evaluates prompt complexity in <0.2ms to route simple classification queries to high-speed economy models.
• Multi-Modal Vision Cache: 64-bit Perceptual Hashing (dHash) for visual screenshot and document deduplication.
• Zero-Knowledge Privacy Vault: Reversible tokenized masking of sensitive credentials and identifiers prior to upstream forwarding.
• Token Jitter SSE Replayer: Poisson-distributed streaming playback (~65 tok/s, <10ms TTFT) for natural streaming UI rendering.

Target Audience

OmniCache is designed for production infrastructure and local developer workflows:

• AI Infrastructure Engineers: Teams seeking to optimize upstream API token volume without modifying core application code.
• Autonomous Coding Agents: Developers running multi-turn reasoning loops (Claude Code, Cursor, AutoGen) with repetitive file inspection cycles.
• Real-Time Voice Pipelines: Voice agent architectures (LiveKit, Pipecat, Vapi) where <1ms LLM lookup latency eliminates telephone conversational delay.
• Self-Hosted AI Deployments: Environments requiring local zero-network data persistence.

Comparison

• Versus LiteLLM / Portkey: Those tools require external vector databases (Redis, Qdrant) which introduce 30ms to 80ms network latency. OmniCache executes local in-memory cosine matching in <0.8ms with zero external services.
• Versus Native Provider Prompt Caching: Provider prompt caching only covers exact input prefixes over 1024 tokens and does not optimize output generation latency. OmniCache serves full cached completions in under 1ms.
• Versus Redis Semantic Cache: Traditional semantic caches apply a static global threshold that can corrupt structured JSON or code syntax. OmniCache applies dynamic intent gating based on payload semantics.

Setup and Usage

git clone https://github.com/13manmayarai-hash/omnicache-proxy.git
cd omnicache-proxy
pip install starlette uvicorn httpx
python3 main.py

Integration via standard client SDK:

from openai import OpenAI

client = OpenAI(
    api_key="your-backend-api-key",
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain binary search trees"}]
)
print(response.choices[0].message.content)

The web metrics dashboard runs locally at http://localhost:8000/dashboard.

Licensed under MIT. Technical feedback, benchmarks, and pull requests are welcome.

Comments

[Bhavin Sheth]

The intent-based thresholds are a really interesting touch, especially for code and structured JSON where a “close enough” cache hit can cause real problems. Curious to see how the latency holds up under larger cache sizes.

[Rajiv Prasad]

Thanks for the great question! You've touched on two of the hardest trade-offs in semantic caching:

1. Precision & Safety on Code / JSON: To prevent subtle code regressions or schema mismatches, we don't just rely on raw cosine similarity:
2. Schema & Tool Hashing: We compute cryptographic hashes of any requested tools/functions (tools_hash) and JSON schemas (schema_hash). If the requested output shape or tool definitions differ, it triggers an instant cache MISS regardless of prompt similarity.

3. Dynamic Intent Strictness: When code syntax or structured data intent is detected, the similarity threshold tightens to 0.985+ (or falls back to exact L1 hashing), whereas creative/high-temperature queries bypass L2 entirely.

4. Latency Under Larger Cache Sizes (10k–100k+ entries):
   Naive semantic caches degrade quickly because linear cosine scanning is O(N). In OmniCache v2.5+:

5. Multi-Table Hyperplane LSH & FAISS HNSW: We partition vector memory with deterministic Locality-Sensitive Hashing (LSH) and FAISS indexers. At scale, this reduces retrieval to O(1) candidate bucket lookups.

6. Tenant-Scoped Partitioning: Indices are partitioned by tenant/workspace, keeping the active search space compact.

7. Benchmark Latencies: In our benchmarks with 10,000+ cached vectors, intent lookup and candidate pruning stay sub-millisecond (~0.05ms in-memory, <1ms via Redis).

If you'd like to test your own workload latency, you can run omnicache benchmark directly from the CLI!

[Suraj Suradkar]

The agent tool-loop caching is the part I find most interesting. For things like read_file or git status, the hard problem isn't cache speed but knowing when the result is still valid. Repo state probably needs to be part of the cache key, otherwise stale context can quietly steer the next tool call.