Building OmniCache: A sub-millisecond Python proxy that cuts LLM bills by 50%

Rajiv Prasad — Thu, 03 Sep 2026 02:49:59 +0000

GitHub Repository: https://github.com/13manmayarai-hash/omnicache-proxy (MIT Licensed)

What My Project Does

OmniCache is an open-source, zero-dependency reverse proxy and semantic caching engine written in pure Python. It sits between client applications and model backends (OpenAI, Anthropic, Google Gemini, local vLLM / Ollama) to serve cached responses in under 1 millisecond.

Key Technical Components:

Sub-Millisecond Vector Caching (<0.8ms): In-memory 512-d feature projection embedder executing in CPU cache without remote embedding overhead.
Dynamic Intent Gating: Adaptive similarity thresholds (Code: 0.98, JSON Schema: 1.0, Creative: Auto-Bypass, FAQ: 0.92) to prevent syntax and logic hallucinations.
Agent Tool-Loop Accelerator: Caches idempotent tool executions (read_file, git status, grep) for coding agents like Claude Code and Cursor, reducing agent turnaround from 15s to 350ms.
Adaptive Cost Cascade: Evaluates prompt complexity in <0.2ms to route simple classification queries to high-speed economy models.
Multi-Modal Vision Cache: 64-bit Perceptual Hashing (dHash) for visual screenshot and document deduplication.
Zero-Knowledge Privacy Vault: Reversible tokenized masking of sensitive credentials and identifiers prior to upstream forwarding.
Token Jitter SSE Replayer: Poisson-distributed streaming playback (~65 tok/s, <10ms TTFT) for natural streaming UI rendering.

Target Audience

OmniCache is designed for production infrastructure and local developer workflows:

AI Infrastructure Engineers: Teams seeking to optimize upstream API token volume without modifying core application code.
Autonomous Coding Agents: Developers running multi-turn reasoning loops (Claude Code, Cursor, AutoGen) with repetitive file inspection cycles.
Real-Time Voice Pipelines: Voice agent architectures (LiveKit, Pipecat, Vapi) where <1ms LLM lookup latency eliminates telephone conversational delay.
Self-Hosted AI Deployments: Environments requiring local zero-network data persistence.

Comparison

Versus LiteLLM / Portkey: Those tools require external vector databases (Redis, Qdrant) which introduce 30ms to 80ms network latency. OmniCache executes local in-memory cosine matching in <0.8ms with zero external services.
Versus Native Provider Prompt Caching: Provider prompt caching only covers exact input prefixes over 1024 tokens and does not optimize output generation latency. OmniCache serves full cached completions in under 1ms.
Versus Redis Semantic Cache: Traditional semantic caches apply a static global threshold that can corrupt structured JSON or code syntax. OmniCache applies dynamic intent gating based on payload semantics.

Setup and Usage

git clone https://github.com/13manmayarai-hash/omnicache-proxy.git
cd omnicache-proxy
pip install starlette uvicorn httpx
python3 main.py

Integration via standard client SDK:

from openai import OpenAI

client = OpenAI(
    api_key="your-backend-api-key",
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain binary search trees"}]
)
print(response.choices[0].message.content)

The web metrics dashboard runs locally at http://localhost:8000/dashboard.

Licensed under MIT. Technical feedback, benchmarks, and pull requests are welcome.

DEV Community: Rajiv Prasad

Building OmniCache: A sub-millisecond Python proxy that cuts LLM bills by 50%

What My Project Does

Target Audience

Comparison

Setup and Usage