GitHub Repository: https://github.com/13manmayarai-hash/omnicache-proxy (MIT Licensed)
What My Project Does
OmniCache is an open-source, zero-dependency reverse proxy and semantic caching engine written in pure Python. It sits between client applications and model backends (OpenAI, Anthropic, Google Gemini, local vLLM / Ollama) to serve cached responses in under 1 millisecond.
Key Technical Components:
- Sub-Millisecond Vector Caching (<0.8ms): In-memory 512-d feature projection embedder executing in CPU cache without remote embedding overhead.
- Dynamic Intent Gating: Adaptive similarity thresholds (Code: 0.98, JSON Schema: 1.0, Creative: Auto-Bypass, FAQ: 0.92) to prevent syntax and logic hallucinations.
- Agent Tool-Loop Accelerator: Caches idempotent tool executions (read_file, git status, grep) for coding agents like Claude Code and Cursor, reducing agent turnaround from 15s to 350ms.
- Adaptive Cost Cascade: Evaluates prompt complexity in <0.2ms to route simple classification queries to high-speed economy models.
- Multi-Modal Vision Cache: 64-bit Perceptual Hashing (dHash) for visual screenshot and document deduplication.
- Zero-Knowledge Privacy Vault: Reversible tokenized masking of sensitive credentials and identifiers prior to upstream forwarding.
- Token Jitter SSE Replayer: Poisson-distributed streaming playback (~65 tok/s, <10ms TTFT) for natural streaming UI rendering.
Target Audience
OmniCache is designed for production infrastructure and local developer workflows:
- AI Infrastructure Engineers: Teams seeking to optimize upstream API token volume without modifying core application code.
- Autonomous Coding Agents: Developers running multi-turn reasoning loops (Claude Code, Cursor, AutoGen) with repetitive file inspection cycles.
- Real-Time Voice Pipelines: Voice agent architectures (LiveKit, Pipecat, Vapi) where <1ms LLM lookup latency eliminates telephone conversational delay.
- Self-Hosted AI Deployments: Environments requiring local zero-network data persistence.
Comparison
- Versus LiteLLM / Portkey: Those tools require external vector databases (Redis, Qdrant) which introduce 30ms to 80ms network latency. OmniCache executes local in-memory cosine matching in <0.8ms with zero external services.
- Versus Native Provider Prompt Caching: Provider prompt caching only covers exact input prefixes over 1024 tokens and does not optimize output generation latency. OmniCache serves full cached completions in under 1ms.
- Versus Redis Semantic Cache: Traditional semantic caches apply a static global threshold that can corrupt structured JSON or code syntax. OmniCache applies dynamic intent gating based on payload semantics.
Setup and Usage
git clone https://github.com/13manmayarai-hash/omnicache-proxy.git
cd omnicache-proxy
pip install starlette uvicorn httpx
python3 main.py
Integration via standard client SDK:
from openai import OpenAI
client = OpenAI(
api_key="your-backend-api-key",
base_url="http://localhost:8000/v1"
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Explain binary search trees"}]
)
print(response.choices[0].message.content)
The web metrics dashboard runs locally at http://localhost:8000/dashboard.
Licensed under MIT. Technical feedback, benchmarks, and pull requests are welcome.
Top comments (1)
The intent-based thresholds are a really interesting touch, especially for code and structured JSON where a “close enough” cache hit can cause real problems. Curious to see how the latency holds up under larger cache sizes.