Sulci 1.0: a context-aware semantic cache for agents, and the number most caches don’t publish

Stable library, hosted tier generally available, every benchmark figure reproducible from the public repo

Today Sulci is 1.0, and the hosted tier is generally available.

Sulci is an open-source, context-aware semantic cache for LLM apps and AI agents. When a new query means the same thing as a past one, you get the cached answer instead of another model call, even when the words are different. It's Apache 2.0, it runs on infrastructure you already have, and it's three lines around an existing call.

Why agents need this

We instrumented an agent workload: 50 sessions, 200 LLM dispatches each, the ReAct-ish mix you get when a tool-using agent works a task.

• With a cold, empty cache, 33.4% [30.00–35.00%] of dispatches could be served from cache.
• Warm, within the same session, 99.7% [99.69–99.81%].
• The median session sent 2 of its 200 dispatches to the model. The p95 session sent 53 [46–64], mostly cold ones, and we publish that too.

None of this means agents are broken. Loops re-derive the same intermediate facts, planners re-summarise the same context at every step, retries re-ask the failing question slightly rephrased. The dispatch stream is mostly near duplicates, and the bill scales with the redundancy, not with the work.

The number most caches don't publish

A semantic cache has one failure people remember: a confident wrong answer, served because two questions looked alike. A cache tuned only for hit rate reaches 100% by answering everything.

So our benchmark corpus is built so that 39% of test queries have no correct answer cached, and we publish what the cache does with them. Four seeded draws, threshold 0.85, the shipped MiniLM embedder:

	measured	range across draws
Repeat questions served from cache	99.9%	[99.85–99.95%]
Correctly declined when it has no answer	98.9%	[98.15–99.38%]
Of cache hits, share that are correct	93.8%	[92.13–94.69%]
Cache hit, p50	87ms	[86.956–87.729ms], quiet M-series machine

87ms is almost entirely the embedding, not the lookup, and it depends on your hardware. Measure it on yours.

Reproduce one draw in a clone of the repo (the benchmark isn't in the PyPI package):

git clone https://github.com/sulci-io/sulci-oss && cd sulci-oss
pip install -e ".[sqlite]"
python benchmark/run.py --use-sulci --fresh --no-sweep --agent --seed 1 --out /tmp/sulci-bench

Then --seed 2, 3 and 42. If you get something different, open an issue with your seed and your machine.

How you use it

pip install "sulci[sqlite]"

from sulci import Cache

cache = Cache(backend="sqlite", threshold=0.85)

response, similarity, context_depth = cache.get("How do I rotate an API key?")
if response is None:
    response = call_your_model(...)
    cache.set("How do I rotate an API key?", response)

threshold= can be overridden on each call, so a strict path and a loose path can share one cache.

It remembers the conversation

Sulci is context-aware. Pass a session_id and each lookup blends the current query with the conversation's recent turns: 70% the query, 30% a decaying history. "How do I fix it?" after a Docker crash and "How do I fix it?" after a billing error search with two different vectors.

We're careful about what that does and doesn't promise. Blending changes where a follow-up searches; it doesn't make a stored follow-up belong to one conversation. On our corpus, on MiniLM, it didn't measurably improve resolution accuracy (more below). So the hosted tier adds two guards:

• Blended lookups have to clear a stricter bar (0.80, against 0.65 for plain lookups).
• A follow-up that only makes sense inside its conversation is never stored as a shared answer. The API says so in its response: stored: false, reason: "context_dependent".

It's on by default there. In the library you opt in with context_window and a session_id, and tune query_weight, context_decay and context_threshold yourself.

Adopt it without rewriting anything

Whatever your agent stack looks like, there's a door that costs a line or less:

You have	You add
LangChain or LangGraph	set_llm_cache(SulciCache(backend="sqlite")): one line, every LLM call and graph node cached
LlamaIndex	wrap your LLM in SulciCacheLLM and pass session_id=
LiteLLM	from sulci.integrations.litellm import install and call it; LiteLLM's 100+ providers are now cached
Any OpenAI or Anthropic client, in any language	pip install "sulci[proxy]", run sulci-proxy and set OPENAI_BASE_URL / ANTHROPIC_BASE_URL. No code change. Hits come back with x-sulci-cache: hit
An agent that can't import Python	pip install "sulci[mcp]" for sulci-mcp, an MCP server (stdio, SSE or streamable HTTP, with a read-only mode)
TypeScript, Go, Java…	npm install @sulci/sulci, or the HTTPS API

The same session_id carries through the library, LlamaIndex, LiteLLM, the proxy (as an x-sulci-session-id header), the MCP server, the TypeScript client and the HTTP API. Storage is SQLite to start, then Chroma, Qdrant, FAISS, Redis (plain Redis, no modules) or Milvus, with the same three lines.

What 1.0 means

The library is declared stable. From 1.0.0, Semantic Versioning holds for the public API in docs/API-SURFACE.md: no breaking change within 1.x, additions in minor releases, fixes in patch releases.

The hosted tier is generally available

Keep your self-hosted backend and pass an API key, and the free dashboard shows hit rates and cost saved per deployment. Only a fixed allowlist of eight operational fields is sent. Your queries, responses and embeddings stay where they are.

Or let us host the cache: Cache(backend="sulci", api_key="sk-sulci-..."). In that mode your query text and cached responses do reach us, because that's what hosting the cache means. The privacy policy spells out both modes.

Pro and Business are generally available — self-serve signup with a 14-day or 50k-request trial, and a one-parameter upgrade from self-hosted. OSS-Connect (10,000 managed requests a month, no card) stays free. Pro is $29/month and Business $149/month. Details at https://sulci.io/pricing.

What we're deliberately not claiming

• No cost-per-query figure and no speedup multiple. Both depend on your model, your token counts and how repetitive your traffic is, and no benchmark of ours predicts that. https://sulci.io/roi does the arithmetic with your numbers instead of ours.
• No context-aware accuracy delta. Sulci blends exponentially decayed prior turns into the lookup vector, and it ships. On the shipped embedder, on our corpus, it showed no measurable gain in resolution accuracy: MiniLM already answers most conversational follow-ups statelessly. Earlier numbers showing a large gain were measured against a TF-IDF baseline with far more headroom. We withdrew them rather than shrinking them.

Try it

• GitHub: https://github.com/sulci-io/sulci-oss
• Docs and a live in-browser demo (no signup): https://sulci.io/demo
• PyPI: https://pypi.org/project/sulci/
• Pricing: https://sulci.io/pricing

Apache 2.0. A U.S. patent application covering aspects of context-aware semantic caching is pending (No. 64/018,452); it is currently assigned to Kathiravan Sengodan. The Apache-2.0 patent grant in the LICENSE file applies to the open-source contribution as licensed thereunder.