DEV Community: Vectorlink Labs

We're Probably Caching the Wrong Things in AI

Vectorlink Labs — Wed, 08 Jul 2026 13:55:06 +0000

Most AI caching today focuses on things like:

Embeddings
Retrieved documents
Prompt templates
Final responses

All of these are valuable.

But they have one thing in common.

They optimize around the reasoning process.

The model still has to read the same context, connect the same ideas, and perform much of the same reasoning on every request.

It reminds me of the early evolution of web infrastructure.

At first, servers recomputed everything.

Eventually, we realized that the computation itself often had value—and caching it became one of the biggest performance improvements.

I wonder if AI infrastructure is heading in a similar direction.

Instead of asking:

"What output should we cache?"

Maybe the better question is:

"What understanding is worth preserving?"

Imagine reusing synthesized understanding instead of repeatedly reconstructing it from the same documents.

That's the direction we've been exploring with Coalent - treating context as something that can be intelligently reused rather than rebuilt from scratch every time.

If you're curious, Coalent is live : https://coalent.ai

Curious how others are thinking about caching beyond embeddings and response caching. What approaches have you found promising?

Token Costs Aren't the Only Cost

Vectorlink Labs — Tue, 07 Jul 2026 13:10:12 +0000

Whenever people talk about production AI, the conversation usually starts with token costs.

And that's fair.

Inference isn't free.

But I've started wondering if we're optimizing the wrong thing.

The real cost isn't always the tokens.

It's the repeated reasoning.

Think about a typical enterprise AI assistant.

For every request, it reads:

Internal documentation
Product manuals
Company policies
API documentation
Knowledge base articles

Most of that information changes slowly.

Yet every request asks the model to reconstruct almost the same understanding from scratch.

It's like hiring a consultant every morning and spending the first hour explaining your business before asking today's question.

The answer is new.

The understanding usually isn't.

That made me think: instead of only optimizing retrieval or reducing token counts, should we also optimize how understanding itself is reused?

This question is actually what led us to build Coalent - an open source context engineering library focused on reusing synthesized context rather than repeatedly reconstructing it from the same documents.

The goal isn't to cache answers.

It's to preserve the expensive understanding that sits between retrieval and generation, while keeping answers grounded in the latest source data.

If you're curious, Coalent is live : https://coalent.ai

I'm curious how others are approaching this.

Are you optimizing token usage, caching retrieved chunks, reusing synthesized context or taking a completely different approach?

Stop Caching LLM Responses. Cache the Thinking Instead.

Vectorlink Labs — Mon, 06 Jul 2026 12:53:37 +0000

One of the biggest surprises I had while working on RAG systems wasn't retrieval.

It was what happened after retrieval.

Most conversations around RAG optimization focus on:

Better embeddings
Better rerankers
Better chunking
Better vector databases

All important.

But here's something I kept noticing.

Every query forces the model to reconstruct the same understanding from scratch.

Imagine your pipeline looks like this:

User Question
      ↓
Retrieve Documents
      ↓
LLM reads everything
      ↓
Builds understanding
      ↓
Generates answer

Even if another user asks a nearly identical question five minutes later, the model repeats almost all of that expensive reasoning.

Not because it has to.

Because we've mostly been caching outputs, not understanding.

The expensive part isn't always generation

Generation usually gets the attention because it's visible.

The hidden cost is rebuilding context.

The model has to:

Connect documents
Identify relationships
Discard irrelevant facts
Resolve contradictions
Build a coherent understanding before it can answer

That work gets thrown away after every request.

What if we cached the synthesized context?

Instead of caching the final answer, cache the structured understanding that the model already built.

That allows future requests to reuse reasoning while still generating fresh answers from up-to-date data.

The benefit is twofold:

Fewer tokens
Lower latency

without serving stale responses.

This also changes how we think about RAG

Instead of optimizing only retrieval, we start optimizing context construction itself.

Retrieval finds information.

Context construction creates understanding.

Those aren't the same problem.

This line of thinking is what eventually led us to build Coalent.

Coalent is an open-source context engineering library that focuses on reusing synthesized context instead of repeatedly reconstructing it from the same retrieved documents.

Rather than caching answers, the goal is to preserve the expensive understanding that happens between retrieval and generation—while still allowing responses to stay grounded in the latest source data.

As AI applications become more agentic, I think this distinction will matter more and more.

I'm curious how others are approaching this.

Are you caching answers, retrieved chunks, synthesized context, or something else entirely?

If you're interested in exploring this approach,
Coalent is open source: https://github.com/Vectorlink-Labs/coalent

I'd genuinely love to hear feedback from anyone experimenting with context engineering, RAG optimization or AI agent memory. Different perspectives are always valuable and I'm curious how others are tackling the same challenges.

Stale RAG vs. expensive RAG: how to cache RAG context without serving outdated answers

Vectorlink Labs — Wed, 01 Jul 2026 12:31:04 +0000

If you run a RAG system in production, you eventually hit a dilemma that has nothing to do with your model and everything to do with your cache.

Cache the answers to save tokens and latency, and one day a source document changes — but your cache keeps cheerfully serving the answer it built from the old document. Nobody gets an error. The number is just quietly wrong.

Cache nothing, and every single call re-retrieves the same chunks, re-reads them, and re-pays the full context bill to rebuild an understanding you already built five minutes ago for a nearly identical question.

Stale or expensive. Most teams pick "expensive" because at least it's correct, then bolt on a TTL and hope. This post is about why the TTL doesn't save you, and about two specific, mechanical fixes that let you cache RAG context and stay fresh. I maintain an open-source library called Coalent that implements both, so I'll use it for the runnable examples — but the two ideas are portable and worth stealing even if you never pip install anything.

Failure mode 1: the stale RAG cache (and why a TTL won't save you)

Here's the standard "answer cache" sitting in front of retrieval:

answer = cache.get(query)
if answer is None:
    chunks = retriever.retrieve(query)
    answer = llm.synthesize(query, chunks)
    cache.set(query, answer, ttl=3600)
return answer

This works until billing.md changes. The refund window goes from 30 days to 14. Your cache has an answer keyed on "what is our refund policy?" that says 30, and it will keep saying 30 for up to an hour — or forever, if the same question keeps refreshing a TTL that never expires under load.

The reason this is hard is that the cache key (the query) has no relationship to the thing that changed (the source). You cached an answer; you threw away the fact that this particular answer was derived from billing.md. So when billing.md changes, you have no way to find the answers that depended on it.

The TTL is a confession that you can't answer the question "which cached units are now wrong?", so you guess a time and blow everything away on a timer. Too short and you've rebuilt a perfectly good cache for nothing. Too long and you serve stale data. There is no good number, because freshness is an event, not a duration.

The fix: invalidate by provenance, not by clock

The fix is to record, for every cached unit, which sources it cited — its provenance — and to invalidate by set membership when a source changes. No source change, no invalidation. A source changes, and only the units that actually cited it get marked dirty.

That requires a cache that keeps the derivation link instead of discarding it. In Coalent, the cached unit retains the source spans it was built from, so a change event is a reverse-index lookup:

from coalent import SemanticCache, LLMSynthesizer, OpenAIProvider, OpenAIEmbedder

cache = SemanticCache(
    retriever,                               # bring your own: vector DB, GraphRAG, tools
    LLMSynthesizer(OpenAIProvider(), model="gpt-4o-mini"),
    embedder=OpenAIEmbedder(),
)

r = cache.get("what is our refund policy?")
# r.context["understanding"]  -> summary + claims + facts, just the decision-relevant slice
# r.cache_hit                 -> served from cache or freshly built

# A source changed. This is surgical, not a flush:
result = cache.source_changed("billing.md", text="...refunds within 14 days...")
# result.dirtied            -> units that actually cited billing.md AND whose content changed
# result.skipped_unchanged  -> matched the source but the bytes were identical -> left fresh

Two details that matter in practice:

Skip the no-ops. source_changed hashes the new text and compares it to the hash stored on each unit's provenance. If a CI job rewrites billing.md byte-for-byte, nothing is invalidated. You don't pay to rebuild understanding that didn't change.
Lazy rebuild. A dirtied unit isn't rebuilt immediately. It's marked stale and re-materializes on the next read that asks for it. Sources that change but are never queried again cost you nothing.

The honest caveat: invalidation today is per-artifact. If you chunk one giant doc into 200 pieces that all share a single artifact_id, a change anywhere in that doc dirties everything derived from it — more rebuilding than strictly necessary. Span-level granularity (invalidate only the units whose specific cited span changed) is the planned improvement. If your sources map roughly one-document-to-one-id, you're already in the good case.

Failure mode 2: the lossy-summary gap (a cache hit worse than plain RAG)

Now the other direction. To cut tokens, you cache a summary of the retrieved context instead of the raw chunks. Great — until a later query needs a detail the summary dropped.

You retrieved a dense billing doc, summarized it to "Refunds are available within the policy window, with some exceptions," and cached that. Then someone asks: "Can I get a refund on a gift-card purchase?" The summary is on-topic, so it "hits" — and answers from a blob that silently dropped the gift-card exception. Your cache just returned an answer worse than plain RAG would have, because plain RAG still had the raw chunk with the exception in it. The cache hit, and the hit was a downgrade.

This is the subtle one. Staleness is at least detectable in principle. The lossy-summary gap looks like a perfectly healthy cache hit. Your hit rate goes up. Your accuracy goes down. Nobody notices until a customer does.

The fix: keep the raw, and add a semantic coverage floor

Two parts.

First, never throw away the raw evidence. Cache the cheap summarized understanding for the common case, but keep the chunks attached to the unit so the detail is always one hop away.

Second — and this is the part most "semantic cache" implementations skip — measure whether the hit actually covers the query before you trust it. A cache hit means "a unit on this topic exists," not "this unit answers this question." Those are different claims, and conflating them is exactly the gift-card bug.

Coalent scores coverage per claim. It embeds each claim/fact in a unit separately and takes the max cosine similarity against the query. If the query's best match against any stored claim is below a floor, the hit escalates: it pulls fresh raw evidence for that specific query and answers from it — still counted as a hit, but no longer answering from a summary that's missing the point.

r = cache.get("can I get a refund on a gift-card purchase?")

r.coverage    # 0..1: how well the unit's best claim addresses THIS query
r.escalated   # True -> coverage was under the floor, so it pulled fresh raw
r.context["raw"]  # the raw evidence, present once escalated

One thing to be precise about: escalation only kicks in on a warm hit — a unit on this topic already exists but doesn't cover this specific query. A cold, first-ever query is just a normal miss that retrieves and builds fresh, so it gets the full evidence anyway. Escalation is for the dangerous in-between: when the cache has something on-topic but incomplete, and would otherwise answer from it.

Why per-claim and not one similarity score over the whole summary? Because a single embedding of the whole unit is a centroid. An on-topic-but-uncovered query ("gift-card refund") sits close to the centroid of a refund unit and looks "covered" even though the specific fact is missing. Scoring each claim separately is what catches the gap. And because it's semantic, "nicked card" still matches a claim about a "stolen card" — a keyword gate would miss that. (That semantic behaviour assumes a real embedder — OpenAIEmbedder or a local model; on the zero-dependency HashingEmbedder default, matching degrades to keyword overlap, so use a real embedder in production.)

The cheap cosine check decides the clear cases for free. For the ambiguous middle band, you can opt into a stricter coverage_scorer (a cross-encoder or LLM-entailment check) that only fires on the genuinely uncertain hits — containment-grade accuracy without paying for it on every query.

The metric that ties it together is the escalation rate:

cache.stats()["escalation_rate"]   # fraction of hits that had to fall back to raw

If that climbs, your cached understanding is systematically under-covering real queries — a signal to deepen what you cache (or lower the floor). It's also your honest readout of when the cache has quietly degraded into doing plain RAG. You want to see that, not discover it from a support ticket.

What this actually buys you (with the caveats)

I benchmarked this against feeding full retrieved context to the model on every call, with an independent gpt-4o judge, on number-dense documents, with a source change midway through the run to test invalidation. Results:

~66% fewer context tokens per read typically, up to ~75% on large/dense docs.
Accuracy matched full-context RAG: 100% on gpt-4o-mini. On a stricter gpt-4o grader, ~93% vs full-context's 96% in that run — Coalent occasionally drops a secondary qualifier from an answer. That's a real gap, not a rounding error, and worth knowing before you ship.

So: roughly two-thirds the context tokens per read at near-parity accuracy, and — the part the token number doesn't show — it stays correct when a source changes, because invalidation is provenance-scoped instead of a TTL guess.

The takeaway you can use anywhere

You don't need my library to apply the two ideas. The structural lessons are:

Don't key a cache on the query alone — record what each cached unit was derived from. Provenance is what lets you invalidate by event (a real source change) instead of by clock (a hopeful guess). It also turns invalidation into set membership, which has zero false negatives over the provenance you recorded.
A cache hit is a hypothesis, not a guarantee. Keep the raw evidence, and put a coverage check under every hit so an on-topic-but-incomplete unit escalates instead of confidently answering from less than RAG had.

Get those two right and the stale-vs-expensive dilemma mostly dissolves: you cache aggressively for reuse, invalidate surgically for freshness, and fall back to raw exactly when — and only when — the cache can't actually answer the question.

Coalent is Apache-2.0, pure Python, zero required dependencies, if you want a reference implementation to read or build on: github.com/Vectorlink-Labs/coalent.

I'm Nisarg Pujara, and I maintain Coalent — so take the benchmark numbers as an invitation to run your own on your own corpus, which is the only benchmark that should ever convince you.

Resources

Interested in trying Coalent or learning more?

🌐 Website: https://coalent.ai
⭐ GitHub: https://github.com/Vectorlink-Labs/coalent
📦 PyPI: https://pypi.org/project/coalent/
📚 Documentation: https://coalent.ai/docs