---
title: "KV-Cache Poisoning: Securing Multi-Tenant LLM APIs"
published: true
description: "Shared KV-cache pools create cross-tenant leakage risks in LLM APIs. Enforce isolation with cache namespacing, eviction policies, and gRPC interceptors."
tags: security, api, architecture, cloud
canonical_url: https://mvpfactory.co/blog/kv-cache-poisoning-multi-tenant-llm-apis
---
## What We Will Build
In this workshop, you will learn to secure a multi-tenant LLM inference service against KV-cache poisoning and cross-tenant context leakage. By the end, you will have three concrete defenses in place: tenant-namespaced cache keys, per-tenant eviction policies, and a gRPC interceptor that enforces isolation across your entire request pipeline — without adding meaningful latency.
## Prerequisites
- A running multi-tenant LLM inference service on a shared inference cluster
- Redis for semantic/KV-cache storage
- Kotlin or Python for gateway and interceptor code
- Basic familiarity with gRPC and cache eviction strategies
---
## The Problem Nobody Talks About Until It Is Too Late
The KV-cache — the key-value store that enables efficient attention computation by reusing prior token representations — is safe by construction in single-tenant deployments. In a shared, multi-tenant inference cluster, it becomes a liability.
Here is the gotcha that will save you hours: without explicit namespacing, a cached prefix from Tenant A can be matched and served to Tenant B when their prompts share a common structure — a shared system prompt template, for example. A malicious actor can go further, crafting prompts that populate the cache with poisoned completions that influence subsequent outputs for other tenants. Semantic similarity-based caches compound this: a tenant can retrieve completions generated for a semantically similar but legally distinct query belonging to a different tenant.
These are the leakage vectors that matter in practice:
| Vector | Mechanism | Severity |
|---|---|---|
| Exact prefix cache hit | Shared system prompt matched across tenants | High |
| Semantic cache collision | Embedding proximity triggers wrong cache entry | High |
| Prompt injection via cache | Malicious completion stored, later retrieved | Critical |
| Eviction side-channel | Timing analysis of hit/miss to infer tenant activity | Medium |
---
## Step 1 — Namespace Every Cache Key
Let me show you a pattern I use in every project. The minimum viable fix is namespacing every cache key with a cryptographically derived tenant identifier. This is non-negotiable — everything else builds on it.
python
import hashlib
def build_cache_key(tenant_id: str, prompt: str, model_id: str) -> str:
tenant_hash = hashlib.sha256(tenant_id.encode()).hexdigest()[:16]
prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
return f"{tenant_hash}:{model_id}:{prompt_hash}"
For semantic caches, this means maintaining a **separate embedding index per tenant** — not a shared index with a tenant filter. A shared index with a `WHERE tenant_id = X` clause is still vulnerable to timing attacks and implementation bugs. Partition at the storage layer, not the query layer.
---
## Step 2 — Enforce Per-Tenant Eviction Policies
Eviction is where most implementations break down. A global LRU cache evicts based on system-wide recency, which creates two problems: a high-traffic tenant can evict a low-traffic tenant's entries (a soft denial-of-service vector), and cached completions may persist beyond a tenant's contractual data retention window.
Here is the minimal setup to get this working:
yaml
cache_policy:
tenant_a:
max_entries: 10000
ttl_seconds: 3600
eviction: lru
tenant_b:
max_entries: 5000
ttl_seconds: 900 # stricter retention SLA
eviction: lru
In Redis, this maps to logical keyspace partitions with per-prefix `SCAN`-based TTL enforcement, or dedicated keyspaces with eviction policies set per-tenant via `CONFIG SET`.
---
## Step 3 — The gRPC Interceptor Pattern
The docs do not mention this, but the cleanest enforcement mechanism is a gRPC server interceptor. It handles tenant identity propagation, cache key construction, and policy enforcement in a single auditable layer:
kotlin
class TenantIsolationInterceptor : ServerInterceptor {
override fun interceptCall(
call: ServerCall,
headers: Metadata,
next: ServerCallHandler
): ServerCall.Listener {
val tenantId = headers.get(TENANT_ID_KEY)
?: throw StatusRuntimeException(Status.UNAUTHENTICATED)
val context = Context.current()
.withValue(TENANT_CONTEXT_KEY, TenantContext(tenantId))
return Contexts.interceptCall(context, call, headers, next)
}
}
The tenant context flows through the entire request pipeline — cache key construction, eviction policy lookup, audit logging — without requiring individual services to re-authenticate. The latency overhead for this interceptor chain runs under 1ms at p99, negligible against inference latency.
---
## Step 4 — Sanitize at the Gateway, Not Inside the Model
Before a request reaches the inference server, route it through a sanitization layer at the gateway. Three reasons this is the right place for it:
1. It is centralized — no per-service duplication of sanitization logic.
2. It runs before any caching, so poisoned inputs never populate the cache.
3. It can be updated independently of model deployment cycles.
Minimum gateway sanitization checklist:
- Strip sequences attempting to override system prompt context (`Ignore all previous instructions...`)
- Validate that user-injected content cannot escape its designated role boundary
- Enforce per-tenant prompt length limits
- Log and alert on structures matching known injection signatures
---
## Gotchas
**Semantic index partitioning is not optional.** A shared embedding index with a tenant filter at query time is still vulnerable to timing-based side channels. Partition at the storage layer, not the query layer — this is the distinction that matters.
**TTL must map to data agreements, not convenience.** Cache TTL policies must reflect each tenant's data retention SLA. This is a compliance obligation. Map cache eviction directly to tenant contracts and enforce it at the infrastructure layer before your legal team asks why you did not.
**Namespace on day one.** Cache keys, semantic indexes, and eviction quotas must all be scoped to a cryptographically derived tenant identifier before you onboard your second customer. This is not a post-incident retrofit. It is a day-one architecture requirement.
---
## Conclusion
KV-cache isolation in multi-tenant LLM deployments is not a performance concern — it is a security and compliance requirement. Cache key namespacing gives you the baseline. Per-tenant eviction policies enforce retention SLAs. The gRPC interceptor pattern gives you zero-latency, auditable enforcement across your entire pipeline. Sanitize at the gateway, not inside the model. Build this before an incident forces the issue.
---
*Resources: [Redis CONFIG SET](https://redis.io/commands/config-set/) · [gRPC Kotlin interceptors](https://grpc.io/docs/languages/kotlin/)*
Top comments (0)