aryan p

Posted on Jul 25

CacheGuard

#cybersecurity #llm #monitoring #security

How I Built CacheGuard: Securing GenAI Prompt Caches from Timing Side-Channel Attacks with SigNoz

Introduction

Prompt caching is one of the most impactful optimizations in modern GenAI infrastructure. Services like vLLM, OpenAI's prompt caching, and Anthropic's cached prefixes can reduce Time-To-First-Token (TTFT) by up to 85% and slash API costs dramatically. But there's a dark side that nobody talks about:

What if an attacker could use those speed improvements against you?

That's the problem I set out to solve with CacheGuard — an observability and active defense middleware that detects and neutralizes timing side-channel attacks on LLM prompt caches, built on top of SigNoz for real-time threat visualization.

In this blog, I'll walk you through the security vulnerability, how CacheGuard defends against it, and how SigNoz became the backbone of the entire observability pipeline.

The Problem: Timing Side-Channel Attacks on LLM Caches

When an LLM prompt cache is active, there's a measurable difference in response times:

Cache HIT: ~20ms (the response was already computed and stored)
Cache MISS: ~200ms+ (the LLM has to generate the response from scratch)

This timing gap is the vulnerability. An attacker doesn't need to see the cached data directly. They just need to measure how fast the server responds. If a response comes back suspiciously fast, they know that exact prompt (or a very similar one) was recently asked by another user — potentially leaking sensitive business data, proprietary prompts, or private conversations.

This is called a Timing Side-Channel Attack, and it's a well-documented class of vulnerability in computer security (similar to CPU cache attacks like Spectre and Meltdown, but applied to AI infrastructure).

Why This Matters for Enterprises

Imagine a multi-tenant AI platform where:

Company A asks: "Summarize our Q4 merger strategy with Acme Corp"
An attacker from Company B sends the same prompt and gets a 20ms response
The attacker now knows Company A is planning a merger with Acme Corp — without ever seeing the actual response

This is a real, exploitable vulnerability in any shared LLM caching layer.

The Solution: CacheGuard

CacheGuard is a Python middleware that sits between your application and the LLM API. It works in two phases:

Phase 1: Threat Detection & Observability

CacheGuard intercepts every LLM request using a monkey-patched httpx.post method. For each request, it:

Measures TTFT (Time-To-First-Token) to determine if it was a cache hit or miss
Calculates a Risk Score using statistical analysis of the timing distribution
Runs a Kolmogorov-Smirnov (KS) Test to detect bimodal timing distributions (a telltale sign of systematic cache probing)
Tracks Prefix Entropy using a trie data structure to detect brute-force enumeration attacks (when entropy drops, it means someone is systematically guessing cached prefixes)

All of this data is emitted as OpenTelemetry spans and metrics, which flow directly into SigNoz for visualization.

Phase 2: Active Defense

When CacheGuard detects an attack, it doesn't just alert — it fights back autonomously:

Adaptive Jitter: Injects random delays (50-500ms) into responses to mask the timing difference between cache hits and misses. The attacker can no longer distinguish fast (cached) from slow (uncached) responses.
Cache Bypass: Forces the LLM to regenerate the response from scratch, even if it's cached. This eliminates the timing signal entirely for suspicious requests.
Tenant Isolation: Quarantines the attacker's session completely, assigning them a unique cache salt so they can never access another tenant's cached data.

The defense mode escalates progressively: monitor → jitter → bypass → isolate.

How I Used SigNoz

SigNoz was the perfect choice for this project because CacheGuard generates three types of telemetry — traces, metrics, and logs — and SigNoz handles all three natively through OpenTelemetry.

Setting Up the Stack

The entire infrastructure runs locally via Docker Compose. SigNoz provides the collector (OTLP endpoint on port 24317/24318), the ClickHouse database for storage, and the web UI for visualization on port 8090.

# From our casting.yaml - Foundry deployment config
apiVersion: v1alpha1
kind: Installation
metadata:
  name: cacheguard
spec:
  deployment:
    flavor: compose
    mode: docker

OpenTelemetry Instrumentation

CacheGuard uses the OpenTelemetry Python SDK to emit rich, structured telemetry:

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Every LLM call becomes a span with security metadata
with tracer.start_as_current_span("cacheguard.llm.call") as span:
    span.set_attribute("cacheguard.side_channel.risk_score", risk_score)
    span.set_attribute("cacheguard.defense.mode", defense_action.mode)
    span.set_attribute("cacheguard.defense.jitter_ms", defense_action.jitter_ms)
    span.set_attribute("gen_ai.prompt_cache.hit", is_cache_hit)

Each span carries over 15 custom attributes that capture the full security context of every single LLM request. This makes SigNoz's trace explorer incredibly powerful for debugging — you can click on any individual request and see exactly why it was flagged, what defense was applied, and how the risk score was calculated.

Building the Dashboard

I built 3 core dashboard panels in SigNoz using raw ClickHouse SQL queries against the trace data:

1. Security SLO (Compliance %)

This panel shows what percentage of traffic is "safe" (risk score below 0.75). During normal operations, this sits at 95-100%. When an attacker starts probing, you can watch it drop in real-time.

SELECT 
    (countIf(attributes_number['cacheguard.side_channel.risk_score'] < 0.75) / count()) * 100 AS value 
FROM signoz_traces.distributed_signoz_index_v3 
WHERE serviceName = 'demo-runner-client'

2. Prompt Cache Hit Rate

This time-series chart visualizes the cache performance over time, proving the value of caching while simultaneously showing when the defense middleware forces cache bypasses during an attack.

SELECT 
    toStartOfInterval(timestamp, INTERVAL 1 MINUTE) as ts,
    (countIf(attributes_bool['gen_ai.prompt_cache.hit'] = true) / count()) * 100 as value
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'demo-runner-client'
GROUP BY ts ORDER BY ts

3. Active Defense Actions

The most visually dramatic panel — it shows the exact moments when CacheGuard autonomously detected an attacker and deployed countermeasures. You can see the jitter injections spike during an attack and tenant isolations activate when the threat is severe.

SELECT 
    toStartOfInterval(timestamp, INTERVAL 1 MINUTE) as ts,
    countIf(attributes_string['cacheguard.defense.mode'] = 'jitter') as value
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'demo-runner-client'
GROUP BY ts ORDER BY ts

Why SigNoz Was the Right Choice

Native OpenTelemetry Support: SigNoz speaks OTLP natively. I didn't need any adapters, translators, or proprietary SDKs. The standard OpenTelemetry Python SDK sends data directly to SigNoz's collector.
ClickHouse Backend: The ability to write raw ClickHouse SQL queries against the trace data was a game-changer. Instead of being limited to pre-built metric types, I could calculate complex security analytics (like the KS test anomaly count) directly from span attributes.
Unified Traces + Metrics + Logs: Security observability requires correlating across all three signals. When a log says "Timing Side-Channel Risk Detected", I need to click through to the exact trace that triggered it and see the associated metrics. SigNoz handles this correlation natively.
Self-Hosted & Open Source: For a security-focused project, sending sensitive telemetry data to a third-party SaaS platform would be ironic. SigNoz runs entirely within our Docker Compose stack, keeping all data local.

The Demo: Watching an Attack in Real-Time

The demo_runner.py script simulates a complete attack scenario in 4 acts:

Act 1 - Normal Traffic: Legitimate users make a mix of cached and uncached requests. The Security SLO stays at ~95%.
Act 2 - Heavy Cache Usage: A performance-focused user hammers the cache, demonstrating the cost savings and hit rate.
Act 3 - Attacker Probing: An attacker begins systematically probing the cache. CacheGuard detects the bimodal timing distribution via the KS test and begins injecting jitter.
Act 4 - Full Defense: The attacker continues, but now every response is jittered, bypassed, or isolated. The attacker sees only noise — the side-channel is completely neutralized.

On the SigNoz dashboard, you can watch this story unfold in real-time: the Security SLO dips, the Active Defense counters spike, and the system autonomously recovers — all without human intervention.

Key Takeaways

LLM prompt caching introduces real security vulnerabilities that the industry hasn't fully addressed yet. Timing side-channels are not theoretical — they're exploitable today.
Observability is the foundation of security. You can't defend against what you can't see. By instrumenting every LLM call with OpenTelemetry and visualizing it in SigNoz, CacheGuard turns invisible timing patterns into actionable threat intelligence.
Active defense beats passive alerting. Instead of just sending a Slack notification when an attack is detected, CacheGuard autonomously injects noise, bypasses the cache, and isolates the attacker — all in milliseconds.
SigNoz + OpenTelemetry is a powerful combination for building custom observability solutions. The ability to query raw trace data with ClickHouse SQL gives you unlimited flexibility to build domain-specific dashboards.

Try It Yourself

The entire project is open source and can be deployed locally in minutes:

git clone https://github.com/aryanpenny/CacheGuard.git
cd CacheGuard
docker compose -f pours/deployment/compose.yaml up -d
pip install -r requirements.txt
python mock_llm.py &
python demo_runner.py

Then open http://localhost:8090 to see SigNoz in action. Follow the DASHBOARD_SETUP.md guide to create the 3 core dashboard panels.

GitHub: https://github.com/aryanpenny/CacheGuard

Built for the SigNoz Hackathon. CacheGuard demonstrates how OpenTelemetry and SigNoz can be used beyond traditional APM — as the foundation for real-time AI security observability and autonomous threat mitigation.

DEV Community