Reducing LLM Costs by Measuring Prompt Caching

LLM (Large Language Model) API calls can become a significant expense due to token-based costs as applications scale. Prompt caching aims to directly reduce this cost by returning cached responses for previously answered or similar queries instead of re-querying the LLM. This strategy not only lightens the financial burden but also significantly shortens response times.

In this post, we will cover the fundamental principles of prompt caching, different implementation approaches, and its key role in cost optimization. Especially in areas with high volumes of repetitive queries, such as financial technology applications or production planning systems, implementing caching is an indispensable step for operational efficiency.

What is Prompt Caching and Why is it Important?

Prompt caching is the process of storing a prompt sent to an LLM and its corresponding response in a cache mechanism. When the same or a semantically similar prompt arrives again, the cached response is returned quickly instead of making an API call to the LLM provider. This fundamental mechanism is critical for the scalability and cost-effectiveness of LLM-based applications.

Especially in applications with high-volume or repetitive queries, sending every request to the LLM API increases both latency and multiplies token-based costs. Caching eliminates this repetitive load, improving user experience and preserving the operational budget. For instance, in an ERP system of a manufacturing company, hundreds of daily reports or similar queries from financial calculators being served from the cache instead of going to the LLM every time leads to noticeable savings in a short period.

Key Advantages of Prompt Caching

The main advantages that prompt caching provides to businesses and developers can be summarized as follows:

• Cost Reduction: The most obvious benefit is the reduction in token costs paid per LLM API call. This difference can be very striking, especially for expensive models or high-volume usage. OpenAI applies a lower charge for cached input tokens than standard input fees. Google Gemini API also offers significantly lower costs for cached inputs, with discounts up to 90% in some cases.
• Performance Improvement: Retrieving a response from the cache is much faster than making an API call and waiting for a response. This directly improves the application's overall response time and user experience. OpenAI's prompt caching feature can reduce latency by up to 80%.
• API Rate Limit Management: Reduces the risk of hitting API rate limits imposed by LLM providers. When the cache hit rate is high, the number of external API calls decreases.
• Service Reliability: The ability to serve responses from the cache even during temporary outages or slowdowns of the LLM provider makes the application more resilient.

💡 Cost-Effectiveness

If you want to reduce costs in your LLM applications, one of the first areas you should focus on is prompt caching. It is possible to achieve a high cache hit rate, especially for frequently used or relatively static prompts.

Types of Prompt Caching and Architectural Approaches

Prompt caching can be implemented in various ways depending on different needs and application scenarios. Fundamentally, it is divided into two main categories based on how similar the prompt needs to be: Exact Match Caching and Semantic Caching.

Exact Match Caching

Exact match caching checks if an incoming prompt has the exact same character string as a stored prompt in the cache. If an exact match is found, the corresponding cached response is returned. This method is generally the simplest and fastest caching strategy.

How it Works:

1. The hash value of the incoming prompt is calculated.
2. This hash value is searched in the cache.
3. If a matching hash is found, the associated response is returned.
4. If no match is found, the prompt is sent to the LLM, the response is received, and the prompt-response pair is added to the cache.

Advantages: Easy implementation, fast lookup.
Disadvantages: Even a small character difference is perceived as a different prompt, which can lower the cache hit rate. For example, "What is the capital of Turkey?" and "What is Turkey's capital?" would be considered different prompts.

Semantic Caching

Semantic caching analyzes the meaning of the incoming prompt to determine if it is semantically similar to prompts in the cache. This approach allows returning a response from the cache for prompts that carry similar meaning, even if there isn't an exact match. Libraries like RedisVL provide a SemanticCache interface to cache LLM responses based on semantic similarity, leveraging Redis's built-in caching capabilities and vector search.

How it Works:

1. The incoming prompt is converted into a vector representation using an embedding model.
2. This prompt embedding is compared with other prompt embeddings in the cache using a similarity search (e.g., cosine similarity).
3. If a prompt showing similarity above a certain threshold is found, the associated response is returned.
4. If no match is found, the prompt is sent to the LLM, the response is received, and the prompt-response pair is added to the cache.

Advantages: High cache hit rate, more flexible.
Disadvantages: More complex implementation, additional embedding model cost and latency, difficulty in setting the similarity threshold. In my side product's financial calculators, I evaluated the potential of semantic caching to allow users to quickly get the correct answer even if they asked the same question with different words.

Architectural Approaches

Prompt caching can be positioned at different layers depending on your application's architecture:

• In-Application Cache: Stored in the application's own memory space (e.g., functools.lru_cache or a dictionary in Python). Suitable for a single application instance.
• Shared Cache Service: An external caching system like Redis or Memcached is used. Allows sharing the cache among multiple application instances or microservices. This is generally the preferred approach in enterprise software architectures.
• LLM Provider-Side Cache: Some LLM providers offer an internal caching mechanism on their side.
  • OpenAI: Provides an automatically operating prompt caching mechanism for gpt-4o and newer models. This feature reduces latency and cost if the initial part of the prompt (prefix) is found in the cache. For GPT-5.6 and subsequent model families, tokens written to the cache are billed at 1.25 times the rate of uncached input tokens.
  • Google Gemini API: Offers two different caching mechanisms:
    • Implicit caching: Automatically enabled in Gemini 2.5 and newer models, providing cost savings on cache hits.
    • Explicit caching: A feature that developers can manually enable, which guarantees cost savings by caching specific content and referring to this cache in subsequent requests. TTL (Time-To-Live) can be set for explicit caching and defaults to 1 hour. Write-to-cache costs can be 25% to 100% more expensive than standard input token costs depending on the TTL, while read-from-cache tokens are 90% cheaper.

ℹ️ Hybrid Approaches

Depending on the complexity and scale of the application, hybrid solutions combining different caching types and architectural approaches may be preferred. For example, an in-application exact match cache can be used for very frequent and exact match expected prompts, while a Redis-based semantic cache can be used for less frequent but semantically similar prompts.

Cost Measurement and Optimization Metrics

Monitoring the right metrics is vital to evaluate the effectiveness of prompt caching and quantify cost reduction. Simply implementing the cache is not enough; you must continuously measure to understand how much you're saving and where you can optimize further.

Core Cost Metrics

When measuring your LLM API usage, the main metrics to pay attention to are:

• Total Token Usage: The total number of prompt and response tokens sent to the LLM within a specific time frame. This is the basis of the LLM provider's billing.
• Total API Calls: The total number of calls made to the LLM API.
• Average Token Cost: The average cost for each token used, calculated according to the provider's pricing model. Prompt and response tokens may often be priced differently.

Caching-Specific Optimization Metrics

To directly measure the impact of caching on performance and cost, the following metrics should be monitored:

• Cache Hit Rate: The ratio of requests served from the cache to the total number of requests. A high hit rate indicates the effectiveness of the caching strategy.
  

  Cache Hit Rate = (Number of Requests Served from Cache / Total Number of Requests) * 100
  

• Cache Miss Rate: The ratio of requests not found in the cache and sent to the LLM to the total number of requests. A high miss rate may indicate a need to review the caching strategy.

• Saved API Calls: The number of API calls saved from being sent to the LLM thanks to caching.

• Saved Tokens: The number of tokens saved from being sent to the LLM thanks to caching.

• Estimated Cost Savings: The estimated monetary savings calculated based on the number of saved tokens and the average token cost.

Let's consider an example scenario (these numbers are for illustrative purposes only and may not reflect current prices):
Suppose you received 1,000,000 LLM prompts in a month. Each prompt generates an average of 100 prompt tokens and 200 response tokens. Let the prompt token cost be 0.00001 USD, and the response token cost be 0.00002 USD.
Total tokens: 1,000,000 * (100+200) = 300,000,000 tokens.
Total cost: (1,000,000 * 100 * 0.00001) + (1,000,000 * 200 * 0.00002) = 1,000 USD + 4,000 USD = 5,000 USD.

If you achieve a 50% cache hit rate:
500,000 requests are served from the cache.
500,000 requests go to the LLM.
New total cost: (500,000 * 100 * 0.00001) + (500,000 * 200 * 0.00002) = 500 USD + 2,000 USD = 2,500 USD.
With a 50% hit rate, your cost is halved. Such financial calculations clearly demonstrate the value of caching.

Monitoring and Reporting

It is important to set up a monitoring and reporting infrastructure to regularly collect and visualize these metrics. Tools like Prometheus, Grafana, and the ELK Stack can be used in this process. Especially in a high-volume system, tracking this data in real-time allows you to make quick adjustments to caching strategies. In the backend of my own side product, I used to pull such metrics from Redis and generate daily reports to observe cost trends.

Prompt Caching Implementation Strategies

When implementing prompt caching, several strategies and practical steps need to be considered. A successful implementation requires not only writing code but also making the right caching policies and architectural decisions.

1. Determining Cache Location

The cache location should be decided based on the application's scale and architecture:

• For a Single Service: For simple Python applications or standalone microservices, in-application solutions like functools.lru_cache may suffice.
  

  from functools import lru_cache
  
  @lru_cache(maxsize=128) # Default maxsize is 128. None can be used for unlimited cache.
  def call_llm_with_cache(prompt: str) -> str:
      print(f"Going to LLM: {prompt[:50]}...")
      # Simulated LLM call
      import time
      time.sleep(1)
      return f"LLM Response: {prompt}"
  
  print(call_llm_with_cache("Hello world"))
  print(call_llm_with_cache("Hello world")) # Will come from cache
  

• For Distributed Systems: For microservice architectures or systems running multiple application instances, a distributed caching solution like Redis is essential.
  

  import redis
  import hashlib
  import json
  
  # Update your Redis connection details
  r = redis.Redis(host='localhost', port=6379, db=0)
  
  def get_llm_response(prompt: str) -> str:
      prompt_hash = hashlib.md5(prompt.encode('utf-8')).hexdigest()
      cached_response = r.get(prompt_hash)
  
      if cached_response:
          print(f"Cache Hit for: {prompt[:50]}...")
          return json.loads(cached_response.decode('utf-8'))['response']
      else:
          print(f"Cache Miss, going to LLM: {prompt[:50]}...")
          # Simulated LLM call
          import time
          time.sleep(2)
          llm_response = f"LLM Response: {prompt}"
          # The setex command automatically deletes the key after a specified time (in seconds).
          r.setex(prompt_hash, 3600, json.dumps({"response": llm_response})) # Cache for 1 hour
          return llm_response
  
  print(get_llm_response("How is the weather today?"))
  print(get_llm_response("How is the weather today?")) # Will come from cache
  

2. Cache Invalidation Strategies

Correct invalidation strategies must be determined to keep cached data up-to-date:

• TTL (Time-To-Live): Automatic deletion of cached data after a specified period. Suitable for dynamic data. The explicit caching feature in Google Gemini API supports TTL and defaults to 1 hour. RedisVL SemanticCache also supports TTL policies.
• Manual Invalidation: The application explicitly clears the cache when data changes. For critical and infrequently changing data.
• LFU (Least Frequently Used) / LRU (Least Recently Used): Evicting the least frequently used or least recently used items when the cache is full. functools.lru_cache uses this strategy.

3. Key Generation Strategies

The cache key should create a unique representation of the prompt. For exact match, the hash of the prompt itself is used, while for semantic caching, prompt embeddings are used.

• Exact Match: hashlib.md5(prompt.encode('utf-8')).hexdigest() is a common method.
• Semantic Caching: Passing the prompt through an embedding model, storing the resulting vector in a vector database (like Pinecone, Weaviate, Qdrant), and performing a similarity search. Vector databases like Pinecone, Weaviate, and Qdrant are commonly used for semantic search and RAG (Retrieval-Augmented Generation) systems.

4. Cache Architecture Selection

• Single-Layer: Only exact match or only semantic caching.
• Multi-Layered: First, a fast exact match cache is checked; if there's no hit, the more costly semantic cache is consulted. This can provide a good balance between performance and hit rate.

⚠️ Cache Size Management

It is important to limit cache size and regularly clean up old/unused items. An infinitely growing cache can lead to memory issues and performance degradation. This management becomes more critical in distributed caching systems. OpenAI states that prompts that are not regularly used are automatically removed from the cache.

Challenges and Trade-offs

While prompt caching offers many advantages, it also brings some challenges and trade-offs that need to be considered. Understanding these points will help you determine the most suitable caching strategy for your application.

1. Cache Invalidation Complexity

One of the biggest challenges is determining when cached data becomes stale. This is critical, especially for prompts containing dynamic or time-sensitive information. For example, the answer to a prompt like "What are the exchange rates in Turkey today?" changes constantly, and keeping it in the cache for too long would lead to incorrect information.

• Stale Data Risk: Presenting incorrect or outdated data negatively impacts user experience and can lead to wrong decisions. To manage this risk, short TTLs or manual invalidation mechanisms triggered by data updates may be necessary.
• Performance vs. Freshness: Longer TTLs provide higher cache hit rates and cost savings but increase the risk of data staleness. Finding a balance between these two factors depends on the application's requirements.

2. Cost and Complexity of Semantic Caching

While semantic caching offers a higher hit rate than exact match caching, it introduces additional costs and operational complexity:

• Embedding Model Cost: Using an embedding model to convert prompts into vectors requires additional API call costs or computational resources if a local model is used.
• Vector Database Management: Storing embeddings and performing similarity searches requires a dedicated vector database or a database with vector search capabilities. Solutions like Pinecone, Weaviate, and Qdrant are used in this area. This implies additional infrastructure and management overhead.
• Threshold Adjustment: Setting the similarity threshold correctly is difficult. A too-low threshold can lead to irrelevant responses being returned from the cache; a too-high threshold can reduce the hit rate. This is a process that requires continuous experimentation and adjustment. For RedisVL SemanticCache, the distance_threshold parameter determines this threshold.

3. Cache Size and Management

The cache consumes the application's memory or storage resources. In applications with high volume and diverse prompts, cache sizes can grow rapidly.

• Resource Consumption: Especially in in-application caches, this can lead to memory leaks or application crashes. In distributed caches, the resource consumption and cost of services like Redis increase. There are storage costs for explicit caching in Google Gemini API.
• Data Management: Deciding which data to keep in the cache is important. Prioritizing the most frequently accessed or most costly queries for caching improves efficiency.

4. LLM API Changes and Incompatibility

LLM models and APIs are constantly evolving. An update to a model or the introduction of a new model can invalidate cached responses.

• Model Versioning: Including the version of the LLM model used in the cache key can mitigate such problems. This way, when a new model version is used, there is no conflict with old cached data. OpenAI states that caching behavior has changed for GPT-5.6 and subsequent model families.

ℹ️ Trade-off Analysis

When implementing prompt caching, you need to constantly balance cost savings, performance improvement, data freshness, and implementation complexity. Since each application's requirements are different, there is no "one size fits all" solution.

From Mustafa Erbay's Perspective: Real-World Scenarios

In my nearly 20 years of experience in system and software development, performance and cost optimization have always been a priority. Since LLMs entered our lives, these principles have been carried into AI application architectures with techniques like prompt caching. In particular, I have clearly seen the value of prompt caching in the AI-powered operations of one of my side products, where I use multi-LLM provider fallback (Gemini Flash, Groq, Cerebras, OpenRouter).

When developing AI-powered recommendation systems for financial calculators in one of my side products or for operator screens in a manufacturing ERP, I observed that user questions often repeated within certain patterns. For example, answers to questions like "What is the average production time for product X?" or "When is the shipment date for order Y?" usually remain current for a certain period, and going to the LLM every time would create unnecessary costs.

In such scenarios, exact match caching quickly provided high hit rates. However, for more free-form prompts like "Perform a cost analysis for product X" or "Summarize the status of order Y," the potential of semantic caching emerged. Even when users asked the same question with different phrasing, returning responses from the cache for semantically similar prompts reduced latency and helped us keep LLM token costs under control.

Of course, caching is not always a panacea. Especially in LLM applications integrated with RAG (Retrieval-Augmented Generation) patterns, if the freshness of the source data is critical, cache invalidation strategies must be managed much more carefully. My approach is always to first understand the problem and business requirement, then implement the simplest and most cost-effective solution accordingly. Sometimes a simple lru_cache is sufficient at the beginning, but as the system scales, it becomes necessary to switch to a Redis-based distributed and multi-layered cache architecture. When planning these transitions, the existing infrastructure and operational load are always taken into account.

Conclusion

Prompt caching is an indispensable technique for reducing the cost and improving the performance of LLM-based applications. Different approaches such as exact match and semantic caching can be chosen according to your application's needs. For a successful implementation, it is essential to regularly monitor cost metrics, apply correct invalidation strategies, and carefully design the cache architecture.

It should be remembered that caching is a balancing act. Finding the right balance between cost savings, performance improvement, and data freshness is critical for the long-term success of your application. By correctly applying these strategies, we can use the power of LLM technology more economically and efficiently.

Official Resources

• openai.com
• openai.com
• google.com
• openai.com
• googleblog.com
• pipecat.ai
• google.dev
• puter.com