DEV Community: syncore

Slash Your Claude API Costs by 90% with Prompt Caching: A Practical Guide

syncore — Fri, 29 May 2026 19:08:02 +0000

If you are building production-grade AI applications, you already know the pain of LLM API bills. As your context grows—whether you are feeding Claude large codebases, legal documents, or long chat histories—the cost of input tokens scales linearly.

But it doesn't have to.

Anthropic's Prompt Caching feature allows you to cache frequently used context. Instead of paying full price to process the same system instructions or reference documents over and over, you can reuse them for a 90% discount on input tokens, while simultaneously cutting latency by up to 80%.

In this hands-on guide, you will learn how to implement prompt caching in Python, understand the pricing dynamics, and walk away with production-ready code.

How Prompt Caching Works (The TL;DR)

Normally, every time you send a request to Claude, the API processes your entire prompt from scratch.

With Prompt Caching, you define specific cache breakpoints in your prompt. When Claude processes a request containing a breakpoint, it saves that prefix of the prompt to a temporary cache (with a 5-minute Time-To-Live, refreshed on every hit).

Here is the pricing breakdown for Claude 3.5 Sonnet:

Base Input Tokens: $3.00 / million tokens
Cache Write (Creation) Tokens: $3.75 / million tokens (a 25% premium to write to cache)
Cache Read (Hit) Tokens: $0.30 / million tokens (a 90% saving)

If you query a 50,000-token document 10 times, you go from paying $1.50 per request to just $0.15 per request after the first turn.

Implementation 1: Caching Large Reference Documents

To use prompt caching, you must target a minimum token threshold. For Claude 3.5 Sonnet and Claude 3 Haiku, the minimum cacheable prompt size is 1,024 tokens (roughly 750 words). For Claude 3 Opus, it is 2,048 tokens.

Here is how to cache a massive reference document using the Python SDK. We will use the cache_control parameter with the type ephemeral.

import os
from anthropic import Anthropic

# Initialize the client (ensure ANTHROPIC_API_KEY is in your environment)
client = Anthropic()

# Let's simulate a large knowledge base (must be > 1024 tokens to cache)
large_knowledge_base = """
SYSTEM RULES AND KNOWLEDGE BASE (v4.2):
[Imagine 2,000 words of complex enterprise documentation, API specs, or code guidelines here...]
""" * 50 

def query_with_cached_context(user_query: str):
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=500,
        temperature=0,
        system=[
            {
                "type": "text",
                "text": large_knowledge_base,
                # This flag tells Anthropic to cache everything up to this block
                "cache_control": {"type": "ephemeral"}
            }
        ],
        messages=[
            {"role": "user", "content": user_query}
        ]
    )

    # Inspect token usage to verify caching behavior
    usage = response.usage
    print(f"Response: {response.content[0].text[:100]}...")
    print(f"Input Tokens: {usage.input_tokens}")
    print(f"Cache Creation Tokens (Write): {getattr(usage, 'cache_creation_input_tokens', 0)}")
    print(f"Cache Read Tokens (Hit): {getattr(usage, 'cache_read_input_tokens', 0)}")
    print("-" * 40)

# First run: Cache Miss (Creates the cache)
print("--- Run 1 (Cache Miss / Creation) ---")
query_with_cached_context("What version of the system rules is this?")

# Second run: Cache Hit (Reads from cache)
print("--- Run 2 (Cache Hit) ---")
query_with_cached_context("Summarize the main rules in one sentence.")

When you run this, the first request will show a high number of cache_creation_input_tokens. The second request (executed within 5 minutes) will show those same tokens registered under cache_read_input_tokens, billing you at the 90% discounted rate.

Implementation 2: Multi-Turn Conversations

Prompt caching is incredibly powerful for chatbots. By caching the conversation history, you only pay full price for the latest user message.

To make this work, place your cache breakpoint at the end of the conversation history. Anthropic allows up to 4 cache breakpoints per request.

from anthropic import Anthropic

client = Anthropic()

# We start a conversation and cache the history as it grows
conversation_history = [
    {"role": "user", "content": "Let's write a python script step-by-step. First, write a function to fetch data from an API."},
    {"role": "assistant", "content": "Here is a basic function using `requests`:\n\n```

python\nimport requests\ndef fetch_data(url):\n    return requests.get(url).json()\n

```"},
    # We place the cache breakpoint on the most recent message to keep the history cached
    {
        "role": "user", 
        "content": "Now, add error handling to that function.",
        "cache_control": {"type": "ephemeral"}
    }
]

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=800,
    messages=conversation_history
)

print(f"Assistant: {response.content[0].text}")
print(f"Cache Hits: {getattr(response.usage, 'cache_read_input_tokens', 0)} tokens")

Practical Takeaways for Developers

To get the most out of prompt caching, structure your API calls with these rules in mind:

Order Matters: Claude processes prompts from top to bottom. If any character changes before a cache breakpoint, the entire cache after that point is invalidated. Always put your static content (system prompts, documentation, instructions) at the very beginning, and dynamic content (user queries) at the end.
Mind the Limit: Caching only triggers if your prompt prefix meets the minimum token limit (1,024 for Sonnet/Haiku). Small prompts will not benefit from caching.
Track Your Metrics: Always log cache_read_input_tokens and cache_creation_input_tokens in your telemetry. This ensures you can audit your actual cost savings.

What will you build?

With a 90% cost reduction on large contexts, you can now build features that were previously cost-prohibitive: codebase-wide QA bots, real-time document synthesizers, and highly contextual agents.

Are you already using prompt caching in your projects? Share your cost-saving wins or ask your integration questions in the comments below!

How to Build a Production-Ready Claude Chatbot in Python

syncore — Fri, 29 May 2026 18:27:02 +0000

Anthropic's Claude 3.5 Sonnet has set a new benchmark for intelligence, speed, and cost-effectiveness. But there is a massive gulf between running a simple API script and deploying a production-ready chatbot.

In a production environment, your chatbot needs to handle state (conversation memory), stream responses in real-time to prevent user drop-off, and gracefully handle API rate limits and network failures.

In this guide, we will build a production-grade Claude chatbot in Python using the official Anthropic SDK, complete with memory management, streaming, and robust error handling.

Prerequisites

First, make sure you have the Anthropic Python SDK installed and your API key set as an environment variable.

pip install anthropic
export ANTHROPIC_API_KEY="your-api-key-here"

Step 1: Building the Core Streaming and Memory Engine

A production chatbot cannot make users wait 10 seconds for a full paragraph to generate. We must stream responses chunk-by-chunk. Additionally, we need to maintain a thread history so Claude remembers the context of the conversation.

Here is the core engine utilizing Python generators to stream tokens in real-time:

import os
from typing import Generator, List, Dict
from anthropic import Anthropic

class ClaudeChatEngine:
    def __init__(self, system_prompt: str = "You are a helpful, concise assistant."):
        # Initialize the client. It automatically looks for ANTHROPIC_API_KEY in env
        self.client = Anthropic()
        self.model = "claude-3-5-sonnet-20241022"
        self.system_prompt = system_prompt
        self.history: List[Dict[str, str]] = []

    def send_message(self, user_message: str) -> Generator[str, None, None]:
        """
        Sends a message to Claude, updates history, and yields tokens as they arrive.
        """
        # Append user message to state history
        self.history.append({"role": "user", "content": user_message})

        assistant_response = ""

        # Initiate a streaming request
        with self.client.messages.stream(
            model=self.model,
            max_tokens=1024,
            system=self.system_prompt,
            messages=self.history
        ) as stream:
            for text in stream.text_stream:
                assistant_response += text
                yield text

        # Append Claude's completed response to history to maintain context
        self.history.append({"role": "assistant", "content": assistant_response})

# Usage Example
if __name__ == "__main__":
    bot = ClaudeChatEngine()
    print("Chatbot initialized. Type 'exit' to quit.\n")

    while True:
        user_input = input("You: ")
        if user_input.lower() == "exit":
            break

        print("Claude: ", end="", flush=True)
        for token in bot.send_message(user_input):
            print(token, end="", flush=True)
        print("\n")

Step 2: Making it Production-Ready (Error Handling & Resilience)

The code above works perfectly under ideal conditions. But in production, network requests fail, APIs rate-limit you, and unexpected errors occur.

To make this production-ready, we need to wrap our API calls in a resilient layer that catches:

RateLimitError: When you exceed your tokens-per-minute (TPM) or requests-per-minute (RPM).
APIConnectionError: When network issues prevent connection to Anthropic's servers.
APIStatusError: When the API returns a non-200 HTTP code (e.g., overloaded servers or invalid requests).

Let's refactor our engine to handle these scenarios gracefully, incorporating exponential backoff for rate limits.

import time
import logging
from anthropic import (
    Anthropic, 
    APIConnectionError, 
    RateLimitError, 
    APIStatusError
)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("ClaudeChat")

class ResilientClaudeChatEngine(ClaudeChatEngine):
    def send_message_safe(self, user_message: str, max_retries: int = 3) -> Generator[str, None, None]:
        """
        Sends a message with exponential backoff for rate limits and robust error handling.
        """
        self.history.append({"role": "user", "content": user_message})
        assistant_response = ""
        delay = 1.0  # Initial delay for backoff

        for attempt in range(max_retries):
            try:
                with self.client.messages.stream(
                    model=self.model,
                    max_tokens=1024,
                    system=self.system_prompt,
                    messages=self.history
                ) as stream:
                    for text in stream.text_stream:
                        assistant_response += text
                        yield text

                # If we successfully finished streaming, break out of retry loop
                break

            except RateLimitError as e:
                logger.warning(f"Rate limited (429). Retrying in {delay}s... Error: {e}")
                if attempt == max_retries - 1:
                    yield "\n[Error: The system is currently busy. Please try again in a moment.]"
                    return
                time.sleep(delay)
                delay *= 2  # Exponential backoff

            except APIConnectionError as e:
                logger.error(f"Failed to connect to [Anthropic API](https://console.anthropic.com): {e}")
                yield "\n[Error: Connection issue. Please check your internet connection.]"
                return

            except APIStatusError as e:
                logger.error(f"Anthropic API returned status code {e.status_code}: {e.message}")
                yield f"\n[Error: An API error occurred (Status {e.status_code}).]"
                return

            except Exception as e:
                logger.error(f"Unexpected error: {str(e)}")
                yield "\n[Error: An unexpected system error occurred.]"
                return

        # Save the history only if we got a successful response
        if assistant_response:
            self.history.append({"role": "assistant", "content": assistant_response})

Key Takeaways for Production

Keep Context Under Control: Claude 3.5 Sonnet has a 200k context window, but sending the entire history on every message increases latency and costs. Implement a sliding window or summarization strategy if your chat sessions exceed 20-30 messages.
System Prompts: Always pass system-level instructions (like persona, guardrails, and output formatting constraints) via the system parameter, not as a user message. This ensures the model adheres strictly to your rules.
Environment Variables: Never hardcode your ANTHROPIC_API_KEY. Use env vars or secret managers (like AWS Secrets Manager or HashiCorp Vault) to inject keys at runtime.

What's Next?

You now have a production-ready core engine for a Claude chatbot. Your next step is to wrap this engine in an API layer like FastAPI to expose it to your frontend, or deploy it directly to a serverless environment.

Are you building with Claude? Let me know in the comments below what challenges you're facing with context windows or latency!