DEV Community: Nitheesh gaddam

A Deep Dive into Amazon Bedrock Prompt Caching for Claude 4.6

Nitheesh gaddam — Sun, 26 Jul 2026 12:43:47 +0000

Have you ever noticed that your GenAI applications are spending massive amounts of time and money re-reading the exact same setup text?
Every time a user asks a short question in a chatbot, the Large Language Model (LLM) must re-read your entire 2,000-word corporate playbook, your agent's system rules, and the full chat history from scratch.
This phase is called the pre-fill math phase, and it drives up both your cloud bill and your user latency (Time-to-First-Token).With Amazon Bedrock Prompt Caching for Claude 4.6 (both Sonnet 4.6 and Opus 4.6), this problem is completely solved. You can achieve up to a 90% cost reduction on input tokens and an 85% drop in latency by using a clever architectural shortcut.

Here is exactly how it works under the hood, how AWS maintains it across API requests, and how to implement it using Python.

The Secret Architecture: Model Inference vs. AWS Infrastructure

Prompt caching is a beautiful team effort between the AI model hardware and the AWS cloud infrastructure.

The Model Level (The Brains): Inside Claude 4.6, text is processed through mathematical matrices called KV (Key-Value) Caches. Instead of re-reading text, the GPUs calculate the meaning of your system instructions once and build a "mathematical profile." When a cache point is triggered, the model freezes this calculated KV state inside the GPU memory.

2.The AWS Bedrock Level (The Manager): Normally, an LLM wipes its memory the millisecond an API call finishes. AWS Bedrock changes this. It takes your static prompt, creates a secure, unique cryptographic hash (fingerprint), and pins that KV memory block alive.

When your next API request comes in, AWS Bedrock instantly hashes the new incoming prompt text. If the top section matches a saved fingerprint, Bedrock's router bypasses the standard pre-fill setup and routes your request directly to the GPU holding your frozen mathematical profile.

It is exactly like loading a "Save Game" file instead of restarting a video game from Level 1 on every single turn!

The Golden Rules of Bedrock Caching

Before writing the code, keep these rules in mind to ensure your cache actually hits:

1.The Thresholds: Your cached text must meet minimum size requirements. For Claude Sonnet 4.6, your static content must be at least 1,024 tokens. For Claude Opus 4.6, it requires 4,096 tokens.

2.The 5-Minute Window: The cache stays alive for a default Time-To-Live (TTL) of 5 minutes. However, every time a user makes a new request and hits the cache, that 5-minute countdown timer resets back to zero.

3.Order Matters: AWS reads your prompts sequentially. You must put your heavy, fixed instructions first, drop your cache bookmark, and append your changing user messages at the very end. If a single character changes before your cache marker, the cache breaks!

Implementation: Prompt Caching with Python (Boto3)

The cleanest way to handle a multi-turn chatbot with prompt caching is using AWS Bedrock's Converse API. By putting a cachePoint at the end of your system configuration, your fixed instructions stay frozen while your dynamic chat messages grow freely.

import boto3

# Initialize the Bedrock Runtime client
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

# Target Model: Claude Sonnet 4.6 (Minimum threshold: 1,024 tokens)
MODEL_ID = "anthropic.claude-sonnet-4-6"

# 1. DEFINE FIXED SYSTEM INSTRUCTIONS (Ensure this exceeds 1,024 tokens)
BASE_SYSTEM_PROMPT = """
You are a highly specialized corporate HR assistant for Acme Corp.
You must adhere strictly to the following guidelines:
1. Always maintain a professional tone.
2. Rely only on company procedures outlined below.
... Imagine 1,000+ tokens of corporate documentation/policies placed here ...
"""

# 2. STRUCTURE SYSTEM PARAMETER WITH A CACHE POINT
# We place the cachePoint marker IMMEDIATELY following our static text block.
system_configuration = [
    {"text": BASE_SYSTEM_PROMPT},
    {"cachePoint": {"type": "default"}}
]

# Track our growing chat conversation history
conversation_history = []

def run_chat_turn(user_input):
    global conversation_history

    # Append the newest user message to our history tracking
    conversation_history.append({
        "role": "user",
        "content": [{"text": user_input}]
    })

    print(f"\n--- Processing Chat Request (Turns: {len(conversation_history)//2 + 1}) ---")

    # Invoke the Bedrock Converse API
    response = bedrock.converse(
        modelId=MODEL_ID,
        system=system_configuration,   # The fixed cached system instructions
        messages=conversation_history,  # The growing dynamic chat history
        inferenceConfig={"maxTokens": 500, "temperature": 0.4}
    )

    # Save the assistant's reply back to history
    assistant_message = response["output"]["message"]
    conversation_history.append(assistant_message)

    # Extract usage metrics to verify cache hits!
    metrics = response["usage"]
    read_from_cache = metrics.get("cacheReadInputTokens", 0)
    written_to_cache = metrics.get("cacheWriteInputTokens", 0)
    standard_input = metrics.get("inputTokens", 0)

    print(f"AI Response snippet: {assistant_message['content'][0]['text'][:80]}...")
    print(f" 💾 Tokens Written to Cache (First Turn): {written_to_cache}")
    print(f" 🔥 Tokens Read From Cache (Discounted!): {read_from_cache}")
    print(f" 📝 Standard Input Tokens Processed: {standard_input}")

# ==========================================================
# SIMULATING THE CONVERSATION
# ==========================================================

# Turn 1: Cache Miss / Cache Write
# Bedrock reads the instructions fully, builds the KV cache, and saves it.
run_chat_turn("Hello! What are the official office hours for the engineering team?")

# Turn 2: Cache Hit!
# The system instructions are read directly from cache memory. Only new text is computed.
run_chat_turn("Great. And what is the standard protocol if I need to request time off?")

The VerdictBy keeping your architecture split cleanly between fixed inputs (cached) and dynamic inputs (uncached), you shift the heavy lifting away from continuous, expensive compute cycles and onto smart cloud routing.If you are building RAG applications, enterprise chatbots, or complex agent workflows on AWS, prompt caching isn't just an optimization feature—it is a production requirement for building fast, cost-effective AI systems.

AWS #AmazonBedrock #GenerativeAI #Claude #Python #CloudComputing

How Instagram Scales Tagging for Billions of Users

Nitheesh gaddam — Sat, 17 Jan 2026 06:40:31 +0000

Have you ever wondered what happens in the milliseconds between hitting "Share" on a photo and your friend receiving a notification that they’ve been tagged? On the surface, tagging is a simple feature. At Instagram’s scale, it is a masterclass in distributed systems design.

To handle millions of tags per minute, Instagram moves away from a single "do-it-all" database and instead uses a specialized Microservices Tech Stack.

The Core Architecture: A Four-Pillar Approach
The secret to Instagram's speed lies in using the right tool for the right job. Here is how the four main components work in harmony:

1. The Source of Truth: Sharded PostgreSQL
Every tag needs a permanent home. Instagram uses PostgreSQL, but with a twist: Logical Sharding.

How it works: Your data isn’t in one giant table; it’s partitioned across hundreds of databases based on your User_ID.

The Benefit: When you view a post, the system knows exactly which shard to query, ensuring that retrieving tag coordinates and usernames is lightning-fast and consistent.

2. The Speed Demon: Redis Caching
When a hashtag like #nature goes viral, thousands of writes happen every second.

The Role of Redis: Instead of hammering the main database to update "post counts," Instagram uses Redis—an in-memory data store.

The Benefit: It acts as a high-speed scoreboard, incrementing hashtag counts and storing "Hot Post" lists so the Explore page loads instantly.

3. The Search Engine: Elasticsearch
Standard databases are terrible at text search. If you search for "summ," a SQL database would struggle to find "#summer" among billions of rows.

The Solution: Instagram pipes caption data into Elasticsearch.

The Benefit: It builds an Inverted Index (mapping words to Post IDs), allowing for fuzzy matching and near-instant discovery of trending topics.

4. The Reliable Messenger: Apache Kafka
Tagging a friend triggers a chain reaction: a notification is sent, the "Photos of You" section updates, and the search index is refreshed.

The Role of Kafka: It acts as a Message Queue. The main app simply "drops a note" in Kafka and moves on.

The Benefit: This "asynchronous" processing ensures that if the notification service is busy, your photo upload isn't slowed down. The work happens reliably in the background.

Key Takeaways for Developers
Decouple your services: Use queues (Kafka) so your main API stays fast.

Pick the right DB: Use SQL for consistency, but NoSQL or Search Engines (Elasticsearch) for discovery.

Shard early: Horizontal scaling is the only way to survive "Instagram-level" traffic.

How GraphRAG Works

Nitheesh gaddam — Sat, 03 Jan 2026 04:51:50 +0000

GraphRAG has two main phases: Indexing (preprocessing the dataset) and Querying (answering questions).

Indexing Phase (Offline, Expensive but Done Once)

Text Chunking — Split the input text into manageable chunks.
Entity Extraction — Use an LLM to identify entities (people, places, organizations, concepts) and relationships from each chunk.
Build Knowledge Graph — Create a graph where nodes are entities and edges are relationships (with descriptions).
Community Detection — Apply graph algorithms (e.g., Leiden algorithm) to identify clusters of closely related entities (communities).
Hierarchical Summarization — Generate summaries for each community at multiple levels (bottom-up hierarchy: detailed low-level communities → higher-level aggregated summaries).
The result is a structured index: the graph + pre-generated community summaries.

This captures implicit connections across the entire dataset that vector embeddings alone miss.

Querying Phase

Local Queries (specific details): Retrieve relevant subgraphs or text chunks near mentioned entities.
Global Queries (broad understanding):
Select relevant community summaries (based on similarity to the query).
Use the LLM to generate partial answers from each summary.
Aggregate and summarize the partial answers into a final coherent response.

This "map-reduce" style over communities enables holistic reasoning.
Why It's Better Than Standard RAG

Comprehensiveness: Captures broader themes and connections → answers are more complete.
Diversity: Reduces repetition and surfaces varied perspectives.
Empowerment: Provides grounded, evidence-based insights for complex datasets (e.g., conflicting news sources).
Experiments in the paper (on datasets ~1 million tokens) show GraphRAG outperforming baseline RAG by 70-80% on metrics like comprehensiveness and diversity for global questions.

Practical Details

Open-source implementation: Available on GitHub (microsoft/graphrag).
Costs: Indexing is LLM-intensive (many calls for extraction and summarization), but querying is efficient.
Later improvements (post-paper): Things like LazyGraphRAG (more cost-efficient), DRIFT search, dynamic community selection, and auto-tuning for new domains.

In summary, GraphRAG represents a major advancement in making LLMs reason over large, private, narrative-rich datasets by leveraging graph structures for "global sensemaking." It's particularly useful when standard RAG gives incomplete or superficial answers.