What is Amazon Bedrock?
as per the official documentation of AWS the definition of Amazon bedrock as below:
"Amazon Bedrock is a fully managed service that provides secure, enterprise-grade access to high-performing foundation models from leading AI companies, enabling you to build and scale generative AI applications."
so you can consider it as a managed API hub that allow managed access to all available models in the market with an easy way and low configuration.
before proceeeding with the article we will list some definitions that you always see in most of the market and their use cases
- RAG ( Knowledge bases ): Managed Retrieval-Augmented Generation pipeline and used for enterprise document Q&A, internal wikis
- Agents & AgentCore: Autonomous multi-step orchestration and tool calling and used for automated workflows, customer refunds, IT ops
- Guardrails: Enterprise safety layer, PII masking, hallucination prevention and used for compliance enforcement, prompt injection defense.
- Bedrock Flows: Visual and code-defined workflow builder and used for Content pipelines, multi-prompt approvals.
- Model Customization: Fine-tuning, pre-training, and model distillation and used for domain-specific models (Legal, Medical, Finance).
- Prompt caching: Prompt caching is an optional feature that you can use with supported models on Amazon Bedrock to reduce inference response latency and input token costs. By adding portions of your context to a cache, the model can use the cache to skip recomputation of inputs, allowing Bedrock to share in the compute savings and lower your response latencies.
Architecture Diagram
So basically this diagram explains how the cache will work as we have the below
Tier 1 (Application Level - Amazon ElastiCache): Bypasses Amazon Bedrock completely to eliminate 100% of invocations and tokens for identical or semantically similar queries.
Tier 2 (Model Level - Amazon Bedrock Prompt Caching): Reduces input token costs by up to 90% and latency by up to 85% when calls must hit Bedrock by reusing internal KV attention states for long prompt prefixes
Sequence Flow Diagram
so for step by step detailed steps of the sequence diagram:
Phase 1: Incoming Request & Tier 1 Lookup
Step 1 (Client / App --> Application Service / Lambda):
The client submits a user prompt to the backend service.
Step 2 (Application Service / Lambda --> Amazon ElastiCache):
Before touching Amazon Bedrock, the application backend inspects ElastiCache:
Computes a SHA-256 hash of the prompt for an exact match.
Runs a K-Nearest Neighbors vector search to detect semantic similarity.
Path A: Tier 1 Cache Hit (ElastiCache)
If the query or a semantically equivalent query exists in ElastiCache:
Step 3 (Amazon ElastiCache --> Application Service / Lambda):
ElastiCache returns the cached JSON response.Step 4 3a (Application Service / Lambda --> Client / App):
The application returns the answer directly to the user.
Latency: ~5–20 ms
Cost: $0 in Bedrock LLM tokens.
Path B: Tier 1 Cache Miss (Proceed to Amazon Bedrock)
If ElastiCache does not have the answer, the request moves down to Tier 2:
- Step 5 3b (Application Service / Lambda --> Amazon Bedrock Runtime): The backend invokes the Bedrock converse() API, attaching a cachePoint checkpoint inside the payload to separate static system context (documents/instructions) from the user's dynamic query.
Sub-Branch B1: Tier 2 Prompt Cache Hit (Bedrock)
If Bedrock has already cached the static prefix (KV attention state) within its 5-minute rolling TTL:
Step 6 4a (Amazon Bedrock Runtime --> Foundation Model):
Bedrock instructs the Foundation Model (Amazon Nova) to reuse the precomputed Key-Value (KV) attention states, skipping prompt processing for the prefix.
Step 7 (Foundation Model --> Amazon Bedrock Runtime):
The model generates new tokens based only on the dynamic query suffix.
Step 8 (Amazon Bedrock Runtime --> Application Service / Lambda):
Bedrock sends the completion back to the application.
Cost Savings: 90% discount on input tokens behind the cachePoint.
Speed: Significantly faster Time-To-First-Token (TTFT).
Sub-Branch B2: Tier 2 Full Cache Miss (Bedrock)
If this is the first time the static prompt is being processed or the 5-minute cache TTL expired:
Step 9 4b (Amazon Bedrock Runtime --> Foundation Model):
Bedrock computes the entire prompt (static prefix + dynamic query) and writes the static prefix into Bedrock's ephemeral cache for future requests.
Step 10 (Foundation Model --> Amazon Bedrock Runtime):
The model generates completion tokens.
Step 11 (Amazon Bedrock Runtime --> Application Service / Lambda):
Bedrock returns the full completion response to the application backend at standard pricing.
Phase 2: Post-Processing & Cache Write-Back
Step 12 5 (Application Service / Lambda --> Amazon ElastiCache):
Now that a fresh response was generated by Bedrock, the Lambda backend computes its embedding vector and stores the prompt-response pair in ElastiCache. Future identical or similar queries will now hit Tier 1 (Path A).
Step 13 6 (Application Service / Lambda-->Client / App):
The backend delivers the final generated response to the client.
Wrapping Up Part 1 & What’s Next in Part 2
By implementing this Two-Tier Caching Strategy, we move away from treating LLM infrastructure as a black box and start optimizing it with classical software engineering patterns.
Tier 1 (ElastiCache): Cuts out 100% of token costs and invocations for identical or semantically duplicate queries at sub-20ms speeds.
Tier 2 (Bedrock Prompt Caching): Slashes input token costs by up to 90% and latency by up to 85% when long static context must hit the model.
Moving to part 2 where we are going for the hands-on and implementation.
Top comments (0)