Caching is one of the most effective ways to improve application performance while reducing costs. In this guide, I'll show you how to implement a cache-aside pattern using DynamoDB, ElastiCache Redis, AWS Lambda, and API Gateway - all provisioned with Terraform.
📘 Part 1: Infrastructure Setup
1. DynamoDB Table
First, let's create a Products table in DynamoDB:
resource "aws_dynamodb_table" "products" {
name = "Products"
billing_mode = "PAY_PER_REQUEST"
hash_key = "productId"
attribute {
name = "productId"
type = "S"
}
}
2. ElastiCache Redis Cluster
We'll deploy Redis inside a VPC for better security and performance:
resource "aws_elasticache_cluster" "products_cache" {
cluster_id = "products-cache"
engine = "redis"
node_type = "cache.t3.micro"
num_cache_nodes = 1
parameter_group_name = "default.redis6.x"
engine_version = "6.x"
port = 6379
security_group_ids = [aws_security_group.redis.id]
subnet_group_name = aws_elasticache_subnet_group.redis.name
}
resource "aws_elasticache_subnet_group" "redis" {
name = "redis-subnet-group"
subnet_ids = [aws_subnet.private_a.id, aws_subnet.private_b.id]
}
3. Networking Configuration
Proper VPC setup is crucial:
resource "aws_vpc" "main" {
cidr_block = "10.0.0.0/16"
}
resource "aws_subnet" "private_a" {
vpc_id = aws_vpc.main.id
cidr_block = "10.0.1.0/24"
availability_zone = "us-east-1a"
}
resource "aws_security_group" "redis" {
name = "redis-sg"
description = "Allow access to Redis"
vpc_id = aws_vpc.main.id
ingress {
from_port = 6379
to_port = 6379
protocol = "tcp"
security_groups = [aws_security_group.lambda.id]
}
}
📘 Part 2: Lambda Function Logic
Here's our Python Lambda function implementing the cache-aside pattern:
import os
import json
import boto3
import redis
from datetime import datetime
# Initialize clients
dynamodb = boto3.resource('dynamodb')
products_table = dynamodb.Table('Products')
# Redis connection
redis_client = redis.Redis(
host=os.environ['REDIS_HOST'],
port=6379,
decode_responses=True
)
def lambda_handler(event, context):
product_id = event['pathParameters']['productId']
cache_key = f"product:{product_id}"
# Try to get from Redis first
cached_product = redis_client.get(cache_key)
if cached_product:
print("Cache hit!")
return {
'statusCode': 200,
'body': cached_product
}
print("Cache miss - fetching from DynamoDB")
# Get from DynamoDB
response = products_table.get_item(Key={'productId': product_id})
if 'Item' not in response:
return {'statusCode': 404, 'body': 'Product not found'}
product = response['Item']
product_json = json.dumps(product)
# Cache with 5 minute TTL
redis_client.setex(cache_key, 300, product_json)
return {
'statusCode': 200,
'body': product_json
}
📘 Part 3: API Gateway Configuration
Let's expose our Lambda through API Gateway:
resource "aws_api_gateway_rest_api" "products_api" {
name = "products-api"
}
resource "aws_api_gateway_resource" "product" {
rest_api_id = aws_api_gateway_rest_api.products_api.id
parent_id = aws_api_gateway_rest_api.products_api.root_resource_id
path_part = "product"
}
resource "aws_api_gateway_resource" "product_id" {
rest_api_id = aws_api_gateway_rest_api.products_api.id
parent_id = aws_api_gateway_resource.product.id
path_part = "{productId}"
}
resource "aws_api_gateway_method" "get_product" {
rest_api_id = aws_api_gateway_rest_api.products_api.id
resource_id = aws_api_gateway_resource.product_id.id
http_method = "GET"
authorization = "NONE"
}
resource "aws_api_gateway_integration" "lambda" {
rest_api_id = aws_api_gateway_rest_api.products_api.id
resource_id = aws_api_gateway_resource.product_id.id
http_method = aws_api_gateway_method.get_product.http_method
integration_http_method = "POST"
type = "AWS_PROXY"
uri = aws_lambda_function.get_product.invoke_arn
}
📘 Part 4: Monitoring and Optimization
1. Adding TTLs
We already implemented TTLs in our Lambda function (setex with 300 seconds), but let's add CloudWatch metrics to track cache performance:
from aws_lambda_powertools import Metrics
metrics = Metrics()
def lambda_handler(event, context):
# ... existing code ...
if cached_product:
metrics.add_metric(name="CacheHits", unit="Count", value=1)
# ... return cached product ...
else:
metrics.add_metric(name="CacheMisses", unit="Count", value=1)
# ... fetch from DynamoDB ...
2. Terraform for Monitoring
Add CloudWatch alarms and dashboards:
resource "aws_cloudwatch_dashboard" "cache" {
dashboard_name = "cache-performance"
dashboard_body = jsonencode({
widgets = [
{
type = "metric"
x = 0
y = 0
width = 12
height = 6
properties = {
metrics = [
["AWS/Lambda", "CacheHits", "FunctionName", aws_lambda_function.get_product.function_name],
["AWS/Lambda", "CacheMisses", "FunctionName", aws_lambda_function.get_product.function_name]
]
period = 300
stat = "Sum"
region = "us-east-1"
title = "Cache Performance"
}
}
]
})
}
3. Complete Terraform Workflow
For a production setup, add a CI/CD pipeline:
resource "aws_codepipeline" "deploy_pipeline" {
name = "products-api-deployment"
role_arn = aws_iam_role.codepipeline.arn
artifact_store {
location = aws_s3_bucket.artifacts.bucket
type = "S3"
}
stage {
name = "Source"
action {
name = "Source"
category = "Source"
owner = "ThirdParty"
provider = "GitHub"
version = "1"
output_artifacts = ["source_output"]
configuration = {
Owner = "your-github-org"
Repo = "products-api"
Branch = "main"
OAuthToken = var.github_token
}
}
}
stage {
name = "Terraform"
action {
name = "Apply"
category = "Build"
owner = "AWS"
provider = "CodeBuild"
input_artifacts = ["source_output"]
version = "1"
configuration = {
ProjectName = aws_codebuild_project.terraform.name
}
}
}
}
Results and Observations
After implementing this architecture, you should see:
- Average latency reduction from ~100ms (DynamoDB) to ~5ms (Redis) for cache hits
- Reduced DynamoDB RCU consumption (and costs) by 80-90% for frequently accessed items
- More consistent performance under load
Final Thoughts
This cache-aside pattern is just one of many ways to optimize DynamoDB performance. For production workloads, consider:
- Adding write-through caching for data modifications
- Implementing cache invalidation strategies
- Monitoring Redis memory usage and eviction policies
- Considering DAX for alternative DynamoDB caching
The complete Terraform code is available in this GitHub repo.
Have you implemented similar caching patterns? Share your experiences in the comments!
Top comments (0)