DEV Community

Cover image for Building an AI Agent That Responds to Real-Time Events with AWS Bedrock, Kinesis, DynamoDB, and S3
Jubin Soni
Jubin Soni Subscriber

Posted on

Building an AI Agent That Responds to Real-Time Events with AWS Bedrock, Kinesis, DynamoDB, and S3

#aws #bedrock #kinesis #dynamodb

Most recommendation systems are batch jobs. They crunch last night's data, write a recommendations table, and serve it all day. That works fine until your user watches three thriller movies in a row at 9pm and your system is still recommending rom-coms because the batch hasn't run yet.

In this post I'll walk through building an agent system that reacts to streaming user behavior in real time using:

  • Amazon Kinesis to ingest and route events
  • AWS Lambda to process, enrich, and trigger reasoning
  • Amazon Bedrock as the reasoning and recommendation layer
  • DynamoDB to store user profiles and recommendation cache
  • S3 for raw event archiving and model artifacts

By the end you'll have an architecture where a user's recommendation set updates within seconds of their behavior changing.

Architecture Overview

Architecture description

The system has three layers:

Layer Services Responsibility
Ingest Kinesis Data Streams, Kinesis Firehose Capture and fan-out user events
Process & Reason Lambda, Amazon Bedrock Agent Enrich events, build context, invoke LLM
Store & Serve DynamoDB, S3 Persist profiles, cache recs, store artifacts

The key design decision is keeping the hot path (Kinesis → Lambda → Bedrock → DynamoDB) fully async and the serving path (API → DynamoDB cache) completely decoupled. The user never waits for Bedrock to respond; they get the last cached recommendation set while a fresh one is already being computed in the background.

Event Flow

Event description

Here's what happens end to end when a user clicks on a product:

  1. The app publishes a user.interaction event to Kinesis Data Streams
  2. Kinesis fans the event out to two consumers: Lambda Processor and Kinesis Firehose
  3. Firehose archives the raw event to S3 (cheap, durable, great for retraining later)
  4. Lambda enriches the event with user history from DynamoDB User Profiles, then invokes the Bedrock Agent
  5. The Bedrock Agent reasons over the enriched context (recent events + profile + item catalog embeddings from S3) and writes a fresh recommendation set to DynamoDB Rec Cache
  6. The client app reads recommendations from the cache via a lightweight Lambda API — no Bedrock latency in the hot path

Code: Publishing Events to Kinesis

This is your app-side producer. Keep it thin — just serialize and publish. Do all enrichment downstream.

import boto3
import json
import uuid
from datetime import datetime, timezone

kinesis = boto3.client('kinesis', region_name='us-east-1')

def publish_interaction(user_id: str, item_id: str, event_type: str, metadata: dict = {}):
    """
    Publish a user interaction event to Kinesis Data Streams.
    Partition key is user_id so all events for a user land on the same shard.
    """
    event = {
        'event_id':   str(uuid.uuid4()),
        'user_id':    user_id,
        'item_id':    item_id,
        'event_type': event_type,          # 'click', 'purchase', 'dwell', 'skip'
        'timestamp':  datetime.now(timezone.utc).isoformat(),
        'metadata':   metadata,
    }

    response = kinesis.put_record(
        StreamName='user-interactions',
        Data=json.dumps(event).encode('utf-8'),
        PartitionKey=user_id,              # consistent routing per user
    )

    return response['SequenceNumber']


# Example call
publish_interaction(
    user_id='u_8821',
    item_id='prod_thriller_042',
    event_type='purchase',
    metadata={'price': 14.99, 'category': 'thriller', 'session_id': 'sess_xyz'}
)
Enter fullscreen mode Exit fullscreen mode

Tip: Use user_id as the partition key so all events for a given user land on the same shard and arrive in order. This matters when Lambda is building a recency-ordered event window.

Code: Lambda Processor — Enrich and Invoke Bedrock

This is the core of the pipeline. The Lambda reads from the Kinesis stream, pulls user context from DynamoDB, and invokes the Bedrock Agent with a structured prompt.

import boto3
import json
import os
from datetime import datetime, timezone

dynamodb  = boto3.resource('dynamodb')
bedrock   = boto3.client('bedrock-agent-runtime', region_name='us-east-1')

profiles_table = dynamodb.Table(os.environ['PROFILES_TABLE'])   # DynamoDB User Profiles
rec_table      = dynamodb.Table(os.environ['REC_CACHE_TABLE'])  # DynamoDB Rec Cache

AGENT_ID      = os.environ['BEDROCK_AGENT_ID']
AGENT_ALIAS   = os.environ['BEDROCK_AGENT_ALIAS']
MAX_HISTORY   = 20  # last N events to include in context


def handler(event, context):
    for record in event['Records']:
        # Kinesis payload is base64-encoded
        payload = json.loads(record['kinesis']['data'])
        process_event(payload)


def process_event(payload: dict):
    user_id  = payload['user_id']
    item_id  = payload['item_id']
    evt_type = payload['event_type']

    # 1. Fetch user profile + recent history from DynamoDB
    response = profiles_table.get_item(Key={'user_id': user_id})
    profile  = response.get('Item', {'user_id': user_id, 'history': [], 'preferences': {}})

    # 2. Append current event and trim to window
    profile['history'].append({
        'item_id':    item_id,
        'event_type': evt_type,
        'timestamp':  payload['timestamp'],
        'metadata':   payload.get('metadata', {}),
    })
    profile['history'] = profile['history'][-MAX_HISTORY:]

    # 3. Write enriched profile back
    profiles_table.put_item(Item=profile)

    # 4. Build prompt for Bedrock Agent
    prompt = build_personalization_prompt(profile)

    # 5. Invoke Bedrock Agent
    agent_response = bedrock.invoke_agent(
        agentId=AGENT_ID,
        agentAliasId=AGENT_ALIAS,
        sessionId=user_id,           # session per user keeps conversational context
        inputText=prompt,
    )

    # 6. Parse streaming response chunks
    recommendations = parse_agent_response(agent_response)

    # 7. Write to recommendation cache
    rec_table.put_item(Item={
        'user_id':         user_id,
        'recommendations': recommendations,
        'generated_at':    datetime.now(timezone.utc).isoformat(),
        'ttl':             int(datetime.now(timezone.utc).timestamp()) + 3600,  # 1hr TTL
    })


def build_personalization_prompt(profile: dict) -> str:
    history_summary = '\n'.join([
        f"- [{e['event_type'].upper()}] item={e['item_id']} category={e['metadata'].get('category','unknown')}"
        for e in profile['history'][-10:]
    ])
    return f"""You are a real-time personalization agent.

User profile: {json.dumps(profile.get('preferences', {}))}

Recent interactions (most recent last):
{history_summary}

Based on this behavior, return exactly 5 personalized item recommendations as a JSON array.
Each item must include: item_id, category, reasoning (1 sentence), confidence_score (0-1).
Return only valid JSON. No explanation outside the JSON block."""


def parse_agent_response(agent_response) -> list:
    full_text = ''
    for event in agent_response['completion']:
        if 'chunk' in event:
            full_text += event['chunk']['bytes'].decode('utf-8')
    try:
        # Extract JSON from response
        start = full_text.index('[')
        end   = full_text.rindex(']') + 1
        return json.loads(full_text[start:end])
    except (ValueError, json.JSONDecodeError):
        return []
Enter fullscreen mode Exit fullscreen mode

Code: Serving Recommendations via Lambda API

The serving layer never touches Bedrock. It reads purely from the DynamoDB cache, keeping p99 latency well under 10ms.

import boto3
import json
import os
from datetime import datetime, timezone

dynamodb  = boto3.resource('dynamodb')
rec_table = dynamodb.Table(os.environ['REC_CACHE_TABLE'])

FALLBACK_RECS = ['popular_001', 'popular_002', 'popular_003']  # cold-start fallback


def handler(event, context):
    user_id = event['pathParameters']['userId']

    response = rec_table.get_item(Key={'user_id': user_id})
    item     = response.get('Item')

    if not item:
        # Cold start: user has no history yet
        return api_response(200, {
            'user_id':         user_id,
            'recommendations': FALLBACK_RECS,
            'source':          'fallback',
            'generated_at':    None,
        })

    age_seconds = (
        datetime.now(timezone.utc) -
        datetime.fromisoformat(item['generated_at'])
    ).total_seconds()

    return api_response(200, {
        'user_id':         user_id,
        'recommendations': item['recommendations'],
        'source':          'cache',
        'generated_at':    item['generated_at'],
        'cache_age_sec':   int(age_seconds),
    })


def api_response(status: int, body: dict) -> dict:
    return {
        'statusCode': status,
        'headers': {
            'Content-Type':                'application/json',
            'Access-Control-Allow-Origin': '*',
        },
        'body': json.dumps(body),
    }
Enter fullscreen mode Exit fullscreen mode

Service Comparison: Why Each AWS Service?

Service Why it's here Alternative considered
Kinesis Data Streams Ordered, replayable, millisecond latency fan-out SQS (no ordering guarantee per user), EventBridge (higher latency)
Kinesis Firehose Zero-code delivery to S3 for archiving Writing to S3 directly in Lambda (adds failure surface)
Lambda Event-driven, scales to 0, tight Kinesis integration ECS Fargate (overkill for stateless enrichment)
Amazon Bedrock Managed LLM with agent runtime, no infra to maintain Self-hosted model on SageMaker (more control, much more ops)
DynamoDB Single-digit ms reads, TTL support, scales automatically RDS (too slow for hot path), ElastiCache (extra cost for separate store)
S3 Cheap durable archive + model artifact store DynamoDB for raw events (expensive and unnecessary)

Things to Watch in Production

Bedrock latency is variable. Claude Sonnet typically responds in 1-4 seconds but can spike. Since recs are written async to cache, this doesn't affect user-facing latency, but it does affect freshness. Monitor bedrock:InvokeAgent duration in CloudWatch.

Kinesis shard scaling. One shard handles 1MB/s write or 1000 records/s. At 10k active users you'll need to plan shard count carefully. Use Enhanced Fan-Out if you have multiple Lambda consumers reading the same stream.

DynamoDB TTL for cache eviction. The serving Lambda sets a 1-hour TTL on each rec entry. If Bedrock hasn't updated the cache in over an hour (e.g. Lambda errors), users fall back to the popular items list. Adjust TTL based on how stale you can tolerate.

Cold start users. New users have no history so the Bedrock prompt has nothing useful to reason over. I recommend a popularity-based fallback as shown in the serving Lambda, and switching to personalized recs after the user's first 3-5 interactions.

Wrapping Up

The pattern here is worth generalizing: keep the reasoning layer (Bedrock) fully off the hot serving path. Write results to a fast cache (DynamoDB), serve from the cache, and let the agent pipeline update it continuously in the background. This gives you the intelligence of an LLM-powered agent without the latency of one.

The same pattern applies to fraud scoring, content moderation queues, ops alerting — anywhere you need a reasoning system that reacts to real-time streams without blocking the user experience.

References

Top comments (0)