Sujitha Rasamsetty

Posted on Jan 2

Serverless Logging for Amazon Bedrock: From Invocation to Insight

#aws #awscommunity #awsbedrock

A hands-on, step-by-step guide to setting up model response logging, error alerts, and actionable insights using AWS Lambda, Amazon Bedrock, S3, CloudWatch, and Athena.

🧠 Why This Matters

As Generative AI adoption grows within enterprises, using services like Amazon Bedrock becomes a preferred approach for building scalable LLM-powered applications. However, as models get integrated into business-critical flows, tracking what is invoked, how it's performing, and where it fails becomes vital.

This PoC demonstrates a serverless, zero-infrastructure approach to:

Log model prompts and responses
Catch and alert on errors in real-time
Store and analyze logs with Athena

For stakeholders, it offers:

Engineering: Track model behavior over time.
Product: See usage patterns or prompt gaps.
Compliance: Maintain traceability of model interactions.

Why We Built This PoC

As enterprises integrate generative AI into realworld applications from chatbots to knowledge assistants ensuring operational visibility becomes critical.

However, when using Amazon Bedrock, developers typically focus only on model integration. What’s often missed is observability — knowing:

What prompts are being sent?
What responses are returned?
What errors occur and when?

This PoC answers that gap.

Purpose

We built this PoC to:

Track model usage (prompts, responses)
Monitor for failures (e.g., access issues, quota breaches)
Alert stakeholders in real-time
Store structured logs for audit/compliance
Enable analytics using SQL (Athena),without extra infrastructure

How This Helps the IT Industry

Cloud Architects / DevOps: Build fault-tolerant LLM apps with integrated monitoring and logging pipelines.
Product Teams: Analyze prompts, user behavior, or failure trends to improve UX.
Security & Compliance: Ensure that model usage is tracked and auditable,a growing requirement in regulated industries.
Startups: Use this serverless pattern to scale quickly without managing infrastructure.

Architecture Overview

Here’s what we’re building:

Architecture Overview: Serverless Logging for Amazon Bedrock

This architecture illustrates a serverless logging and monitoring pipeline for Amazon Bedrock model invocations. It is designed to capture model responses, log interactions, and notify stakeholders of any issues, all without maintaining any infrastructure.

Here's what happens step-by-step:

End User

Initiates a request (e.g., a prompt) that is handled by a serverless backend.
AWS Lambda

A Lambda function acts as the compute layer, invoking the Amazon Bedrock model (e.g., Claude, Titan, or any supported foundation model) and capturing the response.
Amazon Bedrock

Processes the request and returns the generated response back to Lambda.
S3 (Audit Logs)

The prompt and response pair (along with a timestamp) is stored in Amazon S3 in structured JSON format, organized into success and error folders for easy retrieval and analysis.
Athena

Athena queries the audit logs directly from S3 to extract insights, track usage, or debug issues using SQL, without needing to move data.
CloudWatch

Monitors the Lambda function’s execution and captures any error logs or anomalies.
SNS (Simple Notification Service)

When an error is detected (e.g., failed Bedrock call or S3 write), CloudWatch triggers an alarm that notifies stakeholders via an SNS topic (email/SMS/Slack).

Step 1: Set Up IAM Roles and Policies

Lambda Execution Role

Create a role with the following:

AmazonBedrockFullAccess (or scoped-down invoke permissions)
AmazonS3FullAccess (or fine-grained access to specific bucket/prefix)
CloudWatchLogsFullAccess

Attach the role to the Lambda function.

S3 Bucket Policy

Allow the Lambda role to PutObject to the audit log bucket:

{
  "Effect": "Allow",
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::bedrock-model-audit-logs-sujitha/bedrock-backups/*"
}

Step 2: Create Lambda Function

Use Python for the function.

Key features:

Receives prompt via event
Invokes Bedrock model (Titan / Claude / others)
Stores response in S3 (success or error folder)
Logs to CloudWatch
Sends SNS alert on failure

import json
import boto3
import uuid
import datetime

bedrock = boto3.client("bedrock-runtime")
s3 = boto3.client("s3")

S3_BUCKET = "bedrock-model-audit-logs-sujitha"
S3_PREFIX = "bedrock-backups"

def lambda_handler(event, context):
    prompt = event.get("prompt", "Default prompt")
    timestamp = datetime.datetime.now().isoformat()
    interaction_id = str(uuid.uuid4())

    try:
        # Bedrock inference
        response = bedrock.invoke_model(
            modelId="amazon.titan-text-express-v1",
            body=json.dumps({
                "inputText": prompt,
                "textGenerationConfig": {
                    "maxTokenCount": 500,
                    "temperature": 0.7,
                    "topP": 0.9
                }
            }),
            contentType="application/json",
            accept="application/json"
        )

        result = json.loads(response["body"].read().decode("utf-8"))
        output = result.get("results", [{}])[0].get("outputText", "")

        # Save success log
        s3.put_object(
            Bucket=S3_BUCKET,
            Key=f"{S3_PREFIX}/success/interaction-{interaction_id}.json",
            Body=json.dumps({
                "timestamp": timestamp,
                "prompt": prompt,
                "response": output
            })
        )

        return {
            "statusCode": 200,
            "body": json.dumps({
                "prompt": prompt,
                "response": output
            })
        }

    except Exception as e:
        error_message = str(e)

        # Save error log
        s3.put_object(
            Bucket=S3_BUCKET,
            Key=f"{S3_PREFIX}/error/interaction-{interaction_id}.json",
            Body=json.dumps({
                "timestamp": timestamp,
                "prompt": prompt,
                "error": error_message
            })
        )

        return {
            "statusCode": 500,
            "body": json.dumps({"error": error_message})
        }

Step 3: Create S3 Folder Structure

CloudWatch Metrics & Error Alerts

To ensure production-grade reliability, we monitor our Lambda function using Amazon CloudWatch.

Custom Metric: `Errors`

We use the built-in Lambda metric:

Namespace: AWS/Lambda
Metric name: Errors
Dimension: FunctionName = bedrock-model-backup
Period: 10 seconds
Statistic: Average

This metric tracks how often the Lambda function fails — for example, when the Bedrock invocation fails or when S3 writes are denied.

🚨 Alarm Setup

A CloudWatch alarm is created with:

Threshold: ≥ 1 error
Evaluation Period: 1 datapoint (10 seconds)
TreatMissingData: missing
Alarm Action: Triggers an SNS Topic to notify stakeholders

This ensures real-time notifications when even a single error occurs helpful during early testing or production monitoring.

You’ll receive emails like this from Amazon SNS, alerting you to issues immediately:

ALARM: "bedrock-error-alarm"

Threshold Crossed: 1 out of the last 1 datapoints was ≥ 1.0

FunctionName = bedrock-model-backup

Why This Helps

Engineering teams get real-time visibility into model failures.
Product managers can track stability over time.
Stakeholders are kept in the loop via email/SMS.

Step 4: Set Up SNS Alert

Create an SNS topic: bedrock-model-errors.
Subscribe to your email to the topic.
Use the topic ARN in Lambda.

Step 5: Set Up Athena for Log Querying

Create Database

CREATE DATABASE IF NOT EXISTS bedrock_audit_logs;

Create Table (Success Logs)

CREATE EXTERNAL TABLE IF NOT EXISTS bedrock_audit_logs.success_logs (
  `timestamp` string,
  `prompt` string,
  `response` string
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
  'ignore.malformed.json' = 'true'
)
LOCATION 's3://bedrock-model-audit-logs-sujitha/bedrock-backups/success/.'
TBLPROPERTIES ('has_encrypted_data'='false');

Repeat similarly for error_logs table with an error column.

Run Queries

SELECT prompt, response, timestamp
FROM bedrock_audit_logs.success_logs
ORDER BY timestamp DESC;

Conclusion: Wrapping It All Together

In a world rapidly adopting Generative AI, observability is not optional especially when models power customer-facing or compliance-sensitive applications.

This Proof of Concept (PoC) offers a fully serverless, production-grade pattern to:

Log every Amazon Bedrock prompt and response
Detect and alert on errors in real-time
Store structured audit logs for compliance and monitoring
Run post-hoc analysis using SQL, without managing infrastructure

By combining Lambda, Amazon Bedrock, S3, CloudWatch, SNS, and Athena, we’ve created a scalable blueprint that any team can adopt and extend.

Whether you're building internal tools or external-facing AI apps, this architecture ensures visibility, accountability, and agility.

If you found this helpful, don’t forget to ❤️ and follow for more serverless + GenAI guides!

Questions or suggestions? Drop them in the comments 👇

About the Author

Sujitha Rasamsetty is an AWS Community Builder in AI Engineering and a Data Scientist at Relanto, with a strong passion for emerging technologies in cloud computing and artificial intelligence.

In her role, she works hands on with data, cloud architectures, and AI-driven solutions, focusing on building scalable, secure, and production-ready systems. Her interests span across machine learning, generative AI, cloud-native architectures, and data platforms, where she enjoys bridging the gap between advanced analytics and real-world cloud implementations.

Sujitha actively shares her learning and experiences with the community through blogs, discussions, and technical knowledge sharing, with a strong belief in learning in public and growing together as a community.

📌 Let’s Connect:

For updates, discussions, and questions, feel free to connect with her on LinkedIn:

👉 https://www.linkedin.com/in/sujitharasamsetty/

DEV Community

Serverless Logging for Amazon Bedrock: From Invocation to Insight

🧠 Why This Matters

Why We Built This PoC

Purpose

How This Helps the IT Industry

Architecture Overview

Architecture Overview: Serverless Logging for Amazon Bedrock

Here's what happens step-by-step:

Step 1: Set Up IAM Roles and Policies

Lambda Execution Role

S3 Bucket Policy

Step 2: Create Lambda Function

Key features:

Step 3: Create S3 Folder Structure

CloudWatch Metrics & Error Alerts

Custom Metric: `Errors`

🚨 Alarm Setup

Why This Helps

Step 4: Set Up SNS Alert

Step 5: Set Up Athena for Log Querying

Create Database

Create Table (Success Logs)

Run Queries

Conclusion: Wrapping It All Together

About the Author

Top comments (0)

🧠 Why This Matters

Why We Built This PoC

Purpose

How This Helps the IT Industry

Architecture Overview

Architecture Overview: Serverless Logging for Amazon Bedrock

Here's what happens step-by-step:

Step 1: Set Up IAM Roles and Policies

Lambda Execution Role

S3 Bucket Policy

Step 2: Create Lambda Function

Key features:

Step 3: Create S3 Folder Structure

CloudWatch Metrics & Error Alerts

Custom Metric: Errors

🚨 Alarm Setup

Why This Helps

Step 4: Set Up SNS Alert

Step 5: Set Up Athena for Log Querying

Create Database

Create Table (Success Logs)

Run Queries

Conclusion: Wrapping It All Together

About the Author

Custom Metric: `Errors`