Assist In Root Cause Analysis | 🏗️ Build A Debugging Dashboard

Exam Guide: Developer - Associate
🏗️ Domain 4: Troubleshooting And Optimization
📘 Task 1: Assist In A Root Cause Analysis

When something breaks in production, you need to find out why. Fast. This task tests your ability to use CloudWatch Logs, Logs Insights, CloudWatch metrics, dashboards, and CloudTrail to debug application issues. Writing Logs Insights queries. Understanding Lambda-specific log fields. Using Embedded Metric Format for custom metrics. And recognizing common failure patterns across AWS service integrations.

---

📘 Concepts

CloudWatch Logs Architecture

Component	What It Is	Example
Log Group	A collection of log streams that share retention and access settings	/aws/lambda/my-function
Log Stream	A sequence of log events from a single source	2024/01/15/[$LATEST]abc123
Log Event	A single log entry with a timestamp and message	{"level":"ERROR","message":"DynamoDB timeout"}
Retention	How long logs are kept (1 day to 10 years, or never expire)	30 days for production
Subscription Filter	Real-time stream of log events to another destination	Stream to Kinesis, Lambda, or OpenSearch
Metric filter	Extract metric values from log text patterns	Count ERROR occurrences

💡 Lambda automatically creates a log group named /aws/lambda/{function-name}. Each function instance creates a new log stream. Log retention defaults to "Never expire". Always set a retention policy to control costs. You need logs:CreateLogGroup and logs:PutLogEvents permissions in the Lambda execution role.

CloudWatch Logs Insights Query Syntax

Logs Insights is a query language for searching and analysing log data. The exam tests specific query commands:

Command	Purpose	Example
fields	Select which fields to display	fields @timestamp, @message
filter	Filter log events by condition	filter @message like /ERROR/
stats	Aggregate data	stats count(*) by bin(5m)
sort	Order results	sort @timestamp desc
limit	Cap the number of results	limit 20
parse	Extract fields from unstructured log text	parse @message "user=* action=*" as user, action
display	Choose which fields appear in results	display @timestamp, @message

Common Query Patterns

Find the most recent errors

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 25

Count errors per 5-minute window

filter @message like /ERROR/
| stats count(*) as errorCount by bin(5m)
| sort errorCount desc

Find slow Lambda invocations

filter @duration > 5000
| fields @requestId, @duration, @billedDuration, @memorySize, @maxMemoryUsed
| sort @duration desc
| limit 10

Parse custom log fields

parse @message '"level":"*","requestId":"*","error":"*"' as level, reqId, error
| filter level = "ERROR"
| stats count(*) by error

Find cold starts

filter @type = "REPORT" and @initDuration > 0
| fields @requestId, @initDuration, @duration, @memorySize
| sort @initDuration desc
| limit 20

💡The parse command uses glob patterns (*) to extract fields from unstructured text. The stats command supports count(), sum(), avg(), min(), max(), pct() (percentiles). Use bin() to group by time intervals. The @ prefix denotes system-generated fields like @timestamp, @message, @logStream, etc.

Lambda-Specific Log Fields

Lambda automatically adds these fields to REPORT lines:

Field	What It Contains	Why It Matters
@type	Log line type: START, END, REPORT	Filter for REPORT to get invocation summaries
@requestId	Unique invocation ID	Correlate logs for a single invocation
@duration	Actual execution time (ms)	Performance monitoring
@billedDuration	Rounded up to nearest ms (or 1ms minimum)	Cost tracking
@memorySize	Configured memory (MB)	Right-sizing analysis
@maxMemoryUsed	Peak memory used (MB)	Detect memory pressure
@initDuration	Cold start initialization time (ms)	Only present on cold starts
@xrayTraceId	X-Ray trace ID	Link logs to traces

Memory right-sizing query

filter @type = "REPORT"
| stats avg(@maxMemoryUsed) as avgMem,
        max(@maxMemoryUsed) as peakMem,
        avg(@memorySize) as configuredMem,
        avg(@maxMemoryUsed) / avg(@memorySize) * 100 as utilizationPct

💡 If @maxMemoryUsed is consistently close to @memorySize, the function may be running out of memory. If it's consistently low (under 50%), you're over-provisioned and wasting money. The @initDuration field only appears on cold starts. If you see it frequently, consider provisioned concurrency.

CloudWatch Embedded Metric Format (EMF) vs PutMetricData

There are two ways to publish custom metrics.

Aspect	Embedded Metric Format (EMF)	PutMetricData API
How It Works	Write structured JSON to stdout: CloudWatch extracts metrics automatically	Call the CloudWatch API directly
Latency	Async (written with logs)	Sync (API call blocks execution)
Cost	Log ingestion cost only (no per-metric API charge)	$0.01 per 1,000 PutMetricData API calls
Dimensions	Up to 30 dimensions	Up to 30 dimensions
Resolution	Standard (60s) or high (1s)	Standard (60s) or high (1s)
Batching	Automatic (one log line = one metric data point)	Manual (up to 1,000 values per API call)
Lambda Overhead	None. Just a print statement	API call adds latency to each invocation
Best For	Lambda functions, high-throughput applications	EC2 applications, infrequent metric publishing

EMF format

import json

def lambda_handler(event, context):
    # Your business logic here
    order_total = process_order(event)

    # Emit EMF metric. CloudWatch extracts this automatically
    print(json.dumps({
        "_aws": {
            "Timestamp": 1234567890000,
            "CloudWatchMetrics": [{
                "Namespace": "OrderService",
                "Dimensions": [["Environment", "OrderType"]],
                "Metrics": [
                    {"Name": "OrderTotal", "Unit": "None"},
                    {"Name": "ProcessingTime", "Unit": "Milliseconds"}
                ]
            }]
        },
        "Environment": "production",
        "OrderType": "standard",
        "OrderTotal": order_total,
        "ProcessingTime": 145
    }))

PutMetricData approach

import boto3

cloudwatch = boto3.client('cloudwatch')

def lambda_handler(event, context):
    order_total = process_order(event)

    # Sync API call — adds latency
    cloudwatch.put_metric_data(
        Namespace='OrderService',
        MetricData=[{
            'MetricName': 'OrderTotal',
            'Value': order_total,
            'Unit': 'None',
            'Dimensions': [
                {'Name': 'Environment', 'Value': 'production'},
                {'Name': 'OrderType', 'Value': 'standard'}
            ]
        }]
    )

💡> EMF is the preferred approach for Lambda because it adds zero latency. You're just writing to stdout, which Lambda already sends to CloudWatch Logs. CloudWatch automatically extracts the metrics from the structured JSON. Use PutMetricData when you need to publish metrics from EC2 or when you need more control over metric timing.

Common Integration Failure Patterns

Scenarios to identify the root cause. Know these patterns:

Symptom	Likely Cause	How to Debug	Fix
502 Bad Gateway (API GW)	Lambda returned invalid response format	Check Lambda logs for response structure	Return statusCode, headers, body
504 Gateway Timeout (API GW)	Lambda exceeded API Gateway's 29-second timeout	Check @duration in logs	Reduce Lambda timeout, optimize code, or use async
429 Too Many Requests	Lambda concurrency throttled or API GW rate limit	Check Throttles metric	Increase reserved concurrency or request limit increase
Access Denied	Missing IAM permissions	Check CloudTrail for AccessDenied events	Update IAM policy
Task timed out (Lambda)	Function exceeded its configured timeout	Check @duration vs timeout setting	Increase timeout or optimize code
Out of memory	Function exceeded configured memory	Check @maxMemoryUsed vs @memorySize	Increase memory allocation
Connection timeout	Lambda in VPC can't reach internet or service	Check VPC config, NAT Gateway, security groups	Add NAT Gateway or VPC endpoint
ResourceNotFoundException	DynamoDB table or resource doesn't exist in the region	Check region configuration	Verify resource exists in the correct region

502 Bad Gateway Deep Dive

The most common Lambda + API Gateway error. API Gateway expects this exact response format:

# CORRECT: API Gateway accepts this
return {
    'statusCode': 200,
    'headers': {'Content-Type': 'application/json'},
    'body': json.dumps({'message': 'success'})  # body must be a STRING
}

# WRONG: causes 502
return {
    'statusCode': 200,
    'body': {'message': 'success'}  # body is a dict, not a string!
}

# WRONG: causes 502
return "success"  # not the expected format at all

# WRONG: causes 502 (unhandled exception)
def handler(event, context):
    raise Exception("oops")  # no try/except = 502

💡 A 502 from API Gateway almost always means the Lambda function returned a response that doesn't match the expected format, or the function threw an unhandled exception. Check the Lambda logs first: the error will be there. The API Gateway execution logs (if enabled) will show "Malformed Lambda proxy response."

CloudTrail for Debugging Permission Issues

CloudTrail Feature	What It Does	Use Case
Event History	Last 90 days of management events (free)	Quick lookup of recent API calls
Trail	Continuous logging to S3 (all events)	Long-term audit, compliance
Data Events	S3 object-level and Lambda invocation events	Debug specific resource access
Insights	Detect unusual API activity	Identify anomalous patterns

Finding AccessDenied events

1. Open CloudTrail console → Event history
2. Filter by Event name or Error code
3. Look for events with errorCode: AccessDenied or errorCode: UnauthorizedAccess
4. The event details show:

• userIdentity: who made the call (which role/user)
• eventName: what API was called (e.g., PutItem)
• requestParameters: what resource was targeted
• errorMessage: the specific permission that's missing

💡 CloudTrail logs every AWS API call. When you get an AccessDenied error, CloudTrail tells you exactly which principal tried to call which API on which resource and which permission was missing. This is the fastest way to debug IAM issues. Event history is free and covers the last 90 days.

Deployment Failure Troubleshooting

Failure Type	Where to Look	Common Causes
CloudFormation stack failure	CloudFormation Events tab	IAM permissions, resource limits, invalid template
CodeBuild failure	CodeBuild build logs	Missing dependencies, test failures, buildspec syntax
CodeDeploy failure	CodeDeploy deployment logs	Hook script errors, health check failures, timeout
Lambda deployment failure	CloudWatch Logs + CodeDeploy	New version crashes, alarm triggers rollback
SAM deploy failure	CloudFormation Events + SAM CLI output	Transform errors, packaging issues, capability missing

💡 CloudFormation failures show the specific resource and error in the Events tab. Look for CREATE_FAILED or UPDATE_FAILED status. The most common cause is insufficient IAM permissions because the CloudFormation role doesn't have permission to create the resource. For SAM, make sure you include CAPABILITY_IAM and CAPABILITY_AUTO_EXPAND when deploying.

---

💡 Build A Debugging Dashboard

Build a Debugging Dashboard from scratch using the AWS Console:

• A Lambda function that emits structured logs and EMF custom metrics
• CloudWatch Logs Insights queries that find errors, slow invocations, and cold starts
• A CloudWatch dashboard displaying key application metrics
• Simulated failures (timeout, permission denied, 502) with debugging walkthroughs
• CloudTrail queries to find access denied events

---

Prerequisites

• An AWS Account
• Python

---

Part I

Create a Lambda Function with Structured Logging and EMF

Step 01: Create the Function

Open the Lambda console → Create function

• Function name: DebugDemoFunction
• Runtime: Python 3.13

Click Create function

Step 02: Add Structured Logging with EMF

import json
import time
import os
import random

def emit_emf_metric(metric_name, value, unit, dimensions):
    """Emit a CloudWatch metric using Embedded Metric Format."""
    print(json.dumps({
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": "DebugDemo",
                "Dimensions": [list(dimensions.keys())],
                "Metrics": [{"Name": metric_name, "Unit": unit}]
            }]
        },
        **dimensions,
        metric_name: value
    }))

def lambda_handler(event, context):
    start_time = time.time()
    action = event.get('action', 'process')
    request_id = context.aws_request_id

    # Structured log entry
    log_entry = {
        "level": "INFO",
        "requestId": request_id,
        "action": action,
        "timestamp": time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
        "functionName": context.function_name,
        "memoryLimit": context.memory_limit_in_mb
    }

    try:
        if action == 'slow':
            # Simulate a slow operation
            delay = random.uniform(2, 5)
            log_entry["level"] = "WARN"
            log_entry["message"] = f"Slow operation detected: {delay:.1f}s"
            time.sleep(delay)

        elif action == 'error':
            # Simulate an error
            raise ValueError("Invalid order data: missing required field 'customerId'")

        elif action == 'oom':
            # Simulate high memory usage
            log_entry["message"] = "Processing large dataset"
            data = "x" * (50 * 1024 * 1024)  # 50MB string

        else:
            # Normal processing
            log_entry["message"] = "Order processed successfully"
            time.sleep(random.uniform(0.1, 0.5))

        print(json.dumps(log_entry))

        # Emit EMF metrics
        processing_time = (time.time() - start_time) * 1000
        emit_emf_metric("ProcessingTime", processing_time, "Milliseconds",
                        {"Environment": "demo", "Action": action})
        emit_emf_metric("SuccessCount", 1, "Count",
                        {"Environment": "demo", "Action": action})

        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'Processed successfully',
                'action': action,
                'requestId': request_id,
                'processingTime': f"{processing_time:.0f}ms"
            })
        }

    except Exception as e:
        processing_time = (time.time() - start_time) * 1000
        log_entry["level"] = "ERROR"
        log_entry["error"] = str(e)
        log_entry["errorType"] = type(e).__name__
        print(json.dumps(log_entry))

        emit_emf_metric("ErrorCount", 1, "Count",
                        {"Environment": "demo", "Action": action})

        return {
            'statusCode': 500,
            'body': json.dumps({
                'error': str(e),
                'requestId': request_id
            })
        }

Click Deploy

Step 03: Generate Test Data

Go to the Test tab

Create and run these test events multiple times:

Normal processing

{"action": "process"}

Slow operation

{"action": "slow"}

Error case

{"action": "error"}

⚠️ Run each test event 5-10 times to generate enough log data

---

Part II

Write CloudWatch Logs Insights Queries

Step 01: Open Logs Insights

Open the CloudWatch console → ** ▼ Logs** → Log Management

Select the log group: /aws/lambda/DebugDemoFunction

⚠️ Set the time range to the last 1 hour

→ View in Logs Insights

Query 01: Find All Errors

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 25

Click Run query. You should see the structured error logs from the error test events.

Query 02: Analyze Invocation Performance

filter @type = "REPORT"
| stats avg(@duration) as avgDuration,
        max(@duration) as maxDuration,
        min(@duration) as minDuration,
        count(*) as invocations

Query 03: Find Cold Starts

filter @type = "REPORT" and @initDuration > 0
| fields @requestId, @initDuration, @duration, @maxMemoryUsed
| sort @initDuration desc

Query 04: Parse Structured Logs for Error Breakdown

parse @message '"level":"*","requestId":"*"' as level, reqId
| filter level = "ERROR"
| stats count(*) as errorCount by level

Query 05: Memory Utilization Analysis

filter @type = "REPORT"
| stats avg(@maxMemoryUsed) as avgMemUsed,
        max(@maxMemoryUsed) as peakMemUsed,
        avg(@memorySize) as configuredMem,
        avg(@maxMemoryUsed) / avg(@memorySize) * 100 as utilizationPct

Query 06: Slow Invocations Over Time

filter @type = "REPORT"
| stats avg(@duration) as avgDuration,
        pct(@duration, 95) as p95Duration,
        pct(@duration, 99) as p99Duration
        by bin(5m)
| sort bin(5m) asc

💡 Save useful queries by clicking **Save in Logs Insights. You can also add query results directly to a CloudWatch dashboard. The pct() function calculates percentiles: pct(@duration, 95) gives you the 95th percentile duration, which is more useful than averages for identifying tail latency.**

---

Part III

Create a CloudWatch Dashboard

Step 01: Build the Dashboard

Open the CloudWatch console → Dashboards → Create dashboard

Dashboard name: DebugDemo-Dashboard

Click Create dashboard

Step 02: Add an Invocation Count Widget

Click Add widget → Number

Select Lambda → By Function Name → DebugDemoFunction → Invocations

Period: 5 minutes

Click Create widget

Step 03: Add an Error Rate Widget

Click Add widget → Line

Add two metrics:

• Lambda → By Function Name → DebugDemoFunction → Errors
• Lambda → By Function Name → DebugDemoFunction → Invocations

Click Create widget

Step 04: Add a Duration Widget

Click Add widget → Line

Select Lambda → By Function Name → DebugDemoFunction → Duration

Add statistics: Average, p99, Maximum

Click Create widget

Step 05: Add a Custom EMF Metric Widget

Click Add widget → Number

Select DebugDemo (custom namespace) → Environment, Action → ProcessingTime

Click Create widget

Step 06: Add a Logs Insights Query Widget

Click Add widget → Logs table

Select log group: /aws/lambda/DebugDemoFunction

Enter query:

filter @message like /ERROR/
| fields @timestamp, @message
| sort @timestamp desc
| limit 10

Click Create

Step 07:* Click **Save dashboard

---

Part IV

Simulate and Debug Common Failures

Step 01: Simulate a 502 Bad Gateway

Create a new Lambda function: BadResponseFunction and add this intentionally broken code

import json

def lambda_handler(event, context):
    # BUG: body should be a string, not a dict
    return {
        'statusCode': 200,
        'body': {'message': 'this will cause a 502'}  # Wrong!
    }

Create an API Gateway REST API pointing to this function

💡 Test the API endpoint. You'll get a 502

⚠️ Debug steps:

• Check Lambda logs → the function executed successfully (no error)
• Check API Gateway execution logs → "Malformed Lambda proxy response"
• Fix: Change 'body': {'message': '...'} to 'body': json.dumps({'message': '...'})

Step 02: Simulate a Timeout

On DebugDemoFunction, go to Configuration → General configuration → Edit

Set Timeout to 3 seconds

Run the slow test event (which sleeps 2-5 seconds)

⚠️ Debug steps:

• Check logs → Task timed out after 3.00 seconds
• Check the REPORT line → @duration close to 3000ms
• Query in Logs Insights:

   filter @message like /Task timed out/
   | fields @requestId, @duration, @memorySize

Fix: Increase timeout or optimize the slow operation

Step 03: Simulate Permission Denied

On DebugDemoFunction, update the code to call DynamoDB:

import boto3
import json

dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):
    table = dynamodb.Table('NonExistentTable')
    try:
        table.put_item(Item={'PK': 'test', 'SK': 'test'})
    except Exception as e:
        print(json.dumps({
            "level": "ERROR",
            "error": str(e),
            "errorType": type(e).__name__
        }))
        return {
            'statusCode': 500,
            'body': json.dumps({'error': str(e)})
        }

💡Run the test. You'll get an AccessDeniedException

Debug steps:

• Check Lambda logs → AccessDeniedException
• Open CloudTrail → Event history
• Filter by Event name: PutItem
• Find the event with errorCode: AccessDenied
• The event shows which role tried to access which resource

---

Part V

Use CloudTrail to Find Access Denied Events

Step 01: Search Event History

Open the CloudTrail console → Event history

Change the filter to Error code

Type AccessDenied and press Enter

You should see the PutItem event from the previous step

Step 02: Examine the Event Details

Click on the event to expand it

Key fields to examine

• userIdentity.arn: the Lambda execution role that made the call
• eventName: PutItem
• requestParameters: the table name and item
• errorCode: AccessDeniedException
• errorMessage: the specific permission needed

💡To fix: add dynamodb:PutItem permission to the Lambda execution role for the specific table ARN

Step 03: Create a Logs Insights Query for CloudTrail

If you have a CloudTrail trail sending logs to CloudWatch:

fields eventTime, eventName, errorCode, errorMessage,
       userIdentity.arn as principal,
       requestParameters.tableName as table
| filter errorCode = "AccessDenied"
| sort eventTime desc
| limit 20

💡 CloudTrail Event history is free and covers the last 90 days of management events. For longer retention or data events (S3 object access, Lambda invocations), you need a trail. When debugging permission issues, CloudTrail is your first stop, it tells you exactly what permission is missing, who tried to use it, and on which resource.

---

🏗️ What You Built | 📘 Exam Concepts Recap

What You Built	Exam Concept
Emitted EMF metrics from structured logs	Zero-latency custom metrics via Embedded Metric Format
Wrote CloudWatch Logs Insights queries	fields, filter, stats, parse, sort, pct() syntax
Queried @initDuration for cold starts	Reading Lambda REPORT fields for performance
Built a CloudWatch dashboard with mixed widgets	Single pane of glass for application health
Reproduced and fixed a 502 Bad Gateway	Malformed Lambda proxy response (body must be string)
Reproduced a timeout and read the REPORT line	Diagnosing Task timed out from logs
Found AccessDenied events in CloudTrail	Debugging permission issues: principal, API, resource

---

⚠️ Clean Up Protocol

1. Lambda → Delete DebugDemoFunction and BadResponseFunction
2. API Gateway → Delete any test APIs created
3. CloudWatch → Delete the DebugDemo-Dashboard
4. CloudWatch → Delete log groups for the Lambda functions
5. IAM → Delete Lambda execution roles

---

Key Takeaways for the Exam

1. CloudWatch Logs Insights uses fields, filter, stats, parse, sort, and limit commands. Know the syntax for each.
2. Lambda REPORT lines contain @duration, @billedDuration, @memorySize, @maxMemoryUsed, and @initDuration (cold starts only). Use these for performance analysis.
3. EMF is preferred over PutMetricData for Lambda because it adds zero latency. Metrics are extracted from structured JSON written to stdout. Use PutMetricData for EC2 or infrequent publishing.
4. 502 Bad Gateway from API Gateway almost always means the Lambda response format is wrong: body must be a string, not a dict.
5. 504 Gateway Timeout means Lambda exceeded API Gateway's 29-second limit. The Lambda timeout must be less than 29 seconds for synchronous API Gateway integrations.
6. CloudTrail logs every AWS API call. Use Event history to find AccessDenied errors. It shows the principal, API, resource, and missing permission.
7. Metric filters extract CloudWatch metrics from log text patterns. Subscription filters stream log events to Kinesis, Lambda, or OpenSearch in real time.
8. CloudWatch dashboards can include metrics widgets, Logs Insights query results, and alarms giving you a single pane of glass for application health.
9. The parse command in Logs Insights extracts fields from unstructured text using glob patterns (*). The stats command supports count(), avg(), sum(), min(), max(), and pct() for percentiles.
10. When debugging deployment failures, check CloudFormation Events (stack failures), CodeBuild logs (build failures), and CodeDeploy logs (deployment failures). Each service has its own log location.

---

Additional Resources

• CloudWatch Logs Insights language query syntax
• Specification: Embedded metric format
• Monitoring, debugging, and troubleshooting Lambda functions
• Working with CloudTrail event history
• Creating a customized CloudWatch dashboard
• Troubleshooting issues in Lambda

---

🏗️