DEV Community

Cover image for Instrument Code For Observability | 🏗️ Build An Observable Order Service
AWS Community Builders profile image Ntombizakhona Mabaso
Ntombizakhona Mabaso Subscriber for AWS Community Builders

Posted on

Instrument Code For Observability | 🏗️ Build An Observable Order Service

#aws #certification #cloud #developer

Exam Guide: Developer - Associate
🏗️ Domain 4: Troubleshooting And Optimization
📘 Task 2: Instrument Code For Observability

Observability goes beyond monitoring: it's about understanding the internal state of your system from its external outputs. This task tests your ability to instrument applications with AWS X-Ray for distributed tracing, implement structured logging with correlation IDs, set up CloudWatch alarms and notifications, and build health check endpoints. X-Ray concepts (traces, segments, subsegments, annotations vs metadata), structured logging best practices, and alarm configuration.

📘Concepts

Logging vs Monitoring vs Observability

These three concepts build on each other.

Aspect Logging Monitoring Observability
What It Is Recording discrete events Tracking predefined metrics over time Understanding system behaviour from outputs
Question It Answers What happened? Is the system healthy? Why is the system behaving this way?
Data Type Text entries (structured or unstructured) Numeric time-series data Logs + metrics + traces combined
Proactive vs Reactive Reactive. Search after an issue Proactive. Alerts before users notice Both. Explore unknown unknowns
AWS Service(s) CloudWatch Logs CloudWatch Metrics, Alarms X-Ray, CloudWatch Logs Insights, ServiceLens
Example ERROR: DynamoDB timeout on table Orders Lambda Duration > 5s for 3 consecutive periods Trace shows DynamoDB call took 4.8s due to throttling

The Three Pillars Of Observability

Pillar What It Provides AWS Service
Logs Detailed event records for debugging CloudWatch Logs
Metrics Numeric measurements for alerting and trending CloudWatch Metrics
Traces Request flow across distributed services AWS X-Ray

Monitoring tells you something is wrong. Observability tells you why. Metrics show a problem (high latency) but you need traces to find the root cause (a specific downstream service is slow)

AWS X-Ray Concepts

💡 X-Ray is the answer to "How do you find which service is causing latency?"

X-Ray provides distributed tracing. It tracks requests as they flow through your application across multiple services.

Concept What It Is Example
Trace The complete journey of a single request through all services API Gateway → Lambda → DynamoDB → SNS
Segment A unit of work done by a single service The Lambda function's execution
Subsegment A subdivision of a segment for more detail A specific DynamoDB call within the Lambda function
Trace ID Unique identifier for the entire trace 1-5f84c7a1-abcdef1234567890
Sampling Controls what percentage of requests are traced Default: first request each second + 5% of additional
Service Map Visual representation of your architecture and request flow Shows services as nodes with latency and error rates
Trace Map Visual representation of a single trace Shows the path of one request through services

X-Ray Annotations vs Metadata

Feature Annotations Metadata
What It Stores Key-value pairs (string, number, boolean) Any JSON-serializable object
Indexed Yes. Searchable in the X-Ray console No. Not searchable
Use For Filtering and grouping traces Storing detailed debug data
Max Size 50 annotations per trace No practical limit
Example userId: "CUST-001", orderType: "premium" Full request/response bodies, stack traces
Search Syntax annotation.userId = "CUST-001" Cannot search by metadata

Use annotations for values you want to search and filter by user IDs, order types, environment names.
Use metadata for detailed debugging data you only need when examining a specific trace.
Annotations are indexed and searchable, metadata is not.

💡"How do you find all traces for a specific user?" The answer is annotations.

X-Ray Sampling Rules

Setting Default Purpose
Reservoir 1 per second Guaranteed minimum traces per second
Rate 5% (0.05) Percentage of additional requests to trace
Service Name * (all) Which services the rule applies to
HTTP Method * (all) Filter by GET, POST, etc.
URL Path * (all) Filter by specific paths
Priority 10000 (default) Lower number = higher priority

Custom Sampling Rule Example:

  • Trace 100% of requests to /api/orders (high-value endpoint)
  • Trace 1% of requests to /api/health (noisy, low-value)

💡 The default sampling rule traces the first request each second plus 5% of additional requests. This keeps costs manageable for high-traffic applications. You can create custom rules to trace more (or fewer) requests for specific paths or services. Sampling rules are evaluated in priority order. Lower priority number wins.

X-Ray Integration with AWS Services

Service How to Enable What Gets Traced
Lambda Toggle "Active tracing" in function configuration Function execution, SDK calls
API Gateway Enable X-Ray tracing on the stage API request processing
ECS/Fargate Run X-Ray daemon as sidecar container Application SDK calls
EC2 Install and run X-Ray daemon Application SDK calls
Elastic Beanstalk Enable in .ebextensions or console Application SDK calls
SNS/SQS Automatic when upstream service is traced Message propagation

💡 For Lambda, enabling "Active tracing" is all you need for basic traces. To add custom subsegments and annotations, you need the X-Ray SDK in your code. The X-Ray daemon runs automatically in Lambda yiu don't need to install it. For EC2 and ECS, you must install and run the daemon yourself.

Structured Logging Best Practices

Practice Why It Matters Example
Use JSON Format Machine-parseable, works with Logs Insights {"level":"ERROR","message":"timeout"}
Include Correlation ID Track requests across services {"correlationId":"abc-123","service":"orders"}
Include Request Context Debug without searching multiple logs {"requestId":"req-1","userId":"user-5"}
Use Log Levels Filter by severity DEBUG, INFO, WARN, ERROR, FATAL
Avoid Sensitive Data Security and compliance Never log passwords, tokens, PII
Include timestamps Accurate event ordering ISO 8601 format: 2024-01-15T10:30:00Z

Correlation IDs for Distributed Tracing

A Correlation ID is a unique identifier that follows a request across all services:

Client → API Gateway → Lambda A → SQS → Lambda B → DynamoDB
         correlationId: "abc-123" flows through every service
Enter fullscreen mode Exit fullscreen mode
Where to Set It How
API Gateway Pass X-Correlation-Id header or generate in mapping template
Lambda Read from event headers or generate if missing
SQS Include in message attributes
SNS Include in message attributes
DynamoDB Not applicable (use in logs only)

💡 Correlation IDs complement X-Ray traces. X-Ray gives you the visual trace map. Correlation IDs let you search logs across services for the same request. Use both together: add the correlation ID as an X-Ray annotation so you can find the trace from the log and vice versa.

CloudWatch Alarms

Alarm Component What It Does Options
Metric What you're monitoring Any CloudWatch metric (built-in or custom)
Statistic How to aggregate data points Average, Sum, Min, Max, p99, SampleCount
Period Time window for each data point 10s, 30s, 60s, 300s, etc.
Evaluation periods How many periods to evaluate 1-100
Datapoints to alarm How many periods must breach threshold 1 to evaluation periods
Threshold The value that triggers the alarm Any number
Comparison How to compare metric to threshold >, >=, <, <=
Actions What happens when alarm fires SNS notification, Auto Scaling, EC2 action, Lambda
Missing data How to treat gaps in data missing, notBreaching, breaching, ignore

Alarm States:

State Meaning
OK Metric is within the threshold
ALARM Metric has breached the threshold
INSUFFICIENT_DATA Not enough data to determine state

Composite Alarms

Feature Standard Alarm Composite Alarm
Based On A single metric Multiple alarms combined with AND/OR/NOT
Use Case Monitor one metric Reduce alarm noise, complex conditions
Example Error rate > 5% Error rate > 5% AND latency > 2s
Actions SNS, Auto Scaling, EC2, Lambda SNS, Lambda

💡 Composite alarms reduce alert fatigue. Instead of getting separate alerts for high errors and high latency, create a composite alarm that only fires when both conditions are true. Use AND for "all conditions must be true" and OR for "any condition triggers." Composite alarms can suppress actions on child alarms to prevent duplicate notifications.

Health Check Patterns

Pattern What It Checks Response
Shallow Health Check Is the service running? 200 OK (no dependency checks)
Deep Health Check Are all dependencies healthy? 200 OK with dependency status
Liveness Probe I_s the process alive?_ 200 OK (minimal check)
Readiness Probe Can the service handle requests? 200 OK only when fully initialized

Deep Health Check Response Example: 

{
  "status": "healthy",
  "timestamp": "2026-07-20T10:30:00Z",
  "dependencies": {
    "dynamodb": {"status": "healthy", "latency": "12ms"},
    "sqs": {"status": "healthy", "latency": "8ms"},
    "external-api": {"status": "degraded", "latency": "2500ms"}
  }
}
Enter fullscreen mode Exit fullscreen mode

💡Health checks should have their own endpoint (e.g., /health) that doesn't require authentication. Shallow checks are fast and cheap: use them for load balancer health checks. Deep checks verify dependencies: use them for monitoring dashboards. Don't make deep health checks too expensive. Cache dependency status for a few seconds.

🏗️ Build An Observable Order Service

Build an Observable Order Service from scratch using the AWS Console:

  • A Lambda function with X-Ray active tracing enabled
  • Custom X-Ray subsegments and annotations in the code
  • Structured JSON logging with correlation IDs
  • CloudWatch alarms with SNS email notifications
  • A health check endpoint that verifies dependencies
  • Traces visible in the X-Ray console service map

Prerequisites

Part I

Create a Lambda Function with X-Ray Tracing

Step 01: Create the Function

Open the Lambda console → Create function

  • Function name: ObservableOrderService
  • Runtime: Python 3.13

→ Click Create function

Step 02: Enable X-Ray Active Tracing

Go to Configuration → Monitoring and operations tools → Edit

→ Under CloudWatch Application Signals and AWS X-Ray → ✔ Lambda services traces

→ Click Save

Step 03: Add the X-Ray SDK Layer

⚠️ The instrumented code below uses aws_xray_sdk for custom subsegments and annotations. That package is **not in the Lambda runtime (and the AWSSDKPandas layer doesn't include it), so you'll get No module named 'aws_xray_sdk' unless you add it. Build a layer in CloudShell (no local setup needed):**

mkdir -p xray-layer/python
pip install aws-xray-sdk -t xray-layer/python/
cd xray-layer && zip -r ../xray-sdk-layer.zip python/ && cd ..
Enter fullscreen mode Exit fullscreen mode

⚠️ Download it (Actions → Download file → xray-sdk-layer.zip), then in the Lambda console:

→ Layers → Edit → Add layer → Create a new layer

→ Name: aws-xray-sdk

→ Source code: Create from a .zip file → Choose file → Upload the zip → Compatible runtimes optional: Python 3.13 → Create → Copy arn

→ Open ObservableOrderService → Code → Layers → Edit → Add a layer → Specify an ARN → Paste arn → Verify → Add

💡 Prefer no layer? You can skip the SDK entirely and rely on active tracing alone It auto-traces the invocation and any boto3/HTTP calls without importing anything. You'd just lose in-code custom subsegments/annotations. If you go that route, remove the aws_xray_sdk imports and xray_recorder calls from the code below.

⚠️ X-Ray permissions: if traces don't appear, attach AWSXRayDaemonWriteAccess to the function's execution role (Configuration → Permissions → role → Add permissions). Active tracing needs xray:PutTraceSegments and xray:PutTelemetryRecords.

⚠️ Alternatively, you can use the aws_xray_sdk by packaging it with your function code. For this tutorial, we'll use the built-in boto3 tracing capabilities.

Step 04: Add Instrumented Code

Replace the function code with:

import json
import time
import boto3
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

patch_all()

def generate_correlation_id(event):
    headers = event.get('headers', {}) or {}
    return headers.get('X-Correlation-Id',
           headers.get('x-correlation-id',
           f"gen-{int(time.time() * 1000)}"))

def structured_log(level, message, correlation_id, **kwargs):
    print(json.dumps({
        "timestamp": time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
        "level": level, "message": message,
        "correlationId": correlation_id,
        "service": "ObservableOrderService", **kwargs
    }))

def lambda_handler(event, context):
    correlation_id = generate_correlation_id(event)
    action = event.get('action', event.get('path', '/process'))

    # Facade-segment fix: annotate our OWN subsegment, not the Lambda root.
    with xray_recorder.in_subsegment('request-handler') as subsegment:
        subsegment.put_annotation('correlationId', correlation_id)
        subsegment.put_annotation('action', str(action))
        subsegment.put_metadata('event', event, 'request')

        structured_log("INFO", "Request received", correlation_id,
                       action=action, requestId=context.aws_request_id)

        if action in ('/health', 'health'):
            return health_check(correlation_id)

        return process_order(event, correlation_id, context)

def health_check(correlation_id):
    with xray_recorder.in_subsegment('HealthCheck') as subsegment:
        subsegment.put_annotation('checkType', 'deep')
        dependencies = {}
        try:
            start = time.time()
            boto3.client('sts').get_caller_identity()
            latency = int((time.time() - start) * 1000)
            dependencies['sts'] = {'status': 'healthy', 'latency': f'{latency}ms'}
        except Exception as e:
            dependencies['sts'] = {'status': 'unhealthy', 'error': str(e)}
        overall = 'healthy' if all(d['status'] == 'healthy' for d in dependencies.values()) else 'degraded'
        structured_log("INFO", f"Health check: {overall}", correlation_id, dependencies=dependencies)
        return {
            'statusCode': 200 if overall == 'healthy' else 503,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps({'status': overall, 'correlationId': correlation_id, 'dependencies': dependencies})
        }

def process_order(event, correlation_id, context):
    with xray_recorder.in_subsegment('ValidateInput') as subsegment:
        order_id = event.get('orderId', f"ORD-{int(time.time())}")
        customer_id = event.get('customerId')
        subsegment.put_annotation('orderId', order_id)

        if not customer_id:
            subsegment.put_annotation('validationResult', 'failed')
            structured_log("WARN", "Missing customerId", correlation_id, orderId=order_id)
            return {
                'statusCode': 400,
                'headers': {'Content-Type': 'application/json'},
                'body': json.dumps({'error': 'customerId is required', 'correlationId': correlation_id})
            }
        subsegment.put_annotation('validationResult', 'passed')

    with xray_recorder.in_subsegment('ProcessBusinessLogic') as subsegment:
        subsegment.put_annotation('orderId', order_id)
        subsegment.put_annotation('customerId', customer_id)
        processing_time = 0.1 + (hash(order_id) % 5) / 10
        time.sleep(processing_time)
        subsegment.put_metadata('processingDetails',
            {'orderId': order_id, 'processingTime': f'{processing_time:.1f}s', 'items': event.get('items', [])}, 'order')

    structured_log("INFO", "Order processed successfully", correlation_id,
                   orderId=order_id, customerId=customer_id, processingTime=f"{processing_time:.1f}s")
    return {
        'statusCode': 200,
        'headers': {'Content-Type': 'application/json'},
        'body': json.dumps({'message': 'Order processed', 'orderId': order_id,
                            'customerId': customer_id, 'correlationId': correlation_id})
    }
Enter fullscreen mode Exit fullscreen mode

→ Click Deploy

⚠️ If the aws_xray_sdk import fails, you'll need to add it as a Lambda layer. Create a layer with the X-Ray SDK package, or use the simplified version without the SDK (X-Ray will still trace boto3 calls automatically with active tracing enabled).

Part II

Add Custom Subsegments and Annotations

Step 01: Test the Function

→ Go to the Test tab

→ Create a test event:

{
  "action": "process",
  "orderId": "ORD-2026-001",
  "customerId": "CUST-100",
  "items": [{"productId": "PROD-A", "quantity": 2}],
  "headers": {"X-Correlation-Id": "test-corr-001"}
}
Enter fullscreen mode Exit fullscreen mode

→ Click Test → ⚠️ **Verify the response includes the correlation ID

→ Run it several more times with different order IDs

Step 02: Test the Health Check

→ Create another test event

{
  "action": "health"
}
Enter fullscreen mode Exit fullscreen mode

→ Click Test → ⚠️ Verify the health check response with dependency status

Step 03: Test Validation Failure

Create a test event without customerId

{
  "action": "process",
  "orderId": "ORD-2026-002"
}
Enter fullscreen mode Exit fullscreen mode

→ Click Test → ⚠️ verify the 400 response with validation error

Part III

View Traces in the X-Ray Console

Step 01: Open the X-Ray Trace Map

→ Open the X-Ray console (or CloudWatch → Trace Map → ObservableOrderService)

→ Set the time range to the last 30 minutes

→ You should see a service map showing:

  • Client → ObservableOrderService (Lambda Context) → ObservableOrderService (Lambda Function)
  • If the function called other AWS services, those appear as nodes too

Step 02: Examine Individual Traces

→ Click on the ObservableOrderService node in the service map

→ Click View traces

→ Click on a specific trace to see the timeline:

  • The Lambda segment (total execution time)
  • Initialization subsegment (cold start, if applicable)
  • ValidateInput subsegment (your custom subsegment)
  • ProcessBusinessLogic subsegment (your custom subsegment)
  • HealthCheck subsegment (for health check requests)

Step 03: Search Traces by Annotation

→ Go to Traces

→ In the filter expression box, type:

   annotation.correlationId = "test-corr-001"
Enter fullscreen mode Exit fullscreen mode

→ Click Run query → You should see only the traces with that correlation ID

💡This is exactly why annotations are indexed and metadata is not. You can search for annotation.userId = "CUST-100" to find all traces for a specific user. You cannot search by metadata values. Use annotations for the fields you'll want to filter by. Use metadata for the detailed data you'll examine once you've found the right trace.

Part IV

Set Up CloudWatch Alarms with SNS Notifications

Step 01: Create an SNS Topic

→ Open the SNS console → Topics → Create topic

→ Type: Standard

→ Name: OrderServiceAlerts

→ Click Create topic

Step 02: Create an Email Subscription

→ Click Create subscription

→ Protocol: Email

→ Endpoint: your email address

→ Click Create subscription

→ Check your email and confirm the subscription

Step 03: Create an Error Rate Alarm

→ Open the CloudWatch console → Alarms → Create alarm

→ Click Select metric

→ Navigate: Lambda → By Function Name → ObservableOrderService → Errors

→ Click Select metric

  • Statistic: Sum
  • Period: 60 seconds
  • Threshold type: Static
  • Whenever Errors is: Greater than 0
  • Datapoints to alarm: 1 out of 1

→ Notification:

  • Alarm state trigger: In alarm
  • SNS topic: OrderServiceAlerts

→ Alarm name: OrderService-Errors

→ Click Create alarm

Step 04: Create a Duration Alarm

→ Create another alarm:

  • Metric: Lambda → ObservableOrderService → Duration
  • Statistic: p95
  • Period: 300 seconds (5 minutes)
  • Threshold: Greater than 3000 (3 seconds)
  • Datapoints to alarm: 2 out of 3
  • SNS topic: OrderServiceAlerts
  • Alarm name: OrderService-HighLatency

Step 05: Create a Composite Alarm

→ Go to Alarms → Create alarm → Create composite alarm

→ Alarm rule:

   ALARM("OrderService-Errors") AND ALARM("OrderService-HighLatency")
Enter fullscreen mode Exit fullscreen mode

→ Notification: OrderServiceAlerts

→ Alarm name: OrderService-Critical

→ Click Create alarm

⚠️ This composite alarm only fires when both errors AND high latency occur simultaneously indicating a real problem, not just a transient spike.

Part V

Verify the Complete Observability Stack

Generate Traffic and Verify

→ Go back to the Lambda function and run various test events:

  • Normal orders (several times)
  • Health checks
  • Validation failures (missing customerId)

→ Check CloudWatch Logs:

  • Open the log group /aws/lambda/ObservableOrderService
  • Verify structured JSON logs with correlation IDs
  • Use Logs Insights to query: 
   parse @message '"correlationId":"*","service":"*"' as corrId, svc
   | filter corrId like /test-corr/
   | fields @timestamp, @message
   | sort @timestamp desc
Enter fullscreen mode Exit fullscreen mode

→ Check X-Ray:

  • View the trace map. Verify your function appears
  • Search traces by annotation
  • Examine subsegment timing

→ Check CloudWatch Alarms:

  • Verify alarms are in OK state
  • Trigger errors to test the alarm → check your email for the SNS notification

→ Check the Dashboard (optional):

  • Create a dashboard combining Lambda metrics, custom metrics, and Logs Insights queries

💡 A fully observable application has all three pillars working together: structured logs with correlation IDs (searchable in Logs Insights), metrics with alarms (proactive alerting), and traces with annotations (distributed request tracking). The correlation ID ties them all together. From a log entry, you can find the X-Ray trace, and from a trace, you can search the logs.

🏗️ What You Built | 📘 Exam Concepts Recap

What You Built Exam Concept
Added structured logging with correlation IDs Tracking a request across services
Created custom X-Ray subsegments Measuring specific code sections for bottlenecks
Added X-Ray annotations and metadata Annotations are indexed/searchable. Metadata is not
Searched traces by annotation.correlationId Filtering traces by indexed fields
Enabled active tracing on Lambda X-Ray daemon runs automatically, no install
Created error and latency CloudWatch alarms Proactive alerting with SNS notifications
Built a composite alarm with AND logic Reducing alert noise by combining conditions
Tied logs + metrics + traces via correlation ID The three pillars of observability working together

⚠️ Clean Up Protocol

  1. Lambda → Delete ObservableOrderService
  2. SNS → Delete the OrderServiceAlerts topic (this also deletes subscriptions)
  3. CloudWatch → Delete all three alarms (Errors, HighLatency, Critical composite)
  4. CloudWatch → Delete the log group /aws/lambda/ObservableOrderService
  5. IAM → Delete the Lambda execution role

Key Takeaways

  1. Observability = logs + metrics + traces. Monitoring tells you something is wrong. Observability tells you why. X-Ray is the answer when the question asks about finding the root cause of latency in distributed systems.
  2. X-Ray annotations are indexed and searchable: use them for filtering traces by user ID, order ID, etc. Metadata is not searchable: use it for detailed debugging data.
  3. X-Ray subsegments let you measure specific parts of your code. Create subsegments for database calls, external API calls, and business logic to identify bottlenecks.
  4. Correlation IDs track a request across services. Generate one at the entry point and pass it through headers, message attributes, and log entries. Add it as an X-Ray annotation.
  5. Structured JSON logging is essential for Logs Insights queries. Include level, message, correlation ID, timestamp, and relevant context in every log entry.
  6. CloudWatch alarms have three states: OK, ALARM, INSUFFICIENT_DATA. Configure TreatMissingData carefully. notBreaching is usually the right choice for Lambda.
  7. Composite alarms combine multiple alarms with AND/OR logic to reduce alert noise. They only fire when the combined condition is true.
  8. X-Ray sampling defaults to 1 request/second + 5% of additional requests. Create custom rules to trace more of high-value endpoints and less of health checks.
  9. Health checks should have shallow (is it running?) and deep (are dependencies healthy?) variants. Use shallow for load balancers, deep for dashboards.
  10. For Lambda, enable Lambda service traces in the function configuration. The X-Ray daemon runs automatically no installation needed. Use the X-Ray SDK to add custom subsegments and annotations.

Additional Resources

🏗️

Top comments (0)