DEV Community

Cover image for Query Structured Lambda Logs by a Nested JSON Field
miruky
miruky

Posted on

Query Structured Lambda Logs by a Nested JSON Field

#aws #lambda #cloudwatch #tutorial

Introduction

Hi, I'm miruky.

Searching an unstructured log message often means parsing text after the fact. AWS Lambda advanced logging controls can capture Python logging output as JSON, and CloudWatch Logs Insights can address structurally nested JSON fields with dot notation.

This Console walkthrough sends three distinct event bodies to one function. Each application record contains an order object with a nested customer tier; an Insights query for order.customer.tier = "gold" returns one row for each distinct gold order and excludes the standard order.

The exercise uses three short Lambda invocations, a one-day log group, and a narrowly scoped log query. Current Lambda and CloudWatch pricing terms are linked below.

1. Prepare the execution role and log group

Use the English AWS Console in United States (N. Virginia). Lambda, CloudWatch Logs, and every invocation remain in us-east-1; IAM itself is global.

The English AWS Console shows United States (N. Virginia) before the regional resources are created.

The header confirms United States (N. Virginia) while the Lambda Console is in English. This fixes the Region for the function, log group, invocations, and query.

In IAM, search for exact role name miruky-ajojxevzcpsbmryk and confirm that it is absent. Do not open or modify a partial match.

The IAM role list shows zero matches for the generated role name.

The exact filter reports 0 matches and No matches for miruky-ajojxevzcpsbmryk. That empty result establishes the global IAM resource boundary before creation.

Create the role for the Lambda AWS service and attach AWSLambdaBasicExecutionRole.

The generated IAM role has the AWSLambdaBasicExecutionRole managed policy attached.

The role's permissions table shows one attached AWS managed policy: AWSLambdaBasicExecutionRole. The role uses the Lambda service trust relationship. Screenshots omit account-specific identifiers.

Open CloudWatch Logs and confirm that miruky-dwjrjfrjsvxegfiu does not exist. In this run, the account had no log groups in us-east-1, so the list itself established the empty baseline without a name filter.

The CloudWatch log-group list is empty before the generated group is created.

The page reports Log groups (0) and There are no log groups. This confirms that the generated name is available before creation. Create miruky-dwjrjfrjsvxegfiu and set Retention to 1 day, which is one of the supported retention periods.

The generated CloudWatch log group has a one-day retention period.

The log-group row shows miruky-dwjrjfrjsvxegfiu with retention set to 1 day.

By default, Lambda creates /aws/lambda/<function-name> when a function is initially invoked. Selecting an existing custom log group before any invocation keeps the controlled records in the generated group instead. One-day retention also prevents this small validation log from persisting indefinitely.

2. Create a function with JSON logging

Open AWS Lambda. In this run, the Region had no functions, so the Console opened its getting-started page instead of a function list.

The AWS Lambda getting-started page offers Create a function while the Region has no functions.

The empty onboarding state shows Create a function without any function rows. Use it to enter the generated name miruky-lkddcrnpcpqhkitg and choose the supported Python 3.14 runtime.

The create-function form uses the generated name and Python 3.14 runtime.

The form shows miruky-lkddcrnpcpqhkitg and Python 3.14.

Expand Additional settings, choose Custom execution role, and select miruky-ajojxevzcpsbmryk in the side panel.

The custom execution role panel selects the generated existing Lambda execution role.

The panel shows Configure custom execution role and selects miruky-ajojxevzcpsbmryk.

Open Monitoring and operations tools and edit Logging configuration. Select custom log group miruky-dwjrjfrjsvxegfiu, set Log format to JSON, and keep Application log level at INFO.

The Lambda logging configuration uses JSON, INFO, and the generated custom log group.

The logging configuration selects JSON, application level INFO, and only miruky-dwjrjfrjsvxegfiu as the CloudWatch log group. System fields will still exist, but the later query will not select them.

Changing the format affects new log entries, not entries already stored. This controlled group begins empty, so all application records used below share the intended JSON format.

3. Emit a nested object with Python logging

Replace the starter code with this function and choose Deploy:

import logging

logger = logging.getLogger()


def lambda_handler(event, context):
    # Keep application dimensions structured for direct field-level queries.
    logger.info(
        "order routed",
        extra={
            "order": {
                "ref": event["order_ref"],
                "customer": {"tier": event["tier"]},
            },
            "route": event["route"],
        },
    )
    return {"logged": True}
Enter fullscreen mode Exit fullscreen mode

The deployed Python code logs a nested order object through the logging extra argument.

The editor shows the standard logging library, one logger.info call, an order object with nested customer and tier keys, and a separate route field inside extra.

The function relies on Lambda's INFO application filter instead of overriding the level in code, following the JSON-logging guidance for Python. With Lambda JSON logging enabled, the standard Python logger adds values supplied through extra to the JSON record alongside Lambda-managed fields such as timestamp, level, message, and request ID. The controlled query below verifies that order remains a nested object. The code does not print the entire incoming event.

4. Generate three controlled records

Use one Private Lambda test event named miruky-zpgwxaehrfxbobzs with the initial input:

{
  "order_ref": "order-blue",
  "tier": "gold",
  "route": "priority"
}
Enter fullscreen mode Exit fullscreen mode

Save the test event, invoke the function once, and confirm that the response is {"logged":true}.

The private test-event editor contains the initial gold input before invocation.

The editor shows event name miruky-zpgwxaehrfxbobzs with the fixed order-blue, gold, and priority values.

Edit the same private test event, replace its body with the standard order, save, and invoke once:

{
  "order_ref": "order-green",
  "tier": "standard",
  "route": "normal"
}
Enter fullscreen mode Exit fullscreen mode

The reused private test-event editor contains the standard-tier input before invocation.

The same saved event name now carries order-green, tier standard, and route normal. Confirm that the invocation returns {"logged":true}. This record is the negative control for the query.

Edit the event one more time and invoke another gold order:

{
  "order_ref": "order-violet",
  "tier": "gold",
  "route": "priority"
}
Enter fullscreen mode Exit fullscreen mode

The reused private test-event editor contains the final gold input before invocation.

The final editor state uses order-violet, tier gold, and route priority under the same event name. Confirm that the invocation returns {"logged":true}. The log group should now contain all three distinct order_ref values.

The same test event supplies three inputs, each with a distinct order_ref value. A repeated click can create another record with the same reference, so the query below handles reruns explicitly.

5. Query the nested tier field

Open CloudWatch Logs Insights, select only miruky-dwjrjfrjsvxegfiu, and choose a time range that covers the three invocations. The active Logs Insights experience may represent the chosen group in its data-source control or in an account-specific generated SOURCE line. Leave a generated line in place and replace only the remaining query body with:

fields order.ref, order.customer.tier, route
| filter message = "order routed"
| filter order.customer.tier = "gold"
| sort @timestamp asc
| dedup order.ref
Enter fullscreen mode Exit fullscreen mode

Logs Insights selects only the generated log group and filters the nested customer tier.

The selected data source limits the query to miruky-dwjrjfrjsvxegfiu. The query begins with fields to retain only the three application columns, while dot notation addresses the nested order.customer.tier value. After sorting, dedup order.ref keeps the earliest result for each distinct order reference. dedup is last because only limit may follow it in Logs Insights QL.

Run the query. The result should contain two rows: order-blue and order-violet, each with tier gold and route priority.

The query returns the two distinct gold order records and their nested tier values.

The result rows show order-blue and order-violet, each with tier gold and route priority. For a rerun, Showing 2 of N matched means that dedup reduced repeated gold records to two distinct references instead of hiding them with a narrower time window.

order-green should be absent because its nested tier is standard. The query selects only the three application fields; @message and @requestId are not needed to identify each order or its route.

Wrap-up

Lambda emitted structured application records through the Python logger's extra argument. CloudWatch Logs Insights evaluated the nested order.customer.tier value directly and kept only the two distinct gold-tier orders.

Structured logging moves useful dimensions out of ad hoc message parsing. A stable logging contract still matters; each investigation should limit its log groups, time range, and output columns.

Thanks for reading this far.

See you in the next one.

Disclosure: This article was written with AI assistance and independently verified against the linked primary sources and observed results.

References

Top comments (0)