miruky

Posted on Sep 6

Find One Failed Lambda Invocation with CloudWatch Logs Insights

#aws #lambda #cloudwatch #tutorial

Introduction

Hi, I'm miruky.

A Lambda error is visible while a test result remains open, but the durable troubleshooting trail lives in CloudWatch Logs. CloudWatch Log Analytics can run a Logs Insights query across the selected log group without opening individual log streams.

This Console run invokes one tiny Python function three times: a control invocation succeeds, the failure invocation raises an intentional error, and the same control succeeds again afterward. The handler writes VALIDATION_FAILURE exactly once when that controlled failure occurs, so a narrowly scoped query can map one matching log event back to the one failed invocation in this run.

The function runs only three times and the log group keeps data for one day. Lambda execution, CloudWatch log ingestion and storage, and data scanned by Logs Insights can still incur charges, so check the current pricing pages linked below.

1. Create the role and log group

This run uses the AWS Console in English and United States (N. Virginia). Lambda, CloudWatch Logs, and every test invocation stay in us-east-1; IAM remains a global service.

The header confirms United States (N. Virginia) while the Lambda Console is in English. This fixes the Region shared by the function, log group, and query.

The exact filter for miruky-fsvgfpmfvqosvpil returns no role. That empty result establishes the IAM resource boundary before creation.

In IAM, use that exact role-list filter before choosing Create role. Select the Lambda use case, attach the AWS managed policy AWSLambdaBasicExecutionRole, name the role miruky-fsvgfpmfvqosvpil, and create it. That managed policy supplies the CloudWatch Logs write permissions required by the function; this exercise adds no application-service permissions.

The role view confirms miruky-fsvgfpmfvqosvpil and one attached AWS managed policy. The policy selected in the preceding wizard was AWSLambdaBasicExecutionRole; this run added no inline or unrelated managed policy.

Open CloudWatch, choose Log groups, and search for miruky-jsykmubbcucoxtqr. Create a Standard log group with that exact name, then set its retention to 1 day.

The log-group row shows miruky-jsykmubbcucoxtqr, Standard, and 1 day. This destination exists before the function sends its initial log event.

2. Create a function with a custom log destination

In Lambda, search for miruky-xypwnfccbzlnwnmd and confirm that the exact function name is unused. Choose Create function, select Author from scratch, and use the Python 3.14 runtime. Under the execution-role settings, enable Custom execution role, choose the existing role miruky-fsvgfpmfvqosvpil, and save that role selection.

The form shows miruky-xypwnfccbzlnwnmd, Python 3.14, and miruky-fsvgfpmfvqosvpil together. These values fix the runtime and role before creation.

After creation, open Configuration, then Monitoring and operations tools, and edit the logging configuration. Keep CloudWatch Logs as the destination, select a Custom log group, enter miruky-jsykmubbcucoxtqr, and retain the plain-text log format.

The logging settings show CloudWatch Logs, Custom, miruky-jsykmubbcucoxtqr, and Text. The function is therefore pointed at the pre-created one-day group.

Replace the generated function code with this handler and choose Deploy:

def lambda_handler(event, context):
    # Raise only for the controlled failure event used by the later query.
    if event.get("mode") == "fail":
        raise RuntimeError("VALIDATION_FAILURE order-001")

    return {"status": "ok"}

The success path returns normally. Only the failure path emits the exact marker, so the later query has a controlled signal rather than a broad search for every occurrence of the word “error.”

The deployed editor contains the fixed ok response and VALIDATION_FAILURE order-001 exception marker. No event value is written back to the response.

3. Produce one control and one failure

On the Code tab, expand TEST EVENTS. Create a private test event named miruky-yzgswgzuasvcgvbe with this JSON, save it, and invoke the function once:

{
  "mode": "succeed"
}

Invoke the saved event once. This control checks the function and its execution role before the intentional exception is introduced.

The cropped result reports Status: Succeeded and shows "status": "ok" for the control event. Execution logs and request identifiers are excluded.

Create another private test event named miruky-bakfjxlqakdamrjk with {"mode":"fail"} and run it once.

The cropped result reports Status: Failed, RuntimeError, and VALIDATION_FAILURE order-001. Dynamic request and execution-environment values are outside the image.

The first two invocations differ by only one event value. Do not publish the request ID, complete log stream name, execution-environment identifier, function ARN, role ARN, or account menu while recording this evidence.

Select miruky-yzgswgzuasvcgvbe again and invoke the unchanged function once more. This recovery control checks that the intentional exception did not alter the handler or its configuration.

The recovery control reports Status: Succeeded and shows "status": "ok" again. The function is operational after the isolated failure, while only the one failure invocation contains the query marker.

4. Count the failed invocation without revealing its ID

CloudWatch Logs delivery can take several minutes. Open Log Analytics under Logs in CloudWatch. Log Analytics is now the default unified Console experience, and it runs CloudWatch Logs Insights queries alongside Live Tail and Contributor Insights.

Use Query by to target a log-group prefix, enter miruky-jsykmubbcucoxtqr, and confirm that the Console reports one matched log group. Keep the time window at 1,800 seconds, which is 30 minutes immediately after this run.

The scope shows miruky-jsykmubbcucoxtqr, View matched log groups (1), Last 1800 seconds, and Standard. No unrelated log group is included.

Run this query:

SOURCE logGroups(namePrefix: ["miruky-jsykmubbcucoxtqr"], class: "STANDARD") START=-1800s END=0s |
filter @message like /VALIDATION_FAILURE/
| stats count(*) as failed_invocations

SOURCE fixes the Standard-class log-group prefix and the time boundary without embedding a log-group ARN. The filter keeps only the controlled marker, and count(*) returns one matching failure record without displaying request identifiers or raw log messages.

The result table contains failed_invocations with value 1. Above it, the Console shows Complete: Showing 1 of 1 matched. and Query executed for 1 log group.

The expected value is 1. If the result is 0, keep the same log-group prefix and narrow time range, wait for log delivery, and rerun the query rather than widening it across unrelated groups.

This one-to-one interpretation belongs to this handler: one failed invocation raises one exception containing VALIDATION_FAILURE. If an application can write the same marker more than once per invocation, count(*) counts matching log events rather than invocations. Use structured application fields, a metric, or another correlation design before applying this query shape to production traffic.

Wrap-up

The control invocation returned normally before and after one intentional invocation raised the fixed runtime error. The Logs Insights query matched the handler's one failure record while keeping request identifiers and raw log messages out of the result.

The same query shape avoids manually opening individual log streams: use a specific application marker, select only the relevant log group, choose a tight time window, and return an aggregate instead of the underlying records.

The plain-text marker keeps this validation result directly visible in the failure line. For a production function, prefer a stable field in structured JSON unless existing tooling requires text; AWS recommends JSON because Logs Insights can search and filter named fields without parsing free-form prose. Set retention and access controls to match the sensitivity and operational lifetime of those logs.

Thanks for reading this far.

See you in the next one.

Disclosure: This article was written with AI assistance and independently verified against the linked primary sources and observed results.

DEV Community