miruky

Posted on Sep 8

Trigger AWS Lambda from DynamoDB Streams in the AWS Console

#tutorial #aws #dynamodb #lambda

Introduction

Hi, I'm miruky.

The Enabled label on a Lambda trigger confirms that an event source mapping exists. It does not prove that a table change reached the function, that the function handled the record, or that the result landed in the intended log group. I wanted this Console run to keep those states separate and make the final evidence narrow enough to publish without exposing an account-scoped ARN or a complete DynamoDB event.

The validation uses one on-demand table, one Python function, one custom CloudWatch log group, and one synthetic item. I create it in a pending state, wait for the insertion record, and then update it to complete. The last query must show both changes in order.

DynamoDB Streams keeps item-level change records for up to 24 hours. Lambda polls the stream through an event source mapping and invokes the function with batches of records. The application does not need to poll the table, but there are still four distinct states to verify: the stream is enabled, the mapping is enabled, the insert was delivered, and the update was delivered.

All regional resources in this article use us-east-1, shown in the Console as United States (N. Virginia). IAM is a global service, so its role does not have a Region selector. The run can incur charges for DynamoDB requests, Lambda execution, CloudWatch log ingestion and storage, and CloudWatch Logs Insights data scanned. AWS does not charge separately for GetRecords calls that Lambda makes for a DynamoDB trigger.

What the Console path contains

The Console labels this integration as a DynamoDB trigger, which compresses several AWS resources into one phrase. The table owns the DynamoDB stream. Lambda owns the event source mapping that reads that stream. The function owns the handler code, and its execution role authorizes the stream and logging calls made during processing.

The event source mapping is the active poller. It stores the source, starting position, batch settings, and enabled state, then invokes Lambda when records are available. The function does not call GetRecords in the handler, and the DynamoDB application does not call Lambda. This division matters when a query is empty: table writes can exist while a new mapping is still starting, and an enabled mapping can exist before any write has happened.

This is a Console-only walkthrough. The signed-in identity needs permission to create and configure every resource used below. If an organization restricts IAM role creation or managed-policy attachment, use an administrator-provided equivalent instead of weakening the guardrail.

1. Fix the Region before creating resources

The run begins with one Regional boundary for the table, function, mapping, and logs. Keeping that boundary fixed prevents a table in one Region from being paired with a function in another; cross-Region DynamoDB triggers are not supported.

The header shows United States (N. Virginia) while the DynamoDB Console is in English. This is the Region used by the table, stream, Lambda function, event source mapping, and CloudWatch log group throughout the run.

Open the AWS Console, switch the interface to English, and choose United States (N. Virginia) before creating a resource. The retained header confirms the selection without showing the account menu.

The resource names in the screenshots are generated for this validation. Use your own unique names if you repeat the steps. The synthetic item key and status values can remain item-001, pending, and complete because they are application data rather than AWS control-plane resource names.

2. Create an empty DynamoDB table

Open DynamoDB, choose Tables, and create a table. Enter miruky-kzfxrkmuilklwyvp for the table name and configure a single partition key named id with type String. Do not add a sort key or secondary index for this run.

Keep the default table settings. The current DynamoDB default is on-demand capacity mode, which bills reads and writes by request instead of reserving provisioned throughput. This validation performs only the item operations needed for the two stream records, although account-level pricing and free-tier eligibility still apply.

The form contains miruky-kzfxrkmuilklwyvp, the id partition key, and the String type. Default settings remains selected, and no item exists when the table is created.

Submit the form and wait for the table to become active. A stream enabled on an empty table gives TRIM_HORIZON a clean starting point later. It also lets the pre-write Logs Insights query distinguish an empty baseline from a filter that accidentally hides earlier test output.

3. Enable new and old images on the stream

Open the active table and choose Exports and streams. In DynamoDB stream details, turn on the stream and select New and old images.

DynamoDB offers four stream view types. Key-only records include the primary key, New image includes the item after a change, Old image includes the item before a change, and New and old images includes both images when they apply. This function logs only the new status, but selecting both images makes the MODIFY record suitable for later comparisons without recreating the stream.

The table now shows stream status On and view type New and old images. No application write has occurred, so the stream is active without a retained item-001 record.

A stream view type cannot be edited in place. Changing it requires disabling the current stream and enabling a new one, which produces a different stream descriptor. Confirm the view type here before configuring the Lambda trigger.

eventName identifies INSERT, MODIFY, or REMOVE. The dynamodb object carries Keys and the images allowed by the view type. DynamoDB retains attribute type wrappers, so this handler indexes S for its two string fields and logs only NewImage.

4. Give the consumer stream and log permissions

Open IAM, choose Roles, and create a role for the Lambda service. Name it miruky-aptegpkapbaqlkjx and attach the AWS managed policy AWSLambdaDynamoDBExecutionRole.

That managed policy currently allows the DynamoDB Streams read calls and CloudWatch Logs write calls required by this integration, with resource scope *. It matches the Console trigger documentation; for production, review whether a customer-managed policy can narrow the stream and log resources.

The role trust relationship must allow the lambda.amazonaws.com service principal to assume the role. Attaching permissions to a role with the wrong trust policy will not give the function an executable identity.

The role page shows miruky-aptegpkapbaqlkjx, Permissions policies (1), and an AWS managed policy row. The policy name is truncated in the Console table, so the exact AWSLambdaDynamoDBExecutionRole selection comes from the preceding creation step and the linked AWS policy reference rather than from this crop.

The stream mapping uses the role to obtain stream records, while the function uses the same role when it creates a log stream and writes log events. Creating the role before the function makes both dependencies available in the function creation form.

5. Create a focused CloudWatch log group

Open CloudWatch, choose Log groups, and create a log group named miruky-fkkeimmjwetqcevh. Select the Standard log class and set retention to 1 day.

Lambda can send logs to an existing custom log group instead of its usual /aws/lambda/<function-name> group. A dedicated group isolates this query, while one-day retention limits how long the validation output remains.

The row shows miruky-fkkeimmjwetqcevh, Standard, and 1 day. The group exists before the function is invoked, so Lambda only needs to create its runtime log stream and put events into this destination.

Custom Lambda log group names must follow the CloudWatch Logs naming rules and must not begin with aws/. The generated name shown here avoids that reserved prefix.

6. Create the Lambda function

Open Lambda, choose Create function, and use Author from scratch. Enter miruky-nvgccstdrxrmkgsk for the function name, select Python 3.14, and choose the existing role miruky-aptegpkapbaqlkjx.

Python 3.14 is a supported Lambda managed runtime based on Amazon Linux 2023 at validation time. Check the current runtime table before repeating the article later.

After the function is created, open its logging configuration and select miruky-fkkeimmjwetqcevh as the custom log group. Keep the log format as text because the handler emits one deliberately prefixed line that Logs Insights can parse.

The logging configuration shows miruky-fkkeimmjwetqcevh as the CloudWatch log group and Text as the Log format. The runtime and execution role were selected during function creation; this crop is limited to the log destination so it does not expose account-scoped role details.

Replace the generated code with this handler and choose Deploy:

import json


def lambda_handler(event, context):
    processed = []

    # Keep public validation output limited to synthetic fields.
    for record in event.get("Records", []):
        dynamodb = record.get("dynamodb", {})
        new_image = dynamodb.get("NewImage", {})
        processed.append(
            {
                "event": record.get("eventName"),
                "id": dynamodb["Keys"]["id"]["S"],
                "status": new_image.get("status", {}).get("S"),
            }
        )

    print("STREAM_RESULT " + json.dumps(processed, separators=(",", ":")))
    return {"processed": len(processed)}

The handler does not print the complete event or context. Only three synthetic fields reach the STREAM_RESULT line that Logs Insights filters and parses.

NewImage exists for the INSERT and MODIFY records used here. A REMOVE record has no new image, so this compact handler would log a null status for deletion. Handling deletes, malformed records, and partial batch failures belongs in application code rather than in this delivery check.

The editor shows Current deployment state, the event field, and the STREAM_RESULT prefix. The code block above holds the complete handler because the compact Console line extends beyond the editor width.

7. Add the DynamoDB trigger

In the Lambda function overview, choose Add trigger and select DynamoDB. Choose the stream for miruky-kzfxrkmuilklwyvp, set Batch size to 10, choose Trim horizon for the starting position, and leave Enable trigger selected.

The trigger creates an event source mapping. Lambda polls the stream, assembles batches, invokes the function, and tracks shard progress. Batch size 10 is a ceiling; Lambda can invoke with fewer records.

Trim horizon starts at the oldest retained record. Mapping creation is eventually consistent, and AWS warns that Latest can miss records written while polling starts. TRIM_HORIZON avoids that gap; the empty table has nothing earlier to replay.

Submit the mapping and wait for it to become enabled. Writing the item while the mapping is still being created would combine two unknowns: whether the source contains a record and whether Lambda has begun polling it.

The trigger list shows DynamoDB: miruky-kzfxrkmuilklwyvp and state: Enabled. The source ARN is excluded from the retained image because it contains an account identifier and is not needed to prove the selected table and current mapping state.

Enabled confirms the mapping configuration, not record delivery or successful logging. The empty query, insertion checkpoint, and later update keep those states separate while the query and log group remain unchanged.

8. Establish an empty log baseline

Open CloudWatch Logs Insights and select only miruky-fkkeimmjwetqcevh. Use a short time range that covers the current validation session, then run this query before writing an item:

fields @timestamp, @message
| filter @message like /STREAM_RESULT/
| parse @message /STREAM_RESULT (?<stream_result>.*)$/
| sort @timestamp asc
| display stream_result

The filter discards Lambda runtime lines such as start, end, and report messages. The named regular expression capture stores the text after STREAM_RESULT in stream_result, and the last query line limits the result table to that extracted field. Do not add raw log fields to public evidence.

The query returns no matched stream_result row before inserting the item. This baseline confirms that the selected group and time range do not contain output from an earlier invocation.

An empty result alone cannot prove that the trigger works. Its job is narrower: it establishes the before-state. The two item operations and the final query provide the delivery evidence.

9. Create the item and wait for INSERT

Return to the DynamoDB table, choose Explore table items, and create this item:

{
  "id": "item-001",
  "status": "pending"
}

After using Explore table items, save the item and keep the query unchanged until its stream_result contains the insertion outcome for item-001 in the pending state. Do not update the item yet.

Waiting for INSERT gives each operation an independent checkpoint. An immediate update could place both records in one batch before insert delivery had been observed.

The stream itself records each item modification once. Lambda event source mappings process records at least once, so a function can still receive a duplicate after a retry. Production consumers should make repeated processing harmless instead of treating one invocation as an exactly-once guarantee.

10. Update the item to complete

Once the insertion log is present, change the item state from pending to complete without changing the partition key. Both records will then describe modifications to the same DynamoDB item.

The saved item now shows item-001 with status set to complete. The preceding create operation used the same key with pending, so the stream should classify this update as MODIFY.

A write that leaves every attribute unchanged does not produce a DynamoDB stream record. Changing pending to complete gives the update an observable data difference and keeps the expected result precise.

11. Verify INSERT and MODIFY

Rerun the unchanged Logs Insights query over a time range that includes both writes until the update record arrives.

The final result contains exactly the two focused rows observed in this run:

[{"event":"INSERT","id":"item-001","status":"pending"}]
[{"event":"MODIFY","id":"item-001","status":"complete"}]

INSERT/pending appears before MODIFY/complete, and both entries use item-001. That order matches the two modifications made to one primary key. The result column excludes the account ID, stream ARN, log stream name, request ID, sequence number, and complete DynamoDB event.

Each visible array contains one record despite the batch-size ceiling of 10. Downstream behavior must not assume a fixed number per invocation.

This result proves the path used in the article: the table emitted the two stream records, the event source mapping delivered them, the handler extracted the selected fields, and the custom log group received the output. It does not prove exactly-once processing, production retry behavior, event filtering, or partial-batch recovery.

If a row has not arrived, keep the item, log group, and query fixed. Include the write time and mapping startup delay in the query range. If INSERT exists but MODIFY does not, first confirm that the item contains complete, then rerun the same query. For a function error, inspect the custom group privately, correct and deploy the handler, and use a new synthetic key so an earlier retry cannot imitate recovery.

Wrap-up

The useful boundary in this walkthrough is the gap between configuration and delivery. An enabled stream and an enabled trigger establish that the route exists; the empty baseline followed by INSERT/pending and MODIFY/complete establishes that records traveled through it.

DynamoDB retains stream data for up to 24 hours and preserves modification order for one item. Lambda polls those records through an event source mapping and processes them at least once. For production code, keep the handler idempotent, decide how partial batch failures should be reported, and log only the fields needed to operate the workload.

Thanks for reading this far.

See you in the next one.

Disclosure: This article was written with AI assistance and independently verified against the linked primary sources and observed results.

DEV Community