Build a DynamoDB Outbox and an Idempotent SQS Consumer in the Console

Introduction

Hi, I'm miruky.

A database commit can succeed while the caller believes processing failed. Retrying then becomes a correctness problem: did the business effect already happen?

I built this in the AWS Console and failed the consumer after its transaction. You need resource-creation and role-passing permissions; no terminal is required.

1. Two transactions, with a retryable gap between them

The command transaction stores the order and its outbox event together. The consumer transaction stores an inbox marker and updates the sales summary together; the queue between them can deliver repeatedly.

The relay cannot atomically send to SQS and advance its stream checkpoint, so the consumer must tolerate repeated delivery. An outbox preserves event intent; it does not provide exactly-once transport.

Deduplication uses the application identifier event_id. A payload digest rejects conflicting content under that identity, independently of transport IDs.

2. A small validation budget

The target is below $5 in us-east-1, excluding tax and unrelated usage. This is not a spending cap; the final billed amount is unverified.

A deliberately oversized one-day allowance is approximately $0.58 before free-tier benefits: 10,000 Lambda invocations at 256 MB and 10 billed seconds, 100,000 SQS request units, 100,000 DynamoDB write units, 50,000 read units, 0.1 GB of ingested logs, and tiny stored data. These are cost-model allowances, not instructions to generate traffic. Transactional reads/writes consume additional units; count billing units, not business events. Current rates: Lambda, SQS, DynamoDB, and CloudWatch (checked September 5, 2026).

Use only the supplied inputs. This setup needs no NAT gateway, customer-managed KMS key, provisioned polling, public endpoint, or continuous producer.

3. Create the storage and queues

Prepare three short-retention log destinations. Keep a mapping if you substitute these random resource names.

The header shows United States (N. Virginia), and Create log group has Retention setting set to 1 day with Standard selected. Create three groups using the same settings, without a customer-managed encryption key, so each function has its own destination.

Function purpose	Log group
Command	miruky-agosvhxwcqobcbnx
Relay	miruky-hhndjnjpuqvfgxob
Consumer	miruky-xmtozprxyfsivjnx

The command table stores business state and outgoing event intent. Different key prefixes distinguish them.

In DynamoDB's Create table form, use miruky-gohtgdyurfkykbbq with Partition key PK and Sort key SK, both String. These keys support the order and outbox items without secondary indexes.

The defaults show DynamoDB Standard, On-demand, and AWS owned key. Retain those settings and create the table; wait for creation to finish before configuring its change stream.

The table's DynamoDB stream details panel provides Turn on. Use it to configure the stream that will feed the relay, rather than adding a message send to the command transaction.

Select New image and finish enabling the stream. The relay needs the complete inserted outbox body, so key-only records would not contain its input; retain the resulting stream ARN for the relay policy.

Create miruky-zlsawfrdkceltmge with the same PK and SK string keys. Keep this second table on demand too; it stores inbox markers and the sales summary, and does not need a stream.

In SQS, create miruky-cbezwpehumcfgypl as Standard, with Message retention period 4 Days and Receive message wait time 20. This queue is the failure destination, so create it before the delivery queue that references it.

Keep encryption enabled with Amazon SQS key (SSE-SQS) and finish creating the queue. This choice does not introduce a customer-managed KMS key into the exercise.

Create miruky-qbqexveshjqbcqzz as Standard, with Visibility timeout 60, retention 1 Days, and receive wait 20. The 60-second visibility matches AWS's six-times-timeout recommendation for the 10-second consumer used here. Lambda/SQS configuration

Enable Dead-letter queue, choose the queue ending in miruky-cbezwpehumcfgypl, and set Maximum receives to 5, then choose Create queue. Keep SSE-SQS here too; record this delivery queue's URL and ARN for the function configuration and policies.

4. Give each function its own permissions

The functions need different data access. Three roles make those boundaries explicit.

Choose AWS service, select Lambda as the service and use case, then continue with Next. Repeat for the three roles below, using the corresponding inline permission policy rather than broad managed policies.

Purpose	Role	Inline policy
Command	miruky-pvdxcewewpfupyyc	miruky-mzcyfrqfmwwljmcd
Relay	miruky-ghicniunbdkuqqju	miruky-fnvmisucvthsrwio
Consumer	miruky-ewfkyewmdvtrriar	miruky-mocgbtjiycynmyid

Replace the policy placeholders with your table, current stream, delivery-queue, and log-group ARNs. Each log resource ends in :* to address its streams.

The Inline policy summary shows the command policy name and access to CloudWatch Logs and DynamoDB. The full documents below define the actual action/resource boundaries; the summary alone is not a permission audit.

Command policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteOrderAndOutbox",
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem"
      ],
      "Resource": "SOURCE_TABLE_ARN"
    },
    {
      "Sid": "WriteFunctionLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "COMMAND_LOG_GROUP_ARN:*"
    }
  ]
}

Relay policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ForwardOutbox",
      "Effect": "Allow",
      "Action": "sqs:SendMessage",
      "Resource": "DELIVERY_QUEUE_ARN"
    },
    {
      "Sid": "ReadSourceStream",
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator"
      ],
      "Resource": "SOURCE_STREAM_ARN"
    },
    {
      "Sid": "DiscoverStreams",
      "Effect": "Allow",
      "Action": "dynamodb:ListStreams",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": "us-east-1"
        }
      }
    },
    {
      "Sid": "WriteFunctionLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "RELAY_LOG_GROUP_ARN:*"
    }
  ]
}

Consumer policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteInboxAndAggregate",
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem"
      ],
      "Resource": "PROJECTION_TABLE_ARN"
    },
    {
      "Sid": "ConsumeDeliveryQueue",
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "DELIVERY_QUEUE_ARN"
    },
    {
      "Sid": "WriteFunctionLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "CONSUMER_LOG_GROUP_ARN:*"
    }
  ]
}

TransactWriteItems authorization uses its component item operations, hence the PutItem and UpdateItem permissions. ListStreams has no resource-level scope, so its wildcard is constrained by Region; the other stream reads use the exact stream ARN. DynamoDB authorization reference

Your signed-in operator also needs resource-creation permissions, iam:PassRole for these roles, and DLQ-redrive permissions. Those operator permissions do not belong in the consumer role. Redrive permissions

5. Create the three functions

Each handler is short enough to paste into the browser editor. The exercise uses the runtime-provided Boto3 SDK; package and pin dependencies separately when moving this code into a maintained application. Runtime dependencies

Choose Author from scratch and Python 3.14. Leave Durable execution and EC2 capacity provider off; these handlers use ordinary Lambda execution.

Under Additional settings, select the matching Custom execution role and choose Create function. Leave ARM64 architecture, Function URL, and VPC off for the tested x86 configuration.

Purpose	Function
Command	miruky-rpxwrwxhrayiuybs
Relay	miruky-gwydnurrkyqxojss
Consumer	miruky-bjftppshbgtwjjhy

Repeat for each function. Identical execution limits preserve the queue visibility calculation.

Set Memory to 256 MB and Timeout to 10 seconds, retain Ephemeral storage 512 MB and SnapStart None, then choose Save. These limits bound individual invocations, not total account spending.

Select Custom, enter the corresponding Custom log group, and leave Add required permissions unchecked before Save. The dedicated role already contains the logging permissions for its precreated group. Custom log groups

Add SOURCE_TABLE with the source-table name and choose Save. Configure the relay's QUEUE_URL with the delivery queue URL and the consumer's PROJECTION_TABLE with the projection-table name; do not add the fault setting yet.

Replace lambda_function.py with the matching handler below, then choose Deploy for each function. The handler entry point remains lambda_function.lambda_handler; do not send test traffic until both event-source mappings exist.

Command handler

The transaction conditionally writes both items. A retry with identical order content returns the existing identity; a conflicting request fails.

import json
import os
import re

import boto3
from botocore.config import Config

# Keep the database commit and the event intent in the same transaction.
db = boto3.resource('dynamodb', config=Config(
    connect_timeout=2, read_timeout=2,
    retries={'mode': 'standard', 'total_max_attempts': 2},
)).meta.client
TABLE = os.environ['SOURCE_TABLE']


def lambda_handler(event, context):
    order_id = event.get('order_id')
    amount = event.get('amount_cents')
    if not isinstance(order_id, str) or not re.fullmatch(r'[a-z0-9-]{1,40}', order_id):
        raise ValueError('order_id must contain 1-40 lowercase letters, digits or hyphens')
    if type(amount) is not int or not 1 <= amount <= 1000000:
        raise ValueError('amount_cents must be an integer from 1 to 1000000')
    event_id = 'order-created:' + order_id
    body = json.dumps({'event_id': event_id, 'type': 'OrderCreated',
                       'order_id': order_id, 'amount_cents': amount},
                      sort_keys=True, separators=(',', ':'))
    order = {'PK': 'ORDER#' + order_id, 'SK': 'STATE', 'kind': 'ORDER',
             'order_id': order_id, 'amount_cents': amount, 'status': 'ACCEPTED',
             'event_id': event_id}
    outbox = {'PK': 'OUTBOX#' + event_id, 'SK': 'EVENT',
              'kind': 'OUTBOX', 'event_id': event_id, 'body': body}
    try:
        db.transact_write_items(TransactItems=[
            {'Put': {'TableName': TABLE, 'Item': order,
                     'ConditionExpression': 'attribute_not_exists(PK)'}},
            {'Put': {'TableName': TABLE, 'Item': outbox,
                     'ConditionExpression': 'attribute_not_exists(PK)'}},
        ])
    except db.exceptions.TransactionCanceledException:
        stored = db.get_item(TableName=TABLE, Key={'PK': order['PK'], 'SK': 'STATE'},
                             ConsistentRead=True).get('Item')
        if stored != order:
            raise
        return {'result': 'ALREADY_ACCEPTED', 'event_id': event_id}
    return {'result': 'ACCEPTED', 'event_id': event_id}

Outbox relay

The relay forwards the stored body unchanged. A failed send reports the DynamoDB sequence number for retry.

import os

import boto3
from botocore.config import Config

# A successful queue send is not atomic with the stream checkpoint.
sqs = boto3.client('sqs', config=Config(
    connect_timeout=2, read_timeout=2,
    retries={'mode': 'standard', 'total_max_attempts': 2},
))
QUEUE_URL = os.environ['QUEUE_URL']


def lambda_handler(event, context):
    for record in event['Records']:
        image = record.get('dynamodb', {}).get('NewImage', {})
        if record.get('eventName') != 'INSERT' or image.get('kind', {}).get('S') != 'OUTBOX':
            continue
        try:
            sqs.send_message(QueueUrl=QUEUE_URL, MessageBody=image['body']['S'])
            print('FORWARDED ' + image['event_id']['S'])
        except Exception as error:
            print('RELAY_RETRY ' + type(error).__name__)
            return {'batchItemFailures': [
                {'itemIdentifier': record['dynamodb']['SequenceNumber']}
            ]}
    return {'batchItemFailures': []}

Idempotent consumer

The conditional inbox insert and aggregate increment share one transaction. After a canceled transaction, a strongly consistent marker read distinguishes a matching duplicate from an unresolved failure or conflicting payload.

import hashlib
import json
import os
import re

import boto3
from botocore.config import Config

# The inbox marker and the aggregate change succeed or fail together.
db = boto3.resource('dynamodb', config=Config(
    connect_timeout=2, read_timeout=2,
    retries={'mode': 'standard', 'total_max_attempts': 2},
)).meta.client
TABLE = os.environ['PROJECTION_TABLE']
FAULT_ORDER = os.environ.get('FAULT_AFTER_COMMIT', '')


def process(body):
    message = json.loads(body)
    if not isinstance(message, dict) or set(message) != {'event_id', 'type', 'order_id', 'amount_cents'}:
        raise ValueError('Unexpected event schema')
    order_id = message['order_id']
    amount = message['amount_cents']
    if not isinstance(order_id, str) or not re.fullmatch(r'[a-z0-9-]{1,40}', order_id):
        raise ValueError('Invalid order_id')
    if message['type'] != 'OrderCreated' or message['event_id'] != 'order-created:' + order_id:
        raise ValueError('Invalid event identity')
    if type(amount) is not int or not 1 <= amount <= 1000000:
        raise ValueError('Invalid amount_cents')
    digest = hashlib.sha256(json.dumps(message, sort_keys=True,
                                      separators=(',', ':')).encode()).hexdigest()
    key = {'PK': 'INBOX#' + message['event_id'], 'SK': 'EVENT'}
    result = 'APPLIED'
    try:
        db.transact_write_items(TransactItems=[
            {'Put': {'TableName': TABLE,
                     'Item': {**key, 'kind': 'INBOX', 'digest': digest},
                     'ConditionExpression': 'attribute_not_exists(PK)'}},
            {'Update': {'TableName': TABLE, 'Key': {'PK': 'SALES', 'SK': 'TOTAL'},
                        'UpdateExpression': 'SET kind = :kind ADD order_count :one, revenue_cents :amount',
                        'ExpressionAttributeValues': {':kind': 'SUMMARY', ':one': 1, ':amount': amount}}},
        ])
    except db.exceptions.TransactionCanceledException:
        stored = db.get_item(TableName=TABLE, Key=key, ConsistentRead=True).get('Item')
        if not stored or stored.get('digest') != digest:
            raise
        result = 'DUPLICATE_IGNORED'
    print(result + ' ' + message['event_id'])
    # This controlled fault happens after the committed effect, not before it.
    if order_id == FAULT_ORDER:
        raise RuntimeError('INJECTED_AFTER_COMMIT')


def lambda_handler(event, context):
    failures = []
    for record in event['Records']:
        try:
            process(record['body'])
        except Exception as error:
            print('RETRY ' + type(error).__name__ + ': ' + str(error))
            failures.append({'itemIdentifier': record['messageId']})
    return {'batchItemFailures': failures}

The fault is checked after both the applied and duplicate branches. Leaving it enabled therefore keeps the message failing until the DLQ threshold, without repeating the sales increment.

6. Connect the event sources

Connect the source table to the relay and the delivery queue to the consumer. Neither mapping targets the command.

On the relay's Add trigger page, select DynamoDB, the source table, and Activate trigger. Set Batch size 1, Starting position Trim horizon, and Batch window 0; leave Enable EventCount metrics unchecked.

Under Additional settings, set Retry attempts 100 and Maximum age of record 3600, leaving On-failure destination empty. This is a bounded relay demonstration: exhausted retries or an expired record can stop forwarding, despite the outbox item remaining stored. Stream mapping parameters

Enable Report batch item failures, keep Concurrent batches per shard 1, and enter the following Filter criteria. Finish creating the mapping; only inserted outbox items should reach the relay. DynamoDB filtering

{"eventName":["INSERT"],"dynamodb":{"NewImage":{"kind":{"S":["OUTBOX"]}}}}

On the consumer's Add trigger page, select SQS and the delivery queue with Activate trigger checked. Leave Provisioned mode unconfigured and Enable EventCount metrics unchecked; the next settings govern batch failure handling.

Set Batch size 1, Batch window 0, Maximum concurrency 2, and enable Report batch item failures, then choose Add. Wait for both mappings to activate; otherwise the handler's failure response will not have the intended retry behavior. Partial batch responses

7. Commit the first order

Start with one input. The tables distinguish acceptance from counting.

Create a Private test event with the Event JSON below. Save it, then invoke the command once; this is a business input, not a fabricated stream or SQS event envelope.

{"order_id":"order-a","amount_cents":1000}

The Test result reports Executing function: succeeded and returns ACCEPTED with order-created:order-a. That response establishes command acceptance; inspect storage next to confirm the resulting items.

The source-table scan returns ORDER#order-a and OUTBOX#order-created:order-a. Both records belong to the command transaction; now inspect the projection table after asynchronous delivery has had time to run.

The projection scan shows INBOX#order-created:order-a and the SALES/TOTAL summary with order_count 1 and revenue_cents 1000. This is the baseline business effect against which the duplicate test will be compared.

8. Send the same application event again

A fresh transport message can carry an already-processed business event. Send the following body to the delivery queue once.

Use Send and receive messages, enter the body below, and choose Send message. The green confirmation shows the send succeeded; do not poll this delivery queue while Lambda is its consumer.

{"event_id":"order-created:order-a","type":"OrderCreated","order_id":"order-a","amount_cents":1000}

The consumer log search for order-created:order-a shows APPLIED followed by DUPLICATE_IGNORED. The duplicate branch actually ran; an unchanged table alone would not prove processing.

9. Fail after the sales update

The harder case is a committed database effect followed by an unsuccessful message-processing response. Enable the controlled fault before submitting the second order.

Add FAULT_AFTER_COMMIT with value order-b to the consumer and choose Save. After the update completes, invoke the command once with this new input and wait several minutes for the delivery retries.

{"order_id":"order-b","amount_cents":2500}

The log sequence starts with APPLIED order-created:order-b and RETRY RuntimeError: INJECTED_AFTER_COMMIT. Later attempts report DUPLICATE_IGNORED before failing again, which is the intended post-commit fault rather than a failed database write.

The projection already contains both inbox markers and a summary of order_count 2, revenue_cents 3500. The message can still fail while its business effect is committed; the DLQ body provides the next piece of evidence.

The received message's Body retains order-created:order-b and 2500. I inspected it with one bounded receive from the DLQ; Console receives also increment receive counts, so they must not be mistaken for additional Lambda attempts. DLQ receive behavior

10. Recover, redrive, and inspect the business result

Remove the injected fault before returning the message to its consumer. Otherwise the deliberately failing branch will send it through the same retry cycle.

After editing and saving the consumer environment, Environment variables (1) contains only PROJECTION_TABLE, and the update-success banner is visible. The fault setting is gone; the redrive can now test the duplicate branch without the injected exception.

For the DLQ redrive, choose Redrive to source queue(s) and Custom max velocity, with Messages per second 1. Start the redrive after these settings; a low rate makes this single-message recovery observable. Native redrive

The redrive reaches 100% and Successfully completed, with destination Source queue(s). Task completion confirms the move, not the business result; inspect the consumer and projection once more.

The final log entry at 2026-09-05T14:36:30.630Z is DUPLICATE_IGNORED order-created:order-b, after the five earlier failing attempts. No later injected-error entry appears in this result, so compare the queue counters and stored total next.

Both queues show Messages available 0 and Messages in flight 0. These are approximate transport counters, so the projection remains the decisive evidence for whether sales were added twice. SQS metrics

The final scan still shows both inbox markers, order_count 2, and revenue_cents 3500. Redrive did not add another business effect for the retained event identity, completing the recovery test.

What this design does not prove

The guarantee is one database effect per retained application identity under these write rules. Expiring an inbox marker reopens its duplicate window; this exercise does not configure TTL.

The single SALES/TOTAL item is a contention point, not a throughput architecture. There is no global ordering guarantee and no atomicity with an external payment or email.

DynamoDB Streams retains records for 24 hours, while this relay's record-age setting is only one hour. A production design needs reconciliation from retained outbox records, or another durable recovery path, for a longer outage; neither is implemented here. Streams retention

Wrap-up

The useful result was a failed delivery whose sales update had already committed. Keeping the inbox marker and that update in the same transaction made the subsequent retries and redrive non-additive.

Thanks for reading this far. See you in the next one.

Disclosure: This article was written with AI assistance and independently verified against the linked primary sources and observed results.

References

• DynamoDB transaction APIs
• Lambda with Amazon SQS
• SQS dead-letter queues