DEV Community: CK

The Missing Link: Triggering Serverless Events from Legacy Databases with AWS DMS

CK — Wed, 07 Jan 2026 19:09:05 +0000

We live in a world where we want everything to be Event-Driven. We want a new user registration in our SQL database to immediately trigger a welcome email via SES, update a CRM via API, and start a Step Function workflow.

If you are building greenfield on DynamoDB, this is easy (DynamoDB Streams). But what if your data lives in a legacy MySQL monolith, an on-premise Oracle DB, or a standard PostgreSQL instance?

You need Change Data Capture (CDC). You need to stream these changes to the cloud.

Naturally, you look at AWS DMS (Database Migration Service). It’s perfect for moving data. But then you hit the wall:

The Problem: AWS DMS cannot target an AWS Lambda function directly.

You cannot simply configure a task to say "When a row is inserted in Table X, invoke Function Y".

So, how do we bridge the gap between the "Old World" (SQL) and the "New World" (Serverless)? We need a glue service. While many suggest Kinesis, often the most robust and cost-effective answer is Amazon S3.

Here is the architecture pattern I use to modernize legacy backends without rewriting them.

The Architecture: The "S3 Drop" Pattern

The flow works like this:

Source: DMS connects to your Legacy Database and captures changes (INSERT/UPDATE/DELETE) via the transaction logs.
Target: DMS writes these changes as JSON files into an S3 Bucket.
Trigger: S3 detects the new file and fires an Event Notification.
Compute: Your Lambda Function receives the event, reads the file, and processes the business logic.

Why S3 instead of Kinesis or Airbyte?

You might face two common alternatives when designing this:

1. Why not Kinesis Data Streams?
Kinesis is faster (sub-second latency). However, S3 is often superior for this specific use case because:

Cost: S3 is incredibly cheap compared to a provisioned Kinesis stream (especially if the legacy DB is quiet).
Observability: You can literally see the changes as files in your bucket. It makes debugging legacy data migration 10x easier.
Batching: DMS writes to S3 in batches. This naturally prevents your Lambda from being overwhelmed if the database takes a massive write hit.

2. Why not Airbyte or Fivetran?
Tools like Airbyte are fantastic for ELT (Extract Load Transform) pipelines, typically moving data to a warehouse like Snowflake every 15 or 60 minutes.
However, our goal here is Event-Driven capability. We want to trigger a Lambda function as close to "real-time" as possible. AWS DMS offers continuous replication (CDC), giving us a granular stream of events that batch-based ELT tools often miss. Furthermore, staying 100% AWS native simplifies IAM governance in strict enterprise environments.

The Implementation Guide

Here are the specific settings you need to make this work smoothly.

1. The DMS Endpoint Settings

When creating your Target Endpoint in DMS (pointing to S3), don't just use the defaults. You want the data to be readable by your Lambda.

Use these Extra Connection Attributes:

dataFormat=json;
datePartitionEnabled=true;

dataFormat=json: By default, DMS might use CSV. JSON is much easier for your Lambda to parse using standard libraries.

datePartitionEnabled=true: This organizes your files in S3 by date (/2023/11/02/...), preventing a single folder from containing millions of files.

2. Understanding the Event Structure

When DMS writes a file, it looks like this inside:

{
    "data": { "id": 101, "username": "jdoe", "status": "active" },
    "metadata": { "operation": "insert", "timestamp": "2023-11-02T10:00:00Z" }
}
{
    "data": { "id": 102, "username": "asmith", "status": "pending" },
    "metadata": { "operation": "update", "timestamp": "2023-11-02T10:05:00Z" }
}

You get the Operation (was it an insert or an update?) and the Data in one clean package.

3. The Lambda Logic

AWS DMS does not write a valid JSON array (e.g., [{...}, {...}]). It writes Line-Delimited JSON (NDJSON).

If you try to json.loads() the entire file content at once, your code will crash. You must iterate line-by-line.

Here is the Python boilerplate to handle this correctly:

import boto3
import json

s3 = boto3.client('s3')

def handler(event, context):
    # 1. Get the S3 Key from the event trigger
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']

        print(f"Processing file: {key}")

        # 2. Fetch the actual file generated by DMS
        obj = s3.get_object(Bucket=bucket, Key=key)
        content = obj['Body'].read().decode('utf-8')

        # 3. Parse DMS JSON (NDJSON / Line-Delimited)
        # CRITICAL: Do not use json.loads(content) directly!
        for line in content.splitlines():
            if not line.strip(): continue # Skip empty lines

            row = json.loads(line)

            # 4. Filter: Only process what you need
            operation = row.get('metadata', {}).get('operation')

            if operation == 'insert':
                user_data = row.get('data')
                print(f"New User Detected: {user_data['username']}")
                # trigger_welcome_email(user_data)

            elif operation == 'update':
                print(f"User Updated: {row['data']['id']}")

Summary

You don't need to refactor your entire legacy database to get the benefits of Serverless.

By using AWS DMS to unlock the data and S3 as a reliable buffer, you can trigger modern Lambda workflows from 20-year-old databases with minimal friction. It is a pattern that prioritizes stability and observability over raw speed—a trade-off that is usually worth it in enterprise migrations.

Mastering AWS Cross-Account Secrets with Terraform & KMS

CK — Wed, 03 Dec 2025 14:05:31 +0000

The Multi-Account Reality Check

In modern cloud architecture, multi-account strategies are the norm. We separate development from production, and often centralized shared services into their own hubs.

A very common scenario is having a central "Security" or "Shared Services" AWS account holding sensitive database credentials in AWS Secrets Manager, which need to be accessed by a Lambda function running in a completely different "Workload" account.

It sounds simple on paper:

Create the Secret in Account A.

Attach a resource policy to the secret allowing Account B to read it.

Give the Lambda in Account B IAM permission to read the secret.

You deploy your Terraform, invoke your Lambda, and... fail. You get an AccessDeniedException or a vague KMS error.

I recently ran into this exact wall. Here is why it happens, and the Terraform pattern to fix it.

The "Gotcha": It’s Rarely IAM, It’s Usually KMS

When debugging cross-account access failures with Secrets Manager (or S3, or SQS), 90% of the time developers focus on IAM policies. But when secrets are involved, you are fighting a two-front war: authentication (IAM) and cryptography (KMS).

By default, when you create a secret in Secrets Manager without specifying encryption settings, AWS encrypts it using the AWS-managed key for the service (alias/aws/secretsmanager).

Here is the trap: You cannot modify the Key Policy of an AWS-managed key. It is designed to only trust principals within the same account.

No matter how wide open you make your IAM policies, the external account will never be allowed to decrypt the payload. The door is unlocked, but the box is welded shut.

The Solution: Customer Managed Keys (CMK)

To enable cross-account access, you must take control of encryption. You need to create a Customer Managed Key (CMK) in KMS and explicitly tell it to trust the external account.

Here is the complete Terraform implementation to securely share secrets across accounts.

Prerequisites

We need the AWS Account ID of the "consumer" account (the one running the Lambda function that needs the secret). Let's define it as a variable:

variable "external_consumer_account_id" {
  description = "The AWS Account ID that needs read access to the secrets."
  type        = string
  # Example: "123456789012"
}
# Helper to get current account ID for policy definitions
data "aws_caller_identity" "current" {}

Step 1: The KMS Key (The Gatekeeper)

This is the most critical step. We create a symmetric KMS key. The Key Policy must allow two things:

The current account's root user to manage the key (otherwise you might lock yourself out).

The external account root user to perform cryptographic operations (Decrypt, Describe).

resource "aws_kms_key" "cross_account_secrets_key" {
  description             = "KMS Key for cross-account RDS credentials"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Enable IAM User Permissions for current account"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
        }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow External Account to Decrypt"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${var.external_consumer_account_id}:root"
        }
        # The external account needs to decrypt the secret payload
        Action = [
          "kms:Decrypt",
          "kms:DescribeKey"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_kms_alias" "secrets_key_alias" {
  name          = "alias/cross-account-secrets-key"
  target_key_id = aws_kms_key.cross_account_secrets_key.key_id
}

Step 2: The Secret (Using the Key)

Now we create the secret. The crucial part here is the kms_key_id argument. If you omit this, Terraform will happily use the default key, and cross-account access will break.

resource "aws_secretsmanager_secret" "database_credentials" {
  name        = "prod/rds/read-replica-creds"
  description = "Database credentials accessible by workload accounts"

  # CRITICAL: Force the use of our custom KMS key
  kms_key_id  = aws_kms_key.cross_account_secrets_key.id
}

(Note: I'm only showing the secret container creation here).

Step 3: The Secret Policy (Granting Access)

Finally, we need to tell Secrets Manager itself that the external account is allowed to ask for the value. We attach a resource policy to the secret.

resource "aws_secretsmanager_secret_policy" "database_credentials_policy" {
  secret_arn = aws_secretsmanager_secret.database_credentials.arn

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "AllowExternalRead"
        Effect    = "Allow"
        Principal = { 
          AWS = "arn:aws:iam::${var.external_consumer_account_id}:root" 
        }
        Action    = "secretsmanager:GetSecretValue"
        Resource  = "*"
      }
    ]
  })
}

Summary: The Full Picture

For a successful cross-account secret retrieval, three gates must open simultaneously. If any one of these fails, the request fails:

The KMS Key Policy (in the source account) must trust the destination account to Decrypt.

The Secret Resource Policy (in the source account) must trust the destination account to GetSecretValue.

The IAM Role attached to the Lambda/EC2 (in the destination account) must have permissions to perform both actions on the respective ARNs.

Using default AWS keys is the most common trap in multi-account architectures. By shifting to Customer Managed Keys and handling policy definitions explicitly in Terraform, you ensure secure, repeatable cross-account access.