DEV Community: ほうき星

Rethinking DynamoDB Encryption : From Attribute-Level Encryption to Beacons

ほうき星 — Thu, 27 Aug 2026 13:19:42 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

DynamoDB の暗号化を改めて考える：属性レベル暗号化から Beacons まで #AWS - Qiita

はじめにこんにちは、ほうき星 @H0ukiStar です。皆さんは DynamoDB を暗号化して利用していますでしょうか？ DynamoDB を暗号化なしで作成することはできませんので、この問いに「暗号化してなーい！」という返答は帰ってこないかと思います。 Dyna...

qiita.com

Introduction

Hello, I'm @H0ukiStar.

Are you encrypting your DynamoDB data? Since you cannot create a DynamoDB table without encryption, the answer is always "yes" — whether you realize it or not.

By default, DynamoDB encrypts data at rest using an AWS owned key on the server side. You can choose from three different KMS keys for this server-side encryption depending on your requirements.

Additionally, DynamoDB supports client-side encryption, which allows you to encrypt data at the attribute level before sending it to DynamoDB.

In this article, I revisit the various encryption approaches available for DynamoDB. In particular, I walk through client-side attribute-level encryption hands-on — from basic encryption to Beacons (Searchable Encryption).

Encryption Options Supported by DynamoDB

As mentioned above, DynamoDB supports three types of encryption: server-side encryption, client-side encryption, and encryption in transit.

The overall picture of DynamoDB encryption is shown below:

Server-Side Encryption

When creating a DynamoDB table, there is no option to disable encryption — server-side encryption at rest is always applied. DynamoDB transparently encrypts the table when it is persisted to disk and decrypts it when you access the data.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EncryptionAtRest.html

You can choose from three KMS keys for server-side encryption:

Key Type	Manager	Cost	Use Case
AWS owned key (default)	AWS	Free	No special requirements
AWS managed key (`aws/dynamodb`)	AWS (user can view)	KMS API call charges	Need to audit key usage via CloudTrail
Customer managed key (CMK)	User	KMS key charges + API charges	Need rotation control, key policy management, or cross-account access

The selection criteria can be summarized as follows:

No special key management requirements: AWS owned key (default)
Need to audit key usage via CloudTrail: AWS managed key
Need to manage key lifecycle and access policies yourself: Customer managed key

Client-Side Encryption

In addition to server-side encryption at rest, you can perform client-side encryption. With client-side encryption, the client application encrypts the data itself before sending it to DynamoDB, ensuring that unencrypted data is never exposed to any third party, including AWS.

https://docs.aws.amazon.com/database-encryption-sdk/latest/devguide/client-server-side.html

Client-side encryption is performed using either the AWS Database Encryption SDK for DynamoDB or the Amazon DynamoDB Encryption Client, and you can select encryption targets on a per-attribute basis.

Crypto Action Types

With attribute-level encryption using the AWS Database Encryption SDK for DynamoDB, you specify a "Crypto Action" for each attribute to determine how it should be encrypted and/or signed. There are four options:

Action	Encrypt	Sign	Use Case
`ENCRYPT_AND_SIGN`	Yes	Yes	Attributes requiring both confidentiality and integrity (e.g., email, address)
`SIGN_ONLY`	No	Yes	Attributes not requiring encryption but needing tamper detection (e.g., partition key, sort key)
`SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT`	No	Yes	Signed and included in the encryption context (e.g., identifiers for audit logs)
`DO_NOTHING`	No	No	Attributes requiring neither encryption nor signing (attributes you may freely add or change later)

⚠️ Warning: Partition keys and sort keys cannot be encrypted. DynamoDB uses these values to determine item placement and build indexes, so encrypting them would prevent server-side routing. These attributes can only be assigned SIGN_ONLY or SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT.

⚠️ Warning: SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT is an action added in the AWS Database Encryption SDK and is not available in the Amazon DynamoDB Encryption Client. For details on the differences between the two SDKs, see SDK History and Migration below.

Keyring (Key Provider) Options

The following keyrings are primarily available for client-side encryption:

AWS KMS Keyring: Leverages KMS key management capabilities with low operational overhead
Raw AES Keyring: For cases where you manage keys yourself without KMS
Raw RSA Keyring: For cases using asymmetric keys
AWS KMS Hierarchical Keyring: Improves performance through branch key caching (required when using Beacons)

This article demonstrates samples using the AWS KMS Keyring and the Hierarchical Keyring.

Supplement: Encryption in Transit

DynamoDB encrypts data in transit between client and server using HTTPS.

The encryption best practices recommend additional security measures for data in transit:

While DynamoDB encrypts data in transit by using HTTPS by default, additional security controls are recommended. You can use any of the following options:

AWS Site-to-Site VPN connection using IPsec for encryption.

AWS Direct Connect connection to establish a private connection.

AWS Direct Connect connection with AWS Site-to-Site VPN connection for an IPsec-encrypted private connection.

If access to DynamoDB is required only from within a virtual private cloud (VPC), you can use a VPC gateway endpoint and allow only resources in the VPC to access it. This prevents the traffic from traversing the public internet.

https://docs.aws.amazon.com/prescriptive-guidance/latest/encryption-best-practices/dynamodb.html

Hands-On

Server-Side Encryption

Server-side encryption at rest is determined by the "encryption key" you select when creating a DynamoDB table.

Since DynamoDB handles server-side encryption transparently, no application-level changes are required. Simply select a key when creating the table and you're done.

Additionally, you can change the encryption key after table creation without any downtime.

You select the KMS key for a table when you create or update the table. You can change the KMS key for a table at any time, either in the DynamoDB console or by using the UpdateTable operation. The process of switching keys is seamless and does not require downtime or degrade service.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.usagenotes.html#dynamodb-kms

Attribute-Level (Client-Side) Encryption

This is the main topic of this article. We perform attribute-level encryption using the AWS Database Encryption SDK for DynamoDB (Java) and the DynamoDB Encryption Client (Python).

DynamoDB Table Structure and CloudFormation Template

The DynamoDB tables, KMS key, and other resources used in the samples are defined in the following CloudFormation template.

The sample uses a table storing user information with the following attributes:

Attribute	Type	Crypto Action	Reason
`pk`	String (Partition Key)	`SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT`	Key attributes cannot be encrypted. Included in encryption context for auditability
`sk`	String (Sort Key)	`SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT`	Same as above
`email`	String	`ENCRYPT_AND_SIGN`	PII — encrypt
`name`	String	`ENCRYPT_AND_SIGN`	PII — encrypt
`age`	Number	`ENCRYPT_AND_SIGN`	PII — encrypt
`status`	String	`SIGN_ONLY`	Not sensitive, but tamper protection needed

sample.yaml

AWSTemplateFormatVersion: "2010-09-09"
Description: >-
  DynamoDB Client-Side Encryption sample resources.
  Creates a Users table (with GSIs for Standard/Compound Beacons),
  a KeyStore table for Hierarchical Keyring, and a KMS key.

Parameters:
  UsersTableName:
    Type: String
    Default: Users
    Description: Name of the DynamoDB table for the sample application.

  KeyStoreTableName:
    Type: String
    Default: KeyStore
    Description: Name of the DynamoDB table used as the KeyStore for Beacons (branch key management).

Resources:
  # ===========================================================================
  # KMS Key - Client-Side Encryption & KeyStore branch key wrapping
  # ===========================================================================
  EncryptionKey:
    Type: AWS::KMS::Key
    Properties:
      Description: KMS key for DynamoDB client-side encryption sample
      KeyUsage: ENCRYPT_DECRYPT
      KeySpec: SYMMETRIC_DEFAULT
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Sid: AllowRootAccountFullAccess
            Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
            Action: "kms:*"
            Resource: "*"

  EncryptionKeyAlias:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: alias/ddb-cse-sample
      TargetKeyId: !Ref EncryptionKey

  # ===========================================================================
  # Users Table - Sample application table
  # ===========================================================================
  UsersTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Ref UsersTableName
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: pk
          AttributeType: S
        - AttributeName: sk
          AttributeType: S
        # Standard Beacon: email
        - AttributeName: aws_dbe_b_email
          AttributeType: S
        # Compound Beacon: status + email
        - AttributeName: aws_dbe_b_status_email
          AttributeType: S
      KeySchema:
        - AttributeName: pk
          KeyType: HASH
        - AttributeName: sk
          KeyType: RANGE
      GlobalSecondaryIndexes:
        # GSI for Standard Beacon
        - IndexName: email-index
          KeySchema:
            - AttributeName: aws_dbe_b_email
              KeyType: HASH
          Projection:
            ProjectionType: ALL
        # GSI for Compound Beacon
        - IndexName: status-email-index
          KeySchema:
            - AttributeName: aws_dbe_b_status_email
              KeyType: HASH
          Projection:
            ProjectionType: ALL

  # ===========================================================================
  # KeyStore Table - Branch key management for Hierarchical Keyring / Beacons
  # ===========================================================================
  # The AWS Database Encryption SDK expects the KeyStore table to have
  # a specific schema:
  #   - Partition key: "branch-key-id" (String)
  #   - Sort key: "type" (String)
  KeyStoreTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Ref KeyStoreTableName
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: branch-key-id
          AttributeType: S
        - AttributeName: type
          AttributeType: S
      KeySchema:
        - AttributeName: branch-key-id
          KeyType: HASH
        - AttributeName: type
          KeyType: RANGE

Outputs:
  KmsKeyArn:
    Description: ARN of the KMS key for client-side encryption
    Value: !GetAtt EncryptionKey.Arn

  KmsKeyAlias:
    Description: Alias of the KMS key
    Value: !Ref EncryptionKeyAlias

  UsersTableName:
    Description: Name of the Users DynamoDB table
    Value: !Ref UsersTable

  UsersTableArn:
    Description: ARN of the Users DynamoDB table
    Value: !GetAtt UsersTable.Arn

  KeyStoreTableName:
    Description: Name of the KeyStore DynamoDB table
    Value: !Ref KeyStoreTable

  KeyStoreTableArn:
    Description: ARN of the KeyStore DynamoDB table
    Value: !GetAtt KeyStoreTable.Arn

This template creates the following resources:

Resource	Description
`EncryptionKey` (KMS)	Symmetric key used for client-side encryption and KeyStore branch key wrapping (automatic rotation enabled)
`UsersTable`	Main table for the sample application
`KeyStoreTable`	Branch key management table for Hierarchical Keyring, created with the schema required by the SDK (`branch-key-id` / `type`)

The Users table also has two GSIs for Standard Beacon / Compound Beacon searches:

GSI Name	Key Attribute	Purpose
`email-index`	`aws_dbe_b_email`	Equality search on email using Standard Beacon
`status-email-index`	`aws_dbe_b_status_email`	Compound condition search on status + email using Compound Beacon

Basic Put / Get Sample

AWS Database Encryption SDK (Java)

Here is a sample of attribute-level encryption using the AWS Database Encryption SDK (Java).

The author verified this with the following environment:

Amazon Corretto 21 (Java 21)
Dependency (Apache Maven 3.9):

  <dependency>
      <groupId>software.amazon.cryptography</groupId>
      <artifactId>aws-database-encryption-sdk-dynamodb</artifactId>
      <version>3.9.0</version>
  </dependency>

We use DynamoDbEncryptionInterceptor to configure an encryption client and perform item put and get operations.

⚠️ Warning: Replace arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id with your own KMS key ARN.

Sample Code : BasicPutGetExample.java

package com.example.ddbcse;

import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;
import software.amazon.cryptography.dbencryptionsdk.dynamodb.DynamoDbEncryptionInterceptor;
import software.amazon.cryptography.dbencryptionsdk.dynamodb.model.DynamoDbTableEncryptionConfig;
import software.amazon.cryptography.dbencryptionsdk.dynamodb.model.DynamoDbTablesEncryptionConfig;
import software.amazon.cryptography.dbencryptionsdk.structuredencryption.model.CryptoAction;
import software.amazon.cryptography.materialproviders.IKeyring;
import software.amazon.cryptography.materialproviders.MaterialProviders;
import software.amazon.cryptography.materialproviders.model.CreateAwsKmsMrkMultiKeyringInput;
import software.amazon.cryptography.materialproviders.model.MaterialProvidersConfig;

import java.util.HashMap;
import java.util.Map;

public class BasicPutGetExample {

    private static final String TABLE_NAME = "Users";
    private static final String KMS_KEY_ARN =
        "arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id";

    public static void main(String[] args) {
        // =======================================================
        // 1. Create the keyring
        // =======================================================
        final MaterialProviders matProv = MaterialProviders.builder()
                .MaterialProvidersConfig(MaterialProvidersConfig.builder().build())
                .build();
        final CreateAwsKmsMrkMultiKeyringInput keyringInput =
                CreateAwsKmsMrkMultiKeyringInput.builder()
                        .generator(KMS_KEY_ARN)
                        .build();
        final IKeyring kmsKeyring = matProv.CreateAwsKmsMrkMultiKeyring(keyringInput);

        // =======================================================
        // 2. Define crypto actions per attribute
        // =======================================================
        final Map<String, CryptoAction> attributeActions = new HashMap<>();
        attributeActions.put("pk", CryptoAction.SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT);
        attributeActions.put("sk", CryptoAction.SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT);
        attributeActions.put("email", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("name", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("age", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("status", CryptoAction.SIGN_ONLY);

        // =======================================================
        // 3. Table encryption configuration
        // =======================================================
        final DynamoDbTableEncryptionConfig tableConfig =
                DynamoDbTableEncryptionConfig.builder()
                        .logicalTableName(TABLE_NAME)
                        .partitionKeyName("pk")
                        .sortKeyName("sk")
                        .attributeActionsOnEncrypt(attributeActions)
                        .keyring(kmsKeyring)
                        .build();

        final Map<String, DynamoDbTableEncryptionConfig> tableConfigs = new HashMap<>();
        tableConfigs.put(TABLE_NAME, tableConfig);

        // =======================================================
        // 4. Create DynamoDbEncryptionInterceptor
        // =======================================================
        final DynamoDbEncryptionInterceptor interceptor =
                DynamoDbEncryptionInterceptor.builder()
                        .config(DynamoDbTablesEncryptionConfig.builder()
                                .tableEncryptionConfigs(tableConfigs)
                                .build())
                        .build();

        // =======================================================
        // 5. Create encryption-enabled DynamoDB client
        // =======================================================
        final DynamoDbClient ddb = DynamoDbClient.builder()
                .overrideConfiguration(
                        ClientOverrideConfiguration.builder()
                                .addExecutionInterceptor(interceptor)
                                .build())
                .build();

        // =======================================================
        // 6. Write an item
        // =======================================================
        final HashMap<String, AttributeValue> item = new HashMap<>();
        item.put("pk", AttributeValue.builder().s("USER#001").build());
        item.put("sk", AttributeValue.builder().s("PROFILE").build());
        item.put("email", AttributeValue.builder().s("taro@example.com").build());
        item.put("name", AttributeValue.builder().s("太郎").build());
        item.put("age", AttributeValue.builder().n("30").build());
        item.put("status", AttributeValue.builder().s("active").build());

        System.out.println("=== Put item (plaintext) ===");
        item.forEach((k, v) -> System.out.println("  " + k + " = " + v));

        ddb.putItem(PutItemRequest.builder()
                .tableName(TABLE_NAME)
                .item(item)
                .build());
        System.out.println("Put item completed.\n");

        // =======================================================
        // 7. Get item with plain client to verify encrypted state
        // =======================================================
        final Map<String, AttributeValue> key = new HashMap<>();
        key.put("pk", AttributeValue.builder().s("USER#001").build());
        key.put("sk", AttributeValue.builder().s("PROFILE").build());

        final DynamoDbClient plainClient = DynamoDbClient.builder().build();
        GetItemResponse rawResponse = plainClient.getItem(GetItemRequest.builder()
                .tableName(TABLE_NAME)
                .key(key)
                .build());
        System.out.println("=== Get item (raw / encrypted) ===");
        rawResponse.item().forEach((k, v) -> System.out.println("  " + k + " = " + v));
        System.out.println();

        // =======================================================
        // 8. Read item (automatic decryption)
        // =======================================================
        GetItemResponse response = ddb.getItem(GetItemRequest.builder()
                .tableName(TABLE_NAME)
                .key(key)
                .build());
        System.out.println("=== Get item (decrypted) ===");
        response.item().forEach((k, v) -> System.out.println("  " + k + " = " + v));
    }
}

Verifying the Encrypted Item

The sample above puts an item, retrieves it with a plain DynamoDB client (no Interceptor) to verify the encrypted state, and then retrieves it with the encryption client to show the decrypted result. Running the sample produces output like the following:

=== Put item (plaintext) ===
  sk = AttributeValue(S=PROFILE)
  name = AttributeValue(S=太郎)
  pk = AttributeValue(S=USER#001)
  email = AttributeValue(S=taro@example.com)
  age = AttributeValue(N=30)
  status = AttributeValue(S=active)
Put item completed.

=== Get item (raw / encrypted) ===
  status = AttributeValue(S=active)
  aws_dbe_foot = AttributeValue(B=SdkBytes(bytes=0x0376e53f...))
  aws_dbe_head = AttributeValue(B=SdkBytes(bytes=0x02010fce...))
  pk = AttributeValue(S=USER#001)
  email = AttributeValue(B=SdkBytes(bytes=0x0001c341...))
  name = AttributeValue(B=SdkBytes(bytes=0x0001737c...))
  age = AttributeValue(B=SdkBytes(bytes=0x0002c79e...))
  sk = AttributeValue(S=PROFILE)

=== Get item (decrypted) ===
  name = AttributeValue(S=太郎)
  sk = AttributeValue(S=PROFILE)
  pk = AttributeValue(S=USER#001)
  email = AttributeValue(S=taro@example.com)
  age = AttributeValue(N=30)
  status = AttributeValue(S=active)

Looking at the raw (encrypted) item, we can observe the following:

email, name, age are stored as binary (B type) — they are encrypted
pk, sk, status remain as plaintext (S type) — signed only, not encrypted
aws_dbe_head and aws_dbe_foot metadata attributes have been automatically added by the SDK

You can also verify this from the DynamoDB console — the "Explore table items" view shows the same encrypted state.

Amazon DynamoDB Encryption Client (Python)

Here is a sample of attribute-level encryption using the Amazon DynamoDB Encryption Client (Python).

The author verified this with the following environment:

Python 3.14
Dependencies:

  dynamodb-encryption-sdk==3.3.0
  boto3==1.43.73

We use the EncryptedTable class to create an encryption-enabled table resource and perform put and get operations. This class transparently handles encryption and decryption internally, so it can be used just like a regular boto3 Table.

⚠️ Warning: Replace arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id with your own KMS key ARN.

Sample Code : basic_put_get_example.py

"""DynamoDB Client-Side Encryption - Basic Put/Get Example (Python)

Performs attribute-level encryption and executes item put and get.
Uses AWS KMS to provide cryptographic materials.
"""

from typing import Any

import boto3
from dynamodb_encryption_sdk.encrypted.table import EncryptedTable
from dynamodb_encryption_sdk.identifiers import CryptoAction
from dynamodb_encryption_sdk.material_providers.aws_kms import (
    AwsKmsCryptographicMaterialsProvider,
)
from dynamodb_encryption_sdk.structures import AttributeActions

TABLE_NAME: str = "Users"
KMS_KEY_ARN: str = (
    "arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id"
)


def main() -> None:
    # =======================================================
    # 1. Create KMS CMP (Cryptographic Materials Provider)
    # =======================================================
    kms_cmp: AwsKmsCryptographicMaterialsProvider = (
        AwsKmsCryptographicMaterialsProvider(key_id=KMS_KEY_ARN)
    )

    # =======================================================
    # 2. Define crypto actions per attribute
    # =======================================================
    actions: AttributeActions = AttributeActions(
        default_action=CryptoAction.ENCRYPT_AND_SIGN,
        attribute_actions={
            # Partition key and sort key cannot be encrypted (EncryptedTable auto-sets SIGN_ONLY)
            "pk": CryptoAction.SIGN_ONLY,
            "sk": CryptoAction.SIGN_ONLY,
            "status": CryptoAction.DO_NOTHING,  # Neither signed nor encrypted
        },
    )

    # =======================================================
    # 3. Create regular DynamoDB table resource
    # =======================================================
    table = boto3.resource("dynamodb").Table(TABLE_NAME)

    # =======================================================
    # 4. Create encryption-enabled table
    # =======================================================
    encrypted_table: EncryptedTable = EncryptedTable(
        table=table,
        materials_provider=kms_cmp,
        attribute_actions=actions,
    )

    # =======================================================
    # 5. Write an item
    # =======================================================
    item: dict[str, Any] = {
        "pk": "USER#002",
        "sk": "PROFILE",
        "email": "jiro@example.com",
        "name": "次郎",
        "age": 26,
        "status": "active",
    }

    print("=== Put item (plaintext) ===")
    for k, v in item.items():
        print(f"  {k} = {v}")

    encrypted_table.put_item(Item=item)
    print("Put item completed.\n")

    # =======================================================
    # 6. Get with plain table resource to verify encrypted state
    # =======================================================
    key: dict[str, Any] = {"pk": "USER#002", "sk": "PROFILE"}

    raw_response: dict[str, Any] = table.get_item(Key=key)
    raw_item: dict[str, Any] = raw_response["Item"]
    print("=== Get item (raw / encrypted) ===")
    for k, v in raw_item.items():
        print(f"  {k} = {v!r}")
    print()

    # =======================================================
    # 7. Get with encrypted table resource (automatic decryption)
    # =======================================================
    decrypted_response: dict[str, Any] = encrypted_table.get_item(Key=key)
    decrypted_item: dict[str, Any] = decrypted_response["Item"]
    print("=== Get item (decrypted) ===")
    for k, v in decrypted_item.items():
        print(f"  {k} = {v!r}")


if __name__ == "__main__":
    main()

Verifying the Encrypted Item

Similar to the AWS Database Encryption SDK (Java) version, we retrieve the item with a plain table resource (no encryption) to verify the encrypted state, then retrieve it with EncryptedTable to show the decrypted result. Running the sample produces output like the following:

=== Put item (plaintext) ===
  pk = USER#002
  sk = PROFILE
  email = jiro@example.com
  name = 次郎
  age = 26
  status = active
Put item completed.

=== Get item (raw / encrypted) ===
  *amzn-ddb-map-desc* = Binary(b'\x00\x00\x00\x00...')
  status = 'active'
  *amzn-ddb-map-sig* = Binary(b'\xa1\xef\x02]...')
  pk = 'USER#002'
  email = Binary(b'\x8e\x02\xd8V...')
  name = Binary(b'\x04\x8ceI...')
  age = Binary(b'\xec_x\xe7...')
  sk = 'PROFILE'

=== Get item (decrypted) ===
  status = 'active'
  pk = 'USER#002'
  email = 'jiro@example.com'
  name = '次郎'
  age = Decimal('26')
  sk = 'PROFILE'

Looking at the raw (encrypted) item, we can observe the following:

email, name, age are stored as binary (Binary type) — they are encrypted
pk, sk, status remain as plaintext
*amzn-ddb-map-desc* and *amzn-ddb-map-sig* metadata attributes have been automatically added by the SDK

Cross-SDK Decryption Is Not Possible

The AWS Database Encryption SDK and the DynamoDB Encryption Client use different encryption formats with different metadata attribute names (aws_dbe_head / aws_dbe_foot vs. *amzn-ddb-map-desc* / *amzn-ddb-map-sig*).

Therefore, items encrypted by one SDK cannot be decrypted by the other. For example, attempting to decrypt an item encrypted by the DynamoDB Encryption Client (Python) using the AWS Database Encryption SDK (Java) results in the following error:

[WARNING]
software.amazon.awssdk.core.exception.SdkClientException: Unable to unmarshall response (Encrypted item missing expected header and footer attributes). Response Code: 200, Response Text: OK (SDK Attempt Count: 1)
...
Caused by: software.amazon.cryptography.dbencryptionsdk.dynamodb.itemencryptor.model.DynamoDbItemEncryptorException: Encrypted item missing expected header and footer attributes

ℹ️ Note: Cross-language decryption within the same SDK family is supported.

AWS Database Encryption SDK:

The AWS Database Encryption SDK for DynamoDB is available in multiple programming languages. The language implementations are designed to be fully interoperable and to offer the same features, although they might be implemented in different ways. Typically, you use the library that is compatible with your application.

https://docs.aws.amazon.com/database-encryption-sdk/latest/devguide/configure.html

Amazon DynamoDB Encryption Client:

The Amazon DynamoDB Encryption Client is available for the following programming languages. The language-specific libraries vary, but the resulting implementations are interoperable. For example, you can encrypt (and sign) an item with the Java client and decrypt the item with the Python client.

https://docs.aws.amazon.com/database-encryption-sdk/latest/devguide/programming-languages.html

For details on the differences and history of the two SDKs, see SDK History and Migration below.

Querying Encrypted Attributes

When attribute-level encryption is introduced, there are constraints on querying encrypted attributes.

Constraints on Filter and Condition Expressions for Encrypted Attributes

Since encrypted attributes cannot be decrypted on the DynamoDB server side, the following operations do not work:

FilterExpression: Even if you filter with a condition like email = :val, the server compares against the encrypted binary value, so it won't match
ConditionExpression: For the same reason, conditional writes don't work either
GSI / LSI key attributes: For the same reason, specifying an encrypted attribute as an index key produces no meaningful query results

Example: Filtering on an encrypted attribute with plaintext (no match)

System.out.println("=== Scan with FilterExpression: email = 'taro@example.com' (String) ===");
ScanResponse response1 = ddb.scan(ScanRequest.builder()
        .tableName(TABLE_NAME)
        .filterExpression("email = :val")
        .expressionAttributeValues(Map.of(
                ":val", AttributeValue.builder().s("taro@example.com").build()))
        .build());
System.out.println("  Items found: " + response1.count());

=== Scan with FilterExpression: email = 'taro@example.com' (String) ===
  Items found: 0

If you know the exact encrypted binary, a filter can match. However, you cannot derive the encrypted binary at FilterExpression construction time for the following reasons, making search effectively impossible:

In AES-GCM, a different nonce (IV) is generally used for each encryption operation, so even encrypting the same plaintext with the same key produces a different ciphertext each time
Therefore, it is impossible to pre-compute the ciphertext stored in the table from a plaintext search value

Example: Filtering with the known encrypted binary (matches)

byte[] encryptedEmailBytes = Base64.getDecoder().decode(
        "AAHDQc/juSMNHkMPz+zNnOIBZKen9tEsHf0X7VaL7CFHKQ==");
ScanResponse response2 = ddb.scan(ScanRequest.builder()
        .tableName(TABLE_NAME)
        .filterExpression("email = :val")
        .expressionAttributeValues(Map.of(
                ":val", AttributeValue.builder()
                        .b(SdkBytes.fromByteArray(encryptedEmailBytes))
                        .build()))
        .build());
System.out.println("  Items found: " + response2.count());

  Items found: 1

So how can you search on encrypted attributes? This is where Searchable Encryption (Beacons) comes in.

Beacons (Searchable Encryption)

The AWS Database Encryption SDK provides Beacons as a mechanism to enable querying on encrypted attributes.

⚠️ Warning: The Amazon DynamoDB Encryption Client does not support Beacons.

How Beacons Work

Beacons compute an HMAC from the plaintext value of an encrypted attribute and store a portion (truncated HMAC) as a "beacon" in a separate attribute. This beacon value cannot be used to recover the original plaintext, but since the same plaintext always produces the same beacon value, it can be used for equality queries.

Since a hash value is used rather than the plaintext itself, there are the following characteristics (caveats):

Because beacon values are truncated hashes, different plaintexts may produce the same beacon value (collisions)
Shorter beacon lengths increase collisions but make it harder to guess the original value (better security)
Longer beacon lengths reduce collisions and improve query precision but increase the risk of statistical inference

Beacon Types

Type	Purpose
Standard Beacon	Equality search on a single attribute
Compound Beacon	Search combining multiple attributes

Standard Beacon Sample

The following example configures a Standard Beacon for the email attribute, enabling email-based searches even in an encrypted state.

Using Beacons requires a Hierarchical Keyring, which in turn requires a pre-created KeyStore (branch key management table).

⚠️ Warning: Replace arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id with your own KMS key ARN.

Sample Code : StandardBeaconExample.java

package com.example.ddbcse;

import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;
import software.amazon.awssdk.services.kms.KmsClient;
import software.amazon.cryptography.dbencryptionsdk.dynamodb.DynamoDbEncryptionInterceptor;
import software.amazon.cryptography.dbencryptionsdk.dynamodb.model.*;
import software.amazon.cryptography.dbencryptionsdk.structuredencryption.model.CryptoAction;
import software.amazon.cryptography.keystore.KeyStore;
import software.amazon.cryptography.keystore.model.CreateKeyInput;
import software.amazon.cryptography.keystore.model.CreateKeyOutput;
import software.amazon.cryptography.keystore.model.KeyStoreConfig;
import software.amazon.cryptography.keystore.model.KMSConfiguration;
import software.amazon.cryptography.materialproviders.IKeyring;
import software.amazon.cryptography.materialproviders.MaterialProviders;
import software.amazon.cryptography.materialproviders.model.CreateAwsKmsHierarchicalKeyringInput;
import software.amazon.cryptography.materialproviders.model.MaterialProvidersConfig;

import java.util.*;

public class StandardBeaconExample {

    private static final String TABLE_NAME = "Users";
    private static final String KMS_KEY_ARN =
            "arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id";
    private static final String KEYSTORE_TABLE_NAME = "KeyStore";

    public static void main(String[] args) {
        // =======================================================
        // 1. Create KeyStore client
        // =======================================================
        final KeyStore keyStore = KeyStore.builder()
                .KeyStoreConfig(KeyStoreConfig.builder()
                        .ddbTableName(KEYSTORE_TABLE_NAME)
                        .kmsConfiguration(KMSConfiguration.builder()
                                .kmsKeyArn(KMS_KEY_ARN)
                                .build())
                        .logicalKeyStoreName(KEYSTORE_TABLE_NAME)
                        .ddbClient(DynamoDbClient.builder().build())
                        .kmsClient(KmsClient.builder().build())
                        .build())
                .build();

        // =======================================================
        // 2. Create branch key
        //    (If already created, you can specify the existing branchKeyId directly)
        // =======================================================
        System.out.println("Creating branch key...");
        final CreateKeyOutput branchKeyOutput = keyStore.CreateKey(
                CreateKeyInput.builder().build());
        final String branchKeyId = branchKeyOutput.branchKeyIdentifier();
        System.out.println("Branch key created: " + branchKeyId);

        // =======================================================
        // 3. Create Hierarchical Keyring
        // =======================================================
        final MaterialProviders matProv = MaterialProviders.builder()
                .MaterialProvidersConfig(MaterialProvidersConfig.builder().build())
                .build();

        final IKeyring hierarchicalKeyring =
                matProv.CreateAwsKmsHierarchicalKeyring(
                        CreateAwsKmsHierarchicalKeyringInput.builder()
                                .keyStore(keyStore)
                                .branchKeyId(branchKeyId)
                                .ttlSeconds(600L)
                                .build());

        // =======================================================
        // 4. Define Standard Beacon
        // =======================================================
        List<StandardBeacon> standardBeacons = new ArrayList<>();
        standardBeacons.add(StandardBeacon.builder()
                .name("email")
                .length(30)  // Beacon length (in bits)
                .build());

        // =======================================================
        // 5. Configure Beacon Version
        // =======================================================
        List<BeaconVersion> beaconVersions = new ArrayList<>();
        beaconVersions.add(BeaconVersion.builder()
                .version(1)
                .keyStore(keyStore)
                .keySource(BeaconKeySource.builder()
                        .single(SingleKeyStore.builder()
                                .keyId(branchKeyId)
                                .cacheTTL(600)
                                .build())
                        .build())
                .standardBeacons(standardBeacons)
                .build());

        // =======================================================
        // 6. Search configuration (SearchConfig)
        // =======================================================
        SearchConfig searchConfig = SearchConfig.builder()
                .versions(beaconVersions)
                .writeVersion(1)
                .build();

        // =======================================================
        // 7. Define crypto actions per attribute
        // =======================================================
        final Map<String, CryptoAction> attributeActions = new HashMap<>();
        attributeActions.put("pk", CryptoAction.SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT);
        attributeActions.put("sk", CryptoAction.SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT);
        attributeActions.put("email", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("name", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("age", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("status", CryptoAction.SIGN_ONLY);

        // =======================================================
        // 8. Table encryption config (Beacons enabled)
        // =======================================================
        final DynamoDbTableEncryptionConfig tableConfig =
                DynamoDbTableEncryptionConfig.builder()
                        .logicalTableName(TABLE_NAME)
                        .partitionKeyName("pk")
                        .sortKeyName("sk")
                        .attributeActionsOnEncrypt(attributeActions)
                        .keyring(hierarchicalKeyring)
                        .search(searchConfig)
                        .build();

        final Map<String, DynamoDbTableEncryptionConfig> tableConfigs = new HashMap<>();
        tableConfigs.put(TABLE_NAME, tableConfig);

        // =======================================================
        // 9. Create DynamoDbEncryptionInterceptor
        // =======================================================
        final DynamoDbEncryptionInterceptor interceptor =
                DynamoDbEncryptionInterceptor.builder()
                        .config(DynamoDbTablesEncryptionConfig.builder()
                                .tableEncryptionConfigs(tableConfigs)
                                .build())
                        .build();

        // =======================================================
        // 10. Create encryption-enabled DynamoDB client
        // =======================================================
        final DynamoDbClient ddb = DynamoDbClient.builder()
                .overrideConfiguration(
                        ClientOverrideConfiguration.builder()
                                .addExecutionInterceptor(interceptor)
                                .build())
                .build();

        // =======================================================
        // 11. Write an item
        // =======================================================
        final HashMap<String, AttributeValue> item = new HashMap<>();
        item.put("pk", AttributeValue.builder().s("USER#100").build());
        item.put("sk", AttributeValue.builder().s("PROFILE").build());
        item.put("email", AttributeValue.builder().s("hanako@example.com").build());
        item.put("name", AttributeValue.builder().s("花子").build());
        item.put("age", AttributeValue.builder().n("25").build());
        item.put("status", AttributeValue.builder().s("active").build());

        System.out.println("=== Put item (plaintext) ===");
        item.forEach((k, v) -> System.out.println("  " + k + " = " + v));

        ddb.putItem(PutItemRequest.builder()
                .tableName(TABLE_NAME)
                .item(item)
                .build());
        System.out.println("Put item with beacon completed.\n");

        // =======================================================
        // 12. Raw get with plain client (verify beacon attribute)
        // =======================================================
        final DynamoDbClient plainClient = DynamoDbClient.builder().build();
        final Map<String, AttributeValue> key = new HashMap<>();
        key.put("pk", AttributeValue.builder().s("USER#100").build());
        key.put("sk", AttributeValue.builder().s("PROFILE").build());

        GetItemResponse rawResponse = plainClient.getItem(GetItemRequest.builder()
                .tableName(TABLE_NAME)
                .key(key)
                .build());
        System.out.println("=== Get item (raw) - beacon attribute visible ===");
        rawResponse.item().forEach((k, v) -> System.out.println("  " + k + " = " + v));
        System.out.println();

        // =======================================================
        // 13. Query using beacon (GSI: email-index)
        // =======================================================
        QueryResponse queryResponse = ddb.query(QueryRequest.builder()
                .tableName(TABLE_NAME)
                .indexName("email-index")
                .keyConditionExpression("aws_dbe_b_email = :emailVal")
                .expressionAttributeValues(Map.of(
                        ":emailVal",
                        AttributeValue.builder().s("hanako@example.com").build()))
                .build());
        System.out.println("=== Query via beacon (email-index) ===");
        System.out.println("  Items found: " + queryResponse.count());
        queryResponse.items().forEach(i -> {
            System.out.println("  ---");
            i.forEach((k, v) -> System.out.println("    " + k + " = " + v));
        });
    }
}

Running the sample above produces output like the following:

Creating branch key...
Branch key created: 7b700228-d086-4446-bf71-bb4e95e95907
=== Put item (plaintext) ===
  ...
Put item with beacon completed.

=== Get item (raw) - beacon attribute visible ===
  status = AttributeValue(S=active)
  aws_dbe_foot = AttributeValue(B=SdkBytes(bytes=...))
  aws_dbe_head = AttributeValue(B=SdkBytes(bytes=...))
  pk = AttributeValue(S=USER#100)
  email = AttributeValue(B=SdkBytes(bytes=...))
  name = AttributeValue(B=SdkBytes(bytes=...))
  aws_dbe_b_email = AttributeValue(S=176be958)
  age = AttributeValue(B=SdkBytes(bytes=...))
  sk = AttributeValue(S=PROFILE)

=== Query via beacon (email-index) ===
  Items found: 1
  ---
    sk = AttributeValue(S=PROFILE)
    name = AttributeValue(S=花子)
    pk = AttributeValue(S=USER#100)
    age = AttributeValue(N=25)
    email = AttributeValue(S=hanako@example.com)
    status = AttributeValue(S=active)

Looking at the raw get result, in addition to the attributes we saw in the basic encryption sample (aws_dbe_head, aws_dbe_foot, encrypted binary attributes), there is a new attribute aws_dbe_b_email. This is the beacon attribute automatically computed and stored by the SDK. Its value 176be958 is a short string — the truncated HMAC of hanako@example.com.

The GSI email-index uses aws_dbe_b_email as its partition key. At query time, the SDK computes the beacon value from the plaintext hanako@example.com and searches the GSI. As a result, the encrypted item is found, and the SDK automatically decrypts it, returning the plaintext values.

Important Considerations When Using Beacons

Key considerations when making data searchable with Beacons:

KeyStore table must be created in advance: A separate DynamoDB table for managing beacon keys must be created, along with a branch key
Hierarchical Keyring is required: Beacons require the use of a Hierarchical Keyring
Beacon length tradeoff: When determining beacon length, consider the expected record count and security requirements > AWS documentation provides recommended values based on dataset size: > https://docs.aws.amazon.com/database-encryption-sdk/latest/devguide/choosing-beacon-length.html#beacon-length-example
Equality matches only: Standard Beacons only support equality (=) queries
Existing data not searchable: Beacons compute beacon values at write time, so enabling beacons does not make existing data searchable (re-writing is required)
Not supported with DynamoDB Enhanced Client: Searchable Encryption can only be used with DynamoDbEncryptionInterceptor + the low-level DynamoDB API

Compound Beacon Sample

When you need searches combining multiple attributes, use Compound Beacons. For example, combining status (SIGN_ONLY) and email (ENCRYPT_AND_SIGN) to search for "status is active AND email is hanako@example.com".

A Compound Beacon generates a single beacon value by concatenating each part's value with a split character, enabling compound condition equality searches with a single GSI query.

⚠️ Warning: Replace arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id with your own KMS key ARN.

Sample Code : CompoundBeaconExample.java

package com.example.ddbcse;

import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;
import software.amazon.awssdk.services.kms.KmsClient;
import software.amazon.cryptography.dbencryptionsdk.dynamodb.DynamoDbEncryptionInterceptor;
import software.amazon.cryptography.dbencryptionsdk.dynamodb.model.*;
import software.amazon.cryptography.dbencryptionsdk.structuredencryption.model.CryptoAction;
import software.amazon.cryptography.keystore.KeyStore;
import software.amazon.cryptography.keystore.model.CreateKeyInput;
import software.amazon.cryptography.keystore.model.CreateKeyOutput;
import software.amazon.cryptography.keystore.model.KeyStoreConfig;
import software.amazon.cryptography.keystore.model.KMSConfiguration;
import software.amazon.cryptography.materialproviders.IKeyring;
import software.amazon.cryptography.materialproviders.MaterialProviders;
import software.amazon.cryptography.materialproviders.model.CreateAwsKmsHierarchicalKeyringInput;
import software.amazon.cryptography.materialproviders.model.MaterialProvidersConfig;

import java.util.*;

/**
 * Compound Beacon sample.
 * Uses a Compound Beacon combining status (SIGN_ONLY) and email (ENCRYPT_AND_SIGN)
 * to perform compound condition searches like "status = active AND email = hanako@example.com".
 *
 * Table: Users (deployed via sample.yaml)
 * GSI: status-email-index (aws_dbe_b_status_email)
 */
public class CompoundBeaconExample {

    private static final String TABLE_NAME = "Users";
    private static final String KMS_KEY_ARN =
            "arn:aws:kms:ap-northeast-1:123456789012:key/your-key-id";
    private static final String KEYSTORE_TABLE_NAME = "KeyStore";

    public static void main(String[] args) {
        // =======================================================
        // 1. Create KeyStore client
        // =======================================================
        final KeyStore keyStore = KeyStore.builder()
                .KeyStoreConfig(KeyStoreConfig.builder()
                        .ddbTableName(KEYSTORE_TABLE_NAME)
                        .kmsConfiguration(KMSConfiguration.builder()
                                .kmsKeyArn(KMS_KEY_ARN)
                                .build())
                        .logicalKeyStoreName(KEYSTORE_TABLE_NAME)
                        .ddbClient(DynamoDbClient.builder().build())
                        .kmsClient(KmsClient.builder().build())
                        .build())
                .build();

        // =======================================================
        // 2. Create branch key
        // =======================================================
        System.out.println("Creating branch key...");
        final CreateKeyOutput branchKeyOutput = keyStore.CreateKey(
                CreateKeyInput.builder().build());
        final String branchKeyId = branchKeyOutput.branchKeyIdentifier();
        System.out.println("Branch key created: " + branchKeyId);

        // =======================================================
        // 3. Create Hierarchical Keyring
        // =======================================================
        final MaterialProviders matProv = MaterialProviders.builder()
                .MaterialProvidersConfig(MaterialProvidersConfig.builder().build())
                .build();

        final IKeyring hierarchicalKeyring =
                matProv.CreateAwsKmsHierarchicalKeyring(
                        CreateAwsKmsHierarchicalKeyringInput.builder()
                                .keyStore(keyStore)
                                .branchKeyId(branchKeyId)
                                .ttlSeconds(600L)
                                .build());

        // =======================================================
        // 4. Define Standard Beacon (required for Compound Beacon's encrypted part)
        // =======================================================
        List<StandardBeacon> standardBeacons = new ArrayList<>();
        standardBeacons.add(StandardBeacon.builder()
                .name("email")
                .length(30)
                .build());

        // =======================================================
        // 5. Define Compound Beacon
        //    Combining status (SIGN_ONLY) and email (ENCRYPT_AND_SIGN)
        // =======================================================
        List<CompoundBeacon> compoundBeacons = new ArrayList<>();
        compoundBeacons.add(CompoundBeacon.builder()
                .name("status_email")
                .split("#")  // Split character (must not appear in any part's plaintext)
                .encrypted(List.of(
                        EncryptedPart.builder()
                                .name("email")
                                .prefix("E-")
                                .build()))
                .signed(List.of(
                        SignedPart.builder()
                                .name("status")
                                .prefix("S-")
                                .build()))
                .build());

        // =======================================================
        // 6. Configure Beacon Version
        // =======================================================
        List<BeaconVersion> beaconVersions = new ArrayList<>();
        beaconVersions.add(BeaconVersion.builder()
                .version(1)
                .keyStore(keyStore)
                .keySource(BeaconKeySource.builder()
                        .single(SingleKeyStore.builder()
                                .keyId(branchKeyId)
                                .cacheTTL(600)
                                .build())
                        .build())
                .standardBeacons(standardBeacons)
                .compoundBeacons(compoundBeacons)
                .build());

        // =======================================================
        // 7. Search configuration (SearchConfig)
        // =======================================================
        SearchConfig searchConfig = SearchConfig.builder()
                .versions(beaconVersions)
                .writeVersion(1)
                .build();

        // =======================================================
        // 8. Define crypto actions per attribute
        // =======================================================
        final Map<String, CryptoAction> attributeActions = new HashMap<>();
        attributeActions.put("pk", CryptoAction.SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT);
        attributeActions.put("sk", CryptoAction.SIGN_AND_INCLUDE_IN_ENCRYPTION_CONTEXT);
        attributeActions.put("email", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("name", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("age", CryptoAction.ENCRYPT_AND_SIGN);
        attributeActions.put("status", CryptoAction.SIGN_ONLY);

        // =======================================================
        // 9. Table encryption config (Beacons enabled)
        // =======================================================
        final DynamoDbTableEncryptionConfig tableConfig =
                DynamoDbTableEncryptionConfig.builder()
                        .logicalTableName(TABLE_NAME)
                        .partitionKeyName("pk")
                        .sortKeyName("sk")
                        .attributeActionsOnEncrypt(attributeActions)
                        .keyring(hierarchicalKeyring)
                        .search(searchConfig)
                        .build();

        final Map<String, DynamoDbTableEncryptionConfig> tableConfigs = new HashMap<>();
        tableConfigs.put(TABLE_NAME, tableConfig);

        // =======================================================
        // 10. Create DynamoDbEncryptionInterceptor
        // =======================================================
        final DynamoDbEncryptionInterceptor interceptor =
                DynamoDbEncryptionInterceptor.builder()
                        .config(DynamoDbTablesEncryptionConfig.builder()
                                .tableEncryptionConfigs(tableConfigs)
                                .build())
                        .build();

        // =======================================================
        // 11. Create encryption-enabled DynamoDB client
        // =======================================================
        final DynamoDbClient ddb = DynamoDbClient.builder()
                .overrideConfiguration(
                        ClientOverrideConfiguration.builder()
                                .addExecutionInterceptor(interceptor)
                                .build())
                .build();

        // =======================================================
        // 12. Write multiple items
        // =======================================================
        List<Map<String, AttributeValue>> items = List.of(
                Map.of(
                        "pk", AttributeValue.builder().s("USER#201").build(),
                        "sk", AttributeValue.builder().s("PROFILE").build(),
                        "email", AttributeValue.builder().s("hanako@example.com").build(),
                        "name", AttributeValue.builder().s("花子").build(),
                        "age", AttributeValue.builder().n("25").build(),
                        "status", AttributeValue.builder().s("active").build()),
                Map.of(
                        "pk", AttributeValue.builder().s("USER#202").build(),
                        "sk", AttributeValue.builder().s("PROFILE").build(),
                        "email", AttributeValue.builder().s("taro@example.com").build(),
                        "name", AttributeValue.builder().s("太郎").build(),
                        "age", AttributeValue.builder().n("30").build(),
                        "status", AttributeValue.builder().s("active").build()),
                Map.of(
                        "pk", AttributeValue.builder().s("USER#203").build(),
                        "sk", AttributeValue.builder().s("PROFILE").build(),
                        "email", AttributeValue.builder().s("hanako@example.com").build(),
                        "name", AttributeValue.builder().s("花子(退会済み)").build(),
                        "age", AttributeValue.builder().n("25").build(),
                        "status", AttributeValue.builder().s("inactive").build()));

        for (Map<String, AttributeValue> item : items) {
            ddb.putItem(PutItemRequest.builder()
                    .tableName(TABLE_NAME)
                    .item(item)
                    .build());
            System.out.println("Put: " + item.get("pk").s() + " / status=" + item.get("status").s()
                    + " / email=" + item.get("email").s());
        }
        System.out.println();

        // =======================================================
        // 13. Raw get with plain client (verify Compound Beacon attribute)
        // =======================================================
        final DynamoDbClient plainClient = DynamoDbClient.builder().build();
        final Map<String, AttributeValue> key = Map.of(
                "pk", AttributeValue.builder().s("USER#201").build(),
                "sk", AttributeValue.builder().s("PROFILE").build());

        GetItemResponse rawResponse = plainClient.getItem(GetItemRequest.builder()
                .tableName(TABLE_NAME)
                .key(key)
                .build());
        System.out.println("=== Get item (raw) - beacon attributes visible ===");
        rawResponse.item().forEach((k, v) -> System.out.println("  " + k + " = " + v));
        System.out.println();

        // =======================================================
        // 14. Query using Compound Beacon
        //     status = "active" AND email = "hanako@example.com"
        // =======================================================
        System.out.println("=== Query via compound beacon (status-email-index) ===");
        System.out.println("  Condition: status=active AND email=hanako@example.com");
        QueryResponse queryResponse = ddb.query(QueryRequest.builder()
                .tableName(TABLE_NAME)
                .indexName("status-email-index")
                .keyConditionExpression("aws_dbe_b_status_email = :val")
                .expressionAttributeValues(Map.of(
                        ":val",
                        AttributeValue.builder().s("S-active#E-hanako@example.com").build()))
                .build());
        System.out.println("  Items found: " + queryResponse.count());
        queryResponse.items().forEach(i -> {
            System.out.println("  ---");
            i.forEach((k, v) -> System.out.println("    " + k + " = " + v));
        });
    }
}

The sample writes three items:

Item	status	email
USER#201	active	hanako@example.com
USER#202	active	taro@example.com
USER#203	inactive	hanako@example.com

Running the sample produces output like the following:

Creating branch key...
Branch key created: 781aefa0-5d25-440f-bbe7-106fa81bee14
Put: USER#201 / status=active / email=hanako@example.com
Put: USER#202 / status=active / email=taro@example.com
Put: USER#203 / status=inactive / email=hanako@example.com

=== Get item (raw) - beacon attributes visible ===
  aws_dbe_b_status_email = AttributeValue(S=S-active#E-3d042b97)
  status = AttributeValue(S=active)
  aws_dbe_foot = AttributeValue(B=SdkBytes(bytes=...))
  aws_dbe_head = AttributeValue(B=SdkBytes(bytes=...))
  pk = AttributeValue(S=USER#201)
  email = AttributeValue(B=SdkBytes(bytes=...))
  name = AttributeValue(B=SdkBytes(bytes=...))
  aws_dbe_b_email = AttributeValue(S=3d042b97)
  age = AttributeValue(B=SdkBytes(bytes=...))
  sk = AttributeValue(S=PROFILE)

=== Query via compound beacon (status-email-index) ===
  Condition: status=active AND email=hanako@example.com
  Items found: 1
  ---
    name = AttributeValue(S=花子)
    sk = AttributeValue(S=PROFILE)
    pk = AttributeValue(S=USER#201)
    email = AttributeValue(S=hanako@example.com)
    age = AttributeValue(N=25)
    status = AttributeValue(S=active)

From the raw get result, we can see the Standard Beacon (aws_dbe_b_email = 3d042b97) and the Compound Beacon (aws_dbe_b_status_email = S-active#E-3d042b97). The Compound Beacon value is a concatenation of S-active (status plaintext with prefix S-) and E-3d042b97 (email beacon value with prefix E-), joined by the split character #.

The query result returns only USER#201, which matches both status = "active" and email = "hanako@example.com". The SDK automatically converts the search value S-active#E-hanako@example.com to S-active#E-3d042b97 and executes a single query against the GSI.

Note that the split character must be a character that does not appear in any part's plaintext value. In this example, # is used, but a different character should be chosen if # could appear in the plaintext.

Key Considerations

Here is a summary of important points to be aware of when introducing client-side encryption.

Using Filter/Condition Expressions on Encrypted Attributes

FilterExpression and ConditionExpression do not work on encrypted attributes. If you need to search on encrypted attributes, consider using Beacons.

Signature Verification Errors

Adding, deleting, or modifying attributes in an item will cause the signature to no longer match, resulting in an error on retrieval (tamper detection). Care must be taken when changing the attribute structure of signed items.

For example, deleting or modifying a signed attribute via the management console produces the following errors:

When a signed attribute is deleted:

StructuredEncryptionException: Schema changed : something that was signed is now unsigned.

When a signed attribute value is modified:

StructuredEncryptionException: Signature of record does not match the signature computed when the record was encrypted.

Increased Item Size

Encryption increases item size for the following reasons:

Encrypted values are stored as binary type and become larger than the original due to padding
Metadata attributes (aws_dbe_head, aws_dbe_foot) are added
When using Beacons, beacon attributes are also added

Therefore, design with margin against DynamoDB's item size limit (400KB).

Data Not Visible in Console or CLI

Encrypted attributes are stored as binary values, so their contents cannot be viewed in the management console or AWS CLI.

Difficulty Adding/Changing Beacons After the Fact

When using Beacons, it is extremely important to determine search patterns (which attributes to search on) in advance. While DynamoDB always requires upfront access pattern planning, Beacons make this even more critical.

If you want to add a new beacon later, the following steps are required:

Add the new beacon definition and create a corresponding GSI
Re-write all existing data (existing items don't have the new beacon attribute and won't be searchable)

Additionally, if you rotate the branch key, the HMAC key used for beacon value derivation changes, requiring all items' beacon values to be recalculated (re-written).

With regular DynamoDB, simply adding a GSI automatically indexes existing data, but this is not the case with Beacons.

PartiQL (ExecuteStatement) Is Not Supported

The DynamoDbEncryptionInterceptor in the AWS Database Encryption SDK does not support PartiQL (ExecuteStatement / BatchExecuteStatement). If your existing application accesses DynamoDB using PartiQL, you need to refactor those calls to use the standard DynamoDB API (PutItem / GetItem / Query, etc.) before introducing attribute-level encryption.

SDK History and Migration

The DynamoDB client-side encryption library has undergone name changes and generational transitions to reach its current form.

Timeline

Date	Event
May 2018	Amazon DynamoDB Encryption Client released for Java and Python
July 2022	DynamoDB Encryption Client Java v1.x, Python v1.x/v2.x entered end-of-support phase
June 2023	Library renamed to AWS Database Encryption SDK. Released as a new-generation SDK with a new encryption format and Searchable Encryption (Beacons) for Java
Present	AWS Database Encryption SDK available for Java, .NET, Rust. Python continues with the legacy DynamoDB Encryption Client v3.x

https://docs.aws.amazon.com/database-encryption-sdk/latest/devguide/document-history.html

Two SDK Families

As described above, two SDK families currently coexist for DynamoDB client-side encryption:

	AWS Database Encryption SDK (New Generation)	Amazon DynamoDB Encryption Client (Legacy)
Supported Languages	Java, .NET, Rust	Java, Python
Metadata Attributes	`aws_dbe_head` / `aws_dbe_foot`	`amzn-ddb-map-desc` / `amzn-ddb-map-sig`
Searchable Encryption	Supported (Beacons)	Not supported
Interoperability	Cross-language decryption within the same SDK family	Cross-language decryption within the same SDK family

Migration

The two SDK families cannot encrypt/decrypt each other's data. Therefore, migrating from the legacy DynamoDB Encryption Client to the AWS Database Encryption SDK follows a phased approach (handle both formats on read → switch writes to the new format).

https://docs.aws.amazon.com/database-encryption-sdk/latest/devguide/ddb-java-migrate.html#ddb-java-v2-to-v3

Conclusion

This article covered DynamoDB encryption from server-side encryption to client-side attribute-level encryption.

Server-side encryption is handled transparently by DynamoDB, so you rarely need to think about it in day-to-day development. However, understanding the key options and audit requirements remains important.

On the other hand, client-side attribute-level encryption is a powerful way to protect data from third parties including AWS, but it comes with tradeoffs such as query constraints and increased item size. In particular, if you need to search on encrypted attributes, Beacons must be introduced, and the added design complexity should be considered upfront.

Note that the AWS Database Encryption SDK for DynamoDB currently supports only Java, .NET, and Rust. If you want to perform client-side encryption with Python, you'll need to use the legacy DynamoDB Encryption Client (v3.x), but be aware that newer features like Beacons are not available.

When considering adoption, I recommend first identifying "which attributes are sensitive and which attributes need to be searchable," then deciding on crypto actions and whether Beacons are needed.

I hope this article helps improve your understanding of DynamoDB encryption.

Benchmarking AWS Lambda Functions with Side Effects Using AWS Lambda Power Tuning

ほうき星 — Fri, 24 Jul 2026 13:01:57 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

副作用のある Lambda を AWS Lambda Power Tuning でチューニングする方法 #lambda - Qiita

はじめにこんにちは、ほうき星 @H0ukiStar です。 AWS Lambda は、割り当てるメモリのサイズを増やすことによって CPU をはじめとする実行リソースも比例して割り当てられる特徴があります。そのため、割り当てるメモリサイズを変更することにより処理時間も...

qiita.com

Introduction

Hello, I'm @H0ukiStar.

One of the characteristics of AWS Lambda is that increasing the allocated memory also proportionally increases the CPU and other execution resources available to the function.

As a result, changing the memory allocation can significantly affect execution time. Depending on the workload, allocating more memory may reduce the execution time enough to lower the overall execution cost.

However, the optimal memory size depends on the workload, and manually evaluating every memory configuration from 128 MB to 10,240 MB is impractical.

One tool that helps solve this problem is AWS Lambda Power Tuning.

In this article, I'll show you how to use AWS Lambda Power Tuning with Lambda functions that have side effects, such as writing to or deleting data from a database.

What is AWS Lambda Power Tuning?

AWS Lambda Power Tuning is an open-source tool for finding the optimal memory configuration for AWS Lambda functions.

As mentioned earlier, Lambda functions can be configured with memory sizes ranging from 128 MB to 10,240 MB. Increasing the memory allocation also proportionally increases the CPU and other execution resources.

Since Lambda pricing is based on allocated memory × execution duration, increasing the memory allocation can actually reduce the total execution cost if it shortens the execution time sufficiently.

Manually running the same function across multiple memory configurations and comparing the results is tedious and time-consuming.

AWS Lambda Power Tuning automates this process by executing your Lambda function with multiple memory settings and visualizing the performance and cost of each configuration.

This allows you to choose the optimal memory size based on actual measurements rather than intuition.

The project was created by Alex Casalboni and is available on GitHub:

alexcasalboni / aws-lambda-power-tuning

AWS Lambda Power Tuning is an open-source tool that can help you visualize and fine-tune the memory/power configuration of Lambda functions. It runs in your own AWS account - powered by AWS Step Functions - and it supports three optimization strategies: cost, speed, and balanced.

AWS Lambda Power Tuning

AWS Lambda Power Tuning is a state machine powered by AWS Step Functions that helps you optimize your Lambda functions for cost and/or performance in a data-driven way.

The state machine is designed to be easy to deploy and fast to execute. Also, it's language agnostic so you can optimize any Lambda functions in your account.

Basically, you can provide a Lambda function ARN as input and the state machine will invoke that function with multiple power configurations (from 128MB to 10GB, you decide which values). Then it will analyze all the execution logs and suggest you the best power configuration to minimize cost and/or maximize performance.

Note

Please note that the input function will be executed in your AWS account and perform real HTTP requests, SDK calls, cold starts, etc. The state machine also supports cross-region invocations and you can enable parallel execution to generate…

View on GitHub

Deployment

AWS Lambda Power Tuning can be deployed in several ways, such as cloning the GitHub repository and deploying it with AWS SAM, or deploying it directly from the Serverless Application Repository (SAR).

In this article, we'll use the SAR deployment because it's the quickest way to get started.

Open the AWS Lambda Power Tuning application in the Serverless Application Repository and click Deploy.
In the AWS Management Console, review the application parameters, acknowledge IAM role creation, and deploy the application.
Wait until the deployment status becomes Create complete.

Running AWS Lambda Power Tuning

AWS Lambda Power Tuning is implemented as an AWS Step Functions state machine.

To start tuning, execute the state machine with the ARN of the target Lambda function and the desired configuration.

Open powerTuningStateMachine and choose Start execution.
Enter the following JSON as the input, then choose Start execution.

  {
    "lambdaARN": "<Target Lambda ARN>",
    "powerValues": [128, 256, 512, 1024],
    "num": 10,
    "payload": {}
  }

Parameters

powerValues: Memory configurations to test

num: Number of executions for each memory configuration

payload: Event payload passed to the target Lambda function

Viewing the Results

After the execution finishes, open the Execution input and output tab.

The output contains a visualization URL, which displays an interactive report showing execution time, cost, and the recommended memory configuration.

In the example below, 512 MB provides the lowest execution cost.

Why Side Effects Become a Problem

AWS Lambda Power Tuning executes the target Lambda function as follows:

Executes the function num times sequentially for each memory configuration (default behavior)
Executes different memory configurations in parallel

Because of this behavior, Lambda functions that interact with external systems—such as databases—may not execute under identical conditions.

For example, a function that:

Reads from a database
Inserts records
Updates records
Deletes records

changes the external state every time it runs.

To obtain reliable benchmark results, you must restore the execution environment before every invocation.

Depending on your workload, simply preparing the required data may not be sufficient.

Each execution may require its own isolated dataset to ensure that previous executions do not affect subsequent ones.

For example, if multiple executions share the same database records, updates or deletions performed by earlier executions may change the behavior of later executions, resulting in inaccurate benchmark results.

Fortunately, AWS Lambda Power Tuning provides two properties specifically designed for this purpose:

preProcessorARN
postProcessorARN

These allow you to execute arbitrary Lambda functions immediately before and after every benchmark execution.

Preparing and Cleaning Up Data with preProcessorARN / postProcessorARN

AWS Lambda Power Tuning allows you to specify Lambda functions for preprocessing and postprocessing by using the preProcessorARN and postProcessorARN properties.

This makes it possible to prepare and clean up database data before and after every benchmark execution, even when the target Lambda function has side effects.

{
  "lambdaARN": "<Target Lambda ARN>",
  "powerValues": [128, 256, 512, 1024],
  "num": 10,
  "payload": {},
  "preProcessorARN": "<Pre-processing Lambda ARN>",
  "postProcessorARN": "<Post-processing Lambda ARN>"
}

When preProcessorARN and postProcessorARN are specified, every benchmark execution follows this sequence:

One important thing to note is how payload is handled.

According to the executor logic:

function Executor:
  iterate from 0 to num:
    [payload = execute Pre-processor (payload)]
    results = execute Main Function (payload)
    [execute Post-processor (results)]

The original payload is first passed to the pre-processing Lambda.

The return value of the pre-processing Lambda then becomes the input to the target Lambda function.

Therefore, if your target Lambda requires the original payload, your pre-processing Lambda must return it (or return a modified payload that includes the required information).

Note

This behavior only occurs when preProcessorARN is specified.

Without a pre-processing Lambda, the original payload is passed directly to the target Lambda function.

Conclusion

AWS Lambda Power Tuning is an excellent tool for optimizing Lambda memory allocation.

However, if your Lambda function modifies external resources such as a database, you may assume that Power Tuning cannot be applied safely.

As shown in this article, using preProcessorARN and postProcessorARN allows you to prepare and clean up your environment before and after every benchmark execution, ensuring that each run starts from the same state.

It's also important to understand how payload is passed between the pre-processing Lambda and the target Lambda, as this behavior can be surprising the first time you use it.

If you're planning to benchmark a Lambda function with side effects, I hope this article helps you apply AWS Lambda Power Tuning with confidence.

I Built a CLI Tool to Delete Default VPCs Across All AWS Regions

ほうき星 — Thu, 04 Jun 2026 12:39:47 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

AWS 全リージョンのデフォルト VPC を一括削除する CLI ツールを作った #Python - Qiita

はじめにこんにちは、ほうき星 @H0ukiStar です。 AWS アカウントを作成すると、デフォルト VPC と呼ばれる VPC が各リージョンに 1 つずつ作成されます。このデフォルト VPC はパブリックサブネットのみで構成されており、これらのサブネットでは E...

qiita.com

Introduction

Hello, I’m @H0ukiStar.

When you create an AWS account, a VPC called the default VPC is automatically created in each region.

This default VPC consists only of public subnets, and the default subnets are configured to automatically assign public IP addresses when launching EC2 instances. When launching an EC2 instance from the AWS Management Console, this default VPC is also selected by default, which can lead to resources being created with unintended network configurations depending on your environment.

For this reason, if the default VPC is not needed in your organization’s network design, some teams choose to delete it in advance as part of their operational baseline.

In this article, I’ll introduce a CLI tool I created to delete default VPCs across all available regions in an AWS account.

CLI Tool for Deleting Default VPCs: aws-default-vpc-cleaner

The tool is available in the following repository:

H0ukiStar / aws-default-vpc-cleaner

A tool to delete default VPCs and related resources across all AWS regions.

AWS Default VPC Cleaner

A tool to delete default VPCs and related resources across all AWS regions.

AWSアカウント上のすべてのリージョンに存在するデフォルトVPCと関連リソースを削除するツール。

Features / 機能

Multi-Region Support / 複数リージョン対応: Delete default VPCs across all AWS regions or specific regions / すべてのAWSリージョンまたは特定のリージョンのデフォルトVPCを削除
Dry Run Mode / ドライランモード: List resources without deleting them / 削除せずにリソースをリスト表示
Safe Deletion / 安全な削除: Deletes resources in the correct order to avoid dependency issues / 依存関係の問題を回避するために正しい順序でリソースを削除
Multi-Language / 多言語対応: Supports English and Japanese output / 英語と日本語の出力をサポート
Verbose Mode / 詳細モード: Detailed logging of operations / 操作の詳細なログ出力

Deleted Resources / 削除されるリソース

The tool deletes the following resources in order: ツールは以下のリソースを順番に削除します:

Internet Gateways (detached then deleted) / インターネットゲートウェイ（デタッチ後に削除）
Subnets / サブネット
Route Tables (excluding main route table) / ルートテーブル（メインルートテーブルを除く）
Security Groups (excluding default security group) / セキュリティグループ（デフォルトセキュリティグループを除く）
Network ACLs (excluding default ACL) / ネットワークACL（デフォルトACLを除く）
VPC / VPC

Installation / インストール

Prerequisites / 前提条件

Python 3.10 or higher /…

View on GitHub

Installing the Tool

Build artifacts for both Linux and Windows are available in the Releases section.

Below is an installation example on CloudShell (Amazon Linux 2023).

$ wget https://github.com/H0ukiStar/aws-default-vpc-cleaner/releases/download/v1.0.0/aws-default-vpc-cleaner-v1.0.0-linux-x64
～omitted～
2026-05-21 14:35:22 (29.2 MB/s) - ‘aws-default-vpc-cleaner-v1.0.0-linux-x64’ saved [34495320/34495320]
$ chmod +x aws-default-vpc-cleaner-v1.0.0-linux-x64 
$ 
$ ./aws-default-vpc-cleaner-v1.0.0-linux-x64 --help
usage: aws-default-vpc-cleaner [-h] [--regions REGION [REGION ...]] [--dry-run] [--yes] [--lang {en,ja}] [--verbose] [--version]

Delete default VPCs and related resources across AWS regions

options:
  -h, --help            show this help message and exit
  --regions REGION [REGION ...]
                        Specify target regions (default: all regions)
  --dry-run             List resources without deleting them
  --yes, -y             Skip confirmation prompts
  --lang {en,ja}        Language for output (en/ja)
  --verbose, -v         Enable verbose output
  --version             show program's version number and exit

Examples / 使用例:
  # Delete default VPCs in all regions (with confirmation)
  # 全リージョンのデフォルトVPCを削除 (確認あり)
  aws-default-vpc-cleaner

  # Delete default VPCs in specific regions
  # 特定リージョンのデフォルトVPCを削除
  aws-default-vpc-cleaner --regions us-east-1 us-west-2

  # List default VPCs without deleting (dry-run)
  # 削除せずにデフォルトVPCをリストアップ (ドライラン)
  aws-default-vpc-cleaner --dry-run

  # Delete without confirmation
  # 確認なしで削除
  aws-default-vpc-cleaner --yes

  # Use Japanese language output
  # 日本語出力を使用
  aws-default-vpc-cleaner --lang ja

  # Verbose output
  # 詳細出力
  aws-default-vpc-cleaner --verbose

Deleting Default VPCs with the Tool

Before deleting default VPCs, it’s a good idea to first use the --dry-run option to check what resources will be affected.

As shown below, the tool lists default VPCs in all available regions, along with related resources such as internet gateways and subnets.

$ ./aws-default-vpc-cleaner-v1.0.0-linux-x64 --dry-run
Starting AWS Default VPC Cleaner...
DRY RUN MODE - No resources will be deleted
Fetching available regions...
Found 17 regions
Default VPC found: vpc-0dfe2040fc98de87c
Default VPC found: vpc-0088a011ef4fd1a43
～omitted～
Processing region: ap-south-1
Would delete InternetGateway: igw-0d46d61ebaff6cb2d
Would delete Subnet: subnet-0c5d458b14c891426
Would delete Subnet: subnet-0f875743062c40fd8
Would delete Subnet: subnet-07a484491ecaaf387
Would delete VPC: vpc-0dfe2040fc98de87c
Region ap-south-1 completed
～omitted～

=== Summary ===
Regions processed: 17
VPCs deleted: 0
Resources deleted: 0
Errors: 0

Operation completed successfully

If the resources look correct, you can run the tool without --dry-run to actually delete them.

$ ./aws-default-vpc-cleaner-v1.0.0-linux-x64 --yes
Starting AWS Default VPC Cleaner...
Fetching available regions...
Found 17 regions
Default VPC found: vpc-0dfe2040fc98de87c
Default VPC found: vpc-0088a011ef4fd1a43
～omitted～
Processing region: ap-south-1
Deleted InternetGateway: igw-0d46d61ebaff6cb2d
Deleted Subnet: subnet-0c5d458b14c891426
Deleted Subnet: subnet-0f875743062c40fd8
Deleted Subnet: subnet-07a484491ecaaf387
Deleting VPC: vpc-0dfe2040fc98de87c
VPC deleted successfully: vpc-0dfe2040fc98de87c
Region ap-south-1 completed
～omitted～

=== Summary ===
Regions processed: 17
VPCs deleted: 17
Resources deleted: 89
Errors: 0

Operation completed successfully

Available Options

`--dry-run`

As shown in the example above, this option lists the default VPCs and related resources that would be deleted, but does not actually delete anything.

Instead of deleting resources immediately, it is recommended to review the list first and confirm that the targets are correct.

`--yes`

If --yes is not specified, the tool prompts for confirmation before deleting resources.

If --yes is specified, deletion proceeds without confirmation.

`--regions REGION [REGION ...]`

Specifies which regions to target for deleting default VPCs.

If not specified, the tool runs against all available regions.

`--lang {en,ja}`

Specifies the language used for the tool’s output.

`--verbose`

Enables verbose output during execution.

Conclusion

Default VPCs are a convenient feature automatically created when an AWS account is provisioned, but depending on your environment, they may not always be necessary. When multiple regions are involved, deleting them manually can also become tedious.

I created aws-default-vpc-cleaner as a CLI tool to make it easier to review and remove default VPCs across all available regions in one go.

I hope this tool and article are helpful for anyone facing the same challenge.

Trying AWS IoT Core's New GetConnection and ListSubscriptions APIs

ほうき星 — Fri, 29 May 2026 14:10:41 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

AWS IoT Core 単体で MQTT クライアントの接続状態を取得できるようになったので試す #awsIoT - Qiita

はじめにこんにちは、ほうき星 @H0ukiStar です。本日 AWS IoT Core に以下のようなアップデートがありました。 Posted on: May 28, 2026 Today, AWS IoT Core launches two new MQTT c...

qiita.com

Introduction

Hello, I'm @H0ukiStar.

Today, AWS IoT Core announced the following update.

Posted on: May 28, 2026
Today, AWS IoT Core launches two new MQTT connection management APIs, GetConnection and ListSubscriptions, enabling you to easily access MQTT client connection and subscription information for your Internet of Things (IoT) devices. These APIs help you troubleshoot connectivity issues, monitor client behavior, and audit connection patterns across your device fleet.

https://aws.amazon.com/about-aws/whats-new/2026/05/aws-iot-core-apis-mqtt/

Previously, checking device connection status required using Fleet Indexing or implementing your own state management with features such as Device Shadow. With this update, it is now possible to check device connection status directly using AWS IoT Core alone.

I previously introduced methods for checking connection status using Fleet Indexing and Device Shadow in the following article, so feel free to check it out as well.

AWS IoT Core でモノの接続状態をオンデマンドで取得する2つの方法 #lambda - Qiita

はじめにこんにちは、ほうき星です。皆さんは、AWS IoT Core に接続した自作 IoT デバイス（Raspberry Pi / ESP32 など）が「今オンラインなのか、それともオフラインなのか」を確認したいと思ったことはありませんか？私は自作した IoT ...

qiita.com

Trying It Out

Let's try the new APIs.

At the time of writing, these APIs do not appear to be available in the AWS SDK yet, so the API requests must be signed manually using SigV4.

This time, I used awscurl for signing. You can install it with the following command:

pip install awscurl

You can also retrieve the API endpoint using the AWS CLI as shown below.

aws iot describe-endpoint --endpoint-type iot:Data-ATS

{
    "endpointAddress": "abcd1234567890-ats.iot.ap-northeast-1.amazonaws.com"
}

GetConnection

This API allows you to retrieve information such as the connection status and Keep Alive configuration of a specified MQTT client.

docs.aws.amazon.com

Send the following request using awscurl.

awscurl \
  --service iotdevicegateway \
  --region ap-northeast-1 \
  "https://abcd1234567890-ats.iot.ap-northeast-1.amazonaws.com/connections/<clientId>"

[!WARNING]
For SigV4 signing, you must specify iotdevicegateway as the service name instead of iot.

You can retrieve information such as whether the client is currently connected and when the connection was established.

{
    "connected": true,
    "cleanSession": true,
    "clientId": "clientId",
    "thingName": "thingName",
    "keepAliveDuration": 15,
    "connectedSince": 1780061470032
}

You can also retrieve socket information by adding includeSocketInformation=true.

awscurl \
  --service iotdevicegateway \
  --region ap-northeast-1 \
  "https://abcd1234567890-ats.iot.ap-northeast-1.amazonaws.com/connections/<clientId>?includeSocketInformation=true"

{
    "connected": true,
    "cleanSession": true,
    "clientId": "clientId",
    "thingName": "thingName",
    "sourceIp": "sourceIp",
    "sourcePort": sourcePort,
    "targetIp": "targetIp",
    "targetPort": 8883,
    "keepAliveDuration": 15,
    "connectedSince": 1780061470032
}

ListSubscriptions

This API allows you to retrieve the topics subscribed to by a specified client.

docs.aws.amazon.com

Send the following request using awscurl.

awscurl \
  --service iotdevicegateway \
  --region ap-northeast-1 \
  "https://abcd1234567890-ats.iot.ap-northeast-1.amazonaws.com/connections/<clientId>/subscriptions"

[!WARNING]
For SigV4 signing, you must specify iotdevicegateway as the service name instead of iot.

You can retrieve the topics currently subscribed to by the specified client as shown below.

{
    "nextToken": "",
    "subscriptions": [
        {
            "topicFilter": "$aws/things/<thingName>/shadow/update/delta",
            "qos": 0
        }
    ]
}

Conclusion

With the newly introduced GetConnection and ListSubscriptions APIs, it is now possible to retrieve MQTT client connection status and subscription information directly from AWS IoT Core.

Previously, checking connection status often required custom implementations using Fleet Indexing, Device Shadow, or Lifecycle Events. With these new APIs, monitoring client state should become much simpler.

Although the APIs do not yet appear to be available in the AWS SDK, they can already be used today by manually signing requests with SigV4.

If you're interested, give them a try.

Automatically Healing CloudFormation Drift with Durable Functions

ほうき星 — Sun, 17 May 2026 02:34:59 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

Durable Functions を用いて CloudFormation のドリフトを自動修復する #AWS - Qiita

はじめにこんにちは、ほうき星 @H0ukiStar です。皆さんは昨年(2025年)の11月に CloudFormation がアップデートされ、ドリフト状態の修正に利用可能なドリフト認識変更セットが追加されたことをご存じでしょうか？本機能の登場以前は、ドリフ...

qiita.com

Introduction

Hello, I’m @H0ukiStar.

Did you know that in November 2025, AWS introduced a new CloudFormation feature called drift-aware change sets?

https://aws.amazon.com/about-aws/whats-new/2025/11/configuration-drift-enhanced-cloudformation-sets/

Before this feature was introduced, repairing drift typically required making a temporary dummy change to the stack, updating the stack, and then reverting the change afterward.

Now, by specifying --deployment-mode REVERT_DRIFT when creating a change set, CloudFormation can recognize differences between the IaC definition and the actual infrastructure, and automatically generate a change set specifically for drift remediation.

In this article, I’ll show how I implemented a Configuration Healing mechanism using Durable Functions that:

Detects CloudFormation stack drift
Automatically creates a drift-aware change set
Repairs the drift automatically

Trying Drift-Aware Change Sets

Deploying a Sample Stack

First, let’s see how drift-aware change sets work.

I deployed the following CloudFormation template with the stack name test.

AWSTemplateFormatVersion: 2010-09-09
Description: Stack for testing drift-aware change sets

Resources:
  TestParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: /drift-test/sample
      Type: String
      Value: initial-value
      Description: Parameter for drift testing

Immediately after deployment, the stack is not drifted.

[!WARNING]
The stack was deployed using a dedicated CloudFormation execution role.

The IAM role template is omitted here for brevity.
Please refer to the following GitHub repository for the complete template definition.

sample-aws-cfn-configuration-healing

Introducing Drift

Next, I intentionally introduced drift by modifying the SSM parameter value with the following command.

aws ssm put-parameter \
  --name /drift-test/sample \
  --value updated-value \
  --overwrite

You can confirm from the CloudFormation drift detection screen that the stack is now drifted.

Repairing Drift with a Drift-Aware Change Set

On the CloudFormation drift screen, CloudFormation displays guidance indicating that the drift can be repaired using a drift-aware change set.

Select Create change set.

Make sure the change set type is set to Drift-aware change set, then follow the instructions to create it.

If you review the generated change set, you can see that CloudFormation correctly detects the drifted resource and prepares the remediation changes automatically.

After executing the change set, the drift is resolved.

Configuration Healing with Durable Functions

CloudFormation drift detection and change set creation are asynchronous operations, which means polling is required until completion.

Implementing this behavior using only standard Lambda functions can quickly become complicated due to retry handling, wait control, and state management.

This time, I used Durable Functions to simplify these wait operations and state transitions.

Durable Functions are designed for long-running workflows, making them a great fit for asynchronous APIs like CloudFormation.

Especially for workflows like:

Start operation
Wait for completion
Check status
Proceed to the next step

Durable Functions make the orchestration logic much easier to implement and maintain.

Workflow

The implementation follows the workflow below to detect and repair CloudFormation stack drift.

Durable Functions make it straightforward to implement waiting logic for both drift detection and change set creation.

In addition, by passing CreateChangeSetOnly: true in the Lambda event payload, the workflow can stop after creating the change set without executing it automatically.

Durable Functions Implementation Example

The Durable Functions implementation follows the workflow shown above.

The full source code, including the SAM template, is available in the following repository.

H0ukiStar / sample-aws-cfn-configuration-healing

Sample implementation of an AWS Lambda function to detect and automatically heal CloudFormation stack configuration drift.

CloudFormation Configuration Healing with Durable Functions

Automatic detection and healing of CloudFormation stack drift using AWS Lambda with Durable Functions.

This sample demonstrates how to automatically detect configuration drift in CloudFormation stacks and heal them by creating and executing change sets using the AWS Durable Execution SDK for Python.

Features

Automatic Drift Detection: Detects configuration drift in CloudFormation stacks
Automatic Healing: Creates and executes change sets to restore the stack to its desired state
Durable Functions: Uses AWS Lambda Durable Execution SDK to handle long-running operations reliably
SNS Notifications: Sends notifications about drift detection and healing operations
Error Handling: Comprehensive error handling for drift detection and change set operations

Installation

Deploy using the AWS SAM CLI with the following commands:

cd configuration-healing
sam build
sam deploy --guided

During the guided deployment, you will be prompted to provide:

SNS Topic ARN: The ARN of an…

View on GitHub

# The complete implementation is available in the GitHub repository above.
# The code is omitted here for brevity.

Verification

Next, I intentionally modified the value of the SSM Parameter created in the test stack to introduce drift, then executed the deployed Lambda function.

aws lambda invoke \
  --function-name arn:aws:lambda:<region>:<account-id>:function:cfn-drift-healing:Alias \
  --invocation-type Event \
  --cli-binary-format raw-in-base64-out \
  --payload '{"StackName": "test"}' \
  response.json

From the logs, you can confirm that the automatic remediation proceeds as follows:

Start drift detection
Detect stack drift
Create a drift-aware change set
Execute the change set
Resolve the drift

You can also confirm the remediation completion through SNS notifications.

You can also stop the workflow after creating the change set by passing CreateChangeSetOnly: true in the event payload.

This allows a human operator to review and execute the change set manually.

aws lambda invoke \
  --function-name arn:aws:lambda:<region>:<account-id>:function:cfn-drift-healing:Alias \
  --invocation-type Event \
  --cli-binary-format raw-in-base64-out \
  --payload '{"StackName": "test", "CreateChangeSetOnly": true}' \
  response.json

Conclusion

CloudFormation drift-aware change sets make it much safer and easier to repair configuration drift than before.

In this article, I implemented a Configuration Healing mechanism using Durable Functions that automatically detects and repairs CloudFormation drift.

Even when using IaC, long-running environments inevitably experience unintended configuration changes over time.

By combining this workflow with services like EventBridge Scheduler for periodic execution, you can continuously and automatically remediate drift caused by:

Unintended manual changes
Temporary fixes that were never reverted
Configuration updates forgotten over time

This helps maintain infrastructure consistency and improve long-term IaC governance.

If fully automated remediation feels too risky for production environments, you can instead stop after creating the change set and require human review before execution, as demonstrated earlier in this article.

I hope this article serves as a useful example of implementing Configuration Healing with CloudFormation.

How to Create an EventBridge Scheduler with ActionAfterCompletion in CloudFormation

ほうき星 — Sun, 26 Apr 2026 09:09:56 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

CloudFormation で未対応の ActionAfterCompletion を指定した EventBridge Scheduler を作成する #AWS - Qiita

はじめにこんにちは、ほうき星 @H0ukiStar です。皆さんは CloudFormation で ActionAfterCompletion を指定した EventBridge Schedulder を作成したいと思ったことはありませんか？私はあります。 Eve...

qiita.com

Introduction

Hi, I'm @H0ukiStar.

Have you ever wanted to create an EventBridge Scheduler with ActionAfterCompletion specified in CloudFormation?

I have.

The CloudFormation resource for creating EventBridge Scheduler schedules, AWS::Scheduler::Schedule, does not currently support the ActionAfterCompletion property, as shown in the official documentation below:

https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-scheduler-schedule.html

My assumption is that this limitation exists because when ActionAfterCompletion is set to DELETE, the schedule resource is automatically deleted after execution completes. If CloudFormation allowed this property, the resource would disappear outside of CloudFormation’s control, potentially causing drift.

However, there are cases where it is useful to create a schedule with ActionAfterCompletion from CloudFormation. For example, if you repeatedly deploy one-time schedules through CloudFormation, being able to specify ActionAfterCompletion: DELETE allows the schedule to clean itself up automatically after execution.

In this article, I will show how to achieve this by using a CloudFormation custom resource.

What is a CloudFormation Custom Resource (`AWS::CloudFormation::CustomResource`)?

A CloudFormation custom resource allows you to create resources or execute APIs that are not natively supported by CloudFormation.

It can invoke an SNS topic or a Lambda function, and the invoked function can perform tasks such as:

Creating AWS resources not supported by native CloudFormation resources
Integrating with external services through APIs
Automating initialization or registration processes

https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-cloudformation-customresource.html

In this article, we will use a custom resource backed by AWS Lambda to create an EventBridge Scheduler schedule with ActionAfterCompletion.

Sample Lambda Function for Creating a Schedule Resource with `ActionAfterCompletion`

A CloudFormation custom resource has a ServiceToken property, which is used to specify the ARN of the Lambda function that handles the resource lifecycle.

The Lambda function referenced by ServiceToken can either be defined inline in the CloudFormation template or deployed in advance and referenced externally.

In this example, I prepared the Lambda function beforehand and deployed it using AWS SAM.

The complete source code, including the SAM template, is available in the following repository:

https://github.com/H0ukiStar/sample-aws-cfn-schedule-with-action-after-completion

# lambda_function.py
from __future__ import print_function
import re
import json
from typing import Optional
from datetime import datetime

import boto3
import urllib3
from pydantic import BaseModel, ValidationError


class FlexibleTimeWindowProperty(BaseModel):
    """
    Flexible time window configuration for EventBridge Scheduler.
    """
    MaximumWindowInMinutes: Optional[int] = None
    Mode: Optional[str] = None


class TargetProperty(BaseModel):
    """
    Target configuration for EventBridge Scheduler.
    """
    class DeadLetterConfigProperty(BaseModel):
        """
        Dead letter queue configuration.
        """
        Arn: Optional[str] = None

    class EventBridgeParametersProperty(BaseModel):
        """
        EventBridge event parameters.
        """
        DetailType: Optional[str] = None
        Source: Optional[str] = None

    class KinesisParametersProperty(BaseModel):
        """
        Kinesis stream parameters.
        """
        PartitionKey: Optional[str] = None

    class RetryPolicyProperty(BaseModel):
        """
        Retry policy configuration.
        """
        MaximumEventAgeInSeconds: Optional[int] = None
        MaximumRetryAttempts: Optional[int] = None

    class SageMakerPipelineParametersProperty(BaseModel):
        """
        SageMaker Pipeline parameters.
        """
        class PipelineParameterListItemProperty(BaseModel):
            """
            Individual pipeline parameter.
            """
            Name: Optional[str] = None
            Value: Optional[str] = None

        PipelineParameterList: Optional[list[PipelineParameterListItemProperty]] = None

    class SqsParametersProperty(BaseModel):
        """
        SQS queue parameters.
        """
        MessageGroupId: Optional[str] = None

    Arn: Optional[str] = None
    DeadLetterConfig: Optional[DeadLetterConfigProperty] = None
    # EcsParameters: Optional[dict] = None
    EventBridgeParameters: Optional[EventBridgeParametersProperty] = None
    Input: Optional[str] = None
    KinesisParameters: Optional[KinesisParametersProperty] = None
    RetryPolicy: Optional[RetryPolicyProperty] = None
    RoleArn: Optional[str] = None
    SageMakerPipelineParameters: Optional[SageMakerPipelineParametersProperty] = None
    SqsParameters: Optional[SqsParametersProperty] = None


class ScheduleProperty(BaseModel):
    """
    EventBridge Scheduler schedule configuration.
    """
    ActionAfterCompletion: Optional[str] = None
    Description: Optional[str] = None
    EndDate: Optional[datetime] = None
    FlexibleTimeWindow: Optional[FlexibleTimeWindowProperty] = None
    GroupName: Optional[str] = None
    KmsKeyArn: Optional[str] = None
    Name: str
    ScheduleExpression: Optional[str] = None
    ScheduleExpressionTimezone: Optional[str] = None
    StartDate: Optional[datetime] = None
    State: Optional[str] = None
    Target: Optional[TargetProperty] = None


def lambda_handler(event: dict, context: object) -> None:
    """
    AWS Lambda handler for CloudFormation custom resource managing EventBridge Scheduler schedules.

    Handles Create, Update, and Delete operations for EventBridge Scheduler schedules
    as a CloudFormation custom resource. Supports ActionAfterCompletion property to enable
    automatic schedule actions (e.g., deletion) after completion.

    Parameters
    ----------
    event : dict
        Lambda event object containing CloudFormation request details.
    context : object
        Lambda context object containing runtime information.

    Returns
    -------
    None
        Sends response to CloudFormation via HTTP callback.

    Notes
    -----
    - For Create: Creates a new schedule and returns its ARN
    - For Update: Updates existing schedule or creates new one if Name changed
    - For Delete: Deletes the schedule unless it failed during creation/update
    """
    print(f"{event['ResourceProperties']=}")

    try:
        schedule_property: ScheduleProperty = ScheduleProperty(
            **event["ResourceProperties"]
        )
    except ValidationError as e:
        send(event, context, "FAILED", {}, reason=f"Resource properties pre validation failed: {e.errors()}")
        return

    scheduler_client = boto3.client("scheduler")
    request_type: str = event["RequestType"]
    print(f"{request_type=}")

    if request_type == "Create":
        try:
            response: dict = scheduler_client.create_schedule(
                **schedule_property.model_dump(exclude_none=True)
            )
            send(event, context, "SUCCESS", {"Arn": response["ScheduleArn"]}, physicalResourceId=schedule_property.Name)
        except Exception as e:
            send(event, context, "FAILED", {}, reason=f"{e}", physicalResourceId="CREATE_FAILED")
        return
    elif request_type == "Update":
        try:
            old_schedule_property: ScheduleProperty = ScheduleProperty(
                **event["OldResourceProperties"]
            )
        except ValidationError as e:
            send(event, context, "FAILED", {}, reason=f"Old resource properties pre validation failed: {e.errors()}")
            return

        physical_resource_id: str = event["PhysicalResourceId"]

        # Create a new schedule if Name has changed
        if old_schedule_property.Name != schedule_property.Name:
            try:
                # CloudFormation will automatically delete the old schedule when the physical ID changes, so no explicit deletion is needed in Update.
                response: dict = scheduler_client.create_schedule(
                    **schedule_property.model_dump(exclude_none=True)
                )
                send(event, context, "SUCCESS", {"Arn": response["ScheduleArn"]}, physicalResourceId=schedule_property.Name)
            except Exception as e:
                send(event, context, "FAILED", {}, reason=f"{e}", physicalResourceId="UPDATE_FAILED")
            return

        # Update the schedule if Name is the same
        try:
            existing_schedule: dict = scheduler_client.get_schedule(Name=physical_resource_id)
            existing_schedule_property: ScheduleProperty = ScheduleProperty(**existing_schedule)

            update_params: dict = existing_schedule_property.model_dump(exclude_none=True)
            new_params: dict = schedule_property.model_dump(exclude_none=True)
            update_params.update(new_params)

            response: dict = scheduler_client.update_schedule(**update_params)
            send(event, context, "SUCCESS", {"Arn": response["ScheduleArn"]}, physicalResourceId=schedule_property.Name)
        except Exception as e:
            send(event, context, "FAILED", {}, reason=f"{e}", physicalResourceId=physical_resource_id)
        return
    elif request_type == "Delete":
        physical_resource_id: str = event["PhysicalResourceId"]

        # Skip deletion and return success if CREATE_FAILED (schedule does not exist) or UPDATE_FAILED (should not delete)
        if physical_resource_id == "CREATE_FAILED" or physical_resource_id == "UPDATE_FAILED":
            send(event, context, "SUCCESS", {})
            return

        try:
            scheduler_client.delete_schedule(Name=physical_resource_id)
            send(event, context, "SUCCESS", {})
        except scheduler_client.exceptions.ResourceNotFoundException:
            # If the schedule is already deleted, consider it a success
            send(event, context, "SUCCESS", {})
        except Exception as e:
            send(event, context, "FAILED", {}, reason=f"{e}")
        return
    send(event, context, "FAILED", {}, reason=f"Unsupported request type: {request_type}")
    return


# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT-0
http = urllib3.PoolManager()


def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None):
    responseUrl = event['ResponseURL']

    responseBody = {
        'Status': responseStatus,
        'Reason': reason or "See the details in CloudWatch Log Stream: {}".format(context.log_stream_name),
        'PhysicalResourceId': physicalResourceId or context.log_stream_name,
        'StackId': event['StackId'],
        'RequestId': event['RequestId'],
        'LogicalResourceId': event['LogicalResourceId'],
        'NoEcho': noEcho,
        'Data': responseData
    }

    json_responseBody = json.dumps(responseBody)

    print("Response body:")
    print(json_responseBody)

    headers = {
        'content-type': '',
        'content-length': str(len(json_responseBody))
    }

    try:
        response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody)
        print("Status code:", response.status)

    except Exception as e:
        print("send(..) failed executing http.request(..):", mask_credentials_and_signature(e))


def mask_credentials_and_signature(message):
    """
    Mask AWS credentials and signatures in error messages.

    Redacts sensitive AWS credential and signature information from messages
    before logging to prevent credential exposure.

    Parameters
    ----------
    message : str
        Error message or string that may contain AWS credentials.

    Returns
    -------
    str
        Message with credentials and signatures masked.

    Notes
    -----
    Masks the following AWS authentication parameters:
    - X-Amz-Credential
    - X-Amz-Signature
    """
    message = re.sub(r'X-Amz-Credential=[^&\s]+', 'X-Amz-Credential=*****', message, flags=re.IGNORECASE)
    return re.sub(r'X-Amz-Signature=[^&\s]+', 'X-Amz-Signature=*****', message, flags=re.IGNORECASE)

Note

Pydantic validation in this sample is mainly used to convert values such as datetime and int. Validation for allowed strings or numeric ranges is left to the EventBridge Scheduler API through boto3.

Warning

EcsParameters is not implemented in this sample.

Verification

The following sample CloudFormation template creates an EventBridge Scheduler schedule with ActionAfterCompletion enabled:

AWSTemplateFormatVersion: 2010-09-09
Description: Sample template for managing EventBridge Scheduler with ActionAfterCompletion using a CloudFormation custom resource

Resources:
  CustomScheduleWithActionAfterCompletion:
    Type: Custom::ScheduleWithActionAfterCompletion
    Properties:
      ServiceTimeout: 30
      # Replace with the ARN of the deployed custom resource Lambda function
      ServiceToken: arn:aws:lambda:ap-northeast-1:123456789012:function:cfn-custom-resource-schedule-with-aac

      Name: schedule-with-aac
      ActionAfterCompletion: DELETE

      FlexibleTimeWindow:
        Mode: OFF

      # Replace with the desired schedule expression
      ScheduleExpression: at(2026-04-30T00:00:00)

      ScheduleExpressionTimezone: Asia/Tokyo

      Target:
        # Replace with the ARN of the target resource to invoke
        Arn: arn:aws:lambda:ap-northeast-1:123456789012:function:example-function

        # Replace with the ARN of the IAM role that EventBridge Scheduler assumes
        RoleArn: arn:aws:iam::123456789012:role/example-scheduler-role

Warning

Replace ServiceToken with the ARN of the deployed Lambda function

Replace Target with the desired schedule target configuration

Deploy the above template with CloudFormation. If the schedule is created successfully with ActionAfterCompletion set, the custom resource is working as expected.

Notes

As mentioned earlier, when ActionAfterCompletion is set to DELETE, the schedule resource is automatically removed after execution.

CloudFormation custom resources do not support drift detection, so even if the schedule is deleted by EventBridge Scheduler, the CloudFormation stack will not be marked as drifted.

However, when deleting the CloudFormation stack, the custom resource Lambda function receives a Delete event. Therefore, it is a good idea to handle ResourceNotFoundException as SUCCESS:

try:
    scheduler_client.delete_schedule(Name=physical_resource_id)
    send(event, context, "SUCCESS", {})
except scheduler_client.exceptions.ResourceNotFoundException:
    send(event, context, "SUCCESS", {})
except Exception as e:
    send(event, context, "FAILED", {}, reason=f"{e}")

Conclusion

In this article, I showed how to create an EventBridge Scheduler schedule with ActionAfterCompletion using a CloudFormation custom resource.

There are many cases where AWS SDK APIs support features that CloudFormation does not yet expose as native resources or properties. The same custom resource pattern introduced here can be applied to those situations as well.

I hope this article helps someone facing the same challenge.

Why You Should Use context.logger in Lambda Durable Functions

ほうき星 — Thu, 19 Mar 2026 13:25:20 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

Lambda Durable Functions のロギングには context.logger を使用しよう #Python - Qiita

はじめにこんにちは、ほうき星 @H0ukiStar です。皆さんは Lambda Durable Functions を使っていますでしょうか？ Lambda Durable Functions は Lambda 関数単体で Step Functions のようなマル...

qiita.com

Introduction

Hi, I'm @H0ukiStar.

Have you tried using Lambda Durable Functions?

Lambda Durable Functions is a feature that allows you to implement multi-step workflows—similar to Step Functions—using only a single Lambda function.

When implementing logging in Lambda Durable Functions, it is recommended to use context.logger and step_context.logger. But why is that?

In this article, I’ll explain the reason based on actual behavior and implementation details.

Logging with the Standard Method

First, in a typical Lambda function (Python), logging is usually done using the built-in logging module, like this:

import logging

logger: logging.Logger = logging.getLogger()
logger.setLevel(logging.INFO)

logger.info("Loading lambda_function.py")

Below is a sample where logging is performed using the logging module inside both steps and the lambda_handler in Lambda Durable Functions:

import logging


from aws_durable_execution_sdk_python.config import Duration
from aws_durable_execution_sdk_python.execution import durable_execution
from aws_durable_execution_sdk_python.context import DurableContext, StepContext, durable_step


logger: logging.Logger = logging.getLogger()
logger.setLevel(logging.INFO)


@durable_step
def step_one(step_context: StepContext, arg: int) -> str:
    logger.info("Hello from step_one")
    return f"from step_one: {arg=}"


@durable_step
def step_two(step_context: StepContext, arg: int) -> str:
    logger.info("Hello from step_two")
    return f"from step_two: {arg=}"


@durable_step
def step_three(step_context: StepContext, arg: int) -> str:
    logger.info("Hello from step_three")
    return f"from step_three: {arg=}"


@durable_execution
def lambda_handler(event: dict, context: DurableContext) -> None:
    msg_one: str = context.step(step_one(123))
    logger.info(msg_one)

    msg_two: str = context.step(step_two(456))
    logger.info(msg_two)

    context.wait(Duration.from_seconds(30))

    msg_three: str = context.step(step_three(789))
    logger.info(msg_three)

Execution Result

When you run this code and check the logs in CloudWatch, you will see output like this:

In summary, the logs behave as follows:

step_one() runs → "Hello from step_one" is logged
The return value "from step_one: arg=123" is logged (①)
step_two() runs → "Hello from step_two" is logged
The return value "from step_two: arg=456" is logged (②)
Execution pauses with context.wait()
After resuming, "from step_one: arg=123" is logged again (①')
"from step_two: arg=456" is logged again (②')
step_three() runs → "Hello from step_three" is logged
The return value "from step_three: arg=789" is logged
You can see that logs before and after context.wait() are duplicated.

Recommended Logging Method

As mentioned earlier, in Lambda Durable Functions:

Use context.logger in the handler
Use step_context.logger inside steps

Logging—I use context.logger for the main handler and step_context.logger inside steps. The context logger suppresses duplicate logs during replay.

Build multi-step applications and AI workflows with AWS Lambda durable functions | AWS News Blog

aws.amazon.com

Here is the updated version of the previous code using context.logger:

from aws_durable_execution_sdk_python.config import Duration
from aws_durable_execution_sdk_python.execution import durable_execution
from aws_durable_execution_sdk_python.context import DurableContext, StepContext, durable_step


@durable_step
def step_one(step_context: StepContext, arg: int) -> str:
    step_context.logger.info("Hello from step_one")
    return f"from step_one: {arg=}"


@durable_step
def step_two(step_context: StepContext, arg: int) -> str:
    step_context.logger.info("Hello from step_two")
    return f"from step_two: {arg=}"


@durable_step
def step_three(step_context: StepContext, arg: int) -> str:
    step_context.logger.info("Hello from step_three")
    return f"from step_three: {arg=}"


@durable_execution
def lambda_handler(event: dict, context: DurableContext) -> None:
    msg_one: str = context.step(step_one(123))
    context.logger.info(msg_one)

    msg_two: str = context.step(step_two(456))
    context.logger.info(msg_two)

    context.wait(Duration.from_seconds(30))

    msg_three: str = context.step(step_three(789))
    context.logger.info(msg_three)

Execution Result

The output now looks like this:

Unlike the previous case, there are no duplicated logs before and after context.wait().

Why This Is Recommended

1. Logs in the handler are not duplicated

From the results above, logs written using context.logger are not duplicated.

This is because Lambda Durable Functions uses a mechanism called replay.

When execution resumes after context.wait(), the function does NOT continue from where it left off. Instead, the handler is executed again from the beginning.

During this process:

Completed steps are skipped using checkpoint logs
Their results are reused instead of re-executing the step

When your function resumes after a pause or interruption, the SDK performs replay:

Load checkpoint log: The SDK retrieves the checkpoint log for this execution from Lambda.

Run from beginning: The SDK invokes your handler function from the start, not from where it paused.

Skip completed durable operations: As your code calls durable operations, the SDK checks each against the checkpoint log. For completed durable operations, the SDK returns the stored result without executing the operation code.

docs.aws.amazon.com

As a result:

Step functions are skipped → no duplicate logs inside steps
But handler code runs again → duplicate logs when using logging

On the other hand, context.logger is implemented to suppress duplicate logs during replay:

https://github.com/aws/aws-durable-execution-sdk-python/blob/main/src/aws_durable_execution_sdk_python/logger.py#L118-L131

2. Logs can be viewed in the "Logger output" tab

You might think:

"If step functions are skipped anyway, why not just use logging?"

While that works for avoiding duplication, there is another benefit.

Logs written using context.logger or step_context.logger can be viewed in the "Logger output" tab in the Durable Execution management console.

This allows you to:

View logs per execution
See timestamps and messages in a structured way

Internally, this is just a CloudWatch Logs Insights query:

fields timestamp, message
    | filter executionArn like "arn:aws:lambda:{region}:{accountId}:function:{functionname}:{version or alias}/durable-execution/{executionId}/{invocationId}"

You could replicate this format manually, but it's much easier to just use step_context.logger.

Conclusion

In this article, we explored why context.logger and step_context.logger are recommended in Lambda Durable Functions, based on both behavior and SDK implementation.

Lambda Durable Functions is a powerful feature that enables multi-step workflows within a single Lambda function.

I hope this article helps you when implementing Durable Functions in your own projects.