DynamoDb stream lambda trigger

Konstantin — Wed, 14 Apr 2021 08:55:54 +0000

Assignment

Let's take a look on a typical scenario, once it is required to subscribe to DynamoDb stream events and trigger AWS lambda function. In theory and in tutorials the task looks quite simple:

setup a lambda
add event source with the table attached. Sample:

private subscribeToDynamoDbStream(
    lambdaRole: Role,
    layer: LayerVersion,
    applicationName: string,
    targetTable: ITable) {
    const lambda = new Function(this, `${applicationName}-lambda`, 
    {
          code: Code.fromAsset('functions/custom-trigger/src'),
          functionName: `${applicationName}-lambda`,
          handler: handler,
          layers: [layer],
          role: lambdaRole,
          runtime: Runtime.NODEJS_12_X,
     });

     lambda.addEventSource(new DynamoEventSource(targetTable, {
     startingPosition: StartingPosition.TRIM_HORIZON, }));
}

This scenario works well in case of the single repository, once lambda, dynamodb and dynamodb stream are provisioned in the same stack.

Issue

In more complex layout the dynamodb itself is created and corresponding dynamodb stream is provisioned in a different stack. The dynamodb arn is saved to the SSM parameter store, or the name is known.
So, first import table:

private importTargetTable(environment: string, tableArn: string) {
    const table = Table.fromTableArn(this, 'ImportedTable', tableArn);
    return table;
  }

Then call subscribeToDynamoDbStream function:

const importedTable = importTargetTable(this.environment, this.tableArn);
this.subscribeToDynamoDbStream(lambdaRole, layer, applicationName, importedTable);

The result is unexpected: DynamoDB Streams must be enabled on the table [tablename]. The AWS console clearly shows that the Dynamodb stream is enabled. Quick googling recommends the most obvious thing: enable already enabled feature. CDK source code inspection reveals that during fromTableArn invocation the stream arn most probably has not been loaded.

Solution

First of all there should be DynamoDb stream arn exported to SSM during DynamoDB provisioning:

const targetTable = new Table(this, `targeTableName`, {
   partitionKey: { ... },
   sortKey: { ... },
   ...
   stream: StreamViewType.NEW_AND_OLD_IMAGES,
   tableName: `targetTableName`
    });
   // Export target table stream arn to SSM
   new StringParameter(this.ctx, `targetTable-stream-arn`, {
      description: "Exported stream arn",
      parameterName: `/targeTable-stream-arn`,
      stringValue: targetTable.tableStreamArn!
    });

In a stack where a lambda is created and subscription to the dynamodb stream is setup, tableStreamArn is read from SSM.

Option 1

Change the way how ITable is constructed:

private importTargetTable(environment: string, tableArn: string, tableStreamArn: string) {
    const table = Table.fromTableAttributes(this, 'ImportedTable', {
        tableArn: tableArn,
        tableStreamArn: tableStreamArn
    return table;
  }

Option 2

Technically speaking AWS lambda with DynamoDb stream trigger does not have to be aware of DynamoDb table (and it's arn). Option 2 could be considered as an improvement of the code above: 'Invent and Simplify'.
Once the lambda is created, EventSourceMapping object could be used to enable DynamoDb stream trigger for the lambda usin tableStreamArn only:

const lambda = new Function(this, ...);
new EventSourceMapping(this, `lambda-trigger-mapping`, {
    eventSourceArn: tableStreamArn,
    startingPosition: StartingPosition.TRIM_HORIZON,
    // any necessary options of a choice, e.g. batchSize, retryAttempt, etc.
    target: lambda
});

I hope that this post would help you in your CDK learning journey.
Best wishes..

Painful CDK - 1

Konstantin — Wed, 07 Apr 2021 09:18:29 +0000

AWS CDK is a great tool. Although it sometimes works not as expected. In this post, I'd like to share some CDK pain.

Let's take a look on the simple piece of code below. My goal is to create a new Cognito user pool authorizer and use it to protect a newly created endpoint with a lambda integration.

// create cognito authorizer:
const authorizer = new apigateway.CfnAuthorizer(this.ctx, "restApiMyResourceAuthorizer", {
      restApiId: this.restApi.restApiId,
      type: "COGNITO_USER_POOLS",
      identitySource: "method.request.header.Authorization",
      name: "restApiMyResourceAuthorizer",
      providerArns: [".."],
 });
// create a resource
const myResource: apigateway.Resource = this.restApi.root.addResource("myResource");
// add an endpoint to the Rest API resource with authorizer
myResource.addMethod(lambdaDeploymentOptions.httpMethod, new apigateway.LambdaIntegration(lambda),{
      authorizationType: apigateway.AuthorizationType.COGNITO,
      authorizer: {
        authorizerId: authorizer.node.id,
        authorizationType: apigateway.AuthorizationType.COGNITO,
      }
});

This gives:

Invalid authorizer ID specified. Setting the authorization type to CUSTOM or COGNITO_USER_POOLS requires a valid authorizer.

There are some opened questions in the internet on this topic.
Solution I've found is a bit hacky:

const method = myResource.addMethod(lambdaDeploymentOptions.httpMethod, new apigateway.LambdaIntegration(lambda),{
    authorizationType: apigateway.AuthorizationType.COGNITO,
    authorizer: {
    authorizerId: lambdaDeploymentOptions.authorizer.node.id,
    authorizationType: apigateway.AuthorizationType.COGNITO,
  }
});

const child = method.node.findChild('Resource') as apigateway.CfnMethod;
child.addPropertyOverride('AuthorizationType', 'COGNITO_USER_POOLS');
child.addPropertyOverride('AuthorizerId', { Ref: lambdaDeploymentOptions.authorizer.logicalId });

In case of questions: "why CfnAuthorizer, not CognitoUserPoolsAuthorizer. The reason is simple - CognitoUserPoolsAuthorizer does not allow to set Token Validation regular expression (identityValidationExpression).

P.S.
There is a general guide how to solve issues with CDK. "Escape hatches": https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html