DEV Community: Roger Chi

AppSync Cache Eviction with Lambda Data Sources

Roger Chi — Sun, 01 Feb 2026 02:49:27 +0000

AppSync is a fully managed, serverless GraphQL API service from AWS. It has the concept of data sources, which are direct connectors to various back end services and data stores, which are configured using either request and response templates when using the VTL resolver, or using some special JavaScript exports when using the JavaScript runtime resolver. This is not to be confused with the Lambda data source, which allows custom logic to be written in an AWS Lambda function (NodeJS or any other supported runtime or image), and which is a separate concept and layer apart from the JavaScript resolver. If you've only ever used AppSync with Lambda data sources, you may never have needed to work with the resolver mapping templates or code directly, as the service uses a default mapping configuration if one is not provided.

One feature of AppSync is the ability to enable a managed caching layer, which allows GraphQL responses to be stored in a cache and returned without requesting the data directly from the configured data source. One key functionality for a cache is the ability to evict cached entries -- important especially if a mutation runs that would cause a cached query to become stale and return incorrect results. This ability is found in the extensions.evictFromApiCache() function available only in the VTL mapping templates or JavaScript resolver code. This article describes the way to utilize this extension if you are using a Lambda data source.

The Pattern

The implementation involves two parts:

Lambda returns eviction metadata alongside the actual result
Response mapping template processes the eviction and cleans the response

Lambda Response Structure

Your Lambda function should return eviction instructions as a sibling field:

exports.handler = async (event) => {
  const bowlId = event.arguments.bowlId;

  // Update your data
  const fruitType = 'apple';
  const quantity = 5;
  await updateDatabase(bowlId, fruitType, quantity);

  // Return result with eviction metadata
  return {
    __evictFromApiCache: [['Query', 'fruit', { 'context.arguments.bowlId': bowlId }]],
    type: fruitType,
    quantity: quantity
  };
};

The __evictFromApiCache array contains:

'Query' - Operation type to evict
'fruit' - Field name to evict
{ 'context.arguments.bowlId': bowlId } - Cache key arguments

VTL Response Mapping Template

#if (!$util.isNull($ctx.result['__evictFromApiCache']))
  #foreach($evict in $ctx.result['__evictFromApiCache'])
    $extensions.evictFromApiCache($evict[0], $evict[1], $evict[2])
  #end

  $util.qr($ctx.result.remove('__evictFromApiCache'))
#end

$util.toJson($ctx.result)

JavaScript Response Mapping Template

import { util, extensions } from '@aws-appsync/utils';

export function request(ctx) {
  return {
    operation: 'Invoke',
    payload: { field: ctx.info.fieldName, arguments: ctx.args }
  };
}

export function response(ctx) {
  if (ctx.result.__evictFromApiCache) {
    ctx.result.__evictFromApiCache.forEach(evict => {
      extensions.evictFromApiCache(evict[0], evict[1], evict[2]);
    });
    delete ctx.result.__evictFromApiCache;
  }

  return ctx.result;
}

CDK Implementation

VTL Resolver

const dataSource = api.addLambdaDataSource('PickFruitVtlDataSource', pickFruitLambda);

dataSource.createResolver('PickFruitVtlResolver', {
  typeName: 'Mutation',
  fieldName: 'pickFruitVtl',
  responseMappingTemplate: appsync.MappingTemplate.fromString(`
#if (!$util.isNull($ctx.result['__evictFromApiCache']))
  #foreach($evict in $ctx.result['__evictFromApiCache'])
    $extensions.evictFromApiCache($evict[0], $evict[1], $evict[2])
  #end
  $util.qr($ctx.result.remove('__evictFromApiCache'))
#end

$util.toJson($ctx.result)
  `),
});

JavaScript Resolver

const dataSource = api.addLambdaDataSource('PickFruitJSDataSource', pickFruitLambda);

new appsync.Resolver(this, 'PickFruitJSResolver', {
  api: this.api,
  typeName: 'Mutation',
  fieldName: 'pickFruitJS',
  dataSource: dataSource,
  runtime: appsync.FunctionRuntime.JS_1_0_0,
  code: appsync.Code.fromInline(`
import { util, extensions } from '@aws-appsync/utils';

export function request(ctx) {
  return {
    operation: 'Invoke',
    payload: { field: ctx.info.fieldName, arguments: ctx.args }
  };
}

export function response(ctx) {
  if (ctx.result.__evictFromApiCache) {
    ctx.result.__evictFromApiCache.forEach(evict => {
      extensions.evictFromApiCache(evict[0], evict[1], evict[2]);
    });
    delete ctx.result.__evictFromApiCache;
  }

  return ctx.result;
}
  `),
});

Example Implementation

To demonstrate this pattern, I built a complete example with two GraphQL APIs backed by DynamoDB:

Non-cached API - No caching, always hits Lambda (~5 seconds)
Cached API - 5-minute cache with three mutation variants

GraphQL Schema

type Fruit {
  type: String
  quantity: Int
}

type Query {
  fruit(bowlId: Int!): Fruit
}

type Mutation {
  pickFruit(bowlId: Int!): Fruit      # No cache eviction
  pickFruitVtl(bowlId: Int!): Fruit   # VTL cache eviction
  pickFruitJS(bowlId: Int!): Fruit    # JS cache eviction
}

Lambda Implementation

The Lambda functions include a 5-second delay to make cache behavior visible:

exports.handler = async (event) => {
  await new Promise(resolve => setTimeout(resolve, 5000)); // Simulate work

  const bowlId = event.arguments?.bowlId || event.bowlId;
  const fruits = ['apple', 'banana', 'orange', 'grape', 'mango'];
  const fruitType = fruits[Math.floor(Math.random() * fruits.length)];
  const quantity = Math.floor(Math.random() * 10) + 1;

  // Update DynamoDB
  await ddb.send(new PutItemCommand({
    TableName: process.env.TABLE_NAME,
    Item: {
      bowlId: { N: String(bowlId) },
      fruitType: { S: fruitType },
      quantity: { N: String(quantity) }
    }
  }));

  // Return with eviction metadata (for VTL/JS mutations only)
  return {
    __evictFromApiCache: [['Query', 'fruit', { 'context.arguments.bowlId': bowlId }]],
    type: fruitType,
    quantity
  };
};

Test Results

Running the test script demonstrates the cache invalidation behavior:

[21:02:17.937] === Test Script ===

[21:02:17.947] 1. Pick fruit for bowl 1 (non-cached API)
[21:02:23.565] Result: 10x cantaloupe

[21:02:23.566] 2. Get fruit from non-cached API (3 times, ~5s each)
[21:02:28.915]   1. 10x cantaloupe (5349ms)
[21:02:34.034]   2. 10x cantaloupe (5118ms)
[21:02:39.179]   3. 10x cantaloupe (5144ms)

[21:02:39.180] 3. Get fruit from cached API (3 times, first ~5s, rest cached)
[21:02:39.411]   1. 4x grape (231ms)
[21:02:39.529]   2. 4x grape (118ms)
[21:02:39.637]   3. 4x grape (107ms)

[21:02:39.637] 4. Pick fruit without eviction (cached API)
[21:02:44.905] New fruit: 6x jackfruit

[21:02:44.906] 5. Get fruit from both APIs (cached API is stale)
[21:02:50.331] Non-cached API: 6x jackfruit
[21:02:50.331] Cached API (stale): 4x grape

[21:02:50.332] 6. Pick fruit with VTL eviction
[21:02:55.835] New fruit: 1x clementine

[21:02:55.836] 7. Get fruit from cached API (cache evicted, 3 times)
[21:03:01.171]   1. 1x clementine (5335ms)
[21:03:01.449]   2. 1x clementine (277ms)
[21:03:01.562]   3. 1x clementine (113ms)

[21:03:01.563] 8. Pick fruit with JS eviction
[21:03:07.213] New fruit: 4x strawberry

[21:03:07.214] 9. Get fruit from cached API (cache evicted, 3 times)
[21:03:12.436]   1. 4x strawberry (5222ms)
[21:03:12.537]   2. 4x strawberry (101ms)
[21:03:12.652]   3. 4x strawberry (114ms)

[21:03:12.653] === Test Complete ===

Key Observations

Non-cached API: Every query takes ~5 seconds (always hits Lambda)
Cached API: First query ~5s, subsequent queries ~100ms (cache hit)
Stale cache: Step 5 shows cached API returning old data after mutation without eviction
Successful eviction: Steps 7 and 9 show cache cleared after VTL/JS mutations (first query ~5s, then cached)

The timing differences clearly demonstrate when the cache is working versus when it's been properly evicted and needs repopulation.

Cache Key Format

The eviction key must exactly match your caching configuration. For a query with:

cachingConfig: {
  ttl: cdk.Duration.seconds(300),
  cachingKeys: ['$context.arguments.bowlId'],
}

Use the eviction key format:

{ 'context.arguments.bowlId': bowlId }

This pattern ensures your mutations properly invalidate the specific cache entries they affect, maintaining data consistency while preserving the performance benefits of caching.

Complete working example available on GitHub

I wrote an article describing a simple strategy for discovering service relationships in an EventBridge-based event-driven system.

Roger Chi — Mon, 12 Jan 2026 14:03:58 +0000

Simple Service Relationships in EventBridge

Roger Chi for AWS Community Builders ・ Jan 12

#aws #serverless #eventdriven #awscdk

Simple Service Relationships in EventBridge

Roger Chi — Mon, 12 Jan 2026 00:21:44 +0000

Event-driven architectures are powerful. Services communicate through events, staying loosely coupled while enabling complex workflows. This loose coupling comes with a tradeoff: as your system grows, understanding how services connect becomes increasingly difficult.

"Which services consume the OrderCreated event?"
"What happens if we change the PaymentCompleted event schema?"
"How do events flow through our system?"

In this article, I'll show you how to build a service graph visualization for EventBridge-based architectures using two complementary discovery mechanisms:

Producer Discovery: Using EventBridge Schema Registry to automatically discover which services produce which events
Consumer Discovery: Using a tagging convention on EventBridge Rules to identify which services consume which events

By combining these two data sources, we can generate an accurate, up-to-date visualization of how services communicate through events.

Setting Up Schema Registry for Producer Discovery

The first piece of our puzzle is discovering which services produce which events. EventBridge Schema Registry with automatic discovery handles this for us.

When you enable schema discovery on an event bus, EventBridge automatically creates schemas for every unique event type it sees. The schema names follow the pattern {source}@{DetailType}, for example: order-service@OrderCreated. This gives us what we need: a mapping from event sources to the events they produce.

AWS-CDK Setup

Here's TypeScript AWS-CDK code:

import * as cdk from 'aws-cdk-lib';
import * as events from 'aws-cdk-lib/aws-events';
import * as eventschemas from 'aws-cdk-lib/aws-eventschemas';
import type { Construct } from 'constructs';

export interface SchemaDiscovererStackProps extends cdk.StackProps {
  eventBus: events.IEventBus;
}

export class SchemaDiscovererStack extends cdk.Stack {
  public readonly discoverer: eventschemas.CfnDiscoverer;

  constructor(scope: Construct, id: string, props: SchemaDiscovererStackProps) {
    super(scope, id, props);

    this.discoverer = new eventschemas.CfnDiscoverer(this, 'SchemaDiscoverer', {
      sourceArn: props.eventBus.eventBusArn,
      description: `Schema Discoverer for ${props.eventBus.eventBusName} event bus`,
    });
}

Or, in pure CloudFormation:

CloudFormation Setup

AWSTemplateFormatVersion: '2010-09-09'
Description: EventBridge Schema Registry with automatic discovery

Parameters:
  EventBusName:
    Type: String
    Default: default
    Description: Name of the EventBridge event bus to discover schemas from

Resources:
  SchemaDiscoverer:
    Type: AWS::EventSchemas::Discoverer
    Properties:
      SourceArn: !Sub 'arn:aws:events:${AWS::Region}:${AWS::AccountId}:event-bus/${EventBusName}'
      Description: !Sub 'Schema Discoverer for ${EventBusName} event bus'

Outputs:
  DiscovererId:
    Description: ID of the Schema Discoverer
    Value: !Ref SchemaDiscoverer

Schema Limit Considerations

By default, Schema Registry has a limit of 200 discovered schemas per registry. If your bus has more than 200 unique event types, you'll need to request a limit increase through AWS Support.

Establishing a Rule Tagging Convention for Consumer Discovery

Schema Registry tells us who produces events, but we also need to know who consumes them. EventBridge Rules define event consumers, and there's a simple way to add metadata for which services are consuming which events: tag every EventBridge Rule with a Service tag containing the consuming service's name.

The Tagging Convention

Tag Key: Service
Tag Value: Service name

This convention is straightforward to implement and gives us everything we need to discover consumers.

CDK Implementation

Here's how to create a service stack that automatically tags all rules with the service name:

import * as cdk from 'aws-cdk-lib';
import * as events from 'aws-cdk-lib/aws-events';
import type { Construct } from 'constructs';

export interface ServiceStackProps extends cdk.StackProps {
  serviceName: string;
  consumedEvents: string[];
  eventBus: events.IEventBus;
}

export class ServiceStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: ServiceStackProps) {
    super(scope, id, props);

    const { serviceName, consumedEvents, eventBus } = props;

    // Apply Service tag at stack level - propagates to all rules
    cdk.Tags.of(this).add('Service', serviceName);

    // Create a Rule for each consumed event
    for (const consumedEvent of consumedEvents) {
      const rule = new events.Rule(this, `Rule-${consumedEvent}`, {
        ruleName: `${serviceDefinition.name}-${consumedEvent}`,
        eventBus,
        eventPattern: {
          detailType: [consumedEvent],
        },
      });
    }
  }
}

By applying the tag at the stack level with cdk.Tags.of(this).add('Service', serviceName), every rule created in the stack automatically inherits the tag.

CloudFormation Example

For CloudFormation users, here's how to create a tagged rule:

Resources:
  TaggedRule:
    Type: AWS::Events::Rule
    Properties:
      Name: !Sub '${ServiceName}-${DetailType}'
      EventBusName: !Ref EventBusName
      EventPattern:
        detail-type:
          - !Ref DetailType
      State: ENABLED
      Tags:
        - Key: Service
          Value: !Ref ServiceName

The key is being consistent. Every team creating EventBridge Rules needs to follow this convention. Consider adding validation in your CI/CD pipeline to ensure all rules have the Service tag. We handle this by creating a custom internal projen project type.

Discovering Producers and Consumers

With Schema Registry capturing producers and tagged Rules identifying consumers, we can now query both to build our service graph.

Producer Discovery

The ProducerDiscovery code queries Schema Registry and parses schema names to extract source/detail-type pairs:

import {
  SchemasClient,
  ListSchemasCommand,
} from '@aws-sdk/client-schemas';

export interface ProducerRecord {
  source: string;
  detailType: string;
  schemaName: string;
}

export class ProducerDiscovery {
  private readonly client: SchemasClient;
  private readonly registryName: string;

  constructor(registryName: string, client?: SchemasClient) {
    this.registryName = registryName;
    this.client = client ?? new SchemasClient({});
  }

  async discover(): Promise<ProducerRecord[]> {
    const producers: ProducerRecord[] = [];
    let nextToken: string | undefined;

    do {
      const command = new ListSchemasCommand({
        RegistryName: this.registryName,
        NextToken: nextToken,
      });

      const response = await this.client.send(command);

      if (response.Schemas) {
        for (const schema of response.Schemas) {
          if (schema.SchemaName) {
            const parsed = this.parseSchemaName(schema.SchemaName);
            if (parsed) {
              producers.push({
                source: parsed.source,
                detailType: parsed.detailType,
                schemaName: schema.SchemaName,
              });
            }
          }
        }
      }

      nextToken = response.NextToken;
    } while (nextToken);

    return producers;
  }

  private parseSchemaName(schemaName: string): { source: string; detailType: string } | null {
    const separatorIndex = schemaName.indexOf('@');
    if (separatorIndex === -1) return null;

    const source = schemaName.substring(0, separatorIndex);
    const detailType = schemaName.substring(separatorIndex + 1);

    if (!source || !detailType) return null;
    return { source, detailType };
  }
}

Consumer Discovery

The ConsumerDiscovery code lists all Rules, retrieves their Service tags, and extracts detail-type patterns:

import {
  EventBridgeClient,
  ListRulesCommand,
  ListTagsForResourceCommand,
} from '@aws-sdk/client-eventbridge';

export interface ConsumerRecord {
  serviceName: string;
  detailType: string;
  ruleName: string;
}

export class ConsumerDiscovery {
  private readonly client: EventBridgeClient;
  private readonly eventBusName: string;

  constructor(eventBusName: string, client?: EventBridgeClient) {
    this.eventBusName = eventBusName;
    this.client = client ?? new EventBridgeClient({});
  }

  async discover(): Promise<ConsumerRecord[]> {
    const consumers: ConsumerRecord[] = [];
    const rules = await this.listAllRules();

    for (const rule of rules) {
      if (!rule.Name || !rule.Arn) continue;

      const serviceName = await this.getServiceTag(rule.Arn);

      if (!serviceName) {
        console.warn(`Skipping rule "${rule.Name}": missing Service tag`);
        continue;
      }

      const detailTypes = this.extractDetailTypes(rule.EventPattern);

      for (const detailType of detailTypes) {
        consumers.push({
          serviceName,
          detailType,
          ruleName: rule.Name,
        });
      }
    }

    return consumers;
  }

  private async listAllRules() {
    const rules: any[] = [];
    let nextToken: string | undefined;

    do {
      const command = new ListRulesCommand({
        EventBusName: this.eventBusName,
        NextToken: nextToken,
      });

      const response = await this.client.send(command);
      if (response.Rules) rules.push(...response.Rules);
      nextToken = response.NextToken;
    } while (nextToken);

    return rules;
  }

  private async getServiceTag(ruleArn: string): Promise<string | undefined> {
    const command = new ListTagsForResourceCommand({ ResourceARN: ruleArn });
    const response = await this.client.send(command);

    return response.Tags?.find(tag => tag.Key === 'Service')?.Value;
  }

  private extractDetailTypes(eventPattern?: string): string[] {
    if (!eventPattern) return [];

    try {
      const pattern = JSON.parse(eventPattern);
      const detailType = pattern['detail-type'];

      if (Array.isArray(detailType)) {
        return detailType.filter(item => typeof item === 'string');
      }
      if (typeof detailType === 'string') {
        return [detailType];
      }
      return [];
    } catch {
      return [];
    }
  }
}

Rules without a Service tag are skipped with a warning. This helps identify rules that haven't adopted the tagging convention yet.

Generating the Service Graph

Now we combine producer and consumer data to generate a visual graph. The ServiceGraphGenerator code creates nodes for each service and directed edges based on shared detail-types.

export interface ServiceNode {
  id: string;
  name: string;
  producedEvents: string[];
  consumedEvents: string[];
}

export interface ServiceEdge {
  from: string;
  to: string;
  detailTypes: string[];
}

export interface ServiceGraph {
  nodes: ServiceNode[];
  edges: ServiceEdge[];
}

export class ServiceGraphGenerator {
  private readonly producers: ProducerRecord[];
  private readonly consumers: ConsumerRecord[];

  constructor(producers: ProducerRecord[], consumers: ConsumerRecord[]) {
    this.producers = producers;
    this.consumers = consumers;
  }

  generateGraph(): ServiceGraph {
    const services = new Map<string, { produces: Set<string>; consumes: Set<string> }>();
    const eventTypes = new Map<string, { producers: Set<string>; consumers: Set<string> }>();

    // Process producers
    for (const producer of this.producers) {
      if (!services.has(producer.source)) {
        services.set(producer.source, { produces: new Set(), consumes: new Set() });
      }
      services.get(producer.source)!.produces.add(producer.detailType);

      if (!eventTypes.has(producer.detailType)) {
        eventTypes.set(producer.detailType, { producers: new Set(), consumers: new Set() });
      }
      eventTypes.get(producer.detailType)!.producers.add(producer.source);
    }

    // Process consumers
    for (const consumer of this.consumers) {
      if (!services.has(consumer.serviceName)) {
        services.set(consumer.serviceName, { produces: new Set(), consumes: new Set() });
      }
      services.get(consumer.serviceName)!.consumes.add(consumer.detailType);

      if (!eventTypes.has(consumer.detailType)) {
        eventTypes.set(consumer.detailType, { producers: new Set(), consumers: new Set() });
      }
      eventTypes.get(consumer.detailType)!.consumers.add(consumer.serviceName);
    }

    // Generate nodes
    const nodes: ServiceNode[] = Array.from(services.entries()).map(([name, data]) => ({
      id: name,
      name,
      producedEvents: Array.from(data.produces).sort(),
      consumedEvents: Array.from(data.consumes).sort(),
    }));

    // Generate edges (aggregate multiple detail-types per service pair)
    const edgeMap = new Map<string, Set<string>>();
    for (const [detailType, data] of eventTypes) {
      for (const producer of data.producers) {
        for (const consumer of data.consumers) {
          if (producer === consumer) continue; // Skip self-loops

          const edgeKey = `${producer}|${consumer}`;
          if (!edgeMap.has(edgeKey)) edgeMap.set(edgeKey, new Set());
          edgeMap.get(edgeKey)!.add(detailType);
        }
      }
    }

    const edges: ServiceEdge[] = Array.from(edgeMap.entries()).map(([key, types]) => {
      const [from, to] = key.split('|');
      return { from: from!, to: to!, detailTypes: Array.from(types).sort() };
    });

    return { nodes: nodes.sort((a, b) => a.name.localeCompare(b.name)), edges };
  }

  toMermaid(): string {
    const graph = this.generateGraph();
    const lines: string[] = ['graph LR'];

    for (const node of graph.nodes) {
      const nodeId = node.id.replace(/-/g, '_');
      lines.push(`    ${nodeId}["${node.name}"]`);
    }

    for (const edge of graph.edges) {
      const fromId = edge.from.replace(/-/g, '_');
      const toId = edge.to.replace(/-/g, '_');
      const label = edge.detailTypes.join(', ');
      lines.push(`    ${fromId} -->|"${label}"| ${toId}`);
    }

    return lines.join('\n');
  }

  toDot(): string {
    const graph = this.generateGraph();
    const lines: string[] = [
      'digraph ServiceGraph {',
      '    rankdir=LR;',
      '    node [shape=box, style=rounded];',
      '',
    ];

    for (const node of graph.nodes) {
      const nodeId = node.id.replace(/-/g, '_');
      lines.push(`    ${nodeId} [label="${node.name}"];`);
    }

    lines.push('');

    for (const edge of graph.edges) {
      const fromId = edge.from.replace(/-/g, '_');
      const toId = edge.to.replace(/-/g, '_');
      const label = edge.detailTypes.join('\\n');
      lines.push(`    ${fromId} -> ${toId} [label="${label}"];`);
    }

    lines.push('}');
    return lines.join('\n');
  }
}

Example Output

For an e-commerce system with five services (order, payment, inventory, shipping, notification), the generated Mermaid diagram looks like this:

graph LR
    inventory_service["inventory-service"]
    notification_service["notification-service"]
    order_service["order-service"]
    payment_service["payment-service"]
    shipping_service["shipping-service"]
    inventory_service -->|"InventoryReserved"| order_service
    inventory_service -->|"InventoryReserved"| shipping_service
    order_service -->|"OrderCancelled, OrderCreated"| inventory_service
    order_service -->|"OrderCancelled, OrderCreated"| payment_service
    order_service -->|"OrderCreated"| notification_service
    payment_service -->|"PaymentCompleted"| notification_service
    payment_service -->|"PaymentCompleted"| order_service
    payment_service -->|"PaymentCompleted"| shipping_service
    payment_service -->|"PaymentFailed"| inventory_service
    shipping_service -->|"ShipmentDispatched"| notification_service

The graph clearly shows:

Order service produces events consumed by payment, inventory, and notification services
Payment service produces events consumed by order, shipping, inventory, and notification services
The notification service is a pure consumer—it doesn't produce events that other services consume

Putting It All Together

Here's how to use these components together:

async function generateServiceGraph() {
  const registryName = 'discovered-schemas';
  const eventBusName = 'default';

  // Discover producers from Schema Registry
  const producerDiscovery = new ProducerDiscovery(registryName);
  const producers = await producerDiscovery.discover();

  // Discover consumers from EventBridge Rules
  const consumerDiscovery = new ConsumerDiscovery(eventBusName);
  const consumers = await consumerDiscovery.discover();

  // Generate the graph
  const generator = new ServiceGraphGenerator(producers, consumers);

  // Output as Mermaid
  console.log(generator.toMermaid());

  // Or output as Graphviz DOT
  console.log(generator.toDot());
}

Conclusion

It is a relatively low lift effort to be able to understand service relationships in event-driven architectures using EventBridge. By combining EventBridge Schema Registry for producer discovery with a simple tagging convention for consumer discovery, you can automatically generate accurate visualizations of how events flow through your system.

The code in this article is available as a complete, deployable CDK application.

AWS AppConfig multi-accounts, multi-environments with ease

Roger Chi — Mon, 27 Jan 2025 16:40:29 +0000

AppConfig is AWS's native solution for feature flagging and dynamic configuration management. It provides a comparable feature set to third party feature flagging tools such as LaunchDarkly or StatSig. Coming from an organization that used these third party tools, there were some hurdles that I encountered when implementing AppConfig for our applications with the same ease as these third party tools. In this article, I will describe these challenges and our solutions to them. Specifically, I will cover these topics:

AWS Organizations setup with an AppConfig shared service account
Configuring IAM roles and Systems Manager Parameters to simplify application access to the correct environments configured in AppConfig
Setting up AppConfig Agent with Lambda functions, including Docker Image Functions

Authorization and access compared to third party tools

If you're familiar with using a feature flagging tool such as LaunchDarkly, you will be used to using API keys that are associated with specific environments. This abstracts the association between an environment (a set of flags/configs tied to a deployment environment, such as dev, staging, or prod) and the application. The application client just needs to be configured with the correct API key, and will pull flags that are specific to that environment without needing to be explicitly configured with information about that environment.

With AppConfig, there is no concept of API keys. All access to the service is through the AWS native IAM integration. So the question is how best to set up AppConfig and the IAM roles allowed to access it in a multi-account, multi-environment AWS Organization?

AWS Organization setup

The structure we settled on is:

A separate, shared services AWS account dedicated to AppConfig
IAM roles specific to an AppConfig "Application"
For each Application and Environment within that application, an IAM role that grants access to get configurations specific to that environment
SSM Systems manager parameters for each Application deployed to each workload account that specifies the role to assume and the environment name
AWS IAM Identity Center permission set granting access to the AppConfig account

CloudFormation templates

To implement this structure, we'll need two CloudFormation templates:

A template for the AppConfig account that creates the application, environments, and cross-account IAM roles
A template for the workload accounts (dev/prod) that creates the SSM parameters

AppConfig Account Template

Deploy this template to your central AppConfig account. It creates an AppConfig application with dev and prod environments, plus the IAM roles that will be assumed by applications in the dev and prod OUs. The AssumeRolePolicyDocument can be modified to support your specific AWS Organization structure (for example, granting access to a specific account, etc.).

The IAM roles created in the AppConfig account need policies that grant access to the appropriate AppConfig resources. The policy below shows the permissions needed:

Basic AppConfig read operations like GetConfiguration and StartConfigurationSession scoped to the specific environment
Additional list/describe operations needed by the AppConfig Agent to discover configurations and lookup ids based on names.

# Deploy this template to the AppConfig account
Parameters:
  ApplicationName:
    Type: String
    Description: Name of the AppConfig application
    Default: myapp

  prodOuId:
    Type: String
    Description: ID of the Production OU

  sdlcOuId:
    Type: String
    Description: ID of the SDLC OU

  sandboxOuId:
    Type: String
    Description: ID of the Sandbox OU

  organizationPrincipalId:
    Type: String
    Description: ID of the Organization Principal

  organizationRootOuId:
    Type: String
    Description: ID of the Organization Root OU

Resources:
  Application:
    Type: AWS::AppConfig::Application
    Properties:
      Name: !Ref ApplicationName
      Description: 'Feature flags and configuration for applications'

  ProdEnvironment:
    Type: AWS::AppConfig::Environment
    Properties:
      ApplicationId: !Ref Application
      Name: prod
      Description: 'Production environment'

  DevEnvironment:
    Type: AWS::AppConfig::Environment
    Properties:
      ApplicationId: !Ref Application
      Name: dev
      Description: 'Development environment'

  ProdAccessRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub 'appconfig-${ApplicationName}-prod'
      Description: !Sub 'Cross-account access role for ${ApplicationName} Production AppConfig environment'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              AWS: '*'
            Condition:
              'ForAnyValue:StringLike':
                'aws:PrincipalOrgPaths': !Sub '${organizationPrincipalId}/${organizationRootOuId}/${prodOuId}/*'
      Policies:
        - PolicyName: AppConfigAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - appconfig:GetConfiguration
                  - appconfig:StartConfigurationSession
                  - appconfig:GetLatestConfiguration
                Resource: !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/environment/${ProdEnvironment}/configuration/*'
              - Effect: Allow
                Action:
                  - appconfig:GetApplication
                  - appconfig:GetEnvironment
                  - appconfig:GetConfigurationProfile
                  - appconfig:ListApplications
                  - appconfig:ListConfigurationProfiles
                  - appconfig:ListEnvironments
                  - appconfig:ListHostedConfigurationVersions
                Resource: 
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}'
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/environment/*'
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/configurationprofile/*'

  DevAccessRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub 'appconfig-${ApplicationName}-dev'
      Description: !Sub 'Cross-account access role for ${ApplicationName} Development AppConfig environment'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              AWS: '*'
            Condition:
              'ForAnyValue:StringLike':
                'aws:PrincipalOrgPaths': 
                  - !Sub '${organizationPrincipalId}/${organizationRootOuId}/${sdlcOuId}/*'
                  - !Sub '${organizationPrincipalId}/${organizationRootOuId}/${sandboxOuId}/*'
      Policies:
        - PolicyName: AppConfigAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - appconfig:GetConfiguration
                  - appconfig:StartConfigurationSession
                  - appconfig:GetLatestConfiguration
                Resource: !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/environment/${DevEnvironment}/configuration/*'
              - Effect: Allow
                Action:
                  - appconfig:GetApplication
                  - appconfig:GetEnvironment
                  - appconfig:GetConfigurationProfile
                  - appconfig:ListApplications
                  - appconfig:ListConfigurationProfiles
                  - appconfig:ListEnvironments
                  - appconfig:ListHostedConfigurationVersions
                Resource: 
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}'
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/environment/*'
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/configurationprofile/*'

Outputs:
  ApplicationId:
    Description: AppConfig Application ID
    Value: !Ref Application

  ProdEnvironmentId:
    Description: Production Environment ID
    Value: !Ref ProdEnvironment

  DevEnvironmentId:
    Description: Development Environment ID
    Value: !Ref DevEnvironment

Workload Account Template

Deploy this template to each of your workload accounts (development and production). It creates SSM parameters that applications will use to discover the correct IAM role to assume and environment ID to use.

# Deploy this template to each workload account (dev/prod)
Parameters:
  ApplicationName:
    Type: String
    Description: Name of the AppConfig application
    Default: myapp

  AppConfigAccountId:
    Type: String
    Description: AWS Account ID where AppConfig is deployed

  Environment:
    Type: String
    Description: Environment name (dev or prod)
    AllowedValues: [dev, prod]

Resources:
  AppConfigRoleArnParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub '/appconfig/${ApplicationName}/role-arn'
      Type: String
      Value: !Sub 'arn:aws:iam::${AppConfigAccountId}:role/appconfig-${ApplicationName}-${Environment}'
      Description: !Sub 'Cross-account role ARN for ${ApplicationName} AppConfig access'

  AppConfigEnvironmentNameParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub '/appconfig/${ApplicationName}/environment-name'
      Type: String
      Value: !Ref Environment
      Description: !Sub 'AppConfig Environment Name for ${ApplicationName} ${Environment} environment'

When deploying these templates:

First deploy the AppConfig account template
Then deploy the workload account template to each account, using:
- The appropriate Environment value (dev/prod)
- The AppConfigAccountId where you deployed the first template

This setup provides a consistent way for applications to discover and access their AppConfig configuration, regardless of which account or environment they're running in. In the next section, we'll look at how applications can use these SSM parameters to access their configuration.

Setting up AppConfig Agent for Lambda functions

CDK code

const APP_NAME = 'myapp';

export function enableAppConfig(fn: LambdaFunction) {
  const appConfigRoleArn = StringParameter.fromStringParameterName(
    fn,
    'AppConfigRoleArn',
    `/appconfig/${APP_NAME}/role-arn`,
  ).stringValue;
  const appConfigEnvironment = StringParameter.fromStringParameterName(
    fn,
    'AppConfigEnvironment',
    `/appconfig/${APP_NAME}/environment-name`,
  ).stringValue;
  const appConfigRole = Role.fromRoleArn(fn, 'AppConfigRole', appConfigRoleArn);
  fn.addEnvironment('AWS_APPCONFIG_EXTENSION_ROLE_ARN', appConfigRoleArn);
  fn.addEnvironment('APPCONFIG_ENVIRONMENT', appConfigEnvironment);
  if (!fn.role) {
    throw new Error('Lambda role is undefined');
  }
  appConfigRole.grantAssumeRole(fn.role);
}

export function enableAppConfigLayer(fn: LambdaFunction) {
  let appConfigLayer: ILayerVersion;
  if (fn.architecture === Architecture.ARM_64) {
    appConfigLayer = LayerVersion.fromLayerVersionArn(
      fn,
      'AppConfigLayer',
      'arn:aws:lambda:us-west-2:359756378197:layer:AWS-AppConfig-Extension-Arm64:131',
    );
  } else {
    appConfigLayer = LayerVersion.fromLayerVersionArn(
      fn,
      'AppConfigLayer',
      'arn:aws:lambda:us-west-2:359756378197:layer:AWS-AppConfig-Extension:229',
    );
  }
  fn.addLayers(appConfigLayer);
}

export class DockerImageFunction extends CdkDockerImageFunction {
  constructor(scope: Construct, id: string, props: DockerImageFunctionProps) {
    super(scope, id, props);
    enableAppConfig(this);
  }
}

export class NodejsFunction extends CdkNodejsFunction {
  constructor(scope: Construct, id: string, props: NodejsFunctionProps) {
    super(scope, id, props);
    enableAppConfig(this);
    enableAppConfigLayer(this);
  }
}

export class PythonFunction extends CdkPythonFunction {
  constructor(scope: Construct, id: string, props: PythonFunctionProps) {
    super(scope, id, props);
    enableAppConfig(this);
    enableAppConfigLayer(this);
  }
}

Docker Image Functions

Docker image functions can't use the AppConfig Agent Lambda Layer, so we need to build our own. This Dockerfile builds a base image with the AppConfig extension, and then builds the application on top of that. The extension is downloaded from the AWS public registry, and then installed into the /opt/extensions directory. The application is then built on top of that base image. The access tokens are only used in the first stage, so they are not included in the final image.

# First stage to fetch AppConfig extension
FROM amazon/aws-cli:latest as appconfig-extension

ARG INCLUDE_APPCONFIG=0
ARG AWS_DEFAULT_REGION
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_SESSION_TOKEN

ENV AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
ENV AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}

# Install unzip before trying to use it
RUN yum install -y unzip

# Create directory and download extension if enabled
RUN mkdir -p /opt/extensions && \
    if [ "$INCLUDE_APPCONFIG" = "1" ]; then \
        aws lambda get-layer-version-by-arn \
        --arn arn:aws:lambda:us-west-2:359756378197:layer:AWS-AppConfig-Extension:229 \
        --query 'Content.Location' --output text | xargs curl -o layer.zip && \
        unzip -j layer.zip "extensions/*" -d /opt/extensions && \
        rm layer.zip; \
    fi

# Main build stage
FROM python:3.13-slim as base

# Copy the AppConfig extension from the first stage
RUN mkdir -p /opt/extensions
COPY --from=appconfig-extension /opt/extensions /opt/extensions
RUN ls -la /opt/extensions

Conclusion

This architecture provides several key benefits for managing feature flags and configuration across a multi-account AWS organization:

Centralized Management: By hosting AppConfig in a dedicated account, we create a single source of truth for all feature flags and configuration. This makes it easier to manage and audit configuration changes across environments.
Secure Access Control: Using AWS Organizations and OU-based IAM roles ensures that:
- Production applications can only access production configurations
- Development applications can only access development configurations
- Access is automatically granted to new accounts added to the appropriate OUs
Simple Application Integration:
- Applications only need to know their application name to discover the appropriate role and environment
- The same application code works across all environments without environment-specific configuration
- SSM Parameters provide a consistent interface for applications to discover their AppConfig settings
Lambda Integration: The provided CDK constructs make it easy to add AppConfig support to Lambda functions:
- Automatic layer addition for Node.js and Python functions
- Docker image support with a base image that includes the AppConfig extension
- Environment-aware configuration without code changes

This solution achieves the same simplicity of third-party feature flag services while maintaining the benefits of AWS's native services and security model. Teams can deploy applications across accounts and environments without worrying about AppConfig access configuration, while security teams can maintain strict access controls through AWS Organizations.

Bonus, for users of org-formation-cli

Tasks file:

Parameters:
  <<: !Include '../../_parameters.yml'

  appName:
    Type: String
    Default: 'myapp'

AppConfigMyApp:
  Type: update-stacks
  Template: ./application.yml
  StackName: !Sub '${resourcePrefix}-appconfig-${appName}'
  StackDescription: !Sub 'AppConfig Application, Environments, Roles and Parameters for ${appName}'
  TerminationProtection: false
  DefaultOrganizationBindingRegion: 'us-west-2'
  OrganizationBindings:
    ApplicationBinding:
      Account: !Ref AppConfigAccount
    ProdParametersBinding:
      IncludeMasterAccount: false
      OrganizationalUnit:
        - !Ref ProdOu
      Region: 'us-west-2'
    DevParametersBinding:
      IncludeMasterAccount: false
      OrganizationalUnit:
        - !Ref SdlcOu
        - !Ref SandboxOu
      Region: 'us-west-2'
  Parameters:
    applicationName: !Ref appName
    prodOuId: !Ref ProdOu
    sdlcOuId: !Ref SdlcOu
    sandboxOuId: !Ref SandboxOu
    organizationPrincipalId: !Ref organizationPrincipalId
    organizationRootOuId: !Ref organizationRootOuId
    appConfigAccountId: !Ref AppConfigAccount

application.yml:

Parameters:
  applicationName:
    Type: String
    Description: Name of the AppConfig application

  prodOuId:
    Type: String
    Description: ID of the Production OU

  sdlcOuId:
    Type: String
    Description: ID of the SDLC OU

  sandboxOuId:
    Type: String
    Description: ID of the Sandbox OU

  organizationPrincipalId:
    Type: String
    Description: ID of the Organization Principal

  organizationRootOuId:
    Type: String
    Description: ID of the Organization Root OU

  appConfigAccountId:
    Type: String
    Description: ID of the AppConfig account

Resources:
  # AppConfig Resources - Created in AppConfig Account
  Application:
    Type: AWS::AppConfig::Application
    OrganizationBinding: !Ref ApplicationBinding
    Properties:
      Name: !Ref applicationName
      Description: 'Feature flags and configuration for applications'

  ProdEnvironment:
    Type: AWS::AppConfig::Environment
    OrganizationBinding: !Ref ApplicationBinding
    Properties:
      ApplicationId: !Ref Application
      Name: prod
      Description: 'Production environment'

  DevEnvironment:
    Type: AWS::AppConfig::Environment
    OrganizationBinding: !Ref ApplicationBinding
    Properties:
      ApplicationId: !Ref Application
      Name: dev
      Description: 'Development environment'

  ProdAccessRole:
    Type: AWS::IAM::Role
    OrganizationBinding: !Ref ApplicationBinding
    Properties:
      RoleName: !Sub 'appconfig-${applicationName}-prod'
      Description: !Sub 'Cross-account access role for ${applicationName} Production AppConfig environment'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              AWS: '*'
            Condition:
              'ForAnyValue:StringLike':
                'aws:PrincipalOrgPaths': !Sub '${organizationPrincipalId}/${organizationRootOuId}/${prodOuId}/*'
      Policies:
        - PolicyName: AppConfigAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - appconfig:GetConfiguration
                  - appconfig:StartConfigurationSession
                  - appconfig:GetLatestConfiguration
                Resource: !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/environment/${ProdEnvironment}/configuration/*'
              - Effect: Allow
                Action:
                  - appconfig:GetApplication
                  - appconfig:GetEnvironment
                  - appconfig:GetConfigurationProfile
                  - appconfig:ListApplications
                  - appconfig:ListConfigurationProfiles
                  - appconfig:ListEnvironments
                  - appconfig:ListHostedConfigurationVersions
                Resource: 
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}'
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/environment/*'
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/configurationprofile/*'

  DevAccessRole:
    Type: AWS::IAM::Role
    OrganizationBinding: !Ref ApplicationBinding
    Properties:
      RoleName: !Sub 'appconfig-${applicationName}-dev'
      Description: !Sub 'Cross-account access role for ${applicationName} Development AppConfig environment'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              AWS: '*'
            Condition:
              'ForAnyValue:StringLike':
                'aws:PrincipalOrgPaths': 
                  - !Sub '${organizationPrincipalId}/${organizationRootOuId}/${sdlcOuId}/*'
                  - !Sub '${organizationPrincipalId}/${organizationRootOuId}/${sandboxOuId}/*'
      Policies:
        - PolicyName: AppConfigAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - appconfig:GetConfiguration
                  - appconfig:StartConfigurationSession
                  - appconfig:GetLatestConfiguration
                Resource: !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/environment/${DevEnvironment}/configuration/*'
              - Effect: Allow
                Action:
                  - appconfig:GetApplication
                  - appconfig:GetEnvironment
                  - appconfig:GetConfigurationProfile
                  - appconfig:ListApplications
                  - appconfig:ListConfigurationProfiles
                  - appconfig:ListEnvironments
                  - appconfig:ListHostedConfigurationVersions
                Resource: 
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}'
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/environment/*'
                  - !Sub 'arn:aws:appconfig:*:${AWS::AccountId}:application/${Application}/configurationprofile/*'

  # SSM Parameters - Created in Production Accounts
  ProdAppConfigRoleParameter:
    Type: AWS::SSM::Parameter
    OrganizationBinding: !Ref ProdParametersBinding
    Properties:
      Name: !Sub '/appconfig/${applicationName}/role-name'
      Type: String
      Value: !Sub 'appconfig-${applicationName}-prod'
      Description: !Sub 'Cross-account role name for ${applicationName} AppConfig access'

  ProdAppConfigRoleArnParameter:
    Type: AWS::SSM::Parameter
    OrganizationBinding: !Ref ProdParametersBinding
    Properties:
      Name: !Sub '/appconfig/${applicationName}/role-arn'
      Type: String
      Value: !Sub 'arn:aws:iam::${appConfigAccountId}:role/appconfig-${applicationName}-prod'
      Description: !Sub 'Cross-account role ARN for ${applicationName} AppConfig access'

  ProdAppConfigEnvironmentParameter:
    Type: AWS::SSM::Parameter
    OrganizationBinding: !Ref ProdParametersBinding
    Properties:
      Name: !Sub '/appconfig/${applicationName}/environment-id'
      Type: String
      Value: !Ref ProdEnvironment
      Description: !Sub 'AppConfig Environment ID for ${applicationName} Production environment'

  ProdAppConfigEnvironmentNameParameter:
    Type: AWS::SSM::Parameter
    OrganizationBinding: !Ref ProdParametersBinding
    Properties:
      Name: !Sub '/appconfig/${applicationName}/environment-name'
      Type: String
      Value: 'prod'
      Description: !Sub 'AppConfig Environment Name for ${applicationName} Production environment'

  # SSM Parameters - Created in Development Accounts
  DevAppConfigRoleParameter:
    Type: AWS::SSM::Parameter
    OrganizationBinding: !Ref DevParametersBinding
    Properties:
      Name: !Sub '/appconfig/${applicationName}/role-name'
      Type: String
      Value: !Sub 'appconfig-${applicationName}-dev'
      Description: !Sub 'Cross-account role name for ${applicationName} AppConfig access'

  DevAppConfigRoleArnParameter:
    Type: AWS::SSM::Parameter
    OrganizationBinding: !Ref DevParametersBinding
    Properties:
      Name: !Sub '/appconfig/${applicationName}/role-arn'
      Type: String
      Value: !Sub 'arn:aws:iam::${appConfigAccountId}:role/appconfig-${applicationName}-dev'
      Description: !Sub 'Cross-account role ARN for ${applicationName} AppConfig access'

  DevAppConfigEnvironmentParameter:
    Type: AWS::SSM::Parameter
    OrganizationBinding: !Ref DevParametersBinding
    Properties:
      Name: !Sub '/appconfig/${applicationName}/environment-id'
      Type: String
      Value: !Ref DevEnvironment
      Description: !Sub 'AppConfig Environment ID for ${applicationName} Development environment'

  DevAppConfigEnvironmentNameParameter:
    Type: AWS::SSM::Parameter
    OrganizationBinding: !Ref DevParametersBinding
    Properties:
      Name: !Sub '/appconfig/${applicationName}/environment-name'
      Type: String
      Value: 'dev'
      Description: !Sub 'AppConfig Environment Name for ${applicationName} Development environment'

Outputs:
  ApplicationId:
    Description: AppConfig Application ID
    Value: !Ref Application
    Export:
      Name: !Sub '${AWS::StackName}-application-id'

  ProdEnvironmentId:
    Description: Production Environment ID
    Value: !Ref ProdEnvironment
    Export:
      Name: !Sub '${AWS::StackName}-prod-env-id'

  DevEnvironmentId:
    Description: Development Environment ID
    Value: !Ref DevEnvironment
    Export:
      Name: !Sub '${AWS::StackName}-dev-env-id'

  ProdAccessRoleArn:
    Description: ARN of the Production access role
    Value: !GetAtt ProdAccessRole.Arn
    Export:
      Name: !Sub '${AWS::StackName}-prod-role-arn'

  DevAccessRoleArn:
    Description: ARN of the SDLC access role
    Value: !GetAtt DevAccessRole.Arn
    Export:
      Name: !Sub '${AWS::StackName}-dev-role-arn'

Practical DynamoDB - Locking Reads

Roger Chi — Thu, 05 Dec 2024 16:37:57 +0000

Amazon's DynamoDB is promoted as a zero-maintenance, virtually unlimited throughput and scale* NoSQL database. Because of its' very low administrative overhead and serverless billing model, it has long been my data store of choice when developing cloud applications and services.

The Challenge of Concurrent Access

Event-driven architectures commonly built on AWS services like Lambda, EventBridge, and Step Functions frequently use DynamoDB as their primary data store. In these systems, multiple serverless functions or processes can be triggered simultaneously by events, and they could be attempting to access and modify the same DynamoDB items. This creates unique concurrency challenges.

Consider these common scenarios:

Multiple Lambda functions responding to an SQS queue of orders, all trying to update the same inventory record
EventBridge rules triggering parallel processes that need to modify shared configuration data
Step Functions running concurrent workflows that interact with the same customer record
API Gateway endpoints receiving near-simultaneous requests to update a user's status

Traditional applications might handle these scenarios through application-layer coordination or database transaction isolation levels. However, serverless event-driven systems require different approaches due to their distributed nature.

There are three broad approaches to handling concurrency in these situations:

No locking: Ideal if you don't need to read an item's contents before updating it. You might only use an update condition ensuring the last updated time is less than the event time to prevent out-of-order processing.
Optimistic locking: you read the item and only update it if the last updated timestamp or version remains the same as when you read it initially. This is useful for cases where conflicts may be rare, or the cost of retrying is low.
Pessimistic locking (the focus of this article): you acquire an exclusive lock on the item prior to any processing, preventing any other concurrent processes from beginning their work. You release the lock after performing your work and as you update the item.

Generally, any time you need strict consistency when reading before writing in a distributed system, you must use some form of concurrency control mechanism.

Understanding Lock Management Through Condition Expressions

DynamoDB's condition expressions provide a powerful way to implement self-managing pessimistic locks. Let's explore how this works by examining our lock attributes:

{
    id: "item-123",          // Primary key
    data: { ... },           // Your actual item data
    lockTime: 1635789600,    // Unix timestamp when lock expires
    lockedBy: "process-456"  // Identifier of the locking process
}

The magic of this implementation lies in how we use DynamoDB's condition expressions to manage lock acquisition and expiration automatically. Let's look at how this works in practice:

Decision Flow Diagram

DynamoDB's Serial Write Guarantee

One of DynamoDB's key characteristics that makes this locking pattern reliable is its guarantee of serial writes at the item level. When multiple processes attempt to write to the same item simultaneously, DynamoDB processes these writes one at a time in the order they are received.

Let's understand this through an example. Imagine three processes attempting to acquire a lock on the same item at almost the same time:

// All three processes execute this nearly simultaneously
const params = {
  TableName: 'MyTable',
  Key: { id: 'item-123' },
  UpdateExpression: 'SET lockTime = :lockTime, lockedBy = :processId',
  ConditionExpression: 'attribute_not_exists(lockTime) OR lockTime < :now',
  ExpressionAttributeValues: {
    ':lockTime': lockExpiration,
    ':processId': processId,
    ':now': now,
  },
};

DynamoDB will handle these requests serially, meaning:

The first request to reach DynamoDB will be evaluated completely
If it succeeds, it will acquire the lock and update the item
Only then will DynamoDB evaluate the second request
The second request will fail because the lock now exists and isn't expired
The third request will similarly fail for the same reason

This serial processing means we don't need additional synchronization mechanisms beyond DynamoDB's condition expressions. There's no possibility of a "race condition" where two processes think they've acquired the lock simultaneously, because DynamoDB's serial write guarantee prevents this scenario.

This behavior complements our locking pattern in several ways:

Lock acquisition is guaranteed to be exclusive because of serial write processing
We don't need distributed coordination or consensus protocols
The system naturally handles contention through DynamoDB's built-in request queuing
Failed condition checks happen quickly, allowing processes to retry or move on

When combined with condition expressions and atomic updates, this serial write behavior creates a foundation for building reliable distributed primitives like our locking system. It's worth noting that while writes are serial, reads can happen concurrently - which is why our pattern always uses write operations (UpdateItem) even when reading data, to ensure we're part of the serial write queue.

Atomic Lock Acquisition

async function acquireLockAndRead(itemId, processId) {
  const lockDuration = 30; // seconds
  const now = Math.floor(Date.now() / 1000);
  const lockExpiration = now + lockDuration;

  // Create the update command object for lock acquisition
  const command = new UpdateItemCommand({
    TableName: 'MyTable',
    Key: { id: { S: itemId } }, // Note: v3 SDK requires explicit AttributeValue types
    UpdateExpression: 'SET lockTime = :lockTime, lockedBy = :processId',
    ConditionExpression: 'attribute_not_exists(lockTime) OR lockTime < :now',
    ExpressionAttributeValues: {
      ':lockTime': { N: lockExpiration.toString() }, // Numbers must be strings in v3
      ':processId': { S: processId },
      ':now': { N: now.toString() },
    },
    ReturnValues: 'ALL_NEW',
  });

  try {
    // Send the command using the DynamoDB client
    const result = await client.send(command);
    return result.Attributes;
  } catch (error) {
    if (error.name === 'ConditionalCheckFailedException') {
      throw new Error(
        'Failed to acquire lock - item is locked by another process',
      );
    }
    throw error;
  }
}

The condition expression makes this function powerful and elegant. It allows us to acquire a lock in two scenarios:

When no lock exists (attribute_not_exists(lockTime))
When an existing lock has expired (lockTime < :now)

By using ReturnValues: 'ALL_NEW', we get both the lock and the item data in a single atomic operation. This eliminates the need for an additional read after acquiring the lock. This saves on costs (additional RCUs), complexity (you would want a consistent read if not returning the values, and an additional read means this whole process is three API operations instead of two, a 50% increase), and total processing time (the throughput of this system is highly dependent on the total time it takes between the initial lock and the releasing update).

Protected Updates with Lock Verification

When updating data under a lock, we verify that we still hold a valid lock:

async function updateAndReleaseLock(itemId, processId, updateData) {
  const now = Math.floor(Date.now() / 1000);

  // Create the update command object for the protected update
  const command = new UpdateItemCommand({
    TableName: 'MyTable',
    Key: { id: { S: itemId } },
    UpdateExpression: 'SET #data = :newData REMOVE lockTime, lockedBy',
    ConditionExpression: 'lockedBy = :processId AND lockTime > :now',
    ExpressionAttributeNames: {
      '#data': 'data',
    },
    ExpressionAttributeValues: {
      ':newData': marshall(updateData), // Use marshall utility for complex objects
      ':processId': { S: processId },
      ':now': { N: now.toString() },
    },
    ReturnValues: 'ALL_NEW',
  });

  const result = await client.send(command);
}

Lock Lifecycle Example

Please see this repo for a complete example of this pattern: rogerchi/ddb-locking-read

Conclusion

By leveraging DynamoDB's condition expressions, we can create an elegant, self-managing locking system that requires no external cleanup or maintenance. This pattern showcases how DynamoDB's features can be composed to create robust distributed systems primitives.

If you liked this article, consider following me on Bluesky: @rogerchi.com

A Mental Model for DynamoDB Write Limits

Roger Chi — Mon, 04 Mar 2024 02:57:21 +0000

When working with any tool, it is important to understand the limits involved to be able to create a mental model that can be used to most effectively plan and implement the usage of that tool. It might seem easy to think that because DynamoDB bills itself as "unlimited throughput and scale" you do not need to think about limitations on write and read throughput. However, as we will explore in this post, there are actually very well defined limits that we need to take into account when implementing solutions with DynamoDB.

Partition limits

Amazon documents these partition level limits: 3,000 RCU/s and 1,000 WCU/s. Let's verify these limits and put them to the test. We want to understand what these limits are for an UpdateItem command and if the limits are any different for direct PutItem commands.

Test setup

CDK code can be found here: https://github.com/rogerchi/ddb-writes-model

We create Items of a predefined size by filling them with random binary data. We then set up lambda functions which will spam our DynamoDB Item with either 10,000 atomic updates, or 10,000 full puts. We set up a stream handler to capture the times of the first and last updates to get an idea of how much time it takes to complete these 10,000 operations.

`UpdateItem` code

We first initialize an Item with a count of 0 and then run this lambda function against it.

async function updateItem(id: string) {
  const params = {
    TableName: TABLE_NAME,
    Key: {
      pk: { S: id },
      sk: { S: 'ITEM' },
    },
    UpdateExpression: 'set #count = #count + :one',
    ExpressionAttributeNames: {
      '#count': 'count',
    },
    ExpressionAttributeValues: {
      ':one': { N: '1' },
    },
  };

  let attempt = 0;
  while (true) {
    try {
      const command = new UpdateItemCommand(params);
      await client.send(command);
      return;
    } catch (error) {
      attempt++;
      console.error(`Update failed for item ${id}. Attempt ${attempt}.`);
    }
  }
}

interface Event {
  id: string;
}

exports.handler = async (event: Event) => {
  const id = event.id;
  if (!id) {
    throw new Error('Event does not contain an ID.');
  }

  const totalUpdates = 10000;
  const commands = [];

  for (let i = 0; i < totalUpdates; i++) {
    commands.push(updateItem(id));
  }

  await Promise.all(commands);

  console.log('All updates complete.');
};

`UpdateItem` results (10,000 updates)

Item Size (bytes)	Total time (seconds)	Op/s	WCU/s
1000	10	1000	1000
2000	20	500	1000
4000	40	250	1000

As we can see, the results match the expected write throughput described in the documentation. Does the throughput change if we are using the PutItem instead of the UpdateItem command, since the underlying service doesn't need to retrieve an existing item to update it?

`PutItem` code

async function putItem(id: string, itemBody: any, count: number) {
  const updatedItem = { ...itemBody, count: { N: `${count}` } };
  const params = {
    TableName: TABLE_NAME,
    Item: updatedItem,
  };

  let attempt = 0;
  while (true) {
    try {
      const command = new PutItemCommand(params);
      await client.send(command);
      return;
    } catch (error) {
      attempt++;
      console.error(`Put failed for item ${id}. Attempt ${attempt}.`, error);
    }
  }
}

interface Event {
  id: string;
  size: number;
}

exports.handler = async (event: Event) => {
  const { id, size } = event;

  const data = generateData(id, size);
  const item = marshall({
    pk: id,
    sk: 'PUTS',
    size,
    data: Uint8Array.from(data),
  });

  const totalUpdates = 10000;
  const commands = [];

  for (let i = 0; i < totalUpdates; i++) {
    commands.push(putItem(id, item, i));
  }

  await Promise.all(commands);

  console.log('All updates complete.');
};

`PutItem` results

Item Size (bytes)	Total time (seconds)	Op/s	WCU/s
1000	10	1000	1000
2000	20	500	1000
4000	40	250	1000

Our PutItem results are identical to our UpdateItem results, and conform with the idea that the maximum write throughput for an Item is 1,000 WCU/s.

`TransactWriteItem`

TransactWriteItem costs two WCU per 1KB to write the Item to the table. One WCU to prepare the transaction, and one to commit the transaction. You might assume that for a 1KB Item, given what we learned above, we will be able to perform 500 transactions per second. Does that prove out?

Code

async function transactUpdateItem(id: string) {
  const params = {
    TableName: TABLE_NAME,
    Key: {
      pk: { S: id },
      sk: { S: 'TRXS' },
    },
    UpdateExpression: 'set #count = #count + :one',
    ExpressionAttributeNames: {
      '#count': 'count',
    },
    ExpressionAttributeValues: {
      ':one': { N: '1' },
    },
  };

  let attempt = 0;
  while (true) {
    try {
      const command = new TransactWriteItemsCommand({
        TransactItems: [{ Update: { ...params } }],
      });
      await client.send(command);
      return;
    } catch (error) {
      attempt++;
      console.error(`Update failed for item ${id}. Attempt ${attempt}.`);
    }
  }
}

interface Event {
  id: string;
}

exports.handler = async (event: Event) => {
  const id = event.id;
  if (!id) {
    throw new Error('Event does not contain an ID.');
  }

  const totalUpdates = 10000;
  const commands = [];

  for (let i = 0; i < totalUpdates; i++) {
    commands.push(transactUpdateItem(id));
  }

  await Promise.all(commands);

  console.log('All updates complete.');
};

`TransactWriteItem` results

Item Size (bytes)	Total time (seconds)	Op/s	WCU/s
1000	60	167	333
2000	90	111	444
4000	130	77	615
8000	188	53	851
16000	370	27	864

We see from the TransactWriteItem results that we cannot achieve the full 1000 WCU/s through transactions, and that it seems like there is some fundamental overhead so that smaller items fall far from the 1000 WCU/s throughput and it's not until the item sizes get beyond 8KB that we get closer (but not quite reaching) the 1000 WCU/s partition throughput.

Our mental model for DynamoDB Writes

So what's our foundational mental model for DynamoDB write limits? You need to understand that writes target a specific Item at all times. This Item can have up to 1000 WCUs of Put or Update actions performed on it. This means an item of up to 1KB of size can be written or updated 1,000 times a second. An item of 2KB of size can be written or updated 500 times a second, and an item of 200KB of size can only be written or updated 5 times a second. To be able to exceed these limits, you must shard your writes. But because of instant adaptive capacity, you can think of each Item as being its own separate partition, literally a whole set of infrastructure dedicated to that single Item, which is pretty darn amazing. It's why there's no other service that shards quite like DynamoDB, and why it is the ultimate multi-tenant service, a real example of the power of scale in the cloud.

The caveat is that write throughput is significantly decreased with introducing transactions, from around 167 operations per second for a 1KB item to 27 operations per second for a 16KB item.

About me

I am a Staff Engineer @ Veho with a passion for serverless.

Step Functions Distributed Map and Athena, a match made in cloud heaven

Roger Chi — Sun, 21 May 2023 03:18:18 +0000

Step Functions released a new feature late last year called Distributed Map, which allows the service to coordinate parallel processing over huge datasets (millions of objects). Read more about it in the announcement blog here: https://aws.amazon.com/blogs/aws/step-functions-distributed-map-a-serverless-solution-for-large-scale-parallel-data-processing/

One of the input formats that can be used for the distributed map state is a .csv file in an S3 bucket. This opens up an opportunity to use the optimized Athena integration for Step Functions to generate the input .csv file that can be processed by the distributed map state.

Configuration

The distributed map state is configured to read a .csv file from S3 by using the ItemReader property:

  "ItemReader": {
    "Resource": "arn:aws:states:::s3:getObject",
    "ReaderConfig": {
      "InputType": "CSV",
      "CSVHeaderLocation": "FIRST_ROW"
    },
    "Parameters": {
      "Bucket.$": "$.CsvBucket",
      "Key.$": "$.CsvKey"
    }
  }

The Athena StartQueryExecution task is configured to output CsvBucket and CsvKey to the state:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::athena:startQueryExecution.sync",
  "Parameters": {
    "QueryString": "select * from mytable",
    "WorkGroup": "primary"
  },
  "ResultSelector": {
    "CsvBucket.$": "States.ArrayGetItem(States.StringSplit($.QueryExecution.ResultConfiguration.OutputLocation, ':/'), 1)",
    "CsvKey.$": "States.ArrayGetItem(States.StringSplit($.QueryExecution.ResultConfiguration.OutputLocation, ':/'), 2)"
  },
  "Next": "DistributedMapState"
}

The ResultSelector configuration takes the S3 URI that's in the $.QueryExecution.ResultConfiguration.OutputLocation path and uses intrinsic functions to parse out the Bucket and Key of the CSV result file. Note, this assumes the .csv output is at the root level of the query results bucket.

Example

Here is a simple example of what's possible with this workflow:

In this state machine, we have an Athena query that retrieves a list of S3 URIs containing image files that we want to copy to a separate location for further processing. We use the distributed map state to process these S3 URIs and use the SDK integration to call S3 CopyObject to copy the images to the other location.

Conclusion

Step Functions Distributed Map, along with Athena queries can enable orchestrated Serverless workflows that allow us to process huge amounts of data all through the Step Functions service.

About me

I am a Staff Engineer @ Veho. We are still actively hiring for some key roles! If you are passionate about serverless, or even looking to grow your skills, have a look at our open positions!

Power Up Your AWS CDK Stack Props

Roger Chi — Mon, 14 Nov 2022 15:30:05 +0000

Break out of traditional static stack configuration patterns by leveraging dynamic stack parameters with the AWS CDK

Overview

Traditional infrastructure as code frameworks such as CloudFormation and Terraform allow for basic parameterization by defining static values as parameters or environment variables which allow some flexibility in what infrastructure is deployed based on the template or module that defines the stack. Typically the amount of flexibility is hard coded in the modules and it is difficult to fully customize behavior without breaking into the underlying templates/modules. For example, a stack may allow for some input parameters that define an auto scaling strategy, but the implementation of that strategy needs to be predefined in the templates and only basic static parameters are allowed to be modified during deployment.

The beauty of an infrastructure framework like the AWS CDK which allows one to define infrastructure with an actual programming language means we no longer need to hard code or strictly define various aspects of our infrastructure as static parameters, but instead are allowed to dynamically configure stacks using all the language tools available to us as programmers.

Example

Imagine we deployed an ECS service using AWS Fargate and wanted to set different auto scaling strategies for our various deployments. In a traditional CloudFormation template, you might decide to allow for a configurable scheduled scaling strategy and add these parameters to your template:

  Parameters:
    ScaleOutNum: 30
    ScaleInNum: 10
    ScaleOutExpression: 'cron(00 10 * * ? *)'
    ScaleInExpression: 'cron(00 14 * * ? *)'

You would then configure some scheduled scaling resources inside the template which would take these parameters as inputs. But what if you wanted to have multiple scaling times depending on the day of the week? Or if you wanted to use target scaling instead of scheduled scaling? Or if your development environment didn't need a complex scaling schedule? You would need to refactor your CloudFormation template each time, adding some conditionals, and adjusting the parameter inputs to try to accommodate the different variations of how your infrastructure could be defined.

Enter: the AWS CDK

Instead of defining your scaling strategy inside your infrastructure stack, that custom scaling strategy itself could be a parameter/method that is passed into the stack.

// Define the scaling function:
// It takes in a FargateService as a parameter, and doesn't
// return anything. It only applies scaling methods on the
// service.
export interface FargateServiceStackProps extends StackProps {
  scalingStrategy?: (service: FargateService) => void;
}

Inside our stack, we can use the scaling method provided to apply custom scaling strategies to the FargateService.

export class FargateServiceStack extends Stack {
  constructor(
    scope: Construct,
    id: string,
    { scalingStrategy, ...props }: FargateServiceStackProps
  ) {
    super(scope, id, props);

    let fargateService = {} as FargateService;

    // If there's a scaling strategy passed in to the stack,
    // apply it to this service
    if (scalingStrategy) {
      scalingStrategy(fargateService);
    }
  }
}

The ScalingStrategy can be as simple or as complicated as you'd like. For example, setting a simple fixed capacity strategy might look like this:

(service): ScalingStrategy => {
  service.autoScaleTaskCount({
    minCapacity: 10,
    maxCapacity: 10,
  });
};

One with scheduled scaling and CPU/Memory based scaling:

(service): ScalingStrategy => {
  const scalableTarget = service.autoScaleTaskCount({
    minCapacity: 10,
    maxCapacity: 200,
  });

  scalableTarget.scaleOnSchedule('scale-out', {
    schedule: Schedule.cron({ hour: '10', minute: '00' }),
    minCapacity: 50,
    maxCapacity: 200,
  });

  scalableTarget.scaleOnSchedule('scale-in', {
    schedule: Schedule.cron({ hour: '14', minute: '00' }),
    minCapacity: 10,
    maxCapacity: 200,
  });

  scalableTarget.scaleOnCpuUtilization('CpuScaling', {
    targetUtilizationPercent: 50,
    scaleInCooldown: Duration.minutes(2),
    scaleOutCooldown: Duration.seconds(30),
  });
  scalableTarget.scaleOnMemoryUtilization('MemoryScaling', {
    targetUtilizationPercent: 50,
    scaleInCooldown: Duration.minutes(2),
    scaleOutCooldown: Duration.seconds(30),
  });
};

These scaling strategies can also be built into convenience functions, which leads to your stacks being defined like so:

const app = new App();

new FargateServiceStack(app, 'FargateService-dev', {
  scalingStrategy: fixedCapacity({ fixedCapacity: 10 }),
});

new FargateServiceStack(app, 'FargateService-prod', {
  scalingStrategy: scheduledScalingFn({
    scaleOutMinCapacity: 50,
    scaleOutMaxCapacity: 200,
    scaleOutSchedule: Schedule.cron({ hour: '10', minute: '00' }),
    scaleInMinCapacity: 10,
    scaleInMaxCapacity: 200,
    scaleInSchedule: Schedule.cron({ hour: '14', minute: '00' }),
    cpuUtilizationPct: 50,
    memoryUtilizationPct: 50,
    scaleInCooldown: Duration.minutes(2),
    scaleOutCooldown: Duration.seconds(30),
  }),
});

Example code

All example code can be found in this repo: https://github.com/rogerchi/cdk-dynamic-stack-props

Conclusion

Using the AWS CDK allows us to use all of the possibilities of a full programming language when defining our Cloud applications, including passing in dynamic methods into our stacks, allowing us more control over our infrastructure.

If you liked this article, consider following me on Twitter: @scythide

About me

I am a Staff Engineer @ Veho. We are still actively hiring for some key roles! If you are passionate about serverless, or even looking to grow your skills, have a look at our open positions!

One weird trick for sampling EventBridge events

Roger Chi — Mon, 28 Mar 2022 13:13:28 +0000

Amazon EventBridge is a serverless event bus and a key component of serverless, event-driven applications. However, it can be difficult to construct example event data for testing environments, and developing against mock events can be limiting. You may also wish to perform some continuous testing against production events, but the volume of events may make this infeasible. For all of these reasons, it would be great to be able to sample a random portion of events from a production eventbus. Unfortunately, there is no native functionality to do this, but it can be approximated using this one weird trick!

The trick

It has been possible to match events based on complex content patterns since early 2020. EventBridge events also always include an id field which is in a uuid format. We can create a rule which approximates a given sampling percentage using these two properties. Each character in the id field can be one of 16 values, 0-f (hexadecimal values). This means that we can set the EventPattern to:

{ "id": [{ "prefix": "0" }] }

This should match 1/16th, or 6.25% of our events. We can increase the granularity by increasing the number of digits that we match on. A prefix consisting of two characters will match 0.39% of events, and a prefix consisting of three characters will match 0.0244% of events.

For example, if we want to match 1% of events, we can give a list of prefixes such as:

{
  "id": [
    { "prefix": "00" },  // 0.39%
    { "prefix": "01" },  // 0.78%
    { "prefix": "020" }, // 0.81%
    { "prefix": "021" }, // 0.83%
    { "prefix": "022" }, // 0.85%
    { "prefix": "023" }, // 0.88%
    { "prefix": "024" }, // 0.90%
    { "prefix": "025" }, // 0.93%
    { "prefix": "026" }, // 0.95%
    { "prefix": "027" }, // 0.98%
    { "prefix": "028" }  // 1.00%
  ]
}

Note: There's no guarantee that the uuids generated by EventBridge are uniformly distributed, but for most purposes it should be sufficient.

We can build in logic for generating these prefix lists into a CDK construct.

The SamplingRule construct (code)

All example code can be found in this GitHub repository: rogerchi/cdk-events-sampling

There is currently an open issue in the AWS-CDK repository for defining content-based matchers in an EventBridge Rule, so for our construct we will need to use an escape hatch and drop down to the underlying L1 CloudFormation construct to set the patterns.

import { RuleProps, Rule, CfnRule } from 'aws-cdk-lib/aws-events';
import { Construct } from 'constructs';

export interface SamplingRuleProps extends RuleProps {
  percentage: number;
}

export class SamplingRule extends Rule {
  constructor(
    scope: Construct,
    id: string,
    { percentage, ...props }: SamplingRuleProps
  ) {
    // Construct the base class with a dummy eventPattern
    super(scope, id, { ...props, eventPattern: { version: [''] } });

    // Escape hatch for getting the CfnRule to be able to use content-based
    //   filtering until this issue is resolved:
    //   https://github.com/aws/aws-cdk/issues/6184
    const cfnRule = this.node.defaultChild as CfnRule;
    const idPattern = this.getIdPatternFromPercentage(percentage);
    if (!props.eventPattern) {
      cfnRule.eventPattern = { id: idPattern };
    } else {
      cfnRule.eventPattern = { ...props.eventPattern, id: idPattern };
    }
  }

  // Convert percentage into list of uuid prefix patterns
  private getIdPatternFromPercentage(percentage: number): { prefix: string }[] {
    if (percentage >= 1 || percentage <= 0) {
      throw new Error('Percentage must be between 0 and 1');
    }
    // Get the first three significant hex digits from the percentage
    const hexString = percentage.toString(16).slice(0, 5).slice(-3);

    // Build up list of prefixes, starting from most significant digit
    let prefixes: string[] = [];
    let a;
    for (a = 0x0; a < parseInt(hexString[0], 16); a++) {
      prefixes.push(a.toString(16));
    }
    let b;
    for (b = 0x0; b < parseInt(hexString[1], 16); b++) {
      prefixes.push(a.toString(16) + b.toString(16));
    }
    let c;
    for (c = 0x0; c < parseInt(hexString[2], 16); c++) {
      prefixes.push(a.toString(16) + b.toString(16) + c.toString(16));
    }

    return prefixes.map((prefix) => ({ prefix }));
  }
}

Use cases

Here are some use cases for sampling production events:

Sending a portion of real events to QA/Test and Dev accounts for more reliable real-world tests and developing with real events. (Could first target a Lambda function that anonymizes certain keys in the payload to remove sensitive information)
Sending a portion of real events to a continuous end-to-end testing environment
Continuously spot checking a sample of real events to ensure that they are conforming to published schemas

Testing (code)

The example repo has a CDK application set up to create a few stacks: a production eventbus stack, a development eventbus stack, and a stack that configures event replication from the production eventbus to the development eventbus using a sampling percentage of 3%. We deploy the CDK application to our sandbox account and invoke the scripts/put-events.ts script to put 10,000 events on our production eventbus. We should expect the sampling rule to match around 3% of these events and replicate them into our development stack (of course the development stacks could live in separate development accounts as well).

ts-node scripts/put-events.ts {prodEventBusName}

We check CloudWatch metrics for our event buses and we can see that 2.81% of our events were sampled and sent to our dev eventbus.

Conclusion

My hope is that you will be able to use the SamplingRule construct/pattern to help accelerate your serverless and event-driven application development! Please let me know if this was helpful, and consider following me on Twitter: @scythide

About me

I am a Staff Engineer @ Veho. We recently raised our Series A and B funding rounds and are actively hiring! If you are passionate about serverless, or even looking to grow your skills, have a look at our open positions!

Three serverless, Lambda-less API patterns with AWS CDK

Roger Chi — Thu, 17 Mar 2022 18:21:47 +0000

Serverless, Lambda-less microservice patterns are growing in popularity these days. The goal of this article is to help jump-start your entry into these patterns by describing three different methods for implementing an API boundary using the AWS CDK.

All code can be found in this repo: rogerchi/cdk-functionless-api

API patterns

The three basic patterns that will be described in this article:

Asynchronous, small payload (<256kb, the EventBridge payload limit)
Asynchronous, large payload (<10mb, the API Gateway payload limit)
Synchronous, small payload (<256kb, the Step Functions state size limit)

These examples are bare-bones on purpose, additional functionality such as API Gateway Authorizers, validators, processing the payloads with event targets, etc. can be layered on top to provide a fully robust interface.

Asynchronous APIs

For each of our asynchronous APIs, we will ingest a payload and emit an event to our event bus for asynchronous processing. We will immediately return an id to the caller that is matched with the event, which can be used later to retrieve further information about the request.

Small payload (code)

This example sets up a direct service integration from API Gateway to the EventBridge PutEvents API call. The response from the direct integration is the ID of the event that is created, and which can then be used by the caller to check on progress and by the backend services to update tables, etc. The size of the payload is limited by the maximum size of an EventBridge event: 256kb.

First, we import the constructs required:

import { Construct } from 'constructs';
import { AwsIntegration, IRestApi } from 'aws-cdk-lib/aws-apigateway';
import { IEventBus } from 'aws-cdk-lib/aws-events';
import { Role, ServicePrincipal } from 'aws-cdk-lib/aws-iam';

In this particular example we pass our event bus and REST API into our construct:

interface AsyncEventBridgeApiProps {
  eventBus: IEventBus;
  api: IRestApi;
}

We define our construct:

export class AsyncEventBridgeApi extends Construct {
  constructor(
    scope: Construct,
    id: string,
    { eventBus, api }: AsyncEventBridgeApiProps,
  ) {
    super(scope, id);

We create the credentials role that API Gateway will use to perform PutEvents on our event bus:

const role = new Role(this, 'role', {
  assumedBy: new ServicePrincipal('apigateway.amazonaws.com'),
});

eventBus.grantPutEventsTo(role);

We define the AWS service integration and set up the resource path in the API:

const eventbridgeIntegration = new AwsIntegration({
  service: 'events',
  action: 'PutEvents',
  integrationHttpMethod: 'POST',
  options: {
    credentialsRole: role,
    requestTemplates: {
      'application/json': `
        #set($context.requestOverride.header.X-Amz-Target ="AWSEvents.PutEvents")
        #set($context.requestOverride.header.Content-Type ="application/x-amz-json-1.1")
        ${JSON.stringify({
          Entries: [
            {
              DetailType: 'putEvent',
              Detail: "$util.escapeJavaScript($input.json('$'))",
              Source: 'async-eventbridge-api',
              EventBusName: eventBus.eventBusArn,
            },
          ],
        })}
      `,
    },
    integrationResponses: [
      {
        statusCode: '200',
        responseTemplates: {
          'application/json': JSON.stringify({
            id: "$input.path('$.Entries[0].EventId')",
          }),
        },
      },
    ],
  },
});

const resource = api.root.addResource('async-eventbridge');
resource.addMethod('POST', eventbridgeIntegration, {
  methodResponses: [{ statusCode: '200' }],
});

These header overrides are required to be able to put events to the event bus, we define them in the VTL request template:

#set($context.requestOverride.header.X-Amz-Target ="AWSEvents.PutEvents")
#set($context.requestOverride.header.Content-Type ="application/x-amz-json-1.1")

In this example, we put the whole body of the request into the detail of the event. This part of the VTL template might be customized to perform further processing of the input body:

${JSON.stringify({
  Entries: [
    {
      DetailType: 'putEvent',
      Detail: "$util.escapeJavaScript($input.json('$'))",
      Source: 'async-eventbridge-api',
      EventBusName: eventBus.eventBusArn,
    },
  ],
})}

Our integration response template maps the service response from EventBridge to only give the id of the event back to the caller.

integrationResponses: [
  {
    statusCode: '200',
    responseTemplates: {
      'application/json': JSON.stringify({
        id: "$input.path('$.Entries[0].EventId')",
      }),
    },
  },
],

Invoking the API

We can invoke our API with a payload:

curl -X POST -d '{"message": "Hello world!"}' -H 'Content-Type: application/json' https://xxxxxxxxxx.execute-api.us-east-2.amazonaws.com/prod/async-eventbridge

{"id":"3a82be5b-d2fd-a85b-3ba4-5a37b98dd9e1"}

And see our event in the event bus:

Large payload (code)

In cases where the payload exceeds the 256kb size limit of EventBridge, we can use S3 to store the payload prior to downstream asynchronous processing.

This example sets up a direct service integration from API Gateway to the S3 PutObject API. The response from the direct integration is the ID of the API Gateway request, which is also used as the object key in S3. The size of the payload is limited by the maximum size of an API Gateway payload: 10mb.

First, we import the constructs required:

import { AwsIntegration, IRestApi } from 'aws-cdk-lib/aws-apigateway';
import { IEventBus } from 'aws-cdk-lib/aws-events';
import { Role, ServicePrincipal } from 'aws-cdk-lib/aws-iam';
import { Construct } from 'constructs';
import { EventsEnabledBucket, S3DetailType } from './events-enabled-bucket';

Note: there is an open issue currently for enabling EventBridge S3 bucket notifications, so the sample code includes a workaround construct EventsEnabledBucket.

Again, we pass our event bus and REST API into our construct:

interface AsyncS3ApiProps {
  eventBus: IEventBus;
  api: IRestApi;
}

We define our construct:

export class AsyncS3Api extends Construct {
  constructor(
    scope: Construct,
    id: string,
    { eventBus, api }: AsyncS3ApiProps,
  ) {
    super(scope, id);

We create the bucket and the credentials role that API Gateway will use to perform PutObject on our bucket:

const bucket = new EventsEnabledBucket(this, 'bucket', {
  ruleActions: [S3DetailType.OBJECT_CREATED],
  eventBus,
});

const executionRole = new Role(this, 'ExecutionRole', {
  assumedBy: new ServicePrincipal('apigateway.amazonaws.com'),
});

bucket.grantWrite(executionRole);

We define the AWS service integration and set up the resource path in the API:

function requestTemplate() {
  const statements = [
    '#set($context.requestOverride.path.objectKey = $context.requestId)',
    '$input.body',
  ];
  return statements.join('\n');
}

const s3Integration = new AwsIntegration({
  service: 's3',
  path: `${bucket.bucketName}/{objectKey}`,
  integrationHttpMethod: 'PUT',
  options: {
    requestTemplates: {
      'application/json': requestTemplate(),
    },
    integrationResponses: [
      {
        statusCode: '200',
        responseTemplates: {
          'application/json': JSON.stringify({
            id: '$context.requestId',
          }),
        },
      },
    ],
    credentialsRole: executionRole,
  },
});

const resource = api.root.addResource('async-s3');
resource.addMethod('POST', s3Integration, {
  methodResponses: [{ statusCode: '200' }],
});

For an S3 integration, we use a PUT method and set the path to the bucket name and object key:

service: 's3',
path: `${bucket.bucketName}/{objectKey}`,
integrationHttpMethod: 'PUT',

We override the objectKey with the value of requestId from the API Gateway request:

function requestTemplate() {
  const statements = [
    '#set($context.requestOverride.path.objectKey = $context.requestId)',
    '$input.body',
  ];
  return statements.join('\n');
}

Our integration response template maps the requestId from API gateway to give an id back to the caller.

integrationResponses: [
  {
    statusCode: '200',
    responseTemplates: {
      'application/json': JSON.stringify({
        id: '$context.requestId',
      }),
    },
  },
],

Invoking the API

We can invoke our API with a payload:

curl -X POST -d '{"message": "Hello world!"}' -H 'Content-Type: application/json' https://xxxxxxxxxx.execute-api.us-east-2.amazonaws.com/prod/async-s3

{"id":"24e15c2a-543a-47bf-93be-6dfada6a9820"}

We see the object in S3:

And we see our event in the event bus:

Synchronous APIs (code)

For our synchronous API, we will take advantage of the AWS StepFunctions backed APIs construct.

First, we import the constructs required:

import { IRestApi, StepFunctionsIntegration } from 'aws-cdk-lib/aws-apigateway';
import { Role, ServicePrincipal } from 'aws-cdk-lib/aws-iam';
import {
  JsonPath,
  Pass,
  StateMachine,
  StateMachineType,
} from 'aws-cdk-lib/aws-stepfunctions';
import { Construct } from 'constructs';

In this particular example we pass our REST API into our construct:

interface SyncExpressSfnApiProps {
  api: IRestApi;
}

We define our construct:

export class SyncExpressSfnApi extends Construct {
  constructor(scope: Construct, id: string, { api }: SyncExpressSfnApiProps) {
    super(scope, id);

We create a simple express step function which will be invoked from our API:

const machine = new StateMachine(this, 'sfn', {
  definition: new Pass(this, 'pass', {
    parameters: {
      message: 'Hello from express API',
      echo: JsonPath.stringAt('$.body'),
    },
  }),
  stateMachineType: StateMachineType.EXPRESS,
});

We create the credentials role that API Gateway will use to perform StartSyncExecution on our express step function:

const executionRole = new Role(this, 'ExecutionRole', {
  assumedBy: new ServicePrincipal('apigateway.amazonaws.com'),
});

machine.grantStartSyncExecution(executionRole);

We define the AWS service integration and set up the resource path in the API:

const sfnIntegration = StepFunctionsIntegration.startExecution(machine, {
  credentialsRole: executionRole,
});

const resource = api.root.addResource('sync-sfn');
resource.addMethod('POST', sfnIntegration, {
  methodResponses: [{ statusCode: '200' }],
});

The AWS StepFunctions backed APIs construct returns the state machine output as the response from the API.

Invoking the API

We can invoke our API with a payload and receive a synchronous response:

curl -X POST -d '{"message": "Hello world!"}' -H 'Content-Type: application/json' https://xxxxxxxxxx.execute-api.us-east-2.amazonaws.com/prod/sync-sfn

{"message":"Hello from express API","echo":{"message":"Hello world!"}}

Conclusion

Hopefully these examples will allow you to quickly get up and running with serverless, lambda/function-less APIs!

If you liked this article, consider following me on Twitter: @scythide

DEV Community: Roger Chi

AppSync Cache Eviction with Lambda Data Sources

The Pattern

Lambda Response Structure

VTL Response Mapping Template

JavaScript Response Mapping Template

CDK Implementation

VTL Resolver

JavaScript Resolver

Example Implementation

GraphQL Schema

Lambda Implementation

Test Results

Key Observations

Cache Key Format

I wrote an article describing a simple strategy for discovering service relationships in an EventBridge-based event-driven system.

Simple Service Relationships in EventBridge

Roger Chi for AWS Community Builders ・ Jan 12

Simple Service Relationships in EventBridge

Setting Up Schema Registry for Producer Discovery

AWS-CDK Setup

CloudFormation Setup

Schema Limit Considerations

Establishing a Rule Tagging Convention for Consumer Discovery

The Tagging Convention

CDK Implementation

CloudFormation Example

Discovering Producers and Consumers

Producer Discovery

Consumer Discovery

Generating the Service Graph

Example Output

Putting It All Together

Conclusion

AWS AppConfig multi-accounts, multi-environments with ease

Authorization and access compared to third party tools

AWS Organization setup

CloudFormation templates

AppConfig Account Template

Workload Account Template

Setting up AppConfig Agent for Lambda functions

CDK code

Docker Image Functions

Conclusion

Bonus, for users of org-formation-cli

Practical DynamoDB - Locking Reads

The Challenge of Concurrent Access

Understanding Lock Management Through Condition Expressions

Decision Flow Diagram

DynamoDB's Serial Write Guarantee

Atomic Lock Acquisition

Protected Updates with Lock Verification

Lock Lifecycle Example

Conclusion

A Mental Model for DynamoDB Write Limits

Partition limits

Test setup

UpdateItem code

UpdateItem results (10,000 updates)

PutItem code

PutItem results

TransactWriteItem

Code

TransactWriteItem results

Our mental model for DynamoDB Writes

About me

Step Functions Distributed Map and Athena, a match made in cloud heaven

Configuration

Example

Conclusion

About me

Power Up Your AWS CDK Stack Props

Overview

Example

Enter: the AWS CDK

Example code

Conclusion

Further reading

About me

One weird trick for sampling EventBridge events

`UpdateItem` code

`UpdateItem` results (10,000 updates)

`PutItem` code

`PutItem` results

`TransactWriteItem`

`TransactWriteItem` results