One weird trick for sampling EventBridge events

#serverless #aws #eventbridge #awscdk

Amazon EventBridge is a serverless event bus and a key component of serverless, event-driven applications. However, it can be difficult to construct example event data for testing environments, and developing against mock events can be limiting. You may also wish to perform some continuous testing against production events, but the volume of events may make this infeasible. For all of these reasons, it would be great to be able to sample a random portion of events from a production eventbus. Unfortunately, there is no native functionality to do this, but it can be approximated using this one weird trick!

The trick

It has been possible to match events based on complex content patterns since early 2020. EventBridge events also always include an id field which is in a uuid format. We can create a rule which approximates a given sampling percentage using these two properties. Each character in the id field can be one of 16 values, 0-f (hexadecimal values). This means that we can set the EventPattern to:

{ "id": [{ "prefix": "0" }] }

This should match 1/16th, or 6.25% of our events. We can increase the granularity by increasing the number of digits that we match on. A prefix consisting of two characters will match 0.39% of events, and a prefix consisting of three characters will match 0.0244% of events.

For example, if we want to match 1% of events, we can give a list of prefixes such as:

{
  "id": [
    { "prefix": "00" },  // 0.39%
    { "prefix": "01" },  // 0.78%
    { "prefix": "020" }, // 0.81%
    { "prefix": "021" }, // 0.83%
    { "prefix": "022" }, // 0.85%
    { "prefix": "023" }, // 0.88%
    { "prefix": "024" }, // 0.90%
    { "prefix": "025" }, // 0.93%
    { "prefix": "026" }, // 0.95%
    { "prefix": "027" }, // 0.98%
    { "prefix": "028" }  // 1.00%
  ]
}

Note: There's no guarantee that the uuids generated by EventBridge are uniformly distributed, but for most purposes it should be sufficient.

We can build in logic for generating these prefix lists into a CDK construct.

The SamplingRule construct (code)

All example code can be found in this GitHub repository: rogerchi/cdk-events-sampling

There is currently an open issue in the AWS-CDK repository for defining content-based matchers in an EventBridge Rule, so for our construct we will need to use an escape hatch and drop down to the underlying L1 CloudFormation construct to set the patterns.

import { RuleProps, Rule, CfnRule } from 'aws-cdk-lib/aws-events';
import { Construct } from 'constructs';

export interface SamplingRuleProps extends RuleProps {
  percentage: number;
}

export class SamplingRule extends Rule {
  constructor(
    scope: Construct,
    id: string,
    { percentage, ...props }: SamplingRuleProps
  ) {
    // Construct the base class with a dummy eventPattern
    super(scope, id, { ...props, eventPattern: { version: [''] } });

    // Escape hatch for getting the CfnRule to be able to use content-based
    //   filtering until this issue is resolved:
    //   https://github.com/aws/aws-cdk/issues/6184
    const cfnRule = this.node.defaultChild as CfnRule;
    const idPattern = this.getIdPatternFromPercentage(percentage);
    if (!props.eventPattern) {
      cfnRule.eventPattern = { id: idPattern };
    } else {
      cfnRule.eventPattern = { ...props.eventPattern, id: idPattern };
    }
  }

  // Convert percentage into list of uuid prefix patterns
  private getIdPatternFromPercentage(percentage: number): { prefix: string }[] {
    if (percentage >= 1 || percentage <= 0) {
      throw new Error('Percentage must be between 0 and 1');
    }
    // Get the first three significant hex digits from the percentage
    const hexString = percentage.toString(16).slice(0, 5).slice(-3);

    // Build up list of prefixes, starting from most significant digit
    let prefixes: string[] = [];
    let a;
    for (a = 0x0; a < parseInt(hexString[0], 16); a++) {
      prefixes.push(a.toString(16));
    }
    let b;
    for (b = 0x0; b < parseInt(hexString[1], 16); b++) {
      prefixes.push(a.toString(16) + b.toString(16));
    }
    let c;
    for (c = 0x0; c < parseInt(hexString[2], 16); c++) {
      prefixes.push(a.toString(16) + b.toString(16) + c.toString(16));
    }

    return prefixes.map((prefix) => ({ prefix }));
  }
}

Use cases

Here are some use cases for sampling production events:

Sending a portion of real events to QA/Test and Dev accounts for more reliable real-world tests and developing with real events. (Could first target a Lambda function that anonymizes certain keys in the payload to remove sensitive information)
Sending a portion of real events to a continuous end-to-end testing environment
Continuously spot checking a sample of real events to ensure that they are conforming to published schemas

Testing (code)

The example repo has a CDK application set up to create a few stacks: a production eventbus stack, a development eventbus stack, and a stack that configures event replication from the production eventbus to the development eventbus using a sampling percentage of 3%. We deploy the CDK application to our sandbox account and invoke the scripts/put-events.ts script to put 10,000 events on our production eventbus. We should expect the sampling rule to match around 3% of these events and replicate them into our development stack (of course the development stacks could live in separate development accounts as well).

ts-node scripts/put-events.ts {prodEventBusName}

We check CloudWatch metrics for our event buses and we can see that 2.81% of our events were sampled and sent to our dev eventbus.

Conclusion

My hope is that you will be able to use the SamplingRule construct/pattern to help accelerate your serverless and event-driven application development! Please let me know if this was helpful, and consider following me on Twitter: @scythide

About me

I am a Staff Engineer @ Veho. We recently raised our Series A and B funding rounds and are actively hiring! If you are passionate about serverless, or even looking to grow your skills, have a look at our open positions!