Maximize your EventBridge productivity with evb-cli

The aim of this article is to demonstrate how evb-cli can be used to boost your productivity when working with Amazon EventBridge. I will present use cases and how you can use the tool to solve them.

I assume that you are familiar with or a user of EventBridge. If not, there are some great videos and blog posts available to get started.

This is part one of two covering this tool. Some commands require a back-end deployed to the target account and for the sake of digestability I will cover them in a separate post at a later date.

Table of contents

• Background
• Prerequisites
• Installation
• Schema Registry basics
• Commands
  • evb pattern - generate event patterns from schemas in the Schema Registry
  • evb input - generate InputTransformer based on schemas in the Schema Registry
  • evb code-binding - generate code bindings for you Lambda consumers
  • evb browse - browse where and how events from a given source/detail-type are consumed
  • evb diagram - Visualise your EventBridge architecture
  • evb extract-sam-event - convert SAM notation to AWS::Events::Rule for more complex use cases
  • evb test-event - test event pattern against an event payload

Commands covered in part two:

• evb replay - replay archived events against select target(s)
• evb replay-dead-letter - replay dead letter events
• evb local - local debugging

Background

EventBridge was released in July 2019, but it took about six months before we started using it heavily at MatHem. Up until then we were leveraging SNS and SQS for our events, but as we started playing with EventBridge we quickly noticed how it improved the decoupling of our services and decreased friction between teams.

Prior to EventBridge, when someone from team A wanted to subscribe to messages using filtering from an SNS topic owned by team B, they had to submit a pull request to team B to add the message attribute they wanted to filter on. We found this being backwards and inefficient.

With EventBridge, the producing team doesn't need to know its consumers. The consumers have access to perform filtering on any part of the payload, but they still need to know the contract of the data they can filter on. The release of the EventBridge Schema Registry at re:Invent 2019 meant that the decoupling got even more optimized.

As we started composing patterns we found that although we had removed team friction we still spent a lot of time on getting the patterns right. Also, writing complex patterns is error prone and requires multiple deploys to test and get right, so we began automating the bridge between the Schema Registry and the pattern composition.

The first version of evb-cli had that sole purpose - to generate event patterns to be pasted into the template. As we got really fast at that, we quickly found further bottlenecks, so we added commands for composing InputTransformers, architecture visualization, code bindings, etc. The sections below describe each command in depth.

Prerequisites

Much of the functionality is based on content in the EventBridge Schema Registry. Make sure you have enabled schema discovery on your event buses.

To get the most of this tool you need to have the aws-cli preconfigured with an IAM or SSO user with permissions covering events:Get*, events:List*, schemas:Get*, schemas:List* and events:List*.

You will also need NPM package manager and NodeJS 12+ installed on your system.

Installation

npm install -g @mhlabs/evb-cli

Schema Registry basics

To make the most of this tool it's important to understand the basics and the limitations of the EventBridge Schema Registry.

There are three types of registries;

• AWS event schema registry - the schemas of the events the AWS services produce. These are the events that live on the default event bus
• Discovered schema registry - on a custom eventbus you can enable schema discovery. These schemas for these events end up here. They can be from an SaaS-provider or your own custom events
• Custom schema registry - here you can upload your own schemas

The discoverer ingests a sampling of the real events and analyses the values. An annoying thing here is that it can't handle null very well and seem to create new versions from 2 events where a value is null in one and non-null in the other. This means when we run commands like evb pattern it only knows the structure of the last event it ingested. If that contained many null values it simply won't know about them.

Another important thing to know is that what constitutes a schema is the combination of source and detail-type. For example, the following event will get an entry in the registry under the name my-source@my-detail-type

{
  "source": "my-source",
  "detail-type": "my-detail-type
  ...
}

It's therefore a good practice to stick to the same detail structure for all events sharing that combination.

Commands

The commands of the CLI have come to life organically as we identified the need for them.

evb pattern

Purpose: Quickly and accurately build event patterns to match events from the EventBridge Schema Registry

Example use case: You are tasked with sending a Slack message to a channel each time a CodePipeline execution in any US region fails.

The events from CodePipeline look like this (taken from here):

{
  "version": "0",
  "id": "CWE-event-id",
  "detail-type": "CodePipeline Pipeline Execution State Change",
  "source": "aws.codepipeline",
  "account": "123456789012",
  "time": "2017-04-22T03:31:47Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:codepipeline:us-east-1:123456789012:pipeline:myPipeline"
  ],
  "detail": {
    "pipeline": "myPipeline",
    "version": "1",
    "state": "STARTED",
    "execution-id": "01234567-0123-0123-0123-012345678901"
  }
}

The pattern you're after looks like this:

source:
  - "aws.codepipeline"
detail-type:
  - "CodePipeline Pipeline Execution State Change"
region:
  - prefix: "us-"
detail:
  state:
    - "FAILED"

Even though this is a very simple pattern it still requires googling the event structure if you want to compose it manually.

Using evb pattern you can solve this in seconds:

Note that I'm passing -t template.yaml to the command. This will prompt you with the option to attach the generated pattern to existing resources in your template. These can either be of type AWS::Serverless::Function or AWS::Events::Rule. If you skip the -t flag, the event will be written to the console.

Template injection works for both YAML and JSON, but note that any YAML comments will be stripped during serialization.

evb input

Purpose: Quickly and accurately build InputTransformer CloudFormation objects.

Example use case: Building on the use case above we now want to invoke a Lambda function when events from CodePipeline match our pattern. To make the function code simpler, we only want to forward the properties we actually need from the event payload. Let's say we want to let the Slack users know the name of the pipeline and the region it was running in.

To achieve this we need to compose an InputTransformer block for the Target under the AWS::Events::Rule resource that maps JSON paths form the original payload into a new template:

InputTransformer:
  InputPathsMap:
    region: "$.region"
    pipeline: "$.detail.pipeline"
  InputTemplate: "{\"region\": <region>, \"pipeline\": <pipeline>}" 

This is tedious copy/paste work and, at least for us, the InputTemplate is often a straight off representation of the InputPathsMap. A nice feature would be if the CloudFormation team made InputTemplate optional and default it to mirror the InputPathsMap if omitted.

evb input is very similar to evb-pattern, but the output is different. Also, at the time of writing it doesn't yet support template injection, so a copy/paste from the console is required:

Note that I pass in -f yaml. This is telling the tool to output the result in YAML. Default is JSON.

evb code-binding

Purpose: Generate code bindings in your preferred language for your Lambda function to use.

Example use case: Given the above InputTemplate we now want to receive this as a strongly typed object. For this example I will use C#, but a fill list of supported languages can be found in quicktype's documentation

In the following video I generate two types of bindings; one for the InputTemplate and one for the entire original event.

To get the optimized code binding for my InputTemplate (FromTemplate.cs), I run evb code-binding and point it at my template. At first I get prompted which language I want to generate for. Next it asks me in which registry the schema can be found. Since we're working with AWS produced events, it's under aws.events. If it's from a custom event bus, then you'd select discovered-schemas. Lastly, I select which Rule and Target combination I want to generate for. Here we only have one choice, so I select that one.

For the sake of demo I also generate a class for the entire original untransformed event (FromSchema.cs). I do this by skipping to provide the -t flag. The tool now takes me through prompts to select language, registry, source and detail-type. As you can see, the output is very verbose and the majority of properties will not be of any interest to the function.

evb browse

Purpose: Quickly find usages of events conforming to a given schema.

Example use cases:

• I could be a producer of an event and I've sent broken data for a period of time and I need to alert downstream teams
• I'm about to start working on a feature and I want to check if something similar already exists in the system

Here we can see our Lambda target that consumes events belonging to the aws.codepipeline@CodePipelinePipelineExecutionStateChange schema along with some other pre-existing consumers. I can view the event pattern and any input transformations.

Another way to browse events is to visualize them using evb diagram as described below.

evb diagram

Purpose: Visualize how events are flowing within an event bus

Example use cases:

• To have a discussion around during project planning meetings
• To quickly get a mental view of how things hang together

In my next post I will go through setting up the evb-local back-end. With that deployed on the target account you are able to see the events as JSON in real-time as they happen.

evb extract-sam-event

Purpose: To convert a SAM EventBridgeRule to an AWS::Event::Rule to get more control over the rule.

Example use case: Often when you need an EventBridge to Lambda integration it's tempting to start with the EventBridgeRule SAM shorthand. Under the hood, the SAM macro inflates this into a full AWS::Events::Rule and an AWS::Lambda::Permission, but using this disables you from using the powerful InputTransformer. I've opened an issue about this, so please upvote.

I've found myself refactoring a SAM EventBridgeRule into the real thing enough times to automate it with a simple command. Again, note that if you have comments in your YAML, then these will be stripped using this command.

evb test-event

Purpose: Tests an event payload against rules on an event bus.

Example use cases:

• You have been given an example payload to consume and you want to test your rule.
• You are a producer of events and want to see which rules match it.
• Integration tests

Conclusion

In this post I've gone through how we optimize many of the tasks developers working with EventBridge encounter - from building patterns, input transformers, code bindings to gaining insights into the system by browsing schema usage or visualizing the architecture.

In my next post I will focus on local debugging and replaying of archived events.

Comments

[Pubudu Jayawardana]

Wow this is awesome! I’m definitely going to give it a try