Finally, Real Event Buses!

#aws #eventdriven

I haven't been excited about many new AWS features this year, but the notification about the relaunch of EventBridge event buses caught my attention! This update brings long-overdue improvements to EventBridge and will be a real benefit for organizations trying to build an event-driven platform. This update addresses many of the multi-account and multi-team challenges we've seen for years. I've seen an organization develop its own serverless eventing platform to avoid these limitations in EventBridge. These improvements truly are long overdue.

The Problem with Classic Buses

With classic EventBridge buses, you have to manage your rules and targets in the same account and region as the bus you are listening to. For small applications or utilities that listen to AWS events for that account, this isn't a problem. If you are building a multi-region or multi-account application, things get challenging quickly. If you're building a true event bus, that means either everyone needs access to the central account where the bus was created, or a platform team manages all subscriptions. You could forward events to buses in different accounts, but then you lose central visibility into what is connected to the bus.

If your application needs to run in multiple regions, you can send events to buses in different regions, but there is no built-in deduplication. So if you need all events to be present in both regions, you must craft the filtering criteria to avoid creating a loop with the buses.

With classic EventBridge buses, you are also limited to AWS's event format. You can include whatever structure you want in your message details, but AWS sets the overall structure, which is pivotal to how classic EventBridge buses work. You also have no options for ordering within a classic bus; if you need your messages to arrive in order, you have to implement something else to sort them further down the line.

That said, classic EventBridge buses are not deprecated. You can migrate to the newer buses if you want; AWS has some guidance, but you can also keep running your existing event bus.

What's New?

These new event buses change some of the concepts used in EventBridge. Instead of rules, the new event buses use a subscriber concept. Subscribers determine the events you want, using filters, and the one target that receives them. Subscribers can optionally include a transform that modifies the event before passing it to the target. Your subscribers end up encapsulating all the details to get specific events to a particular target.

You can use AWS RAM (Resource Access Manager) to share the bus with your organization or specific AWS accounts. Once you have shared the bus, the target account needs to accept the invitation in the RAM console. After that, the bus shows up in the target account's bus list, where they can add subscribers or publish events. You can see basic information about the subscribers added to your bus, such as name, ordering type, account ID, and creation and update dates, but you don't see what they integrate with, their tags, or other details.

If you have an organization that follows AWS Well-Architected principles and uses different accounts for different applications, this allows you to provide a centrally managed bus that different applications, and presumably teams, can publish and subscribe to. With this central management, you can monitor the rate of events flowing through the bus and, if required, revoke account access, for example, due to a cybersecurity incident involving one application.

These new event buses offer the option to choose between unordered and FIFO delivery per subscriber. This, combined with batching configuration, transformations, and filtering, gives you flexibility in how downstream applications receive events. Part of this is event deduplication, which can be either content-based or ID-based.

Another area where you have more flexibility now is in the types and formats of your events. Multiple content types are supported; at launch, they are JSON, Avro, Protobuf, and octet streams. You can also adjust what the events delivered to your target look like; you choose whether you want the raw payload, metadata applied by AWS included, or payload adjusted by a JSONata expression. This means you want to use your own format, AWS's metadata, or something like the CNCF-backed CloudEvents specification.

Retention of events is built into the new event bus; you can retain events for as little as 1 day or as long as 1 year. When you add a subscriber, you can choose a starting point for the events it receives. Often, that will be the latest event after the subscription is created, but it can also be earlier within your retention window. This allows you to replay events to a new subscriber. Unfortunately, there isn't a replay API to let you arbitrarily replay events to a subscriber. Instead, you would need to create a new subscriber to the same target as your existing subscriber, and specify the time span of events you want replayed. Once events reach the end of the timestamp, they will stop being delivered.

Exciting, but Use Caution

Keep in mind, this new version of the EventBridge bus is still new and was only deployed in 14 regions at release. It's likely to receive some patches and adjustments in the coming months. Prototype away, but I'd be cautious about using it for critical production systems for the next month or two. I don't expect major issues, but you should weigh the risks.

CloudFormation has been updated to include the new resource types for this functionality. However, some of the CloudFormation validation tooling has not been updated. The CDK is not updated for these new event buses. They only support the L1 constructs that mirror the CloudFormation resources, but those aren't even complete and require escape hatches to function. I've submitted a bug report to the CDK project to let them know about the issue.

This lack of robust infrastructure-as-code support is my primary reason for advising caution. Unless you are using CloudFormation directly, give it some time. The CDK and Terraform don't have good support yet, and that takes a little time.

Use Case

I'm excited by this new event bus because of the functionality I have in mind for it. At work, we have a system with some event-driven aspects, but bringing it together into an event bus will give us much greater flexibility. We also have an active-standby architecture, and I want to shift that towards an active-active configuration. One challenge we have is that some of our events must be ordered. It's a long road, but this new option for an event bus gives me some hope.

I've created a very basic example of this architecture using the CDK, escape hatches and all, with a multi-region configuration where the buses forward events across regions and Lambda functions subscribe to log the events they receive.

My proof of concept uses content-based deduplication and works well; aside from requiring escape hatches, I'm happy with how it turned out. In a more fully developed application, I would expect many subscribers to use a dead-letter queue. In the system I am envisioning, I would still need to implement idempotency. The Powertools for Lambda (Python, TypeScript) might help with that, but that is a PoC for another time.

jbutz / aws-relaunched-eventbridge-poc

Proof of concept using the relaunched AWS EventBridge event bus

AWS New EventBridge Custom Event Bus PoC

This proof of concept uses the relaunched EventBridge event bus. It creates an event bus in two regions, with messages forwarded between regions, and a Lambda function in each region to log the events they receive.

Setup

Ensure you have Node.js 22 or higher installed on your machine
Ensure you have the AWS CLI installed on your machine
Configure a terminal session with AWS programmatic credentials
Install this repo's dependencies
```
npm ci
```
Run the CDK's boostrap command to ensure you AWS account is configured correctly
```
npx cdk boostrap
```
Deploy the AWS resources
```
npm run deploy
```

Useful commands

npm run build type-check the project
npm run watch watch for changes and type-check
npm run test perform the jest unit tests
npx cdk deploy deploy this stack to your default AWS account/region
npx cdk diff compare deployed stack with current state
npx cdk synth…

View on GitHub

As I said, I'm excited about this relaunch of EventBridge event buses, and I look forward to what this will allow us to build. Still, for now, I'm limiting myself to proofs of concept while IaC support is more fully built out and more regions have the service relaunched.