Amazon EventBridge - How to manage Bus to Bus recursive message passing

#aws #serverless #eventdriven #eventbus

It is possible for an EventBridge bus to have a rule that sends messages to another EventBridge bus. But, what happens if the second EB bus has a rule that sends any message it receives back to the first EB bus? As expected, there will be a recursive data transfer between two EB buses.

In this article, let’s discuss how to address such recursive messages passing between two EB buses.

First, let's look at the architecture.

You can find the CDK project that includes the source code to set up this system in your AWS environment using AWS CDK and Python here: https://github.com/pubudusj/eb-to-eb

How it works

Once the project is set up, this will create 2 EB buses. Each bus will have two rules.

One rule has the other EB bus as the target. The other rule will have a CloudWatch group as a target which will log each message received into the bus. Also each bus will have a Dead Letter Queue setup to preserve any messages that cannot be processed.

To test, you can send any message into EB bus A.

The expected behaviour is that the message is being sent between the EB buses as a loop, so that there will be a record in CloudWatch Log group for each receipt.

The actual behaviour is that, the message is sent from bus A to bus B. CloudWatch groups A has the message logged. Also, CloudWatch group B has logged the message which means the message was received to bus B.

However, those are the only available CloudWatch log records, which means the message was not sent back to bus A.

Instead, if you check the bus B’s dead letter queue, you can see the message is available there. Also, if you check the message attributes, you can see the error code as “THIRD_ACCOUNT_HOP_DETECTED”. And the error message says — “Event ingestion rejected for event [event id] because an event can be sent to an event bus target only once. This event was previously delivered to an event bus target.”

Which means, at bus B, it failed to process the message when try to send it back to bus A.

Explanation

Well, this is the default behaviour of Amazon EventBridge Event Buses. EventBridge Bus has a mechanism to not send the same message to another event bus target more than once. In such a scenario, it will simply reject the message, and in this case, since we have DLQ set up, the message was sent there.

Because of this feature of Amazon EventBridge, recursion loop is prevented. So, you don’t need to take any additional steps to prevent the recursion here.