Implement content enricher pattern easily with Message Silo

Daniel Berki — Fri, 02 Jun 2023 08:55:10 +0000

About the pattern

When sending messages from one system to another it is common for the target system to require more information than the source system can provide.
With content enricher pattern you can add an enricher in the middle to pick up messages/events and enrich them before sending downstream to consumers.

In this article, we will see how the above pattern can be implemented in the simplest way, without any extra infrastructure!

What is Message Silo?

Message Silo is an open-source platform/tool for Developers to auto-correct dead-lettered messages and make integration simpler for event-driven systems.

It also has a CLI called siloctl.

The example case

Suppose we have a service that can send latitude and longitude coordinates in a message to a message queue (e.g.: AWS SQS).

However, we on the consumer side would need more than this, our application would like to display the name of the country belonging to the coordinate, the city, zip code and other information.

The message must therefore be enriched. For this we can use some kind of reverse geocode API (e.g.: https://geocode.maps.co).

The flow looks something like this:

In the Enrichment Phase we will call the external API to get the extra data & transform the message to fit into the consumer expectations.

Let's do it!

Let's see how this all looks in Message Silo!

Step 1. Define the YAML file(s)

Based on the Message Silo Docs I created a YAML file that can correlate with our workflow:

example.yaml

apiVersion: v1
kind: Connection
name: input
type: AWS_SQS
queueName: test-q-input
region: eu-north-1
accessKey: XXXXX
secretAccessKey: XXXXX
receiveMode: ReceiveAndDelete
enrichers:
 - reverseGeocoding
 - messageTransformator
target: output
---
apiVersion: v1
kind: Enricher
type: API
name: reverseGeocoding
method: GET
url: "https://geocode.maps.co/reverse?lat={latitude}&lon={longitude}"
---
apiVersion: v1
kind: Enricher
type: Inline
name: messageTransformator
function: "(x) => { return { latitude: x.lat, longitude: x.lon, name: x.display_name, address: x.address }; }"
---
apiVersion: v1
kind: Target
name: output
type: API
url: "https://myapi.mydomain.com/location/add"

Step 2. Apply our solution

I downloaded the siloctl and created my account, so I can apply my changes with a single command:

siloctl apply -f c:/temp/example.yaml

Step 3. Enjoy! :)

The established Message Silo connection monitors the messages coming to test-q-input AWS SQS and automatically transforms the messages with the help of Enrichers (in 2 steps: reverseGeocoding, messageTransformator) and then forwards them to the specified API endpoint (output Target).

So what we did is we wrote our workflow in a Message Silo configuration:

What were the benefits of using Message Silo?

We didn't have to touch our existing code
No additional infrastructure elements had to be installed
We didn't have to write and host the enrichment phase
Our system elements are remains decoupled