Broadcasting messages from one source to multiple destinations (or consumers) is a commonly seen pattern in distributed computing solutions.
A very simple case is when we would like to trigger multiple workers at the same time to run a parallel job. The trigger can be executed, e.g., as a CloudWatch cron-like event.
Here, the same message is delivered to multiple consumers. In terms of a specific implementation, using AWS cloud components, the above diagram can be transformed into the following architecture diagram:
Now, let say that we would like that pattern an event bus, on which messages to different consumers are published into the same communication channel, as depicted on the figure below.
The communication channel could be, e.g. SNS topic, and the consumers, as in the previous example, can be an AWS Lambdas. Multiple producers (marked with different colours) push messages to the communication bus and the consumers read the messages from it.
This solution has one major drawback, the same message will be delivered to multiple consumers, even if the consumer cannot handle that message. So for example, an orange consumer will receive message marked with green, or red. In principle, nothing bad could happen if the consumers can filter out such messages. However, this will involve additional computational time and cost.
We could think about several solutions to this kind of problem. The filtering, as said before, can be delegated to the consumers. There could be multiple communication channels, one per every message type, so the consumers can be linked directly with the appropriate channel. But on the cost of complicated architecture.
Instead of making the architecture complex, or move the filtering to the consumer code, we could use message filtering available in SNS service. By this, the consumers can get only a subset of the messages that are pushed to the topic.
The filtering is done on the attributes and values of the attributes assigned to the published message. A very basic example, using a serverless framework, is shown below
functions: hndl1: handler: handler.fun1 events: - sns: arn: !Ref NotifyTopic topicName: notifyTopic filterPolicy: messageType: - hndl1 messageContext: - running
Here, the lambda hndl1 will be triggered by an incoming message on SNS topic only if the message will have two attributes:
messageContext of values
Currently, SNS policy filtering can handle only attributes of types:
The filter values can be either matched exactly or treated as prefixes. In the previous example, the values were matched exactly. In the following example, we use keywords
prefix to indicate that the value should be treated as a string prefix. Moreover, the keyword
anything-but allows defining the values the attribute should not have (blacklist).
functions: hndl2: handler: handler.fun2 events: - sns: arn: !Ref NotifyTopic topicName: notifyTopic filterPolicy: messageType: - hndl2 messageContext: - incontext - prefix: type_ messageStage: - anything-but: - restart
It is worth to mention that
prefix and constant values can be joined together. In the above example,
messageContext matches either with
incontext value or with
type_abc … etc.
The same does not hold for
anything-but keyword, which cannot be mixed with exact values.
Apart from working with strings, filters can also operate on numerical values. This is done by using keyword numeric, as in the example below
functions: hndl3: handler: handler.fun3 events: - sns: arn: !Ref NotifyTopic topicName: notifyTopic filterPolicy: messageType: - hndl3 cost: - numeric: - ">" - 0
Here, the message with attributes messageType equal to hndl3 and cost greater than zero. The numeric filter accepts the following operators:
<=. In addition, you can define a range of values. In the example below, the value of cost attribute has to be in the range (0, 200) to be accepted.
cost: - numeric: - ">" - 0 - "<" - 200
You can even mix numeric and string attribute values, as in the example below
cost: - unknown - numeric: - "=" - 100
cost attribute has to be either
SNS filtering can help you distribute the messages to right consumers, reducing your computational cost on a little price of writing filters.
It is quite robust tool and can be used for all kind of receiver the SNS subscription can handle.