DEV Community: Solace Developers

A Simpler Architecture for Handling High Connection Counts and Throughput

arih1299 — Fri, 22 Oct 2021 13:00:47 +0000

I stumbled upon a blog post by Qiscus.com that talks about the architecture that lets them handle 10 million concurrent connections and throughput of 5 million messages per minute. This is the high-level Qiscus architecture shared in the blog post, which jives with what I learned from a talk their CTO Evan Purnama gave in Jakarta a couple years ago.

Pretty cool, right?

In that talk, Evan spoke about the architecture and design decisions, and the post has some of the information there as well. Here’s a few of the key points I noted from this architecture.

MQTT was chosen as the fronting layer because it’s lightweight, requires minimal client resources, and topics can be very specific, and support wildcard patterns.
They opted to decouple the layer that handles the huge incoming traffic and the layer that services the many different consumers of those events for performance and independent scalability reasons.
Message persistency is required so the system can tolerate failed consumers in a way that lets reconnected consumers pick things up from the last message they did receive, and absorb bursts of traffic in a way that lets downstream apps consume incoming data at whatever pace works for them.
Kafka was chosen for the consumption layer because of its scalability and replay capability.

Alternative Architecture

Looking at the requirements and design decision assumptions, I had been thinking if I could come up with a different approach. After a little thinking and a lot of procrastination, here’s what I came up with. Naturally, I’m using Solace PubSub+ Platform as the event platform.

Figure 2: Alternative architecture with a Solace-enabled event mesh.

With this alternative architecture, let’s start with the MQTT requirement. Solace PubSub+ Event Broker supports MQTT 3.1 and 5, along with REST and WebSocket, all without any add-ons or proxies. Events coming in via MQTT can be consumed in any other protocols/APIs used by the consumers on the right hand side, such as JMS, AMQP, WebSocket, MQTT, or even via a REST webhook mechanism.

Message persistence is taken care of with the guaranteed messaging capability of Solace PubSub+ Event Broker, and yes it can do replay as well. No biggies there. In fact, shock absorption and lossless guaranteed delivery is one of the things Solace is known for by our customers globally.

The key distinction between Qiscus’ architecture and the one I’ve presented here is that there is logically just one fabric for the messaging platform, something called an “event mesh” — like service mesh but for events instead of services. With an event mesh, there is no need for applications to move and/or translate the events from connectivity layer to the processing layer.

Tiered Architecture Working as One

This architecture is still decoupled for performance and scalability, by creating a layered architecture for the connectivity side and the processing/consumption side. The connectivity layer consists of multiple PubSub+ brokers deployed as the needed scale as well as location requirements. After all, these brokers can run on most if not all public and private clouds, containers, or virtualization platforms.

The connectivity layer then streams the events to the processing layers in a smart, efficient, and guaranteed manner, so we don’t lose messages along the way. The processing layer then scales as needed to stream those events to the back-end applications.

The Crux of EDA: Topic Routing

One major thing to note here ̶ these back-end applications have all the flexibility of the topic subscriptions with wildcards just like the one you see with MQTT protocol. If you don’t really get what this means, it could be because you are led to believe topic creation is required before anything else, and that it is expensive, and you shouldn’t have the luxury or agility of fine-grained topic addressing. In that case, you’re missing out on one of the great things about EDA, and you should really go watch a video on this topic.

Figure 3: A sneak peek of how topic routing with wildcards works

The Event Mesh

An event mesh, as you should know by now, isn’t a single event broker, nor a cluster of brokers acting as one, but a network of several nodes or pairs or clusters of event brokers working together much like how IP routers work to form a TCP/IP network.

In this alternative architecture, there’s no “subscribers” moving events from the connectivity layer to the processing layer. That’s because these PubSub+ brokers are linked to form an event mesh. A different application of event mesh is distributing events simply by leveraging the topic routing. This is really useful for multi-sites or even hybrid-cloud topology, instead of relying on cumbersome and expensive cross-site replications.

Event Mesh and Internet of Things

Internet of Things (IoT) is closely related to the use of MQTT protocol, but MQTT is being used for more than just IoT, just like Qiscus’ and many other mobile applications or even real-time dashboards.

Event mesh has a few more tricks up its sleeves for IoT solutions. One is the tiering or layering of brokers to form a tree topology that can support larger connection counts and greater geographic distributions. Another is bidirectional communications, i.e. not just streaming events from devices to back end servers, but also sending out events to devices or mobile apps as part of command and control interactions.

Our CTO Shawn McAllister produced a fun demo of those capabilities, and hope you’ll watch what I playfully like to call “The Architects’ Guide to Honking the Horn”.

Performance

While I didn’t do a performance test for this architecture, there is a public report on Solace PubSub+ performance numbers for several possible combinations of deployment types, payload sizes, delivery mode, and client connections.

Conclusion

To recap, check out this summary table for the features and requirements discussed in this post.

What I want to bring up with this alternative architecture is simplicity. Yes, simplicity. Fewer moving parts is better. Leave the event distribution and delivery to the infrastructure and provide support for the many open standard protocols and APIs to avoid rewrites.

I hope this architecture gives you some ideas for your own architecture and set of challenges. Get in touch for an ideas exchange or just quick chats in my LinkedIn or post your questions in the Solace Developer Community. And lastly, head out to Solace.Dev if you are a developer and want to get hands-on with event mesh!

The post A Simpler Architecture for Handling High Connection Counts and Throughput appeared first on Solace.

PubSub+ Message Handling Features: Reject-Message-to-Sender-on-Discard

Leah Robert — Thu, 19 Aug 2021 14:11:27 +0000

Thus far in the message handling features blog series we have looked at message expiry and dead message queues. In this blog post we will cover the persistent messaging feature Reject Message to Sender on Discard. A long name, and an important endpoint property for defining certain scenarios when publishers should receive negative acknowledgements. For short, we will refer to the feature in this blog as Reject-to-Sender.

What is the Reject-to-Sender endpoint property?

Reject-to-Sender defines when a publisher should receive a negative acknowledgement if a message is rejected from an endpoint. This property is typically used when there are multiple consumers that require the exact same data. The configuration could look something like the below; each consumer has its own queue, and each queue has the same subscription.

Each of these consumers will receive the same messages because they are using persistent delivery and they all have the same subscription. However, if one consumer is slow or offline, its queue could fill up.

In this scenario, messages could still be stored in Q1 and Q2, but as Q3 is full, it would not have any room for new messages. Reject-to-Sender being enabled on these queues ensures that a message is stored on all 3 queues, or none of the queues. Almost like an ‘all or nothing’ property for endpoints that share subscriptions.

When the message is rejected from Q3, an error (or negative acknowledgement) is sent back to the publisher and the message is not stored on any of the three queues; it is then up to the publisher to resend the message.

Now, if Q3 had Reject-to-Sender disabled in this scenario, the message would be stored on Q1 and Q2, and the publisher would not receive any error message. This is regardless of whether Q1 and Q2 have the Reject-to-Sender property enabled or not.

How do I use Reject-to-Sender?

Reject-to-Sender is the property of a guaranteed endpoint and can be enabled/disabled as an endpoint setting. There are three options when configuring Reject-to-Sender:

‘When Queue Enabled’ refers to the queue admin status – if the queue is enabled, Reject-to-Sender is enabled
- If the queue is administratively shutdown, this queue will not have an impact on sending negative acknowledgements to the publisher. Negative acknowledgements are only sent if a message is rejected & the queue is enabled. This is the most common setting for the Reject-to-Sender property.
‘Always’ treats Reject-to-Sender as enabled even when the queue is administratively shutdown
- This is the least common setting for this property as messages would always be negative acknowledged to the publisher when a queue is shutdown.
‘Never’ treats Reject-to-Sender as disabled
- This setting is used when a queue should not be included in the ‘all or nothing’ functionality. An example is when a queue bridges data from a Production environment to UAT/Test. If this queue were to fill up, you would not want it to affect Production processing of data.

By default, Reject-to-Sender is set as ‘When Queue Enabled’ for queues, and is disabled for topic-endpoints.

When messages are not able to be saved to a queue, the queue statistics will increment.

There is another scenario to consider: a message is published as persistent, however there is no matching subscription on the broker. For this scenario, there is another Reject-to-Sender property located in the authorization object called a Client-Profile. While I won’t be going into full detail of the Client-Profile object in this blog, the property name is called Reject-Messages-to-Sender-On-No-Subscription-Match-Discard. This property ultimately provides the same behaviour as the queue property, but for specific scenarios where there are no subscriptions for a published persistent message.

If you found this post useful, visit the queue properties section of our docs, and our PubSub+ for Developers page for more information. If you have any questions about queues and topic endpoints, try posting them to the Solace Developer Community.

The post PubSub+ Message Handling Features: Reject-Message-to-Sender-on-Discard appeared first on Solace.

PubSub+ Message Handling Features: Dead Message Queues

Leah Robert — Tue, 15 Jun 2021 14:35:32 +0000

In the first blog in this series, I explained the concept of message expiry, or message-TTL. Here I’ll look at dead message queues (DMQs), which assumes you have background knowledge on Solace endpoints and Solace persistent messaging.

What is a Dead Message Queue?

A dead message queue, sometimes referred to as a dead letter queue, is an endpoint to store messages that have lost delivery eligibility in their original endpoint, and stores messages for special processing or delayed consumption.

You might be wondering how a message can lose delivery eligibility, especially when it is originally published as persistent. There are two ways messages can be deemed ineligible for delivery:

** **** Message Expiry:** The first post in this series goes into more detail on message expiry, but as a summary it is when a message is published to an endpoint with a timer, or time-to-live. When this timer expires, the message is removed from the original endpoint and can be deleted or can be moved to a DMQ.
Poison Message: The second reason a message can lose delivery eligibility is if it is redelivered from an endpoint to a consumer too many times. This scenario could occur when a consumer is unable to process a message and crashes before sending an application acknowledgement. When that application reconnects to the endpoint, it will receive the same message and would be stuck in a loop. In this case, the message is referred to as a poison message. To ensure an application does not get stuck in this loop, an endpoint can be set to redeliver a message a maximum number of times with the setting ‘Maximum Redelivery Count’. Once exceeded, the message will be removed from the endpoint or can be moved to a DMQ.

How do I use a Dead Message Queue with Solace PubSub+ Event Broker?

A DMQ on Solace PubSub+ Event Broker is configured as a normal queue and can be created in the same manner on the Queue page of PubSub+ Broker Manager.

Once created, the DMQ is configured on the original endpoint in the ‘DMQ Name’ field.

Note that messages will not be moved into a DMQ unless they are published with the flag ‘DMQ-Eligible’. Without this set, the messages will be deleted from the original queue when they lose delivery eligibility.

That is an important point that bears repeating: Messages must be DMQ-Eligible!

** ** If you want messages that expire to go to a DMQ, you need:

Queue with ‘Respect-TTL’ enabled, a DMQ configured, and optionally a ‘Maximum-TTL’ specified
Published messages can have a ‘Message-TTL’ and must be ‘DMQ-Eligible’

If you want messages that reach the maximum redelivery count to go to a DMQ, you need:

Queue with ‘Maximum Redelivery Count’ set (Redeliver Forever disabled), and a DMQ configured
Published messages must be ‘DMQ-Eligible’

When a message moves from an endpoint to its DMQ, the below highlighted statistics will reflect this.

Why would I use a DMQ?

A DMQ is useful for special processing when messages are unable to be consumed on their original endpoint. This could be for expedited processing of orders, or the DMQ consumer may process the message in a different way than the original consumer.

Take for example an order management system that must process orders in a timely manner. For orders that are not processed in the desired amount of time, they could be expired to a dead message queue for expedited processing.

Another good example is error handling: a message that can’t be consumed from the original queue is moved to a DMQ to be inspected and processed.

Conclusion

I hope I’ve helped you feel more comfortable with what DMQs are and how to use them. If you found this post useful, you might want to check out the message properties section of our docs, and our PubSub+ for Developers page for more information. If you have any questions about queues and topic endpoints, try posting them to the Solace Developer Community!

The post PubSub+ Message Handling Features: Dead Message Queues appeared first on Solace.

Solace PubSub+ vs Kafka: Implementation of the Publish-Subscribe Messaging Pattern

Himanshu Gupta — Tue, 08 Jun 2021 19:09:06 +0000

This blog post is the second part of a series comparing Apache Kafka with Solace’s PubSub+ Event Broker. In the first, my colleague Ush introduced the two very different internal architectures that Kafka and Solace PubSub+ have. In this post, I’ll explain how pub/sub messaging pattern implementation differs in these two brokers.

Publish/Subscribe, commonly known as pub/sub, is a popular messaging pattern which is commonly used in today’s systems to help them efficiently distribute data and scale among other things. The pub/sub messaging pattern can be easily implemented through an event broker such as Solace PubSub+, Kafka, RabbitMQ, and ActiveMQ.

When you use an event broker, you have a set of applications known as producers and another set of applications known as consumers. Producers are responsible for publishing data to the broker and similarly, consumers are responsible for consuming data from the event broker. By introducing a broker to our architecture, we have removed the need for producers to directly communicate with the consumers. This ensures we have a loosely coupled architecture. Additionally, our broker now becomes responsible for managing connections, security, and subscription interests, instead of implementing this logic in the applications themselves.

Different event brokers implement the pub/sub messaging pattern differently. It is important to understand their implementations when deciding which event broker to use for your specific use case.

Pub/Sub in Kafka

To understand Kafka’s implementation of pub/sub, you need to understand topics, producers, and consumers.

Topics

As described in part one, in Kafka messages are produced and consumed via topics. They are immutable log files, persisted on disk, to which data is appended sequentially. Given that Kafka is a distributed system, you need to create topics and decide how you want to partition and replicate them across your brokers. While topics can be created by default if they don’t exist, it is best practice to manually create them with appropriate settings before using them.

Producer

A Kafka producer can be programmed to write data to a topic. If the topic has 3 partitions, data will be written to all 3 partitions in a round-robin fashion. This leads to a significant problem. Because our data is scattered across multiple partitions, we don’t get ordering across partitions. Additionally, Kafka publisher applications batch writes to improve performance as they publish data to Kafka brokers.

Consumer

If we want to consume the data we just wrote to our topic, we will need to create a consumer application and connect to our Kafka cluster. Your Kafka consumer can easily subscribe to a topic and consume the necessary data. However, as mentioned earlier, there is an issue caused by topic partitions. Because our data was written to 3 partitions, we have lost message ordering. When our consumer subscribes to the topic and consumes the data, it will get unordered messages. Depending on your system, this can be a critical issue.

To overcome this issue, you will need to use a key when publishing data so that all the messages pertaining to a specific key will always go to the same partition and hence preserve ordering. However, as you may have guessed already, with this workaround, you lose simple the ability to have balanced parallelization resulting in some partitions overflowing while others will be lightly used.

Furthermore, in Kafka, messages are polled instead of pushed to the consumer. When you program your consumer application, you are expected to provide a timer to constantly poll for data. As you can imagine, this can be highly inefficient if your application is frequently polling for data, especially when there is no data available.

Pub/Sub in Solace PubSub+

Solace’s PubSub+ Event Broker is an enterprise-grade event broker which supports open APIs and protocols. It comes with numerous enterprise-grade features straight out-of-box such as high-availability, disaster recovery, and security so enterprises don’t have to implement them.

Topics

As you can see in the previous part, the topics of PubSub+ are very different from those of Kafka. Unlike Kafka’s topics, PubSub+ topics are dynamic and don’t need to be manually created. They are simply metadata that you use to describe your messages as you publish them. Additionally, PubSub+ topics are hierarchical, which means consumers can use wildcards to filter based on different levels in a topic.

PubSub+ topics don’t need to be partitioned or replicated. The fact that they don’t need to be created and persisted to disk means that they barely require any maintenance. As your system scales, there is no need to rebalance your topics as you do in Kafka.

Publisher

PubSub+ publishers can be coded using any of the supported open APIs and protocols such as JMS, MQTT, AMQP, and REST. To publish data, your publisher application just needs to specify which topic to publish the data to. And that’s it.

Subscriber

Similar to Kafka, PubSub+ subscribers can be coded to consume from a topic by specifying the entire topic. Or you can subscribe to multiple topics using wildcards (* and >). Moreover, you can use these wildcards to dynamically filter data.

For example, if your publisher is publishing data for all US stocks from different exchanges to topics such as EQ/US/NYSE/AAPL and EQ/US/NASDAQ/FB, my consumer can subscribe to different datasets by subscribing to any of the following topics:

EQ/> – for all equities stocks
*/*/NYSE/> – for all stocks traded on NYSE
*/*/*/FB – for all messages for Facebook traded on different exchanges

It should be noted that with this approach PubSub+ Event Broker efficiently filters the datastreams to the clients as opposed to Kafka, where filtering happens on the client side or via an intermediary KStreams process. This saves both network bandwidth and compute resources on your consumer processes.

Furthermore, PubSub+ subscribers are push based instead of poll based. Messages will be pushed to subscribers rather than constantly poll for data.

Conclusion

To sum it up, the pub/sub messaging pattern implementation exists in both Solace PubSub+ and Kafka but they differ significantly. While PubSub+ Event Broker utilizes dynamic, hierarchical topics with wildcard support, Kafka uses flat topics partitioned across different brokers. Additionally, Solace PubSub+ Event Broker supports open APIs and protocols which prevents vendor lock-in whereas while Kafka is open-source, it does not support open APIs and protocols natively.

In the next post, we will discuss how filtering is achieved in both brokers.

The post Solace PubSub+ vs Kafka: Implementation of the Publish-Subscribe Messaging Pattern appeared first on Solace.

Solace PubSub+ vs Kafka: The Basics

Ushnash (Ush) shukla — Mon, 31 May 2021 18:16:51 +0000

This is the first in a series of blog posts in which we’ll compare Solace PubSub+ Event Broker with Apache Kafka. Our goal is to help you understand the salient differences between the platforms, arming you with the knowledge you need to select the right streaming platform for your organization. For a quick 5-minute overview, check out this video.

About Apache Kafka

Apache Kafka is a distributed data-ingest and event streaming platform developed in 2011 by engineers at LinkedIn for consuming large volumes of log data. Although commonly called a broker, Kafka is more akin to a distributed data store than a traditional message broker. For example, Kafka only supports publish/subscribe messaging by default, and uses the concept of an ever-increasing commit log, where messages are generally retained indefinitely.

Operationally, Apache Kafka brokers run in a cluster coordinated by Apache Zookeeper. Producers publish records to consumers on named topics. Messages, termed records, are prepended to predefined topics and identified with an offset. Topics may be spread across multiple Kafka brokers using a strategy known as topic partitioning, wherein each broker stores a subset of records intended for a given topic. Consumers ingest messages from all partitions (i.e., the entire topic).

About PubSub+ Event Broker

Solace PubSub+ Event Broker offers event streaming along with traditional message broker functionality. Initially developed in 2001 as a purpose-built messaging appliance for the capital markets, the broker has since been virtualized and implemented in both standalone software and SaaS form-factors.

Solace PubSub+ supports several messaging exchange patterns including publish/subscribe, FIFO queueing, and asynchronous request/reply, and both persistent and non-persistent qualities of service. With persistent messaging, producers send events to consumers using queues; events arrive at the queue in FIFO order and are removed only once receipt is confirmed by the consumer. Non-persistent messaging supports pub/sub whereby one or more publishers send messages to multiple subscribers across topics. Unlike Apache Kafka, topics in Solace PubSub+ Event Broker are not partitioned or persistent — senders and receivers exchange messages across a single, reliable session that does not persist messages when consumers are offline.

To understand more about Solace topics vs Kafka topics, check out this popular video by Developer Advocate Aaron Lee or check out his blog post Solace Topics vs. Kafka Topics: What’s the Difference?

Conclusion

As you can see, Kafka and Solace PubSub+ have two very different internal architectures. In the next post, my colleague Himanshu Gupta will compare each platform’s publish/subscribe functionality in greater detail. Meanwhile, you can visit our website to read more about how Solace PubSub+ compares to Kafka or check out this handy one-page datasheet.

The post Solace PubSub+ vs Kafka: The Basics appeared first on Solace.

Demonstrating a Hybrid Architecture with RESTful and Streaming APIs

Himanshu Gupta — Wed, 19 May 2021 13:00:53 +0000

In my last post I introduced the importance of event-driven architecture and endorsed a hybrid architecture consisting of both RESTful and streaming APIs. Now I’d like to walk you through a simple demo that shows the benefit of using a broker, such as Solace PubSub+ Event Broker, that can handle multiple open protocols and APIs.

Besides allowing microservices to publish and consume events in real time, a major benefit of using an event broker is integration. A typical enterprise has all kinds of in-house and vendor products with different APIs and it can be extremely painful to get these applications to talk to each other.

A modern event broker must support multiple open APIs and protocols without having to install new plug-ins or manage proxies. For example, PubSub+ Event Broker supports open protocols like AMQP, MQTT, REST, WebSocket, and APIs in multiple languages, which means you can have one application publishing data via REST and downstream applications consuming that data via AMQP and/or WebSocket. And if tomorrow, you onboard a new vendor application that supports MQTT only then you can easily use MQTT to stream data to the event broker.

Hybrid Architecture

In the previous post, I explained our hybrid architecture in which data is published to the event broker via REST and the broker is responsible for translating that event to the protocol desired by the downstream services.

This demo consists of the following components:

Solace PubSub+ Event Broker
Publishers
1. REST publisher; simple REST POST command(s) via cURL
2. Java publisher; using streaming API
Consumers
1. REST WebHooks; an endpoint will be invoked every time a message is received/enqueued.
2. Java consumer; using MQTT protocol

Here is what it will look like:

As you can see in the diagram, there are some objects created in the event broker as well. There are two types of delivery modes supported by PubSub+: direct and guaranteed. In direct messaging, there is no persistence. It is popularly used for high-throughput and low latency use cases where you can afford some data loss. For critical data, where you want zero message loss, you need guaranteed messaging.

In this example, you’ll use guaranteed messaging and apply persistence via a queue. Our two publishers will publish messages to a well-defined topic with multiple levels (i.e., prices/java/stocks/{ticker}). This allows our subscribers to consume the messages via topic subscriptions mapped to queues. For example, thre’s a queue called queue_java_consumer for java consumer and added a topic subscription prices/>. This will allow messages published by both REST and Java publishers to be enqueued in this queue.

Similarly, there’s a separate queue for our REST consumer called queue_rest_consumer with a different topic subscription prices/rest/>. This means only messages published by the REST publisher will be enqueued in this queue.

Queues

Before you start setting up our publishers and consumers, let’s create the queues that will hold the messages based on topic subscriptions mapped to them. Create two queues:

queue_java_consumer with topic prices/> mapped to it
queue_rest_consumer with topic prices/rest/ mapped to it

You can create a queue by going to the Queues tab on PubSub+ Manager and clicking on + Queue. Once the queue has been created, click on it and add the appropriate subscription.

Publishers

REST publisher

You can publish messages to the broker via cURL commands by executing a POST command against a URL which will contain the host and port of the broker and REST service running on the broker. It will also contain the topic address you want to publish the message to. For example, to publish a message to a local broker running to the topic prices/rest/stocks/aapl, run this command:


curl -X POST http://127.0.0.1:9000/prices/rest/stocks/aapl --header "Content-Type: application/json" -d '{"name":"aapl", "price":"130"}'

The URL that’s used contains the host, port, and topic address in this syntax: http(s)://:/

You can find more information about REST publishers here.

Java publisher

Depending on the protocol you want to use (SMF, MQTT, AMQP), you can use the corresponding API. Solace has a command-line utility tool for demos and tests, called sdkperf, freely available for download here. It comes in different flavors based on the protocol you want to use. This demo uses sdkperf-jcsmp that uses Solace’s SMF protocol. Here is a guide on how to use sdkperf.

Here is the command that starts the Java publisher which will connect to the local broker and publish 10,000 messages at the rate of 1 message per second to the topic prices/java/stocks/aapl.

> bash sdkperf_java.sh -ptl="prices/java/stocks/aapl" -mt=persistent -mn=10000 -mr=1 -msa=100 -cip=localhost:55555

Consumers

REST consumer

The REST consumer will be a webservice endpoint which is invoked every time the queue receives a message. You will first need to have a webservice which accepts POST commands.

Webservice

Let’s spin up a webservice which will expose a REST endpoint. This endpoint will just echo the arguments once invoked. Here is sample python code to spin up the webservice:


from flask import Flask, json, request
from flask import *

companies = [{"name": "ibm", "price": 134}, {"name": "msft", "price": 255}]

api = Flask( __name__ )

@api.route('/', methods=['GET'])
def get_companies():
  return json.dumps(companies)

@api.route('/', methods=['POST'])
def post_companies():
  name = request.args.get("name")
  price = request.args.get("price")
  print(json.dumps(request.json))
  return json.dumps(request.json), 201

if __name__ == ' __main__':
    api.run(host='0.0.0.0')

I ran the script on an EC2 instance and issued a GET request against post 5000 and here is the output:


[{"name": "ibm", "price": 134}, {"name": "msft", "price": 255}]

This means the webservice is working, so you can issue a POST command:


$ curl -X POST -H "Content-Type: application/json" -d '{"name":"nvda","price":"609"}' http://ec2-34-201-735.compute-1.amazonaws.com:5000

{"name": "nvda", "price": "609"}

So the POST command works as well.

REST consumer, queue binding, and RDP

As you can tell from the architecture diagram shown earlier, there is something different about the REST consumer. That’s because you need to create some additional objects on the broker to set it up. These objects are REST consumer, queue binding, and REST delivery point(RDP).

A REST consumer with Solace is not just a microservice that polls the broker with GET commands. That wouldn’t be appropriate given that Solace PubSub+ is all about event-driven architecture. Instead, PubSub+ utilizes REST WebHooks via POST commands so that the updates are pushed in real-time.

When the publisher publishes a message to a topic, depending on the topic subscriptions mapped to queues, they will be routed to the appropriate queues. For a REST endpoint to be invoked, you need to create a REST delivery endpoint (RDP) on the broker. The RDP will consist of a REST consumer which will contain information about the HTTP endpoint to invoke among other things and a queue binding which will bind a queue to the consumer. This queue binding will ensure that when a message is enqueued in the queue, the broker will invoke the endpoint with the correct method (POST or PUT).

Creating an RDP

To create a REST delivery endpoint on the broker, go to the PubSub+ Manager, click on the message VPN (i.e., default) and click on the Client Connections tab. On the following page, click on REST and then, click on + Rest Delivery Point.

Give your RDP a name, enable it, and click on apply.

Creating a REST consumer

Click on the RDP and then the REST Consumers tab. Then, click on + REST Consumer and give it a name. On the following page, enable the consumer and add host/port of the endpoint you wish to invoke when a message is received. In this case, that would be http://ec2-34-201-77-35.compute-1.amazonaws.com:5000/.

Select the appropriate method to invoke, which in this case is POST. There are a lot of other settings you can configure but for now, leave them as default.

Creating a queue binding

Go to Client Connections > Queue Bindings and click on + Queue Binding. Select the queue_rest_consumer queue and click on Create.

Since the specific endpoint is /, set Post Request Target to that.

With all the necessary objects created on the broker, you are ready to publish your first message via REST.

Java Consumer

You will be using sdkperf-jcsmp to spin up a Java consumer quickly via this command:


> bash sdkperf_java.sh -cip=localhost:55555 -sql=queue_java_consumer -md -pe

It will connect to the broker and bind to the queue_java_consumer queue you created earlier.

Running the Demo

Phew, everything is set and you’re ready to run the demo. Best practice is to run the consumers first. Your REST consumer is already running, so run the Java consumer with the command shown above.

Then, start the Java publisher which will start publishing to topic prices/java/stocks/aapl.

As soon as you run the Java publisher, you will notice that your Java consumer will start picking up the messages.

You should also see that the messages are making it to your queue queue_java_consumer.

Now, publish a message via REST using cURL.

As you can see, you immediately see the output on the webservice logs. This means that the message was published to the broker on the topic prices/rest/stocks/aapl and was enqueued in the queue queue_rest_consumer. As soon as the message was enqueued, the configured endpoint was invoked which resulted in the webservice logging that output.

Now, to make things interesting, you can change the topic that the Java publisher is publishing to so that those messages make it to the webservice as well. However, because you’re not setting a JSON payload for your sdkperf Java publisher, the logged output might not be pretty. Let’s see what happens!

I will use the same Java publisher but change the topic to prices/rest/stocks/aapl.

As you can see in the above screenshot, my messages published by Java publisher are not only making it to the Java consumer (as before) but also to the webservice!

Wrap-up

As you saw in the demo above, the system consisted of 2 publishers and 2 consumers. They were using REST and streaming APIs. You can get more creative as well and add publishers and consumers that use MQTT or AMQP.

The beauty of using an event broker is that it allows you to decouple your microservices. You can easily modify your architecture tomorrow by adding new microservices or removing old ones and it would have no impact on downstream consumers as long as they get the messages they are interested in consuming.

As you evolve your system from a batch-driven monolith to event-driven architecture consisting of microservices, remember that you might still need RESTful services (at least in the intermediate phase) which means you need to pick a broker that supports multiple protocols.

The post Demonstrating a Hybrid Architecture with RESTful and Streaming APIs appeared first on Solace.

PubSub+ Messaging Handling Features: Message Expiry

Leah Robert — Tue, 11 May 2021 13:15:07 +0000

Message handling features can help administrators and developers control how messages behave on the event broker and how they are received by consumers. This is the first in a series of blog posts in which I’ll introduce the various message handling features supported by Solace PubSub+ Event Brokers, starting with message expiry. Before reading this you should be familiar with both Solace endpoints and Solace persistent messaging.

What is Message Expiry?

Message expiry is a timer that dictates the lifespan of the message. The message will persist on an endpoint until that timer expires. Without this timer, a message will persist on a Solace endpoint indefinitely, or until it is consumed by an application or deleted by an administrator.

Message expiry is also commonly referred to as ‘message time-to-live’ or ‘message-TTL’.

How do I use message expiry with Solace PubSub+ Event Broker?

When considering message expiry, you need to consider a couple of required configurations. First, your endpoint must ‘Respect-TTL’ – this setting informs the endpoint to actively check messages for a TTL value. Next, you must set the TTL for the message.

There are two options to set the TTL:

Set ‘Maximum TTL’ on the endpoint. This will apply to all messages that are stored in the endpoint and is measured in seconds.
The publisher optionally sets the ‘Message-TTL’ on the message before sending it. This method allows the publisher to decide how long each message should live and if each message requires a TTL. This value is measured in milliseconds.

It is possible to set both the Message-TTL and the queue Max-TTL. The lower of the two is used for expiry.

Once a message has expired, the default behavior is that the message is deleted from the endpoint. It is also possible to have the message move to a Dead-Message-Queue, for special error or behaviour handling. When a message is expired from the endpoint, the discard statistics will reflect this.

What are some use cases for message expiry?

Message expiry, especially when set on the queue itself, is particularly useful in development environments where messages are sent to queues and are not consumed. To prevent the queue from filling up, you can use TTL to remove messages from a development queue as needed.

A use case example of message expiry could be an order management system that must process orders in a timely manner. For orders that are not processed in the desired amount of time, they could be expired to a Dead-Message-Queue for special, expedited processing or for error handling.

Another example could be an application that receives flight statuses. If a flight status is not consumed within a couple of hours, possibly due to the consuming application being unavailable, the flight status may no longer be relevant and is no longer required. The message could then be expired and removed from the endpoint.

If you found this post useful, visit the message properties section of our docs, and our PubSub+ for Developers page for more information. If you have any questions about queues and topic endpoints, post them to the Solace Developer Community.

The post PubSub+ Messaging Handling Features: Message Expiry appeared first on Solace.

The Relevance of RESTful Apps in Event-Driven Architecture

Himanshu Gupta — Thu, 29 Apr 2021 23:31:17 +0000

I love REST-based applications, especially when I am developing a new app and I need to get some data from another application. Learning from the team that their application has a REST API which is well documented and accessible via Swagger API is the best thing ever…besides having some sunny days after a freezing winter in NYC.

You take a look at the Swagger page, figure out the REST endpoints you need to hit to get the data you need, and you’re good to go! But as you start to work on integrating these REST calls in your shiny new application, you need to build yet another integration. You realize you have done this a few times already before. You have worked on numerous existing and new applications that need to either share data with other apps or consume data from them.

You’ve built such one-to-one integrations so many times that it’s a piece-of-cake by now, but it does take your valuable time and resources. You have to communicate with the other team, read the Swagger documentation, create JIRA tickets, have sprint meetings, test the API, implement failure logic (retries/time-outs) in case the endpoint is not available or is taking too long and, most annoyingly, poll for data at regular intervals.

Wouldn’t it be nice to not go through so much trouble when adding new microservices? You might still need to create JIRA tickets and hold sprint meetings, but many of the other steps can be avoided.

There are many use cases where REST is just the right way to build an application or microservice, but there is more and more demand for applications to become real-time. If your application is customer-facing, you know customers are demanding a more responsive, real-time service. You simply cannot afford to not process data in real-time anymore. Batch processing is just not sufficient in many modern use cases.

RESTful services, inherently, must poll for data as opposed to being executed/triggered based on an event. RESTful services are akin to the kid on a road trip continuously asking “Are we there yet?” “Are we there yet?” “Are we there yet?”, and just when you thought the kid had gotten some sense and would stop bothering you, he asks again: “Are we there yet?”

Additionally, RESTful services communicate _synchronously _as opposed to asynchronously. A synchronous call is blocking, which means your application cannot do anything but wait for the response. Alternatively, an asynchronous call is non-blocking providing your application with the freedom to continue processing. This can lead to much faster applications and improve customer experience.

Of course, there are some calls that must be synchronous. For example, if your app is responsible for opening a bank account, you need to run security/background checks before approving the account. Other services, however, like the one responsible for updating customer’s address, don’t demand immediate action and acknowledgement.

Advantage of Being Real-Time

Remember those pre-COVID days when we used to be able to fly? My favorite example is Delta’s mobile app — it’s one of the best apps of all the airlines I have flown. As soon as I am eligible for early check-in, I get a push notification with confirmation. As I pass through security, the app keeps me up to date about my plane’s status. If the gate gets changed at the last minute, I’m informed right away. Good thing, because it wouldn’t do me any good to know my gate changed from C-1 to B-10 thirty minutes after the fact if my flight’s door’s close in 20 minutes. If you check in your bag on the jetway because the flight is full, your app will confirm the scan, and when you land, you’ll get an alert letting you know which carrousel to pick up your bag from!

The key here is not that the information is made available, but that it’s made available in real time as events occur. That’s what makes a difference to the user experience, and I have picked Delta over other airlines simply due to their excellent mobile app and user experience.

To be able to provide such an experience, applications need to implement event-driven architecture using event broker (such as Solace PubSub+) and streaming APIs.

With such an architecture, your applications will publish data to your event broker in real time and your broker will be responsible for routing those events to any downstream subscriber(s). For example, as soon as I check in at the airport, an event can be published to a well-defined topic: delta/customer/{customerID}/{airportID}/checkedIn. That’s all that specific application needs to do. It doesn’t need to worry about calling some REST endpoint, waiting for it to reply back, figuring out failure/retry logic, etc. And, in the future, if it needs to publish the same event to another app, it doesn’t need to worry about integrating with another REST endpoint. In fact, it doesn’t need to do anything. The downstream subscriber simply subscribes to the topic and gets the event.

Once the event has been published, it is the router’s job to ensure it is delivered to all the downstream subscribers interested in this event. This is known as pub/sub messaging. Solace PubSub+ Event Broker supports guaranteed messaging which means zero message loss. If your subscribers are online, they can subscribe using wildcards such as delta/customer/> or delta/customer/*/JFK/> and the event(s) will be pushed to them. If one or more subscribers are not online for some reason, events will be enqueued in queues with ordering preserved. Whenever the subscriber comes online, it will simply consume those messages and pick up where it left off.

These are just some of the benefits of implementing event-driven architecture. There are plenty more such as scalability, agility, high-availability, and efficiency (via pub/sub). However, as with anything, make sure to do a deep dive, document all the requirements, and see if event-driven architecture makes sense for your use case.

Coexistence of REST APIs and Streaming APIs

As I mentioned earlier, REST APIs are not going anywhere. There are many use cases that don’t need the additional complexity of an event broker and streaming APIs. For example, many user-facing applications should be REST-based as it is the standard for web-based applications. For example, if I am accessing a website, I am simply submitting a GET request. If I am entering some user information via a form, it’s a POST request. If I am scanning a QR code to submit a payment, it is just another RESTful endpoint behind the scenes. The user will submit a request over REST which most likely goes to some API gateway and is then routed to a broker. In this case, our publisher is REST-based. However, our downstream subscribers can choose to use REST, streaming APIs, or both!

Modern hybrid architecture with both REST and streaming APIs

This is another benefit of implementing event-driven architecture via Solace PubSub+ Event Broker. It supports a variety of APIs and open protocols which means your applications can choose which language (i.e., Java, Python, C, etc.) or protocol (i.e., MQTT, AMQP, SMF, etc.) to use.

Thus, a modern architecture will most likely not be solely REST-based or event-driven but instead a combination of the two. As your architecture evolves from REST-based synchronous communication to real-time asynchronous communication, there will be some components of it that will continue being REST-based.

To support such a hybrid architecture, you will need an event broker that supports REST out-of-box without any plugins or any proxy services. It will need to handle protocol translation from REST to other open protocols such as MQTT and AMQP reliably.

If your use case does lend to full evolution from REST to streaming, then having a broker which supports REST protocol natively will make it much easier for you to migrate your services. You can select one service at a time to support streaming and others can continue using REST.

In a subsequent post, I will demo a hybrid architecture such as the one shown above. All you need to know for now is that as you consider moving to an event-driven architecture, you don’t and shouldn’t necessarily get rid of all your RESTful services. Your evolved architecture might be a hybrid one!

The post The Relevance of RESTful Apps in Event-Driven Architecture appeared first on Solace.

Event-Driven Architecture Myth Busting — Part 1: Five Common EDA Claims

Tamimi — Thu, 29 Apr 2021 13:50:09 +0000

Here is a claim that I am willing to drop: event-driven architecture (EDA) is filled with myths. Yep, I said it — and that’s a fact! Now that the elephant is out of the room, let me explain. In this blog post, I will be listing a couple of hand-picked claims about event-driven architecture and, you guessed it, I will either bust them or confirm them as fact. Let’s go ahead and delve further into this exploration with our first five EDA claims!

1. I will have to re-design my REST-heavy architecture from scratch to adopt EDA

Poll Results:

A digital transformation journey strategy explicitly takes that into account! It simply answers the question of “how can I liberate my already existing architecture with its legacy tools, tech, and protocols to modernize my infrastructure without throwing away what I already have”. Choosing an event broker to support an event-driven architecture implementation that natively supports protocol translation facilitates the adoption in existing architectures. If the event broker of choice natively supports sending/receiving REST messages, you’ll be able to integrate existing RESTful APIs with messaging APIs and protocols like JMS, MQTT, or AMQP without any extra external adapters, connectors, or bridges.

This is an extremely valuable asset to an event broker as it provides the organization with an architecturally simple way to distribute business events (order placed, payment initiated, room booked, etc.) to core APIs/services. And since digital transformation journeys to adopting event-driven architecture is an iterative process, this pattern of protocol translation support is valuable in the long-term as it provides a gradual path for adoption of an event-driven architecture. Check out these resources to see how you can become an event-driven enterprise in 3 steps or implement event-driven architecture in 6 steps.

Therefore, claiming that a complete redesign of a REST-only architecture is inevitable when adopting event-driven architecture is a complete… BUST!

2. APIs are only asynchronous in EDA

Poll Results:

A digital transformation journey to adopt event-driven architecture involves the process of event-enabling an already existing architecture. This could include a harmonious interaction between microservices and applications that use synchronous REST-based APIs and other components of the system that incorporate asynchronous message-oriented APIs.

A modern event broker should support both synchronous and asynchronous message exchange patterns. We’ve always heard the famous “it depends” answer when asking any architectural question about the choice of technology stack in an architecture. And it’s true – choosing a synchronous or asynchronous strategy to implement a particular feature or solve a specific problem does depend on the use-case. That’s why it’s so important to acknowledge that event-driven architecture does not replace synchronous call-and-response REST altogether, but complements it.

Hence, the myth that APIs are only asynchronous in event-driven architecture is… BUSTED!

3. Designing EDA involves lots of moving parts

Designing and implementing event-driven system requires a comprehensive strategy, and multiple tools. These tools are used to track all the events in the system, the different payload schemas, the documentation of the events, the versioning of the topics, and – on top of that – visualizing the overall event-driven architecture topology between the different application domains in your system.

Organizations use Microsoft Excel to track event topics, schema registries to track the payload schema, Visio diagrams to visualize the constantly changing event-driven architecture topology, wiki pages to document every moving part, and code repositories to store the application code.

Complete event management toolsets like event portals facilitate the design, documentation and governance of events and event-driven applications allowing easier collaboration between different stakeholders. Similar to an API portal that makes REST-focused application development easier, an event portal is a one-stop-shop for all things EDA, starting from designing topics, defining payload schemas, visualizing the topology, defining the contracts between applications and application domains within the organization, all the way to code generation from leveraging asynchronous API specifications like AsyncAPI.

Hence, the myth that designing event-driven architecture involves many moving parts is… CONFIRMED!

4. An event broker is the only thing you need to implement EDA

When deciding whether to adopt an EDA strategy, architects and developers should look at this decision through a platform lens instead of just evaluating event brokers. Event-driven architecture should be implemented with a toolset that enables data transformation, connectivity to a wide range of event sources, and tracing events through an event mesh that cannot be independently provided by an event broker.

Enabling event-driven architecture requires strategies and tools for documentation; governance to help determine the downstream impact of upcoming changes to apps; events and schemas; and topology visualization of many different microservices, along with the relationship of events connecting applications and application domains.

All this cannot be successfully implemented with a standalone event broker only, so the myth that an event broker are the only thing you need to implement event-driven architecture is… BUSTED!

5. Using Asynchronous API patterns means processing is done “later”

When implementing event-driven architecture, a developer can leverage several message exchange patterns, including “fire-and-forget” and request-reply. The fire-and-forget pattern is useful for keeping logs or events for later processing, such as analytics. On the other hand, the request-reply pattern is known for being inherently synchronous, meaning that further processing is blocked until the initial request is completed. I would even argue that implementing a request-reply behavior using messaging APIs is more powerful than REST APIs (obviously this is a biased opinion given my focus on messaging).

The request-reply pattern can be used in the case of an event consumer needing to request information from another provider, requiring a bi-directional exchange of information. Implementing this pattern involves:

A requestor publishing a message on topic A.
The message is then received by the provider that subscribed to topic A.
A response is then published by the provider on topic B.
The response is then received by the original requestor that subscribed on topic B.

Due to the asynchronous nature of the implementation, the requestor can engage in more than one asynchronous request-response conversation at one time.

So, the myth that using an asynchronous API means that processing is only done later is… BUSTED!

Stay tuned for more myth busting (or confirming) in Part 2!

The post Event-Driven Architecture Myth Busting — Part 1: Five Common EDA Claims appeared first on Solace.

See How PubSub+ Event Broker Handles Slow Consumers

arih1299 — Wed, 21 Apr 2021 13:10:16 +0000

If you use a message broker as a communication backbone for your enterprise applications and have had issues caused by a few slow consuming applications, then this is for you. I wrote a quick LinkedIn post about this when a friend asked me what’s so different with how Solace PubSub+ deals with slow consumers. I figured the best way to show this is by doing a quick demo with more than 50 GB messages queued in the broker due to slow consumption. But I want to show more of that demo here, so I can easily refer it back should someone ask me the same question again.

Event brokers are considered infrastructure. We don’t want infrastructure to keep us up at night worrying that we’ll need to force a restart and purge tens of gigabytes of customers’ transactions because some downstream applications slowed down or the traffic just spiked due to some new marketing promotions.

With this blog post I will explain and show how Solace PubSub+ Event Broker keeps on working as expected in the event a consumer slows down or fails so other applications connected to the broker are not affected. I will also show that a large number of pending messages will not impact the fail-over time needed for Solace PubSub+ Event Broker.

The Setup

For this demo, I’m using Solace PubSub+ Cloud simply because it’s the quickest way to go. It literally just took me just few clicks and I only needed to type the broker name. Of course we can set it up using the software on many other platforms if you so wish. The brokers are deployed on AWS and run on m5.xlarge EC2 instances. This is because the environment I’m using is not using the most recent and highly recommended Kubernetes-based deployment.

For the applications, I’m using SDKPerf to simulate the producers of events and the few differently paced events consumers, running on two t2.micro instances.

Figure 1: Create Solace Cloud service

Figure 2: SDKPerf EC2 instances for publishers and subscribers

The Test

Here I’m going to simulate a situation where a broker is serving multiple applications and some of these applications are slow and unable to keep up with their respective incoming traffic. You will be able to see that the broker and the other applications will not be impacted by the slow consumer.

I created three queues, each having different topic subscriptions. If you are not familiar with this concept, just consider that we have different queues with different incoming traffic. You can also take a look at a tutorial on this concept.

Figure 3: List of Queues

Figure 4: The first queue and its subscription

Figure 5: The second queue and its subscription for slow consumer test

Figure 6: The third queue and its subscriptions for a very slow consumer test

I have ten producers sending around up to 10,000 messages per second with a payload size of 1 KB to four different topics, which were then spooled or written into the three different queues we had created earlier. This is because we want to test the spooling or storing of the messages while the designated consumer applications can’t keep up with the pace to immediately process the incoming messages.

In normal condition, all three queues will not have many unprocessed messages at any time because all the consumer applications are able to catch up with the incoming traffic. In this case, the first test queue steadily flows messages in and out to its consumer.

Figure 7: Queue 1 with a consumer fast enough to keep up with incoming traffic

The second queue consumer is a rather slow consumer that is only able to process around 800 messages per second, which is around half of the incoming traffic rate. This means there are a lot of messages waiting to be consumed in this queue.

Figure 8: Queue 2 with a consumer capable of processing only half of the incoming traffic rate

The third queue has a much slower consumer, which can only take one message for every 10 seconds. Consider this as a sample of a normal application that suddenly has a problem and is almost unable to process any messages.

Figure 9: Queue 3 with a very slow consumer

As you can see here, the fast queue consumer is not affected by the other slow consumers. You should see some network limitation based on your environment setup, but the broker maintains good service level for the healthy consumers while keeping spooling messages for the slow consumers to process later on.

Note: This test was using a single t2.micro AWS EC2 instance to run all three consumer applications, and they are likely to hit the network bandwidth capacity of the instance.

Conclusion

The key point to show here is that any healthy consumer will not be affected by the other slow consumers and piling-up messages in the other queues. The Solace PubSub+ Event Broker maintains the service for the other clients while isolating the slow consumers. Compare this to other brokers that will slow down entirely, or even crash due to the building pressure and large spool size. If your broker slows down when messages start piling up due to a slow consumers, take a stab with this free broker.

Bonus Point: Safe, Predictable Failover

When the pile gets way too big, a broker may struggle even to fail-over to its standby/backup broker because the other broker needs to ‘load’ the pending messages before it can start taking new traffic. This usually forces administrators to reset the broker to zero pending messages to allow instant start-up and resume operations. The nice thing with Solace PubSub+ Event Broker is that it will take linear time of fail-over regardless of how big the pending messages pile is. I’ll demonstrate that by stopping the primary broker EC2 instance and see what happens.

Figure 10: The SDKPerf client lost connection after the primary broker shut down.

Figure 11: SDKPerf client got back connection on the 9th attempt of 1 second interval.

Figure 12: Immediately back to business with 50 GB pending messages

Above is the log from the SDKPerf client who is able to resume operations after 9th attempt to reconnect with a 1 second interval. It immediately has access to the same 50 GB messages pending earlier. This is simply because the standby broker keeps its own copy and always be ready to take over and not having to read the data of the primary broker when it’s time for it to take over activity.

I have also recorded this demo as a video for easier and more complete view of the action.

If you like it, give it a try from solace.dev and head over to the Solace Developer Community for any help needed or to share your quick wins!

The post See How PubSub+ Event Broker Handles Slow Consumers appeared first on Solace.

Developing an IoT Hub to Facilitate the Support of Your IoT Architecture

Julian Setiawan — Thu, 01 Apr 2021 13:44:26 +0000

Every quarter, Solace’s PubSub+ Cloud team participates in a two-day hackathon. It’s an opportunity to experiment with the Solace PubSub+ Platform, either by utilizing it in a novel way or extending the platform itself. It’s also a great way to work side by side with others on the team that we haven’t had a chance to work with and possibly dabble in other areas.

Our team’s goal for the hackathon was to provide an IoT hub where users only had to worry about writing code and could leave the other details to us. PubSub+ Event Broker is an extremely powerful and versatile broker with a staggering number of features across many protocols. Within two days we went from an idea to actually creating the IoT hub. This article shares how we used PubSub+ Event Broker: Cloud to develop the hub.

How the IoT Hub Works

The IoT Hub uses PubSub+ Cloud to provision event brokers. When a new device is registered on the hub, a new client username is created on the event broker.

This is as simple as adding a device from the IoT hub’s UI.

How We Developed the IoT Hub

We used the ACL substitution variables of PubSub+ to ensure that each device only has access to their own topic space. This was all done with a straightforward integration with the PubSub+ Cloud REST API and SEMP.

We then offered some useful shortcuts for developers who want to dive straight into coding. There are Java and Python snippets ready to download and run, pre-populated with your connection details.

For easier debugging, we also display recent events within your topic space and provide a way to publish events right from the hub.

./run.sh and you’re publishing and subscribing!

Now we needed devices to use the IoT hub. Our choice? Nerf guns.

We created an internet-connected target and score display. Using code samples and connection details from the hub, we were able to start iterating immediately.

When the target powers on, it sends a registration event to the hub. A micro switch push button is depressed when the target is hit. The button triggers “hit” events to be published to the hub. The score display receives the “hit” events and updates the score.

The hub gave us a fast and simple way to start publishing, subscribing, and seeing events. We were able to add the score display as a second device with the press of a button, giving it its own set of credentials. This quick validation allowed us to concentrate on the actual events instead of debugging connection parameters.

The Hackathon – Working with Students

Solace has participated in many university hackathons, exposing students to event-driven architecture and getting them to think beyond REST. However, many students don’t have much experience with event brokers.

Historically, we helped students sign up for PubSub+ Cloud and guided them through connecting their applications. One of the broker’s greatest assets of being language agnostic and multi-protocol ended up being overwhelming with so many different ways to connect.

We thought the IoT hub could resolve all of these. Choosing MQTT as the protocol, picking which libraries to use for the different languages, and setting up the connection parameters gave students a baseline functionality they could start with. As they explored more advanced use-cases, we could point them to PubSub+ Cloud, but they were armed with a working prototype.

In two days, we had a few tweaks to help us out during an upcoming university hackathon. First, we secured the dashboard with our company’s SSO. This allowed all the Solace volunteers to create new devices (i.e. credentials) and help students debug issues by viewing their connection details. Then, we added a way to generate unauthenticated, hard-to-guess links to the different devices so we could easily onboard the students. Finally, we polished up the interface, added more samples, and hosted the application online.

During the hackathon, we started seeing the benefits immediately. We were able to help students debug simple issues by checking out their connection details and using the hub to publish events to their applications. Instead of the usual questions about how to connect, we heard “How can we tell when a client disconnects?” (will messages) and “Can I publish with one protocol and subscribe with another?” (absolutely!). Without being bogged down by connection issues, students were starting to think beyond the basics and concentrating on the events!

Looking Beyond the Hackathon

The IoT Hub wasn’t the most elaborate use of what Solace has to offer, it was a few opinionated integrations with Solace products that are readily available to anyone. The only difference is that we have the valuable experience of advising countless customers on their event-driven architecture and guiding users to proven patterns.

We hope to bring these lessons to our products. Who knows, maybe sometime in the future you’ll see an “IoT Hub” icon appear in PubSub+ Cloud and you’ll be assured that you have the expertise of Solace powering your products.

The post Developing an IoT Hub to Facilitate the Support of Your IoT Architecture appeared first on Solace.

Introducing the new PubSub+ Messaging API for Python

Andrew MacKenzie — Thu, 18 Feb 2021 18:05:22 +0000

Solace is pleased to announce the release of version 1.0.0 of a new Python client API for PubSub+ messaging applications. The API is available on our downloads page or through PyPi. To install via pip, simply type: pip install solace-pubsubplus

Depending on whose list you look at, Python is anywhere from 1^st to 3^rd in popularity, demand for developers, and use in popular projects among programming languages. This new API fits in nicely with the existing Solace native APIs in C, Java, .Net/C#, and JavaScript/NodeJS.

While the popularity of Python was certainly a factor in creating this new API, as well as customer demand, it’s also important as a way of opening up support for a number of use cases popular in the Python community such as event streams to analytics packages for real-time machine learning, AI, and data analysis.

An event mesh using Solace event brokers enables the distribution and federation of streaming events across your organization, regardless of where the applications live or the language you’re using. With the addition of PubSub+ Messaging API for Python, the event mesh can deliver events to your Python applications in Pandas, NumPy, TensorFlow, Keras, and more. Data is the lifeblood of data science, and event streaming data that is distributed across your event mesh makes that data real-time, up-to-date, and fresh.

If you are looking for a tutorial to learn how to use the Solace PubSub+ Messaging API for Python, you can get started by:

Read our official documentation on our docs site
Access our samples, developer resources and API reference docs
Get help from your peers and participate in the discussions on the Solace Community

You can also subscribe to our channel on YouTube. There you will get a mix of topics such as using the PubSub+ Platform, event-driven architecture, customer testimonials, and, of course, technical content related to writing event-driven applications and the features of PubSub+ Event Broker.

Recently Tamimi, one of our terrific developer advocates, did a live coding session featuring an early access version of the Python API. It is well worth a view.

In addition to the resources above, it is interesting to look at ways that developers are using Python in event-driven applications in general and how they could be written using the Solace PubSub+ Messaging API for Python instead.

Just recently, Solace Developer Community member, Daniel Curtis wrote a blog post “Visualizing Live Streamed Data with Python and Solace PubSub+”. In it, Daniel articulates a use case of tracking NYC taxis using Pandas DataFrames, fed by a streaming subscription to a PubSub+ Event Broker topic, and visualized using Bokeh on a dashboard featuring a map. Daniel used the Paho MQTT Python API for this as the Solace PubSub+ Messaging API for Python was not yet available. He could just as easily have used the new API. The advantages of using Solace PubSub+ Messaging API for Python over an open standard protocol library like Paho MQTT is the ability to use advanced features of the broker that can’t be easily expressed using another open standard library.

Do you have an event-driven use case that would be best implemented in Python and fully utilize the power and breadth of the PubSub+ Platform? It is time to use our new PubSub+ Messaging API for Python!

And this 1.0.0 version of Solace PubSub+ Messaging API for Python is just the beginning. Future releases will add even more features such as request/reply helpers, Kerberos client authentication, session transactions and more. Solace developers are working hard on the Python API to enhance the feature set and cover more use cases. Stay tuned for future releases of this brand new API.

The post Introducing the new PubSub+ Messaging API for Python appeared first on Solace.