DEV Community: Himanshu Gupta

Solace PubSub+ vs Kafka: Implementation of the Publish-Subscribe Messaging Pattern

Himanshu Gupta — Tue, 08 Jun 2021 19:09:06 +0000

This blog post is the second part of a series comparing Apache Kafka with Solace’s PubSub+ Event Broker. In the first, my colleague Ush introduced the two very different internal architectures that Kafka and Solace PubSub+ have. In this post, I’ll explain how pub/sub messaging pattern implementation differs in these two brokers.

Publish/Subscribe, commonly known as pub/sub, is a popular messaging pattern which is commonly used in today’s systems to help them efficiently distribute data and scale among other things. The pub/sub messaging pattern can be easily implemented through an event broker such as Solace PubSub+, Kafka, RabbitMQ, and ActiveMQ.

When you use an event broker, you have a set of applications known as producers and another set of applications known as consumers. Producers are responsible for publishing data to the broker and similarly, consumers are responsible for consuming data from the event broker. By introducing a broker to our architecture, we have removed the need for producers to directly communicate with the consumers. This ensures we have a loosely coupled architecture. Additionally, our broker now becomes responsible for managing connections, security, and subscription interests, instead of implementing this logic in the applications themselves.

Different event brokers implement the pub/sub messaging pattern differently. It is important to understand their implementations when deciding which event broker to use for your specific use case.

Pub/Sub in Kafka

To understand Kafka’s implementation of pub/sub, you need to understand topics, producers, and consumers.

Topics

As described in part one, in Kafka messages are produced and consumed via topics. They are immutable log files, persisted on disk, to which data is appended sequentially. Given that Kafka is a distributed system, you need to create topics and decide how you want to partition and replicate them across your brokers. While topics can be created by default if they don’t exist, it is best practice to manually create them with appropriate settings before using them.

Producer

A Kafka producer can be programmed to write data to a topic. If the topic has 3 partitions, data will be written to all 3 partitions in a round-robin fashion. This leads to a significant problem. Because our data is scattered across multiple partitions, we don’t get ordering across partitions. Additionally, Kafka publisher applications batch writes to improve performance as they publish data to Kafka brokers.

Consumer

If we want to consume the data we just wrote to our topic, we will need to create a consumer application and connect to our Kafka cluster. Your Kafka consumer can easily subscribe to a topic and consume the necessary data. However, as mentioned earlier, there is an issue caused by topic partitions. Because our data was written to 3 partitions, we have lost message ordering. When our consumer subscribes to the topic and consumes the data, it will get unordered messages. Depending on your system, this can be a critical issue.

To overcome this issue, you will need to use a key when publishing data so that all the messages pertaining to a specific key will always go to the same partition and hence preserve ordering. However, as you may have guessed already, with this workaround, you lose simple the ability to have balanced parallelization resulting in some partitions overflowing while others will be lightly used.

Furthermore, in Kafka, messages are polled instead of pushed to the consumer. When you program your consumer application, you are expected to provide a timer to constantly poll for data. As you can imagine, this can be highly inefficient if your application is frequently polling for data, especially when there is no data available.

Pub/Sub in Solace PubSub+

Solace’s PubSub+ Event Broker is an enterprise-grade event broker which supports open APIs and protocols. It comes with numerous enterprise-grade features straight out-of-box such as high-availability, disaster recovery, and security so enterprises don’t have to implement them.

Topics

As you can see in the previous part, the topics of PubSub+ are very different from those of Kafka. Unlike Kafka’s topics, PubSub+ topics are dynamic and don’t need to be manually created. They are simply metadata that you use to describe your messages as you publish them. Additionally, PubSub+ topics are hierarchical, which means consumers can use wildcards to filter based on different levels in a topic.

PubSub+ topics don’t need to be partitioned or replicated. The fact that they don’t need to be created and persisted to disk means that they barely require any maintenance. As your system scales, there is no need to rebalance your topics as you do in Kafka.

Publisher

PubSub+ publishers can be coded using any of the supported open APIs and protocols such as JMS, MQTT, AMQP, and REST. To publish data, your publisher application just needs to specify which topic to publish the data to. And that’s it.

Subscriber

Similar to Kafka, PubSub+ subscribers can be coded to consume from a topic by specifying the entire topic. Or you can subscribe to multiple topics using wildcards (* and >). Moreover, you can use these wildcards to dynamically filter data.

For example, if your publisher is publishing data for all US stocks from different exchanges to topics such as EQ/US/NYSE/AAPL and EQ/US/NASDAQ/FB, my consumer can subscribe to different datasets by subscribing to any of the following topics:

EQ/> – for all equities stocks
*/*/NYSE/> – for all stocks traded on NYSE
*/*/*/FB – for all messages for Facebook traded on different exchanges

It should be noted that with this approach PubSub+ Event Broker efficiently filters the datastreams to the clients as opposed to Kafka, where filtering happens on the client side or via an intermediary KStreams process. This saves both network bandwidth and compute resources on your consumer processes.

Furthermore, PubSub+ subscribers are push based instead of poll based. Messages will be pushed to subscribers rather than constantly poll for data.

Conclusion

To sum it up, the pub/sub messaging pattern implementation exists in both Solace PubSub+ and Kafka but they differ significantly. While PubSub+ Event Broker utilizes dynamic, hierarchical topics with wildcard support, Kafka uses flat topics partitioned across different brokers. Additionally, Solace PubSub+ Event Broker supports open APIs and protocols which prevents vendor lock-in whereas while Kafka is open-source, it does not support open APIs and protocols natively.

In the next post, we will discuss how filtering is achieved in both brokers.

The post Solace PubSub+ vs Kafka: Implementation of the Publish-Subscribe Messaging Pattern appeared first on Solace.

Demonstrating a Hybrid Architecture with RESTful and Streaming APIs

Himanshu Gupta — Wed, 19 May 2021 13:00:53 +0000

In my last post I introduced the importance of event-driven architecture and endorsed a hybrid architecture consisting of both RESTful and streaming APIs. Now I’d like to walk you through a simple demo that shows the benefit of using a broker, such as Solace PubSub+ Event Broker, that can handle multiple open protocols and APIs.

Besides allowing microservices to publish and consume events in real time, a major benefit of using an event broker is integration. A typical enterprise has all kinds of in-house and vendor products with different APIs and it can be extremely painful to get these applications to talk to each other.

A modern event broker must support multiple open APIs and protocols without having to install new plug-ins or manage proxies. For example, PubSub+ Event Broker supports open protocols like AMQP, MQTT, REST, WebSocket, and APIs in multiple languages, which means you can have one application publishing data via REST and downstream applications consuming that data via AMQP and/or WebSocket. And if tomorrow, you onboard a new vendor application that supports MQTT only then you can easily use MQTT to stream data to the event broker.

Hybrid Architecture

In the previous post, I explained our hybrid architecture in which data is published to the event broker via REST and the broker is responsible for translating that event to the protocol desired by the downstream services.

This demo consists of the following components:

Solace PubSub+ Event Broker
Publishers
1. REST publisher; simple REST POST command(s) via cURL
2. Java publisher; using streaming API
Consumers
1. REST WebHooks; an endpoint will be invoked every time a message is received/enqueued.
2. Java consumer; using MQTT protocol

Here is what it will look like:

As you can see in the diagram, there are some objects created in the event broker as well. There are two types of delivery modes supported by PubSub+: direct and guaranteed. In direct messaging, there is no persistence. It is popularly used for high-throughput and low latency use cases where you can afford some data loss. For critical data, where you want zero message loss, you need guaranteed messaging.

In this example, you’ll use guaranteed messaging and apply persistence via a queue. Our two publishers will publish messages to a well-defined topic with multiple levels (i.e., prices/java/stocks/{ticker}). This allows our subscribers to consume the messages via topic subscriptions mapped to queues. For example, thre’s a queue called queue_java_consumer for java consumer and added a topic subscription prices/>. This will allow messages published by both REST and Java publishers to be enqueued in this queue.

Similarly, there’s a separate queue for our REST consumer called queue_rest_consumer with a different topic subscription prices/rest/>. This means only messages published by the REST publisher will be enqueued in this queue.

Queues

Before you start setting up our publishers and consumers, let’s create the queues that will hold the messages based on topic subscriptions mapped to them. Create two queues:

queue_java_consumer with topic prices/> mapped to it
queue_rest_consumer with topic prices/rest/ mapped to it

You can create a queue by going to the Queues tab on PubSub+ Manager and clicking on + Queue. Once the queue has been created, click on it and add the appropriate subscription.

Publishers

REST publisher

You can publish messages to the broker via cURL commands by executing a POST command against a URL which will contain the host and port of the broker and REST service running on the broker. It will also contain the topic address you want to publish the message to. For example, to publish a message to a local broker running to the topic prices/rest/stocks/aapl, run this command:


curl -X POST http://127.0.0.1:9000/prices/rest/stocks/aapl --header "Content-Type: application/json" -d '{"name":"aapl", "price":"130"}'

The URL that’s used contains the host, port, and topic address in this syntax: http(s)://:/

You can find more information about REST publishers here.

Java publisher

Depending on the protocol you want to use (SMF, MQTT, AMQP), you can use the corresponding API. Solace has a command-line utility tool for demos and tests, called sdkperf, freely available for download here. It comes in different flavors based on the protocol you want to use. This demo uses sdkperf-jcsmp that uses Solace’s SMF protocol. Here is a guide on how to use sdkperf.

Here is the command that starts the Java publisher which will connect to the local broker and publish 10,000 messages at the rate of 1 message per second to the topic prices/java/stocks/aapl.

> bash sdkperf_java.sh -ptl="prices/java/stocks/aapl" -mt=persistent -mn=10000 -mr=1 -msa=100 -cip=localhost:55555

Consumers

REST consumer

The REST consumer will be a webservice endpoint which is invoked every time the queue receives a message. You will first need to have a webservice which accepts POST commands.

Webservice

Let’s spin up a webservice which will expose a REST endpoint. This endpoint will just echo the arguments once invoked. Here is sample python code to spin up the webservice:


from flask import Flask, json, request
from flask import *

companies = [{"name": "ibm", "price": 134}, {"name": "msft", "price": 255}]

api = Flask( __name__ )

@api.route('/', methods=['GET'])
def get_companies():
  return json.dumps(companies)

@api.route('/', methods=['POST'])
def post_companies():
  name = request.args.get("name")
  price = request.args.get("price")
  print(json.dumps(request.json))
  return json.dumps(request.json), 201

if __name__ == ' __main__':
    api.run(host='0.0.0.0')

I ran the script on an EC2 instance and issued a GET request against post 5000 and here is the output:


[{"name": "ibm", "price": 134}, {"name": "msft", "price": 255}]

This means the webservice is working, so you can issue a POST command:


$ curl -X POST -H "Content-Type: application/json" -d '{"name":"nvda","price":"609"}' http://ec2-34-201-735.compute-1.amazonaws.com:5000

{"name": "nvda", "price": "609"}

So the POST command works as well.

REST consumer, queue binding, and RDP

As you can tell from the architecture diagram shown earlier, there is something different about the REST consumer. That’s because you need to create some additional objects on the broker to set it up. These objects are REST consumer, queue binding, and REST delivery point(RDP).

A REST consumer with Solace is not just a microservice that polls the broker with GET commands. That wouldn’t be appropriate given that Solace PubSub+ is all about event-driven architecture. Instead, PubSub+ utilizes REST WebHooks via POST commands so that the updates are pushed in real-time.

When the publisher publishes a message to a topic, depending on the topic subscriptions mapped to queues, they will be routed to the appropriate queues. For a REST endpoint to be invoked, you need to create a REST delivery endpoint (RDP) on the broker. The RDP will consist of a REST consumer which will contain information about the HTTP endpoint to invoke among other things and a queue binding which will bind a queue to the consumer. This queue binding will ensure that when a message is enqueued in the queue, the broker will invoke the endpoint with the correct method (POST or PUT).

Creating an RDP

To create a REST delivery endpoint on the broker, go to the PubSub+ Manager, click on the message VPN (i.e., default) and click on the Client Connections tab. On the following page, click on REST and then, click on + Rest Delivery Point.

Give your RDP a name, enable it, and click on apply.

Creating a REST consumer

Click on the RDP and then the REST Consumers tab. Then, click on + REST Consumer and give it a name. On the following page, enable the consumer and add host/port of the endpoint you wish to invoke when a message is received. In this case, that would be http://ec2-34-201-77-35.compute-1.amazonaws.com:5000/.

Select the appropriate method to invoke, which in this case is POST. There are a lot of other settings you can configure but for now, leave them as default.

Creating a queue binding

Go to Client Connections > Queue Bindings and click on + Queue Binding. Select the queue_rest_consumer queue and click on Create.

Since the specific endpoint is /, set Post Request Target to that.

With all the necessary objects created on the broker, you are ready to publish your first message via REST.

Java Consumer

You will be using sdkperf-jcsmp to spin up a Java consumer quickly via this command:


> bash sdkperf_java.sh -cip=localhost:55555 -sql=queue_java_consumer -md -pe

It will connect to the broker and bind to the queue_java_consumer queue you created earlier.

Running the Demo

Phew, everything is set and you’re ready to run the demo. Best practice is to run the consumers first. Your REST consumer is already running, so run the Java consumer with the command shown above.

Then, start the Java publisher which will start publishing to topic prices/java/stocks/aapl.

As soon as you run the Java publisher, you will notice that your Java consumer will start picking up the messages.

You should also see that the messages are making it to your queue queue_java_consumer.

Now, publish a message via REST using cURL.

As you can see, you immediately see the output on the webservice logs. This means that the message was published to the broker on the topic prices/rest/stocks/aapl and was enqueued in the queue queue_rest_consumer. As soon as the message was enqueued, the configured endpoint was invoked which resulted in the webservice logging that output.

Now, to make things interesting, you can change the topic that the Java publisher is publishing to so that those messages make it to the webservice as well. However, because you’re not setting a JSON payload for your sdkperf Java publisher, the logged output might not be pretty. Let’s see what happens!

I will use the same Java publisher but change the topic to prices/rest/stocks/aapl.

As you can see in the above screenshot, my messages published by Java publisher are not only making it to the Java consumer (as before) but also to the webservice!

Wrap-up

As you saw in the demo above, the system consisted of 2 publishers and 2 consumers. They were using REST and streaming APIs. You can get more creative as well and add publishers and consumers that use MQTT or AMQP.

The beauty of using an event broker is that it allows you to decouple your microservices. You can easily modify your architecture tomorrow by adding new microservices or removing old ones and it would have no impact on downstream consumers as long as they get the messages they are interested in consuming.

As you evolve your system from a batch-driven monolith to event-driven architecture consisting of microservices, remember that you might still need RESTful services (at least in the intermediate phase) which means you need to pick a broker that supports multiple protocols.

The post Demonstrating a Hybrid Architecture with RESTful and Streaming APIs appeared first on Solace.

The Relevance of RESTful Apps in Event-Driven Architecture

Himanshu Gupta — Thu, 29 Apr 2021 23:31:17 +0000

I love REST-based applications, especially when I am developing a new app and I need to get some data from another application. Learning from the team that their application has a REST API which is well documented and accessible via Swagger API is the best thing ever…besides having some sunny days after a freezing winter in NYC.

You take a look at the Swagger page, figure out the REST endpoints you need to hit to get the data you need, and you’re good to go! But as you start to work on integrating these REST calls in your shiny new application, you need to build yet another integration. You realize you have done this a few times already before. You have worked on numerous existing and new applications that need to either share data with other apps or consume data from them.

You’ve built such one-to-one integrations so many times that it’s a piece-of-cake by now, but it does take your valuable time and resources. You have to communicate with the other team, read the Swagger documentation, create JIRA tickets, have sprint meetings, test the API, implement failure logic (retries/time-outs) in case the endpoint is not available or is taking too long and, most annoyingly, poll for data at regular intervals.

Wouldn’t it be nice to not go through so much trouble when adding new microservices? You might still need to create JIRA tickets and hold sprint meetings, but many of the other steps can be avoided.

There are many use cases where REST is just the right way to build an application or microservice, but there is more and more demand for applications to become real-time. If your application is customer-facing, you know customers are demanding a more responsive, real-time service. You simply cannot afford to not process data in real-time anymore. Batch processing is just not sufficient in many modern use cases.

RESTful services, inherently, must poll for data as opposed to being executed/triggered based on an event. RESTful services are akin to the kid on a road trip continuously asking “Are we there yet?” “Are we there yet?” “Are we there yet?”, and just when you thought the kid had gotten some sense and would stop bothering you, he asks again: “Are we there yet?”

Additionally, RESTful services communicate _synchronously _as opposed to asynchronously. A synchronous call is blocking, which means your application cannot do anything but wait for the response. Alternatively, an asynchronous call is non-blocking providing your application with the freedom to continue processing. This can lead to much faster applications and improve customer experience.

Of course, there are some calls that must be synchronous. For example, if your app is responsible for opening a bank account, you need to run security/background checks before approving the account. Other services, however, like the one responsible for updating customer’s address, don’t demand immediate action and acknowledgement.

Advantage of Being Real-Time

Remember those pre-COVID days when we used to be able to fly? My favorite example is Delta’s mobile app — it’s one of the best apps of all the airlines I have flown. As soon as I am eligible for early check-in, I get a push notification with confirmation. As I pass through security, the app keeps me up to date about my plane’s status. If the gate gets changed at the last minute, I’m informed right away. Good thing, because it wouldn’t do me any good to know my gate changed from C-1 to B-10 thirty minutes after the fact if my flight’s door’s close in 20 minutes. If you check in your bag on the jetway because the flight is full, your app will confirm the scan, and when you land, you’ll get an alert letting you know which carrousel to pick up your bag from!

The key here is not that the information is made available, but that it’s made available in real time as events occur. That’s what makes a difference to the user experience, and I have picked Delta over other airlines simply due to their excellent mobile app and user experience.

To be able to provide such an experience, applications need to implement event-driven architecture using event broker (such as Solace PubSub+) and streaming APIs.

With such an architecture, your applications will publish data to your event broker in real time and your broker will be responsible for routing those events to any downstream subscriber(s). For example, as soon as I check in at the airport, an event can be published to a well-defined topic: delta/customer/{customerID}/{airportID}/checkedIn. That’s all that specific application needs to do. It doesn’t need to worry about calling some REST endpoint, waiting for it to reply back, figuring out failure/retry logic, etc. And, in the future, if it needs to publish the same event to another app, it doesn’t need to worry about integrating with another REST endpoint. In fact, it doesn’t need to do anything. The downstream subscriber simply subscribes to the topic and gets the event.

Once the event has been published, it is the router’s job to ensure it is delivered to all the downstream subscribers interested in this event. This is known as pub/sub messaging. Solace PubSub+ Event Broker supports guaranteed messaging which means zero message loss. If your subscribers are online, they can subscribe using wildcards such as delta/customer/> or delta/customer/*/JFK/> and the event(s) will be pushed to them. If one or more subscribers are not online for some reason, events will be enqueued in queues with ordering preserved. Whenever the subscriber comes online, it will simply consume those messages and pick up where it left off.

These are just some of the benefits of implementing event-driven architecture. There are plenty more such as scalability, agility, high-availability, and efficiency (via pub/sub). However, as with anything, make sure to do a deep dive, document all the requirements, and see if event-driven architecture makes sense for your use case.

Coexistence of REST APIs and Streaming APIs

As I mentioned earlier, REST APIs are not going anywhere. There are many use cases that don’t need the additional complexity of an event broker and streaming APIs. For example, many user-facing applications should be REST-based as it is the standard for web-based applications. For example, if I am accessing a website, I am simply submitting a GET request. If I am entering some user information via a form, it’s a POST request. If I am scanning a QR code to submit a payment, it is just another RESTful endpoint behind the scenes. The user will submit a request over REST which most likely goes to some API gateway and is then routed to a broker. In this case, our publisher is REST-based. However, our downstream subscribers can choose to use REST, streaming APIs, or both!

Modern hybrid architecture with both REST and streaming APIs

This is another benefit of implementing event-driven architecture via Solace PubSub+ Event Broker. It supports a variety of APIs and open protocols which means your applications can choose which language (i.e., Java, Python, C, etc.) or protocol (i.e., MQTT, AMQP, SMF, etc.) to use.

Thus, a modern architecture will most likely not be solely REST-based or event-driven but instead a combination of the two. As your architecture evolves from REST-based synchronous communication to real-time asynchronous communication, there will be some components of it that will continue being REST-based.

To support such a hybrid architecture, you will need an event broker that supports REST out-of-box without any plugins or any proxy services. It will need to handle protocol translation from REST to other open protocols such as MQTT and AMQP reliably.

If your use case does lend to full evolution from REST to streaming, then having a broker which supports REST protocol natively will make it much easier for you to migrate your services. You can select one service at a time to support streaming and others can continue using REST.

In a subsequent post, I will demo a hybrid architecture such as the one shown above. All you need to know for now is that as you consider moving to an event-driven architecture, you don’t and shouldn’t necessarily get rid of all your RESTful services. Your evolved architecture might be a hybrid one!

The post The Relevance of RESTful Apps in Event-Driven Architecture appeared first on Solace.

Publishing Real-Time Tweets to Solace PubSub+ Event Broker

Himanshu Gupta — Wed, 02 Dec 2020 16:17:49 +0000

If you want to work with real-time streaming data, the two most popular feeds you will come across are Wikipedia updates and Twitter tweets. While capturing Wikipedia updates used to be quite trendy a few years ago, recently, capturing tweets from Twitter and running sentiment analysis on this has become trendier.

In this post, I would like to show you how you can write a Twitter publisher that will use Twitter’s API (under the hood) to get tweets as they are published in real-time and publish them to Solace PubSub+ Event Broker using JMS.

You can find my code on GitHub.

Twitter/PubSub+ Connection Settings

My Twitter publisher is based on Hosebird Client (github) by Twitter which is a Java HTTP client for consuming Twitter’s standard Streaming API. The client is easy to use and if you follow the instructions/explanation on README.md, you will be able to understand how to set it up.

To be able to access Twitter’s API, you will need to get a Twitter developer account and register your app. Once you have been approved, you will have access to consumerKey, consumerSecret, token, and tokenSecretthat you will need to populate in resources/twitter.yaml file.

Similarly, you will need to populate your Solace PubSub+ connection information in resources/broker.yaml. You will need host, vpn, username, and password. You can easily spin up a PubSub+ instance using docker container or through PubSub+ Event Broker: Cloud (free tier account with 60-day free trial).

Subscribing to Tweets

The Twitter API allows you to subscribe to multiple keywords and get real-time tweets that contain those keywords. Since we are going through a pandemic right now, I have decided to pick covid-19 and coronavirus as my keywords. You can pick different keywords.

List<String> terms = Lists.newArrayList("covid-19", "coronavirus"); hosebirdEndpoint.trackTerms(terms);

Publishing to PubSub+ Topic

As soon as we connect, we will start receiving relevant tweets as JSON strings. We are interested in publishing these tweets to topics on Solace PubSub+ Event Broker. Since PubSub+ supports rich hierarchical topics and dynamic filtering using wildcards, we should put useful information on our topics which the consumers can use to filter the data.

With this in mind, I will use “tweets/v1/covid/” + language of the tweet as my topic structure. Each tweet has a field called langwhich tells you the language of the tweet. For example, if the tweet is in English, value of langwill be en. So, for tweets that are in English, they will be published to topic: tweets/v1/covid/en. If a tweet is in French, it will be published to: tweets/v1/covid/fr.

if(msg!=null){     JSONObject jsonMsg = (JSONObject) parser.parse(msg);     String lang = (String) jsonMsg.get("lang");     publishMessageToSolace(session, msg, "tweets/v1/covid/" + lang);      // Control the msg publish rate     Thread.sleep(1000); }

Why did I include language in the topic? I did so because later, I would like my consumers to have the flexibility to filter tweets based on language. For example, I might only want tweets that are in English. With Solace’s support for rich hierarchical topics, we can dynamically filter data using wildcards. My consumer can simply subscribe to tweets/*/*/en and get only English tweets. If my consumer wanted tweets in all languages, it can subscribe to tweets/>.

That’s pretty much it. To get started with my code, you have to add your twitter and broker information in the respective config files, decide which keywords you want to get tweets for, and which topic you want to publish messages to.

Running the Code

Once you start the publisher, you will notice the following output (via IntelliJ):

As you can see, we are getting tweets in different languages: en, pt, nl.

However, our subscriber can decide which topic to subscribe to and here is an example of a subscriber (via Solace’s Try Me! app on the management UI) subscribing to just English tweets via tweets/v1/covid/en topic.

Once you have these tweets being distributed over Solace PubSub+ Event Broker, your applications can use any of the different open APIs and protocols Solace supports to consume this data. For example, you can even create an event mesh and stream this data to an ML application in AWS or GCP to perform sentiment analysis!

The post Publishing Real-Time Tweets to Solace PubSub+ Event Broker appeared first on Solace.

Achieving Event-Driven Architecture with Solace PubSub+ Event Broker and MuleSoft

Himanshu Gupta — Sat, 01 Aug 2020 16:28:41 +0000

One of the benefits of my current job is that I get to work with companies from different industries. As someone who had only worked in financial services before, it has been a very rewarding experience to understand the kinds of challenges companies face in other industries and how they compare to challenges commonly found in financial services. Additionally, I also get to learn about their technology stack.

Recently, I had the opportunity to work with a popular iPaaS (integration platform as a service) solution called MuleSoft which was acquired by Salesforce. MuleSoft, like other integration solutions, makes it as easy as dragging and dropping connectors to connect your applications. Such tools are heavily used to integrate applications from different domains together. For example, you can link your CRM app to your marketing app, invoice app, and analytics app with a simple drag-and-drop.

As you can probably guess, iPaaS tools are built on synchronous RESTful APIs. For your marketing app to be integrated with your CRM app, you need to have appropriate RESTful endpoints. While this works fine, we know that synchronous REST calls have their limitations when compared with the asynchronous calls of event-driven architecture. Instead of polling for data, you would rather have your apps sharing messages via the publish/subscribe messaging that enables event-driven architecture as part of your Mulesoft environment.

What are the advantages of event-driven architecture and publish/subscribe messaging?

You have undoubtedly heard of these terms before because, in the last few years, both have gotten extremely popular. Our world is full of events so your applications need to be event-driven. When a credit card is swiped, an event is fired which triggers a bunch of other downstream events. For example, your phone gets a real-time notification informing you that your card was just swiped, your credit card balance is updated, and so forth.

I have previously written a post about the nature and advantages of event-driven architecture, but here are the main ones:

Perform analytics and actions on real-time data as opposed to stale data obtained from batch processing
Identify issues in real-time instead of waiting till the batch is executed
Push vs pull – events are pushed to your applications as opposed to constantly polling for updates
Loosely coupled applications
Easier to scale
Typical pub/sub benefits such as the efficient distribution of data, persistence, replay, and migration to the cloud

Implementing Event-Driven Architecture with MuleSoft and Solace PubSub+

Now that we have a better idea of why you should consider using pub/sub messaging with your iPaaS, let’s look at how to achieve Mulesoft event-driven architecture. We will use MuleSoft’s Anypoint Studio which you can download for free (there may be a 30-day limit).

For our broker, we will be using PubSub+ Event Broker deployed via Docker container on my laptop, but you can also use PubSub+ Event Broker: Cloud which has a free tier. PubSub+ supports open APIs and protocols, so it’s really easy to use it with other products. For example, to integrate it with Mulesoft, we will be using Mulesoft’s JMS module.

Use Case

We have a CRM system such as Salesforce which pushes updates to a REST endpoint whenever there is an account update such as an account being created. The update contains high-level information about the account itself (name, website etc.), contact information, contract information, and location information (address). We have downstream applications such as marketing app, invoice app, and analytics app which are interested in one or more of these types of information contained in the entire account update.

Here is what our downstream applications are interested in:

Analytics app – interested in high-level account and contract information
Invoice app – interested in contract information only
Marketing app – interested in contact and location information

Our goal is to digest the original payload (XML schema), parse it, and split it into 4 different smaller payloads:

High-level account information
Contact information
Location information
Contract information

Then, we need to publish this data on different topics and have different consumers subscribe to this data depending on their interest.

Notice that despite having multiple apps subscribing to the same data (both analytics and invoice apps are interested in _contrac_t data), we only need to publish it once. This is one of the key benefits of using pub/sub messaging pattern. While it may not seem like a major benefit in our use case, it definitely makes a difference when you are dealing with high data volumes.

Additionally, we are able to dynamically filter the dataset using PubSub+’s rich hierarchical topics instead of having all our applications consume the same data and then having to filter themselves.

Schemas and Topics

One of the benefits of using an iPaaS such as MuleSoft is that it allows you to transform your data, which can come in very handy in event-driven architecture. In our case, we will be ingesting XML payload but the output will be in JSON.

Here is the schema of the original payload (topic: company/sales/salesforce/customerAccount/all/created/v1/{accountId}):

<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:element name="AccountId" type="xs:string"/> <xs:element name="AccountName" type="xs:string"/> <xs:element name="AccountSource" type="xs:string"/> <xs:element name="AnnualRevenue" type="xs:int"/> <xs:element name="BillingCountryCode" type="xs:string"/> <xs:element name="BillingState" type="xs:string"/> <xs:element name="CreatedDate" type="xs:dateTime"/> <xs:element name="CurrencyIsoCode" type="xs:string"/> <xs:element name="IsActive" type="xs:string"/> <xs:element name="IsArchived" type="xs:string"/> <xs:element name="IsDeleted" type="xs:string"/> <xs:element name="LastModifiedById" type="xs:string"/> <xs:element name="LastModifiedDate" type="xs:dateTime"/> <xs:element name="LastReferencedDate" type="xs:dateTime"/> <xs:element name="LastViewedDate" type="xs:dateTime"/> <xs:element name="ContactName" type="xs:string"/> <xs:element name="ContactId" type="xs:int"/> <xs:element name="ContactEmail" type="xs:string"/> <xs:element name="Description" type="xs:string"/> <xs:element name="Industry" type="xs:string"/> <xs:element name="NumberOfEmployees" type="xs:short"/> <xs:element name="Type" type="xs:string"/> <xs:element name="BillingAddress" type="xs:string"/> <xs:element name="Website" type="xs:anyURI"/> <xs:element name="ProductCode" type="xs:int"/> <xs:element name="ContractNumber" type="xs:int"/> <xs:element name="ContractAddress" type="xs:string"/> <xs:element name="Sum\_Units\_Sold\_\_c" type="xs:float"/> <xs:element name="SystemModstamp" type="xs:dateTime"/></xs:schema>

Here are the schemas for the 4 payloads after they have been parsed:

High-level account (topic: company/sales/salesforce/customerAccount/account/created/v1/{accountId}):

{ "$schema": "http://json-schema.org/draft-04/schema#", "type": "object", "properties": { "Body": { "type": "object", "properties": { "AccountId": { "type": "string" }, "AccountName": { "type": "string" }, "Website": { "type": "string" } }, "required": ["AccountId", "AccountName", "Website"] } }, "required": ["Body"]}

Contact (topic: company/sales/salesforce/customerAccount/contact/created/v1/{accountId}):

{ "$schema": "http://json-schema.org/draft-04/schema#", "type": "object", "properties": { "Body": { "type": "object", "properties": { "ContactName": { "type": "string" }, "ContactId": { "type": "string" }, "ContactEmail": { "type": "string" } }, "required": ["ContactName" "ContactId", "ContactEmail"] } }, "required": ["Body"]}

Location (topic: company/sales/salesforce/customerAccount/location/created/v1/{accountId}):

` { "$schema": "http://json-schema.org/draft-04/schema#", "type": "object", "properties": { "Body": { "type": "object", "properties": { "BillingCountryCode": { "type": "string" }, "BillingState": { "type": "string" }, "BillingAddress": { "type": "string" } }, "required": ["BillingCountryCode", "BillingState", "BillingAddress",] } }, "required": ["Body"]} `

Contract (topic: company/sales/salesforce/customerAccount/contract/created/v1/{accountId}):

` { "$schema": "http://json-schema.org/draft-04/schema#", "type": "object", "properties": { "Body": { "type": "object", "properties": { "ContractNumber": { "type": "string" }, "ContractAddress": { "type": "string" } }, "required": ["ContractNumber", "ContractAddress"] } }, "required": ["Body"]} `

Phew, that’s a lot of information to keep track of. We need to know our downstream applications, which events they are interested in, and what the schemas will be for those events. If only there was a design tool which allowed us to do that!

How PubSub+ Event Portal Helps You Manage Event-Driven Architecture

The “event gods” have heard your pleas and granted your wish! Solace recently launched new product called PubSub+ Event Portal that is, as we like to say, “the market’s first event management toolset to design, create, discover, catalog, share, visualize, secure and manage all the events in your enterprise”.

I used it to design the flow of our steps in our Mulesoft event-driven architecture, and here is what it looks like:

As you can see, the Salesforce app at the top publishes the original event containing raw payload, which is then parsed by MuleSoft, and published to PubSub+ for downstream applications to consume — event-driven architecture in action!

We can also create specific events and associate schemas in the Event Portal. For example, as you can see from the image above, our MuleSoft app is publishing accountContractUpdates event which is being subscribed to by InvoicingSystem application. Expanding that event shows us who created it, which topic it is published to, what the associated schema is, and which applications are subscribing and publishing this event.

Again, while they may not seem very impressive right now, imagine how useful Event Portal would be to an organization with hundreds of applications and thousands of events!

Implementing MuleSoft Event-Driven Architecture

Now that we have our events documented, we can start implementing them.

Here is what my Mule workspace looks like:

It consists of two flows: Publisher and Consumer.

The Publisher flow has an http listener which is listening to my endpoint localhost:5001/solace. When I issue a POST request against this endpoint, the listener will capture the payload and pass it to 4 different JMS Publish modules for publishing 4 different events to 4 different topics.

Here is the connection setting for JMS Publish module:

` <jms:config name="Solace" doc:name="JMS Config" doc:id="635cb4d0-a727-4723-8d5f-6da38b806745" ><jms:generic-connection username="mule" password="mule"><jms:connection-factory ><jms:jndi-connection-factory connectionFactoryJndiName="/jms/cf/default"><jms:custom-jndi-name-resolver ><jms:simple-jndi-name-resolver jndiInitialFactory="com.solacesystems.jndi.SolJNDIInitialContextFactory" jndiProviderUrl="tcp://mule:mule@localhost:55555"><jms:jndi-provider-properties ><jms:jndi-provider-property key="Solace\_JMS\_VPN" value="default" /></jms:jndi-provider-properties></jms:simple-jndi-name-resolver></jms:custom-jndi-name-resolver></jms:jndi-connection-factory></jms:connection-factory></jms:generic-connection></jms:config> `

And here is the config xml for one of the JMS Publish modules:

` <jms:publish doc:name="Publish Account to Solace" doc:id="1a24df7f-c33d-4ca1-a551-0d097d85bb65" config-ref="Solace" destination='#[output textns ns0 http://schemas.xmlsoap.org/soap/envelope/---"company/sales/salesforce/customerAccount/account/created/v1/" ++ payload.ns0#Envelope.ns0#Body.AccountId]' destinationType="TOPIC"><jms:message ><jms:body ><![CDATA[#[output application/xmlns ns0 http://schemas.xmlsoap.org/soap/envelope/---{ns0#Envelope: {ns0#Body: {AccountName: payload.ns0#Envelope.ns0#Body.AccountName,Website: payload.ns0#Envelope.ns0#Body.Website}}}]]]></jms:body></jms:message></jms:publish> `

I have a second flow called Consumer which has a simple JMS Consume module listening to a queue followed by a JMS ack module for acknowledging messages. Here is its configuration:

` <flow name="Consumer" doc:id="e705a269-7978-40c5-a0d2-7279567278ad" ><jms:listener doc:name="Listener" doc:id="2c76ddb5-07ff-41aa-9496-31897f19d378" config-ref="Solace" destination="mule" ackMode="MANUAL"><jms:consumer-type ><jms:queue-consumer /></jms:consumer-type></jms:listener><logger level="INFO" doc:name="Logger" doc:id="cb53890f-9bc7-4bad-8512-55586aa61e8d" /><jms:ack doc:name="Ack" doc:id="c2b197a8-2fcb-4954-b748-071ffde36da5" ackId="#[%dw 2.0output application/java---attributes.ackId]"/></flow>You will need to create a queue for the consumer to bind to. In my case, I have created a queue called <code>mule</code> using Solace’s UI and mapped a topic to it: <code>company/sales/salesforce/customerAccount/></code><img src="https://solace.com/wp-content/uploads/2020/07/pubsubplus-mulesoft\_pic-04.png" alt="" width="1024" height="316" class="alignnone size-full wp-image-41324" />Notice that instead of specifying an exact topic, I have used Solace’s wildcard > to select any topic that falls under that hierarchy. In our case, subscribing to this topic will allow us to consume all of the four events.Now we are ready to test our setup. Using Postman, I have sent a POST request:<img src="https://solace.com/wp-content/uploads/2020/07/pubsubplus-mulesoft\_pic-05.png" alt="" width="1024" height="499" class="alignnone size-full wp-image-41323" />And here is the output of my Mule flow:INFO 2020-05-21 13:02:55,528 [[MuleRuntime].uber.03: [solace].Consumer.CPU\_LITE @85a5ceb] [processor: Consumer/processors/0; event: ea2e4270-9b84-11ea-9fe8-a483e79ba806] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: org.mule.runtime.core.internal.message.DefaultMessageBuilder$MessageImplementation{ payload=<?xml version='1.0' encoding='UTF-8'?><ns0:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/"> <ns0:Body> <AccountName>Solace</AccountName> <Website>www.solace.com</Website> </ns0:Body></ns0:Envelope> mediaType=application/xml; charset=UTF-8 attributes=org.mule.extensions.jms.api.message.JmsAttributes@460bec31 attributesMediaType=application/java}INFO 2020-05-21 13:02:55,536 [[MuleRuntime].uber.03: [solace].Consumer.CPU\_LITE @85a5ceb] [processor: Consumer/processors/0; event: ea2e4270-9b84-11ea-9fe8-a483e79ba806] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: org.mule.runtime.core.internal.message.DefaultMessageBuilder$MessageImplementation{ payload={ "Envelope": { "Body": { "ContactName": "Himanshu", "ContactId": "2309402", "ContactEmail": "himanshu@solace.com" } }} mediaType=application/json; charset=UTF-8 attributes=org.mule.extensions.jms.api.message.JmsAttributes@395abcae attributesMediaType=application/java}INFO 2020-05-21 13:02:55,566 [[MuleRuntime].uber.03: [solace].Consumer.CPU\_LITE @85a5ceb] [processor: Consumer/processors/0; event: ea2e4270-9b84-11ea-9fe8-a483e79ba806] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: org.mule.runtime.core.internal.message.DefaultMessageBuilder$MessageImplementation{ payload={ "Envelope": { "Body": { "ContractNumber": "123456", "ContractAddress": "535 Legget Drive, 3rd Floor, Ottawa, Canada" } }} mediaType=application/json; charset=UTF-8 attributes=org.mule.extensions.jms.api.message.JmsAttributes@6a4910db attributesMediaType=application/java}INFO 2020-05-21 13:02:55,574 [[MuleRuntime].uber.03: [solace].Consumer.CPU\_LITE @85a5ceb] [processor: Consumer/processors/0; event: ea2e4270-9b84-11ea-9fe8-a483e79ba806] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: org.mule.runtime.core.internal.message.DefaultMessageBuilder$MessageImplementation{ payload={ "Envelope": { "Body": { "BillingCountryCode": "US", "BillingState": "NY", "BillingAddress": "535 Legget Drive, 3rd Floor, Ottawa, Canada" } }} mediaType=application/json; charset=UTF-8 attributes=org.mule.extensions.jms.api.message.JmsAttributes@130e6e64 attributesMediaType=application/java} `

As you can see, our single consumer in our Consumer Mule flow was able to consume all the messages from our queue (mule) which was subscribing to just one topic.

Of course, to fully leverage the power of pub/sub messaging, event-driven architecture and Mulesoft, you can show having multiple consumers using Solace’s rich hierarchal topics to consume select events.

Get on the road to event-driven architecture!

That’s it for this post! I wanted to show you how you can make iPaaS tools such as MuleSoft even more powerful by adding pub/sub messaging to enable event-driven architecture. With Solace PubSub+, you can easily use JMS standard to connect to it and publish/consume messages.

If you work with an iPaaS, I highly recommend considering PubSub+ to make your applications event-driven. PubSub+ Standard Edition is free to use, even in production!

The post Achieving Event-Driven Architecture with Solace PubSub+ Event Broker and MuleSoft appeared first on Solace.

Integrating PubSub+ Event Broker: Cloud with Lambda via API Gateway

Himanshu Gupta — Tue, 07 Jul 2020 13:27:22 +0000

Serverless architecture is all the rage these days and nothing screams serverless like AWS Lambda. AWS Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources for you. When Lambda was announced a few years ago by AWS, it caused quite a bit of confusion but slowly it has gained popularity and become analogous to serverless architecture.

Lambdas, which is what most people call the services you spin up with Lambda, are powerful because they can be easily triggered and scaled without having to worry about the underlying infrastructure.

In this post, I would like to show you how you can send a message to a PubSub+ Event Broker running as a managed service on PubSub+ Event Broker: Cloud and use it to trigger a lambda through AWS’s API Gateway. And all of this without any code, except for the code you need for your Lambdas.

More specifically, I’ll walk you through the process of publishing a message to a topic that is being subscribed to by a queue. This is known as topic-to-queue mapping in the PubSub+ world. Once the message is received by the queue, it will be forwarded to API Gateway which will trigger the Lambda function.

We will be using REST Delivery Points (RDPs) to achieve this. An RDP is a provisioned object on a Message VPN that facilitates message delivery to REST consumers. For our use case, we will need an RDP to link our queue (which will receive the message) to the AWS API Gateway endpoint.

Here is what the architecture looks like:

Let’s begin!

Creating an AWS Lambda

Lambdas can be simple or complicated. For this blog post, I am going to create an extremely simple one that simply prints the event that triggered it.

To create a Lambda function, go to your AWS Console and select Lambda from Services section. Then, click on Create Function button on the right side.

You will be provided with multiple options on how to create a Lambda. You can pick from existing Lambdas available in AWS Serverless Application Repository, build from templates or create a new one from scratch.

Pick the option Author from scratch, give your function a name, and pick a runtime environment for your function. I have picked my Runtime environment to be Python 3.8 and my function’s name is myFirstLambda. Click on Create Function.

** ** Clicking on Create Function will create your Lambda function and take you to the next screen where you can design/code your function. In the Function code block, you can add your code, which in this case will simply print the event that triggered the Lambda. Here is the code to do that:

<br>import json<br><br> <br><br>def lambda\_handler(event, context):<br><br># TODO implement<br><br>print(\'## EVENT\')<br><br>print(event)<br><br>return {<br><br>\'statusCode\': 200,<br><br>\'body\': event<br><br>}<br><br><strong> </strong><br>

Click Save button on top right to save your changes. You have now created a Lambda function!

Creating an API Gateway endpoint

Now that you have your Lambda, we need an endpoint to trigger it. Here’s how you can do that with an API Gateway:

On the API Gateway page, click on the Create API button on top right. You will then be presented with different options on types of APIs you can create such as HTTP API, WebSocket API, and REST API (public or private). Pick REST API (public) and click on Build button.

On the next page, leave all of the settings as default and give your API a name (i.e. myFirstAPI) and click on Create API button.

Note: Currently, PubSub+ does not support SNI for REST consumers, so make sure your Endpoint Type is set to Regional, instead of Edge Optimized.

Now, you can create different endpoints under your root endpoint and specify which HTTP methods they support. For example, create one called purchase which supports POST method.

Click on Actions and then, Create Resource and fill in the details:

Click the Create Resource button when finished. Then, click on Create Method and select POST from the dropdown menu. Click the little circle with checkmark to finish.

You can now configure what happens when a POST request is issued against your endpoint. In this case, you’ll want it to trigger your Lambda function.

Again, keep all the settings as default and select your Lambda function from the dropdown menu:

Click on Save to save your changes.

Now that you’ve designed your API, you need to deploy it to be able to use it. You can also create different Stages to manage lifecycle of our endpoint.

Click on Actions and then, Deploy API which will pop-up a mini window where you can specify the stage you want to deploy your API to and provide a description for your deployment. Because you don’t have an existing stage, you will see [New Stage] option when you click on the Deployment stage dropdown menu.

Select [New Stage] and enter information about your stage. I called mine PROD.

Click Deploy when finished.

You will now find details about your newly deployed endpoint, specifically, the invoke URL. Mine is https://1v3hgu8bia.execute-api.us-east-1.amazonaws.com/PROD

You’ve now successfully created an endpoint that will trigger your Lambda function when a POST request is issued against it. All that’s left to do now is to integrate your endpoint with PubSub+ Event Broker: Cloud!

PubSub+ Event Broker: Cloud

If you are unfamiliar with PubSub+ Event Broker: Cloud, it is Messaging-as-a-Service provided by Solace. PubSub+ Event Broker: Cloud allows you to easily spin up a PubSub+ Event Broker on any of the major cloud providers. It also has a free tier so you can easily spin up a service to follow along with this post without having to worry about any cost.

Follow instructions here on how to create a service on Solace PubSub+ Event Broker: Cloud.

Creating a Queue with topic subscription

Before you create your RDP, you need to create a queue with the appropriate topic subscription.

Go to your messaging service, click on the Manage tab, and then click on the Message VPN block under PubSub+ Manager Quick Settings.

That will pop up a management UI for your service which you can use to create your queue. Click on Queues on the left navigation bar and then click on +Queue button. Give it a name of the form [rdpname].[consumername]. This syntax is not mandatory, but it will help you identify which queue is bound to which consumer and for which RDP. Because you haven’t yet created an RDP or a consumer, you can call it whatever you like. I called mine myFirstRDP.myRDPConsumer.

Click Create button and then leave the settings as default, click Apply. Your queue has now been created:

Now you need to add a topic subscription to it. To do that, click on the queue and navigate to Subscriptions. Click on + Subscription and enter POST/PROD/purchase and then, click on Create. Note that our topic must be of this syntax: <HTTP_METHOD>/<STAGE>/<endpoint>.

Creating a REST Delivery Point (RDP)

Now that you have your queue ready, you can go ahead and create your RDP. To do that, click on Client Connections on left navigation bar and then REST. Then, click on + REST Delivery Point button. As decided earlier, we will call our RDP myFirstRDP. Toggle the Enabled button to enable the RDP and click Apply. You will notice that the Operational State is currently Down and that’s because you haven’t added a REST consumer yet.

Click on your RDP and navigate to REST Consumers. Click on + REST Consumer and give it a name. We decided earlier (when creating our queue) that your REST consumer will be called myRDPConsumer. Click Create.

On the next screen, you will need to provide the host url for your AWS API Gateway endpoint. This URL is just the base URL and should not include /PROD/purchase. In my case, that URL is 1v3hgu8bia.execute-api.us-east-1.amazonaws.com.

The port number should be set to 443. Toggle the button next to TLS Enabled to enable TLS.

Click on Apply button to create your consumer.

Click on your newly created RDP consumer and navigate to TLS Options on top. Click on + Trusted Common Name and enter *.execute-api.us-east-1.amazonaws.com.

Go back to your RDP and enable it by clicking on the toggle button next to Enabled. You will notice that the Operational State has now changed to Up.

You now need to add your queue binding to your RDP. Click on Queue Bindings and then on + Queue Bindings. Select the queue you created earlier, myFirstRDP.myRDPConsumer, from the dropdown menu. Click Create.

On the following page, enter a Post Request Target. In this case, that would be /PROD/purchase. Make sure you include the “/” in front.

Click Apply. Operation State of your queue binding should be Up.

Adding Amazon’s CA Cert

To make sure your TLS settings work, you need to upload Amazon’s CA Cert to your broker. The certificate can be downloaded from here.

To upload the certificate, go back to your service on PubSub+ Event Broker: Cloud and click on the Certificate Authorities block.

Click on + Add New, give it a name, and paste the content in the popup box. Click on Submit.

That should be it! We can now test to see if our setup works.

Demo Time

A quick recap: With the goal of sending a message to PubSub+ Event Broker and have a Lambda triggered by it, you created an RDP which has a REST consumer that points to our AWS API Gateway endpoint and a queue binding which is subscribed to a topic. You AWS API Gateway endpoint is configured to trigger your Lambda function when a POST request is issued.

So, let’s send a message to our PubSub+ Event Broker and see what happens. You can easily do that by using the Try Me! app on PubSub+ Event Broker: Cloud. Click on Try Me! on the left navigation bar and enter your connection information on the Publisher app.

You can get your credentials by going back to your service and navigating to the Connect tab on the top and then selecting REST.

Click on Connect once you have entered your connection details in the Publisher app.

You can now set your Topic to POST/PROD/purchase and your Message Content to a JSON message:

<br><br>{<br>"body" : "A purchase order has been placed by Solace"<br>}<br><br>

Click on Publish to send the message and see if your Lambda was invoked. To do so, go to your Lambda and click on Monitoring on top and then, click on View logs in CloudWatch.

That will take you to CloudWatch console where you will see a Log Stream. Click on it to see the contents:

If you see something like this, that means your Lambda was successfully invoked, and it consumed the message sent from your Try Me! app. Pretty cool, right? And all with configuration, not coding!

Conclusion

Hope you enjoyed this post and learned something interesting!

The post Integrating PubSub+ Event Broker: Cloud with Lambda via API Gateway appeared first on Solace.

Using Solace PubSub+ with OneTick Time Series Database

Himanshu Gupta — Wed, 17 Jun 2020 19:27:52 +0000

OneTick is a time-series database that is heavily used in financial services by hedge funds, brokerages and investment banks. OneMarketData, the maker of OneTick, describes it as “the premier enterprise-wide solution for tick data capture, streaming analytics, data management and research.” OneTick captures, compresses, archives, and provides uniform access to global historical data, up to and including the latest tick.

I myself have used OneTick for almost 5 years and found it to be a very nicely packaged solution thanks to built-in collectors for popular feeds such as Reuters’ Elektron and Bloomberg’s MBPIPE. OneTick also comes with a GUI that makes it easy for non-developers, such as researchers and portfolio managers, to write advanced queries using pre-built event processors (EPs). The GUI can be used to design queries that you can run from the GUI or APIs that OneTick supports, such as Python.

For example, here is a simple query that would select trade data for AAPL where price is greater than or equal to 21.5 (data is from 2003). In SQL, this would be select * from <database_name> where symbol="AAPL" and price >= 21.5.

OneTick comes with a DEMO database that I will be using for my examples. Here is what the query looks like in OneTick GUI:

When you run this query, you will see the result in OneTick’s GUI.

That’s useful information, but what if…

You wanted to stream this data to your real-time event-driven applications?
You wanted to publish the data once and have many recipients get it?
You wanted to let downstream applications retrieve data via open APIs and message protocols?

Solace PubSub+ Event Broker makes all of that possible!

How PubSub+ Can Help OneTick Developers

Here are some of the many ways OneTick developers can leverage PubSub+:

Consume data from a central distribution bus Many financial companies capture market data from external vendors and distribute it to their applications internally via PubSub+. OneTick developers can now natively consume the data and store it for later use.
Distribute data from OneTick to other applications Instead of delivering data individually to each client/application, OneTick developers can leverage pub/sub messaging pattern to publish data once and have it consumed multiple times.
Migrate to the cloud Have you been thinking about moving your applications and/or data to the cloud? With PubSub+ you can easily give cloud-native applications access to your high-volume data stored in on-prem OneTick instance using an event mesh.
Easier integration PubSub+ supports open APIs and protocols which make it a perfect option for integrating applications. Your applications can now use a variety of APIs to publish to and subscribe data from OneTick.
Replay messages Are you tired of losing critical data such as order flows when your applications crash? Replay functionality helps protect applications by giving them the ability to correct database issues they might have arising from events such as misconfiguration, application crashes, or corruption, by redelivering data to the applications.
Automate notifications It’s always a challenge to ensure all of the consumers are aware of any data issues whether it be an exchange outage leading to data gaps or delay in an ETL process. With PubSub+, OneTick developers can automate the publication of notifications related to their databases and broadcast it to the interested subscribers.

Running an Instance of PubSub+ Event Broker

In the rest of this post, I am going to show you how you can take data from your OneTick database and publish it to PubSub+, and vice versa. To follow along you will need access to an instance of PubSub+. If you don’t, there are two free ways to set up an instance:

Install a Docker container locally or in the cloud
Set up a dev instance on Solace Cloud (easiest way to get started)

Please post on Solace Community if you have any issues setting up an instance.

Publishing data from OneTick to PubSub+

OneTick makes it extremely easy for you to publish data to PubSub+ by simply using a built-in event processor (EP). The EP is called WRITE_TO_SOLACE and is part of OneTick’s Output Adapters (see OneTick docs: docs/ep_guide/EP/OutputAdapters.htm) which contains other EPs such as WRITE_TEXT, WRITE_TO_RAW, and WRITE_TO_ONETICK_DB.

To publish results of our previous query to PubSub+, we would simply add WRITE_TO_SOLACE EP at the end of our query and fill in appropriate connection and other details. Here is what the EP parameters look like:

You can find details about the WRITE_TO_SOLACE EP and its parameters in OneTick docs (docs/ep_guide/EP/OMD/WRITE_TO_SOLACE.htm).

The first few fields in the EP are for your PubSub+ connection details such as HOSTNAME, VPN, USERNAME, and PASSWORD.

The EP supports both direct and guaranteed message delivery. Direct messages are suitable for high throughput/low latency use cases such as market data distribution, and guaranteed messages are appropriate for critical messages that can’t be lost, such as order flow. You can learn more about the two delivery modes here.

The EP also allows you to specify the ENDPOINT you would like to publish data to, and the ENDPOINT_TYPE where you can specify if it is a topic or a queue. Publishing to a topic supports the pub/sub messaging pattern where your application publishes once to the topic and many applications can subscribe to that topic. On the other hand, publishing to a queue is point-to-point messaging because only the consumer binding to that specific queue can consume data published to it. Needless to say, we encourage clients to use the pub/sub messaging pattern as it provides more flexibility.

For this example today, I will publish to this topic: EQ/US/<security_name; such as EQ/US/AAPL.

Finally, you will notice that the EP allows users to specify the message format via MSG_FORMAT field. Currently, you can only pick from NAME_VALUE_PAIRS and BINARY, but OneTick will support additional message formats soon.

Configuring Subscriptions on PubSub+ Broker

Before publishing data to PubSub+, you need to determine how it will be consumed. Applications can either subscribe directly to the topic we are publishing to, or map the topic(s) to a queue and consume them from there. Using a queue introduces persistence so if your consumer disconnects abruptly, your messages won’t be lost. For our example, we will consume from a queue called onetick_queue that I have created from PubSub+’s UI:

PubSub+ allows you to map one or many topics to a queue, which means your consumer only needs to worry about binding to one queue and it can have messages being published to different topics ordered and delivered to that queue.

Additionally, PubSub+ also supports rich hierarchical topics and wildcards so you can dynamically filter the data you are interested in consuming. PubSub+ supports two wildcards: * lets you abstract away one topic level, and > lets you abstract away one or more levels. So, if you wanted to subscribe to all the data for equities, you can simply subscribe to EQ/&gt; and if you wanted to subscribe to data for AAPL from different regions/venues, you could subscribe to EQ/*/AAPL.

You can learn more about Solace topics and wildcards here.

I have subscribed my queue to topic: EQ/US/AAPL.

With that done, you’re ready to run the query!

Once you have run your OneTick query, go to the PubSub+ UI and note that messages are in the queue.

As you can see, there are 8,278 messages in our queue.

This is how easy it is to publish messages from OneTick to PubSub+!

Writing Data from PubSub+ to OneTick

Now it’s time to consume data from PubSub+ and write it to a OneTick database. To be efficient you can use the data in the onetick_queue to write to OneTick db.

To read data from PubSub+, OneTick provides the omd::OneTickSolaceConsumer interface via its C++ API. Use this interface to connect to a Solace broker, make subscriptions, process received messages, and propagate those messages to the Custom Serialization interface.

To make things simple for us, OneTick distribution comes with a C++ example (examples/cplusplus/solace_consumer.cpp) that shows you how to do that. You can go through the code to see how to use the OneTick/Solace API but for the purpose of this post, we will simply run the code. However, before we can write the data from PubSub+ to a OneTick database, we need to create/configure that database.

I have created a database called solace by following these steps:

Create data directory for our database. I have created mine at: C:/omd/one_market/data/data/solace
Create a database entry in your locator file. Here is my locator entry for solace db:

<db ID="solace" SYMBOLOGY="SOLACE" ><LOCATIONS><location access\_method="file" location="C:\omd\data\solace" start\_time="20000101000000" end\_time="20300101000000" /></LOCATIONS><RAW\_DATA><RAW\_DB ID="PRIMARY" prefix="solace."><LOCATIONS><location mount="mount1" access\_method="file" location="C:\omd\data\solace\raw\_data" start\_time="20000101000000" end\_time="20300101000000" /></LOCATIONS></RAW\_DB> </RAW\_DATA></db>

Now that we have our database configured, we are ready to run our sample code. We will need to provide the following details to run the example:

PubSub+ connection details: host:port, username, password, vpn
endpoint: which topic/queue to subscribe to
OneTick db: name of database we will be writing data to

If the queue doesn’t exist, our code will administratively create it and map topics to it. In our case, we already have a queue (onetick_queue) that we will be consuming from.

As soon as we run the example, it will consume all the messages from the queue:

C:\omd\one\_market\_data\one\_tick\examples\cplusplus>solace\_consumer.exe -hostname <host\_name>.messaging.solace.cloud -username <username> -password <password> -vpn himanshu-demo-dev -queue onetick\_queue -dbname solace20200509021426 INFO: Relevant section in locator <DB DAY\_BOUNDARY\_TZ="GMT" ID="solace" SYMBOLOGY="SOLACE" TIME\_SERIES\_IS\_COMPOSITE="YES" > <LOCATIONS > <LOCATION ACCESS\_METHOD="file" END\_TIME="20300101000000" LOCATION="C:\omd\data\solace" START\_TIME="20000101000000" /> </LOCATIONS> <RAW\_DATA > <RAW\_DB ID="PRIMARY" PREFIX="solace." > <LOCATIONS > <LOCATION ACCESS\_METHOD="file" END\_TIME="20300101000000" LOCATION="C:\omd\data\solace\raw\_data" MOUNT="mount1" START\_TIME="20000101000000" /> </LOCATIONS> </RAW\_DB> </RAW\_DATA> </DB>

After running the example, you will notice new raw data files have been created:

C:\omd\data\solace\raw\_data>dir Directory of C:\omd\data\solace\raw\_data05/11/2020 05:41 PM <DIR> .05/11/2020 05:41 PM <DIR> ..05/11/2020 05:41 PM 245,644 solace.mount1.2020051121403905/11/2020 05:41 PM 0 solace\_mount1\_lock 2 File(s) 245,644 bytes 2 Dir(s) 172,622,315,520 bytes free

Now, we need to run OneTick’s native_loader_daily.exe to archive the data:

C:\omd\one\_market\_data\one\_tick\bin>native\_loader\_daily.exe -dbname solace -datadate 2020051120200511214237 INFO: Native daily loader started20200511214237 INFO:  BuildNumber 20200420120000Command and arguments: native\_loader\_daily.exe  -dbname solace -datadate 20200511...20200511214237 INFO: Picked up file C:\omd\data\solace\raw\_data/solace.mount1.20200511214039 in raw db PRIMARY20200511214237 INFO: Total processed messages: 8,468 (1,721,228 bytes)20200511214237 INFO: Messages processed in last 0.046 seconds: 8,468 (1,721,228 bytes) at 184,652.958 msg/s (37,533,046.948 bytes/s)20200511214237 INFO: Last measured message latency: NaN20200511214237 INFO: Total processed messages: 8,468 (1,721,228 bytes)20200511214237 INFO: Messages processed in last 0.005 seconds: 0 (0 bytes) at 0.000 msg/s (0.000 bytes/s)20200511214237 INFO: Last measured message latency: NaN20200511214237 INFO: persisting sorted batch20200511214237 INFO: done persisting sorted batch20200511214237 INFO: done persisting sorted batches for database solace, starting phase 2 of database load. Total symbols to load: 1, number of batches: 120200511214237 INFO: beginning sorted files cleanup20200511214237 INFO: done with sorted files cleanup20200511214237 INFO: loading index file; total records: 120200511214237 INFO: loading symbol file20200511214237 INFO: renaming old archive directory, if exists20200511214237 INFO: Load Complete for database solace for 20200511Total ticks 8278Total symbols 120200511214237 INFO: Processing time: 1

As you can see, 8,278 ticks have been archived – which is the same number of messages that were in the queue. You can now go to OneTick GUI and query the data from solace db:

That’s it!

I’ve shown you how easy it is to publish data from OneTick to PubSub+, read it back, and write to a OneTick database.

We are excited to have OneTick users like yourself take advantage of features like rich hierarchical topics , wildcards , support for open APIs and protocols and the dynamic message routing that enables event mesh.

If you would like to learn more about the API, please reach out to your OneTick account manager or support. If you would like to learn more about how to get started with PubSub+, please reach out to us here or check out our Solace with OneTick page.

The post Using Solace PubSub+ with OneTick Time Series Database appeared first on Solace.

Integrating PubSub+ Cloud with Lambda via API Gateway

Himanshu Gupta — Thu, 21 May 2020 14:45:35 +0000

Serverless architecture is all the rage these days and nothing screams serverless like AWS Lambda. When Lambda was announced a few years ago by AWS, it caused quite a bit of confusion but slowly it has gained popularity and has become analogous to serverless architecture.

AWS Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources for you.

AWS

Lambdas are super powerful because they can be easily triggered and scaled without having to worry about the underlying infrastructure.

In this post, I would like to show you how you can send a message to a PubSub+ broker running as a managed service on PubSub+ Cloud and use it to trigger a lambda through AWS’s API Gateway. And all of this without any code, except for the code you need for your Lambdas.

More specifically, we will publish a message to a topic that is being subscribed to by a queue. This is known as topic-to-queue mapping in the PubSub+ world. Once the message is received by the queue, it will be forwarded to API Gateway which will trigger the Lambda function.

Here is what our architecture looks like:

Let’s begin!

Creating an AWS Lambda

Lambdas can be extremely simple or highly complicated. For this blog post, I am going to create an extremely simple one that simply prints the event that triggered it.

To create a Lambda function, go to your AWS Console and select Lambda from Services section. Then, click on Create Function button on the right side.

Clicking on Create Function will create your Lambda function and will take you to the next page where you can design/code your function. In the Function code block, you can add your code. As mentioned earlier, we will only be printing our event that triggered the Lambda. Here is the code to do that:

import json

def lambda_handler(event, context):
    # TODO implement
    print('## EVENT')
    print(event)
    return {
        'statusCode': 200,
        'body': event
    }

Click Save button on top right to save your changes. We have now created a Lambda function!

Creating an API Gateway endpoint

Now that we have our Lambda, we need an endpoint to trigger it. We will use API Gateway to do that.

On the next page, leave all of the settings as default and give your API a name (i.e. myFirstAPI) and click on Create API button.

Note: Currently, PubSub+ does not support SNI for REST consumers so we need to make sure that our Endpoint Type is set to Regional, instead of Edge Optimized.

Now, we can create different endpoints under our root endpoint and specify which HTTP methods they support. For example, let’s create one called purchase which supports POST method.

Click on Actions and then, Create Resource and fill in the details:

Click the Create Resource button when finished. Then, click on Create Methodand select POST from dropdown menu. Click the little circle with checkmark to finish.

You can now configure what happens when a POST request is issued against your endpoint. We would like it to trigger our Lambda function.

Again, keep all the settings as default and select your Lambda function from dropdown menu:

Click on Save to save your changes.

Now that we have designed our API, we need to deploy it to be able to use it. We can also create different Stages to manage lifecycle of our endpoint.

Click on Actions and then, Deploy API which will pop-up a mini window where you can specify the stage you want to deploy your API to and provide a description for your deployment. Because we don’t have an existing stage, you will see [New Stage] option when you click on Deployment stage dropdown menu.

Select [New Stage] and enter information about your stage. I am calling mine: PROD.

Click Deploy when finished.

You will now find details about your newly deployed endpoint, specifically, the invoke URL. Mine is: https://1v3hgu8bia.execute-api.us-east-1.amazonaws.com/PROD

We have now successfully created an endpoint which will trigger our Lambda function when a POST request is issued against it. All that’s left to do now is to integrate our endpoint with PubSub+ Cloud!

PubSub+ Cloud

If you are unfamiliar with PubSub+ Cloud, it is Messaging-as-a-Service provided by Solace. PubSub+ Cloud allows you to easily spin up a PubSub+ event broker on any of the major cloud providers. It also has a free tier so you can easily spin up a service to follow along with this post without having to worry about any cost.

Follow instructions here on how to create a service on Solace Cloud.

Creating a Queue with topic subscription

Before we create our RDP, we need to create a queue with the appropriate topic subscription.

Go to your messaging service, click on Manage tab, and then click on Message VPN block under PubSub+ Manager Quick Settings.

That will popup a management UI for your service which you can use to create your queue. Click on Queues on the left navigation bar and then click on + Queue button. Give it a name of the form [rdpname].[consumername]. This syntax is not mandatory but it will help you identify which queue is bound to which consumer and for which RDP. Because we haven’t created either an RDP or a consumer, you can call it whatever you like. I am calling mine: myFirstRDP.myRDPConsumer.

Click Create button and then leave the settings as default, click Apply. Your queue has now been created:

Now we need to add a topic subscription to it. To do that, click on the queue and navigate to Subscriptions. Click on + Subscription and enter POST/PROD/purchase and then, click on Create. Note that our topic must be of this syntax: <HTTP_METHOD>/<STAGE>/<endpoint>.

Creating a REST Delivery Point (RDP)

Now that we have our queue ready, we can go ahead and create our RDP. To do that, click on Client Connections on left navigation bar and then REST. Then, click on + REST Delivery Point button. As decided earlier, we will call our RDP: myFirstRDP. Toggle the Enabled button to enable the RDP and click Apply. You will notice that the Operational State is currently Down and that’s because we haven’t added a REST consumer yet.

Click on your RDP and navigate to REST Consumers. Click on + REST Consumer and give it a name. We decided earlier (when creating our queue) that our REST consumer will be called: myRDPConsumer. Click Create.

On the next page, we will need to provide the host url for our AWS API Gateway endpoint. This URL is just the base URL and should not include /PROD/purchase. In my case, that URL is: 1v3hgu8bia.execute-api.us-east-1.amazonaws.com

The port number should be set to 443. Toggle the button next to TLS Enabled to enable TLS.

Click on Apply button to create your consumer.

Click on your newly created RDP consumer and navigate to TLS Options on top. Click on + Trusted Common Name and enter *.execute-api.us-east-1.amazonaws.com.

Go back to your RDP and enable it by clicking on the toggle button next to Enabled. You will notice that the Operational State has now changed to Up.

We now need to add our queue binding to our RDP. Click on Queue Bindings and then, + Queue Bindings. Select the queue we had created earlier, myFirstRDP.myRDPConsumer, from the dropdown menu. Click Create.

On the following page, enter a Post Request Target. In our case, that would be /PROD/purchase. Make sure you include the “/” in front.

Click Apply. Operation State of your queue binding should be Up.

Adding Amazon’s CA Cert

To make sure our TLS settings work, we need to upload Amazon’s CA Cert to our broker. The certificate can be downloaded from here.

To upload the certificate, go back to your service on PubSub+ Cloud and click on Certificate Authorities block.

Click on + Add New, give it a name, and paste the content in the popup box. Click on Submit.

That should be it! We can now test to see if our setup works.

Demo Time

A quick recap. Our goal is to send a message to PubSub+ broker and have a lambda triggered by it. To achieve this, we created an RDP which has a REST consumer that points to our AWS API Gateway endpoint and a queue binding which is subscribed to a topic. Our AWS API Gateway endpoint is configured to trigger our Lambda function when a POST request is issued.

So, let’s send a message to our PubSub+ broker and see what happens. We can easily do that by using the Try Me! app on PubSub+ Cloud. Click on Try Me! on left navigation bar and enter your connection information on the Publisher app.

You can get your credentials by going back to your service and navigating to Connect tab on top and then selecting REST.

Click on Connect once you have entered your connection details in the Publisher app.

We can now set our Topic to POST/PROD/purchase and our Message Content to a JSON message:

{
"body" : "A purchase order has been placed by Solace"
}

Click on Publish to send the message. Now, we need to verify whether our Lambda was invoked. To do so, we can go to our Lambda and click on Monitoring on top and then, click on View logs in CloudWatch.

That will take you to CloudWatch console where you will see a Log Stream. Click on it to see the contents:

As you can see, our Lambda was successfully invoked and it consumed the message we sent it from our Try Me! app. We were able to accomplish this with mostly configuration settings and without having to code anything.

Hope you enjoyed this post and learned something interesting!

What are the Differences Between IaaS, PaaS, and SaaS?

Himanshu Gupta — Thu, 21 May 2020 06:26:19 +0000

As a solutions architect, I talk regularly to developers and architects about IaaS, PaaS, and SaaS. These terms, which are relatively new, can sometimes be confusing so I’d like to give a brief description of what they are and how they differ from each other.

As shown here, IaaS, PaaS, and SaaS stand for infrastructure, platform and software as a Service, respectively. The core idea behind ‘aaS’ is that you can focus on building products and services critical to your business and let other companies build and manage non-core services you need, whether as part of your own offering or just to run your business.

These days, you can take pretty much add ‘aaS’ to anything, like “data” (DaaS) and “integration platform” which introduces the potentially confusing “iPaaS” which differs from both IaaS and PaaS! In the last few years, new businesses have popped up providing X, whether it be a product, application, feature or anything really, as a service to other businesses or consumers.

Overview: How did we get here?

Before we dive into what these 3 terms mean, let’s take a step back and look at how we got here. Life was simple when all developers had to worry about were monolithic applications running on-premises. As a company, you would either have your own datacenter or rent some space at a datacenter managed by another company. Your networking and sysadmin teams would make sure all the servers were properly connected and working for your developers to use. Your developers would build robust and scalable applications (hopefully) that got deployed to your servers. Simple, right?

Infrastructure as a Service (IaaS)

Most mid-sized business and large enterprises, still manage their own servers and datacenters, but are adopting the cloud. For almost a decade most companies have realized they don’t want to be in the business of managing their own infrastructure (servers/datacenters) and would rather have someone else, such as Amazon or Google, do it for them. Startups these days are going straight to the cloud, which allows them to be extremely flexible and scalable amongst other things. This has led cloud providers such as AWS and GCP to provide you with servers and manage them for you – that’s infrastructure as a service!

For example, you can easily spin up Solace PubSub+ Event Broker software on a linux server using an Amazon AMI in less than a minute. Compare this to having to buy a server, set it up, install it in a datacenter, and deploy software on it. Depending on the size of the company, this can take weeks, if not months.

Platform as a Service (PaaS)

As companies started adopting IaaS and migrating to the cloud, it became apparent that it wasn’t so easy. A key benefit of the cloud is elasticity. It’s easy to spin up servers when demand is high and shut them down when demand is low, but, developing applications for your limited servers in your own datacenter is one thing. Developing applications to run in the cloud where your application might be deployed to hundreds of servers on peak days like Black Friday, accessible by way of the Internet or WAN links, might not be as stable as it is in your own private datacenter.

All of this came with a lot of overhead for developers, and for DevOps team which now had to build and manage cloud-native applications. This led to platform as a service which lets developers develop cloud-native applications and manage the overhead associated with orchestrating them. Popular PaaS offerings include AWS Elastic Beanstalk, Pivotal Cloud Foundry, Kubernetes and OpenShift. I recently blogged about how to deploy Solace’s broker in OpenShift.

Software as a Service (SaaS)

As cloud migration has picked up, it has further fueled the build vs buy debate. Every company is faced with the option to either build something themselves and hence, have complete control over it or buy it from another company so they can focus on its core business at the cost of control. Additionally, while you might have the resources to build the product initially, you still have to dedicate significant resources to maintain and upgrade the product going forward. As a consequence, more and more companies are realizing that they would rather buy non-core products/software from other companies and focus on their core business. For example, you can use Solace Cloud to spin up a PubSub+ Event Broker without needing any hardware and have it managed by Solace.

So what are the differences between IaaS, PaaS, and SaaS?

The main difference between IaaS, PaaS, and SaaS is how much they abstract away from end-users. The end product is an application that you want to use. For that to happen, you need to manage: datacenters, servers, networks, virtualization, operating systems, runtime, storage, middleware, data, applications, and monitoring.

Here is a useful table which shows which parts are being abstracted away from the end-user in each model:

SaaS applications are completely managed by someone else. They can be easily accessed through a web browser, so you don’t need to worry about installing them on an on-prem server. They are provided as managed service which takes away the overhead of regularly upgrading the underlying software.
PaaS provides you with everything you need to have your application running except for the actual application itself. You are responsible for writing the application and managing it but everything else, such as servers, runtime, and orchestration of your microservices, is provided.
IaaS is the most basic (or flexible) of them all and only provides the underlying infrastructure, upon which you must set up your own runtime environments.

Picking the right model for your company

With so many options, it can be a little daunting to decide which one is the best for your company. Thankfully, you don’t have to pick just one. You can go with a hybrid model where your core applications will run on-prem in your own datacenter and the rest of your in-house applications will run on a PaaS such as OpenShift. You might also decide to have some database services run on cloud such as AWS Redshift. Finally, you might want to limit all your vendor software, such as HR/payroll management software (i.e. Workday), monitoring tools (i.e. New Relic), and event brokers (i.e. Solace), to SaaS.

Picking the right model comes down to how much flexibility you want and how little you want to manage. If a product is core to your business and you need to heavily customize it, then going with a SaaS model is not the right option. However, there is no need to have your own convoluted HR/payroll management system when a generic SaaS solution would suffice. Finally, you have to consider the sensitivity of these applications and the data they manage. Some applications manage “crown jewel” processes and information that you won’t want to trust to a SaaS.

Conclusion

As you can see, there are a lot of similarities and differences between IaaS, PaaS, and SaaS. If you are a developer, you must be familiar with all three and how they differ. As an architect, you may be responsible for deciding which model, or hybrid approach, is right for your company based on your requirements, and resources. Fortunately, you can pick which model works best for you for each of your applications. I hope I’ve helped you understand a little bit better which one might be right for you.

The post What are the Differences Between IaaS, PaaS, and SaaS? appeared first on Solace.

Configuring PubSub+ Event Broker for High Availability in AWS

Himanshu Gupta — Thu, 12 Mar 2020 18:45:38 +0000

When architecting a system, it’s important to make sure it’s redundant, resilient, and fault-tolerant so it can carry on if a node fails, or even if all the nodes in one datacenter fail.

PubSub+ Event Broker can be deployed in a high availability (HA) group, and configured for disaster recovery (DR). In this post, I’ll focus on deploying for HA in AWS. There is an AWS CloudFormation template available for you to easily spin up your HA group, but I’d like to show you how to do it manually so you understand how it works.

How does HA work with PubSub+ Event Broker?

A typical HA deployment of Solace’s PubSub+ Event Broker consists of three nodes:

Primary
Backup
Monitor

The primary and backup brokers are set up in active-standby configuration, while the third acts as a monitoring node. The two brokers have their own storage, so they share nothing and are completely independent. The monitoring node is just responsible for maintaining quorum between the primary and backup brokers.

When a message is published to the primary broker, it is persisted locally and synchronously pushed to the backup broker. Once the backup broker receives the message, it acknowledges receipt to the primary broker, at which point the primary broker sends an acknowledgement back to the publisher.

If there is a subscriber interested in this message, it will be forwarded to that subscriber. Otherwise, the message will be forwarded later. Once the subscriber receives the message, it will send a confirmation back to the primary broker. Upon receiving this receipt, the primary and standby brokers will both delete their copies of the message.

Note that at any given time, only the active broker will accept connections. For example, let’s say your primary broker is your active broker. If and when the primary broker fails, the backup broker will become the active broker and start accepting connections. When the primary broker becomes available again, you can either configure your system to automatically make it the active broker again, or leave the secondary broker active. It’s recommended that you run the primary and secondary brokers on different servers within the same datacenter.

This HA setup makes systems very resilient, because if the primary broker fails, the backup will quickly take over and minimize the impact to your system.

Now that we know how an HA group works, let’s spin up three instances of PubSub+ Event Broker on AWS and configure them to be part of an HA group.

Launching EC3 instances with PubSub+ Event Broker

Let’s see how we can configure three broker instances to be our primary broker, secondary broker, and monitoring node. We will be running three instances of PubSub+ Event Broker on three different EC2 instances on AWS. See my blog post on how to launch an EC2 instance with PubSub+Event Broker.

Once you have an EC2 instance up and running, you should use the security group that was created for this instance and attach it to the backup and monitoring EC2 instance. Do not create all 3 instances with 3 different security groups.

I also edited the security group that was created for the first EC2 instance to allow all incoming traffic from any instance that is attached to this security group. You can do this by going to the security group and under Inbound section, clicking Edit , then clicking Add Rule , selecting All Traffic and in the Source field, enter the security group Id. Here is what it should look like:

Again, this is just for demo purposes to make sure all our instances can speak to each other.

Configuring HA group

Now that we have all 3 EC2 instances with PubSub+ Event Broker up and running, we can begin to configure them as part of one HA group. The following steps have been thoroughly documented in the HA Group Configuration section of Solace PubSub+ Technical Documentation.

Before we begin, we need to update our router name on each instance. For an HA group to be configured, the router names must match the node name. When we launched EC2 instance from Solace AMI, we were given a default router name (it’s usually the private IP address). We can use the given router name as our node name, but I would rather give our nodes more descriptive names such as ha-demo-primary, ha-demo-backup, and ha-demo-monitor.

Primary node

/ Activate Solace CLI[sysadmin@ip-172-31-26-146 ~]$ solacectl cliSolace PubSub+ Standard Version 9.3.1.5Operating Mode: Message Routing Nodeip-172-31-26-146> enableip-172-31-26-146# configure/ Change router name to primaryip-172-31-26-146(configure)# router-name ha-demo-primaryThis command causes a reload of the system.Do you want to continue (y/n)? y

Note that a router name change requires a reboot of PubSub+ Event Broker so you will have to wait a minute or so before it comes back up. Run the following command once the broker has been rebooted to confirm the router name was changed:

ip-172-31-26-146> show router-nameRouter Name:          ha-demo-primaryMirroring Hostname:   NoDeferred Router Name: ha-demo-primaryMirroring Hostname:   No

Once you have updated the router name, you will need to run some commands to:

Create and designate different instances as primary , backup, and monitor
Provide the authentication key

From Solace docs: pre-shared-key is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It’s used to provide authentication between nodes in a HA Group and must be the same on each node. I used Base64 Encode to create my key. Once you have your key, run the following commands and replace my key with the one you created.

ip-172-31-26-146(configure)# hardware message-spool shutdownAll message spooling will be stopped.Do you want to continue (y/n)? yip-172-31-26-146(configure)# redundancyip-172-31-26-146(configure/redundancy)# switchover-mechanism hostlistip-172-31-26-146(configure/redundancy)# exitip-172-31-26-146(configure)# redundancyip-172-31-26-146(configure/redundancy)# groupip-172-31-26-146(configure/redundancy/group)# create node ha-demo-primaryip-172-31-26-146(configure/redundancy/group/node)# connect-via 172.31.26.146ip-172-31-26-146(configure/redundancy/group/node)# node-type message-routing-nodeip-172-31-26-146(configure/redundancy/group/node)# exitip-172-31-26-146(configure/redundancy/group)# create node ha-demo-backupip-172-31-26-146(configure/redundancy/group/node)# connect-via 172.31.26.208ip-172-31-26-146(configure/redundancy/group/node)# node-type message-routing-nodeip-172-31-26-146(configure/redundancy/group/node)# exitip-172-31-26-146(configure/redundancy/group)# create node ha-demo-monitorip-172-31-26-146(configure/redundancy/group/node)# connect-via 172.31.22.77ip-172-31-26-146(configure/redundancy/group/node)# node-type monitor-nodeip-172-31-26-146(configure/redundancy/group/node)# exitip-172-31-26-146(configure/redundancy/group)# exitip-172-31-26-146(configure/redundancy)# authenticationip-172-31-26-146(configure/redundancy/authentication)# pre-shared-key key c29sYWNlaXNhZ3JlYXRldmVudHBsYXRmb3Jtd2hpY2h5b3VzaG91bGR1c2Vmb3JhbGx5b3VyZXZlbnRpbmduZWVkcw==ip-172-31-26-146(configure/redundancy/authentication)# exitip-172-31-26-146(configure/redundancy)# active-standby-role primaryip-172-31-26-146(configure/redundancy)# no shutdown

You will mostly run the same commands on your backup node that you ran on your primary node. The only difference is that you will designate it as a backup node by running this command instead:

active-standby-role backup

Here are the commands you need to run on your backup node:

ip-172-31-26-208> enableip-172-31-26-208# configureip-172-31-26-208(configure)# router-name ha-demo-backupThis command causes a reload of the system.Do you want to continue (y/n)? yip-172-31-26-208(configure)# [sysadmin@ip-172-31-26-208 ~]$ solacectl cliSolace PubSub+ Standard Version 9.3.1.5Operating Mode: Message Routing Nodeip-172-31-26-208> show router-nameRouter Name:          ha-demo-backupMirroring Hostname:   NoDeferred Router Name: ha-demo-backupMirroring Hostname:   Noip-172-31-26-208> enableip-172-31-26-208# configureip-172-31-26-208(configure)# hardware message-spool shutdownAll message spooling will be stopped.Do you want to continue (y/n)? yip-172-31-26-208(configure)# redundancyip-172-31-26-208(configure/redundancy)# switchover-mechanism hostlistip-172-31-26-208(configure/redundancy)# groupip-172-31-26-208(configure/redundancy/group)# create node ha-demo-primaryip-172-31-26-208(configure/redundancy/group/node)# connect-via 172.31.26.146ip-172-31-26-208(configure/redundancy/group/node)# node-type message-routing-nodeip-172-31-26-208(configure/redundancy/group/node)# exitip-172-31-26-208(configure/redundancy/group)# create node ha-demo-backupip-172-31-26-208(configure/redundancy/group/node)# connect-via 172.31.26.208ip-172-31-26-208(configure/redundancy/group/node)# node-type message-routing-nodeip-172-31-26-208(configure/redundancy/group/node)# exitip-172-31-26-208(configure/redundancy/group)# create node ha-demo-monitorip-172-31-26-208(configure/redundancy/group/node)# connect-via 172.31.22.77ip-172-31-26-208(configure/redundancy/group/node)# node-type monitor-nodeip-172-31-26-208(configure/redundancy/group/node)# exitip-172-31-26-208(configure/redundancy/group)# exitip-172-31-26-208(configure/redundancy)# authenticationip-172-31-26-208(configure/redundancy/authentication)# pre-shared-key key c29sYWNlaXNhZ3JlYXRldmVudHBsYXRmb3Jtd2hpY2h5b3VzaG91bGR1c2Vmb3JhbGx5b3VyZXZlbnRpbmduZWVkcw==ip-172-31-26-208(configure/redundancy/authentication)# exitip-172-31-26-208(configure/redundancy)# active-standby-role backupip-172-31-26-208(configure/redundancy)# no shutdownip-172-31-26-208(configure/redundancy)#

Monitor node

The commands you need to run on your monitor node are slightly different but for the most part, you are doing the same sort of configuration that you did on primary and backup nodes.

Before changing the router name on the monitor node, you need to run this command:

[sysadmin@ip-172-31-22-77 ~]$ solacectl cliSolace PubSub+ Standard Version 9.3.1.5Operating Mode: Message Routing Nodeip-172-31-22-77> enableip-172-31-22-77# reload default-config monitoring-nodeThis command causes a reload of the system which will discard all configurationand messaging data stored on this system.Do you want to continue (y/n)? y

This will restart your node with default configs and designate it as a monitoring node (instead of a message routing node).

Now, you can change the router name like we did with our primary and backup nodes.

ip-172-31-22-77(configure)# router-name ha-demo-monitorThis command causes a reload of the system.Do you want to continue (y/n)? y

Confirm the name was changed by running this command:

ip-172-31-22-77# show router-nameRouter Name:          ha-demo-monitorMirroring Hostname:   NoDeferred Router Name: ha-demo-monitorMirroring Hostname:   No

You can now go ahead and run the following commands (just like you did with primary and backup nodes).

ip-172-31-22-77(configure)# redundancyip-172-31-22-77(configure/redundancy)# switchover-mechanism hostlistip-172-31-22-77(configure/redundancy)# groupip-172-31-22-77(configure/redundancy/group)# create node ha-demo-primaryip-172-31-22-77(configure/redundancy/group/node)# connect-via 172.31.26.146ip-172-31-22-77(configure/redundancy/group/node)# node-type message-routing-nodeip-172-31-22-77(configure/redundancy/group/node)# exitip-172-31-22-77(configure/redundancy/group)# create node ha-demo-backupip-172-31-22-77(configure/redundancy/group/node)# connect-via 172.31.26.208ip-172-31-22-77(configure/redundancy/group/node)# node-type message-routing-nodeip-172-31-22-77(configure/redundancy/group/node)# exitip-172-31-22-77(configure/redundancy/group)# create node ha-demo-monitorip-172-31-22-77(configure/redundancy/group/node)# connect-via 172.31.22.77ip-172-31-22-77(configure/redundancy/group/node)# node-type monitor-nodeip-172-31-22-77(configure/redundancy/group/node)# exitip-172-31-22-77(configure/redundancy/group)# exitip-172-31-22-77(configure/redundancy)# authenticationip-172-31-22-77(configure/redundancy/authentication)# pre-shared-key key c29sYWNlaXNhZ3JlYXRldmVudHBsYXRmb3Jtd2hpY2h5b3VzaG91bGR1c2Vmb3JhbGx5b3VyZXZlbnRpbmduZWVkcw==ip-172-31-22-77(configure/redundancy/authentication)# exitip-172-31-22-77(configure/redundancy)# no shutdown

Once you are done configuring, you can run the following command on any of the nodes to confirm that all three nodes are Online and part of your HA group:

ip-172-31-26-146> show redundancy groupNode Router-Name   Node Type       Address           Status-----------------  --------------  ----------------  ---------ha-demo-backup     Message-Router  172.31.26.208     Onlineha-demo-monitor    Monitor         172.31.22.77      Onlineha-demo-primary\*   Message-Router  172.31.26.146     Online\* - indicates the current node

That’s it! You now have a functional HA group with your primary and backup brokers and monitoring node.

Note that by default, guaranteed messaging is disabled in HA group and can only be enabled once you have an HA group setup. It is recommended that you enable guaranteed messaging but for the purpose of this post, I will leave it as disabled.

More info on your HA group

You can get more information about your redundancy settings by running this command:

ip-172-31-26-146> show redundancyConfiguration Status     : EnabledRedundancy Status        : UpOperating Mode           : Message Routing NodeSwitchover Mechanism     : HostlistAuto Revert              : NoRedundancy Mode          : Active/StandbyActive-Standby Role      : PrimaryMate Router Name         : ha-demo-backupADB Link To Mate         : UpADB Hello To Mate        : DownPrimary Virtual Router  Backup Virtual Router----------------------  ----------------------Activity Status                Local Active            ShutdownRouting Interface              intf0:1                 intf0:1Routing Interface Status       UpVRRP Status                    InitializeVRRP Priority                  250Message Spool Status           AD-DisabledPriority Reported By Mate      Standby

You can get even more information by running this command: show redundancy detail (I am not going to show the output here).

I hope you found this post useful. For more information, visit PubSub+ for Developers. If you have any questions, post them to the Solace Developer Community.

The post Configuring PubSub+ Event Broker for High Availability in AWS appeared first on Solace.

Controlling an Arduino-based car with Solace PubSub+ Event Broker

Himanshu Gupta — Mon, 09 Mar 2020 18:21:17 +0000

Let me start with a disclaimer that I am very new to Arduino. Despite having an Electrical Engineering degree, I had never really played around Arduino until very recently. The code referenced in this post was handed down to me by my colleagues at Solace.

Recently, I presented for the first time at Solace’s meetup in New York. For this meetup, we decided to cover a specific feature of Solace PubSub+ which we thought our users and prospects would find interesting: Replay. PubSub+’s Replay feature is relatively new but extremely powerful and one that many users have welcomed.

Every meetup’s success depends on two factors:

Food and drinks
Demo

Food and drinks were a no-brainer. We got a bunch of delicious pizza and lots of craft beer (lagers, pilsners, sours, and even rosé cider!).

However, having a cool demo for your meetup requires a lot of planning. We could have easily demoed Replay via PubSub+ admin UI. While it would have gotten the point across, it wouldn’t have been visually appealing. After a lot of discussions, we decided to use a car powered by Arduino and controlled by PubSub+. The idea was simple and visually appealing. We would build an Arduino-powered car and control it by sending MQTT messages via PubSub+. Then, we will replay those messages and show the car repeating its earlier steps (this is when the crowd goes “oooh aaaah”).

In this post, I would like to show you how you can build your own Arduino-powered car and control it with PubSub+ Event Broker. Let’s get started.

Here are the steps to achieve our goal:

Assemble car from the car kit
Configure code and upload to Arduino chip
Send messages to control the car

Assemble Car from the Car Kit

** ** I don’t want to document all the steps I followed to assemble the car because, to be honest, I was just following instructions from other blogs. The instructions that came with the kit were not very helpful and very difficult to understand.

You can buy the kit from Amazon. More details about the kit can be found on this site. This YouTube video was very helpful and I highly recommend using it as a starting point.

The assembled car should look like this:

Configuring Code and Uploading to Arduino Chip

Once you have the car assembled, it is time to upload code to the Arduino chip. You can find the code on Solace’s github. In the repo, you will find car_demo/arduino/src/car_kit.ino which is the code that you will need to open using Arduino IDE and upload it to chip. Note that the following instructions are meant for the ESP8266 chip only.

Downloading Arduino IDE

To be able to upload code to the chip, you will need to download and use the Arduino IDE. I recommend downloading the software instead of just using the browser version. Once installed, open car_kit.ino in the Arduino IDE.

Configuring code

Before you can upload the code, you need to modify the code slightly to get it to work for you. You need to make a minimum of two changes:

Enter wifi details so the chip can connect to the Internet
Enter MQTT broker details

Modify the following lines by entering your Wifi details:

const char\* ssid = "<ssid>";const char\* password =  "<ssid\_pwd>";

Modify the following lines by entering your MQTT broker details:

const char\* mqttServer = "";const int mqttPort = ;const char\* mqttUser = "";const char\* mqttPassword = "";

You will need an MQTT broker running so you can enter the relevant connection settings. Of course, I would recommend using Solace’s PubSub+ Event Broker which supports MQTT. You can spin up an instance via Solace Cloud for free.

Configuring board

Before uploading this code to your Arduino chip, you will need to configure your IDE to work with the specific model of your chip as well as install PubSubClient and ArduinoJson libraries.

Arduino IDE supports a lot of ‘boards’/chips out-of-box but sadly, doesn’t support the ESP8266 chip. So, we need to add it manually to Arduino’s Board Manager. To do so, go to Arduino > Preferences and enter the URL: http://arduino.esp8266.com/stable/package\_esp8266com\_index.json

Click OK.

Now, go to Tools > Boards > Boards Manager , and search for ESP8266. You will now see ESP8266 board by ESP8266 Community. Click on Install to install it. You can also choose to install a specific version if needed. We are using 2.6.3 currently.

Installing necessary libraries

Now, you are ready to install PubSubClient and ArduinoJson libraries. You can easily do that by going to Tools > Manage Libraries and simply searching for them and installing them.

Once you have these libraries installed, you are ready to upload the code to the Arduino chip. Plug your chip into your computer via Micro USB cable. You will see a blue LED light as soon as you plug in the chip. If you don’t see the light, try toggling the white switch on the chip to turn it on.

Finally, go to Sketch > Upload to upload the code to your chip.

That’s it!

Sending MQTT messages via PubSub+

You can now go to PubSub+ UI and send some sample messages to your chip to control the car.

Getting the Chip ID

If you are using the exact code from car_kit.ino, you will need to publish messages to car/drive/${chipID} topic where chipID is your Chip’s ID. You can get the Chip ID by looking at the output of your code in the Serial Monitor. To be able to run the Serial Monitor from Arduino, you will need to make sure your chip is plugged in via USB to your laptop. When you upload the code, you can run the Serial Monitor to see the output.

Here is the output of my Serial Monitor:

As shown in the screenshot above, my Chip ID is 6994174.

Here is some good documentation on how to use Arduino’s Serial Monitor.

Sending MQTT Messages

The payload of your message should be in JSON format and should include three values:

l – power for the left motor with a range of -100 to 100
r – power for the right motor with a range of -100 to 100
d – duration in milliseconds

Negative power will make the motor turn in the opposite direction compared to positive power. For example, l:100, r:100 and d:500 will make the car move forward for 500 milliseconds. If you want to turn right, you can set l to 0 and if you want to turn left, you can set r to 0, respectively. You can control how far the car drives by adjusting the duration (d) parameter.

Here is a sample payload:

{"l": 100,"r": 100,"d": 700}

Solace PubSub+ Try Me! Publisher

It’s up to you how you would like to publish these JSON messages. For development, you can simply use Solace’s Try Me! tab and use the Publisher app there to publish these messages.

Of course, you can build a nice shiny app or program a joystick to send such messages. It’s up to you how creative you want to get!

Architecture

Now that we have our car running, let’s take a step back and look at the architecture of our setup:

On the left side, we have our assembled car powered by an Arduino chip and subscribing to an MQTT topic with QoS 1. You can see where we specify the subscription in the code:

// Subscribe to car specific drive commands char carDriveTopic[40]; sprintf(carDriveTopic, "car/drive/%lu", chipId); client.subscribe(carDriveTopic, 1);

As you can see, we are subscribing to car/drive/${chipId} with QoS of 1. Specifying QoS to be 1 causes PubSub+ to create an internal queue and map car/drive/${chipId} topic to the queue. When we publish JSON messages to the car/drive/${chipId} topic, our messages are sent to the queue and then to the Arduino chip.

On the right side, we have our Controller UI which is simply the javascript Publisher app on Try Me! tab on Solace Cloud.

Needless to say, the MQTT broker being used here is Solace’s PubSub+ Event Broker running on AWS.

That’s pretty much it! I hope you found this post useful. For more information, visit PubSub+ for Developers. If you have any questions, post them to the Solace Developer Community.

The post Controlling an Arduino-based car with Solace PubSub+ Event Broker appeared first on Solace.

Launching an Amazon EC2 instance with Solace PubSub+ Event Broker

Himanshu Gupta — Tue, 04 Feb 2020 02:35:15 +0000

In this post, I would like to show you how you can quickly spin up an Amazon EC2 instance with the Solace PubSub+ Event Broker deployed on it. Of course, you can always just use Solace Cloud and deploy on AWS through it in seconds but where is the fun in that?

I love AMIs and I cannot lie. Gone are the days when you had to read lengthy documentation and figure out how to manually install binaries. Gone are the days when you had to worry about installing different package managers to manually install all the required libraries via pip or conda.

We live in a brave new world where with just few clicks, we can spin up an Amazon EC2 instance with all the necessary binaries and libraries installed with proper configurations to quickly get started. Personally, as a developer who loves to try new technologies and who does a lot of demos, this is so convenient.

Solace PubSub+ Event Broker is “unified event broker technology available as run-anywhere software, purpose-built hardware, and a managed service that you can use together to stream events across your distributed enterprise”. PubSub+ Standard Edition is free to use, even in production.

Amazon Machine Image (AMI)

Amazon has AMI Marketplace where you can pick from numerous paid and free AMIs. In case you are not familiar with AMIs, they provide the information required to launch an EC2 instance; Solace also has an AMI available for PubSub+Event Broker: Software.

You will find necessary information on the AMI page about the instance and what it contains. In my opinion, the most important section is the Pricing Information section which tells you the estimated cost of launching an instance from this AMI.

As you can see, there is no cost to using the PubSub+ Event Broker: Software itself but there is cost involved for the underlying AWS infrastructure since you can’t use the free tier Amazon EC2 instance to run Solace PubSub+ Event Broker. Keep this in mind as you launch your own EC2 instance.

The Usage Information section is also very helpful. You can find links to several necessary resources there which help you get started.

Alright, let’s go ahead and launch an EC2 instance from the AMI. You can do it straight from the current page or you can go to the EC2 page and launch an instance from there. We will do it from the EC2 page.

Launching an Amazon EC2 instance from Solace PubSub+ Event Broker: Software AMI

Once you are there, click the Launch instance button and select Launch instance.

Great, you will now be taken to a page where you can pick an AMI from which you will launch your EC2 instance.

Click the AWS Marketplace tab and search for Solace. You will see Solace’s PubSub+ AMI as the result of your query. Click Select to proceed.

On the next page, you will see the pricing and usage information which we saw earlier. Click Continue.

Then, you will be asked to pick the underlying EC2 instance. There are minimum requirements to use PubSub+ Event Broker software so you cannot use t2.micro (free tier). The recommended instance is General Purpose t2.large (2 vCPUs, 8 GiB Memory) so we will stick to that for this demo. Note that different instances cost differently, so pick wisely.

Click Configure Instance Details. On this page, you can select which VPC you would like to deploy your instance into and pick IAM role to attach to the instance among other details. For our purpose, we will stick with the defaults. The only thing I changed is that I assigned my existing admin IAM role to my EC2 instance.

Click Add Storage and leave it as it is and click Add Tags.

This is not mandatory but I like to add tags to my EC2 instances so I can easily recognize them later. I added the ‘Name’ tag to my EC2 instance.

Click Configure Security Group to configure your security settings. The default option is to create a new Security Group with the recommended rules, or you can pick an existing Security Group. For our use case, we will create a new group with the recommended settings. Click Review and Launch. Review the details and click Launch. You will be asked to select an existing Key Pair or create a new one. I have selected an existing Key Pair which I will be using to log in to this EC2 instance.

And, that’s it! We now have an EC2 instance spinning up from Solace’s PubSub+ Event Broker software AMI.

Next, we will take PubSub+ Event Broker for a spin.

Configuring PubSub+ Event Broker

There is not much to configure, which is the beauty of this whole setup, except for an admin user which we will use to later log in to PubSub+ Manager.

First, let’s log in to our EC2 instance using the Key Pair which we selected when we launched our EC2 instance.

Go to your EC2 page and you should see your new EC2 instance there:

Select your EC2 instance and click Connect to get the ssh command. Note that you need to use the sysadmin user instead of root.

MacBook-Pro:Downloads hgupta$ ssh -i "himanshu-tokyo.pem" sysadmin@ec2-13-115-6-241.ap-northeast-1.compute.amazonaws.comSolace PubSub+ Standard

You can now see the Solace PubSub+ Event Broker instance running:

[sysadmin@ip-172-31-28-133 ~]$ solacectl cliSolace PubSub+ Standard Version 9.3.1.5Operating Mode: Message Routing Node

This proves that we were able to successfully launch an Amazon EC2 instance with Solace PubSub+ Event Broker on it. We now need to create an admin user which we will use to log in to Solace PubSub+ Manager. We can do so through Solace CLI.

First, let’s activate the CLI:

[sysadmin@ip-172-31-28-133 ~]$ solacectl cliSolace PubSub+ Standard Version 9.3.1.5Operating Mode: Message Routing Node

Now, we can run the following commands to create a new user called admin with the password admin (obviously, don’t do it in production).

ip-172-31-28-133&amp;amp;gt; enableip-172-31-28-133# configureip-172-31-28-133(configure)# create username admin password admin

That should do it.

Accessing PubSub+ Manager

Now that we have our admin user created, we can use its credentials to log in to PubSub+ Manager.

Go to the following URL on your browser: http://:8080

You will see the Solace PubSub+Event Broker Software Standard login page:

And that’s it! This whole process took me 5-10 minutes.

Once you are done playing around with PubSub+Event Broker, you can easily terminate your EC2 instance from your EC2 instance page. Make sure to do so to avoid any infrastructure charges.

I hope you found this post useful. For more information, visit PubSub+ for Developers. If you have any questions, post them to the Solace Developer Community.

The post Launching an Amazon EC2 instance with Solace PubSub+ Event Broker appeared first on Solace.