DEV Community: arih1299

A Simpler Architecture for Handling High Connection Counts and Throughput

arih1299 — Fri, 22 Oct 2021 13:00:47 +0000

I stumbled upon a blog post by Qiscus.com that talks about the architecture that lets them handle 10 million concurrent connections and throughput of 5 million messages per minute. This is the high-level Qiscus architecture shared in the blog post, which jives with what I learned from a talk their CTO Evan Purnama gave in Jakarta a couple years ago.

Pretty cool, right?

In that talk, Evan spoke about the architecture and design decisions, and the post has some of the information there as well. Here’s a few of the key points I noted from this architecture.

MQTT was chosen as the fronting layer because it’s lightweight, requires minimal client resources, and topics can be very specific, and support wildcard patterns.
They opted to decouple the layer that handles the huge incoming traffic and the layer that services the many different consumers of those events for performance and independent scalability reasons.
Message persistency is required so the system can tolerate failed consumers in a way that lets reconnected consumers pick things up from the last message they did receive, and absorb bursts of traffic in a way that lets downstream apps consume incoming data at whatever pace works for them.
Kafka was chosen for the consumption layer because of its scalability and replay capability.

Alternative Architecture

Looking at the requirements and design decision assumptions, I had been thinking if I could come up with a different approach. After a little thinking and a lot of procrastination, here’s what I came up with. Naturally, I’m using Solace PubSub+ Platform as the event platform.

Figure 2: Alternative architecture with a Solace-enabled event mesh.

With this alternative architecture, let’s start with the MQTT requirement. Solace PubSub+ Event Broker supports MQTT 3.1 and 5, along with REST and WebSocket, all without any add-ons or proxies. Events coming in via MQTT can be consumed in any other protocols/APIs used by the consumers on the right hand side, such as JMS, AMQP, WebSocket, MQTT, or even via a REST webhook mechanism.

Message persistence is taken care of with the guaranteed messaging capability of Solace PubSub+ Event Broker, and yes it can do replay as well. No biggies there. In fact, shock absorption and lossless guaranteed delivery is one of the things Solace is known for by our customers globally.

The key distinction between Qiscus’ architecture and the one I’ve presented here is that there is logically just one fabric for the messaging platform, something called an “event mesh” — like service mesh but for events instead of services. With an event mesh, there is no need for applications to move and/or translate the events from connectivity layer to the processing layer.

Tiered Architecture Working as One

This architecture is still decoupled for performance and scalability, by creating a layered architecture for the connectivity side and the processing/consumption side. The connectivity layer consists of multiple PubSub+ brokers deployed as the needed scale as well as location requirements. After all, these brokers can run on most if not all public and private clouds, containers, or virtualization platforms.

The connectivity layer then streams the events to the processing layers in a smart, efficient, and guaranteed manner, so we don’t lose messages along the way. The processing layer then scales as needed to stream those events to the back-end applications.

The Crux of EDA: Topic Routing

One major thing to note here ̶ these back-end applications have all the flexibility of the topic subscriptions with wildcards just like the one you see with MQTT protocol. If you don’t really get what this means, it could be because you are led to believe topic creation is required before anything else, and that it is expensive, and you shouldn’t have the luxury or agility of fine-grained topic addressing. In that case, you’re missing out on one of the great things about EDA, and you should really go watch a video on this topic.

Figure 3: A sneak peek of how topic routing with wildcards works

The Event Mesh

An event mesh, as you should know by now, isn’t a single event broker, nor a cluster of brokers acting as one, but a network of several nodes or pairs or clusters of event brokers working together much like how IP routers work to form a TCP/IP network.

In this alternative architecture, there’s no “subscribers” moving events from the connectivity layer to the processing layer. That’s because these PubSub+ brokers are linked to form an event mesh. A different application of event mesh is distributing events simply by leveraging the topic routing. This is really useful for multi-sites or even hybrid-cloud topology, instead of relying on cumbersome and expensive cross-site replications.

Event Mesh and Internet of Things

Internet of Things (IoT) is closely related to the use of MQTT protocol, but MQTT is being used for more than just IoT, just like Qiscus’ and many other mobile applications or even real-time dashboards.

Event mesh has a few more tricks up its sleeves for IoT solutions. One is the tiering or layering of brokers to form a tree topology that can support larger connection counts and greater geographic distributions. Another is bidirectional communications, i.e. not just streaming events from devices to back end servers, but also sending out events to devices or mobile apps as part of command and control interactions.

Our CTO Shawn McAllister produced a fun demo of those capabilities, and hope you’ll watch what I playfully like to call “The Architects’ Guide to Honking the Horn”.

Performance

While I didn’t do a performance test for this architecture, there is a public report on Solace PubSub+ performance numbers for several possible combinations of deployment types, payload sizes, delivery mode, and client connections.

Conclusion

To recap, check out this summary table for the features and requirements discussed in this post.

What I want to bring up with this alternative architecture is simplicity. Yes, simplicity. Fewer moving parts is better. Leave the event distribution and delivery to the infrastructure and provide support for the many open standard protocols and APIs to avoid rewrites.

I hope this architecture gives you some ideas for your own architecture and set of challenges. Get in touch for an ideas exchange or just quick chats in my LinkedIn or post your questions in the Solace Developer Community. And lastly, head out to Solace.Dev if you are a developer and want to get hands-on with event mesh!

The post A Simpler Architecture for Handling High Connection Counts and Throughput appeared first on Solace.

See How PubSub+ Event Broker Handles Slow Consumers

arih1299 — Wed, 21 Apr 2021 13:10:16 +0000

If you use a message broker as a communication backbone for your enterprise applications and have had issues caused by a few slow consuming applications, then this is for you. I wrote a quick LinkedIn post about this when a friend asked me what’s so different with how Solace PubSub+ deals with slow consumers. I figured the best way to show this is by doing a quick demo with more than 50 GB messages queued in the broker due to slow consumption. But I want to show more of that demo here, so I can easily refer it back should someone ask me the same question again.

Event brokers are considered infrastructure. We don’t want infrastructure to keep us up at night worrying that we’ll need to force a restart and purge tens of gigabytes of customers’ transactions because some downstream applications slowed down or the traffic just spiked due to some new marketing promotions.

With this blog post I will explain and show how Solace PubSub+ Event Broker keeps on working as expected in the event a consumer slows down or fails so other applications connected to the broker are not affected. I will also show that a large number of pending messages will not impact the fail-over time needed for Solace PubSub+ Event Broker.

The Setup

For this demo, I’m using Solace PubSub+ Cloud simply because it’s the quickest way to go. It literally just took me just few clicks and I only needed to type the broker name. Of course we can set it up using the software on many other platforms if you so wish. The brokers are deployed on AWS and run on m5.xlarge EC2 instances. This is because the environment I’m using is not using the most recent and highly recommended Kubernetes-based deployment.

For the applications, I’m using SDKPerf to simulate the producers of events and the few differently paced events consumers, running on two t2.micro instances.

Figure 1: Create Solace Cloud service

Figure 2: SDKPerf EC2 instances for publishers and subscribers

The Test

Here I’m going to simulate a situation where a broker is serving multiple applications and some of these applications are slow and unable to keep up with their respective incoming traffic. You will be able to see that the broker and the other applications will not be impacted by the slow consumer.

I created three queues, each having different topic subscriptions. If you are not familiar with this concept, just consider that we have different queues with different incoming traffic. You can also take a look at a tutorial on this concept.

Figure 3: List of Queues

Figure 4: The first queue and its subscription

Figure 5: The second queue and its subscription for slow consumer test

Figure 6: The third queue and its subscriptions for a very slow consumer test

I have ten producers sending around up to 10,000 messages per second with a payload size of 1 KB to four different topics, which were then spooled or written into the three different queues we had created earlier. This is because we want to test the spooling or storing of the messages while the designated consumer applications can’t keep up with the pace to immediately process the incoming messages.

In normal condition, all three queues will not have many unprocessed messages at any time because all the consumer applications are able to catch up with the incoming traffic. In this case, the first test queue steadily flows messages in and out to its consumer.

Figure 7: Queue 1 with a consumer fast enough to keep up with incoming traffic

The second queue consumer is a rather slow consumer that is only able to process around 800 messages per second, which is around half of the incoming traffic rate. This means there are a lot of messages waiting to be consumed in this queue.

Figure 8: Queue 2 with a consumer capable of processing only half of the incoming traffic rate

The third queue has a much slower consumer, which can only take one message for every 10 seconds. Consider this as a sample of a normal application that suddenly has a problem and is almost unable to process any messages.

Figure 9: Queue 3 with a very slow consumer

As you can see here, the fast queue consumer is not affected by the other slow consumers. You should see some network limitation based on your environment setup, but the broker maintains good service level for the healthy consumers while keeping spooling messages for the slow consumers to process later on.

Note: This test was using a single t2.micro AWS EC2 instance to run all three consumer applications, and they are likely to hit the network bandwidth capacity of the instance.

Conclusion

The key point to show here is that any healthy consumer will not be affected by the other slow consumers and piling-up messages in the other queues. The Solace PubSub+ Event Broker maintains the service for the other clients while isolating the slow consumers. Compare this to other brokers that will slow down entirely, or even crash due to the building pressure and large spool size. If your broker slows down when messages start piling up due to a slow consumers, take a stab with this free broker.

Bonus Point: Safe, Predictable Failover

When the pile gets way too big, a broker may struggle even to fail-over to its standby/backup broker because the other broker needs to ‘load’ the pending messages before it can start taking new traffic. This usually forces administrators to reset the broker to zero pending messages to allow instant start-up and resume operations. The nice thing with Solace PubSub+ Event Broker is that it will take linear time of fail-over regardless of how big the pending messages pile is. I’ll demonstrate that by stopping the primary broker EC2 instance and see what happens.

Figure 10: The SDKPerf client lost connection after the primary broker shut down.

Figure 11: SDKPerf client got back connection on the 9th attempt of 1 second interval.

Figure 12: Immediately back to business with 50 GB pending messages

Above is the log from the SDKPerf client who is able to resume operations after 9th attempt to reconnect with a 1 second interval. It immediately has access to the same 50 GB messages pending earlier. This is simply because the standby broker keeps its own copy and always be ready to take over and not having to read the data of the primary broker when it’s time for it to take over activity.

I have also recorded this demo as a video for easier and more complete view of the action.

If you like it, give it a try from solace.dev and head over to the Solace Developer Community for any help needed or to share your quick wins!

The post See How PubSub+ Event Broker Handles Slow Consumers appeared first on Solace.

Event-Driven Logging with Elastic Stack

arih1299 — Wed, 02 Dec 2020 14:00:30 +0000

As part of becoming event-driven, you need to event-enable your logging infrastructure to get the most benefit out of doing so. I’ve written blog posts about how Solace helps with logging and how to integrate Logstash with Solace. These posts assume that you aren’t already using something like Elastic stack. But what if you are? That’s the question I would like to answer with this post by presenting an example where the applications are deployed in a container platform that provides built-in log aggregator and search stack.

The Case with Container Platform

Let’s look at a specific example of an architecture where application logs are streamed into a centralized log system. In this sample, microservices are deployed as containers. These containers can be a Docker environment or Kubernetes environment such as Red Hat OpenShift Container Platform (OCP) or Google Kubernetes Engine (GKE) and many other distributions.

Kubernetes Cluster-level Logging

For Kubernetes platform, some distributions provide a native solution for cluster-level logging, but you can always build such capability on your own. You can use a node-level logging agent that runs on every node or use a sidecar pod for each application. Please refer to this documentation for more information on Kubernetes cluster-level logging.

Figure 1: Simplified Illustration of Kubernetes Cluster-level Logging

The illustration above is using fluentd as the logging agent that runs on all worker nodes and forward those logs to an Elastic stack. So, they are streaming logs and having a centralized logs server, but are they really event-driven?

The Missing Piece

It’s nice for the IT administrators to be able to query against the complete set of logs from their entire Kubernetes cluster from a single web page. It is also nice that application developers don’t need to worry about developing their application to write and send logs in a uniform and centralized manner. But the log data is now accessible only from the log dashboard. It is easy for people to read, but it’s not accessible to any other systems in real time.

The Elastic stack provides a nice tool for technical folks to dig into the logs to find errors or to investigate incidents. It is also very useful to track a specific transaction when there’s a customer complaint. But the customer service officers don’t usually have access to this system, and even if they do, it might not be that useful for them with all the gory details shown in the logs.

What if those logs could be streamed in real-time to the whole enterprise in a way that’s useful to each recipient? What if specific systems such as customer service can subscribe to specific type of events? Any new systems developed in the future can easily subscribe to the stream of logs and have the relevant information in real time, including ad-hoc applications. And even better, it can be done with many languages and open standards protocols.

Event-Enabling Kubernetes Cluster-level Logging

So now you have an Elastic stack as part of your cluster which has all of your cluster logs. Our idea of event-enabling the logging is to make these log events accessible by other systems in your enterprise, not just to the Elasticsearch log store. And what better way to achieve that than publishing the log events into Solace PubSub+ Event Broker, right?

Publishing events to Solace PubSub+ Event Broker can be done with the many supported native and open standard messaging APIs and protocols, and also with REST. If we’re looking at something like a webhook, there is a Webhook action feature provided by Elasticsearch. But this is not the best way to go for two reasons:

The ‘publishing’ will be done by Elasticsearch, but I think it’s better to let fluentd do the publishing, as the log collector who has the events firsthand.
Elasticsearch Actions are executed only if the specified conditions set on the Watchers are met. My understanding is that the Watcher feature is not available for free.

So, what’s the (better) alternative? I’d say the events should be published from the log collector directly instead of from the log store. So, I was looking at fluentd. I decided to use MQTT to publish events out to Solace PubSub+ and I’ll be using this MQTT plugin for this blog. Do note that some distribution’s native cluster-level logging feature might not support changes to their logging collector component.

I’d prefer the idea of having your own dedicated Elastic stack outside of the Kubernetes cluster. This is usually done by customers who bought their own enterprise license of the Elastic stack, or who need much bigger capacity, or simply have an existing stack. We then only need to publish events to Solace PubSub+ from this single external fluentd. Take a look at the following illustration for such approach.

Figure 2: Event Enabling Kubernetes Cluster-level Logging

Topic Routing at the Heart of Event-Driven Architecture

We can’t talk about event-driven architecture without talking about topic. Topic routing is at the heart of what Solace PubSub+ Event Broker does. It enables an efficient and elegant way of routing events between systems and it also has the key capability we need to build a dynamic network as what we call an event mesh. Please check this documentation out for more reference on topic in Solace PubSub+.

Now, we will publish logs to Solace PubSub+ Event Broker with a specific topic. Note that I say ‘with’ not ‘to’ when I talk about topic in Solace PubSub+. That’s because these topics are metadata, part of the messages we send to the broker. They’re not something we need to create beforehand. This is key to having a very flexible and performant event broker, and I invite you to take a look at my buddy Aaron’s video linked in the document referred to in the above paragraph.

To be able to publish to specific topic, we have options to set it up in the log collector itself or later in the MQTT output plugin. Or better yet, you can customize both to get your best fit solution. The idea here is to make sure we have a good design on the topic hierarchy to gain most benefit of these events. Take a look at this blog post for Solace’s best practices on topic architecture or these technical documents for a deeper dive.

We talked about Docker container and Kubernetes platform, but I’ll just quickly refer to how we can customize tag in our Docker log driver documented here. So you can specify some static topic hierarchy as well as some special template markup for that particular container. For Kubernetes, you’ll have to look at your distribution’s documentation, but I would assume it will be very similar to Docker.

I have also linked the MQTT output plugin I am using for this blog, and here’s the sample project to run a container using fluentd with MQTT output plugin configured to publish to Solace PubSub+ Event Broker running as another docker container. Pay attention to the topic_rewrite_pattern and topic_rewrite_replacement options to customize if you want to have a different topic to publish to.

The topic hierarchy enables the event subscribers to subscribe to specific, filtered events based on their needs. And it’s done simply with the topic routing, no logic or filtering effort on the subscribers’ end. For example, if you have multiple clusters and multiple applications, a subscriber can subscribe to only a specific cluster or only to specific applications regardless of where the cluster is.

In the sample project, we are using a sample tag of acme/clusterx/appz/<container id>. In this case, a subscriber can subscribe to acme/clusterx/> to get all log events from cluster X, or subscribe to acme/*/appz/> to subscribe to app Z log events from any cluster. You can also subscribe to a specific log level if that is part of your topic hierarchy. All without having to predefine any topic in the broker! And it’s only gets better when we use it with an event mesh!

An event mesh is a configurable and dynamic infrastructure layer for distributing events among decoupled applications, cloud services and devices. It enables event communications to be governed, flexible, reliable and fast. An event mesh is created and enabled through a network of interconnected event brokers. In other words, an event mesh is an architecture layer that allows events from one application to be dynamically routed and received by any other application no matter where these applications are deployed (no cloud, private cloud, public cloud). This layer is composed of a network of event brokers.

That means topic routing dynamically across multiple sites/environment, on-premises or Cloud, and with many stack and open standards APIs/protocols!

The Event Mesh

Once we event-enable our Kubernetes cluster logging by publishing events into Solace PubSub+, the log events are now accessible in real time for the whole enterprise as well as the external system’s event across Cloud environment.

The first system to benefit is the customer service application, which now can consume the log events in a guaranteed fashion, within its own capacity. The broker will act as a shock-absorber during sudden increase of incoming log events traffic, as well as keeping the events safely stored if the application is down or disconnected for some time.

Since the events are now available on the event mesh, we can expand and have many more systems subscribe to these events. They can use Solace native API (SMF), JMS for the legacy Java enterprise apps, Webhooks, or even Kafka connectors. And they can be subscribing right there on the same datacenter, or somewhere on the Cloud.

The idea of having an event mesh is that these systems can dynamically subscribe to events from anywhere, and the mesh will route the events across. No fixed point-to-point stitching, and no wasted bandwidth for synchronizing events across multi cloud environment when there’s no one subscribing.

Conclusion

I’ve covered the advantages as well as shortcomings of having a centralized logging system such as Kubernetes cluster-level logging and Elastic stack, and ways to event-enable it to realize greater potential of your enterprise logs. The event mesh capability uniquely provided by Solace PubSub+ extends this even more. Hopefully this idea sparked your interests and curiosity to go and event-enable your own systems and reap the event-driven architecture promises! What are you waiting for? Get on to Solace Developers to start building your own event-driven logging with Solace PubSub+ and Elastic!

The post Event-Driven Logging with Elastic Stack appeared first on Solace.

Take Your Distributed System to the Next Level with Event-Driven Logging

arih1299 — Fri, 10 Jul 2020 14:39:56 +0000

Logging is an important aspect of a distributed system, but some applications don’t even have proper logging — some only have file-based logging, others have very basic logging, and some have proper structure with rolling and rotations set up. Many have embarked on the journey of having those logs stored into a database of some sort, for storing and querying the mountains of data flooding in from the many systems in their enterprises. While this approach helps search the contents of the logs, it also presents some challenges. The first challenge is writing those large volumes of logs into the database, and the second problem is the amount of time required to be able to query the large volume of data in the database.

Figure 1: Common Pattern of Logging to a Relational Database

I’ll walk through some common architecture patterns around logging infrastructure, and how Solace offers an event-driven logging alternative that offers a few key advantages over other patterns. Then I’ll share the steps to configure Logstash as the log storage component that consumes a stream of log events via Solace PubSub+ Event Broker.

First Challenge: Ingestion

If you’re using a relational database, the first common problem you’ll encounter is that writing to a relational database is not a fast operation for a database. This isn’t a big deal for small-scale operations, but for many enterprises this proves to be a significant operational issue.

In the worst-case scenario, business processes are stuck waiting for logging operation to complete for several seconds while the actual customer transaction has completed within less than a second. This also bleeds to the infrastructure layer, where you have to increase the scale of your service infrastructure by two-fold or more to support the thousands of transactions that are hogging capacity, mostly to ‘wait’ for the blocking database writing operations.

Some people have learned to not make the database writing operation part of the main business process flow, or not use a relational database at all. There is more than one way to do this, especially with the vast options of technology stack we have today.

One pattern I would like to elaborate on is the use of a queueing mechanism as the intermediary pipeline. In this pattern, applications send the logs as messages to a message broker’s queue and immediately comes back to their transaction flow. This supposedly works better since sending to a queue should be much faster than a database insert operation, and the messages waiting to be written to database are safe and sound in the queue. Sounds clever, right?

It does reduce blocking time to ‘send’ the log entries and buffering if the next step of writing to an actual database becomes slow, but there are big challenges and missing benefits when you only rely on a queue instead of a publish-subscribe pattern. I’ll get back to this in a minute.

Second Challenge: The Query

Once you have all your log entries recorded into a database, it’s all good right? Not always. Depending on what kind of database you are using, writing to and querying the database can take way longer than you’d like them to. You might face problems such as querying the database taking ages to complete, or it simply giving you data from an hour ago.

The Case of Intermediary Queue

Let’s revisit the intermediary queue approach. What could possibly go wrong? Let’s take a closer look.

Figure 2: Log Queue

Imagine what would happen if the database is slow or simply stops working. Eventually, the queue will build up a massive number of unprocessed messages, and the broker will eventually stop ‘acknowledging’ the incoming log messages, leaving the applications again stuck waiting for the logging operations to complete. Kind of the same impact with the direct database write, just at a later time. Imagine if other critical business processes relying on the same message broker are also impacted!

Consider a not-so-distant future, where some other systems need the same set of logs. Then you’re looking at either querying the log database (fully aware of the delayed data and slow response time) or slapping another send-to-queue code into the apps to have a duplicated stream. Doesn’t it sound like much fun, does it?

Event-Driven Logging Architecture

What’s an event-driven logging architecture? Well, each log entry is an event that other enterprise applications can subscribe to, receive, and react to. Source applications just need to worry about emitting each log as an event and that’s it! Those events can be routed to many applications via a range of APIs and protocols, now or later, without touching the producer of those events.

Figure 3: Event-Driven Logging

This diagram illustrates a few key points. An event can be sent simultaneously to many recipients with the publish-subscribe pattern. In this diagram, we’re publishing to topics and consuming from queues. The queue is acting like a subscriber and persists the events for a specific application to consume in a guaranteed fashion. Learn more about topic to queue mapping, one of Solace’s core concepts.

Each application can subscribe to whatever topics indicate the unique set of information they need. Note that this filtering is done by the broker, not after the events have been transferred all the way to the applications. In fact, the benefit of topic routing with wildcards is so great that we produced a video about topic wildcards!

Modern Search Stack and Big Data

Say you’ve modernized your technology stack, including your log database. You now have a bunch of new shiny big data technologies lying around, and a quick new search engine to query your logs. Let me guess, Elasticsearch or Splunk? Apache Spark? Kafka as the ingestion to Hadoop-based platforms? Don’t worry, we’ve got your back!

Figure 4: Event-Driven Logging – Modernized!

It’s the same event-driven logging architecture, but with fresh new applications subscribing to the log events!

I wrote a separate blog on how to integrate Solace PubSub+ Event Broker with Logstash as part of an ELK stack. Check it out!

Also take a look here for more stuff you can integrate with Solace.

What’s Next

With the power of event-driven architecture in your hands now, why don’t you try out stream processing or maybe a machine learning application that also subscribes to your log event stream either directly via topic subscription or in a guaranteed fashion via queue just like our sample with Logstash. The cloud is the limit!

The post Take Your Distributed System to the Next Level with Event-Driven Logging appeared first on Solace.

Integrating Solace PubSub+ with Logstash

arih1299 — Thu, 04 Jun 2020 14:37:51 +0000

Logstash is a free and open source server-side data processing pipeline made by Elastic that ingests data from a variety of sources, transforms it, and sends it to your favorite “stash.” Because of its tight integration with Elasticsearch, powerful log processing capabilities, and over 200 pre-built open-source plugins that can help you easily index your data, Logstash is a popular choice for loading data into Elasticsearch.

I will share here the steps needed to connect Solace PubSub+ with Logstash as a log storage component. For this demo, I’ll be using a Solace PubSub+ Event Broker and a Logstash server. I’ll use dummy log entries instead of real application logging for now. I will set up a Logstash server in my Google Cloud Platform and use Solace PubSub+ Event Broker: Cloud to host my event broker.

Solace PubSub+ Event Broker: Cloud

It’s free to run a Solace PubSub+ Event Broker on Solace PubSub+ Event Broker: Cloud. Just go to https://cloud.solace.com and set up your own instance, then create a new event broker on AWS with a few clicks as shown here.

Figure 1 Create your Solace PubSub+ Event Broker

Don’t blink! No need to go and make a cup of coffee. Your broker should be ready now.

Figure 2 Your shiny new event broker

Figure 3 Connection information

Leave it at that and set up the Logstash server next.

Logstash Server (ELK Stack) on Google Cloud Platform

For the Logstash server, I’m using a Bitnami ELK image readily available from Google Cloud Platform Marketplace.

Figure 4 Set up ELK Server on GCP

Figure 5 Configure the new ELK deployment

Once the deployment is done, you can log in to this new instance via SSH or gcloud or pick your favorite from the options available!

Download Solace JMS API

The first thing you need to do is download the Solace JMS API Library to this server. This library will be used by Logstash to connect to Solace PubSub+ using JMS API. You can put the library anywhere, but for this demo I will create folder /usr/share/jms and put all the JAR files underneath.

You can download the Solace JMS API Library. For this demo, I use wget to download the library directly into the server.

Figure 6 Download the Solace JMS API Library

Once you have the zip file of the Solace JMS API Library, you can extract and copy the JAR files under lib/ directory to our /usr/share/jms folder. You will use these libraries later on when we configure the JMS input for Logstash.

Figure 7 Copy the JAR files

Reset Kibana User Password

You will also use the Kibana dashboard later on to verify that your log entries can be processed correctly by the Logstash JMS input. For that, change the user password with your own password and restart the Apache web server to apply the new setup.

Figure 8 Change ELK default password

Once you’ve restarted the Apache web server, log in to the Kibana by accessing /app/kibana from your web browser.

Figure 9 Test the new Kibana password

Prepare the Logstash Queue

Next you’ll configure Logstash to consume from a Queue to have a guaranteed delivery of log events to Logstash. To do that, go to the Solace PubSub+ Manager by clicking the Manage Service link on the top right corner of your service console.

Figure 10 Manage the event broker

To create a new Queue, simply go to the Queue menu on the left and click the green Create button on the top right. Give any name you want but remember you will use this to configure the Logstash JMS input later on. For now, name it LogstashQ.

Figure 11 Create a new Queue

Keep all the default values for this demo as I will not go into details around Queue for this demo.

Figure 12 Use the default values for the new Queue

Topic to Queue Mapping

This is a very important part of setting up the Queue for our log flow, in which you will make configurations for the Queue to subscribe to the topics that we want to get and later be consumed by Logstash. For this step, go to the Subscriptions tab menu and click the + Subscription button on the top right.

Figure 13 Add subscription to the new Queue

Now, you want to attract all events related to the logs in the Acme enterprise. For that, add a topic subscription with a wildcard after the acme/logs/ hierarchy. You can also have multiple different subscriptions for this Queue if we wish.

Figure 14 Add acme/logs/> subscription

Test the Log Queue

To make sure the mapping is correct, try to publish a fake log event via the Try Me! tab on the service console. Connect both the Publisher and Subscriber components and add acme/logs/> subscription in the Subscriber section.

Now test by sending a dummy log entry line as the Text payload of the test message, and verify that the message is successfully received by the Subscriber component.

Figure 15 Test Publishing a Sample Log Line

Now go to the Queues and into our LogstashQ. Verify that the same message is also safely queued in this queue via the Messages Queued tab.

Figure 16 Log event queued in the new Queue

JNDI Connection Factory

We will be using a JNDI-based connection setup in the Logstash JMS input later on. For that, let’s verify that you have a JNDI Connection Factory for that use. Solace PubSub+ Event Broker comes with an out-of-the-box connection factory object that you can use, or you can create a new one if you want by going to the JMS JNDI menu and Connection Factories tab.

Figure 17 Verify JNDI Connection Factory

Configure Logstash

This Logstash configuration follows the samples already created by the ELK stack image from Bitnami. However, your environment might vary.

Go to /opt/bitnami/logstash and add a new JMS input section to the existing configuration file under the pipeline folder.

root@ari-elk-vm:~# root@ari-elk-vm:~#root@ari-elk-vm:~#root@ari-elk-vm:~# cd /opt/bitnami/logstash/root@ari-elk-vm:/opt/bitnami/logstash# vi pipeline/logstash.confroot@ari-elk-vm:/opt/bitnami/logstash#

Please refer to the Elastic documentation for JMS input plugin for more details. I have created the following configuration as per my newly created Solace PubSub+ Event Broker and the Logstash queue earlier.

Since we are going to consume from a queue, you need to set the pub_sub parameter as false. Pay attention to the jndi_context parameters and make sure you have the correct information based on the connection information seen in PubSub+ Cloud service console earlier.

If you’re not familiar with Solace, pay attention to the principal argument where you need to append ‘@’ followed by the Message VPN name after the username.

input{ beats { ssl => false host => "0.0.0.0" port => 5044 } gelf { host => "0.0.0.0" port => 12201 } http { ssl => false host => "0.0.0.0" port => 8080 } tcp { mode => "server" host => "0.0.0.0" port => 5010 } udp { host => "0.0.0.0" port => 5000 } jms { include\_header => false include\_properties => false include\_body => true use\_jms\_timestamp => false destination => 'LogstashQ' pub\_sub => false jndi\_name => '/jms/cf/default' jndi\_context => { 'java.naming.factory.initial' => 'com.solacesystems.jndi.SolJNDIInitialContextFactory' 'java.naming.security.principal' => 'solace-cloud-client@msgvpn' 'java.naming.provider.url' => 'tcps://yourhost.messaging.solace.cloud:20290' 'java.naming.security.credentials' => 'yourpassword' } require\_jars=> ['/usr/share/jms/commons-lang-2.6.jar', '/usr/share/jms/sol-jms-10.8.0.jar', '/usr/share/jms/geronimo-jms\_1.1\_spec-1.1.1.jar'] }}output{ elasticsearch { hosts => ["127.0.0.1:9200"] document\_id => "%{logstash\_checksum}" index => "logstash-%{+YYYY.MM.dd}" }}

By default, the ELK image from Bitnami does not do auto-reload of the Logstash configuration. In that case, we can restart the Logstash service using the built-in ctlscript.sh script. Check the logstash log after that to make sure that it has successfully connected to our Solace PubSub+ Event Broker. The log should have similar entries as shown below.

Figure 18 Logstash log with JMS input

Also verify that the Logstash connection exists in our Solace PubSub+ Manager, via the Client Connections menu and Solace Clients tab.

Figure 19 Verify the Logstash connection exists

Since you have already tested on sending a fake log entry earlier, by now the Logstash JMS input should have already consumed that message. Verify that by going into the LogstashQ and check the Consumers tab, where you’ll want to look at the “Messages Confirmed Delivery” statistic and make sure the number has increased.

Figure 20 Verify Messages Confirmed Delivery

Create Kibana Index Pattern

To be able to see the log entry in the Kibana dashboard, create a simple index pattern of logstash-* and use the @timestamp field as the time filter.

Figure 21 Create a new index pattern in Kibana

Figure 22 Use @timestamp as the time filter

You can now go back to the home dashboard and should see an entry exists there from the dummy log event. And that’s it! Now you should be able to stream log events by publishing to a PubSub+ topic like acme/logs/app1/tx123/debug, for example.

Figure 23 Verify the log entry shows up in the Kibana dashboard

Summary

So that’s how to integrate Solace PubSub+ Event Broker with Logstash. There are a lot of other stacks that Solace can integrate with as well. Check them out!

The post Integrating Solace PubSub+ with Logstash appeared first on Solace.

How to set up Solace PubSub+ Event Broker with OAuth for MQTT against Keycloak

arih1299 — Thu, 12 Dec 2019 17:12:29 +0000

OAuth 2.0 and OpenID Connect (OIDC) are getting more and more popular as authentication and authorization protocols. OIDC also uses JSON Web Tokens (JWT) as a simple token standard. Another protocol that is gaining popularity is MQTT. Since Solace PubSub+ Event Broker supports all these protocols, why don’t we see how they all work together nicely in a simple demo? We will use the Keycloak server as the authorization server and a simple dotnet core application to build a full end-to-end demo.

Set up the servers

For this blog, I’m running both Solace PubSub+ Event Broker and the Keycloak server as Docker containers on macOS. The configuration steps are the same regardless where we run the servers. One thing to note is that we need connectivity from the Solace PubSub+ Event Broker to the Keycloak server.

Run this command to set up Solace PubSub+ Event Broker software in your local Docker environment.

$ docker run -d --network solace-net -p 8080:8080 -p 1883:1883 --shm-size=2g --env username_admin_globalaccesslevel=admin --env username_admin_password=admin --name=mypubsub solace/solace-pubsub-standard:9.3.0.22

Run this command to set up the Keycloak authorization server in your local Docker environment.

$ docker run -p 7777:8080 \ 
--network solace-net \ 
--name keycloak \ 
-e KEYCLOAK_USER=user \ 
-e KEYCLOAK_PASSWORD=password \ 
-e DB_VENDOR=H2 \ 
-d jboss/Keycloak:7.0.0

If Port 8080 is already used on your local machine, change it to any other available port (the first port in the -p argument).

Using the docker network parameter enables you to access this host using hostname from other Docker instances. If you don’t have one yet, create one with the following command:

$ docker network create solace-net

Once the Keycloak server container is started, we can verify it from the Keycloak homepage.

Figure 1 Keycloak Homepage – use the port we published in the docker run command

Keycloak as the Authorization Server

An authorization server grants clients the tokens they can use to access protected resources. In this setup, we are using the Keycloak server as the authorization server.

In this section, we will set up the user account in the authorization server. We will use this user to get the access and ID token from the authorization server.

The first step is to log in using the username and password defined during the Docker container creation.

Figure 2 Use the username and password defined as environment variable

By default, Keycloak is set up with a built-in realm called Master. For simplicity, we will use this realm for our user. If you want to create a new realm, you can do that as well.

Create a Client

The next step is to create a new client in the realm. We do this by clicking the Create menu on the top right of the clients table.

Figure 3 Client Admin Page

Enter a client ID and choose openid-connect as the client protocol. We will use this client for our OpenID Connect test.

We can leave the Root URL field empty for this demo.

Figure 4 Create a new client

Next, enter the mandatory Redirect URLs for this client. Since we’re not going to use this for the Web, we can use a simple URL such as localhost/* for this demo.

Figure 5 Enter the Redirect URL

Optionally, change the default Access Token Lifespan to a longer period if you want to use a single token for multiple tests spanning several minutes or more.

Figure 6 Change the default Access Token Lifespan

Configure Client Scope for a Custom Audience

Additionally, we will add a custom audience called “pubsub+” to this client for audience validation. Keycloak will add the client ID as the audience attribute value and provide a few ways to add a custom audience. For this test, we create a client scope by the name of “pubsub+” and include a custom audience there. We then include this client scope in the client we created earlier.

Figure 7 Create a client scope to have a custom audience value

Figure 8 Add the client scope to the Solace client

Configure Solace PubSub+ Event Broker

Create an OAuth Provider

The first step is to configure an OAuth provider for OpenID Connect in the Solace PubSub+ Event Broker.

Figure 9 Create a new OAuth provider

We will create the new OAuth provider based on the Keycloak authorization server. We will enable audience validation and authorization group as per our Keycloak client configuration, use the JWKS URL from the Keycloak, and use the preferred_username field from the id_token as the username claim source.

We will look for audience and authorization group claims from the access_token since Keycloak will have those in access_token by default. This is not a mandatory option. Simply configure against how your authorization server would have the claims.

Refer to the screenshot below for the configuration values and don’t forget to enable this provider by toggling the Enabled option on.

Figure 10 Set up a new OAuth provider

We will not configure Token Introspection for this test.

For the username, we will find the username claim from the id_token from the preferred_username attribute rather than from sub. This attribute should carry the value of “user” as the username we use in Keycloak. We will use the username “user” in our sample application.

And as a bonus feature, not really part of OpenID Connect, we can enable API username validation so that the broker validates if the username provided in your application API call matches the username claim extracted from the token.

Figure 11 Set up a new OAuth provider (2)

Enable OAuth

Next, we will set up the Solace PubSub+ Event Broker’s Client Authentication to enable the OAuth Authentication. This is done by toggling the OAuth Authentication switch and then select one of the available OAuth providers as the default provider for OAuth authentication. This value is used when a client using OAuth authentication does not provide the OAuth provider information.

Notice that we disable the Basic Authentication and Client Certificate Authentication for this test to ensure that our broker will only do OAuth authentication.

To keep it simple, we will use default Authorization Type of Internal Database.

Figure 12 Enable OAuth Authentication and set the Default Provider Name

Create an Authorization Group

The next step is to make sure we have configured authorization groups to be used by the broker to validate the authorization claim in the token.

Figure 13 Create an authorization group

Let’s create a sample authorization group by the name “pubsub+” to be used later by the OAuth client.

Figure 14 Enable and select profiles

Make sure to enable this new authorization group and feel free to play around with the ACL and Client Profiles. For now, we will settle with the default profiles for both.

Ready for Test

Now we have the Solace PubSub+ Event Broker and the Keycloak authorization server configured, we are ready to run some tests.

Sample Project

This is a sample .Net Core application to test the OAuth authentication and authorization features. It will take the two arguments access_token and id_token and subscribe and publish a message.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <RootNamespace>solace_dotnet_mqtt</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
  </ItemGroup>
  <ItemGroup>
  </ItemGroup>
  <ItemGroup>
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="M2MqttClientDotnetCore" Version="1.0.1"/>
  </ItemGroup>
</Project>

using System; 
using System.Text; 
using M2Mqtt; 
using M2Mqtt.Messages; 

namespace solace_dotnet_mqtt 

{
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length < 2) {
                Console.WriteLine("Usage: dotnet run <access_token> <id_token>");
                Environment.Exit(-1);
            }

            MqttClient client = new MqttClient("localhost");

            client.MqttMsgPublishReceived += client_MqttMsgPublishReceived;
            string clientId = Guid.NewGuid().ToString();
            string solace_oauth_provider = "keycloak-openid";
            string oidcpass = "OPENID~" + solace_oauth_provider + "~" + args[1] + "~" + args[0];
            client.Connect(clientId, "user", oidcpass);
            string strValue = "Hello World!";
            client.Subscribe(new string[] { "test/topic" }, new byte[] { MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE });
            client.Publish("test/topic", Encoding.UTF8.GetBytes(strValue), MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE, false);
        }

        static void client_MqttMsgPublishReceived(object sender, MqttMsgPublishEventArgs e)
        {
            Console.WriteLine("Message received: " + System.Text.Encoding.UTF8.GetString(e.Message));
        }
    }
}

Prepare the Tokens

To get the tokens, we can use tools such as Postman to get new tokens from the Keycloak authorization server.

We can simply create a new request and go to the Authorization tab, select OAuth 2.0 as the type, and click the Get New Access Token button on the right panel.

Figure 15 Use Postman to get the tokens using OAuth 2.0 Authorization

Figure 16 Get to the New Access Token menu

Fill in the token request details as per the sample below. Make sure you enter the correct client ID.

Since we have set the client ID with public access, we don’t need to enter any client secret. And for Scope, we will use openid so that this is handled as OpenID Connect.

For State , we just put any value for this test.

Figure 17 Use Auth URL from the Keycloak server

Figure 18 Use Access Token URL from Keycloak and change the Client Authentication setting

You will be presented with the Keycloak login page to authenticate yourself to be able to get the tokens. Use the username “user” and password “password” that we used when running the Docker container for the Keycloak server.

Once you get the tokens, you can copy both the access_token and id_token for use in the test later.

Figure 19 Copy the Access Token

Figure 20 Copy the id_token

Peek into the Tokens

You can peek into the tokens to see the contents and attribute values. You can go to https://jwt.io and simply paste the token into the Encoded text box on the left.

Figure 21 Decode a JWT access token

The highlighted aud and scope attributes are the ones we use in this test. As you can see, the aud value of pubsub+aud is extracted from the token, as well as the scope of pubsub+.

Figure 22 Decode a JWT id_token

As we can see, the id_token will contain a preferred_username attribute with the value “user”.

Run the Test Program

To test with the provided sample program, simply run the dotnet run command with both tokens as arguments. For this sample, I have simply used localhost for the Solace PubSub+ Event Broker address as we are running it on Docker locally. Upon successful run, the program will simply print out “Message received: Hello World!” to the console.

ari@Aris-MacBook-Pro solace-dotnet-mqtt % dotnet run [access_token] [id_token]Message received: Hello World!^C
ari@Aris-MacBook-Pro solace-dotnet-mqtt %

I hope you find this blog post useful. For more information about the topic, please refer to the following:

The post How to set up Solace PubSub+ Event Broker with OAuth for MQTT against Keycloak appeared first on Solace.

Alternative to GO-JEK’s Kafka Ingestion Architecture

arih1299 — Fri, 27 Sep 2019 15:53:19 +0000

GO-JEK is one of the big names in the South East Asia technology start-up scene. It started off with a ride-hailing service and continues to grow into many areas. With so many services and a huge user base, the engineering behind its services should be an interesting case study. In this article, I will look at GO-JEK’s Kafka ingestion architecture and try to provide an alternative to it.

GO-JEK’s Existing Architecture

A couple of GO-JEK engineers have written articles around GO-JEK’s engineering work. One of them explains how they do their event ingestion into a Kafka cluster. As the figure below shows, GO-JEK’s existing architecture starts with the Producer App and ends with the Mainstream Kafka cluster.

According to the blog post, GO-JEK has solved a few problems and made the following improvements in the architecture:

Producer application developers don’t need to learn a new API , namely the Kafka client API, since they can just use REST API calls.
Messages are now acknowledged back to the producing applications.
No buffering technique or storage is required for the producer applications when the Kafka cluster is not available.

Why an Alternative Architecture

Looking at the existing architecture, I can’t help but think that there can be a simpler architecture with fewer components to solve the same set of problems. As an external party looking at only a blog post, I may have missed some details or made incorrect assumptions. But, the beauty of it is that this would be a fresh point of view and hopefully it will offer some value to the owner of this architecture.

So why is an alternative architecture necessary? Because having a couple of Kafka clusters, Java applications, and a cache to move data shouldn’t need so much setup and overhead. I would like to take a stab at providing a simpler architecture, one that hopefully has fewer moving parts, is built more as an infrastructure rather than part of the business applications, and can avoid additional complexities.

The Proposed Architecture lterations

All we want to do is to get the events from the producer applications across the Kafka cluster. How hard could it be? Well, for starters, the producer applications now need to be able to “talk” to Kafka.

Proposed Architecture Iteration 1

So, REST is what the applications use? That’s totally fine. Then we should have a thing between the applications and the Kafka cluster that connects REST to the applications and talks to Kafka on the other end. And it must be a centralized, shared thing so new applications can just get onboard without the need to keep adding this new thing. This new “REST to KAFKA Thing” must be able to give acknowledgements back to the producing applications as well as buffer the events when the Kafka cluster is not accessible.

Back to the original blog post, there was a comment that suggested the writer use Kafka REST Proxy and Mirror Maker. One of the reasons why the writer did not accept the suggestion is because this suggestion would lack the ability to buffer and retry the event publishing to Kafka. This is where the “Fronting Failover”, “Fronting Worker”, and Redis came into the picture. It looks like a lot of new moving parts being added to deal with the fact that Kafka cluster can be unavailable at times.

Are there any other ideas? Of course.

Proposed Architecture Iteration 2

This time two components are added: a Solace PubSub+ Event Broker and a Solace PubSub+ Connector for Kafka Source. Solace PubSub+ Event Broker gives you the REST interface as well as the guaranteed messaging feature with acknowledgement. So, the producer applications don’t need to worry about losing their events. It also leverages the Solace Messaging API and the Solace PubSub+ Connector for Kafka Source as the source connector to stream the events directly to the Kafka cluster using the Kafka Connect API.

As an enterprise-grade event broker, PubSub+ provides high-availability and replication features without any dependency on other software or technology. This surely helps in terms of fewer moving parts to manage by the operation team. This PubSub+ Event Broker will also queue up the events in case the connector is not able to pass through the events into Kafka. This is where the guaranteed messaging feature comes into play.

The question that may arise is how big of a buffer the message broker could handle if the Kafka cluster is unavailable. No problem. PubSub+ event brokers handle slow consumers in such a way that they do not impact the performance of the producers or fast consumers. For guaranteed messaging, publishers and fast consumers are identified and prioritized over slow or recovering consumers even as message storage to slow consumers continue to increase.

The Solace PubSub+ Event Broker Appliance currently supports up to 6 TB of message spool for this buffering need, while the software allows around 800 GB of message spool. It’s basically a matter of sizing and deployment choice.

Conclusion

Solace PubSub+ fits the bill as an infrastructure thing that does not really need a lot of attention. It is more like your network equipment that just needs to be set up once and can be left untouched most of the time. But the Solace PubSub+ Connector for Kafka is still a piece of software! It is still the better option to integrate with Kafka using the Kafka Connect API. When Kafka or whatever your event store natively supports open standards such as JMS and MQTT, there will be no need for such additional connectors.

So, use REST, checked; message acknowledgement, checked; buffering, checked!

Visit the Solace website to get more information about the Solace PubSub+ Platform and the Solace PubSub+ Connector for Kafka. Share your developer experience and ask questions at Solace Community.

The post Alternative to GO-JEK’s Kafka Ingestion Architecture appeared first on Solace.