DEV Community: Thomas K

Using GraphQL to Query the State of Your Event Mesh

Thomas K — Tue, 16 Feb 2021 15:16:59 +0000

In this blog post, I will explain how you can leverage the power of GraphQL’s extensible API model to better understand the architecture and state of your event mesh. An event mesh is an interconnected network of event brokers that distributes events among decoupled applications, cloud services and devices, anywhere in the world and running in any cloud, edge or on-premises environment.

Here’s the high-level architecture of an event mesh:

In this diagram, you have a network of Solace PubSub+ Event Brokers deployed on-prem, in public clouds, and even in a private cloud. They link together via a protocol called dynamic message routing (DMR). The key thing to understand about DMR is that events will only get routed to where they need to go without the need to pre-define the path or the routing rules.

For example, if you have an Apache Beam[Google Cloud Dataflow](https://cloud.google.com/dataflow) microservice producing an event inGoogle Cloud Platform and you want it to be routed to a microservice in AWS, all the microservice in AWS has to do is express interest in the Beam\Dataflow event via a ‘Topic Subscription’.

REST-ful API to manage/monitor PubSub+ Event Broker

A core component of the event mesh is the event broker. Solace’s PubSub+ Event Broker is a multi-protocol event broker that comes in multiple form factors – hardware appliances, software, and as a service. The event broker handles client connectivity to the event mesh and also routes data to the neighboring nodes in the event mesh.

Every PubSub+ Event Broker exposes a REST-ful management interface – called the Solace Element Management Protocol (SEMP) – that gives you access to various internal statistics and administrative functions for the event broker.

For example, by running a SEMP query against a broker, you can figure out which IP-Addresses/users are connected to the broker. Another query will give you an understanding of which Topic Subscriptions are active on a broker.

GraphQL as an Abstraction Layer on Your Event Mesh

Solace has monitoring solutions, such as Solace PubSub+ Monitor and PubSub+ Insights built on top of the SEMP API and along with gathering logs, but we will be using GraphQL to extend the SEMP API.

By taking advantage of the SEMP API, you can periodically poll all the nodes of your event mesh to store statistics in a database. You can then use a GraphQL server to front the results of the database and can build out an expressive API that will allow you to answer questions such as :

Which node has a subscription active on a specified topic?
Which user(s) is subscribing to a specific topic?
Which links of my event mesh are experiencing the most traffic?
Which topics are being most subscribed to across the event mesh?

A Sample Implementation

I’ve written a basic Node application that queries nodes of the event mesh and store it in a database using Prisma – a powerful JavaScript ORM library. I also wrote a GraphQL server that exposes the tables in Prisma via GraphQL. The utility currently stores and exposes client and topic subscriptions but can easily be extended to cover more use cases.

You can find the GitHub repo/setup instructions here.

Here are some sample queries and the results that you can get with this application stack:

Which node is my topic subscribed to from?


query{
findClientsForSubscription(topicString: "my/topic")
{
   subscription
   clients {
     nodeName
   }
}
}

Result:


{"data": {
  "findClientsForSubscription": [
  {
    "subscription": "my/topic",
    "clients": [
     {
       "nodeName": "Cloud Broker"
     },
     {
       "nodeName": "Local Broker"
     }
    ]
   }
  ]
 }
}

Which user and node is my topic subscribed to from?


Query{
findClientsForSubscription(topicString: "my/topic"){
  subscription
  clients {
    nodeName
    userName
  }
}
}

Response:


{"data": {
  "findClientsForSubscription": [
    {
      "subscription": "my/topic",
      "clients": [
        {
          "nodeName": "Cloud Broker",
          "userName": "local-user"
        },
        {
          "nodeName": "Local Broker",
          "userName": "default"
        }
      ]
    }
  ]
}
}

Conclusion

An event mesh is a powerful platform that dynamically routes events to their intended destination and nowhere else. GraphQL is a very powerful query language that is able to help you find answers to specific questions about your event mesh.

The post Using GraphQL to Query the State of Your Event Mesh appeared first on Solace.

A scalable WebSocket fanout solution for your High Performance Kafka deployment

Thomas K — Tue, 02 Feb 2021 20:52:24 +0000

In this post, I will show how you can take advantage of a high performance Kafka deployment, the Kafka Connect framework and Solace PubSub+ to build out a scalable websocket distribution backbone for a contrived market data distribution use case.

Market data distribution

For the purpose of this article, I'll be focusing on a financial service use case- specifically market data distribution. Typically market data will be streamed over an exchange using a proprietary protocol. For your internal microservices or UIs, you would consume/normalize/and republish over an event broker of your choosing such as Apache Kafka.

Fanning out to thin clients via WebSocket

For a lot of messaging use-cases, one of the end-points you would likely need to stream to is a web browser. Kafka doesn't support the WebSocket protocol that you could use with your HTML5/JS applications. To bridge from Kafka to WebSocket, we will need to introduce two new components:

Solace PubSub+ Event Broker - A multiprotocol event broker that is able to publish and subscribe natively through a variety of protocols such as JMS, AMQP, MQTT, Solace WebSocket, and REST.
Solace PubSub+ Kafka Sink Connector - This will bridge from Kafka event streams into Solace.

Filtering messages to a thin-client

A Kafka Topic is a contiguous stream of messages organized via a partition. A sample topic-partition may consist of 1000's of streams streams of data with the following structure on a Topic

Topic: PRICES
Partition: 1
Offset 0:
{"tickerSymbol":"AAPL","countryCode":"US","exchange":"NASDAQ","bidSize":10.0,"askSize":10.0,"bidPrice":100.0,"askPrice":105.0}
Offset 1:
{"tickerSymbol":"GME","countryCode":"US","exchange":"NYSE","bidSize":10.0,"askSize":10.0,"bidPrice":1000.0,"askPrice":995.0}
Offset 3:
{"tickerSymbol":"MSFT","countryCode":"US","exchange":"NASDAQ","bidSize":10.0,"askSize":10.0,"bidPrice":142.0,"askPrice":143.0}
...

However, it's likely that a user would only want to subscribe to a subset of the streams rather than the entire topic/topic partition... usually via an identifer like a ticker symbol.

As such, a finer level of topic filtering will be required which can be accomodated via the subscription semantics of a protocol like MQTT.

For example, a user may only be interested in the AAPL stock, so they can issue a subscription to Solace PubSub+ with the following semantics:
US/NASDAQ/AAPL

Or if they wanted to subscribe to all US stocks, they can issue a subscription with the following semantics:
US/NASDAQ/#
which will deliver all messages from the NASDAQ market to the client.

This finer grained topic filtering is not something that Kafka provides out of the box. Hence the Kafka Connector would have some simple logic to translate the contents of the payload into a topic to be passed onto Solace. A simplistic implementation of this function would be as follows:

 private class SolMarketDataObject {
        public String tickerSymbol;
        public String countryCode;
        public String exchange;
        public double bidSize;
        public double askSize;
        public double bidPrice;
        public double askPrice;

        public String createSolaceTopic() {
            return countryCode + "/" + exchange + "/" + tickerSymbol;
        }
    }

In addition, the Solace PubSub+ Broker will guarantee order of the streams to the HTML5 front end in the order it was received from the Kafka cluster without having to create any sorts of topics or partitions.

Putting it all together

MarketData distribution usescases typically have high performance requirements. While Apache Kafka has a variety of knobs that you can tune to achieve low latency and high throughput, it usually comes at a cost of an increased probability of message loss.

The folks over at vectorized.io have created a Kafka-compatible message broker called RedPanda from the ground up in C++ which means you can use your existing Kafka apps with it and realize an instantaneous performance improvement.

This also means that it should work with the existing Kafka Ecosystem including the aforementioned Solace PubSub+ Kafka Connector for WebSocket fanout.

Here are the steps you need to take to spin up the entire stack:

Sign up for a FREE Solace PubSub+ Cloud Broker or download and run a FREE Docker container to do the websocket fanout
Download, install and run the RedPanda broker. It comes in many deployment options but I recommend the docker installation for the easiest install
Download the Apache Kafka distribution to get access to the connect framework
Compile the Solace PubSub+ Kafka Sink Connector after adding the following file as a message processor: SolMarketdataProcessor.java. Follow the instructions in the github repo to add the compiled library to the Kafka Connect you got from the Apache Kafka distribution.
Using a javascript library like MQTT.js you can now connect to the MQTT WebSocket port of your Solace PubSub+ broker using the following code as an example:

var mqtt = require('mqtt')
var client  = mqtt.connect('wss://host:port',{username:'username',password:'password'})

client.on('connect', function () {
  client.subscribe('US/NASDAQ/AAPL'); //Establish your topic subscriptions here
})

client.on('message', function (topic, message) {
  // message is Buffer
  console.log(message.toString())

})

You can retrieve the host:port and credentials from the 'Connect' tab of your Solace Cloud Console:

or use ws://localhost:8000 for a broker deployed locally

Here is an example of a market data stream sent over Solace via WebSocket to a web application -

Extending your deployment using a Solace PubSub+ Event Mesh

The Solace PubSub+ Event Brokers allows you to wire together brokers to create a global multi-region (multi-cloud if need be) backbone for your event streams so you are fanning out in geographic proximity to your users.

In addition, once your Kafka streams are liberated onto the Solace PubSub+ Event Mesh, it provides additional benefits some of which I listed here:

Access Control Lists: Provide fine grained permissions on a per topic basis so users can only subscribe to NASDAQ stocks for example.
Throttling Feeds (Eliding): Allows you to throttle feeds at configured interval - so you will only deliver 1 message per second to your web clients even though your Kafka cluster is getting 1000 messages per second
On-Behalf-of Subscription Management - The ability for a microservice to inject subscriptions into a a WebSocket connection's session. This gives you the ability to add/remove subscriptions based on an entitlements lookup for example.

How to Use Rust, WebAssembly, JavaScript, and PubSub+ to Run Native Apps in Your Browser

Thomas K — Thu, 15 Oct 2020 13:00:06 +0000

As the popularity of JavaScript has improved over the years, desktop applications have increasingly started moving away from thick-clients using languages such as .NET and Java to thin-clients like those built out with JavaScript. Frameworks such as OpenFin and Electron have exploded in growth in recent years because of this trend. However, browser-based JavaScript applications do not have the ability to call native code so browser-based JavaScript applications have been relegated, for the most part, to the presentation layer while more heavy weight processing has been moved to the server side. While this works well for a lot of the use cases as desktop and laptop computers continue to get upgraded with more cores, a tremendous amount of processing capability is being left unused.

In this blog post, I will explain how using technology like Rust, WebAssembly, JavaScript (via the Svelte framework), and Solace PubSub+ enables you to reclaim some of that processing power while you continue to use your modern HTML5 applications through a very basic implementation of an options pricer in the browser.

Whew … those were a lot of buzzwords. Let’s start by breaking each of them down.

Rust

Rust is an exciting new programming language from the fine folks over at the Mozilla foundation. It’s a low-level language that offers extreme performance with no run-time since it targets the LLVM. This makes it ideal for embedded/low powered devices. Other notable features are the lack of a garbage collector, strong compile time checks (such asthe borrow checker), support for functional programming constructs (such as closures), and object oriented constructs (via traits). If you’d like to learn more, I highly recommend the Rust Book.

WebAssembly

WebAssembly is a binary instruction format that is designed as a compilation target for programming languages allowing them to be deployed onto the web. It also allows for inter-op with JavaScript through a wasm.js module that you can load in your client-side applications. This allows you to take advantage of native code in your web applications.

Rust also includes first-class support for compiling down to WebAssembly. With its strong memory safety paradigms, it is an ideal language for this purpose.

Svelte

Svelte is a no-nonsense JavaScript framework. It primarily differs from the more popular frameworks such as Angular, Vue, and ReactJS due to the fact that it compiles down to pure JavaScript without the need for a heavy framework library to be loaded, making it blazingly fast and the resultant apps extremely small.

Solace PubSub+

Solace PubSub+ is an advanced event broker that has many advanced features such as the ability to form an event-mesh, native multi-protocol support, and it’s FREE! Solace PubSub+ will form a core part of the architecture facilitating communication between WebAssembly, JavaScript, and a Rust process.

An Example Using Options and Options Pricing

An option is a derivative financial instrument that gives the buyer the right to purchase/sell an asset at a specific price. It also allows for the seller of the options contract to generate income on assets that they have in their portfolio by taking a view on whether they see an upswing or downswing in the price of an asset.

Let’s take an example. Say you think the price of AAPL is going to go above $125 from its current price of $100 after they announce the next iPhone. In such a situation, you can buy a CALL option for a strike price at $125 that expires at the end of September. What this means is that if by the end of September, the price of the AAPL stock reaches $150, having the option contract gives you the right to buy AAPL stock at $125 and you can instantly pocket the profit of $25 by selling AAPL back at $150 (minus how much it costs to purchase the option contract of course). However, if the price never reaches $125 by the end of September, your investment in the option becomes worthless and you lost your money. Likewise, a PUT option gives you the ability to sell an asset at a price in case you believe the asset will go down by a certain amount.

Pricing an option is not a straightforward task as there are multiple factors associated with the price known as ‘The Greeks’. While the math behind pricing an option goes beyond the scope of this blog post, I recently stumbled upon a blog post by Ronnie Chowdhury entitled Probably the fastest Black Scholes Pricer in the world. I recommend reading through the post but the most interesting technical bit of information for me was a topic called SIMD that allows the CPU to perform multiple operations with one instruction set.

With SIMD, instead we could instead act on multiple numbers together.

From the above we are iterating over each value and adding them one by one. But you would hope the compiler is clever enough to see that actually there’s an instruction _mm_add_ps() that could do 8 additions in 1 instruction. That’s *8x * faster.”

I was able to make some small changes to Ronnie’s code which you can find here to allow it to be compiled to WebAssembly.

Bringing it all together

So, we have a highly performant options pricer implemented in Rust. I also created a Svelte WebApp that compiles the Rust code to WebAssembly and creates a binding to call this code from JavaScript. I also implemented a very bare-bones Rust Options Market Data Simulator that publishes option prices to Solace PubSub+ which you can find here. The Svelte app will consume these feeds from a Solace PubSub+ Event Broker and make a call to WebAssembly to price the option feed (note: the simulator spits out random data so the option prices generated by the pricer will not be realistic).

Here is an image of the application in action:

To run this yourself, follow the instructions in this repo.

Conclusion

While WebAssembly interop with JavaScript is arguably in its infancy and will see tighter integration and improved performance over time, hopefully the information and example in this post opens up your mind to the possibilities of using more of your desktop/laptop cores that are sitting idle on your computer for your modern web applications.

The post How to Use Rust, WebAssembly, JavaScript, and PubSub+ to Run Native Apps in Your Browser appeared first on Solace.

Linking Vuex State Machines Across Your Vue Client Applications with Solace PubSub+

Thomas K — Thu, 30 Apr 2020 14:46:37 +0000

Vue.js is a JavaScript framework developed by Evan You that’s making significant inroads as the framework of choice for modern single-page applications. One reason it has become so popular is that it comes with components any single-page-application is expected to have, such as a router, a developer UI to manage your application, and a state management library called Vuex. To learn more about Vue.js, visit the Getting Started with Vue site.

What is Vuex?

According to the official web site for Vuex, it is a state management pattern and library for Vue.js applications. It serves as a centralized store for all the components in an application, with rules ensuring that the state can only be mutated in a predictable fashion. It also integrates with Vue’s official devtools extension to provide advanced features such as zero-config time-travel debugging and state snapshot export / import.

Key to the architecture of Vuex are the concepts of state, views, and actions:

State serves as the source of truth that drives our applications
The view is a declarative mapping of the state
Actions are the ways the state could change in reaction to user inputs from the view

A simplistic illustration of a Vuex workflow

The Vuex store is instantiated within your application as a singleton, which means there is a single instance of the store that lives within your Vue application, and your various components look to it as the source of truth.

What makes Vuex even more powerful and easier to use is its deep integration into the Vue.Js DevTool Chrome Extension, which allows you to look at the sequence of state mutations and even replay state changes.

Example of the Vuex view in the Vuex DevTool browser extension

Why are state machines important for your JavaScript applications?

State machines are becoming an increasingly important part of any modern application framework. With the pervasiveness of single-page application frameworks, it is also becoming a de facto part of front-end applications.

Depending on the JavaScript framework you are using, there are a variety of choices such as Vuex for Vue and Redux for React. There is also a JavaScript framework-agnostic library called XState.

If you are building an application, it will almost have ‘state’ associated with it. For front-end applications, your application state may include whether the user is logged in or whether the user is an admin. Compared to sprinkling and checking the logic for these transitions throughout your application, handling these states from a single location in your codebase is easier and less prone to errors. This is where a state machine library comes into play – typically exposed as a ‘singleton’ and accessed from within your application, it provides a single source of truth for the state of your applications.

Implementing Vuex for a turn-based game

Now let’s implement a basic Vuex container (state, actions, and mutations) for a turn-based game. Here are the key components we will need to implement:

State
- player – The ID of the player (Player1or Player2 )
- isTurn – Boolean value that represents whether it is the player’s turn
Actions
- logIn – sets the player’s id
- makeMove – Sends the move to the other player and swaps the turn off for the current player
- receiveMove – Receives a move from the other player and activates the current player’s turn

Here is the bare-bones implementation:

export default new Vuex.Store({
  state: {
    player: '',
    isTurn: false
  },
  mutations: {
    LOGIN(state, player) {
      state.player = player
      if (state.player === 'Player1') state.isTurn = true
    },
    TOGGLE_TURN(state) {
      state.isTurn = !state.isTurn
    }
  },
  actions: {
    login({ commit }, player) {
      console.log('Attempting to login with ' + player)
      commit('LOGIN', player)
    },
    makeMove({ commit, getters }, move) {
      if (getters.isTurn()) {
        //TODO: Send a move to the other player
      }
    },
    receiveMove({ commit, getters }, otherPlayerMove) {
      if (!getters.isTurn()) {
        console.log(
          `Received move from other player ${JSON.stringify(otherPlayerMove)}`
        )
        commit('TOGGLE_TURN')
      }
    }
  },
  getters: {
    isTurn: state => () => {
      return state.isTurn
    },
    getPlayer: state => () => {
      return state.player
    }
  }
})

Implementation gaps

Now you’ve implemented a bare-bones skeleton of your Vuex store, but you may have noticed there are a few gaps.

You aren’t sending the ‘move’ object anywhere.
How do you get notified if another player made a move?

Suppose this was a two-player game, the sequence expected would be as follows:

Player1 and Player2 log in.
Player1 has his isTurn flag set to true and Player2 has his isTurn flag set to false.
Player1 sends his move over to Player2.
Player1’s isTurn flag is set to false.
Player2 receives the move and sets his isTurn flag to true.

We need a mechanism to establish communication between two different Vuex instances

Using Solace PubSub+ Event Broker to distribute your Vuex state

Solace PubSub+ Event Broker is a multi-protocol, fully featured and free-to-use event broker. One of the many transports that it supports is WebSocket for both Node.js and a browser, which makes it ideal for solving this particular problem.

The APIs and protocols that Solace PubSub+ Event Broker natively supports

The APIs and protocols that Solace PubSub+ Event Broker natively supports

Getting started with the PubSub+ Event Broker

To get started with PubSub+ Event Broker, you create a new Solace cloud messaging service (for FREE!). If you need help with creating a messaging service, refer to this guide.

Publish events from your Vuex store

Now in your makeMove(…) action, we can change the code as follows:

makeMove({ commit, getters }, move) {
      if (getters.isTurn()) {
        console.log(
          `Sending ${JSON.stringify(move)} to ${getters.getPlayer()}/Move`
        )
        mqttClient.publish(`${getters.getPlayer()}/Move`, move).then(() => {
          commit('SWAP_TURN')
        })
      }
    }

Note you are now sending a message on a well-defined topic ‘Player1/Move’ or ‘Player2/Move’ with the move payload.

Subscribing to move events from the other player

Now how do you get notified when another player, connected from another browser, makes a move?

In your view (in the beforeMount() lifecycle hook, for example) you will simply make use of a subscription function which may look something like this:

//Subscribe to the other player's move message
     this.$mqttClient
       .subscribe(
          'Player2/Move',
         ({ topic, message }) => {
           store.dispatch('receiveMove', message)
           //rest of your business logic
         }
       )
       .then(() => {
         console.log('Succesfully subscribed')
       })
   })

You are adding a subscription to the topic ‘Player2/Move’ if you are ‘Player1’ or ‘Player1/Move’ if you are ‘Player2’. Your callback will instantaneously be triggered once you get a message with the event on the topic.

Your finished product will look something like this across two browser windows:

GitHub Repo

To take a look at a fully built out sample implementation, navigate to this linking-vuex-stores repo .

Conclusion

In this post, I showed you how easy it is to link different Vuex stores together using simple publish/subscribe semantics via Solace PubSub+ Event Broker.

While this works, you may realize that state is being distributed and controlled on the client side. Ideally you would want state to be validated and distributed from the server side. In a future post, we will show you how easy it is to add a node.js server using XState (a popular JavaScript state machine library) to this architecture to control and validate state on the server side while continuing to use Vuex on the client side.

The post Linking Vuex State Machines Across Your Vue Client Applications with Solace PubSub+ appeared first on Solace.

Build a Proximity Detection System with a Raspberry Pi, Solace PubSub+, and JavaScript

Thomas K — Tue, 31 Mar 2020 15:44:20 +0000

Have you ever wanted to tinker around with a Raspberry Pi and IoT but felt that it was too daunting? We’ve developed a set of instructions for you to build a proximity detection system using a Raspberry Pi with an ultra-sonic proximity sensor. The instructions include the materials you need to purchase, how to wire everything together, and the code required to build an application on top of it.

What You Will Build

In the codelab, it runs you through step by step instructions of setting up your Raspberry Pi, wiring it to a proximity sensor, building an application in Javascript that reads the events from the proximity sensor, and publishing it to a Solace PubSub+ Event Broker: Cloud.

You can now build microservices over proximity events triggered when someone comes close to the device!

JavaScript and IoT – Not Your Typical Combination

JavaScript is typically not a language you would associate with IoT, but with the advent of NodeJs, it is being used for more and more use cases outside of traditional web development. One of the frameworks that builds on Node is Johnny-Five – a general purpose IoT framework that works across multiple processor architectures such as SparkFun, Arduino, and Raspberry Pi. The codelab makes use of this framework to gather events from the sensor.

Exposing IoT Events Over Solace PubSub+

Once you’ve detected an object within a certain proximity, what if you wanted to stream this as an event so that it can be picked up by microservices or a dashboard? That’s where Solace PubSub+ Event Broker: Cloud comes into play, a polyglot event broker that allows you to publish IoT events from your Raspberry Pi and subscribe to it with different applications and languages.

Take a look at the codelab here and start building!

The post Build a Proximity Detection System with a Raspberry Pi, Solace PubSub+, and JavaScript appeared first on Solace.

Building Battleship: A Hands-On Way to Learn How to Develop Event-Driven Microservices Using Spring Cloud Stream and TypeScript

Thomas K — Fri, 13 Mar 2020 22:30:58 +0000

The post You Sank My Battleship! A Hands-On Way to Learn How to Develop Event-Driven Microservices Using Spring Cloud Stream and TypeScript appeared first on Solace.

When it comes to building modern applications (and especially microservices), understanding the basics of messaging and event-driven architecture can be critical, difficult, and maybe even boring – especially when you’re presented with a rich choice of languages (Java, TypeScript, etc.), frameworks (Spring Cloud Stream, Aurelia JS, etc.), and patterns to choose from (request/reply, etc.)!

So, what better way to learn new tools and frameworks than by building something fun and familiar – an online version of the game Battleship! Take this free online course and in two hours you’ll build a working game of Battleship using TypeScript in Aurelia JS, Java in Spring Cloud Stream, and Solace PubSub+ Event Broker: Cloud.

You can take the course as a codelab and/or at Udemy.

Is this course for you?

Yes, if you are an enterprise developer building modern, distributed applications. And if you’re interested in learning more about popular and up-and-coming languages/frameworks (TypeScript in Aurelia JS and Java in Spring Cloud Stream). Oh, and if you’re keen to learn more about message exchange patterns and how to leverage them in event-driven design this course is definitely for you.

What should you already know?

There are a couple things you should already have some experience with before signing up for the course:

Previous development experience is required
Basic understanding of transmission control protocol (TCP) and networking is beneficial, but not necessary
Java Development Kit

What you will learn:

How to develop event-driven microservices using Spring Cloud Stream and TypeScript (a superset of JavaScript, a hot new language for web app development)
Best practices to implement event-driven architecture for web applications
Why and how to use request/reply (as well as other message exchange patterns) over an event broker
How to take logic out of the browser and put client-side logic on the server (vs. in the browser)
The value of event-driven design in architecting a distributed system

How to get started: up your game with TypeScript and Java – for free!

The Battleship course is broken into two components: Udemy videos and the codelab. The Udemy videos focus on the theory and fundamentals while the codelab applies the theory and fundamentals to an example application. Each lesson progressively builds in more functionality to the application.

The best way to consume this course is to watch the Udemy videos first and then get your hands dirty by typing out the code snippets found in the codelab.

If videos aren’t your thing, that’s okay — you’ll be able to complete the codelab without watching the Udemy video!

Get started now!

The post You Sank My Battleship! A Hands-On Way to Learn How to Develop Event-Driven Microservices Using Spring Cloud Stream and TypeScript appeared first on Solace.

Creating an OpenFin Event Mesh Across the Hybrid Cloud

Thomas K — Fri, 03 May 2019 01:36:07 +0000

In financial services, there is an old adage “the trend is your friend.” These days, the trend in every financial services firm is from the datacenter to the cloud. A recent survey by Thomson Reuters found that while cloud spend was 30% of financial firms’ IT budgets in 2017, it was expected to reach 47% by this year (2019).

At most financial firms, OpenFin applications need to exchange data with a mix of services that include traditional datacenter resources, newer workloads deployed in the cloud, SaaS services, and a collection of other 3rd-party services. Let’s look at how the back-end landscape is changing:

Datacenter applications. There are a few core applications like order routing and compliance that may never leave the datacenter, but most other applications are ripe for migration to the cloud. This could be via “lift and shift” using virtualization or containers, or a next-gen legacy replacement initiative that leverages powerful new cloud engines like machine learning or streaming analytics. In some cases datacenter applications are becoming hybrid with some of the workloads in the cloud and the rest staying in the datacenter.
SaaS solutions. The FinTech revolution is bringing to market new managed services that can replace traditional in-house applications at lower cost. Examples include fraud detection, data lineage (as part of compliance) and risk management.
3rd-party services. The days of acquiring 3rd-party services over special VPNs or leased lines are coming to an end. When Bloomberg (B-PIPE) and Thomson Reuters (Refinitiv’s Elektron) have a cloud-first strategy, you know it’s officially a trend. OpenFin’s rapidly growing app directory is filled with 3rd-party cloud services.

Achieving Service Location Transparency

As you plan for OpenFin connectivity amidst ongoing services migration, it becomes essential to have a strategy to abstract service location out of app and service interactions. This is equally true for your OpenFin applications interacting with remote services and for the interplay between the services themselves.

This is one of the many benefits of a messaging infrastructure, particularly one that can span both the datacenter and public clouds. A messaging infrastructure lets you declare which systems produce and consume which events, and uses queue or topic names to route the data. In this way, a back-end service can move from the datacenter to the cloud and the broker will dynamically re-route traffic with no changes to any OpenFin apps or other services. The same technique is used for disaster recovery and business continuity; if your primary cloud fails, traffic from your OpenFin apps can be instantly routed to an alternate public or private cloud.

OpenFin Service Integration

The quickest way to get started is to use a multi-protocol broker solely to gateway your OpenFin applications to services across the hybrid cloud. The traffic flow looks like this.

As outlined in the white paper A Guide to Connectivity for OpenFin Architects and Developers, Solace lets OpenFin applications use native web technology and still reach all your cloud, web and legacy services through its unique multi-protocol approach.

Global Event Mesh

Many of our customers also need the same capabilities within individual clouds or datacenters, more like this:

For example, in the diagram above, the left-most applications can use the broker deployed in AWS to manage that cloud’s events and route data to the front office, datacenter, or other clouds. Any application anywhere in the hybrid cloud environment can subscribe to any set of events, and the brokers will dynamically coordinate what is needed where while switching protocols during data delivery as necessary.

Managed Public Cloud Event Mesh

There’s even a managed service offering if you don’t want to manage the cloud infrastructure yourself. You can pay as you go, and Solace will assure uptime and capacity as needed. A single console will let you see all inter-service activity so you always have a complete picture of your data flow.

Which is Right for You?

Whether you’re looking to rationalize service access just for OpenFin or want to transform your entire enterprise, Solace PubSub+ can simplify how your data flows and provide maximum flexibility as you shift workloads to the cloud over time. If you’d like to learn more about Solace PubSub+, PubSub+ Cloud, or a Solace-enabled event mesh, contact us today.

The post Creating an OpenFin Event Mesh Across the Hybrid Cloud appeared first on Solace.

OpenFin Apps Need More Than Just REST

Thomas K — Fri, 03 May 2019 01:16:34 +0000

If you’re an OpenFin developer, you probably prefer to use REST for remote API calls whenever possible. This works great for query-response type requests, but has its limits beyond that, specifically:

Service availability is binary. If the remote service is unavailable or overloaded, your app has to deal with the failure scenarios.
It’s a one-to-one protocol. If your app design is best served with a one-to-many exchange, REST isn’t a great choice.
It only works for web-based services. Many legacy services don’t present a REST API.
Using REST leads to tight coupling of apps to services. The endpoints are embedded into the application.

All of these limitations can be overcome by coupling REST with a multi-protocol message broker that natively supports web-based protocols. Your apps can still use web-friendly API management, and the broker handles service reliability issues, publish/subscribe and queueing semantics, and access to non-web legacy systems.

Let’s take a look at three common scenarios and how OpenFin REST interactions can be improved with a message broker acting as an intermediary.

Request/reply to RESTful service
Request/reply to a legacy service with a messaging interface
REST as a publisher to multiple subscribers

Scenario 1: Request/reply to RESTful service

This is the pattern where REST is most straightforward. You simply POST (or PUT) and specify whether you want the interaction to be synchronous or asynchronous.

I know what you’re thinking:

They both do the same thing, only the first one is simpler! Why add Solace?!?”

There are three primary reasons people put a message broker between their REST POSTs and their RESTful services:

Buffering against failure. A message broker or queue is often used to simplify error handling in the client application. If the remote service is unavailable or heavily loaded, the broker will hold the messages until the service is restored or has caught up. The broker can easily be configured for fault tolerance and disaster recovery to provide instance or site failure recovery. With a direct REST request, if your POST fails it’s on you to manage the retries and handle the many failure scenarios.
Governance. If you have many types of messages going to many types of destinations across the datacenter, cloud and SaaS services, it can be very useful to manage security, logging and other governance operations in a single place.
Publish/subscribe. Sometimes it is useful for the message payload to go to multiple destinations (see scenario 3 later in this post).

REST may be all you need if everything is a service lookup and everything is working, but distributed systems get messy, and a broker improves both reliability and governance.

Scenario 2: Request/reply to a legacy service with a messaging interface

Scenario 2 is still a point-to-point interaction, but this time the remote service does not offer a RESTful interface. We’ve discussed the architectural choices facing this scenario in detail in the whitepaper A Guide to Connectivity for OpenFin Architects and Developers. This time Solace is acting as a protocol and security bridge, terminating the REST request, forwarding the payload to a messaging queue or topic, and doing the same with the responses.

The means of receiving the response will vary depending on the nature of your REST request. If the request is synchronous, Solace will hold the connection open until the associated messaging interaction has completed. If the REST POST is asynchronous, you’ll need a means of receiving the message via callback, or polling for the results. There’s a companion blog post entitled Handling OpenFin Inbound Events from the Hybrid Cloud that covers handling message flow in the other direction.

Scenario 3: REST as a publisher to multiple subscribers

In scenario 3, the client application needs to post information to multiple remote services. Without a broker, the only way to do this is to have the application send the message to each application one by one, or write some kind of intermediary that forwards the message inside a RESTful service.

Solace offers an elegant solution to this by allowing you to use REST to POST the data along with a publishing topic as metadata. Remote services identify themselves as subscribers and PubSub+ publishes onto many different messaging networks on the specified topic. The net effect is that your OpenFin apps use REST and the data automatically goes to all the subscribers, wherever they are. Some subscribers may produce a response, and others may simply listen, for example for logging or archival.

All-of-the-Above

Of course, many applications will want to include exchange patterns that accomplish all three of these scenarios. The all-of-the-above scenario highlights the governance and reliability advantages of using a broker.

If an application needed to use all of the message patterns we have discussed in this blog post, without a message broker, it would need to vary the use of REST, possibly handle security differently for each pattern, and deal with all failure scenarios. With a message broker, it simply POSTs the data to Solace which acts as a security and traffic intermediary, routing to all data sources.

Note: To keep the image above from overcrowding, I did not show the return messages. They would work the same as they did in the prior scenario diagrams above.

Bolstering REST with Enterprise-Grade Resilience and Governance

So there you have it, REST is awesome because it’s so simple—but it has some serious limitations. If operationalizing systems for predictability and high-availability, accessing legacy systems, or taking advantage of messaging-like semantics are requirements for your apps, consider adding a multi-protocol broker to bridge REST into other protocols and bolster your system’s resilience.

Be sure to check out the companion post that discusses receiving notifications and responses from asynchronous REST requests: Handling OpenFin Inbound Events from the Hybrid Cloud.

The post OpenFin Apps Need More Than Just REST appeared first on Solace.

Handling OpenFin Inbound Events from the Hybrid Cloud

Thomas K — Fri, 03 May 2019 00:30:13 +0000

Fully integrating OpenFin applications with remote services requires a plan for how to best interact with those services. Most financial services firms have hundreds of applications and thousands of microservices, some of which consume events and some of which generate them. Some present web-native APIs, and many do not.

We have previously talked about how REST can be enhanced to handle the many outbound exchange patterns. In this post, we’ll be discussing a better way to organize OpenFin inbound events.

What is an Inbound Event?

For the purposes of this post, we’re defining inbound events as any asynchronous event received from a remote source. There are two main types of inbound events:

A notification triggered from a service. In most cases, these will be in response to the user’s interest (explicit or implied) in the associated event. For example, a price alert, a risk threshold breach, a fraud event or news associated with a portfolio asset.
An asynchronous reply to a prior request. JavaScript encourages the use of non-blocking resources, and a remote request may result in a variable amount of time to do some work—from milliseconds to minutes. Most OpenFin apps will want to decouple the request and response and treat the latter as an inbound event. For example, if you submit a basket of trades on behalf of a client, you may want to receive a single event when all fills are complete.

Dealing with Inbound Events

The client side of real-time inbound events is pretty straightforward. You set up a connection and callbacks for the different kinds of events you expect. JavaScript makes it easy to pass control to a function when an event arrives.

The trickier part is choosing what is on the other end of that connection. Your app may be talking to a single remote service or many. It could be inside the datacenter, in a private or public cloud, or a 3rd-party service. The remote service may present a web, messaging or legacy interface. The remote services may also offer different approaches to security.

There are many things to consider to inform the best course of action. Here are three:

The DIY option is to code your own WebSocket server and handle everything yourself. For example, if the remote service does not present a WebSocket event API. Your code will need to integrate with sources, negotiate handshakes, reformat payloads, and send heartbeat pings at intervals to assure the connection stays alive, and handle a range of errors.
Some cloud and web-based APIs will include a WebSocket service interface within their APIs, doing that work for you. You code to an API, and the event handling happens under the covers.
A multi-protocol event broker that supports WebSocket can act as an intermediary. For example, a Solace PubSub+ broker can coordinate with many services and consolidate inbound events back to the OpenFin client through one or more WebSocket links.

If you are interacting with just one remote service, and it implements a WebSocket server, coding direction to its API may be a solid choice. The service will either document how to handle the matching WebSocket client, or it may provide a client-side JavaScript API to handle the connection and event management.

In all other cases your easiest path forward will be to use an event broker to coordinate with the remote services. Let’s take a look at the many ways a Solace PubSub+ broker simplifies your event handling.

Provide the OpenFin App with a Unified Subscriber Model

Whether your back-end is a cloud service with a web-based API, an internal service connected to an AMQP network, or the stodgy old mainframe holding the legacy crown jewels, your OpenFin app simply connects to the broker. The broker handles all the complexity of the various network topologies and differences in connection protocols. The broker will then stream or queue events for consumption by your application. Security is handled by broker-defined access control lists (ACLs) to verify which remote resources each user or application can use and how.

Let’s take a look at how the broker would pass events from a few different service types.

Events from Web-Based Services

It’s always easy to post to a cloud or web-based service using REST, but not as clear how best to receive an asynchronous event back. Often cloud architectures will rely on using some kind of intermediary storage like AWS S3, or a database, then ask the calling app to poll to read the stored data. Sometimes the remote service will include a WebSocket server, and will supply a JavaScript API that emits events to feed your callbacks.

Many of our customers write their remote services to POST events to Solace using REST, and Solace can forward that event onto the client through a WebSocket. In OpenFin, the developer uses the Solace JavaScript API, and Solace uses WebSocket to signal anything that arrives for that app.

Events from a Non-Web Service

If the service does not present a web-friendly API, Solace becomes a peer on any popular messaging network and forwards events back to OpenFin. For example, Solace would subscribe JMS or AMQP topics, or identify queue names on behalf of the OpenFin app, to receive and forward inbound events over a WebSocket. Solace can even reach proprietary protocols like MQ or Kafka through a bridge so those events can reach OpenFin applications as well.

Subscribe to Events from Many Services

In the companion blog post about outbound requests, we showed how Solace allows a REST POST to be mapped into a publish/subscribe network for delivery to many services. The reverse pattern is also true for inbound. An app that receives events from many services can be simplified by having a broker stream or queue the events as an intermediary.

Events from “All of the Above”

A broker is most useful when you have data going to and from many services. In large OpenFin installations, it is very common for many sources and many apps to need to freely exchange information.

Coordinating Inbound Events with a Multi-Protocol Broker

While you may have tight control over your OpenFin environment, the data sources those apps interact with are spread across clouds and datacenters, and have been created with different technologies over a span of decades. Just like a broker can help you do much more with REST, that same architecture can simplify inbound event management as well.

For more on this topic, download our Guide to Connectivity for OpenFin that outlines the three architectural approaches developers and architects may choose to connect OpenFin applications to a variety of data sources:

The post Handling OpenFin Inbound Events from the Hybrid Cloud appeared first on Solace.