DEV Community: Meshvi Patel

How Solace PubSub+ Event Broker Supports Transactions

Meshvi Patel — Thu, 05 Nov 2020 16:43:51 +0000

When drawing up a microservices architecture, you need to consider some design trade-offs. These trade-offs are best expressed by the CAP theorem, which states that it’s impossible for a distributed data store to simultaneously guarantee consistency, availability, and partition tolerance. One of the key decisions you will often have to make is whether you need the strong consistency of ACID transactions (which feature atomicity, consistency, isolation, and durability) or if eventual consistency will suffice.

ACID transactions take away the hassle of ensuring data consistency across all participants of the distributed transaction, but they have a significant impact on scalability, performance, and availability of systems at scale, and they add considerable complexity to run-time operations. Eventual consistency, on the other hand, allows data updates to be a bit more…relaxed. It guarantees that in the absence of new updates, all copies of data converge to a common state eventually. It trades off strong consistency for better performance, availability, and partition tolerance.

Neither approach is perfect for every scenario, and the debate of using strong consistency vs eventual consistency is beyond the scope of this blog post. However, if you are designing a system and have decided you need to use transactions, I’d like to introduce you to transactions in the context of Solace PubSub+ Event Broker, specifically the types of transactions it supports, how it supports them, and how they differ.

Overview of Transactions

Transactions enable enterprise applications to group a set of guaranteed messages to be published and/or consumed as an atomic unit of work. For instance, an application might need to consume a message from a queue endpoint, update some database tables, and publish the message on another queue endpoint, in a single operation. If any of these operations fails or a system failure occurs, the entire operation should roll back to its original state. That is, the published messages are deleted, and any messages to be consumed remain on the endpoints they were spooled to.

Combining operations in a transactional unit ensures the integrity of messaging operations that depend on one another and also ensures the validity and consistency of data.

Reservation systems (airline, train tickets, theatre tickets), travel booking, sales order management are all good examples of transactional processing. Consider an electronic fund transfer in an online banking system where one account is debited and another credited with an equal amount. If the debit and credit operations cannot be executed as a transactional unit, neither should be executed as it leads to an inconsistent state and the bank’s books will not balance at the end of the day. The same applies to a travel booking process for a travel agency that books flights and the hotel upon receiving a customer’s travel request. If the customer cancels the request, both the flight and hotel booking operations needs to be cancelled, or else the systems will be in an inconsistent state with invalid data.

Transactions in Solace PubSub+

Solace PubSub+ Event Broker can support:

Transacted Sessions (a.k.a. local transactions)
XA Transaction Branches (a.k.a. distributed transactions)

Let’s look at what transacted sessions are, and how Solace PubSub+ can support transactions in transacted sessions.

Transacted Sessions

PubSub+ supports transactions in local transacted sessions. A session can be configured as transacted, and the Solace Java, Java RTO, C, .NET, and JMS messaging APIs provide methods for initiating, committing, or rolling back local transactions.

Transacted sessions enable client applications to group multiple message send and/or receive operations together in single atomic transactions. They only involve a single resource—the event broker and hence also known as local transactions.

The scope of a local transaction is always a single session. That is, one or more producer or consumer operations performed in the context of a single session can be grouped into a single local transaction.

Per Client transaction flow in a local transaction

To publish and/or consume messages in a transaction, a client application must create a transacted session within a session. The transaction is automatically initiated once the transacted session is created. Then in that transacted session the client application can establish a producer flow to publish messages in a transaction, and/or establish a consumer flow to receive messages in a transaction.

In Release 9.7 and earlier, a client can publish and/or consume up to a combined maximum of 256 guaranteed messages in a single transaction. As of Release 9.8, a controlled availability feature will allow this limit to be configurable via a client-profile setting for applications that require larger transactions. Using larger transactions can have broker-wide performance implications, so please review your use case with Solace before changing the configuration to allow for larger transactions.

Messages that are published and received in a transaction are staged on the message broker.

To complete all of the message send and receive operations in a transaction, the transaction must be committed. When a transaction is committed, for publish operations, the staged messages are sent to their destinations and for consume operations, the received messages are acknowledged by the consumers and then removed from the endpoints they were spooled to.

If a particular send or receive operation within the transaction fails, an exception is raised. The client application can handle the exception by ignoring it, retrying the operation or rolling back the entire transaction so that all of its publish and receive operations are cancelled. When a rollback is performed, for publish operations, the published messages are deleted and for consume operations, the messages remain on the endpoints on which they were spooled. Once a transaction is completed, another transaction automatically begins.

The main steps for performing local transactions are:

Create transacted sessions and establishing the producer and/or consumer flow to the event broker.
Publish messages in transactions or consuming messages in transactions.
Complete (commit) or cancel (roll back) transactions.

My colleague Sanjeev Nagpal explained how you can initiate transacted sessions and the interactions between client applications and the Solace router and here is a sample you can check out.

XA Transaction Branches (a.k.a Distributed Transactions)

When you break a monolithic application into microservices, each service may have its own database or queueing system that it talks to and the transaction spans across multiple databases or messaging services. This now becomes a distributed transaction, i.e., transactions in transaction branches.

Solace PubSub+ Event Broker also supports transactions in transaction branches within an XA session that can be used in global distributed transactions. XA transaction branches are similar to transactions contained within a local transacted session in that they encapsulate multiple message send and/or receive operations as single atomic units, but transaction branches are to be used in a larger distributed transaction. Hence referred to as distributed transactions. XA transaction branches are only supported by the Solace implementation of the JMS API.

Each of these transaction branches can be included in distributed XA transactions that may span multiple resources, such as database systems and messaging services. An external transaction manager is required to manage the various transaction branches in a distributed transaction in a coordinated way. The transaction manager tracks and manages operations performed by multiple resource managers using a two-phase commit (2PC) protocol defined in the Java Transaction API (JTA), XA Resource API Specification. The interaction between resource managers and the transaction manager are described in the JTA specification.

The Solace JMS implementation Version 7.1 and greater supports the Java Transaction API (JTA) and the XAResource interface. The XAResource interface provides a Java mapping of the XA interface that follows the X/Open CAE Specification. The following figure depicts the conceptual view of a distributed transaction:

Conceptual view of a distributed transaction

By supporting the XAResource interface, a client can use the Solace JMS implementation to create an XA Session in which it can create transaction branches to publish and/or receive a series of Guaranteed messages and participate in distributed transactions. The XAResource relies on Xids that uniquely identify each XA transaction and the global transaction they are associated with.

The transaction manager uses an XAResource object obtained for each XA Session that will participate in a distributed transaction. Using these XAResource objects, the Transaction Manager can in a coordinated fashion, instruct and manage each XAResource to perform specific actions for their respective transaction branches according to a two-phase commit process used for distributed transactions.

In the first phase, the Transaction Manager directs each resource in the distributed transaction to prepare their transactions for a future commit.
In the second phase, the Transaction Manager then directs each resource to do one of the following actions:
- Commit their respective transaction branches. In this case, the work performed by all of the transactions is committed and the entire transaction is atomically committed.
- Roll back their transactions if one or more resources reported that they were unable to prepare their transaction branches for a commit. In this case, none of the work performed in the transactions is committed.

2Phase Commit

Solace does not offer a Transaction Manager; however, to allow you to integrate Solace PubSub+ with Java Platform Enterprise Edition (Java EE) application server environments, Solace provides a Java Connector Architecture (JCA) v1.5 resource adapter for Red Hat JBoss EAP 6.2, WebSphere V7 and V8, as well as WebLogic 11gR1 PS5 and newer. The Solace JCA resource adapter is provided as a standalone resource adapter archive (RAR) file that includes embedded versions of the Solace JMS API libraries.

XA transactions are usually used within Java EE application servers that contain a transaction manager to manage the lifecycle of XA transactions. However, if you want to understand some of the underlying interfaces and constructs used by XA transactions, I have listed below the basic steps for performing transactions in XA branches. These are the steps for a client to perform an XA transaction outside an application server using the Solace JMS implementation.

Obtain an XAConnection Factory and create an XA connection to the event broker.
Create one or more XA sessions. Within an XA Session, individual transaction branches can be created. Although an XA session can contain multiple transaction branches as identified by their unique Xids, only one branch can be active at a time in an XA session.
Establish the publisher and/or consumer flow in the XA session to publish and/or consume messages.
Obtain an XAResource and Xid for the transaction. Once an XA session is created, a transaction does not automatically begin within that XA session. A client must obtain the XAResource to be used by the XA session and the Xid for the transaction to start a transaction within an XA session.
Start the XA transaction branch by invoking the XAResource.start().
Publish messages in the XA transaction branch and/or consume messages in the XA transaction branch.
End the XA transaction branch. An XA End request is used to end the work being performed for a transaction branch. When an XA End request is sent to the Resource Manager (Solace PubSub+ Event Broker), it associates the published and consumed messages with the transaction branch identified by the given Xid, disassociates the XA resource from that transaction branch, and then lets the transaction complete.
Prepare the XA transaction branch for a commit or rollback. A transaction branch that has been ended must be prepared before it can be committed or rolled back through a two-phase commit.
Complete (commit) or cancel (roll back) transactions. Messages that are published or received through an XA transaction are staged on the event broker. The transaction can either be completed through a commit operation or cancelled through a rollback operation. When a transaction branch is successfully committed, the following occurs:
- For publish operations, the staged messages are sent to their destination queue or topic endpoint.
- For receive operations, the received messages are acknowledged by the consumers and then removed from the endpoints they were spooled to.
- The transaction’s state is changed to Free.

A new transaction cannot be started until the current transaction is freed through a successful commit or is rolled back.

Close XA sessions. To cleanly terminate an XA Session, the client should commit or rollback all transactions branches within it, prior to closing the XA Session. The Solace JMS API contains a XATransactions.java example in the samples directory that shows how you can make use of XA transactions in your application. This sample code is manually managing the XA transaction by calling the relevant XAResource methods.

Conclusion

I hope this post has helped you understand the types of transactions supported by Solace PubSub+ Event Broker and how you can use each of them. You may refer our API developer guide for more information on Using Local Transactions and Using XA Transactions. If you have any questions, post them to the Solace Developer Community.

The post How Solace PubSub+ Event Broker Supports Transactions appeared first on Solace.

Why is Middleware Modernization So Important?

Meshvi Patel — Mon, 28 Sep 2020 19:51:26 +0000

Businesses today recognize that there is an unprecedented opportunity in front of them if they can harness technologies like cloud, data analytics, IoT, AI, robotics, blockchain, AR in order to respond to market changes in real-time.

These disruptive innovations are not just accelerating digital transformation and revolutionizing customer expectations, but causing companies to pivot into entirely new business models built around hyper-personalized products and truly interactive services.

With disruptions affecting every sector, digital transformation is really not a choice, but a necessity — to survive and thrive they simply must transform to keep up with evolving technologies, adapt to continuous change, and seize short-lived opportunities that emerge.

However, to realize the benefits of these technological innovations, they must overcome the inevitable: resistance to change.

As business leaders strive to reshape their businesses, some of the key questions they seek answers to include:

How can I shorten the time from the inception of a new business idea to launching the initiative?
How can I deliver improved customer experience and gain customer loyalty?
How can I improve the operational intelligence of the business to make quick and informed decisions?

For IT leaders, all of these questions lead to answers that make them think one thing: We need to to modernize our infrastructure!

** ** Although cost reduction alone can spur modernization initiatives, there are plenty of other reasons it is important.

What Does Modernization Actually Mean?

** ** Modernization means something unique to each organization. Any one or more of these drivers could be the reason for your modernization initiatives.

IT Consolidation : shed costly and outdated legacy systems and replace them with IaaS, PaaS or SaaS
Cloud Adoption : move existing on-premises applications to the cloud with low risk or adopt cloud-native applications
Multi Cloud/Hybrid Cloud : connect legacy applications to new Saas services.
IoT Connectivity : connect to mobile and IoT devices to get sensor readings
Events as First-Class Citizens : Move to microservices or event driven architectures to improve agility and scaling.
Data Analytics and Insights : streaming data to data lakes to yield for accurate data insights to drive decision making and operational intelligence.

Although the drivers for modernization may be different, the end goal remains the same for all: to build a more agile, customer centric, and data-driven business.

Where Modern Middleware Technology Fits In

Middleware glues enterprise apps and services together and is an integral part of modernization initiatives. Cloud-first strategies are demanding a lot from middleware these days and legacy middleware platforms were just not built to address the needs of the modern, dispersed hybrid IT landscape.

As such, a lot of enterprises are turning to iPaaS as a catch-all solution. However, an iPaaS alone will not be ideal for all situations, especially those where interactions between applications and services are RESTful request/reply communication. The tight coupling and synchronous nature of RESTful communications is not scalable for large enterprise deployments.

Some iPaaS solution providers include basic messaging functionality like queueing and guaranteed messaging, but do not support all enterprise-grade features and message exchange patterns. To fill the gaps, you again end up building tactical solutions leveraging cloud native technologies and over time the architecture again becomes complex, heterogenous, and difficult to maintain.

My colleague Jesse Menning recently wrote a series of whitepapers addressing the real-time challenges with iPaaS how to overcome them with event-driven integration as well as how to implement it in 4 detailed steps. In addition, the whitepaper The Time to Modernize your Middleware is Now is a paper that discusses the value of modernizing your middleware and provides guidance towards garnering support from decision-makers in your organization. I highly recommend giving all of them a read.

So, what is the ideal foundational layer to support your middleware modernization? The answer is an event mesh.

A Single Unified Platform for the Distributed Enterprise

An event mesh (an architectural layer of interconnected event brokers) uniquely enables event-driven microservices, supports IoT and mobile use cases, and integrates easily with modern iPaaS, PaaS, and API gateway technologies. It enables you to stream events across your organization and IoT assets to applications and services that need them regardless of the protocols and APIs they use and where they are located (on-premises, private cloud, and/or public cloud).

An event mesh has been proven to allow:

Faster time to market and continuous innovation : since an event mesh enables event-driven communication, it also supports event-driven microservices, which allow developers to create and implement new functionality faster without impacting existing services.
Seamless customer experience: access to real-time data means improved real-time responsiveness to the customers, resulting in an improved customer service.
Improved operational intelligence : rather than waiting for traditional batch or file-based processing, an event mesh enables real-time access to events.

Be it events from your IoT sensor devices, your factories, product line management system, or quality management system, having access to vital events in real-time enables decision-makers in your enterprise to be informed as quickly as possible when making decisions that will impact the whole business.

To learn more about the Solace enabled event-mesh with PubSub+ event brokers, check out this video:

What to Look for in an Event Broker and Supporting Systems

It is important to understand that the technology choices you make today can make or break your modernization efforts for years to come. Since it’s so critical to get your middleware modernization off on the right foot, make sure your event brokers and other tools support these advanced capabilities, even if you don’t need them today:

Simple to deploy, and easy to integrate with your legacy ESB.
Supports phased migration to the cloud by connecting old legacy apps with new modern SaaS applications.
Capable of dynamic message routing or unified connectivity and broker federation to enable an event mesh across clouds for your multi-cloud or hybrid cloud use cases.
Sophisticated message routing via hierarchical topics and wildcard subscriptions for WAN bandwidth optimization.
Open API and protocol support for your IoT use cases
Support for the publish-subscribe messaging pattern for event-driven microservices and event-driven architecture to improve agility and scaling.
Ability to stream events to data lakes for faster and accurate data insights.
Replay, shock absorption, robust security, HA, and disaster recovery.
Single pane of glass management for deploying, managing, and monitoring your event brokers.
Design, creation, and run-time visualization, discovery, and governance of events.

Conclusion

To set yourself up for success, start the process of modernization by keeping an eye out for what’s next. Take a holistic approach to your modernization efforts and not just modernizing in parts. Choosing technologies that are not just modern but future-proof will help you stay ahead of the game!

The post Why is Middleware Modernization So Important? appeared first on Solace.

Why RESTful APIs Can’t Compete with the Event-Driven Approach to Microservices

Meshvi Patel — Wed, 10 Jun 2020 13:30:15 +0000

Can RESTful APIs with microservices deliver when it comes to the critical real-time customer experience? Businesses in all industries and sectors have all sorts of real-time events coursing through them, including human actions, command and control instructions, sensor readings, news feeds, stock ticks, interest rate changes, process alerts, workflow notifications, fault detection, fraud detection – the list goes on!

The number of such events is increasing at an exponential rate. The challenge lies in sensing and capturing and responding to these events in real-time so you can deliver a higher quality of service, offer a better customer experience, and make more informed business decisions. In this post I aim to explain the limitations of RESTful APIs through an ecommerce microservices example. I also explain how an event mesh can help overcome the challenges. Read on to find out more.

The Rise of Microservices + RESTful APIs

To meet the demands of this real-time responsiveness and extreme scale, businesses recognized the need to innovate, modernize and digitally transform. As architectures and technologies evolved, monolithic applications got broken into coarse-grained services (service-oriented architecture) and now into fine-grained services (microservices).

Business capabilities can now be encapsulated as small, autonomous services and exposed via APIs that can be easily consumed by customers, partners, and 3^rd parties. Docker and Kubernetes simplified the containerized deployment and orchestration of services, making it easier to scale and manage them.

The loosely coupled, cloud-native, flexible nature of using different technology stacks and containerized deployments across private cloud, multi-cloud, PaaS, serverless, IoT and edge computing devices have made microservices very popular.

RESTful APIs became a common choice for enabling interactions between microservices because of simplicity – services communicate directly and synchronously with each other over HTTP. Developers who have always programmed by making function/method/RPC calls and database read/writes found it easier to design resource-based APIs using the CRUD actions. And off course rich tooling like OpenAPI (Swagger) made it all quick and easy to use.

Tight Coupling and Orchestration Limit Your Real-Time Interactions with Microservices

** ** As an example, let’s take a look at the RESTful API approach to a microservices use case for processing a customer order on an ecommerce platform.

A customer initiates an order via the Storefront Web app, which acts as the orchestrator by invoking the inventory, billing, and shipping services before acknowledging the order to the customer.

This is the sequence of events:

The Storefront app service calls inventory to reserve the product
The app service waits for the response from inventory
When the inventory system confirms availability, the app service then calls the billing service to validate and charge the customer credit card
The app service waits for the response from billing
When billing confirms that the transaction has succeeded, the app service then calls the shipping service to ship the order

As request/reply interactions, the communications between these microservices using REST APIs are inherently synchronous, so they need to happen one at a time, in a pre-arranged sequence, and each interaction blocks progress of the process until its completion. The interactions between different services, and updates to multiple services, are coordinated by an orchestrator, introducing a single point of failure.

The RESTful API approach for the Storefront Web App microservices architecture described above results in these limitations:

Poor Customer Experience – Response time to the user is a cumulative sum of response times of the inventory, billing, and shipping services. Any issue with either of these services means the user does not get a response.
Cascading Failure Scenarios – If the ”Back In-Stock Notifier” service fails, users will not be notified and the inventory service will never be updated. This would result in a lost order.
Lack of flexibility – To add new business capabilities/services (like fraud detection), you need to change the Storefront service or the billing service.
Poor Scalability/Resource Utilization – The shipping service becomes a bottleneck. To scale the shipping service, you also need to scale the Storefront and the inventory services since they are blocking it – a waste of resources because they are still in a blocked/wait state.

A recent Gartner Report states:

_Applications become harder to scale and the web of connected APIs harder to manage, leading to ever-more tightly coupled systems. To move beyond the connected APIs and to enable more scalable, contextual and responsive digital business, application leaders add event-driven architecture (EDA) to the core of their platform and design capabilities.¹ _

Let’s explore how event-driven architecture addresses these limitations and delivers on the real-time experience.

The Benefits of an Event-Driven Approach over RESTful APIs for Microservices

We can look at processing the same customer order from our previous example but, this time, with an event-driven approach.

The Storefront App, inventory, billing, and shipping services all connect to something called an event mesh. An event mesh is a network of interconnected event brokers that forms a configurable and dynamic infrastructure layer for decoupled applications, systems, and devices. In this case, the event brokers are Solace PubSub+ Event Brokers.

All communications between these services are loosely coupled, asynchronous events. The one-to-many publish/subscribe messaging pattern allowed for data to be published once and received by multiple, interested subscribers simultaneously. Services do not rely on a central controller, i.e. an orchestrator, to send and receive data.

Using this event-driven approach, these are some of the benefits the Storefront App can capitalize on:

Real-time Customer Experience/Better Response Times : Inventory, billing and shipping services are invoked asynchronously. Services can be executed in parallel, making the response time much faster. Real-time transmission of events (push vs. pull) means services interact with each other in a very robust and efficient manner.
Increased Flexibility and Agility: New services (like fraud detection) can be added without impacting other services. Microservices architecture and independent CI/CD pipelines for each means faster time-to-market for new business capabilities.
Improved Operational Efficiencies and Resiliency: If the ”Back In-Stock Notifier” service fails, the message is stored by the event broker and can be delivered when the service comes back online. Reliable delivery ensures eventual consistency and no data loss. A stateful intermediary also allows for additional fault tolerance, speed mismatches, and shock absorption during peak volumes.
Increased Versatility and Future Proof : If the shipping service becomes a bottleneck, it can be scaled by simply spinning up another instance quickly, making it a highly scalable architecture.

And most importantly, it doesn’t just stop here.

How an Event Mesh Supports Event-Driven Architecture

Although financial services have been the early adopters of event-driven architecture for market data and pricing distribution, event-driven thinking can help every business to deliver a quality service and real-time customer experience to gain a competitive advantage in their market. You can easily scale and extend an event mesh to:

Hook into your enterprise’s ERP, CRM, MDM, supply chain, analytics, AI, and ML applications
Provide edge connectivity to your IoT devices and sensors running at the retail stores, warehouses, distribution centre, and suppliers
Connect your services across environments, from on-prem to different private/public clouds

Solace also offers an event portal called PubSub+ Event Portal that makes it easier to design, develop and manage event-centric applications.

Think about your first morning coffee experience at a coffee shop. If the coffee shop followed the one-to-one model of RESTful API interactions with the microservices – one employee serving one customer – the same employee takes the order, serves the coffee, and takes the payment. In this instance, you would be waiting for your favourite coffee until all of the customers ahead of you are served by that one employee. Waiting time is directly proportional to the length of the queue.

If the coffee shop adopted the one-to-many model instead, each employee serves multiple customers. In this model, one employee will take the orders and the payments, and delegate tasks (making the coffee, serving the coffee) other employees— truly an event-driven advantage! Each customer is served quickly and efficiently instead of waiting for one person to do all the work.

Over the years, Solace has helped and continues to help businesses in financial services, aviation, retail, transport, manufacturing, pharmaceuticals, field services, energy, telco, and more in their event-driven journey. If you would like to learn more, it only takes an event (phone call, sms, email) to reach us!

1 Source: Gartner “The 5 Steps Toward Pervasive Event-Driven Architecture” 28 June 2019, Yefim Natis, Massimo Pezzini, Keith Guttridge, Roy Schulte.

The post Why RESTful APIs Can’t Compete with the Event-Driven Approach to Microservices appeared first on Solace.

DEV Community: Meshvi Patel

How Solace PubSub+ Event Broker Supports Transactions

Overview of Transactions

Transactions in Solace PubSub+

Transacted Sessions

XA Transaction Branches (a.k.a Distributed Transactions)

Conclusion

Why is Middleware Modernization So Important?

** **** What Does Modernization Actually Mean?**

Where Modern Middleware Technology Fits In

A Single Unified Platform for the Distributed Enterprise

What to Look for in an Event Broker and Supporting Systems

Conclusion

Why RESTful APIs Can’t Compete with the Event-Driven Approach to Microservices

The Rise of Microservices + RESTful APIs

Tight Coupling and Orchestration Limit Your Real-Time Interactions with Microservices

The Benefits of an Event-Driven Approach over RESTful APIs for Microservices

How an Event Mesh Supports Event-Driven Architecture

What Does Modernization Actually Mean?