Leveraging RabbitMQ and Kafka for Efficient Saga and Pub-Sub Integration

Aung Myat Moe — Mon, 16 Oct 2023 17:30:00 +0000

Sagas and pub-sub are two important patterns in distributed systems. Sagas allow for complex transactions to be coordinated across multiple services, while pub-sub provides a way to decouple services and communicate asynchronously. RabbitMQ and Kafka are two popular messaging systems that can be used to implement sagas and pub-sub. However, each system has its own strengths and weaknesses.

RabbitMQ is a good choice for sagas because it provides support for transactional messages. This means that if one step in a saga fails, all of the previous steps can be rolled back.

Kafka is a good choice for pub-sub because it is highly scalable and can handle very high volumes of data. To implement a saga using RabbitMQ and Kafka, we can use the following approach:

Use RabbitMQ to coordinate the execution of the saga steps.
Use Kafka to communicate between the different services involved in the saga.
Use the transactional features of RabbitMQ to ensure that the saga is either completed successfully or rolled back in its entirety.

This approach allows us to combine the strengths of both messaging systems to create a powerful and flexible solution for sagas and pub-sub. A simple example of a saga is an order processing system. The saga might involve the following steps:

Reserve inventory for the items in the order.
Charge the customer's credit card.
Ship the items to the customer. If any of these steps fail, the saga should roll back all of the previous steps.

To implement this saga using RabbitMQ and Kafka, we could do the following:

Create a RabbitMQ exchange for the saga.
Create a Kafka topic for each step in the saga.
Create a RabbitMQ queue for each service involved in the saga.
Bind the queues to the exchange using routing keys that correspond to the saga steps.

When a customer places an order, the order processing service would publish a message to the RabbitMQ exchange. The exchange would route the message to the queue for the first step in the saga.The service that owns the first step in the saga would then consume the message from its queue. If the step is successful, the service would publish a message to the Kafka topic for the next step in the saga.

The service that owns the next step in the saga would be subscribed to the Kafka topic. When it receives a message, it would consume the message and execute the step.This process would continue until all of the steps in the saga have been completed successfully. If any of the steps fail, the service that owns the step would publish a message to a special RabbitMQ queue for saga compensation.

The saga compensation queue would be consumed by a separate service that would roll back all of the previous steps in the saga. This approach allows us to implement a robust and scalable saga system using RabbitMQ and Kafka. By combining the strengths of RabbitMQ and Kafka, we can create a powerful and flexible solution for sagas and pub-sub. This approach can be used to implement complex distributed systems in a reliable and scalable way.

Sync Microservices System Design in Nustshell

Aung Myat Moe — Mon, 25 Sep 2023 15:27:08 +0000

When I first learned about the system we had to implement, it appeared deceptively simple and concise. However, the reality was far from straightforward. The organization's diverse branches had their databases, and this forced us to reconsider how we would approach the design and development of the system.

The challenge of integrating a legacy system with a modern one became evident, especially as we aimed to incorporate multi-tenancy. While multi-tenancy is suitable for databases sharing the same schema and location, our situation required a different approach. Consider a scenario where we execute a write command on database B - in the event of a bug, we need robust logging and monitoring. To address this concern, we decided to segment our services.

These services are internally composed and exposed via an API gateway, offering a clear separation of internal implementation details from other developers. This approach allows us to foster collaboration with junior team members while still maintaining control over the system's architecture.

To ensure the success of our system, we've applied various design patterns and techniques:

Circuit Breaker : Given the network's inherent unreliability, we've implemented circuit breakers to gracefully handle failures and disruptions.
API Gateway : We've established an API gateway to streamline and manage external communication with our services.
Database per Service : Each service has its dedicated database, ensuring data isolation and scalability.
Schema per Service : We've adopted a schema-per-service approach, allowing each service to manage its data structure independently.
API Composition : Our system leverages API composition to assemble complex functionalities from multiple services.
Sync Communication : Synchronous communication is utilized when needed, ensuring data consistency and reliability.
Service per Team : We've organized teams around specific services to enhance ownership and accountability.
Remote Procedure Invocation (RPI): RPI enables seamless communication between services, enhancing system cohesion.
Self-Registration : Services self-register to simplify dynamic service discovery and orchestration.
Routing : We've implemented routing mechanisms to efficiently direct requests to the appropriate services.
Service Discovery : Dynamic service discovery ensures that services can locate and communicate with each other.
Observability : To monitor system health and performance, we've integrated observability tools and practices.
Handling Failures and Latency : Our system has robust strategies in place to deal with failures and mitigate latency issues.

These design patterns and techniques collectively empower our system to navigate the complex landscape of multi-tenancy, network unpredictability, and evolving requirements while maintaining flexibility, reliability, and manageability."

DEV Community: Myanmar Cyber Youths

Leveraging RabbitMQ and Kafka for Efficient Saga and Pub-Sub Integration

Sync Microservices System Design in Nustshell