Kafka is a powerful stream processing tool, but it's an asynchronous tool. So we know when we send the request but we don't know when the answer will come. And in some cases, there are some synchronous applications which fronts Kafka.
So today we will see the first of 3 cases to make this communication between the synchronous application more efficient depending your case.
1st solution - Kafka Based Solution
This solution is pretty simple. When you are send a message to kafka, you can add 2 headers to inform where the answer should be sent.
- REPLY_TOPIC : Name of the topic where the answer must be sent
- REPLY_PARTITION : Index of the partition in the REPLY_TOPIC where the answer must be sent
Example
In this example we have 2 topics :
- TOPIC REQUEST
- TOPIC RESPONSE
The 2 instances of the API X write in the TOPIC REQUEST and are waiting an answer in the TOPIC RESPONSE. This topic has 4 partitions which are divided equally between the two instances.
Behind these topics, a MicroService Y is reading request from TOPIC REQUEST and is answering on the TOPIC RESPONSE (defined in the REPLY_TOPIC header of the request) on the partition asked by the header REPLY_PARTITION.
Like this when the first instance send a request, it will receive its answer.
Pros & Cons
| Pros | Cons |
|---|---|
| Messages are consumed only once | Messages can be lost on a rebalancing |
| The MicroService Y must be setup to read the headers REPLY_TOPIC and REPLY_PARTITION and use this values to answer | |
| The service account of the microservice Y must be able to write on all the topic it can received on REPLY_TOPIC |
The principal point here is the rebalancing. When you scale up/down or restart a service, the partitions will be rebalanced between all the instances of the service.
If the first instance of the API was listening on the partition 1 and 3 before a restart, after a rebalacing it can listen on 2 other partitions. So, if a request was sent from the first instance of the API with REPLY_PARTITION = 1 and the rebalancing appends, then the message can be lost (if the partition 1 is given to another instance than the first one).
/!\ This solution is working only if you can lose messages.
In the following part, we will see a more data-secure solution!
I hope it will help you! 🍺
You want to support me?
Top comments (0)