🚀 Breaking the Blockade: How We Taught Kafka to "Speak" Like a Synchronous API

Imagine the situation: Our system, let's call it "BSDFlow", is a modern, impressive Event-Driven monster. Everything happens asynchronously, reliably, and scalably through Kafka. Every entity creation and data update flows through the data pipelines like water.

Sounds like an architectural dream, right? Well, there's a catch. And that catch starts when the user at the other end clicks a button.

The Dilemma: When the User Doesn't Want to Wait 🐢

We live in a world of instant gratification. When we click "Save" in React, we expect to see a success message immediately (or an error, if we failed validation).

In a classic Event-Driven architecture, it works something like this:

1. The client sends a command.
2. The server throws the command into Kafka (like a message in a bottle).
3. The server returns an immediate response to the client: "Got it, working on it!".

But the client? They aren't satisfied 😠. This answer tells them nothing. They don't know if the .NET Backend actually processed the data, or if it hit an error along the way. The user needs a final and definitive answer.

This gap, between the speed of the Event and the need for an API response, is the blockade we had to break.

The Magic: A Promise with an ID ✨

The solution we developed allows the user to get an immediate response, while behind the scenes, everything remains asynchronous and safe. We turned our Node.js Middleware into a smart control unit.

The secret lies in the combination of a Promise Map and a Correlation ID.

How Does It Actually Work?

The process consists of three simple but brilliant steps:

1. The Middleware Steps In
When the request arrives from the Frontend, we generate a correlationId – think of it as a unique ID card for the request. We create a Promise, store it in memory within a data structure we called a Promise Map, and just... wait. We launch the message to Kafka, with the ID and the "Reply Topic" name attached to the message headers. The Middleware essentially gets an order: "Stop and await response" (await promise).

2. The Round Trip
The Backend (in our case, a .NET microservice) consumes the command, does the hard work (like a DB update), and at the finish line – sends a reply message to the Reply Topic we defined earlier. The most important part? It attaches the exact same correlationId to the reply.

3. The Resolve
Our Middleware, which is still waiting, constantly listens to the Reply Topic using a dedicated Consumer. The moment an answer arrives, it checks the ID, pulls the matching Promise from the Map, and releases it (resolve).

The result? The client gets a full, final answer, and the user enjoys a smooth experience, without knowing what a crazy journey their message just went through.

Show Me The Code 💻

We've talked a lot, now let's see what this magic looks like in TypeScript. This is the heart of the mechanism in Node.js:

import { v4 as uuidv4 } from 'uuid'; 

// The map that holds all requests waiting for a reply
const pendingRequests = new Map();

async function sendRequestAndWaitForReply(command: any): Promise<any> {
    const correlationId = uuidv4();

    // Create a Promise and store it in the map with a unique ID
    const promise = new Promise((resolve, reject) => {
        // ... It's a good idea to add a Timeout here so we don't wait forever ...
        pendingRequests.set(correlationId, { resolve, reject });
    });

    // Send the message to Kafka (including the correlationId in headers)
    await kafkaProducer.send({ 
        topic: 'commands-topic',
        messages: [{ 
            key: correlationId, 
            value: JSON.stringify(command), 
            headers: { correlationId: correlationId, replyTo: 'reply-topic' } 
        }] 
    });

    return promise; // Wait patiently!
}

// When the answer arrives from the Reply Topic, our code does this:
function handleReplyMessage(message) {
    const correlationId = message.headers['correlationId'];
    const pending = pendingRequests.get(correlationId);

    if (pending) {
        // We found the Promise that was waiting for us!
        pendingRequests.delete(correlationId);
        pending.resolve(message.value);
    }
}

Wrapping Up

Sometimes the best solutions are those that bridge worlds. In this case, bridging the asynchronous world of the Backend with the synchronous need of the Frontend allowed us to maintain a robust architecture without compromising on user experience.

Have you encountered a similar problem? Have you implemented Request-Reply over Kafka differently? I'd love to hear about it in the comments! 👇