Through this small guide, my intention is to enable anyone to understand and begin their journey of working with Kafka in a few minutes.
After following this guide and using the tools mentioned, which is not the only way to achieve the output, you should have a working example on your hands and the understanding to use more advanced concepts.
We shall start by building the individual modules, step-by-step and integrate all of them in the end. So, let's get started!
1. Setting up Kafka
This is where we set up Kafka in our system. So, head down to the official Apache Kafka quick-start guide. Follow the steps till Step 3 and create a topic with any desired name. In my examples I have used the name topic_stream.
2. Creating a Kafka Consumer
Glad you are back! Let's now use the Kafka-node consumer to see how a consumer functions. A consumer simply reads messages from the Kafka topic.
Below, we have created a Consumer that subscribes to topic_stream and prints the data that is produced.
I have setup the Kafka client on port 9092 in step 1. You should change the configuration according to your settings.
const kafka = require('kafka-node');
const Consumer = kafka.Consumer,
client = new kafka.KafkaClient('localhost:9092'),
consumer = new Consumer(
client, [ { topic: 'topic_stream', partition: 0 } ], { autoCommit: false });
consumer.on('message', function (message) {
console.log(message);
});
Pretty straightforward, right?
3. Creating a Socket API
Let's now setup a WebSocket API in our NodeJs service so we start streaming data to a webpage. We use Socket.io in conjunction with express. In a file called server_consumer.js create the following:
const express = require('express');
const port = 3000;
const app = express();
const server = app.listen(port, () => {
console.log(`Listening on port ${server.address().port}`);
});
const io = require('socket.io')(server, {
cors: {
origin: '*',
}
});
io.on('connection', client => {
console.log('Connected', client);
client.on('event', data => {
console.log('Event triggered by client')
});
client.on('disconnect', () => {
console.log('Client disconnected');
});
});
4. Creating the Consumer, Socket API
We now know what a consumer is and how we intend to use the Socket.io API. Let's bring it all together now. Edit the server_consumer.js such that it now contains the Consumer as well:
const kafka = require('kafka-node');
const express = require('express');
const port = 3000;
const app = express();
const Consumer = kafka.Consumer,
client = new kafka.KafkaClient('localhost:9092'),
consumer = new Consumer(
client, [ { topic: 'topic_stream', partition: 0 } ], { autoCommit: false });
const server = app.listen(port, () => {
console.log(`Listening on port ${server.address().port}`);
});
const io = require('socket.io')(server, {
cors: {
origin: '*',
}
});
io.on('connection', client => {
console.log('Connected', client);
consumer.on('message', function (message) {
client.emit('request', message.value);
});
client.on('disconnect', () => {
console.log('Client disconnected');
});
});
5. Creating the frontend using Chart JS
On our web browser, we will create a webpage that invokes the socket API and displays the data using Chart JS. You may use D3.js or any other libraries that you are comfortable with to display the data.
Let's create a simple index.html:
<html>
<head>
<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/chart.js@2.8.0"></script>
<script src="https://cdn.socket.io/socket.io-3.0.1.min.js"></script>
<script type="text/javascript" src="index.js"></script>
</head>
<body>
<canvas id="myChart" width='300' height='100'></canvas>
</body>
</html>
In our index.js, we will make a call to our socket API and initiate listeners for data:
$(document).ready(function(){
const count = 10;
const data = {
labels : [...Array(count).keys()].map(i => i.toString()),
datasets : [
{
label: '# - Streamed number',
backgroundColor: "rgba(50,220,220,0.5)",
data : Array(count).fill(0),
}
]
}
const updateData = function(newVal){
const labels = data["labels"];
const dataSetInitial = data["datasets"][0]["data"];
count++;
labels.push(count.toString());
labels.shift();
const newData = Math.floor(newVal);
dataSetInitial.push(newData);
dataSetInitial.shift();
};
const ctx = document.getElementById("myChart").getContext("2d");
const chart = new Chart(ctx, {
type: 'line',
data,
options: {animation : false}
});
function webSocketInvoke() {
var socket = io('http://localhost:3000');
socket.on('event', (value) => {
updateData(value);
chart.update();
});
}
webSocketInvoke();
});
We will now open the index.html and wait for the data to stream through.
6: Producing data
Let's get to the final step where we produce data that can then be streamed to our setup. I will mention with 2 options to do this:
- Follow the instruction given in Step 4 of the Kafka quick-start guide. That's it.
- If you would like to explore the kafka-node library further, you may experiment and create a
producer.jsfile and run it using your node environment; example file given here.
That's it! You should now have a fully functional Apache Kafka stream at your hands and the understanding on how the end-to-end flow works.
Github link for the project: https://github.com/yuvraj2112/kafka_socket_stream
Top comments (0)