DEV Community

John Bull
John Bull

Posted on • Updated on

Wash away the CRUD with Kafka Streaming Part2

Kafka Streams with your web app.


In the first post, we got Kafka set up and created a message producer to feed some employee data into our Kafka cluster. Post1

In this next one, we will create a consumer and a front end for our application with Express and Vue.js

For our web app to take advantage of the message system we need to create a consumer. We want our data to be available when a user gets to a certain page of our website, so we want to create a little express backend that will call the consumer into action when we want it.


const express = require("express");
const bodyParser = require("body-parser");
const cors = require("cors");
const GetMessage = require("./consumer");

const app = express();
app.use(bodyParser.urlencoded({ extended: true }));

app.get("/api/employees", GetMessage);

app.listen(process.env.PORT || 3000, function() {
  console.log("Kafka Frontend Dev site running");
Enter fullscreen mode Exit fullscreen mode

We are setting up one route that tells express to run the GetMessage function on our consumer.js file. So let's look at that file.

var kafka = require("kafka-node");

var express = require("express");
var router = express.Router();
var messageData = [];

const GetMessage = message => {
  var Consumer = kafka.Consumer;
  var data = [];
  const client = kafka.Client("localhost:2181");
  topics = [
      topic: "employees"
  const options = {
    autoCommit: true,
    fromBeginning: true

Enter fullscreen mode Exit fullscreen mode

Here are our setup and config options to connect to the server and connect to the topic employees.

  var consumer = new kafka.Consumer(client, topics, options);
  this.message = message

   consumer.on("message", function(message, err) {

    if (err) {
    } else {
      console.log("Here is the kafka message... " + JSON.stringify(message));

      if (message.event_data === "emp_chng_02") {
        console.log("A change happened for employee " + message.f_name + " " + message.l_name + ".")

router.get("/api/employees", (req, res) => {
  messageData.forEach(function(message, error) {
    if (message) {
      res.send({key: message});
    } else if (error) {
      res.status(400).send("Error, something went wrong.");

module.exports = router;
Enter fullscreen mode Exit fullscreen mode

Here we are creating the consumer and catching errors as well as running the getMessage function and sending each message we get to the express route. As long as we have our server.js file running in node we should see something when we navigate to localhost:3000/api/employees. We can start to see that we could use messages from Kafka in place of a query from a database as we would in a normal CRUD Express app.

api data

Now let's have some fun and set up a little front end to display our results. I ended up using create-vue-app because I wanted something quick and easy.

yarn global add create-vue-app

yarn create vue-app kafkaFrontend

We should see these files.

├── index.ejs
├── package.json
├── poi.config.js
├── src
│   ├── components
│   │   ├── App.test.js
│   │   └── App.vue
│   ├── index.js
│   └── polyfills.js
├── static
│   └── favicon.ico
└── yarn.lock
Enter fullscreen mode Exit fullscreen mode

We are going to want a component to display the employees. Our template displays a card for each employee and

  <div class="main" id="app">

    <div class="card-container">
        <div class="employee-card" v-for="employee in employees">
        <div>{{employee.f_name }} {{employee.l_name}}</div> 
        <div>Hire date: {{employee.hire_date}}</div>  



import axios from 'axios';

export default {
  name: "Employees",
  data() {
    return {
      employees: [],
      errors: [],


  created() {
    axios.get('http://localhost:3000/api/employees').then((response) => {
      let responseString =
      responseString.forEach(element => {
        element = JSON.parse(element)
        if (element.event_id==="emp_chng_02") {
        } else {
    .catch(e => {
  components: {



 <!-- Scoped component css -->
 <!-- It only affect current component -->
 <style scoped>
.card-container {
  display: flex;
  flex-direction: row;
  flex-wrap: wrap;


.employee-card {
  display: flex;
  flex-direction: column;
  padding: 38px;
  width: 145px;
  height: 86px;
  max-width: 360px;
  margin: 40px;
  background: #fff;
  box-shadow: 0 1px 3px rgba(0, 0, 0, 0.12), 0 1px 2px rgba(0, 0, 0, 0.24);
  transition: all 0.3s cubic-bezier(0.25, 0.8, 0.25, 1);

.employee-card:hover {
  box-shadow: 0 18px 32px rgba(0, 0, 0, 0.25), 0 10px 10px rgba(0, 0, 0, 0.22);

.alert {
  color: #FF4136;
  background: #001f3f;
  border: #001f3f;


Enter fullscreen mode Exit fullscreen mode

In our App.vue file we will need to import our component.

  <div id="app">
    <employees />

import Employees from "./components/employees.vue";

export default {
  name: 'app',
  components: {
Enter fullscreen mode Exit fullscreen mode

We will need to bring it into our index.js file as well.

import Vue from "vue";
import Employees from "./components/employees.vue";
import { library } from '@fortawesome/fontawesome-svg-core'
import { faExclamationTriangle } from '@fortawesome/free-solid-svg-icons'
import { FontAwesomeIcon } from '@fortawesome/vue-fontawesome'


Vue.component('font-awesome-icon', FontAwesomeIcon)

Vue.config.productionTip = false;

new Vue({
  el: '#app',
  render: h => h(Employees),
  data () {
    return {
      info: null
Enter fullscreen mode Exit fullscreen mode

If we run the vue server from the console we should have the employees displayed when we go to the URL http://localhost:8080.
employee site

Lastly, We will do something with our data by adding some event id's and seeing what happens.


Top comments (0)