Streaming Azure Data Pipelines into real-time dashboards

#azure #dataanalytics #devops #analytics

Sometimes organizations are required to respond to events in real-time, allowing them to detect and respond to changes such as security threats, customer-impacting operational issues, business KPIs, system monitoring, etc. Using real-time dashboards can help them make data-driven decisions fast.

What is a real-time dashboard?
A real-time dashboard is a type of visualization that is automatically updated with the most current data available.

Which data sources can be used for real-time dashboards?
The sources of streaming data can be IOT sensors, social media posts, monitoring metrics, or any other time-sensitive data being collected.

Which technology can process streaming events?
There are many technologies that allow producing and consuming streams of events, such as Apache Kafka, Spark and others.
We will be using Azure technologies that will help us hit the road running, in no-time.

In this post I will show you how to create synthetic data, stream it using Azure Events hub, process it with Azure Stream Analytics and showcase it in Power BI in a live dashboard using "streaming datasets".

Let's start

What are we building?

We will:

Run a python script creating synthetic events and sending them to Azure Events hub
Use Azure Stream Analytics to consume the events from Events hub and sending them to Power BI streaming dataset
Visualize in Power BI the events in real time with a dashboard.

Here is the architecture

The recipe
I assume you already have an azure account (click here if you don't)

Create an events hub
In the azure portal create an Events Hub

Click on Create
Fill all the relevant fields
Click "review and create"
Click "create"

Go to your newly created events hub
Create a Shared Access Policy

Copy the Shared Access Policy primary key and the events hub name and paste it into the python script you downloaded from (also see below) github

import datetime
import time
import random
import socket
import struct
import json
from azure.eventhub import EventHubProducerClient, EventData
import psutil
import pyodbc
import pandas as pd

eventHubConnString = "<paste here the events hub shared access signature primary key>"
eventHubName = "<paste here the events hub name>"
producer = EventHubProducerClient.from_connection_string(conn_str=eventHubConnString, eventhub_name=eventHubName)
hostname = socket.gethostname()

def generateRandomIP():
    ip = socket.inet_ntoa(struct.pack('>I', random.randint(1, 0xffffffff)))
    return ip

def getRandomBrowser():
    browsersList = ["Edge", "Chrome", "Safari", "Firefox"]
    browser = random.choice(browsersList)
    return browser

def getRandomMethod():
    methods = [ "GET", "POST", "PUT", "DELETE" ]
    method = random.choice(methods)
    return method

def getRandomURL():
    urls = ["/page1.html", "/page2.html", "/page3.html", "/page4.html", "/page5.html" ,"/page6.html"  ]
    url = random.choice(urls)
    return url

def getCurrentDate():
    today = datetime.datetime.now()
    current = today.strftime("%Y-%m-%d")
    return current

def getCurrentTime():
    today = datetime.datetime.now()
    current = today.strftime("%H:%M:%S")
    return current

def getRandomTemperature():
    return random.randint(0,99)

def generateClickEvent():
    click_event = {
        "tableName" : "clicks",
        "date" : getCurrentDate(),
        "time": getCurrentTime(),
        "ip": generateRandomIP(),
        "browser": getRandomBrowser(),
        "method": getRandomMethod(),
        "url" : getRandomURL(),
        "productID": getRandomProductId()
    }
    return click_event

def getRandomProductId():
    return str(random.randint(0,99999))

def sendToEventsHub(jsonEvent):
    eventString = json.dumps(jsonEvent)
    print(eventString) 

    event_data_batch = producer.create_batch() 
    event_data_batch.add(EventData(eventString)) 
    producer.send_batch(event_data_batch)

def generateClicks():
    try:
        while True:
            clickEvent = generateClickEvent()    
            sendToEventsHub(clickEvent)
            # time.sleep(10)

    except KeyboardInterrupt:
        producer.close()
        conn.close()


if __name__ == "__main__":
    generateClicks()

Create a stream analytics job

Fill the details and click review and create and then Click create
Click on Inputs - Add stream input

Choose your subscription, the events hub we created above, and settings as shown below

Click Outputs and add a Power BI output

Fill all the details as shown below
Please notice that this will authorize the stream analytics job to access your power BI account (I recommend for demo purposes to use the same account as the one logged in into the Azure Portal.

Click on Query
Copy and paste the query snippet below

SELECT 
    DateAdd(s,-3,System.TimeStamp) AS windowstart,
    System.TimeStamp AS windowend,
    browser,
COUNT(*) AS entries
INTO
[powerbi]
FROM
[bddemo-clicks] TIMESTAMP BY EventProcessedUtcTime
GROUP BY browser, TumblingWindow(s, 3)

Save the query
Create a real time dashboard in Power BI
Login to power BI with the same credentials you provided the stream analytics job above
Create a dashboard

Add a tile to your dashboard

Click on Custom Streaming Data

Choose the stream we created in the stream analytics job

Choose the visualization type
We will choose the Line chart and fill the details

Choose a tile name and click Apply

We are done preparing everything!

Let's run our demo

Run the python script provided
Make sure you are authenticated to run your local code and connect to Azure (if not click here and install powershell)

Perform Azure authentication