Real-Time Video Streaming Using Kafka, Flask & OpenCV

1. Overview

This document presents the architecture for a real-time video streaming web application that uses Apache Kafka as the data pipeline, Flask as the web server, OpenCV for video frame processing, and Gunicorn + NGINX for production-grade deployment.
The application captures video (live or file-based), streams it through Kafka, and displays it on the web in real time.

---

2. System Goals

Goal	Description
Real-Time Video Streaming	Provide smooth and low-latency video feed from producers to web clients
Scalability	Kafka enables distributed data handling, supporting horizontal scaling
Modularity	Components are decoupled: producer, broker, consumer
Maintainability	Python-based and well-structured for readability and testing

---

3. Technology Stack

Component	Tool / Library	Role
💻 OS	Ubuntu Server 24.04.2	https://ubuntu.com/download/server
💬 Messaging	Apache Kafka	Message broker for real-time data streaming
🐍 Backend	Python 3, Kafka-Python	Business logic and video frame handling
🌐 Web App	Flask	Micro web framework for streaming video
🖼 Vision	OpenCV	Frame capture, processing and encoding
🚀 Server	Gunicorn + NGINX	WSGI and reverse proxy setup for production

---

4. System Architecture Diagram

4.1 Infrastructure Diagram

This diagram focuses on the deployment and hosting architecture:

4.2 Messaging System Diagram

This focuses on the video processing pipeline and Kafka message flow:

5. Guided Implementation

5.1 Configuring Python

We start by creating a virtual environment:

sudo apt install python3 python3-pip
pip3 install --upgrade pip
pip3 install virtualenv

mkdir kafka-video-streaming
cd kafka-video-streaming
virtualenv venv
source venv/bin/activate

5.2 Installing Requirements

Essential Components for Running the Application in the Browser:

To build and run a real-time web streaming application using Python, we need to set up a few core components that allow smooth communication between the server and the browser. This system relies on:

• Flask – to build the backend web server.
• NGINX – as a reverse proxy for secure and scalable request handling.
• Gunicorn – a production-grade WSGI server to serve the Flask app.
• Kafka – to stream video data between the producer and consumer components.
• OpenCV - an open-source library designed for computer vision and machine learning applications.

pip install kafka
pip install kafka-python

pip install flask

This command also installs essential Flask dependencies, including:

• Werkzeug – a comprehensive WSGI utility library for request/response handling.
• Jinja2 – a templating engine used by Flask.
• MarkupSafe – helps escape characters in Jinja templates.
• ItsDangerous – provides secure signing for data like cookies and tokens.

OpenCV:

pip install opencv-contrib-python

Now we install NGINX with the command:
sudo apt update
sudo apt install nginx

last but not least Gunicorn:

pip install gunicorn

5.2 Confugiring NGINX

Before setting up our custom NGINX configuration for the Flask app, remove the default site configuration to avoid conflicts:

sudo rm /etc/nginx/sites-enabled/default

Now we create a new configuration file inside the sites-available directory. Then, create a symbolic link from this file to the sites-enabled directory to activate the configuration.
This tells NGINX to use our custom settings instead of the default ones.
touch /etc/nginx/sites-available/flask_settings
ln -s /etc/nginx/sites-available/flask_settings /etc/nginx/sites-enabled/flask_settings

Let's open the file located in the sites-enabled directory and add the following configuration to it:

These lines are intended to configure the web server to act as a reverse proxy. Inside the location block, we specify the parameters that enable this behavior:

1. proxy_pass: This line forwards all incoming requests to the target server. In this case, we are using the loopback address (127.0.0.1), and the requests are forwarded to port 8000, which we will discuss shortly.
2. proxy_set_header Host $host;: This line ensures the original host header from the client is passed along with the request to the upstream server.
3. proxy_set_header X-Real-IP $remote_addr;: This line ensures the client’s real IP address is passed to the upstream server, rather than the IP address of the reverse proxy (NGINX).

Now let's restart NGINX and check its status:
systemctl restart nginx
systemctl status nginx.service

After all these configs are done, it's safe to say that we have this architecture:

Nginx acts as a reverse proxy, meaning it receives incoming requests and decides where to forward them next. It also serves static files like images and CSS, and supports encryption via the SSL protocol.
In the second layer, we have Gunicorn. When Nginx receives a request—for example, to www.domain.com—it checks the configuration files and sees that it should forward the request to Gunicorn. We’ve specified port 8000 in the web application configuration, so Nginx forwards the request to Gunicorn through this port.

Gunicorn’s role is to handle dynamic content. It receives the request passed from Nginx via theproxy_pass directive and generates the appropriate dynamic response.

5.3 Kafka Producer – Video Publisher

In this section of the code, we import the required libraries and define the topic. We then define the pub_video function, whose purpose is to stream a pre-recorded video selected by the user. This function takes one parameter: video_file, which is a string representing the path to the video file that will be streamed.

• First statement inside the function assigns the bootstrap server to the producer.
• Second statement opens the video file using OpenCV. To do this, we create a VideoCapture object from the cv2 class. This object takes an argument: either the index of the device to stream from or the name of the video file.
• Third statement prints a simple log message.
• Fourth statement enters a while loop, where the condition is video.isOpened(). In some cases, the video capture may not be initialized correctly, so it's important to check whether the file has been successfully opened before proceeding with streaming.
• Fifth statement: If the video file is successfully opened, the video will be read frame by frame.
• Sixth statement: If the video cannot be opened, an error message is printed and the loop exits.
• Seventh statement: This line converts the frame to PNG format. The ret value indicates whether the frame was successfully captured (True or False). The buffer variable stores the result of applying the imencode() function to the frame, which compresses and encodes the image to memory.
• Eighth statement: Converts the image to bytes format to be transmitted between the producer and the consumer, as required.
• Ninth statement: Adds a short delay using time.sleep(0.2) to give time for each frame to be sent to the Kafka topic.
• Tenth statement: Releases the video file after the streaming is complete. now we write the code for live streaming:

The process is similar to streaming a pre-recorded video, but with a few differences. In the second statement inside the function, we assign the value 0 to cv2.VideoCapture(), which indicates the index of the camera that OpenCV will use to capture input.
As for publishing the video to the Kafka topic, it follows the exact same method. Once the streaming is complete, we call the camera.release() function to release the camera resource.

last snippet of the producer code:

• First statement: Before executing the code, the Python interpreter reads the source file and defines global variables. If the interpreter runs this source file as the main program, it assigns the special variable __name__ the value __main__.
• Second statement: sys.argv is a Python list that contains the arguments passed to the script via the command line. Here, we check the length of this list—if it’s greater than 1, we assume the user has provided a second argument, which will be treated as the path to the video file to be streamed. In this case, we call the pub_video function and pass that file as a parameter.
• Third statement: If the user doesn’t provide any arguments, the program will stream live video by default.

5.4 Kafka Consumer + Flask Web Server

Responsibility: Subscribes to the video-stream topic, retrieves video frames, and sends them as multipart responses to the client browser.

• After importing the required libraries and defining both the Kafka consumer and the Flask app, we define the index() function, which simply returns the home page of the web application (an optional step).
• We then define a route /kaf/ for streaming the video, and set the HTTP method to GET (which can be either GET or POST).
• Next, we define the video() function — this is the core of the streaming logic. Inside this function, we use multipart responses, a key mechanism in streaming applications where each piece of data replaces the previous one. This enables continuous video playback in the browser.
• The idea is that each data chunk is treated as an image, and by replacing each image with the next one in sequence, we simulate a live video stream. To enable real-time frame updates, we use what’s called a multipart boundary, which segments each part of the response.
• We're using the content type: multipart/x-mixed-replace
• This format is ideal for video streaming where each part replaces the previous one in a pipeline-like manner.
• Then we define the kafs() function, which receives images from the Kafka server and converts them into a format compatible with Flask.
• When the program runs, the video() function is called first. It then calls thekafs() function, which handles converting each frame. These converted frames are passed back to video(), which returns them as a response — each one replacing the previous, creating a live stream effect in the browser.
• Finally, we call app.run() in the main program block to launch the Flask server, specifying the host IP address that will serve the video stream.

5.5 Running The App

We run the producer with the folowing commands:
To stream a pre recorded video:
Python <file_name>.py <video_file's name>
To stream a live video:

now we run the app using gunicorn and we open the browser:
gunicorn app_name:app

To see the video we add /kaf to the browser url:

Live Video:

now we re-open the browser:

And Voila!

What's next?

We will build a web application that detects faces and identifies motion by calculating the difference between consecutive video frames. Motion detection is widely used in security applications, especially for identifying moving objects within a monitored area — particularly people. For example, if unusual movement is detected in a surveillance camera feed, the system can raise an alert to indicate suspicious or unexpected activity.
In the next project, we will combine motion detection with face detection to simulate a basic security system — similar to what you might find in intelligent surveillance software.