DEV Community: Rafif

Real-Time Video Streaming Using Kafka, Flask & OpenCV

Rafif — Wed, 23 Jul 2025 11:29:47 +0000

1. Overview

This document presents the architecture for a real-time video streaming web application that uses Apache Kafka as the data pipeline, Flask as the web server, OpenCV for video frame processing, and Gunicorn + NGINX for production-grade deployment.
The application captures video (live or file-based), streams it through Kafka, and displays it on the web in real time.

2. System Goals

Goal	Description
Real-Time Video Streaming	Provide smooth and low-latency video feed from producers to web clients
Scalability	Kafka enables distributed data handling, supporting horizontal scaling
Modularity	Components are decoupled: producer, broker, consumer
Maintainability	Python-based and well-structured for readability and testing

3. Technology Stack

Component	Tool / Library	Role
💻 OS	Ubuntu Server 24.04.2	https://ubuntu.com/download/server
💬 Messaging	Apache Kafka	Message broker for real-time data streaming
🐍 Backend	Python 3, Kafka-Python	Business logic and video frame handling
🌐 Web App	Flask	Micro web framework for streaming video
🖼 Vision	OpenCV	Frame capture, processing and encoding
🚀 Server	Gunicorn + NGINX	WSGI and reverse proxy setup for production

4. System Architecture Diagram

4.1 Infrastructure Diagram

This diagram focuses on the deployment and hosting architecture:

4.2 Messaging System Diagram

This focuses on the video processing pipeline and Kafka message flow:

5. Guided Implementation

5.1 Configuring Python

We start by creating a virtual environment:

sudo apt install python3 python3-pip
pip3 install --upgrade pip
pip3 install virtualenv

mkdir kafka-video-streaming
cd kafka-video-streaming
virtualenv venv
source venv/bin/activate

5.2 Installing Requirements

Essential Components for Running the Application in the Browser:

To build and run a real-time web streaming application using Python, we need to set up a few core components that allow smooth communication between the server and the browser. This system relies on:

Flask – to build the backend web server.
NGINX – as a reverse proxy for secure and scalable request handling.
Gunicorn – a production-grade WSGI server to serve the Flask app.
Kafka – to stream video data between the producer and consumer components.
OpenCV - an open-source library designed for computer vision and machine learning applications.

pip install kafka
pip install kafka-python

pip install flask

This command also installs essential Flask dependencies, including:

Werkzeug – a comprehensive WSGI utility library for request/response handling.
Jinja2 – a templating engine used by Flask.
MarkupSafe – helps escape characters in Jinja templates.
ItsDangerous – provides secure signing for data like cookies and tokens.

OpenCV:

pip install opencv-contrib-python

Now we install NGINX with the command:
sudo apt update sudo apt install nginx

last but not least Gunicorn:

pip install gunicorn

5.2 Confugiring NGINX

Before setting up our custom NGINX configuration for the Flask app, remove the default site configuration to avoid conflicts:

sudo rm /etc/nginx/sites-enabled/default

Now we create a new configuration file inside the sites-available directory. Then, create a symbolic link from this file to the sites-enabled directory to activate the configuration.
This tells NGINX to use our custom settings instead of the default ones.
touch /etc/nginx/sites-available/flask_settings
ln -s /etc/nginx/sites-available/flask_settings /etc/nginx/sites-enabled/flask_settings

Let's open the file located in the sites-enabled directory and add the following configuration to it:

These lines are intended to configure the web server to act as a reverse proxy. Inside the location block, we specify the parameters that enable this behavior:

proxy_pass: This line forwards all incoming requests to the target server. In this case, we are using the loopback address (127.0.0.1), and the requests are forwarded to port 8000, which we will discuss shortly.
proxy_set_header Host $host;: This line ensures the original host header from the client is passed along with the request to the upstream server.
proxy_set_header X-Real-IP $remote_addr;: This line ensures the client’s real IP address is passed to the upstream server, rather than the IP address of the reverse proxy (NGINX).

Now let's restart NGINX and check its status:
systemctl restart nginx
systemctl status nginx.service

After all these configs are done, it's safe to say that we have this architecture:

Nginx acts as a reverse proxy, meaning it receives incoming requests and decides where to forward them next. It also serves static files like images and CSS, and supports encryption via the SSL protocol.
In the second layer, we have Gunicorn. When Nginx receives a request—for example, to www.domain.com—it checks the configuration files and sees that it should forward the request to Gunicorn. We’ve specified port 8000 in the web application configuration, so Nginx forwards the request to Gunicorn through this port.

Gunicorn’s role is to handle dynamic content. It receives the request passed from Nginx via theproxy_pass directive and generates the appropriate dynamic response.

5.3 Kafka Producer – Video Publisher

In this section of the code, we import the required libraries and define the topic. We then define the pub_video function, whose purpose is to stream a pre-recorded video selected by the user. This function takes one parameter: video_file, which is a string representing the path to the video file that will be streamed.

First statement inside the function assigns the bootstrap server to the producer.
Second statement opens the video file using OpenCV. To do this, we create a VideoCapture object from the cv2 class. This object takes an argument: either the index of the device to stream from or the name of the video file.
Third statement prints a simple log message.
Fourth statement enters a while loop, where the condition is video.isOpened(). In some cases, the video capture may not be initialized correctly, so it's important to check whether the file has been successfully opened before proceeding with streaming.
Fifth statement: If the video file is successfully opened, the video will be read frame by frame.
Sixth statement: If the video cannot be opened, an error message is printed and the loop exits.
Seventh statement: This line converts the frame to PNG format. The ret value indicates whether the frame was successfully captured (True or False). The buffer variable stores the result of applying the imencode() function to the frame, which compresses and encodes the image to memory.
Eighth statement: Converts the image to bytes format to be transmitted between the producer and the consumer, as required.
Ninth statement: Adds a short delay using time.sleep(0.2) to give time for each frame to be sent to the Kafka topic.
Tenth statement: Releases the video file after the streaming is complete. now we write the code for live streaming:

The process is similar to streaming a pre-recorded video, but with a few differences. In the second statement inside the function, we assign the value 0 to cv2.VideoCapture(), which indicates the index of the camera that OpenCV will use to capture input.
As for publishing the video to the Kafka topic, it follows the exact same method. Once the streaming is complete, we call the camera.release() function to release the camera resource.

last snippet of the producer code:

First statement: Before executing the code, the Python interpreter reads the source file and defines global variables. If the interpreter runs this source file as the main program, it assigns the special variable __name__ the value __main__.
Second statement: sys.argv is a Python list that contains the arguments passed to the script via the command line. Here, we check the length of this list—if it’s greater than 1, we assume the user has provided a second argument, which will be treated as the path to the video file to be streamed. In this case, we call the pub_video function and pass that file as a parameter.
Third statement: If the user doesn’t provide any arguments, the program will stream live video by default.

5.4 Kafka Consumer + Flask Web Server

Responsibility: Subscribes to the video-stream topic, retrieves video frames, and sends them as multipart responses to the client browser.

After importing the required libraries and defining both the Kafka consumer and the Flask app, we define the index() function, which simply returns the home page of the web application (an optional step).
We then define a route /kaf/ for streaming the video, and set the HTTP method to GET (which can be either GET or POST).
Next, we define the video() function — this is the core of the streaming logic. Inside this function, we use multipart responses, a key mechanism in streaming applications where each piece of data replaces the previous one. This enables continuous video playback in the browser.
The idea is that each data chunk is treated as an image, and by replacing each image with the next one in sequence, we simulate a live video stream. To enable real-time frame updates, we use what’s called a multipart boundary, which segments each part of the response.
We're using the content type: multipart/x-mixed-replace
This format is ideal for video streaming where each part replaces the previous one in a pipeline-like manner.
Then we define the kafs() function, which receives images from the Kafka server and converts them into a format compatible with Flask.
When the program runs, the video() function is called first. It then calls thekafs() function, which handles converting each frame. These converted frames are passed back to video(), which returns them as a response — each one replacing the previous, creating a live stream effect in the browser.
Finally, we call app.run() in the main program block to launch the Flask server, specifying the host IP address that will serve the video stream.

5.5 Running The App

We run the producer with the folowing commands:
To stream a pre recorded video:
Python <file_name>.py <video_file's name>
To stream a live video:

now we run the app using gunicorn and we open the browser:
gunicorn app_name:app

To see the video we add /kaf to the browser url:

Live Video:

now we re-open the browser:

And Voila!

What's next?

We will build a web application that detects faces and identifies motion by calculating the difference between consecutive video frames. Motion detection is widely used in security applications, especially for identifying moving objects within a monitored area — particularly people. For example, if unusual movement is detected in a surveillance camera feed, the system can raise an alert to indicate suspicious or unexpected activity.
In the next project, we will combine motion detection with face detection to simulate a basic security system — similar to what you might find in intelligent surveillance software.

Getting Started with Apache Kafka: A Beginner’s Guide to Real-Time Data Streaming

Rafif — Thu, 03 Apr 2025 14:28:53 +0000

1.Introduction

With the rapid evolution of computing systems, the need for a fast, scalable, and fault-tolerant messaging system has grown significantly. Apache Kafka has emerged as one of the most powerful and widely used messaging systems, providing a highly efficient way to process large volumes of data in real time. Without requiring massive computational resources, Kafka can handle thousands of messages per second with minimal latency, making it a preferred choice for major tech companies like LinkedIn, Twitter, Mozilla, Netflix, and Oracle.

Modern businesses rely on data to understand trends, analyze customer behavior, and automate processes. Kafka plays a crucial role in real-time data processing and predictive analytics by reducing the time between event registration and system response. Originally developed by LinkedIn in 2011 as an open-source project, Kafka was later acquired by Apache and is now further developed by Confluent, founded by Kafka's original creators: Jay Kreps, Neha Narkhede, and Jun Rao.

Kafka's core philosophy revolves around treating data as a continuous stream rather than static storage. This approach is particularly useful in machine learning, security monitoring, and real-time video analytics, where data needs to be processed and responded to instantly.

1.1 Messaging Systems

Messaging systems facilitate communication between applications by transferring data asynchronously. They are categorized into two main types:

1.1.1 Point-to-Point(Queue-Based Messaging)

A producer sends messages to a queue, where a single consumer retrieves them.
Once consumed, messages are removed from the queue.
If the consumer is unavailable, the message remains in the queue until it is processed.

1.1.2 Publish-Subscribe (Pub/Sub)

A publisher sends messages to a topic, which multiple subscribers can read from.
Subscribers must be available to receive messages, or they may be lost.
Unlike the queue model, messages are not deleted after being read by one subscriber.

1.2 Apache Kafka Concept

Kafka is a distributed, real-time streaming platform designed to handle millions of messages per second. It processes continuous data streams by collecting, storing, and distributing records efficiently across different consumers.

Kafka offers three key functions:

Message Publishing & Subscription: Stores and distributes records sequentially, ensuring reliability.
Fault-Tolerance: Ensures system stability even in case of failures.
Real-Time Processing: Supports instant processing of high-speed data streams.

1.3 Why Use Kafka?

Traditional messaging systems often face challenges like high latency and message buildup under heavy traffic. Kafka addresses these limitations with a modern, robust design that offers:

High-throughput message storage that's both scalable and efficient.
Fault-tolerant architecture that ensures no data is lost, even during failures.
Real-time data streaming for instant processing and analytics.
Unified platform that seamlessly combines messaging, storage, and stream processing.

1.4 Kafka Workflow

Even though publish-subscribe (pub/sub) and queuing are different messaging patterns, Kafka combines both to support various use cases. Sometimes, Kafka functions as a traditional topic-based pub/sub system, where data is sent to topics as a continuous stream of records. These records are structured in a sequential and ordered manner, and multiple subscribers (consumers) can process them independently.

1.4.1 Pub/Sub

In the pub/sub model, an application (the producer) connects to Kafka and publishes messages to a topic. Kafka stores these messages in a structured log that is divided into partitions (segments). Multiple consumers can subscribe to a topic, and Kafka ensures that each consumer gets assigned a specific partition. When another application (consumer) connects, it reads and processes records from its assigned partition.

1.4.2 Queuing

In the queuing model, both producers and consumers connect to Kafka in a similar way. However, unlike pub/sub, queuing ensures that each message is delivered to only one consumer for processing. Messages are stored in a queue until a consumer retrieves them.

A key characteristic of this approach is that it does not support multiple consumers processing the same message simultaneously on a single machine. This design is ideal for workloads where requests need to be processed in a sequential manner rather than being handled simultaneously by multiple consumers.

For instance, in large-scale machine learning (ML) applications, requests are often processed one after another and then stored in a queue before being passed to the next stage. Since ML workloads can be computationally heavy, handling requests in a sequential pipeline ensures better resource utilization. This is where Kafka’s consumer API plays a crucial role in managing and distributing these workloads efficiently.

1.5 Kafka Cluster Architecture

Kafka consists of several core components:

1.5.1 Brokers:

A Kafka broker is a server responsible for handling incoming requests and managing topics. It plays a key role in distributing and storing records efficiently.

In a Kafka cluster, there can be one or multiple brokers working together. Each broker holds a copy of the data and manages the topics created within the cluster.

Producers send records to a broker, which then forwards them to the appropriate topic. On the other end, consumers retrieve records from the broker as needed.

The primary reason for using multiple brokers is to take advantage of replication, ensuring data availability and fault tolerance within Kafka.

In Kafka, a topic is divided into 8 partitions to enable parallel processing and scalability. These partitions allow multiple consumers to read data concurrently, ensuring efficient data distribution.

Each broker in the Kafka cluster manages one or more partitions, balancing the load and improving fault tolerance. This partitioning mechanism enhances throughput and performance, making it easier to handle large-scale data streams.

1.5.2 ZooKeeper:

Zookeeper is a distributed coordination service that Kafka uses to maintain synchronization and manage metadata across brokers. It ensures fault tolerance and efficient communication between Kafka components.

Offset Management: Consumers use offsets stored in logs to keep track of their position while reading data.
Cluster Coordination: If a broker fails or a new one joins, Zookeeper helps rebalance the cluster.
Leader Election: It manages leader selection for partitions, ensuring smooth operations.
Service Discovery: Producers and consumers use Zookeeper to locate active brokers and topics.

1.5.3 Producers:

Send messages to Kafka topics.

1.5.4 Consumers:

Retrieve and process messages from topics using offset tracking to ensure correct message order.

2. Install and Run Kafka

2.1 Installation

Kafka requires Java to run. Install OpenJDK 8 using the following command:

sudo apt update
sudo apt install openjdk-8-jdk

Verify the installation:

java -version

Go to the Apache Kafka website and download the latest version. Alternatively, you can use wget to download it directly:

wget https://downloads.apache.org/kafka/<latest_version>/kafka_<latest_version>.tgz

Extract the downloaded file and check its content:

tar -xzf kafka_<latest_version>.tgz
cd kafka_<latest_version>

In this project, we rely on the bin and config directories to run and manage Apache Kafka services.

bin directory: Contains all the necessary shell scripts for starting, stopping, and managing Kafka services, such as running Zookeeper, Brokers, and handling topics.
config directory: Includes configuration files for Kafka, covering settings for Brokers, Zookeeper, Topics, Producers, and Consumers. By exploring the contents of these directories, we can access operational tools that simplify system management and control data flow within Kafka.

2.2 Startup Kafka

To run Kafka, we must first start Zookeeper. Zookeeper is essential for managing the Kafka cluster and ensuring synchronization and coordination between the brokers. As a result, if an error occurs while starting Zookeeper, it will cause Kafka to fail to start due to its heavy dependency on it.
We start Zookeeper using the command:

bin/zookeeper-server-start.sh config/zookeeper.properties

We will get the following output:

The message enclosed in a circle indicates that Zookeeper is listening on port 2181, which is its designated port. From this notification, we can confirm that Zookeeper is running without any issues. Now, to start Kafka, we open a new session within the virtual machine we're working on by pressing Alt + F3 (this may vary depending on the device). After opening the new session and logging in, we can start Kafka using the command:

bin/kafka-server-start.sh config/server.properties

Now we get the following output:

Since it outputted "started," there are no issues. We notice that it assigned the number 0 to the server we just started, which indicates the initial broker number that Kafka creates when it starts.

2.3 Kafka Topics

A topic is a collection of partitions that contain immutable, ordered records, each identified by a unique offset, making the records sequential. The primary goal of having multiple partitions is to allow users to read from the topic in a parallel manner.

To create a topic, we execute the command in a new session (ensure that both Kafka and Zookeeper are running):

bin/kafka-topics.sh –-create –-bootstrap-server localhost:9092 –-replication-factor 1 –-partitions 1 –topic <topic name>

The replication factor indicates the number of copies of the topic that should exist within the Kafka cluster. Its value can be 1, 2, or 3. A value greater than 1 helps store a backup of the data in another broker within the cluster for backup or load balancing purposes. On the other hand, partitions are used to segment the topic, enabling parallel processing. A topic consists of partitions that divide the data across multiple brokers as we mentioned earlier.

After creating the topic, we will send messages from the producer to be received by the consumer. We open both the producer and consumer in separate sessions. we execute the following commands:

producer:

bin/kafka-console-producer.sh --broker-list localhost:9092 -topic <topic name>

Consumer:

bin/kafka-console-consumer.sh localhost:9092 -topic <topic name> --from-beginning

We will notice that the time taken for messages to travel from the producer to the consumer is almost instantaneous, which is one of the main advantages of Kafka—its speed.

And that’s it! In the upcoming chapters, we’ll dive into more advanced features for data streaming—including video and live video streaming using Kafka.