video analytics on edge

INTRODUCTION TO VIDEO ANALYTICS ON THE JETSON NANO

Image and audio processing are the major senses in the five common senses in human beings and animals. They are the senses that provide more comprehensive data about an environment and this data is processed to trigger different reactions to environmental methods.

Machine learning and deep learning has led rise to smart devices with image and audio processing capabilities, these technologies have revolutionized how we interact with machines from domestic appliances to industrial machines, it has allowed machines to have an intelligence of their own and make decisions that reduce losses, human hours in a factory or even turn on mood lights before a party or during that song that I love.

To give the best inference on audio and image data it sometimes requires that several machine learning and deep-learning models in parallel or in series, especially when running the models in low power consumption and low computational powered devices. This has been a big challenge for developers building audio and image processing solutions.

Nvidia discovered this problem and built the NVIDIA DEEPSTREAM SDK which is an accelerated Artificial Intelligence framework to build video analytics pipelines it is built on the opensource G- streamer plugin. It is having been developed to be a deployment tool and to machine learning models deployment. It is primarily developed in the C/C++ but new python bindings have been developed and also a low code graphical composer method.

The basic steps of working of a deep stream application are:

1. Creating a source file where the SDK will get the data to be inferred in the deep stream pipeline video files are converted into h264 format.
2. The data is decoded into image and audio data
3. Running inferencing on the decoded data and this involves running the models and detecting objects in the model
4. A buffer I created to provide the results from the inferencing models and this allows overlaying of bounding boxes on objects detected in a video stream.
5. An output is rendered with to the display showing the video stream with bounding boxes on detected objects or even displaying a number of objects detected in an image.

The deep stream SDK offers developers a solution to create deployment of AI applications and also gives additional features like Smart record features where a stream of video can be recorded in either a local storage or a cloud storage like azure or amazon AWS.

The deep stream SDK can be applied solving multiple problems in different fields like medicine, robotics, self-driving vehicles and many more.

Getting started with deep stream on JETSON Devices.

Nvidia has provided a lineup of jetson devices
• Jetson nano
• Jetson Xavier
• Jetson Orin
With this devices Nvidia has provided a custom operating system built on ubuntu. In this article I was using the jetson nano 2gb version.
The operating systems are referred to as Jetpack there are many versions of jetpack that are tailored to work with the hardware in the Jetson devices.
In my jetson 2gb I used jetpack 4.6.1 which is the latest version of Jetpack which is compatible with jetson nano and jetson nano 2GB lineup. https://developer.nvidia.com/embedded/jetson-nano-developer-kit
After you have installed the jetpack https://developer.nvidia.com/embedded/jetpack on the jetson device we can now download and install the deep stream dependencies depending on the jetson device you have.
To learn more about deep stream SDK https://developer.nvidia.com/deepstream-sdk .

Now installing deep stream.

The deep stream sdk is distributed in three main forms
• Docker containers
• Debian package
• Tar files.
All of the above containers work well with the installation but for the docker containers you have to allow special permission when using RSTP or USB cameras in the deep stream applications.
Installing deep stream
Before installing deep stream some dependencies have to be installed first this include gstreamer , kfka for message broker this feature is for when using deep stream 5.0 and above.
Installing G-Streamer
You must have internet connection for your jetson. copy and paste the following commands.

    sudo apt install \
    libssl1.1 \
    libgstreamer1.0-0 \
    gstreamer1.0-tools \
    gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-bad \
    gstreamer1.0-plugins-ugly \
    gstreamer1.0-libav \
    libgstreamer-plugins-base1.0-dev \
    libgstrtspserver-1.0-0 \
    libjansson4 \
    libyaml-cpp-dev

Installing the kafka protocol

• git clone https://github.com/edenhill/librdkafka.git

library configuration

• cd librdkafka
• git reset --hard 7101c2310341ab3f4675fc565f64f0967e135a6a
• ./configure
• make
• sudo make install

Moving the generated files to the deep stream folders

• sudo mkdir -p /opt/nvidia/deepstream/deepstream-6.1/lib

• sudo cp /usr/local/lib/librdkafka* 

/opt/nvidia/deepstream/deepstream-6.1/lib

do not copy (<>)

Installing latest NVIDIA BSP packages.

• sudo vi /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

this command opens a file that needs to be edited so that it works well with the specific jetson device you have

there are variations for the jetson programs.
• deb https://repo.download.nvidia.com/jetson/common r35.1 main
• deb https://repo.download.nvidia.com/jetson/ r35.1 main
Change the platform to be the version of the device you are using.
• deb https://repo.download.nvidia.com/jetson/common r35.1 main
• deb https://repo.download.nvidia.com/jetson/t194 r35.1 main
( my case I always found the link already in the correct format but ensure yours works well too)

Install the latest NVIDIA V4L2

• sudo apt install --reinstall nvidia-l4t-gstreamer
• sudo apt install --reinstall nvidia-l4t-multimedia
• sudo apt install --reinstall nvidia-l4t-core

now we are almost there

Installing the Deep-stream SDK

(If you want to intall a latest or previous version of deep stream then use the following link

https://docs.nvidia.com/metropolis/deepstream-archive.html )

The steps for installation are very similar.

Using the Debian package.

Download the Debian package from the Nvidia website but first ensure you are signed in.
https://developer.nvidia.com/deepstream-6.1_6.1.1-1_arm64.deb
Navigate to the folder you downloaded the Debian file and run the command.

    sudo apt-get install <<*replace with the file name you downloaded*>>

Running a sample

Navigate to the sample’s directory on the development kit. Enter the following command to run the reference application:

deepstream-app -c <path_to_config_file>