DEV Community: ConnectiveCode

MQTT vs Kafka, HTTP, HTTPS, and CoAP UDP: Optimizing IoT Device Communication

ConnectiveCode — Fri, 04 Aug 2023 10:36:08 +0000

Currently, IoT (Internet of Things) technology, which is beginning to change its terminology to IoE (Internet of Everything), is transforming the way companies manage their devices and sensorize their warehouses and offices. As for vehicle or fleet management, telematics systems have become an essential tool for monitoring and control.

To connect these vehicles, the general public is familiar with the possible hardware devices that, in some cases, are installed in the vehicles. It is even common for our vehicles to have apps with which we can control them.

The great unknown, in many cases, both for the public and for developers, is the communication protocol used.

What does this mean? It refers to how the telemetry device in our vehicle, for example, is able to travel and transmit the read information, and how we can view that information in the app on our smartphone or in a cloud service.

In this article, we will explain and explore the different features of the MQTT (Message Queuing Telemetry Transport) protocol, which is considered the communication protocol par excellence in telemetry applications in vehicles, IoT, IoE, and M2M.

We will also analyze why the use of MQTT optimizes IoT device communication compared to protocols such as Kafka, HTTP, HTTPS, or CoAP UDP.

What is MQTT?

MQTT is an application layer communication protocol based on TCP/IP. It is used for communication between IoT devices and utilizes a publish/subscribe model, where devices subscribe to a topic and only receive relevant messages from their broker.

It was developed by Andy Stanford-Clark and Arlen Nipper in 1999 and designed for data transmission in network environments with limited bandwidth and resources, such as sensors and actuators.

Advantages of MQTT over other communication protocols:

MQTT vs Kafka

MQTT and Kafka are two popular technologies for IoT device communication. Although both are used for transmitting data between devices, there are some key differences between them.

MQTT is a lightweight and efficient messaging communication protocol designed specifically for IoT and M2M devices. It uses a publish/subscribe model, meaning that devices communicate by publishing messages to a channel (topic) and other devices can subscribe to that channel to receive the published messages.

On the other hand, Kafka is a distributed messaging system used for the transmission of large volumes of data in real-time. It is an application layer communication protocol that uses TCP/IP and is used in high-performance environments.

Although Kafka is a solid solution for transmitting large volumes of data, there are several disadvantages compared to MQTT.

Firstly, Kafka is more complex to implement and configure than MQTT. Kafka requires a cluster system, while MQTT can operate on a simple network architecture.
Secondly, Kafka is not as efficient as MQTT in terms of network usage and IoT device power. Kafka continuously sends and receives data, whereas MQTT, with its subscription/publication model, only receives relevant messages for these devices.
Thirdly, in some cases, MQTT is considered more scalable than Kafka due to its decentralized architecture, which allows multiple devices and brokers to connect. This increases the processing and message distribution capacity on the network.

As previously mentioned, MQTT uses a pub/sub model. This means that devices send messages through specific channels (topics) and other devices interested in those messages subscribe to those topics. This model allows for the connection of multiple devices and brokers, increasing message processing and distribution capacity in the network.

Moreover, MQTT is a lightweight and efficient messaging protocol that consumes very few resources and bandwidth. For this reason, it is ideal for IoT devices with limited resources.

On the other hand, Kafka is a distributed data streaming platform designed to handle large volumes of data in real-time. Kafka is based on a cluster architecture, consisting of a set of Kafka brokers that are distributed and scaled horizontally to handle large amounts of data. Kafka uses a publish/subscribe model similar to MQTT, but also allows for real-time data processing through the use of data streams and the event processing API.

Although Kafka is highly scalable and can handle large volumes of real-time data, its centralized architecture can present scalability limitations when using large clusters, as is often the case in IoT device and network use cases. Increasing the number of brokers can lead to network overload and reduce overall performance.

In brief, MQTT is a more suitable option for most IoT device use cases due to its specific characteristics, such as its light weight, scalability, ease of deployment and security. Although Kafka is a more powerful technology and can handle large volumes of data, in most IoT device use cases, MQTT is a better choice due to hardware limitations and the usual low cost of these devices.

MQTT vs HTTP/HTTPS

HTTP/HTTPS (Hypertext Transfer Protocol/Secure Hypertext Transfer Protocol) are communication protocols used in IoT applications, but they have some key differences compared to MQTT in terms of performance and functionality.

HTTP/HTTPS is a web-based communication protocol designed to transmit data over the World Wide Web. It is commonly used in web and mobile applications and can be used in IoT applications to send and receive data. HTTP/HTTPS uses a request-response model, meaning devices send data at regular intervals, regardless of whether there are changes in the data.
HTTP and HTTPS are more suitable for data transmission over the web, while MQTT is more suitable for real-time data transmission between IoT devices. Therefore, in M2M or IoT device use cases, MQTT is a much more energy-efficient protocol.
In terms of scalability, HTTP and HTTPS are less scalable than MQTT. HTTP and HTTPS are designed for data exchange over the web and are not optimized for real-time data transmission between IoT devices.
On the other hand, something we haven't mentioned before is that MQTT is a highly reliable protocol. It is capable of handling unstable connections and data losses, making it a very fault-tolerant option.

MQTT vs CoAp UDP

CoAP (Constrained Application Protocol) is a communication protocol widely used in IoT products. It works over the UDP layer.

Comparing it with MQTT, we can observe certain differences:

MQTT is a much more efficient protocol than CoAP in terms of energy consumption. MQTT uses a subscription/publication model, which means that devices only communicate when new data is available. On the other hand, CoAP uses a request/response model, which means that devices have to constantly communicate to check if new data is available.
MQTT is highly scalable thanks to its subscription/publication model. This means that devices can send data to many subscribers simultaneously, which is ideal for applications that require multiple devices to participate.
MQTT is a more flexible protocol than CoAP. For example, MQTT supports multiple Quality of Service (QoS) levels, allowing devices to control the number of acknowledgements they wish to receive. In this way, we can also flexibly configure their power consumption.
MQTT offers more advanced security features than CoAP, such as end-to-end encryption and device authentication. It supports Transport Layer Security (TLS), which provides end-to-end encryption and ensures that data cannot be intercepted or tampered with.

Telemetry, MQTT and Security

Security is a major concern in any vehicle telemetry system or IoT device. These systems collect and transmit sensitive data, such as vehicle location and driving data. MQTT is a secure protocol that can be used with different levels of security, such as authentication and encryption.

In MQTT, there are different encryption methods that can be used to ensure the privacy and security of the transmitted data. The most common encryption methods in MQTT are described below:

TLS (Transport Layer Security) is a security protocol used to protect the connection between MQTT clients and brokers. TLS uses symmetric and asymmetric encryption to ensure that the data transmitted between clients and brokers is private and secure. To use TLS, it is necessary to configure certificates and encryption keys on both the broker and MQTT clients.
SSL (Secure Sockets Layer) is a security protocol similar to TLS and is used to protect the connection between MQTT clients and brokers. SSL also uses symmetric and asymmetric encryption to ensure the privacy and security of transmitted data. However, SSL is considered obsolete and has been replaced by TLS.
Certificate-based authentication is an encryption method that uses digital certificates to authenticate and authorize MQTT clients. These digital certificates serve to verify the identity of both the client and the MQTT broker before allowing connection and data exchange.
Username and password-based authentication is an encryption method that uses authentication credentials to allow access to MQTT clients. Clients must provide a username and password to connect to the MQTT broker and access transmitted data.

It is important to note that the choice of encryption method in MQTT will depend on the specific security and privacy requirements of each application, as well as the hardware used.

Hardware limits the ability of devices to perform encryption and decryption operations due to resource limitations, especially in IoT devices with limited resources such as connected vehicles.

With less processing power and memory, IoT devices may struggle to perform the encryption and decryption operations necessary to secure MQTT communication. Therefore, appropriate encryption methods compatible with the device's limited resources should be selected.

One way to reduce the load on hardware is to select less resource-intensive encryption methods, such as symmetric encryption instead of asymmetric encryption. Both TLS and SSL use both types of encryption, and while they offer a high level of security, they can be cumbersome for resource-constrained devices.

Another method to reduce the load on hardware is to use end-to-end encryption instead of encryption at the broker. End-to-end encryption means that data is encrypted at the sending device and decrypted at the receiving device, eliminating the need for the broker to perform encryption and decryption operations.

In summary, resource constraints on IoT devices imply the need to carefully select the encryption methods used in MQTT. It is important to consider both security and compatibility with the limited device resources. The security of MQTT communication should not be compromised due to lack of resources, so appropriate measures should be taken to protect the privacy and security of data transmitted on IoT devices.

MQTT & mobility communications

MQTT's ability to connect and communicate devices efficiently has led to smarter vehicle fleet management. Through its publish/subscribe model, MQTT enables vehicles and related devices to transmit and receive relevant data in real time. This has driven the implementation of mobile applications that facilitate vehicle monitoring and control from the convenience of a smartphone. From astara Connect we offer this data for any developer who has in mind to develop a mobility APP, always using the most optimized protocols for each case.

It is interesting to note that MQTT has been widely adopted in the mobility sector due to its technical advantages compared to other protocols. MQTT stands out for its efficiency, scalability and ease of implementation as we have discussed above, which makes it especially suitable for a mobility telemetry system such as astara's with which we ensure that the scalability and volume of data are safe in a system like this.

MQTT has been a key catalyst in the evolution of mobility for people and vehicles, providing a solid and secure technological infrastructure for the effective management and control of fleets in real time. At astara Connect we are developing a telemetry system based on MQTT. Its efficiency, scalability and focus on security allow us to enhance the user experience, making mobility smarter and more connected than ever before. As technology continues to advance, MQTT will continue to play a crucial role in transforming the mobility and driving landscape, facilitating a more efficient and safer future for all. And we will provide much-needed data, always using protocols that ensure the privacy and security of Connect users' data as MQTT does.

In short, in IoT applications with a large volume of devices, we can only say:

we love MQTT!

Unleash the mobility data

ConnectiveCode — Tue, 20 Jun 2023 10:47:17 +0000

Data is the present and the near future of human society. Since the world revolves around data analysis and statistics, it is no coincidence that the most in-demand jobs are data scientists or machine learning specialists. In this post, we'll show you how our data science team uses the power of data to help our customers to succeed in their businesses.

You might not have heard about us, as we are pretty new in the field. We're a company that allows clients to securely connect their vehicles and assets to know how they move.

astara Connect is a platform that acts as a neutral server storing mobility data. We digest the data coming from different providers (APIs, canbus, OEMs, OBD-II), normalize it, and keep it within our geospatial database. But one just does not simply save geospatial data and serve it.

We've created a platform that has intelligence and can interpret your vehicles' data (independently of their sources) and infer their journeys and behaviors.

So let's see how we squeeze the mobility data juice as much as possible.

We will use one of our client's data to show you how we have been helping them to improve their business decisions thanks to its data. This client, an on-demand vehicle subscription company, has hundreds of vehicles of different types: vans, cars and motorbikes, and other hubs around Spain. Henceforth, knowing how their vehicles move and work is vital for their success.

Buckle up! We're going to start talking about statistics and math.

First analysis

Let's start with the most basic analysis. The first step is visualizing the number of trips the client fleet performs during a period of time. The following figure shows our client's evolution of daily trips and trackers.

As it can be expected, the number of trips recorded per day depends linearly on the number of vehicles circulating. This conclusion can be simply deduced by observing the behavior of both overlapped plots: nº of trips and nº of vehicles connected to our platform. Our clients' fleet is wide in-vehicle varieties, including cars, motorbikes, and vans. Ergo, it may be interesting to aggregate the data by vehicle type and observe the results. The following figure groups the data by vehicle type.

This aggregate graph follows the direction of our previous hypothesis. It confirms that, for every vehicle type, the number of trips recorded in our system is linearly proportional to the number of vehicles on the move and, therefore, generating positioning data.

Looking more deeply at the above figure, the time series of the number of trips appears to have a kind of oscillatory or sawtooth wave behavior. Furthermore, the periodicity looks lower than monthly. What do you think it could be? To find out, let's make a plot that will help us understand and clarify the origin of this behavior. The graph will represent the average number of trips per day of the week for each vehicle type.

Eureka! We have found something exciting. The sawtooth wave has a weekly frequency, and the signal behaves differently for each vehicle type. Let's break it per vehicle type:

Vans travel during the working days of the week.
Motorbikes are the opposite, and they move during the weekends (including Fridays).
Cars signal follows the vans behavior but in a smoother way.

Thus, what have we learned from this quick analysis? First, considering that our client follows a subscription model with a diverse fleet, its subscribers could belong to a broad palette of businesses. With this brief study, we have been able to find out our client's business model, giving us insights about where there could be improvements. For instance, in the case of the vans, they are used primarily on working days (Monday to Friday), henceforth rented for business purposes, and more specifically for delivery or last-mile cases. On the other hand, motorcycles are focused on takeaway delivery. The graph shows that these vehicles move mostly during weekends (most takeout meals are served between Friday and Sunday). And in the case of cars, the signal tends to flatten, making the difference between weekends and weekdays smaller because they are rented for personal usage.

In summary, we have analyzed the temporal component of our client trips, but what happens with the spatial one? It is time to dig into the spatial mine and find out what is hidden inside.

The geospatial and temporal analysis

Let's begin putting all the trips on the map.

Indeed, the video is spectacular. Why? Because we used a blank map without drawing any traffic roads! Our client's vehicles have traveled the majority of Spanish highways and roads. We are trying to say that, merely with this representation of the trips, it is possible to undertake a new route map using vehicle positions from this client. Amazing!

Now, knowing where the vehicles have been, let's go back to the business of our client. For them, it could be interesting to know which cities are the most visited ones, so we can infer the most demanded places to circulate or, for example, recommend a new investment in vehicle hubs.

But we are not going to cut down the city's analysis yet. It is time to re-segment the results by vehicle type. Indeed, just by differentiating between types of vehicles, we could suggest which cars are the most interesting ones to store or park in a particular place. For example, it is clear that motorbikes are operating within the downtowns or close to the urban nucleus (Madrid in particular), so it makes no sense to have them available in hubs close to rural areas.

As a final touch, why not combine the spatial and temporal components of the trips in a single feature? For example, the following image shows you a great gif composed of heat maps from the previously analyzed data progressing through several days. This kind of representation can reveal how the business has been growing and how the influence areas are becoming more extensive.

In summary, we believe that by better understanding how your vehicles (or assets) move, we can help you grow your business, save money, and improve how your fleets move.

This study has shown one of our success cases. Simply analyzing the data of three different vehicle types with diverse business cases, we have managed to provide insights about their movement, where they go, and, more importantly, where there should be improvements for the business.

If you have a company where your business is moving vehicles, contact us, we will be more than happy to meet you!

Exploring the dynamics of mobility through the power of heat maps & python

ConnectiveCode — Wed, 26 Apr 2023 11:13:48 +0000

At Astara Connect we deal with millions of data points per month related to our tracking assets (cars, vans, bikes, and any other type of vehicle). When you have tons of data points, one way to start working is by analyzing them visually. Why? Because it will help us to see clusters, patterns, and groups of data.

So, our first goal is to display a large amount of data that is potentially complex and includes a spatial component. Such scenarios are commonly found in a variety of contexts. Technologies like eye tracking, dwell time or click behavior analysis are widely used to optimize web services leveraging data about user preferences and behavior.

Geospatial data sets reflecting traffic flow or weather phenomena are omnipresent in the news, but furthermore commonly used in planning, research, education, politics, etc. Moreover, there is a plethora of scientific fields like nanotechnology or microbiology that use heat maps to generate new insights from all varieties of data.

Figure 1. Visualizing a single car’s trajectory by plotting individual waypoints. Source: Astara Connect

A major part of geospatial information is GIS data, our main data source when we work with traffic and mobility-related phenomena. Which approach would we choose to visualize them? One could start by simply plotting all data points. In Figure 1 we plot the trajectory of a single car during a typical morning. While this approach is straightforward and useful for small data sets, it quickly becomes chaotic and computationally inefficient with increasing data set size.

Consider, for example, the traffic of a large vehicle fleet. Then we speak about an order of magnitude of 1 million data points - per day. Let’s look at what happens during one morning in Madrid (Figure 2).

Figure 2. Visualizing the trajectories of an entire car pool by plotting individual waypoints. Source: Astara Connect

Such an image is of little use. Its calculation and rendering are computationally expensive, and we won’t obtain many insights from it. In most cases, it is actually not necessary to plot all points. There are likely much more elegant ways of visualization.

Displaying all points vs. Displaying a Heat Map

Prior to the advent of computers, professionals who worked with numerical data, such as statisticians and accountants, had to find ways to cope with the challenge of connecting a vast number of values. One method involved color-shading the key values in tables and matrices and then piecing them together to identify distinct patterns. Heat maps evolved from this long-standing practice of matrix displays (Wilkinson & Friendly, 2008).

Fun fact: Many of us may actually remember how a similar visualization technique was used by the alien in the movie Predator while hunting Arnold Schwarzenegger (Figure 3). While we at Astara Connect have much friendlier intentions when studying vehicles and their drivers, we can still learn a lot from the alien's trick. By the way, the movie was released in 1987 - an astonishing 36 years ago!

Figure 3. Alien using thermal vision to hunt its prey in “Predator” (1987). Source: georgeromeros.tumblr.com.

Returning to data science, heat maps generated by infrared cameras and similar technologies are thermograms. They display the intensity of the infrared radiation emitted from an object or a person. The way they are plotted allows us to immediately understand the entire temperature distribution pattern. From this pattern we can obtain insights such as the location of heat spots, like Arnold Schwarzenegger in a forest, but also for example thermal losses in building walls.

While the heat map's historical origin is actually unrelated to heat, it is the most intuitive example, and hence we will use this name. We can abstract the concepts and use them for displaying many other data sets.

Technical background

Mathematically speaking, a spatial heat map is a color-coded 2D density plot. There are two different main classes of heat maps: grid and spatial heat maps.

A grid heat map displays the color variations across a predefined 2D matrix. An example is the above thermograph (Figure 3), in which each pixel is associated with a single temperature point.

On the other hand, a spatial heat map does not use a predefined grid. Instead, the data points themselves define where the color spots are placed across a continuous 2D surface. The click heat map is a clear example of this type (Figure 4).

Figure 4. Click heat map showing where Wikipedia users tend to click. Source: Wikipedia CC BY-SA 3.0.

Conceptually, the creation of such a plot consists of two main steps:

First, every data point is associated with a specific location, for example, an (x,y) location on a screen or a pair of GPS coordinates. Around this location, the data point radiates a small amount of intensity in form of a numeric value. These values are totaled together across all data points, creating a continuous map of intensity.

In the second step, the numeric values are displayed with the help of a color map. Often, one chooses color gradients associated with heat, from blue (cold) to red (hot). However, the color should be chosen to display the data best and correctly.

In principle, we can show the same information using both a grid heat map and a spatial heat map, but depending on the context and the way of data collection, one or the other is likely to be more convenient. For example, a digital photosensor collects data across a grid, while in traffic data sets the position is part of the data point. With traffic data being the core of Astara Connect, in the following lines, we will focus on spatial heat maps.

Creating spatial heat maps in Python

Astara Connect's backend is written in Python 3. This gives us access to a huge pool of powerful data science libraries. Besides popular packages like Pandas and Numpy, there are many specialized ones for GIS data, like Shapely or Geopandas.

For heat maps, there are different Python packages available which greatly simplify their creation and customization. While popular visualization packages like Matplotlib and Plotly also include heat map functionalities, there are some dedicated ones for such tasks, like Seaborn and Folium. Seaborn is a great choice when it comes to grid heat maps whereas the focus of Folium lies on spatial heat maps. In the following, we will work with spatial heat maps and hence have a closer look at Folium.

Folium is a Python wrapper for Leaflet, a widely used JavaScript library for creating versatile interactive maps.

Folium contains wrappers for different leaflet plugins, which we can use to create heat maps of various types. Let’s start with a static Heat map.

At Astara Connect, and generally in the context of mobility, we deal with large amounts of traffic data that we want to analyze to generate insights. So, to start, let’s plot the movements of a single car.

We will need the GPS data (latitude, longitude): 
data_single_car = 
[[37.876495, -5.620123], [37.875622, -5.618808],
...,
[37.842555, -5.762402], [37.842592, -5.762413]]

Create a map base layer:

import folium 

map_h = folium.Map(
    location=location,
    tiles="CartoDB positron",
    zoom_start= 12,
    min_zoom = 6,
    max_zoom = 18)

Add a heat map layer to the map:

from folium.plugins import HeatMap

HeatMap(data_single_car, 
        radius=12, 
        blur=20, 
        min_opacity=.2, 
        gradient = gradient
        ).add_to(map_h)

Save the map:

hm_name="Heat_Map_Astara_Connect.html"
map_h.save(hm_name)

The above code basically transforms the geospatial information (GPS location and point density) into a heat map.

Figure 5. Visualizing a single car’s trajectory using a Folium heat map. Source: Astara Connect

First of all, one can clearly identify the route taken by the vehicle. Additionally, for each spot along the route, the color indicates the local data point density. In a first approximation, this data point density is proportional to the vehicle dwell time. This means that the color reflects how much time vehicles spend in certain areas. In this example, “colder” color tones (blue) reflect brief visits, like taking a road once or twice. The longer or the more frequently a site was visited, like when searching for the last available parking spot within a city center, the color becomes gradually more intense and warmer (eventually yellow and red). Strictly speaking, there are other effects that can cause local or temporal variations in data point density. For example, GPS signal strength can be affected by obstacles such as tunnels or vegetation, which have to be taken into account accordingly.

Such a map lets us immediately identify activity hot spots without plotting all individual paths. Zooming into one of these hot spots still reveals the individual paths that altogether produce the red color (see inset). The color coding allows us to grasp an extra bit of information on a large scale; otherwise, we wouldn’t be able to see all the details. In this case, we have been looking at a single vehicle’s trajectory containing about 6000 data points. Real traffic, however, is created by the combination of hundreds or thousands of vehicles at the same time.

So let’s repeat the same procedure for about 2 million points and explore the full potential of heat maps.

Figure 6. Visualizing the trajectories of an entire car pool during one week using a Folium heat map. Source: Astara Connect

We obtain a complete heat map while still being lightweight and interactive. Keep in mind that plotting this data set point by point would create a 2 GB file! This enables us to extend the analysis to data from entire fleets collected over more extensive periods. The more data, the better.

Folium can create leaflet maps of various types using different freely available tiles. In this way, we can highlight geographical or urban features and blend in one or multiple of our data sets. Depending on the specific use case, this helps us to identify correlations between hot spots of traffic-related metrics (for example, traffic density, traffic flow, local emissions, or air quality) and natural or urban features, such as road planning, construction sites, environmental zones, etc.

Figure 7. Creating Folium heat maps with different map styles. Source: Astara Connect

Dynamic Heat Maps

While static heat maps are already very useful and will cover many of our needs, we can get the most out of our data by leveraging some of the advanced plug-ins that come with Folium.

Many data sets have a temporal component. In the context of mobility, we are confronted with different temporal variations, each with a characteristic time scale. For example, during a winter commuting trip, the drivers’ behavior will correlate with local weather conditions. Throughout the day, traffic density will reach its peak at rush hours when people are commuting and will be lower at night times. Throughout the year, overall traffic flow will be lower around Christmas but can be locally higher when an important highway is closed for reconstruction. Over even more extended periods, traffic noise can correlate with the availability of electric charge stations. Nevertheless, authorities can now leverage mobility data analytics to make data-driven decisions in real-time. For instance, they can opt to implement temporary measures such as opening a lane in the opposite direction or determining the direction of a High-Occupancy Vehicle (HOV) lane, to effectively address any issues that may arise.

By adding a temporal component to heat maps, we could study these phenomena's spatial and temporal aspects. Folium comes with a plugin called Heat MapWithTime, which combines the previously described spatial heat map functionalities with temporal data.

The procedure for creating such dynamic heat maps is brief as follows.

We first separate the data points into certain time intervals in order to discretize the data for display. Vehicle positioning data is usually sampled at a data rate of up to several points per minute, but we will display the heat map evolution at a lower frame rate. We will see that playing with this will allow us to focus on different time scales.

In the second step, the data is displayed using the Folium plug-in for dynamic heat maps.

To start, we will create a heat map that covers one week of traffic data. We will sample this data set once each 24 h. In this way, we can observe the evolution of daily traffic throughout the weekdays.

Like in the previous case, we start with the GPS data: 
data_fleet = 
[40.388369, -3.67176], [40.388361, -3.67178], [40.424472, -3.726995],
...,
[37.383244, -6.062695], [40.29323, -3.979344], [36.511531, -4.635684]]

In addition, we will need the corresponding datetimes for creating the time index: 
data_fleet_ts = 
[2022-11-28 00:00:04+00:00, 2022-11-28 00:00:06+00:00, 2022-11-28 00:00:11+00:00, 
..., 
2022-12-04 23:59:36+00:00, 2022-12-04 23:59:38+00:00,2022-12-04 23:59:56+00:00]

Set parameters on how to downsample the data.
week_days = 7
interval_sec = 86400
total_time = 86400*week_days

Create the time index:

import numpy as np

time_index = []
for _time in np.arange(interval_sec, total_time+interval_sec, interval_sec):
    time_index.append(_time)

Transform datetime entries to relative timestamp and allocate them to index slots

import datetime
import math

geoms_ts_list = []
geoms_ts_sample_ids = []
ref_ts = data_fleet_ts.timestamp_UTC[0].timestamp()

for index, row in data_fleet_ts.iterrows():
    row_ts = data_fleet_ts.timestamp_UTC[index].timestamp()-ref_ts
    geoms_ts_list.append(row_ts)
    geoms_ts_sample_ids.append(math.floor(row_ts/interval_sec))


Create downsampled data set:

current_sample_id = 0
geoms_time_plot = []
geoms_time_plot_sample = []
weight = 0.05
weight *= 120/interval_sec

for idx, sample_id in enumerate(geoms_ts_sample_ids):
    if sample_id == current_sample_id:
        geoms_time_plot_sample.append([data_fleet[idx][0], data_fleet[idx][1], weight])
    elif sample_id == current_sample_id+1:
        geoms_time_plot.append(geoms_time_plot_sample)
        geoms_time_plot_sample = [[data_fleet[idx][0], data_fleet[idx][1], weight]]
        current_sample_id += 1
    if idx == len(geoms_ts_sample_ids)-1:    
        geoms_time_plot.append(geoms_time_plot_sample)
    if geoms_ts_list[idx] >= total_time:    
         break

Create heat map:

Transform timestamps in a human readable time:
time_index_display = [str(datetime.datetime.fromtimestamp(_time+ref_ts-interval_sec)) for _time in time_index] 

Compute heat map

import folium

map_time = folium.Map(
    location=[40.4,-3.7],
    tiles="CartoDB positron",
    zoom_start= 6,
    control_scale=True
)

from folium.plugins import HeatMapWithTime

HeatMapWithTime(data=geoms_time_plot, 
                index = time_index_display,
                auto_play = True, 
                max_speed = 10, 
                radius = 12, 
                blur = .8,
                min_opacity = 0,
                max_opacity = .8,
                ).add_to(map_time)

Save the map:

name = "HeatMapWithTime_AstaraConnect.html"
map_time.save(name)

Figure 8. Dynamic Folium heat map showing a car pool’s daily traffic throughout the week. Source: Astara Connect

The animation covers an entire week from Monday to Sunday. In the beginning, we can observe a red hot spot in the Madrid city center. During the last two days, the weekend, the red spot disappears.

Let’s have a look at a second example. By adjusting the downsampling parameters, we can play with the time scale of the heat map. In order to visualize traffic evolution during one day, we set them in the following way:

Set parameters how to downsample the data.
week_days = 1
interval_sec = 600
total_time = 86400*week_days

Figure 9. Dynamic Folium heat map showing a car pool’s traffic throughout a day. Source: Astara Connect*

This example shows the movement of plenty of vehicles on Monday. One can identify different spots of activity and how they change throughout the day.

With these two examples, we can see how heat maps can be used to visualize different aspects of the same data set. As always, there are many more options to customize these heat maps. The Folium documentation contains more information.

And, last but not least, there are other Python packages to be explored!

What can we get from mobility heat maps?

Mobility is a field that both produces large amounts of data and benefits from the insights obtained from them. Exploring correlations between traffic-related metrics, such as traffic flow, contamination levels, local emissions, and local fuel consumption, and external factors like urban or geographic features, weather conditions, or administrative measures requires the right tools.

Heat maps are a great way to visualize geospatial data sets. They condense many data points down to a color-coded map, which is intuitive to understand and lightweight for convenient handling.

Making use of heat maps and other tools, we can draw data-driven conclusions for optimizing routing and driver experience while at the same time minimizing CO2 emissions and the costs of mobility.

Article generated by Astara Connect tech team.