DEV Community: Carlos Lemus

The Four Stages of IoT - The Data Processing Stage

Carlos Lemus — Fri, 14 Aug 2020 16:17:32 +0000

This is the third post in the series The Four Stages of IoT: A Data-Centric Taxonomy of the IoT and it was originally posted on my website's blog.

Previously, we discussed the connectivity stage, which explored how devices establish a connection with edge devices or the ecosystem backend. Today, we will go one layer deeper and focus on what we do with the data after establishing a reliable connection. I refer to this as the Data Processing Stage.

Data processing can happen in the back end or at the edge, and it involves performing work on the data such as filtering, transformations, enrichment, or analytics.

Types of Data Processing

Data Filtering deals with monitoring device data and accepting or rejecting the entire message or parts of it based on predefined criteria.
For example, adding and removing fields from legacy payload formats to standardize them with newer formats
Data Transformations deals with transformation and manipulation tasks to standardize data and improve readability. For example, expanding the "C" in a temperature field's unit to "Celsius"
Data Enrichment deals with adding contextual data to device messages. For example, appending a customer ID to the current message, based on the device ID.
Data Analytics deals with tasks such as deriving insights from the data in real time and interacting with Machine Learning models. This is perhaps the most critical component of a data-centric IoT architecture For example, deriving a Key Performance Indicator (KPI) based on IoT device data to understand equipment efficiency or executing a preventive maintenance Machine Learning model to address breakage before it happens.

Methods of Data Processing

Although data processing applications can live in physical servers, they are more common on edge computing devices or in the form of cloud-based Serverless Functions, Container Deployments, or Virtual Machines.

Edge Devices

Because of their powerful hardware, Edge devices can typically perform most types of data processing. It is common to see ML models hosted on the Edge device for immediate decision making about how it ought to manage local IoT Devices.

For example, an Edge Device could discard duplicate messages (filtering), attach building information before sending the message to the backend (enrichment), and using an ML model to alert local operators of a device in need of repair (analytics).

Virtual Machines and Servers

Data Processing applications can be hosted as a long-running job inside of virtual machines and servers. This approach is particularly useful when a specific stack not available in other compute platforms is required. For example, if you are required to use legacy applications running on legacy hardware or software, you might be limited to a server or virtual machine.

Containers

Container deployments remove much of the operational burden of maintaining a server, and with managed services like Google Cloud Run or AWS Fargate, it is also possible to only run the application when it is needed—such as when a device message is received.

Containers are preferable over Serverless Functions when large volumes of data need to be processed or when a specific software stack is not available on a Serverless Function platform.

For example, if you were ingesting tens-of-thousands of messages with software requiring disk access, Containers would be an optimal choice. That's because they are easily scalable and configurable, and they are not tightly-coupled with any hardware platform nor operating system. However, this could have a slight added cost of operational management in the form of cluster configuration and container image management.

Serverless Functions

Serverless Functions provide the most hands-off experience in infrastructure management, and they are most apt for sporadic data processing. Writing a Serverless Function is as simple as selecting an environment, writing your application, and deploying it to the cloud.

Furthermore, cloud providers typically bill Functions per-execution, which can make the financial cost of data processing very close to 0 for low-volume workloads.

For example, imagine you have two developers assigned to build an application that processes incoming data for 1000 devices every second. Because the team is small, you want to avoid them as much of the infrastructure heavy-lifting as you can. Serverless Functions would achieve this by abstracting away most of the underlying infrastructure, and it would enable the pair to focus primarily on the code dealing with business and data processing logic. Moreover, the billing cost of running such a function would be roughly $0.30/month for processing data from all 1000 devices each second of the year!¹

An Example Data Processing Application

The following diagram showcases an application which uses an AWS Lambda function to perform all the data processing activities discussed in this post:

Conclusion

The Data Processing Stage provides the applications used to filter, transform, enrich, and analyze data from IoT devices. These applications can live on the Edge device, or server-side as Virtual Machines, container deployments, or serverless functions.

In the next post, we will explore the Data Storage stage, which consists of the file systems, databases, object stores, and "data lakes" that temporarily or permanently hold data about IoT ecosystems.

¹ The assumption here is based on ~2.7 million executions of AWS Lambda using the following configuration: 128MB, 1 second executions, free tier included.

The Four Stages of IoT: The Connectivity Stage

Carlos Lemus — Sun, 26 Jul 2020 20:54:30 +0000

This is the second post in the series The Four Stages of IoT: A Data-Centric Taxonomy of the IoT, and it was originally posted on my website's blog.

In my previous post, I discussed the idea of dividing IoT Architectures into four stages: connectivity, processing, storage, and data access. Today we will explore the Connectivity Stage which is the first step in the journey towards value-generation from IoT Device Data. This stage is composed of the mechanisms whose purpose is to facilitate communications to and from IoT devices: these include services, applications, and communication protocols.

We will now see what some of these mechanisms are and how they are leveraged to stream data to and from backend applications. We will accomplish this by breaking the Connectivity Stage down into three main components: the Field IoT Device, the Edge Device, and the Ecosystem Backend.

THE FIELD IOT DEVICE

"IoT devices" are those collecting data about an environment and acting in response to commands and notifications from a Server: a smart thermostat, a weather station, or a current meter can all be examples of IoT devices.

In this instance, we define a "Server" as any entity which consumes IoT Device Data and pushes notifications and commands back to the device: this could be an actual server, edge device, or a cloud provider.

Connectivity at the Field IoT Device

IoT devices can connect to Servers in one of three ways.

Some devices simply communicate directly with the server through popular protocols like LTE or Wi-Fi. Cellular-enabled fleet tracking systems like those put on cargo trucks are an example of this category of devices, which have enough CPU, memory, and hardware capabilities to transmit data through the Internet independently.

Other devices must first go through an edge device. If you wear a fitness tracker that syncs with your phone and whose data gets uploaded to the cloud, then you have an IoT and Edge device combo (your watch and phone, respectively). Edge Devices can help with local data processing and with saving on data transfer costs through local buffering. They can also serve as a protocol intermediary to assist resource-constrained devices by receiving data in low-power protocols like Bluetooth Low Energy (BLE) and relaying it to the backend over Wi-Fi, Cellular, or Ethernet.

Yet other IoT devices must first communicate with other IoT devices in order to relay their messages to the closest Edge device. For example, Smart Lighting systems often relay their data from light bulb to light bulb before reaching a hub device. Ad-hoc and mesh network protocols are the primary choices for enabling this type of communication.

Data Activity for IoT devices

Once the IoT Field Device is connected, data can be sent, received, and stored as illustrated in the table below.

Data Activity	Description and Examples
Sending Data	Devices send data such as sensor measurements, execution reports, and error logs to a Server.
Receiving Data	Devices subscribe to data from the Server like firmware update commands and the status of job executions through messaging systems MQTT.
Keeping State (available memory, error count, etc.)	A device reports metrics to the Server and relies on it to store them, analyze them, and react to them by pushing commands back to the device.

THE EDGE DEVICE

The role of Edge devices is to collect data from IoT devices and to relay that data to the ecosystem backend. As mentioned above, these devices can come in the form of Smart Lighting hubs, mobile phones, voice assistants, and more.

Connectivity at the Edge Device

Edge Devices use standard IT protocols such as Ethernet, Wi-Fi, or cellular to connect to the backend. Furthermore, they often support low-power protocols such as BLE, ZigBee, and Z-Wave to connect to resource-constrained IoT devices.

Edge Device Software such as AWS IoT Greengrass, Azure IoT Edge, and EdgeX Foundry provide frameworks to connect to IoT devices, to process their data locally, to respond to events at the edge, and finally, to pass the data on to the backend.

Data Activity for Edge Devices

When a Field IoT Device connects to the Edge Device, data can be sent, received, and stored as illustrated in the table below.

Data Activity	Example Data Flow
Transferring data to/from IoT devices	IoT devices use an Edge Device as their Server and send it sensor readings and system logs. The Edge Device in turn may pass this data on to the ecosystem backend.
Keeping State (available memory, error count, etc.)	The Edge device can store a limited amount of state for both itself and the IoT devices which it communicates with. Typically, enough history is kept here to enable local decision making. For example, the Edge Device could increase the speed of a fan after identifying a high temperature excess.

THE ECOSYSTEM BACKEND

Finally we arrive at the Ecosystem Backend where the data is analyzed, processed, stored, and made available in different formats to business intelligence and data analytics applications. The Backend can run on on-premise servers or in cloud platforms such as Amazon Web Services.

Connectivity at Ecosystem Backend

IoT and Edge Devices can both communicate with the Backend.

MQTT is a common protocol for communicating to both Field and Edge devices in solutions like AWS IoT and Azure IoT. Furthermore, in these cloud-based solutions, HTTP is available for applications which require a RESTful interface and WebSockets for integration with real-time web applications.

Data Activity for the Ecosystem Backend

When either a Field IoT Device or an Edge Device connect to the Ecosystem backend, data can be sent, received, and stored as illustrated in the table below.

Data Activity	Description and Examples
Transferring data to/from IoT devices	Devices with sufficient hardware capabilities can send reports, logs, and alerts directly to the Backend over a robust mechanism such as Ethernet.
Transferring data to/from Edge Devices	Edge Devices relay data from IoT devices to the backend, and they also sync up device states, system logs, and edge decision logic.
Keeping State (available memory, error count, etc.)	The Ecosystem Backend is the primary location for storing device state and for applying rules to respond to state events. History of device state is also regularly kept here for auditing and anomaly detection purposes.

Conclusion

The Connectivity Stage provides the means to load data from IoT devices into the Ecosystem backend where it can be analyzed and turned into actionable insights. We reviewed some mechanisms–including services, applications, and communication protocols–that are used to carry out this extraction throughout the entire Connectivity Stage: the Field IoT Device, the Edge Device, and the Ecosystem Backend.

To finalize, here is a diagram depicting an example IoT Architecture which uses several methods for connectivity to transfer different types of data:

In the next post, I will review the Data Processing Stage, which is where data is enhanced with contextual information and analyzed for patterns over time.

The Four Stages of IoT: A Data-Centric Taxonomy of the IoT

Carlos Lemus — Sun, 12 Jul 2020 15:12:25 +0000

This article was originally posted on my website's blog.

The Internet of Things (IoT) isn’t new. It is now a subject with a myriad of books written about it, new exciting college electives, and a generous assortment of products that have spectacularly succeeded or failed. As such, members of the tech community naturally have devised comprehensive, high-level taxonomies of IoT ecosystems.

These usually focus on smart devices and how they, when connected to a network or the Internet, can publish their data to a remote server or cloud provider. What happens to the data from there on is often reduced to a nondescript label of “data analytics.” However, the modern value of IoT is not merely remote data acquisition (we’ve had that for decades): it is the availability of insights about a massive amount of data over an extended time period.

Because of that, in this blog series I'm titling "The Four Stages of IoT," I will advocate for a data-centric taxonomy of the IoT. I will explore how edge and cloud computing are used in tandem today to extract large-scale, powerful, value-generating insights. With data as the epicenter, I will show you, through examples, how to dissect and understand an IoT ecosystem in four stages: connectivity, processing, storage, and access.