DEV Community: EMQ Technologies

How to Use MQTT in C# with MQTTnet

EMQ Technologies — Thu, 24 Oct 2024 08:25:46 +0000

Introduction

With the rise of IoT, the .Net framework has become increasingly popular in building IoT applications. Microsoft's .Net Core and .Net Framework provide developers with a set of tools and libraries to build IoT applications that can run on Raspberry Pi, HummingBoard, BeagleBoard, Pine A64, and more.

MQTTnet is a high-performance .Net library that implements the MQTT protocol. It is open source on GitHub and has a rich set of features, including MQTT 5.0 protocol and TLS/SSL supports.

This blog post demonstrates how to use the MQTTnet library to connect to a serverless MQTT broker. The whole project can be downloaded at MQTT Client Examples.

Prepare an MQTT Broker

EMQX Serverless is an MQTT broker offering on the public cloud with all the serverless advantages. You can start the Serverless deployment in seconds with just a few clicks. Additionally, users can get 1 million free session minutes every month, sufficient for 23 devices to be online for a whole month, making it perfect for tiny IoT test scenarios.

You can follow the guide in this blog to create a serverless deployment for free. Once you have completed the registration process with the online guide, you will get a running instance with the following similar information from the “Overview” in your deployment. We will use the connection information and CA certificate later.

MQTT C# Demo

1. Install .Net and Visual Studio

If you haven't installed the .NET environment on your computer yet, you can visit the official Microsoft documentation for detailed instructions.

Visual Studio is a comprehensive IDE for .NET developers that provides a feature-rich environment for developing, debugging, and deploying .NET applications. You can download and install it here, based on your computer's system and version.

2. Install the MQTTnet package

MQTTnet is delivered via NuGet package manager. To install it, create a Console Application and use NuGet to install the MQTTnet package. For detailed instructions on using NuGet in Visual Studio, refer to the official documentation. If you're using Visual Studio for Mac, refer to install and manage NuGet packages in Visual Studio for Mac.

3. Set up the MQTT connection

To connect to the EMQX Serverless broker, you need to create an instance of the MqttClientOptionsBuilder class and set the necessary options like broker address, port, username, and password. The code snippet below demonstrates how to create an instance of the MqttClientOptionsBuilder:

        string broker = "******.emqxsl.com";
        int port = 8883;
        string clientId = Guid.NewGuid().ToString();
        string topic = "Csharp/mqtt";
        string username = "emqxtest";
        string password = "******";

        // Create a MQTT client factory
        var factory = new MqttFactory();

        // Create a MQTT client instance
        var mqttClient = factory.CreateMqttClient();

        // Create MQTT client options
        var options = new MqttClientOptionsBuilder()
            .WithTcpServer(broker, port) // MQTT broker address and port
            .WithCredentials(username, password) // Set username and password
            .WithClientId(clientId)
            .WithCleanSession()
            .Build();

Please replace the connection parameters with your EMQX connection information and login credentials.

Broker and port: Obtain the connection address and port information from the server deployment overview page.
Topic: Topics are used to identify and differentiate between different messages, forming the basis of MQTT message routing.
Client ID: Every MQTT client must have a unique client ID. You can use Guid.NewGuid() to generate a new unique identifier in .NET.
Username and password: To establish a client connection, please make sure that you provide the correct username and password. The following image shows how to configure these credentials under 'Authentication & ACL - Authentication' on the server side.

4. Using TLS/SSL

When connecting to EMQX Serverless, it is important to note that it relies on a multi-tenant architecture, which enables multiple users to share a single EMQX cluster. In order to ensure the security and reliability of data transmission within this multi-tenant environment, TLS is required. And if the server is utilizing a self-signed certificate, you must download the corresponding CA file from the deployment overview panel and provide it during the connection setup process.

To add TLS and set the certificate file to the MqttClientOptionsBuilder instance, you can use WithTls(). The following code snippet shows how to create a TLS instance of MqttClientOptionsBuilder:

        string broker = "******.emqxsl.com";
        int port = 8883;
        string clientId = Guid.NewGuid().ToString();
        string topic = "Csharp/mqtt";
        string username = "emqxtest";
        string password = "******";

        // Create a MQTT client factory
        var factory = new MqttFactory();

        // Create a MQTT client instance
        var mqttClient = factory.CreateMqttClient();

        // Create MQTT client options
        var options = new MqttClientOptionsBuilder()
            .WithTcpServer(broker, port) // MQTT broker address and port
            .WithCredentials(username, password) // Set username and password
            .WithClientId(clientId)
            .WithCleanSession()
            .WithTls(
                o =>
                {
                    // The used public broker sometimes has invalid certificates. This sample accepts all
                    // certificates. This should not be used in live environments.
                    o.CertificateValidationHandler = _ => true;

                    // The default value is determined by the OS. Set manually to force version.
                    o.SslProtocol = SslProtocols.Tls12; ;

                    // Please provide the file path of your certificate file. The current directory is /bin.
                    var certificate = new X509Certificate("/opt/emqxsl-ca.crt", "");
                    o.Certificates = new List<X509Certificate> { certificate };
                }
            )
            .Build();

5. Connect to the MQTT broker

Now that you have created the MQTT client and set up the connection options, you are ready to connect to the broker. Simply use the PublishAsync method of the MQTT client to establish a connection and start sending and receiving messages.

var connectResult = await mqttClient.ConnectAsync(options);

Here we use asynchronous programming, which allows message publishing while subscribing to prevent blocking.

6. Subscribe to topics

Once connected to the broker, you can verify the success of the connection by checking the value of ResultCode. If the connection is successful, you can subscribe to MQTT topics to receive messages.

if (connectResult.ResultCode == MqttClientConnectResultCode.Success)
        {
            Console.WriteLine("Connected to MQTT broker successfully.");

            // Subscribe to a topic
            await mqttClient.SubscribeAsync(topic);

            // Callback function when a message is received
            mqttClient.ApplicationMessageReceivedAsync += e =>
            {
                Console.WriteLine($"Received message: {Encoding.UTF8.GetString(e.ApplicationMessage.PayloadSegment)}");
                return Task.CompletedTask;
            };

Within this function, you can also print the the corresponding received messages. This allows you to view and process the received data as needed.

7. Publish messages

To send messages to the broker, use the PublishAsync method of the MQTT client. Here is an example for sending messages to the broker in a loop, with one message sent every second:

for (int i = 0; i < 10; i++)
            {
                var message = new MqttApplicationMessageBuilder()
                    .WithTopic(topic)
                    .WithPayload($"Hello, MQTT! Message number {i}")
                    .WithQualityOfServiceLevel(MqttQualityOfServiceLevel.AtLeastOnce)
                    .WithRetainFlag()
                    .Build();

                await mqttClient.PublishAsync(message);
                await Task.Delay(1000); // Wait for 1 second
            }

8. Unsubscribe

To unsubscribe, call:

await mqttClient.UnsubscribeAsync(topic);

9. Disconnect

To disconnect, call:

await mqttClient.DisconnectAsync();

Complete Code

The following code shows how to connect to the server, subscribe to topics, and publish and receive messages. For a complete demonstration of all functions, see the project's GitHub repository.

using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using MQTTnet;
using MQTTnet.Client;
using MQTTnet.Protocol;

class Program
{
    static async Task Main(string[] args)
    {
        string broker = '******.emqxsl.com';
        int port = 8883;
        string clientId = Guid.NewGuid().ToString();
        string topic = "Csharp/mqtt";
        string username = 'emqxtest';
        string password = '**********';

        // Create a MQTT client factory
        var factory = new MqttFactory();

        // Create a MQTT client instance
        var mqttClient = factory.CreateMqttClient();

        // Create MQTT client options
        var options = new MqttClientOptionsBuilder()
            .WithTcpServer(broker, port) // MQTT broker address and port
            .WithCredentials(username, password) // Set username and password
            .WithClientId(clientId)
            .WithCleanSession()
            .WithTls(
                o =>
                {
                    // The used public broker sometimes has invalid certificates. This sample accepts all
                    // certificates. This should not be used in live environments.
                    o.CertificateValidationHandler = _ => true;

                    // The default value is determined by the OS. Set manually to force version.
                    o.SslProtocol = SslProtocols.Tls12; 

                    // Please provide the file path of your certificate file. The current directory is /bin.
                    var certificate = new X509Certificate("/opt/emqxsl-ca.crt", "");
                    o.Certificates = new List<X509Certificate> { certificate };
                }
            )
            .Build();

        // Connect to MQTT broker
        var connectResult = await mqttClient.ConnectAsync(options);

        if (connectResult.ResultCode == MqttClientConnectResultCode.Success)
        {
            Console.WriteLine("Connected to MQTT broker successfully.");

            // Subscribe to a topic
            await mqttClient.SubscribeAsync(topic);

            // Callback function when a message is received
            mqttClient.ApplicationMessageReceivedAsync += e =>
            {
                Console.WriteLine($"Received message: {Encoding.UTF8.GetString(e.ApplicationMessage.PayloadSegment)}");
                return Task.CompletedTask;
            };

            // Publish a message 10 times
            for (int i = 0; i < 10; i++)
            {
                var message = new MqttApplicationMessageBuilder()
                    .WithTopic(topic)
                    .WithPayload($"Hello, MQTT! Message number {i}")
                    .WithQualityOfServiceLevel(MqttQualityOfServiceLevel.AtLeastOnce)
                    .WithRetainFlag()
                    .Build();

                await mqttClient.PublishAsync(message);
                await Task.Delay(1000); // Wait for 1 second
            }

            // Unsubscribe and disconnect
            await mqttClient.UnsubscribeAsync(topic);
            await mqttClient.DisconnectAsync();
        }
        else
        {
            Console.WriteLine($"Failed to connect to MQTT broker: {connectResult.ResultCode}");
        }
    }
}

Test

Run the project in Visual Studio, we can see the output information on the terminal window as follows. The client has successfully connected to the MQTT broker, and received a message every second.

You can also use MQTT Client Tool - MQTTX as another client for the message publishing and receiving the test. If you subscribe the “Csharp/mqtt" topic in MQTTX, you will receive the message every second.

When you publish a message to the topic, the server will receive the message and you can view it both on MQTTX and in the console.

Received message displayed on MQTTX

Received message displayed on terminal

Summary

This blog provides a step-by-step guide on connecting to a serverless MQTT deployment via the MQTTnet library. By following these instructions, you have successfully created a .Net application capable of publishing and subscribing to Serverless MQTT.

Originally published at wwww.emqx.com.

Running EMQX MQTT Broker on Docker (Bonus: Clustering and Persistence)

EMQ Technologies — Wed, 23 Oct 2024 07:18:26 +0000

What Is MQTT on Docker?

MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol designed for low-bandwidth, high-latency networks. Docker is an open-source platform that allows you to automate the deployment, scaling, and management of applications within containers. So, when you run MQTT on Docker, you are deploying the MQTT broker within a Docker container.

This combination provides a scalable and reliable solution to handle data streams in real-time, particularly in IoT applications. The MQTT broker receives messages from publishers (devices or applications that produce data) and dispatches them to subscribers (devices or applications that consume data). By running the broker on Docker, you can make it easier to deploy and scale your MQTT broker.

We’ll show how to deploy MQTT on Docker with EMQX, a popular, open source MQTT broker. EMQX provides powerful capabilities not available in other open source brokers, such as clustering, persistence, and support for very large scale deployments.

Why Should You Run an MQTT Broker on Docker?

Running an MQTT broker on Docker provides several advantages.

Rapid deployment: Docker containers can be started in a matter of seconds, which means your MQTT broker can be up and running quickly.
Isolation: Docker containers run in isolation from each other, ensuring that the broker's processes do not interfere with each other or with the host system. This lets you run multiple instances of the MQTT broker on the same host without any conflicts.
Scalability: Docker allows you to scale your MQTT broker horizontally (by adding more containers) or vertically (by adding more resources to a container). This scalability allows you to handle larger data or transaction volumes. You can automate scalability for large deployments with an orchestrator like Kubernetes (read our guide to MQTT with Kubernetes).
Portability: Docker containers can run on any system that has Docker installed, regardless of the underlying operating system. This means you can build your MQTT broker on your local system, test it, and then deploy it reliably on any cloud or on-premise server.

Setting Up an MQTT Broker in Docker with Clustering and Persistence

EMQX is the leading open source MQTT broker. It provides a Docker Official Image which is available on Docker Hub.

A major advantage of EMQX is that it supports clustering for large scale MQTT deployments. Most other articles on this topic show how to set up the Mosquitto MQTT broker with Docker - but Mosquitto does not support clustering. Additional EMQX features include:

Ability to scale up to 100M+ IoT devices in 1 cluster, while maintaining 1M message per second throughput and sub-millisecond latency.
100% compliant with MQTT 5.0 and 3.x, support for multiple open standard protocols like HTTP, QUIC, and WebSocket.
Secures bi-directional communication with MQTT over TLS/SSL and various authentication mechanisms.
Uses powerful SQL-based rules engine to extract, filter, enrich and transform IoT data in real-time.
Ensures high availability and horizontal scalability with a masterless distributed architecture.
More than 20K+ enterprise users across 50+ countries and regions, connecting 100M+ IoT devices worldwide. Trusted by over 400 customers in mission-critical scenarios including over 70 Fortune 500 companies.

Here is how to use the EMQX Docker image to quickly get up and running with MQTT.

Install Docker

To install Docker, open a terminal window and install Docker using the appropriate command for your operating system. If you're using Ubuntu, for example, you'd use the following command:

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

Once the installation is complete, you can confirm that Docker is running by typing:

sudo systemctl status docker

The output should look like this:

If everything is set up correctly, you should see Docker listed as active (running).

Pull the EMQX Broker Image

To get started with EMQX, the first step is to download the EMQX Docker image from Docker Hub. Open your terminal and run the following command:

docker pull emqx:latest

This command fetches the latest EMQX broker image and prepares it for use on your machine.

Run the EMQX Broker Image

After pulling the image, you can start a new EMQX container. Use the following command to run the EMQX Docker image:

docker run -d --name emqx -p 18083:18083 -p 1883:1883 emqx:latest

We can check if the container is running by executing the docker ps command. The output should look like this:

Here, the -d flag runs the container in detached mode. Ports 18083 and 1883 are mapped to your host machine, allowing you to interact with the EMQX broker. The EMQX broker runs as the Linux user emqx inside the container.

Set MQTT Configuration

You can set configurations via environment variables when running your Docker container. All EMQX settings in etc/emqx.conf can be configured this way (to learn about EMQX configuration options, refer to the official documentation).

Environment variables with the prefix EMQX_ map to key-value pairs in the configuration files. For example, here's how to set the MQTT TCP port to 1883:

docker run -d --name emqx -e EMQX_LISTENERS__TCP__DEFAULT__BIND=1883 -p 18083:18083 -p 1883:1883 emqx:latest

In this example, EMQX_LISTENERS*TCP*DEFAULT__BIND=1883 sets the MQTT TCP port to 1883. The -e flag specifies the environment variable.

Set Up Clustering

Before proceeding with this step, please ensure docker-compose is installed on your system.

For clustering, you can create a docker-compose.yaml file as follows:

version: '3'
services:
  emqx1:
    image: emqx:latest
    container_name: emqx1
    environment:
      - "EMQX_NAME=emqx1"
      - "EMQX_HOST=emqx1.emqx.io"
      - "EMQX_CLUSTER__DISCOVERY_STRATEGY=dns"
      - "EMQX_CLUSTER__DNS__RESOLVER=8.8.8.8"
      - "EMQX_CLUSTER__DNS__INTERVAL=5000"
    networks:
      emqx-net:
        aliases:
          - emqx1.emqx.io
    ports:
      - "1883:1883"
      - "8083:8083"

  emqx2:
    image: emqx:latest
    container_name: emqx2
    environment:
      - "EMQX_NAME=emqx2"
      - "EMQX_HOST=emqx2.emqx.io"
      - "EMQX_CLUSTER__DISCOVERY_STRATEGY=dns"
      - "EMQX_CLUSTER__DNS__RESOLVER=8.8.8.8"
      - "EMQX_CLUSTER__DNS__INTERVAL=5000"
    networks:
      emqx-net:
        aliases:
          - emqx2.emqx.io
    ports:
      - "1884:1883"
      - "8084:8083"

networks:
  emqx-net:
    driver: bridge

Then run the cluster with this command:

docker-compose -p my_emqx up -d

To view the cluster status, run:

docker exec -it emqx1 sh -c "emqx_ctl cluster status"

The output should look like this:

Set Up Persistence

To persist container data, EMQX requires the following directories to be saved:

/opt/emqx/data
/opt/emqx/etc
/opt/emqx/log

To make these directories persistent, use volume mounts in your docker-compose.yaml file like so:

services:
  emqx:
    # ... previous configurations
    volumes:
      - vol-emqx-data:/opt/emqx/data
      - vol-emqx-etc:/opt/emqx/etc
      - vol-emqx-log:/opt/emqx/log

The output should look something like:

To ensure the same state when the container restarts, make sure to specify the EMQX_NAME and EMQX_HOST variables as previously defined.

Large-Scale MQTT Deployment with EMQX

Running MQTT on Docker offers several compelling advantages, including rapid deployment, isolation, scalability, and portability. Leveraging Docker containers for MQTT deployment allows for efficient resource utilization and ease of management.

For those looking to set up MQTT on Docker with clustering and persistence, EMQX proves to be a powerful choice. EMQX's support for clustering, compliance with MQTT standards, and advanced features make it suitable for large-scale MQTT deployments.

Read these blog posts to see how to massively scale up MQTT with EMQX:

Originally published at www.emqx.com

A Cloud-Edge Collaborative Solution for Intelligent Oil & Gas Production and Operation

EMQ Technologies — Thu, 30 Nov 2023 15:13:22 +0000

Introduction

With the reduction of oil and gas reserves and the intensification of global market competition, oil and gas production enterprises are seeking digital transformation to optimize resources and enhance production efficiency. EMQ provides a one-stop cloud-edge collaborative IoT solution that addresses common industry challenges such as data collection, system construction and maintenance. It consolidates data generated by different systems onto one platform, facilitating more precise management and reduced costs.

Based on historical cases, EMQ’s solution can help oil and gas companies save more than 25% in annual collection system maintenance costs and over 40% in data point access costs.

Building Intelligent Oil&Gas System with EMQ

EMQ provides a comprehensive cloud-edge collaborative architecture encompassing industrial data acquisition, edge computing, data integration, and remote operational management. This solution can be used by oil and gas companies for real-time monitoring, equipment management, HSE(Health, Safety, Environment) management, smart production, and intelligent applications. With EMQ, enterprises can achieve greater efficiency and better control over their operations.

Real-time Industrial Data Acquisition

The industrial protocol gateway software, NeuronEX, supports various bus and industrial protocols, such as Modbus, OPC UA, Siemens, ABB, Mitsubishi, and Schneider. It collects real-time data from devices in production sites and then converts them into standard MQTT protocol messages, with data points adhering to unified formatting standards and millisecond-level timestamps. NeuronEX also caches data offline to ensure the integrity of historical data during network interruptions.

NeuronEX’s rule engine enables real-time computation and analysis, message standardization, data filtering, intelligent alarms, business routing, data persistence, and more at the edge. This allows oil and gas producers to preprocess data and process event logic at the edge.

NeuronEX’s HTTP PULL feature allows the perception of the configuration status and operating data of intelligent IT devices at the production field.

Moreover, NeuronEX can integrate AI for intelligent video monitoring, safety management, equipment maintenance, fault prediction, inspection, and safety monitoring through real-time processing of IoT data and video streams.

Enterprise-Grade IoT Message Bus

EMQX Enterprise deployed in the plant-level information center or group cloud provides a unified platform for oil and gas production data on private networks or the internet. This achieves high availability, high concurrency, and low-latency data transmission, analysis, processing, and integration. EMQX Enterprise MQTT platform can simultaneously handle connections from over a million IoT devices and process over a hundred thousand data interactions per second. It is highly suitable for the oil and gas production environment.

What You Can Achieve with EMQ

Comprehensive Perception and Centralized Management

EMQ’s solution facilitates data collection and management from intricate production equipment in oil and gas production, as well as safety and monitoring devices in the field. The cloud-edge collaborative architecture and data aggregation capabilities can address the challenge of centralizing the management of data from originally independent and scattered systems.

Cost Reduction and Efficiency Improvement

The solution simplifies the complex maintenance tasks involved in oil and gas production through the unified intelligent management of equipment access, inspections, configurations, upgrades, updates, and alarm handling. This significantly reduces costs caused by manual operation and improves efficiency.

Scalability and Flexibility for Future Innovation

The solution provides expandable interfaces to facilitate the integration of new systems and technologies into the business platform. At the edge, it supports the integration of algorithms in MATLAB, C, C++, Python, GO, and AI, promoting innovative development in edge intelligence.

Standardized configurations speed up IoT platform construction, making it easier to replicant and scale as the business grows.

The entire solution can be deployed through containerization, supporting cloud-native architecture to achieve unified container orchestration and management. This promotes the integration and innovation of smart oil and gas systems with cloud-native technologies.

Related Products

EMQX Enterprise

EMQX Enterprise is a powerful enterprise-level IoT messaging platform designed for large-scale deployments and high reliability in IoT applications. The following capabilities of EMQX Enterprise can benefit the industry:

High Reliability and Scalability: EMQX Enterprise adopts a distributed architecture with high availability and scalability to handle large-scale concurrent message transmission. It supports horizontal scaling to accommodate the growing number of IoT devices and data traffic, ensuring system stability.
Rich Protocol Support: EMQX Enterprise supports multiple messaging protocols besides the MQTT protocol. It allows developers to extend to support all kinds of private protocols for their application needs.
Data Integration: EMQX Enterprise seamlessly integrates with various data storage services, message queues, cloud platforms, and applications. It can connect with cloud services, enabling remote data transmission and cloud-based analysis.
Security and Authentication: EMQX Enterprise provides robust security features, including TLS/SSL encrypted transmission, client authentication, and access control. It supports various authentication methods such as username/password, X.509 certificates, and OAuth, ensuring the security of IoT communication.
Rule Engine and Data Processing: EMQX Enterprise has a flexible rule engine for real-time data processing and forwarding based on device data. It supports operations such as data filtering, transformation, aggregation, and persistence, helping users analyze and make decisions based on their business needs.
Visual Monitoring and Management: EMQX Enterprise provides an intuitive visual monitoring and management interface, allowing users to monitor IoT devices and message transmission in real-time. Users can view connection status, message traffic, and other metrics and perform device management, troubleshooting, and system configuration operations.

NeuronEX

NeuronEX is software tailored for the industrial sector, focusing on equipment data collection and edge intelligent analysis. It is primarily deployed in industrial settings, facilitating industrial equipment communication, industrial bus protocol acquisition, industrial system data integration, edge-level data filtering and analysis, AI algorithm integration, and integration with IIOT platforms. It offers the following features:

Diverse Connectivity: NeuronEX provides multi-protocol access capability, supporting simultaneous access to dozens of industrial protocols such as Modbus, OPC UA, Ethernet/IP, BACnet, Siemens, Mitsubishi, and more. It enables integration with multiple data sources from systems like MES and WMS within the enterprise. NeuronEX also supports bidirectional data flow, allowing both data acquisition and control command issuance to devices.
Deploy anywhere: Neuron has very low memory footprints and is suitable for running on low-profile architecture devices like x86, ARM, RISC-V, etc. It also supports docker-like containerized deployment, running with other co-located containers in Kubernetes environments.
Data Processing: NeuronEX incorporates 100+ built-in functions, supporting data filtering, data manipulation, device control, and data persistence, storing data in a time-series database.
Algorithm Integration: NeuronEX supports the integration of algorithms written in languages such as C, Python, and Go. It facilitates real-time inference of industrial mechanism models, machine learning, and deep learning models at the edge, enabling alerts and intelligent decision-making.
Edge to Cloud: Through protocols like MQTT and SparkplugB, NeuronEX aggregates and pushes industrial data to cloud platforms. The bidirectional data flow between NeuronEX and the cloud platform establishes cloud-edge data coordination and control synergy, leveraging the platform's big data storage and analysis capabilities to amplify the value of NeuronEX usage.

Originally published at www.emqx.com

From Data to Intelligence: One-Stop MQTT Platform for Smart Factory Advancements

EMQ Technologies — Tue, 21 Nov 2023 12:18:55 +0000

Introduction

In the age of advanced manufacturing, the need for extensive data collection and analysis across all aspects of production is crucial. This data enables factories and enterprises to discover optimization opportunities, improve efficiency and quality, and make informed decisions.

EMQ smart factory solution is designed to establish comprehensive data collection, transmission, distribution, and other mechanisms. This solution empowers factories to rapidly deploy a wide array of intelligent applications, including equipment health management, optimization of energy consumption equipment, production monitoring and analysis, product quality traceability, parameter optimization in the supply chain, predictive maintenance, and defect detection. This facilitates the manufacturing industry's transition towards digital and intelligent transformation.

EMQ Smart Factory Solution: Overcoming Challenges of Manufacturing

EMQ smart factory solution offers real-time monitoring of crucial data such as equipment status, production metrics, and quality markers. This enables factory management to stay up-to-date on production statuses, allowing quick decision-making and adjustments to improve production efficiency and product quality.

In addition to real-time monitoring, this solution also facilitates data analysis and extraction. By utilizing machine learning and AI algorithms, it uncovers valuable insights from vast production data. For instance, it can predict equipment failures and recommend necessary maintenance measures by analyzing equipment performance and operational data, preventing production disruptions and losses.

Eliminating Data Silos

In manufacturing factories, different devices and systems often use diverse protocols and formats, causing a lack of efficient data sharing akin to isolated islands. This results in a scattered distribution of a company's digital resources. EMQ comes to the rescue by enabling seamless access and sharing of extensive real-time data. It supports industrial equipment protocols and can seamlessly amalgamate diverse system data within the organization. With EMQ’s solution, unifying access to millions of data points becomes feasible, effectively eradicating data silos.

Improving Industrial Intelligence

Conventional manufacturing factories often encounter delays in information flow. This can lead to challenges in obtaining real-time data promptly, resulting in excess product inventory and hindered responsiveness to market shifts. EMQ steps in with robust capabilities like data filtering, cleansing, real-time computing, AI-driven analysis, and smart alerts. These tools enhance the intelligence of industrial equipment and enable companies to achieve equipment autonomy, independent decision-making, and nimble production strategies.

Break System Barriers And Enhance Data Collaboration

Collaboration obstacles frequently emerge within diverse departments in manufacturing plants, encompassing production, logistics, and quality control. Issues like limited information exchange and communication barriers often result in suboptimal efficiency in collaborative production efforts. By harnessing the power of flexible open APIs, a multi-layer rule engine, and stream data processing, this solution effectively reduces the burden of data processing at the central end and dismantles barriers between systems. This empowers companies to swiftly create advanced applications tailored to specific business scenarios, facilitating seamless teamwork across various machines and systems.

Agile Response to Business Growth

The architecture boasts remarkable scalability and flexibility. It can seamlessly expand horizontally as the enterprise's information infrastructure and data volume grow. This expansion ensures a robust SLA to effectively address the increasing demands posed by business growth and diverse data sources.

How EMQ Enables Smart Manufacturing

Heterogeneous Data Acquisition

In the industrial domain, a multitude of devices operate using various industrial protocols. Neuron industrial protocol gateway software offers support for common industrial protocols such as Modbus, OPC UA, IEC 61850, and IEC 104. This versatile capability allows for the collection of device data from diverse protocols, consolidating it into a unified data platform.

Furthermore, the solution seamlessly integrates this aggregated data into various other factory systems, encompassing traditional systems like PLC, SCADA, and DCS, as well as production management software like MES, WMS, and ERP. It extends its reach to enterprise service bus (ESB), various databases, and third-party software. Notably, it's also adept at handling unstructured data such as videos and documents, providing comprehensive data integration for enhanced operational insights.

Enterprise-Level MQTT Platform

In contrast to the conventional ISA95 architecture commonly employed in manufacturing, EMQX harnesses the MQTT protocol's publish/subscribe model, effectively decoupling data producers and consumers. This approach offers increased ease and flexibility for applications to access real-time factory data.

The innovative factory data architecture established by the EMQX enterprise-level MQTT platform offers several notable advantages:

It provides a unified data access platform, simplifying application development and integration compared to the traditional point-to-point communication in ISA95 setups.
It enhances data timeliness with low-latency data transmission, enabling intelligent applications such as alarms, instructions, and autonomous equipment operation.
It streamlines data storage by breaking down data silos, facilitating unified data asset management, and aiding in constructing the factory's data model to unlock the full potential of factory data.
Its adaptable architecture expansion features, including high-availability clustering, dynamic horizontal scaling, and support for hot upgrades and configurations, ensure a high level of service agreement (SLA) for the system.

Real-Time Processing and Forwarding

EMQX offers robust capabilities for handling real-time streaming data, addressing both low-latency raw data processing and analysis at the device level, as well as large-scale data conversion and distribution in the cloud or data centers.

With an SQL-like ease of use, it simplifies real-time data tasks such as filtering, transformation, and distribution, thereby maximizing the value derived from data.
It boasts responsive event triggering, allowing for the swift execution of predefined conditional actions to flexibly process real-time data.
EMQX provides an array of built-in functions that enhance the effectiveness and efficiency of its rule engine for data stream processing and manipulation.
It also supports custom function expansions for data processing needs and seamless integration with AI algorithms.

Data Integration

The EMQX enterprise-level MQTT platform stands out for its efficient data storage capabilities. It seamlessly harnesses EMQX's integrated rule engine to swiftly direct a wide range of device data to various database types, including relational, time series, and hybrid databases. This integration results in the creation of a cohesive repository of enterprise data assets, fostering real-time data sharing and collaboration across departments and teams.

Notably, each database type achieves an impressive 100,000 transactions per second (TPS) in terms of data writing performance, supporting the real-time storage of over a million high-throughput data measurement points in industrial applications.

Solution Components

EMQX Enterprise - The One MQTT Platform for IIoT

EMQX Enterprise is the one MQTT platform that helps build and grow your business-critical IoT applications without barriers and limits. It enables you to connect any device, at any scale, anywhere, as well as move and process your IoT data reliably in real-time.

High Reliability and Scalability: EMQX utilizes a distributed architecture with high availability and scalability to handle large-scale concurrent messaging. It supports horizontal scaling to accommodate growing IoT devices and data traffic, ensuring system stability.
Rich Protocol Support: EMQX supports multiple messaging protocols in addition to MQTT. It allows developers to extend to support a variety of industrial protocols to meet their application needs.
Data Integration: EMQX seamlessly integrates with various data storage services, message queues, cloud platforms, and applications. It can connect to cloud services for remote data transfer and cloud-based analytics.
Security Guarantee: EMQX provides strong security features, including TLS/SSL encrypted transmission, client authentication, and access control. It supports multiple authentication methods such as username/password, X.509 certificates, and OAuth to ensure secure IoT communications.
Rule Engine and Data Processing: EMQX has a flexible rule engine for real-time data processing and forwarding. It supports operations such as data filtering, conversion, aggregation and persistence to help users analyze and make decisions based on business needs.
Monitoring and Management: EMQX provides an intuitive visual monitoring and management interface that allows users to monitor IoT devices and messaging in real-time. Users can view connection status, message traffic and other metrics, as well as perform device management, troubleshooting and system configuration operations.

NeuronEX

NeuronEX is a specialized software designed for the collection of equipment data and edge intelligence analysis within the industrial sector. Its primary application is in industrial environments, where it facilitates communication between industrial equipment, collects data from industrial bus protocols, integrates industrial system data, filters and analyzes data at the edge, and seamlessly integrates AI algorithms. It also enables seamless integration with Industrial Internet platforms and facilitates various related functions.

NeuronEX, offered as a commercial software service, encompasses both Neuron and eKuiper.

Originally published at www.emqx.com

Exploring ISA95 Standards in Manufacturing

EMQ Technologies — Mon, 16 Oct 2023 13:43:59 +0000

In today's rapidly evolving industrial landscape, data has become a critical asset for organizations seeking to streamline processes, improve efficiency, and gain competitive advantage. The realm of automation, particularly in the context of ISA-95, presents both opportunities and challenges in effectively managing data.

This blog series examines the challenges of data organization in ISA-95 and explores the concept of a unified namespace and its benefits. We will also outline its implementation, including the use of the ISA-95 Equipment Model Standard and clustering for multi-site data replication.

The first blog explores the fundamental concepts of the ISA-95 standards, their importance in improving manufacturing processes, and their role in fostering cross-functional collaboration, all of which contribute to the optimization and implementation of traditional industrial operations.

Understanding ISA-95 Standards

The ISA-95 standard, also known as ANSI/ISA-95 or ISA-95, is an international standard developed by the International Society of Automation (ISA). It focuses on the integration of enterprise and control systems within the manufacturing industry. ISA-95 stands for "Enterprise-Control System Integration," and it provides a framework for designing and implementing interoperability between an organization's business and manufacturing processes, with the goal of improving the interactions between them.

The primary goal of ISA-95 is to establish a common language and structure for communication and data exchange between different levels of an organization, ranging from the enterprise level, where business decisions are made, down to the control level, where real-time production processes are managed. This standard helps bridge the gap between the information technology (IT) and operational technology (OT) domains.

ISA-95 Hierarchy (Automation Pyramid)

The Manufacturing Automation Pyramid is a conceptual representation of ISA-95 hierarchy that is often associated with industrial automation and control systems. The Automation Pyramid, also known as the automation hierarchy of "Industrial 3.0", illustrates the hierarchical structure of control and automation systems within industrial environments. It's a visualization that helps understand the different levels of control and their interconnections in a manufacturing setting.

The levels of the Automation Pyramid, from bottom to top, typically include:

Level 0: Field Devices and Instruments: This is the lowest level of the pyramid, where physical field devices such as sensors, actuators, and instruments are located. These devices gather data from the manufacturing process and send it to higher levels for processing, and control machinery.
Level 1: Control Devices and PLCs (Programmable Logic Controllers): At this level, basic control functions are executed. PLCs and control devices receive input from field devices and make decisions based on pre-programmed logic. They control processes and machinery directly.
Level 2: Supervisory Control: This level involves supervisory control systems that gather data from multiple PLCs and control devices. It provides real-time monitoring, data aggregation, and limited control capabilities for specific areas or processes.
Level 3: Manufacturing Execution Systems (MES): MES is responsible for managing production scheduling, work orders, quality control, and overall coordination of manufacturing operations. It bridges the gap between the shop floor and enterprise systems.
Level 4: Enterprise Systems: This is the highest level of the pyramid and includes enterprise resource planning (ERP) systems, which manage business operations, including finance, sales, procurement, and planning. Data from the lower levels feed into these systems for higher-level decision-making.

ISA-95 is more than just a pyramid hierarchy. On the other hand, it provides a more comprehensive and standardized approach to integrating enterprise and control systems, helping organizations design and implement interoperability between business and manufacturing processes. The Automation Pyramid can be a useful visualization to consider when thinking about the different levels of control within the context of implementing standards like ISA-95.

Categories of Information Model

The Categories of Information Model refers to a classification system that organizes and categorizes information exchanged between different levels of an organization's manufacturing processes. This model provides a structured framework for defining and understanding the types of information that need to be exchanged to ensure effective communication and integration between business and manufacturing processes.

Control and Monitoring: This category includes information related to the real-time control and monitoring of equipment, processes, and production activities. It involves data such as sensor readings, setpoints, alarms, operational statuses, and other data needed for immediate operational control.
Production Schedule: This category encompasses information about production planning and scheduling. It includes data on production orders, work orders, production sequences, start and end times of tasks, and any changes to the production schedule.
Performance Analysis: This category involves information that is used to analyze the performance of manufacturing processes. It includes data related to cycle times, production rates, downtime, efficiency metrics, quality measurements, and other performance indicators.
Quality and Compliance: Information related to quality control and compliance falls under this category. It includes data about quality standards, inspection results, testing data, non-conformities, corrective actions, and compliance with regulations.
Maintenance and Reliability: This category includes information about equipment maintenance, reliability, and asset management. It covers data on maintenance schedules, maintenance activities, spare parts inventory, equipment condition monitoring, and predictive maintenance.
Material Flow and Inventory: Information regarding the movement of materials, inventory levels, and material requirements is included in this category. It involves data on material consumption, material requests, inventory quantities, and material tracking.
Resource Allocation: This category encompasses information related to the allocation and utilization of resources, including personnel, equipment, tools, and facilities. It includes data on resource availability, assignments, and usage.
Order Fulfillment: Information related to order processing and fulfillment is categorized here. It includes data on customer orders, order status, order changes, shipping details, and delivery schedules.

Equipment Object Model

The Equipment Object Model within the ISA-95 standard focuses on representing the physical and logical equipment and resources used in the manufacturing and production processes. This model provides a structured framework for organizing, categorizing, and managing equipment, allowing for effective monitoring, control, and maintenance within the manufacturing environment.

This model hierarchy is designed to reflect the physical and logical relationships between different equipment units and their respective roles within the production process. While specific terminologies might vary based on the industry and organization, here is a common organization of the Equipment object model hierarchy:

Enterprise: This is the highest level of the hierarchy, representing the entire organization or company. It encompasses all sites and facilities.
Site: A site is a physical location or facility where manufacturing operations occur. It can be a factory, plant, or any other facility. Multiple areas or zones can exist within a site.
Area: An area represents a specific section within a site where a particular type of manufacturing activity takes place. Areas could be designated for different processes, products, or functions.
Unit: A unit refers to a distinct piece of equipment or a specific production unit within an area. Units can be individual machines, assembly lines, or process units. They are the primary operational components of the manufacturing process.
Control Module: A control module represents a functional aspect or module of a unit that can be controlled and monitored independently. It could be a specific subsystem, device, or component within a larger unit.
Component: A component represents smaller parts or sub-components that make up a control module. This level might not be present in all hierarchies and is particularly useful for complex systems.

The purpose of ISA88 is to provide standards and recommended practices as appropriate for the design and specification of batch control systems as used in the process control industries.

Information Exchange between Level 4 (ERP) and Level 3 (MES)

The Information Exchange Model between Level 4 (Enterprise) and Level 3 (Manufacturing Operations Management) within the ISA-95 framework involves the communication and data exchange between the business processes at the enterprise level and the manufacturing operations processes at the operations management level. This exchange is crucial for aligning business strategies, production planning, and execution on the shop floor.

The information models can be classified as Resource, Production Capability, Product Definition, Production Schedule, and Production Performance.

Resource Availability: Level 3 provides real-time updates to Level 4 about the current availability of resources and any potential constraints that might impact production. Based on production orders, Level 4 sends requests to allocate specific resources for production activities.
Production Capability: Level 4 shares information about the manufacturing capacity and constraints with Level 3. Level 3 replies the utilization of manufacturing capabilities to Level 4, indicating how effectively the available resources are being utilized.
Product Definition: Level 4 provides detailed specifications and requirements for the products to be manufactured. Level 3 verifies that the product specifications and requirements from Level 4 are accurate and feasible for production.
Production Schedule: Level 3 receives production plans from Level 4, detailing the sequence and timing of manufacturing activities required to fulfill the orders. Then, Level 3 communicates any changes or updates to the production schedule back to Level 4, such as delays, expedited orders, or adjustments due to resource constraints.
Production Performance: Level 3 shares real-time progress updates with Level 4, including information on quantities produced, completed tasks, and any deviations from the production plan. Level 3 provides data on quality inspections, test results, and any quality-related issues that arise during production.

In general, Level 4 initiates the exchange by sending production orders, work orders, and resource allocation requests to Level 3. Level 3 responds with updates on resource availability, production progress, quality data, and any changes to the production schedule.

Manufacturing Operations Management Activity Model

The Manufacturing Operations Management (MOM) Activity Model is a part of the ISA-95 framework that focuses on breaking down and structuring the activities that occur within the manufacturing operations processes. It provides a detailed view of the tasks and operations that need to be executed on the shop floor to fulfill the requirements specified by higher-level processes, such as production orders and work orders.

The MOM Activity Model serves as a bridge between the high-level business processes defined at the enterprise level (Level 4) and the specific actions carried out on the shop floor at the manufacturing operations management level (Level 3).

Conclusion

In conclusion, the ISA-95 standard has played a significant role in enhancing communication and integration in manufacturing, but adapting it to modern industrial management faces challenges. The evolving complexity of manufacturing, coupled with rapid technological advancements like Industry 4.0, requires flexibility that the standard's structured framework might not fully provide. In modern industrial management, EMQX and Neuron are two essential components that provide an ideal solution. Leveraging them as the MQTT broker and industrial gateway, organizations can elevate connectivity, data handling, and adaptability in dynamic production settings. Together, they seamlessly integrate into today's industrial landscape.

For more information about the benefits of this combination, please refer to: Modern Messaging Infrastructure for Smart Manufacturing.

Originally published at www.emqx.com

10 Important KPIs for Measuring Smart Manufacturing Performance

EMQ Technologies — Thu, 12 Oct 2023 15:16:16 +0000

As described in Part 1, KPIs (Key Performance Indicators) are measurable values that organizations use to evaluate their progress toward achieving specific goals and objectives. In the context of manufacturing, KPIs are crucial for monitoring and optimizing various aspects of the production process. They provide quantifiable metrics that help manufacturing companies track performance, identify areas for improvement, and make informed decisions to enhance efficiency, quality, and overall productivity.

In this blog, Part 2 of the series, we will delve into the 10 crucial KPIs commonly utilized in the manufacturing industry.

KPIs for Smart Manufacturing

1. On-time Delivery

To uphold customer satisfaction, manufacturers must prioritize punctual deliveries. A perfect score of 100% is the goal. When deliveries fall behind schedule, various factors could be at play, such as delayed supplies, impractical production timelines, or frequent equipment issues. Maintaining a strong track record of on-time deliveries is paramount for retaining existing clients and enticing new ones.

Calculation: On-Time Units Delivered ÷ Total Units Delivered

2. Production Schedule Attainment

This aids manufacturers in evaluating the effectiveness of production planning and the alignment of worker performance with set benchmarks. It enables them to identify potential performance issues that might influence timely deliveries. Simply tracking delivery punctuality isn't sufficient; it's equally important to pinpoint the production aspects that are influencing this performance. This analysis facilitates necessary adjustments to enhance delivery efficiency.

Calculation: (Actual Output ÷ Planned Output) x 100

3. Total Cycle Time

This metric gauges the duration from the commencement of production to shipping, encompassing the entire process of fulfilling a customer order. Excluding intervals and idle periods, it strictly measures the active manufacturing time. The comprehensive cycle time encompasses all stages, spanning from raw materials to the final product. This value is an average derived from the cycle times of all orders. Notably, machine cycle time is part of this metric, revealing machine efficiency and necessitating a comparison with the machine's optimal cycle time.

Calculation: Net Production Time (NPT) ÷ Number of Units Produced

4. Throughput

This indicator provides insight into the efficiency of machine performance. It's monitored in real-time to proactively address emerging concerns before they escalate. Optimal results are achieved in throughput when workers are matched with machines aligning with their proficiency. Furthermore, maintaining machines in excellent condition and streamlining processes to minimize operator involvement yields peak throughput levels.

Calculation: Number of Units Produced ÷ Total Time of Production

5. Capacity Utilization

This measurement assists in evaluating a machine's utilization of its highest potential output. Ideally, the objective is to maintain the machine in constant operational order, minimizing any periods of inactivity. Given the substantial investment in production equipment, idleness should be avoided. The emphasis lies in maximizing machine capacity to enhance efficiency and reduce expenses.

Calculation: (Actual Output ÷ Potential Output) x 100

6. Changeover Time

This metric monitors the duration required to complete all tasks within a production run, including activities like unloading/loading, retooling, calibration, and programming for subsequent jobs. This data aids manufacturers in identifying potential areas for enhancement. It might lead to better organization of setup procedures or highlight the need for additional machinery training for staff. Minimizing changeover time translates to reduced production costs, making swift transitions more cost-effective.

Calculation: Time to Produce First Item in a Product Batch – Time to Produce Last Item in a Product Batch

7. Scrap

Materials that fail to meet quality criteria are classified as scrap. Nevertheless, certain manufacturers categorize any unused raw material as scrap. Monitoring scrap materials is essential for cost control and the production of higher-quality goods.

Calculation: Total Scrap ÷ Total Product Run

8. Predictive Maintenance

When evaluating the planned maintenance KPI, it's crucial to include all instances of emergency maintenance within the overall maintenance calculations. Ideally, the proportion of planned maintenance categorized as emergency work orders should remain below 15%. Emergency maintenance incurs significantly higher costs due to factors such as overtime, expedited parts, and disrupted production. It's imperative to minimize emergency maintenance occurrences to prevent negative impacts on profitability, downtime, and employee morale.

Calculation: (Planned Maintenance Time ÷ Total Maintenance Time) x 100

9. Availability

Availability refers to the assessment of machine operational time versus periods of inactivity. Recognizing the extent of downtime within your organization is crucial, given its substantial cost and status as a primary challenge for manufacturers. When determining availability, it's vital to consider both planned and unplanned downtime. Equally important is maintaining a record of downtime causes, enabling subsequent analysis to pinpoint potential enhancements.

Calculation: Uptime ÷ (Uptime + Downtime)

10. Overall Equipment Effectiveness

This vital KPI evaluates equipment productivity by combining availability, performance, and quality factors. However, it doesn't encompass machine downtime or maintenance periods. It's important to recognize that this KPI offers an incomplete perspective. While a high effectiveness rate may be observed, it's advisable to explore potential underlying issues that might contribute to this figure. In next blog Part 3, we will have a more detailed explanation about OEE because of its importance.

Calculation: Availability x Performance x Quality

Achieve Your KPIs with OMH Offering

The Open Manufacturing Hub (OMH) solution presented by EMQ is a reference architecture for building powerful and scalable Industrial IoT applications that keep your KPIs in control.

When deploying the OMH solution, it provides access to a set of pre-defined KPIs that offer two flexible options: first can either use them as they are, or second can customize them to meet specific business goals. In addition to providing these predefined KPIs, OMH also includes a robust business intelligence component. This business intelligence feature helps to delve deeper into the data, facilitating a comprehensive analysis of KPIs. Explore and extract insights to gain a richer understanding of the performance of a particular KPI and its impact on the manufacturing processes. This analytical capability is critical to identifying areas for improvement and optimizing operations.

The OMH solution allows customizing the KPIs based on unique situations. Should a specific scenario require additional measurements or different performance criteria, the organization has the flexibility to define new KPIs that accurately capture the difference of that situation.

All in all, OMH's comprehensive offering includes both preset and customizable KPIs, along with a powerful business intelligence module. This combination enables the organization to harness the power of data analysis to make informed decisions and continuously refine the manufacturing processes based on real-time insights.

Conclusion

KPIs play a pivotal role in guiding significant business choices. Individuals overseeing these metrics in manufacturing should feel responsible for their designated areas and consistently monitor them to drive enhancements when required. Flexibility is key. KPIs may evolve to address specific circumstances. Their focus should remain on vital management concerns. Ideally, a business should work with approximately 10 KPIs, as an excessive number can overwhelm and lose effectiveness.

In addition, an efficient solution is also crucial for optimizing performance and achieving KPIs. With OMH, organizations can easily build powerful, scalable, and data-driven Industrial IoT applications. They will gain real-time insights to optimize industrial processes, improve operational efficiency, and make better decisions.

Originally published at www.emqx.com

Modern Messaging Infrastructure for Smart Manufacturing

EMQ Technologies — Sun, 08 Oct 2023 01:35:20 +0000

Challenges for Traditional Manufacturing

Running a manufacturing business is a multifaceted and demanding endeavor. Almost all companies face significant challenges, including managing large-scale production processes, maintaining product competitiveness, and navigating various operational complexities. To remain competitive, manufacturing companies consistently seek ways to improve their overall performance. Key areas for enhancement typically include:

Operational Efficiency
Quality Assurance
Asset Optimization
Supply Chain Optimization
Product Innovation
Environment Sustainability

Different industries prioritize different challenge areas based on their specific needs and goals. For example, logistics companies may focus on optimizing their supply chain management processes, while food companies may prioritize ensuring the highest standards of food quality. Each industry tailors its focus to meet the unique needs and requirements of its operations. In general, these improvements can be grouped into the six main categories mentioned above.

Industry 4.0 Approach

To address these challenges, a significant number of companies are choosing to adopt Industry 4.0 technologies as part of their strategic approach. The adoption of these advanced technologies is intended to accelerate the digital transformation of their manufacturing facilities. By leveraging Industry 4.0 principles and solutions, companies aim to revolutionize their operations and achieve higher levels of automation, more comprehensive product quality tracking systems, faster production scalability, and improved sustainability.

Blind Spot for Adopting New Technologies

Before budgeting and implementing new technologies, it is critical for companies to assess the capacity and capabilities of their current IT and OT infrastructure.

Interoperability of disparate IT and OT systems is a common issue that most companies would overlook. It's better to:

Assess the compatibility of existing IT and OT systems with new technologies to ensure no compatibility issues between different systems and devices.
Plan for necessary integrations or updates to ensure seamless communication and data exchange.

In addition to interoperability, scalability is an important consideration when adopting new technologies. Companies should also consider the following:

Whether the current infrastructure can scale to support the companies' anticipated growth in data storage and processing requirements.
The requirement of expanding server capacity, increasing storage capacity, and optimizing data management processes when adding large data processing systems.

Legacy Manufacturing Information Infrastructure

In traditional automation systems, direct communication between management-level systems such as ERP or MES and field-level sensors and devices is limited. The primary communication flow follows a hierarchical structure within the automation pyramid, where data from field-level sensors and devices is collected and processed by control-level devices, such as PLCs and SCADAs, before being shared with the management level.

Cascade Data Flow

The flow of information in the automation pyramid generally follows a cascade-like pattern, with data flowing from the bottom (field level) to the top (management level), while control commands or instructions typically flow from the top to the bottom. This pattern of information flow is a fundamental characteristic of the automation pyramid. As a result, communication between the top management IT level and the bottom device OT level is isolated. A large amount of production data is stored or neglected within OT-level systems and devices such as PLCs and SCADA.

Manual Data Analysis for Traditional Manufacturing

In response to this problem, some large companies use data collection software such as Kepware to periodically collect data from field-level devices and control-level SCADA. This data includes various measurements, process variables, alarms, and other relevant information. It is critical to ensure accurate and reliable data collection to provide a solid foundation for subsequent analysis. Spreadsheet tools such as Microsoft Excel or Google Sheets are widely used for basic data analysis. In addition, tools such as Tableau, Power BI, or QlikView allow users to create visually appealing data visualizations that make it easier to explore and present insights. Of course, such data manipulation is part of manual batch processing for data analysis. It is not as efficient as automated and real-time data processing. More components and systems are added to the information infrastructure. However, more communication channels are created, making data exchange more complicated.

Central Manufacturing Data Hub

As mentioned above, new technologies may create compatibility issues with existing components or systems within the automation pyramid. Different devices, protocols, or software may not seamlessly integrate or communicate with each other. Ensuring compatibility and establishing proper interoperability between new and existing technologies can be a significant challenge. Downtime, system reconfiguration, or training requirements can temporarily impact productivity and workflow.

Centralized Data Repository

EMQ introduces a solution called Open Manufacturing Hub (OMH) to ensure interoperability and scalability of the manufacturing information infrastructure by establishing a more flexible and connected approach to data exchange. OMH establishes a centralized data repository to store and manage all data collected from different levels of the automation pyramid. This repository acts as a central hub where data from field-level devices, control-level systems, and management-level applications is consolidated.

In the OMH architecture, data can flow bi-directionally between different levels of the automation pyramid. While the traditional cascade-like flow of data from the field level to the management level remains relevant, the OMH enables more dynamic and direct communication between different levels. Data can be accessed and used by multiple levels, enabling real-time monitoring, analysis and control.

The OMH provides standardized methods for data access via the EMQX broker. It allows authorized users and applications to subscribe to data from any information system in the automation pyramid using consistent and well-defined MQTT interfaces. The OMH also provides standardized methods for device data acquisition through the Neuron gateway. It allows different devices with different data formats to communicate with any other system or device.

IT and OT Convergence

The OMH can facilitate interoperability between IT and OT systems within the automation pyramid by providing a rich set of OT industrial protocol communications and IT application connectors, enforcing MQTT standardization for different data formats. It harmonizes data, ensuring consistency and compatibility between components and systems. Standardization enables smoother collaboration and convergence efforts, breaking down information silos.

By collecting and storing real-time data from OT systems, the OMH enables real-time monitoring and control of manufacturing processes. This facilitates better operational visibility, proactive decision making, and timely intervention to resolve problems or optimize performance. Real-time data from OT systems can be combined with data from IT systems to provide a comprehensive view of the manufacturing environment.

Contextualization

This type of IT and OT data combination is the process of data normalization and contextualization, which means combining data from many different sources, such as field-level devices, control systems, and management-level applications, into a single source of information. This integration enables better contextualization by bringing together data from different parts of the manufacturing process.

In fact, contextualized data is one of the most important features that OMH can provide to analytical applications such as AI/ML. By providing additional context and metadata, AI/ML models can better understand and interpret data, reducing errors and improving accuracy. Therefore, implementing OMH can lead to an improvement in the predictive capabilities of analytic models.

In conclusion, OMH can change the cascade-like data flow in the automation pyramid to a unified namespace architecture. In general, incorporating OMH into the information infrastructure is just like implementing a unified namespace architecture, which provides the following benefits.

Simplified Data Access
Improved Collaboration
Contextualized Data
Simplified IT Administration
Scalability and Flexibility
Interoperability
Data Consistency
Improved Data Integrity
AI/ML Precise Prediction

Multi Sites Manufacturing

By leveraging the high throughput and low latency capabilities of the EMQX broker, OMH can provide high-speed data replication across multiple sites. High-speed data replication across multiple manufacturing sites ensures the consistency of data shared between sites. When data is replicated, each site has access to the same set of information, ensuring that decisions and actions are based on the most current and synchronized data. This promotes operational consistency, reduces inconsistencies, and enables standardized processes across all sites.

Data replication supports scalability and facilitates the expansion of manufacturing operations to new locations. As new sites are added, data can be replicated to those sites, allowing them to quickly access relevant information and integrate into the overall manufacturing ecosystem. This scalability helps streamline operations, accommodate growth, and maintain consistent data across the extended manufacturing network.

Distributed Stream Processing Framework

Replicating data to multiple manufacturing sites enables local data access and processing with our edge processing application eKuiper, which is an edge lightweight IoT data analytics/streaming software. Each site can access and manipulate the replicated data locally, reducing the latency and network traffic associated with accessing data from a central location. This improves performance, enables faster response times, and ensures smoother operations within each site.

On-Premise Central Control Room

With data from multiple applications and systems in multiple locations integrated into the high-speed OMH in the central control room at headquarters, operators and engineers can view real-time data, alarms and performance metrics through visual dashboards and interfaces. This real-time monitoring provides immediate insight into the status of operations, enabling timely responses and proactive management of manufacturing processes.

Virtual Analytic Cloud Platform

By leveraging data from diverse sources and various locations, the implementation of a cloud platform replaces the conventional centralized control room at headquarters. Cloud platforms offer the convenience of remote access to data and applications from anywhere with an Internet connection. This enables operators and engineers to efficiently monitor and manage manufacturing processes, promoting seamless remote collaboration and facilitating informed decision-making across the organization.

Real-time Stream Processing

A hybrid stream processing framework is in place to handle real-time data analysis in both the central control room and the cloud platforms. Analytical tasks are distributed between these environments based on their requirements. The framework includes Apache Kafka, Node-Red, and Apache Flink, which are versatile applications suitable for cloud platforms or on-premises servers. This setup ensures efficient real-time data processing to meet the time constraints of production and effectively addresses the following challenges:

Real-time processing enables rapid response to dynamic changes in the manufacturing environment. It allows the system to continuously monitor data streams, detect deviations, and trigger alerts or automated actions in real time. This agility ensures quick adaptation to evolving conditions, minimizing disruptions and optimizing operational efficiency.
Real-time processing provides the ability to optimize operational efficiency on the fly. By analyzing real-time data, the system can identify process inefficiencies, resource bottlenecks, or deviations from optimal performance. This allows for immediate adjustments, resource allocation optimization, and dynamic process optimization to enhance overall efficiency and productivity.
Real-time processing helps in proactive issue detection and prevention. By monitoring data streams in real time, the system can identify patterns or early warning signs of potential issues before they escalate. This allows operators to take preventive measures, initiate maintenance activities, or implement corrective actions in a timely manner, minimizing downtime and improving reliability.
Real-time processing empowers decision-makers with real-time insights and decision support tools. By analyzing data streams in real time, the system can provide decision-makers with real-time dashboards, alerts, and visualizations. This enables informed decision-making based on up-to-the-minute information, improving response time and overall decision quality.

Conclusion

Successful business transformation requires a shift away from traditional approaches and a renewed focus on scalability and interoperability. This can be achieved by implementing OMH, which centralizes the data repository, creates a contextualized data platform, and delivers the work process in a distributed framework. By addressing the manufacturing challenges, companies can realize the benefits of a scalable and successful data analytics strategy, ultimately leading to a successful adoption of new technologies for Industry 4.0.

Serverless MQTT: Your Key to Efficient IoT Prototyping

EMQ Technologies — Tue, 26 Sep 2023 14:38:34 +0000

The Internet of Things (IoT) is transforming the way we live and work, from smart homes and cities to industrial automation and healthcare. But developing an IoT project can be a complex and time-consuming process, requiring expertise in hardware, software, and cloud services. That's why rapid prototyping has become a popular approach for IoT developers, allowing them to quickly test and iterate their ideas before investing in a full-scale deployment.

IoT prototyping usually contains several key parts like a hardware platform, open-source libraries and frameworks, a cloud service for data storage and management, a simulator that mimics the behavior of real-world devices, etc. One of the essentials is connecting the device to the cloud with an MQTT Broker.

This blog will discuss the optimal device connection approach for quickly implementing an IoT prototype.

Requirements of MQTT Service in IoT Prototyping

MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol that is commonly used in IoT applications to enable communication between devices and servers. MQTT brokers act as intermediaries between clients and clouds, allowing for data to be sent and received in a scalable and efficient manner.

Typically, you can choose a self-hosting MQTT broker or a fully-managed MQTT service for the device connection of your IoT prototyping.

Hosting an MQTT broker requires some technical knowledge, and you need to create cloud-based infrastructure and maintain the instance by yourself. The open-source MQTT broker software installation and configuration may also take some time for beginners. But for a rapid IoT prototype, we should spend limited time in system and device testing rather than establish an MQTT broker.

Fully-managed MQTT service addresses the challenge of infrastructure management, saving your time in setting up an MQTT broker. However, some fully managed services are generally targeted at production environments and do not have particularly low connection specifications. Also, it is best to avoid using the time package to prevent unnecessary resource consumption for a rapid IoT prototype.

Serverless MQTT: A Perfect Option for IoT Prototyping

Serverless MQTT is a cutting-edge technology in the realm of MQTT cloud service. It provides a messaging service that automatically scales based on demand without manual intervention.

EMQX Cloud Serverless, as a newly-offering edition of EMQX Cloud, perfectly meets the requirements we mentioned above. It is a fully managed MQTT service and provides all the capabilities of MQTT 3.1.1 and MQTT 5. Just a few clicks to connect the client to the deployment, and then you can focus on prototyping the application.

EMQX Cloud Serverless is ideal for development and testing scenarios where there are only a few connections or high-throughput situations. Thanks to its large-scale, multi-tenant cluster, EMQX Cloud Serverless adopts a pay-as-you-go pricing model, ensuring cost control by charging based on actual resource consumption rather than time. It eliminates the need for users to pay for idle resources, making it especially beneficial for IoT applications with unpredictable traffic patterns.

Get Your Most Cost-Effective MQTT Service for IoT Prototyping

Pay as You Go

EMQX Cloud Serverless calculates the concurrent clients with sessions. To be more specific, the number of sessions = the number of connected clients + the disconnected clients with sessions retained in the broker. If you have no idea about retained sessions (or persistent sessions), this blog explains it well.

EMQX Cloud Serverless uses Session Minutes and Traffic as its billing units. A Session Minute is counted for each session that connects to the Serverless deployment, regardless of its duration within a minute. Traffic is measured for both inbound and outbound data transfer of the deployment.

Free Quota

Here comes the best part: EMQX Cloud Serverless offers a free quota of billing units every month. You will never get charged if your usage is always within the free quota. Theoretically, you can get a free managed MQTT server forever!

1 million session minutes can support a small number of devices staying connected for quite a long time. In the setup process of Serverless deployment, you can set Spend Limit to 0 to get a free MQTT deployment in seconds. You can read this blog for more information about pricing mode and free quota.

Test out Your Ideas with EMQX Cloud Serverless Today

EMQX Cloud Serverless is an ideal solution for businesses that seek ways to improve efficiency, scalability, and cost-effectiveness. Its architecture in cloud computing enables developers to concentrate on coding and deploying their IoT system without the need to manage the infrastructure. Combined with MQTT, an automated messaging service that can scale up or down based on demand becomes a cutting-edge solution for IoT developers.

You can check out the following links to learn more:

Originally published at www.emqx.com

Build an IoT Time-Series Data Application for Industrial Energy Monitoring with MQTT and Timescale

EMQ Technologies — Tue, 26 Sep 2023 14:31:33 +0000

Introduction

As industrial production continues to grow, monitoring energy consumption in factories is crucial for optimizing energy management strategies and reducing costs. EMQX and Timescale provide a scalable IoT platform for efficient collection and analysis of energy consumption data in real-time. This article shows how to use EMQX to collect data from the plant and integrate it with Timescale for reliable data storage and analysis.

Prerequisites

Git
Docker Engine: v20.10+
Docker Compose: v2.20+

How it Works

This is a simple and effective architecture that avoids complex components. It utilizes the following three key components:

In addition to the basic components, EMQX provides comprehensive observability capabilities. You can use the following components to monitor EMQX metrics and load when the system is running:

Clone the Project Locally

Clone the emqx/mqtt-to-timescaledb repository locally, and initialize the submodule to enable the EMQX Exporter (optional):

git clone https://github.com/emqx/mqtt-to-timescaledb
cd mqtt-to-timescaledb

# Optional
git submodule init
git submodule update

The codebase consists of four parts:

The emqx folder contains EMQX-TimescaleDB integration configurations to automatically create rules and data bridges when launching EMQX.
The emqx-exporter, prometheus and grafana-provisioning folders include observability and energy consumption data visualization configurations for EMQX.
The create-table.sql file defines the database table schema, which will create the tables in Timescale during initialization.
The docker-compose.yml orchestrates all components to launch the project with one click.

Start MQTTX CLI, EMQX, and Timescale

Please make sure you have installed the Docker, and then run Docker Compose in the background to start the demo:

docker-compose up -d

Now, MQTTX CLI will simulate 10 factories connecting to EMQX, and report the energy consumption data of major equipment on their production lines at a frequency of 1 message per second. The energy data in JSON format will be sent to the topic mqttx/simulate/IEM/{clientid}.

All equipment will collect the current instantaneous power usage (randomly simulated, not exceeding max power), calculate the electricity consumption for 1 second, and publish the data to EMQX.

{
    "factory_id": "08",
    "factory": "Miller and Sons",
    "values": {
        "air_compressor_1": 3.72,
        "air_compressor_2": 5.01,
        "lighting": 0.95,
        "cooling_equipment": 23.19,
        "heating_equipment": 52.66,
        "conveyor": 10.66,
        "coating_equipment": 5.21,
        "inspection_equipment": 2.6,
        "welding_equipment": 5.27,
        "packaging_equipment": 7.38,
        "cutting_equipment": 12.56
    },
    "timestamp": 1691144157583
}

EMQX will create a rule to ingest messages from the factory. You can also modify this rule later to add custom processing using EMQX's built-in SQL functions:

SELECT
  payload
FROM
  "mqttx/simulate/#"

After the rules process the data, EMQX will insert the energy consumption data from the message payload into the specified table in Timescale through data bridges. Configuring INSERT SQL on the data bridges allows flexibly accomplishing this operation.

Subscribe to Data from EMQX

Docker Compose has included a subscriber to print all energy consumption data. You can view the data with this command:

$ docker logs -f mqttx
[8/4/2023] [10:15:57 AM] › topic: mqttx/simulate/IEM/mqttx_85df7038
payload: {"factory_id":"08","factory":"Miller and Sons","values":{"air_compressor_1":3.72,"air_compressor_2":5.01,"lighting":0.95,"cooling_equipment":23.19,"heating_equipment":52.66,"conveyor":10.66,"coating_equipment":5.21,"inspection_equipment":2.6,"welding_equipment":5.27,"packaging_equi...

To subscribe and receive the data with any MQTT client:

mqttx sub -t mqttx/simulate/IEM/+

View Enengy Data in Grafana

To view enengy data in the Grafana dashboard, open http://localhost:3000 in your browser, log in with username admin and password public.

After successful login, go to “Home → Dashboards” page and select “Energy Monitoring data”. The dashboard comprehensively displays the key energy consumption metrics of various industrial equipment, including the cumulative energy consumption value of each equipment and the energy consumption share of each plant, which fully presents the real-time energy usage of the industrial system and facilitates data-driven energy-saving management.

Conclusion

In this post, we have explored how to integrate EMQX and Timescale to build an industrial energy monitoring pipeline. By leveraging EMQX as a real-time MQTT broker and utilizing its SQL data integration to ingest data into Timescale, we have created an end-to-end solution for collecting and analyzing time-series energy consumption data.

This demo project provides a starting point for building scalable time-series data platforms, opening opportunities for real-time monitoring, optimization, and intelligence in industrial facilities and other time-sensitive use cases. The reliability of EMQX and the analytical power of Timescale unlock valuable insights from time-series data.

Please Visit the GitHub link for the Demo of ingesting time series data into Timescale.

Originally published at www.emqx.com

MQTT 5.0 Packet Explained 01: CONNECT & CONNACK

EMQ Technologies — Sat, 23 Sep 2023 00:00:01 +0000

In Introduction to MQTT 5.0 Packet, we introduced that MQTT packets are composed of three parts: Fixed Header, Variable Header, and Payload, as well as common concepts in MQTT packets such as Variable Byte Integer and Property. Now, we will further introduce the composition of each type of packet according to its actual use. First, we will focus on the packets used to establish an MQTT connection.

If we want to use MQTT for communication, the first step must be to establish an MQTT connection. Establishing an MQTT connection requires the use of two control packets, which are the CONNECT packet and the CONNACK packet. The CONNECT packet is the first control packet sent by the client to the server after establishing a network connection and is used to initiate a connection request. The server will return a CONNACK packet to inform the client of the connection result.

Sample Packets

We use MQTTX CLI to initiate a connection to a Public MQTT server. In this connection, we set the protocol version to MQTT 5.0, Clean Start to 1, Session Expiry Interval to 300 seconds, Keep Alive to 60, and the username and password to admin and public respectively. The corresponding MQTTX CLI command is:

mqttx conn --hostname broker.emqx.io --mqtt-version 5 \
  --session-expiry-interval 300 --keepalive 60 --username admin --password public

Below is the CONNECT packet sent out by MQTTX CLI captured using the Wireshark tool:

The following is the CONNECT packet sent by MQTTX CLI captured using Wireshark. In Linux, you can use tcpdump to capture the packet first, and then import it to Wireshark for viewing:

10 2f 00 04 4d 51 54 54 05 c2 00 3c 05 11 00 00 01 2c 00 0e 6d 71 74 74 78 5f 30 63 36 36 38 64 30 64 00 05 61 64 6d 69 6e 00 06 70 75 62 6c 69 63

But this is a string of hexadecimal bytes that is not easy to understand unless they are converted into the following format:

Similarly, we also captured the CONNACK packet returned by the public MQTT server:

20 13 00 00 10 27 00 10 00 00 25 01 2a 01 29 01 22 ff ff 28 01

After parsing this string of packet data, we can see that the Reason Code of the CONNACK packet is 0, which means that the connection was successful. The multiple properties that follow provide a list of features supported by the server, such as the maximum packet size supported, whether to support retained messages, etc:

The above examples are to help everyone better understand the structure of MQTT Packets. In actual applications, you can directly view the packet details in Wireshark. Wireshark provides excellent support for MQTT. It directly lists the values of each field for us, without the need for us to analyze ourselves:

Of course, Wireshark has actually listed the values of each field in the packet for us. Through the following introduction to the structure of the CONNECT and CONNACK packets, combined with the packet capture results of Wireshark, you will quickly master these two packets:

CONNECT Packet Structure

Fixed Header

In the Fixed Header of the CONNECT packet, the Packet Type field located in the high 4 bits of the first byte must be 1 (0b0001), and the low 4 bits in the first byte must all be 0.

Therefore, the value of the first byte of the CONNECT packet must be 0x10. We can use this to determine whether a packet is a CONNECT packet.

Variable Header

The Variable Header of the CONNECT packet contains the following fields in order:

Protocol Name: This is a UTF-8 encoded string used to indicate the protocol name. In MQTT, the first two bytes of the UTF-8 encoded string are uniformly used to indicate the length of the actual character data that follows. The protocol name is fixed as MQTT in MQTT 3.1.1 and MQTT 5.0, so the corresponding complete content in hexadecimal bytes is 00 04 4d 51 54 54, where 4d 51 54 54 is the ASCII value of the string MQTT The protocol name in the earliest MQTT 3.1 is MQIsdp, so it corresponds to 00 06 4d 51 49 73 64 70.
Protocol Version: This is a single-byte length unsigned integer used to indicate the protocol version. Currently, there are only three possible values, 3 represents MQTT 3.1, 4 represents MQTT 3.1.1, and 5 represents MQTT 5.0.
Connect Flags: Connection flags, have a length of only one byte, but contain several flags used to indicate connection behavior or whether certain fields exist in the Payload.

User Name Flag: Used to indicate whether the Payload contains the Username.
Password Flag: Used to indicate whether the Payload contains the Password.
Will Retain: Used to indicate whether the will message is a retained message.
Will QoS: Used to indicate the QoS of the will message.
Will Flag: Used to indicate whether the Payload contains relevant fields of the will message.
Clean Start: Used to indicate whether the current connection is a new session or a continuation of an existing session, which determines whether the server will directly create a new session or attempt to reuse an existing session.
Reserved: This is a reserved bit, its value must be 0.
- Keep Alive: This is a double-byte length unsigned integer used to indicate the maximum time interval between two adjacent control packets sent by the client.
- Properties: The table below lists all available properties of the CONNECT packet.

Payload

In the Payload of the CONNECT packet, aside from the Client ID, all other fields are optional. Their existence depends on the value of the corresponding flag in the Connect Flags of the variable header. However, if these exist, they must appear in the order of Client ID, Will Properties, Will Topic, Will Payload, User Name, and Password.

CONNACK Packet Structure

Fixed Header

The value of the high 4 bits in the first byte of the Fixed Header is 2 (0b0010), indicating that this is a CONNACK packet.

Variable Header

The Variable Header of the CONNACK packet contains the following fields in order:

Connect Acknowledge Flags: Connection confirmation flag.
- Reserved (Bit 7 - 1): Reserved bits, must be set to 0.
- Session Present (Bit 0): Used to indicate whether the server is using an existing session to resume communication with the client. Session Present may be 1 only when the client has set Clean Start to 0 in the CONNECT connection.
Reason Code: Used to indicate the result of the connection. The table below lists some common Reason Codes in the CONNACK packet, for a complete list, please refer to the MQTT 5.0 Reason Code Quick Reference Guide.

Properties: The table below lists all available properties for the CONNACK packet.

IdentifierProperty NameType0x11Session Expiry IntervalFour Byte Integer0x21Receive MaximumTwo Byte Integer0x24Maximum QoSByte0x25Retain AvailableBytes0x27Maximum Packet SizeFour Byte Integer0x12Assigned Client IdentifierUTF-8 Encoded String0x22Topic Alias MaximumTwo Byte Integer0x1FReason StringUTF-8 Encoded String0x26User PropertyUTF-8 String Pair0x28Wildcard Subscription AvailableByte0x29Subscription Identifier AvailableBytes0x2AShared Subscription AvailableByte0x13Server Keep AliveTwo Byte Integer0x1AResponse InformationUTF-8 Encoded String0x1CServer ReferenceUTF-8 Encoded String0x15Authentication MethodUTF-8 Encoded String0x16Authentication DataBinary Data

Payload

The CONNACK packet has no Payload.

Conclusion

CONNECT is the first MQTT packet sent by the client after the network connection between the client and the server is established. CONNACK, as the response packet of CONNECT, indicates the connection result through the reason code.

The client and server need to use CONNECT and CONNACK packets to complete the exchange of necessary information, such as the protocol version, Client ID, user name, password used by the client, and the maximum packet size and QoS supported by the server, and whether there is a corresponding session.

The above is an introduction to the MQTT CONNECT and CONNACK packets. In subsequent articles, we will continue to study the structure and composition of packets like PUBLISH, and DISCONNECT.

Originally published at www.emqx.com

4 Reasons Why You Should Adopt MQTT in Unified Namespace

EMQ Technologies — Tue, 19 Sep 2023 22:20:08 +0000

In the last episode of the UNS series, we explained what UNS is and why it is essential for IIoT 4.0. MQTT technology is always a first choice regarding how to build a UNS. Readers unfamiliar with MQTT might be curious why it always appears with UNS in IIoT 4.0 solutions. This episode will walk you through why MQTT is the best partner for building UNS.

What is MQTT

Before entering the world of UNS, it is important to have a basic knowledge of MQTT.

MQTT (Message Queuing Telemetry Transport) is a lightweight, publish-subscribe-based messaging protocol. It was created in 1991 by Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech) to connect oil pipelines over unreliable satellite networks. Thus, since day 1, MQTT has been designed to provide efficient connectivity for resource-constrained devices and low-bandwidth, high-latency, or unreliable networks. Now it has become the de-facto standard of the IoT industry.

For more details on MQTT, please refer to MQTT Protocol: How It Works & Core Concepts Explained.

Why Adopt MQTT in UNS

In our last episode, we learned a common tongue of UNS definition: It is about reflecting your business to the digital infrastructure. To create an effective UNS, it is crucial to have a technology stack that can efficiently connect numerous manufacturing devices and unify the whole business from the physical to the digital world.

This is exactly why MQTT plays a pivotal role in UNS. MQTT broker is commonly used as a digital infrastructure for managing fragmented connections and connecting distributed devices in other industries (such as IoV ), and so is in IIoT. Here are the reasons why you should adopt MQTT in UNS.

MQTT is Lightweight and Scalable

UNS requires a data hub as a single source of truth to connect all of your manufacturing pieces, including PLC, SCADA, MES, and ERP. The amount of real-time data is huge. Usually, data hub faces thousands or even millions of concurrent connections in large-scale solutions.

MQTT has very low overhead and bandwidth consumption thanks to its binary format. This lightweight design minimizes the size of messages and reduces network traffic. This feature perfectly matches UNS and IIoT.

Business owners need to focus more on the operation, not tangling with the expansion of infrastructure resources to support a fast-growing business. This is why the scalability of UNS matters. In the MQTT architecture, some brokers provide superb scalability that can support millions of messages per second, such as EMQX.

Learn more: Reaching 100M MQTT connections with EMQX 5.0

MQTT Connects Everything

MQTT may be a newcomer in the world of IIoT, but it has gained immense popularity. This is because MQTT constitutes the greatest common subset of the old IIoT technology, making it an ideal tool for connecting all the components of your existing system without causing any damage.

Because there is no such technology and products that fit all cases, you cannot build a UNS by OPC UA alone or depend on one protocol solely, especially when you bring the cloud into consideration. UNS must combine different technologies together, and MQTT is that adapter and coordinator to unify them in one namespace.

Many data convention tools and protocol proxy software exist in the open-source world, such as Neuron, ignition and PLC4X. You can connect OPC UA to MQTT, the same story with Modbus and Ethercat.

Read our tutorials on industrial protocol to MQTT bridging:

Bridging OPC UA Data to MQTT for IIoT

Bridging Modbus Data to MQTT for IIoT

In IIoT 3.0, companies that execute digital transformation must move data through different layers of traditional systems, such as MES, ERP, CRM, and WMS. This results in a complex connectivity problem called “data spaghetti”.

With the MQTT broker sitting in the middle, facilitating information exchange, UNS moves data in an efficient manner. This is one of the key reasons why we need MQTT to build UNS.

MQTT is Secure

In the context of UNS, PLCs don't need to report polling requests from unknown sources. It only establishes one long-stand connection to a safe broker, which is known to the whole system and is the only data hub that everything needs to talk to.

MQTT broker not only decouples the inputs and outputs of IIoT 4.0, but also takes over the responsibility of securing the connectivity of fragmented devices and systems. Thanks to the rich security features of MQTT, it is fully competent to act as an isolation data hub for your manufacturing lines, preventing any malformed data penetrates into fragile PLCs. In this way, UNS shifts the security burden to the broker and manages it in a unified place.

MQTT Enables Event-Driven Architecture

For IIoT 4.0, data is primarily generated by devices, not people. And most of the time, PLC readings don’t change, it is no point in transmitting redundant data to UNS and overwhelming your network.

However, once the value of registers changes, it is essential to report it to ERP/MES or cloud immediately to extract the maximum value, since the liquidity and marginal value of information decreases along with time. This is called “report by exception“.

Therefore, an event-driven technology such as MQTT is important to implement UNS. MQTT brokers are aware of client state changes. This awareness is enabled with SparkPlugB and the MQTT Last Will features. With last will and SparkPlugB, MQTT only communicates changes and ensures stale data is not delivered to subscribing clients.

MQTT Topic as Metadata of UNS

UNS is the digital reflection of business in the physical world. It is driven by reality, not the other way around. We need the sublime context of generated data to convert it into information, such as the location of site, and which line and cell PLC belongs to. Fortunately, the Pub/Sub messaging model of MQTT provides an easy way of mapping the context of PLC into the digital world.

UNS with MQTT enables users to browse all the namespaces and features. And the real-time status of every data tag.

The MQTT topic is well-defined as follows:

namespace/group_id/message_type/edge_node_id/[device_id]

From OSISoft AF

MQTT Topics have a hierarchical organization and could be easily imagined as a folder structure. By adding your namespace of each level as a topic, we can build a tree view on top of MQTT.

When subscribers consume data from the topic “Enterprise A/Site A/Area A/Process Cell A/Bio Reactor”, it automatically knows where the data origins and takes necessary action upon it. And they could use wildcards to subscribe to multiple datapoints at the same time.

MQTT could help you define metadata that assures consistency and accuracy across different systems in UNS.

OPC UA: Alternative to MQTT for Building UNS

Precisely speaking, MQTT is the best partner for building UNS. However, it is not compulsory. Therefore, you could build the UNS with other protocols, such as OPC UA and HTTP, or even plain Modbus + TCP/UDP. Among many options, OPC UA remains a popular methodology for IIoT due to the strong commercial backup and the standard's longevity.

The detailed comparison between them can be found at:

A Comparison of IIoT Protocols: MQTT Sparkplug vs OPC-UA

Efficiency Comparison: OPC-UA, Modbus, MQTT, Sparkplug, HTTP

However, it is not one or the other case; the best approach is to combine the advantages from both sides by bridging them together, or using OPC UA via MQTT approach.

The General Architecture of Building UNS with MQTT

Theoretically, UNS is a global namespace that unifies all namespaces of your business. Each factory, each site, and each PLC is a stand-alone namespace.

UNS is to take technology across the business and structure it in a way you can navigate. Thus all other solutions could be built on top of it to their best effort. MQTT is responsible for connecting fragmented technologies from OT, CT and the IT world and seamlessly integrates them together.

Conclusion

In this blog, we discussed the significance of MQTT within UNS and why MQTT is the data pump of all IT systems. Adopting MQTT in a unified namespace offers a multitude of benefits that can revolutionize how we connect and communicate in the digital world. Organizations can streamline their data flow, create more responsive and connected systems, and unlock new opportunities for innovation and growth.

In the upcoming episode, we will explore the innovative contributions of EMQ to UNS regarding MQTT.

Originally published at www.emqx.com

AWS MQTT: Managing IoT Communication in AWS (with Tutorial)

EMQ Technologies — Mon, 18 Sep 2023 15:04:02 +0000

What Is AWS?

AWS, or Amazon Web Services, is the world’s leading cloud platform. It provides a mix of infrastructure as a service (IaaS), platform as a service (PaaS), and packaged software as a service (SaaS) offerings. AWS provides a variety of services that can be used as building blocks to run organizational workloads.

Running your applications in the AWS cloud can help you move faster and save substantial costs, benefiting from the scale and performance of AWS. AWS powers a wide variety of workloads including web and mobile applications, enterprise applications, data processing and warehousing, storage, and many others.

AWS is known for its wide range of features and capabilities. These include on-demand computing power, database storage, content delivery, and advanced analytics capabilities. It allows organizations to tap into a global infrastructure that can serve almost any computing requirement, on-demand and without having to deploy resources on-premises.

What Is MQTT?

MQTT, or Message Queuing Telemetry Transport, is a simple and lightweight messaging protocol designed for constrained devices and low-bandwidth, high-latency networks. The design principles of MQTT are to minimize network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery.

MQTT is often used in Internet of Things (IoT) applications due to its efficient use of network resources, its low power usage, and its minimized data packets. It is most useful in remote locations where a small code footprint is required, or where network bandwidth is at a premium.

The MQTT protocol works by exchanging a series of MQTT control packets in a defined way. The client and server send each other packets to keep the connection alive, to publish data, to acknowledge receipt of data, and to adjust the quality of service (QoS) level of the connection. MQTT is a publish/subscribe messaging protocol designed for lightweight M2M communications. It was originally developed by IBM in the late 1990s and has been used in a variety of IoT settings.

This is part of a series of articles about IoT in the cloud.

AWS IoT Core and MQTT

AWS IoT Core is a managed cloud service that lets connected devices interact with cloud applications and other devices. AWS IoT Core fully supports MQTT, as well as HTTP and WebSockets. With AWS IoT Core, applications can keep track of and communicate with IoT devices, even when they aren’t currently connected.

The AWS IoT Core service allows for secure, bi-directional communication between Internet-connected things (such as sensors, actuators, embedded devices, or smart appliances) and the AWS Cloud. This enables you to collect telemetry data from multiple devices, and store and analyze the data. You can also create applications that enable your users to control these devices from their phones or tablets.

For more information on the features and capabilities of AWS IoT Core, see the official documentation.

Learn more about AWS IoT Core

AWS MQTT Pricing

The cost of using AWS IoT Core is mainly determined by the number of messages published to AWS IoT and delivered to devices or applications. A message is a 512-byte block of data processed by AWS IoT. The total number of messages is the sum of all inbound and outbound messages.

The pricing for AWS IoT Core is based on the number of messages published to AWS IoT (message ingest) and delivered by AWS IoT to devices or applications (message delivery). You are billed separately for each. As of the time of writing, the first 250 million messages per month are charged at $1.00 per million messages, and the price decreases with increasing volume.

However, it's important to note that AWS offers a Free Tier for AWS IoT Core, which includes 250,000 free messages (published or delivered) per month for the first 12 months. After the first 12 months or above the free tier limit, standard AWS IoT Core pricing applies. For up-to-date pricing, refer to the official pricing page.

Quick Tutorial: Using AWS IoT Core with MQTT

This brief guide will show you how to create your first 'thing' object in the AWS IoT ecosystem, connect a device to this object, and monitor the MQTT messages generated by the device. You’ll download software to the IoT device that runs an interactive tutorial, demonstrating the AWS IoT Core service.

Please note:

If your device doesn't have a compatible browser, carry out this tutorial on a computer and transfer the downloaded files to your device.
Your IoT device must be able to interact with port 8443 on your AWS account’s device data endpoint.

To run the interactive tutorial on your IoT device:

From the browser on your IoT device, sign in to AWS IoT Console and select Connect > Connect one device.
Prepare your device according to the instructions and register a new 'thing' under the Register and Secure Your Device section. Choose a unique name for your 'thing' and provide additional configurations if needed. Note that a 'thing' name cannot be modified once created. 📷 Source for this and the following image: AWS
In the Choose Platform and SDK section, select your preferred platform and SDK language. For this tutorial, select the Python SDK on Linux/OS X platform.
AWS IoT now provides a connection kit including various critical files. Download it to your device. Unzip the file by running unzip connect_device_package.zip or an equivalent command.
Open up a terminal window on your device and follow the instructions in the AWS IoT Console. The console will provide a command to run in the directory containing the connection kit file.
After you enter the command, you should see an output in the device's terminal or command window that is similar to the following. This output is from the messages the program is sending to and then receiving back from AWS IoT Core. 📷
While the sample program is running, the test message Hello World! will appear in the terminal on the device.
If you want to see messages from your IoT client in the AWS IoT console, from the Test page of the console, open the MQTT test client.
In the test client, use the Topic filter, and enter a topic such as sdk/test/python to subscribe to messages from your device. The topic filters are case sensitive.
After you subscribe to the test topic, run ./start.sh on your device. Messages should start to appear in the MQTT client, similar to the following. The number in square brackets should increase by one each time a new message is received. { "message": "Hello World!" [1] }

That’s it! You just registered a IoT device with AWS IoT Core and sent MQTT messages to AWS.

Advantages of Using AWS with MQTT

Scalability and Performance

AWS offers virtually unlimited scalability, allowing businesses to quickly scale up or down as their needs change. This is particularly important for IoT projects, where the number of connected devices can easily run into the thousands or even millions.

In terms of performance, AWS offers a high-speed network backbone, ensuring that data is transmitted quickly and reliably. This is crucial for applications that require real-time data, such as monitoring systems or control applications. Furthermore, AWS's global infrastructure ensures that data can be transmitted and received from anywhere in the world, minimizing latency and ensuring optimal performance.

Security and Compliance

Another key advantage of using AWS with MQTT is the strong security and compliance measures it provides. AWS offers a range of security features, including encryption at rest and in transit, fine-grained access control, and the ability to create private networks within the cloud. This ensures that your data is protected at all times, reducing the risk of unauthorized access or data breaches.

In terms of compliance, AWS has been certified against a wide range of international and industry-specific standards, including ISO 27001, PCI DSS, and HIPAA. This means that businesses can confidently use AWS for their MQTT projects, knowing that they are in compliance with these stringent standards.

Integration with Other AWS Services

AWS with MQTT also benefits from seamless integration with other AWS services. For example, data transmitted via MQTT can be stored and analyzed using AWS's data storage and analytics services, such as S3 and Redshift. This allows businesses to easily build comprehensive IoT solutions, combining real-time data with sophisticated analytics.

Furthermore, integrating MQTT with other AWS services can help to streamline operations and reduce costs. For example, by integrating MQTT with AWS's Lambda service, businesses can run code in response to MQTT messages, without the need to provision or manage servers.

Managed Service Benefits

Finally, using AWS with MQTT provides the benefits of a managed service. AWS takes care of all the infrastructure, maintenance, and management, allowing businesses to focus on their core competencies. This reduces the complexity of managing an IoT project, saving time and resources.

In addition, managed services provide a level of reliability and uptime that would be difficult to achieve with an in-house solution. AWS's extensive infrastructure and redundancy measures ensure that the service is always available, minimizing the risk of downtime.

4 Best Practices for AWS MQTT Implementation

Here are a few best practices that can help you make the most of your MQTT implementation in AWS.

1. Proper Configuration and Tuning

When implementing AWS with MQTT, make sure you set up the right security measures, such as enabling encryption and setting up access controls, to protect your data.

Additionally, tuning the performance of your MQTT broker is crucial to ensure optimal performance. This includes adjusting the settings for maximum connections, message size, and message rate, among others. Remember, the goal is to achieve a balance between performance and resource usage.

2. Monitoring and Logging

AWS provides several tools for monitoring your MQTT broker, such as CloudWatch, which provides real-time metrics and alarms, and X-Ray, which provides insights into the behavior of your applications.

Logging, on the other hand, is crucial for troubleshooting and understanding your MQTT traffic. AWS provides CloudTrail, which logs all API calls, allowing you to track who is making calls, the source IP address, and when the calls were made.

3. Security and Compliance Adherence

It is important to regularly review and update your security settings, ensure that your data is encrypted, and keep your software up to date.

In terms of compliance, it is crucial to understand the specific requirements of the standards you are subject to, and to ensure that your AWS MQTT implementation meets these requirements. Regular audits and assessments can help to ensure ongoing compliance.

4. Resource Management

Resource management includes monitoring your resource usage, managing your costs, and ensuring that you have the necessary resources to scale as your needs change.

AWS provides several tools to help with resource management, including Cost Explorer, which provides detailed insights into your costs, and Trusted Advisor, which provides recommendations for optimizing your AWS resources.

AWS IoT Core + EMQX: A Better Choice for IoT Communication

Working with AWS IoT Core can raise some challenges related to management of large-scale MQTT communications. EMQX Cloud can help, with a comprehensive MQTT service that is fully managed and customizable to the needs of users. This includes flexible pricing models, customizable connection specifications, and the freedom to select a cloud provider and integrate with other cloud resources.

Compared to the AWS IoT, EMQX Cloud, can provide a complete solution to match the AWS IoT ecosystem.

EMQX Cloud Products

BYOC (Bring Your Own Cloud): Seamlessly integrates the EMQX MQTT server with Amazon Web Services or other existing cloud services. Keep your data secure in your own cloud and manage it with EMQ's expertise.
Serverless: Provides MQTT services on a secure and scalable cluster with usage-based pricing. The service is completely free within the free quota and supports up to 1000 concurrent connections.
Dedicated Plan: Provides MQTT services on a dedicated EMQX cluster with high performance, reliability, and data integration, without managing your own cluster. Perfect for businesses of all sizes.

Tight Integration with AWS Services

EMQX integrates smoothly with Amazon Web Services:

EMQX Cloud Dedicated cluster can be deployed in 10 regions in AWS, seamlessly integrating with existing resources in AWS.
EMQX Cloud provides data integration with 40+ services (connectors), including databases, message queues, and AWS services like DynamoDB.
You can subscribe to EMQX Cloud from the AWS marketplace, and manage all service bills in AWS Billing.

Benefits of Using EMQX Cloud

Hassle-Free Management: EMQX Cloud fully manages setup, maintenance, and updates.
Cost-Effective: EMQX Cloud offers a variety of pricing models, such as by consumption or by time, which reduce the total cost of ownership.
100% MQTT: 100% compliant with MQTT 3.1, 3.1.1, and 5.0 standards with all 3 QoS level. Seamless integration with all MQTT clients.
Seamless Scalability: Scale up or down on demand without any worries about server capacity or performance.
Up to 99.99% SLA: EMQX Cloud Ensure maximum uptime and availability with our highly available, fault-tolerant architecture.
24 x 7 Technical Support: A dedicated team of experts is always available to assist you with any issues at any time.

EMQX Cloud vs. AWS IoT Core

The table below compares AWS IoT Core with EMQX Cloud in various aspects:

Originally published at www.emqx.com