DEV Community: HiveMQ

Announcing HiveMQ Pulse: The Distributed Data Intelligence Platform

HiveMQ — Tue, 18 Feb 2025 14:00:00 +0000

Author of the blog: Gaurav Suman,Director of Product Marketing at HiveMQ

Over the past several years, there has been a fundamental shift in the requirements for data management. The old approach—rigid, hierarchical structures and siloed systems—can no longer support the demands of modern, data-rich use cases. There’s an ever-increasing demand for productivity, plus the constant pressure of training and retaining the talent needed to deliver on the promises of Industry 4.0 and beyond.

Over the last few weeks, we’ve been showcasing an approach that’s helping forward-thinking companies tackle these challenges: the Unified Namespace (UNS). We’ve learned how it should be understood, how to implement its foundational elements, and crucially, why a single technology or protocol alone (even MQTT) can’t fulfill everything a UNS promises.

HiveMQ has powered UNS implementations at some of the world’s largest energy, pharmaceutical, and logistics organizations. We’ve seen three persistent challenges: first, the difficulty of creating a truly unified, interoperable data model across diverse systems; second, the need for real-time on-location data insights that don’t require perfect connectivity or round trips to the cloud; and third, the pressure to ensure robust security, governance, and compliance at every step. Conventional bolt-on or proprietary platforms miss one or more of these needs, forcing enterprises to compromise.

We kept hearing the same questions from our customers: Can we get a real-time view of our operations, keep control of our data, and make sure the right people have access to the right information—whether at the edge or in the cloud? The short answer is “yes,” and it’s why we’re excited to introduce and announce a private preview of HiveMQ Pulse.

What is HiveMQ Pulse?

HiveMQ Pulse is a real-time, distributed data intelligence platform that delivers the best of both worlds—bringing intelligence to the edge for immediate decision-making while ensuring centralized governance and interoperability across the entire data ecosystem. It allows enterprises to manage, transform, and govern distributed data while maintaining a single, structured view of the enterprise. By historicizing and governing in-flight messages, enabling real-time queries and compute tasks, and applying contextual information, Pulse ensures that data is not only available but also actionable for AI and other critical use cases.

Why Enterprises Need Distributed Data Intelligence

Both IIoT and IoT data have increased in complexity with rapid growth in devices, legacy systems, and disparate data formats, making it harder for enterprises to unify their data in a structured, actionable way. Without a scalable, intelligent approach to managing this complexity, businesses risk falling behind.

Faster access to insights means faster time-to-market and greater ability to make adjustments that prevent downtime, reduce costs, and ultimately increase profits. Research shows that businesses operating in real-time environments achieve 62% higher revenues and make decisions 30% faster, proving that the ability to process, analyze, and act on data as it’s generated directly translates to business success.

That’s why data-driven decision-making is no longer optional—it’s a competitive necessity. Industry leaders recognize that adaptability is key, and organizations that successfully harness real-time intelligence see measurable advantages. This is where HiveMQ Pulse comes in—delivering the unified data management, real-time insights, distributed intelligence, and AI-readiness that enterprises need to succeed.

Unified Data Management

The Unified Namespace pattern has gained significant traction, and HiveMQ has worked with many customers to implement UNS at scale on our MQTT Platform. While MQTT is an ideal technology—and HiveMQ is the ideal broker—for transporting data reliably at scale, there is an absence of tooling for managing the metadata and context of a Unified Namespace. HiveMQ Pulse fills this gap by providing a structured way to manage UNS data transformations efficiently.

Since a UNS approach ensures that all data across an enterprise is organized under a single, centralized data structure, many companies mistakenly assume a data warehouse or traditional enterprise data platform can serve as their UNS. This is not the case. Unlike traditional architectures where transformations and data modifications are performed at individual sites or edge gateways, a UNS enabled by HiveMQ Pulse allows for changes to be made at the namespace level. The underlying system then orchestrates where those changes need to take effect.

For example, instead of manually modifying a data tag at a specific site, the user updates the Unified Namespace. The system then ensures the transformation occurs at the relevant location automatically, eliminating the need to interact with individual gateways. This declarative approach is core to HiveMQ Pulse, enabling it to autonomously determine the type and location of operations to execute when the namespace is modified.

Actionable Insights

Actionable Insights are real-time analytics and calculations that occur as close to the data source as possible, rather than relying on cloud-based processing. This ensures immediate feedback and response, reducing latency and dependency on centralized infrastructure.

For instance, an operational efficiency metric such as Overall Equipment Effectiveness (OEE) can be calculated on-site from existing performance, quality, and availability data without sending raw data to the cloud. This capability enables businesses to derive real-time insights and take corrective action instantly. The ability to generate and apply insights at the point of need is a fundamental shift in data intelligence.

Distributed Intelligence

Distributed Intelligence refers to the decentralization of computing power, ensuring that intelligence and decision-making occur at the appropriate edge locations rather than relying on a central processing hub. This architecture reduces bottlenecks, improves response times, and ensures resilience.

With a distributed intelligence approach:

The Unified Namespace is defined in a centralized plane, but the execution of logic, transformations, and processing happens at the local nodes.
Edge devices autonomously learn and execute tasks based on intelligence that is centrally defined but dynamically distributed, eliminating the need for explicit coordination.

For example, if a script for converting Celsius to Fahrenheit needs to run at a specific filling line, the system automatically determines where it should execute. The user does not need to specify the site or agent responsible for running it—the system handles the orchestration at the edge.

Powers AI

Powering AI means allowing for easy AI model integration and ensuring that data is contextualized, structured, and accessible in a way that supports AI and machine learning applications. The key aspects of an AI-ready system include:

Rich metadata: Every data point is tagged with units, context, and historical changes, ensuring AI models can properly interpret and analyze it.
Lookup capabilities: Data entries include IDs and relationships that allow for easy traceability and contextual understanding.
Future AI integrations: The system is designed to seamlessly integrate with AI models and automation tools by maintaining high data quality and integrity.

For instance, if a dataset containing machine sensor data is stored in a data lake, an AI model can quickly retrieve associated contextual information—such as units, location, equipment type, and past trends—to generate accurate predictions and recommendations.

Under the Hood: The Technology Behind HiveMQ Pulse

HiveMQ Pulse is based on a distributed architecture that overlays an enterprise MQTT deployment. The strength of HiveMQ’s proven MQTT platform is now coupled with an agent-based architecture that captures and shares insights across the enterprise.

The HiveMQ Pulse distributed intelligence platform includes the following components in its deployment.

Pulse Server

Manages information models, authorizes Agents, and orchestrates queries. Scalable for policy enforcement, enables high-throughput processing, and seamless OT-IT integration.

Pulse Agent

Indexes and processes edge data with a distributed calculation engine. Filters, historicizes, and governs in-flight messages while enabling real-time queries and compute tasks. The agents can be deployed on/alongside standards-based endpoints and brokers.

Pulse Client

Secure web app/GUI for managing data models, policies, and queries. Supports UNS modeling, structured data interaction, and simple dashboards for actionable insights.

Together, these components deliver a truly seamless experience where the needs of users and those building and running advanced capabilities can be served equally effectively, and in a distributed manner.

Final Thoughts: A New Era of Distributed Data Intelligence Begins Today

As operations grow more complex, businesses need solutions that don’t just collect data but make it immediately useful—without sacrificing security, governance, or flexibility. HiveMQ Pulse is built to address exactly that need. By delivering real-time intelligence at the source, ensuring data sovereignty, and enabling seamless edge-to-cloud governance, Pulse represents a fundamental shift in how data is accessed, processed, and acted upon.

With unified data management, actionable insights, distributed intelligence, and powering AI use cases, HiveMQ Pulse is more than just a UNS enabler—it’s a new way to think about data architectures. It eliminates data silos, creates true IT-OT convergence, and empowers teams with real-time visibility, ensuring enterprises are ready for the AI-driven future without unnecessary complexity.

Now is the time to move beyond bolted-on, centralized, and reactive data solutions. The industry is changing, and businesses that can act on insights faster, ensure resilience, and maintain control over their data will have the competitive edge.

Get Started with HiveMQ Pulse

HiveMQ Pulse is now available in private preview. Sign up today to explore how it can revolutionize your data strategy.

Identifying, Acquiring and Integrating Plant-Floor Data for Smart Manufacturing

HiveMQ — Thu, 22 Jun 2023 15:27:24 +0000

The success of modern manufacturing enterprises relies heavily on the ability to collect, analyze, and act upon data. It is, therefore, crucial to pinpoint potential data sources and determine the most efficient methods for acquiring data from predominantly legacy systems to achieve desired outcomes. Ensuring a cost-effective, scalable, and replicable solution is essential. Additionally, it is vital to aggregate the collected data to a level where it can be seamlessly integrated with external enterprise systems.

In manufacturing environments, a diverse range of data is generated from various sources on the plant floor, each carrying unique importance and objectives. To effectively manage this data, it is crucial to initially identify the available information and determine the appropriate methods for accessing it.

This article serves as the second instalment in a six-part series titled A Comprehensive Guide To Industrial Data Management for Smart Manufacturing, discussing a practical approach to help you begin implementing data management for smart manufacturing. In part-1 of this series, The Power of Data Management in Driving Smart Manufacturing Success we explored how to establish a well-thought-out strategy for harnessing the power of data in smart manufacturing.

Identifying Potential Data Sources for Smart Manufacturing

Building upon this understanding, we can start identifying potential data sources for smart manufacturing implementation by examining the Computer Integrated Manufacturing (CIM) pyramid.

This reference model, developed in the 1990s, provides a framework for implementing industrial automation. It focuses on collecting, coordinating, sharing, and transmitting data and information between various systems and sub-systems through software applications and communication networks.

Below is a list of possible data sources for smart manufacturing implementation and the reasons why.

Programmable Logic Controllers (PLCs) at level 2
Supervisory Control and Data Acquisition Systems (SCADA) at Level 3
Historians at Level 3

Firstly, Sensors, Actuators, RTUs, CNCs, and other field equipment are not suitable as data sources due to their numerous connections to isolated networks managed by outdated protocols. Furthermore, since PLCs primarily collect and provide their data, there is no need to establish direct connections to these components.

On the other hand, PLCs efficiently handle all the data from sensors and devices at lower levels, processing the information based on their respective scan time resolutions, which can be as low as ten milliseconds. This information is typically time-series data but can also include calculations and alarm data. In turn, they make this information available to higher-level applications through communication protocols like Modbus, OPC DA and OPC UA, making great data sources.

SCADA and Historian systems do not gather all the data provided by the connected PLCs. Rather, they focus on collecting the most critical information and data with lower frequency. Initially designed as consumers of industrial data through an OPC client interface, SCADA and Historian systems have also evolved into producers of industrial data. They now fulfill both roles simultaneously by implementing an OPC UA server interface.

Now that we’ve identified our potential data integration sources, let’s examine each source closely, considering their abilities, pros and cons, and connectivity and data-gathering alternatives.

Integrating Data from Programmable Logic Controllers (PLCs)

Within a large industrial facility, numerous PLCs of varying sizes and capabilities can be found, typically arranged in a hierarchical manner. The highest-level PLCs function as data concentrators and are ideal points for data acquisition. Data Concentration PLCs may sometimes relay their information to a standalone OPC UA server, which would then be used as the access point.

In scenarios where you do not have this kind of hierarchical arrangement of PLCs, you’d need to connect to the primary PLC of each working cell, production line, or plant area to collect data.

Pros of Integrating Data from PLCs

PLCs have swift scanning capabilities that ensure a consistent stream of updated data originating from sensors and other production machinery, with a resolution starting at about tens of milliseconds. Most significantly, PLCs stand out for their reliability, stability, and deterministic nature. They guarantee minimal downtime, an essential feature for uninterrupted data collection, thereby avoiding compromised signal quality or disruptions.

Cons of Integrating Data from PLCs

PLCs are situated at the lower levels of the automation pyramid, meaning data collection occurs close to the hardware with limited abstraction. Such a low level of abstraction introduces complexity in managing what could be thousands of process signals, calculations, and alarms. Moreover, tags may have inconsistent naming conventions across various plant areas, depending on the PLC vendor or the control logic developer. Part 3 of this series discusses contextualizing and normalizing this kind of data before integrating it with enterprise systems.

Integrating Data from Supervisory Control and Data Acquisition (SCADA)

SCADA systems can communicate with factory floor field devices using legacy communication protocols, OPC, or Fieldbus protocols. Typically, SCADA applications acquire data at intervals ranging from half a second to one minute. SCADA systems process thousands of signals, utilizing a standardized approach based on essential parameters for accurate information management, often implementing an OPC UA server interface. These parameters include the tag name, a description, the sampling time, a minimum and a maximum value, and engineering units. SCADA systems, therefore, make good access points for acquiring data with some semblance of a data model.

Pros of Integrating Data from SCADA

A SCADA system primarily functions as a data acquisition system, effectively serving as a robust data concentrator. It collects data from various field devices and PLCs, controlling different production lines or functional areas. With SCADA systems adopting a standardized approach to handling industrial data, often based on a common data model, it simplifies identifying and recognizing data streams required for smart manufacturing implementation.

Furthermore, given their frequent need to interface with MES and ERP systems, they are typically already integrated into the plant or corporate network, which eases the process of transferring data to the enterprise level, thereby streamlining overall data management.

Cons of Integrating Data from SCADA

Compared to PLCs, SCADA systems tend to be less reliable due to several factors. Regular updates to SCADA systems often necessitate application restarts and reboots of the Windows operating system for security patches or installations. Additionally, the modular architecture of SCADA systems may lead to overloads that could disrupt communication tasks essential for data integration with enterprise applications.

Further, direct connection to a SCADA system often requires communication via its API or SDK, which may demand substantial effort to maintain, and update connectors for various SCADA systems while ensuring compatibility. Like PLCs, an alternative solution involves connecting to them through an OPC UA server.

Integrating Data from Historians

The Historian’s ability to interface with various common industrial protocols and Fieldbuses enables it to collect data from various plant-floor devices and systems and log it as time-series data. This data is crucial for tracking and analyzing the performance of machines and systems, and detecting anomalies, which makes it a potential source for data integration.

Pros of Integrating Data from Historians

Historians arrange time-series data using a hierarchical model associated with the asset. This model structures data like branches on a tree, creating an organized, logical system for pinpointing and accessing specific data points.

Much like SCADA systems, Historians are often already integrated into the plant or corporate network, simplifying transferring data to the cloud and enhancing overall data management efficiency. However, unlike SCADA systems, Historians come with superior reliability features, such as built-in data buffering capabilities and store-and-forward mechanisms. This in-built resilience to network disruptions or unforeseen downtime ensures the data’s reliability, provided the source device is available.

Cons of Integrating Data from Historians

Historians are designed to optimize data sampling and storage, but they might only gather a portion of the data, limiting the overall data availability. Additionally, data obtained from Historians is in its raw form and does not deliver a time-specific snapshot of an asset - which is an essential aspect of cloud-based analytic applications.

Like PLCs and SCADA, directly connecting to a Historian using its SDK or API poses a maintenance challenge. Again, the alternative could be to connect to it through an OPC UA server.

Integrating Plant-Floor Data to The Enterprise

When gathering data from established industrial systems, you’re bound to accumulate information from diverse sources based on the advantages and disadvantages I’ve previously mentioned. Regardless of the device acting as a data source, a software layer must be in place that communicates with this source via its unique protocol. This layer should be able to request Tag and Time-Series data, making it accessible to higher levels and external systems through a single standardized interface, typically OPC UA.

Direct connections to PLCs, CNCs, SCADA, Historians, etc., mean your Edge IT infrastructure has to set up and manage multiple protocol endpoints. This approach is neither scalable, secure, nor reliable. Instead, it’s preferable to utilize a connectivity platform to simplify the management of diverse interfaces and protocols, thereby enhancing data integration efficiency.

KEPServerEX, for instance, could be an appropriate choice for efficient data management. It can interact with various devices and machines, irrespective of the manufacturer, and supports numerous communication protocols. Its unique feature exposes your plant-floor data for enterprise integration through a single interface, OPC UA. This significantly eases the process of integrating data into enterprise applications.

Data in a standardized communication interface like OPC UA can be integrated into the enterprise network using a communication protocol like MQTT. MQTT is ideal for this stage of integration for several reasons:

Scalability: MQTT’s publish/subscribe model is scalable, especially when dealing with many devices, making it a good fit for extensive manufacturing setups or those expected to grow significantly.

Cloud Integration: MQTT is a popular choice for cloud integration due to its native support on many IoT platforms. If manufacturing data needs to be integrated with a cloud platform, converting it to MQTT can simplify this task.

Real-time Data Processing: MQTT is suitable for real-time data processing thanks to its lightweight and real-time capabilities. This is advantageous in situations where immediate insights and swift decision-making are crucial.

To enhance the quality and speed of real-time insights, it’s imperative that we first contextualize, normalize, and model the acquired and aggregated data before integrating it. Furthermore, to establish the comprehensive context necessary for implementing smart manufacturing, it’s crucial to integrate your plant floor data with various platforms. These include Manufacturing Execution Systems (MES), Enterprise Resource Planning systems (ERP), and Laboratory Information Management Systems (LIMS), among others. As such, the necessity of introducing DataOps to operationalize data management at this layer becomes clear. This topic will be our main focus in Part 3 of this series.

Conclusion

In this article, we have navigated through identifying data sources and integration opportunities as a first step to implementing a data management strategy for smart manufacturing. We addressed identifying sources of plant-floor data and the procedures for data acquisition. We also highlighted the methods to integrate this data, making it accessible via a standardized interface for enterprise integration.

In Part 3 of this six-part series titled A Comprehensive Guide To Industrial Data Management for Smart Manufacturing, we delve into the methods of transforming, standardizing, normalizing, and modelling the collected data. This process is crucial to ensure that the data can be correctly understood and interpreted, thus enhancing its quality and usefulness.

About the Author

Kudzai Manditereza is a Developer Advocate at HiveMQ and the Founder of Industry40.tv. He is the host of an IIoT Podcast and is involved in Industry4.0 research and educational efforts.

The Power of Data Management in Driving Smart Manufacturing Success

HiveMQ — Thu, 25 May 2023 05:00:00 +0000

Smart Manufacturing encompasses a diverse range of priorities and objectives for companies in the industry. For some, it’s about innovating customer service to stay ahead of emerging competitors, while others focus on improving quality and cost performance. Despite the differing objectives, every smart manufacturing pursuit emphasizes the vital role of data in realizing their desired outcomes.

However, many businesses embark on their smart manufacturing or industry 4.0 journey with isolated digital projects, lacking a comprehensive data management strategy. While these projects may produce positive results, they eventually give rise to various challenges that necessitate reevaluating the approach. Issues include a complex network of digital technologies, an array of disparate solutions that are challenging to scale, an uncoordinated and frequently inefficient digital infrastructure, and escalating costs associated with digital investments arise.

This article serves as the first installment in a six-part series that will explore how to establish a well-thought-out strategy for harnessing the power of data in smart manufacturing. By providing an overview of the importance of data management and its role in driving success, we lay the groundwork for the subsequent articles in this series, which will delve deeper into specific data management techniques.

Aligning Data Management with Business Objectives

It is essential to emphasize that the key to successfully implementing a comprehensive data management strategy is ensuring that it aligns with and supports the overarching business strategy. In many manufacturing businesses, there’s a continuous effort to create consistency in both business and operational methods. This process started by introducing a shared ERP system at the company level and has since expanded to include process and manufacturing operations.

A significant obstacle in promoting uniformity across operations lies in the vast diversity present among physical production facilities. Consequently, numerous companies are identifying clusters of similar operating technologies as a starting point for establishing a degree of consistency. Then they employ data-driven key performance indicators (KPIs) like Overall Equipment Effectiveness (OEE), Lead Time, Time to Resolution (TTR) for quality issues, Product Development Costs, and Supply Chain Cycle Time, among others, to measure the level of achievement for established objectives.

However, it is important to recognize that implementing this measurement system can be more complex than initially anticipated. It goes way beyond developing a weekly or monthly dashboard. Rather, it involves bringing measurement closer to real-time and proactively driving actions that impact business performance. As previously highlighted, the data used for calculating and summarizing KPIs are often intricate, relying on information from numerous sources that do not normally talk to each other. Therefore, a well-orchestrated data management approach is essential in facilitating the automated calculation and aggregation of such KPIs across the organization.

As you will discover in later parts of this series, it is critical that performance calculations utilize trusted data that represents the single source of the truth and, whenever possible, not created in a manual process that is subject to individual bias. The enables autonomous calculation and aggregation of data into a higher-level enterprise data structure, a Unified Namespace.

Addressing the Challenges of Plant-Floor Data

As already ascertained, plant-floor data can potentially drive significant business outcomes, but it often presents challenges due to its raw and unstructured nature. Originally designed for process control, this data can appear in various formats, such as machine, transactional, or time series data, and it lacks context and standardization. Simply, plant-floor data is not immediately compatible with cloud-based enterprise applications. As a result, managing the integration of this complex data to derive meaningful insights from it can be daunting. It requires a data management approach that enforces standardization and repeatability across a manufacturing enterprise.

In addition, data quality remains the foremost challenge for manufacturing companies’ analytics initiatives, with teams often dedicating a significant portion of their time to data preparation and cleaning. Common data quality emanating from manufacturing operations includes issues such as missing or incorrect data, inconsistent data, unsuitable formats, duplicated data, etc. These can be addressed by implementing standardized governance processes.

In addition, legacy industrial systems pose several challenges for smart manufacturing analytics due to their outdated technologies, lack of connectivity, and resistance to change. Some of the key challenges include:

Integration difficulties: Integrating legacy systems with modern smart manufacturing technologies can be complex, time-consuming, and costly. These systems often lack standardized interfaces and use proprietary communication protocols, which is challenging when integrating newer technologies.

Data access and compatibility: Legacy systems often have limited data storage and retrieval capabilities. Extracting, processing, and integrating data from these systems into smart manufacturing platforms may require extensive data manipulation and transformation, which can be time-consuming and error-prone.

Resistance to change: Organizations that rely on legacy systems may face resistance from employees accustomed to using the old systems. This can slow down the adoption of smart manufacturing technologies and hinder overall progress.

A well-defined data management strategy can help alleviate these challenges by developing a systematic approach to integrating data from legacy systems with newer smart manufacturing technologies. This may involve using middleware, data connectors, or custom-built APIs to bridge the communication gap between systems and enable seamless data exchange. Further, data standardization and transformation can be enforced by establishing standardized data formats and structures to facilitate smooth data exchange and processing across different systems.

By implementing a comprehensive data management strategy, organizations can overcome many of the challenges of legacy industrial systems and create a strong foundation for the successful adoption of smart manufacturing technologies.

Leveraging Semantic Data Representation for Interoperability

Implementing capabilities to support a manufacturing company throughout its production cycle is crucial. This includes determining what products to create, the required materials, the production location, specific operations and equipment, and the quality-critical process parameters. Additionally, analyzing past performance is essential for continuous improvement.

It is vital to establish a robust data infrastructure, including accurate models for materials and product families and quality and production-specific specifications to achieve this. Furthermore, it is necessary to consider the production units themselves, ensuring a high degree of repeatability by defining machinery categories or types and setting up instances of those types. Connecting the production unit’s performance specifications with material definitions and quality, food safety, or other compliance-related factors is also important.

Having a strong data model is the foundation for building various tools and systems around it. If the data infrastructure is weak, developers might have to compensate for the shortcomings with additional code in the automation or reporting layers. However, if the data model surrounding materials, processes, production units, and personnel is well-structured, creating custom user experiences or reports becomes significantly easier due to the presence of logical structures. In summary, investing in a solid data model is essential for streamlining operations and supporting growth in the manufacturing industry.

Enabling Seamless Exchange of Data Across the Enterprise

At the heart of smart manufacturing lies data transfer between data producers and consumers for performance analysis and occasionally back to the producers for corrective measures. Smart manufacturing inherently demands integrating data from a diverse range of enterprise components, vendors, and domains in an easily manageable manner. As a result, the effectiveness of a smart manufacturing initiative is directly linked to the openness, flexibility, and scalability of your data exchange architecture.

You can significantly enhance your smart manufacturing initiative by carefully crafting your data management strategy, particularly regarding data exchange capabilities. This will establish a strong foundation upon which you can continuously add and interchange components without constraints. In subsequent sections of this series, we will explore how the publish-subscribe architectural pattern for data exchange, in which data consumers are decoupled from data producers, simplifies data integration for smart manufacturing.

In addition, a key challenge in data integration is the disparity in data exchange methods between the Information Technology (IT) and Operations Technology (OT) domains. Messaging protocols and message formats vary between these domains. Therefore, a well-orchestrated data management strategy will enable you to choose a data transportation and messaging format that effectively spans both domains.

Assessing Digital Capability and Maturity Assessment

Before developing a comprehensive data management strategy for smart manufacturing, it is essential to assess your present digital capability and maturity level—which may vary significantly across sites and business functions—and your desired future state. Digital capability refers to the extent of digital technology available within your company, while digital maturity represents your organization’s readiness to utilize these technologies effectively.

It is self-evident that embarking on a smart manufacturing journey demands a growing level of digital capability and maturity throughout the organization. Therefore, comprehending digital capability and maturity is vital as your company decides on the next steps for laying a data management foundation for smart manufacturing implementation.

When assessing your manufacturing enterprise’s digital capability and maturity, it is crucial to examine various factors, comparing the current state to the ideal target state. These factors and guidelines can help manufacturing companies understand their current position and work towards enhancing their digital capabilities.

For example, you can start by assessing whether data is being collected, stored, and shared effectively across the organization. Your target would be recording and historizing data from all devices and establishing common data definitions across plants. You can also evaluate how easily real-time and historical data can be accessed and ensure its accuracy. Aim for easy access to accurate data, minimizing manual collection or manipulation.

Conclusion

This article explored the importance of effective data management in achieving business objectives through smart manufacturing. We delved into how it tackles the issue of diverse plant-floor data, promotes interoperability through semantic representation, and enables smooth data exchange across the entire enterprise. We also discussed methods for assessing your digital capabilities and maturity to formulate a strategic data management plan.

Check out Part 2 on Identifying, Acquiring and Integrating Plant-Floor Data for Smart Manufacturing, where we take a practical approach to help you begin implementing data management for smart manufacturing. We’ll guide you in identifying data sources and walk you through the process of acquiring and aggregating data from your manufacturing operations to facilitate the implementation of smart manufacturing.

About the Author

Kudzai Manditereza is a Developer Advocate at HiveMQ and the Founder of Industry40.tv. He is the host of an IIoT Podcast and is involved in Industry4.0 research and educational efforts.

Finding the Right MQTT Platform for IoT Data Movement

HiveMQ — Thu, 23 Mar 2023 10:13:12 +0000

Over the past several years, we’ve seen unprecedented growth in the connected world and data driven decision-making. Chances are you found this guide because you are trying to figure out how to effectively use data to build new connected products, achieve more efficient operations, or improve the customer experience. Successful IoT projects and digital transformation depend on having the right data in the right place at the right time.

No matter what you want to do with your data, or whether you are an IT Architect, an Engineer, or a Head of Digital Transformation, you need a plan for putting the right technology infrastructure and IoT protocols in place for reliable, scalable, and secure data movement.

We’ve written a guide to help you make an informed purchase decision for a data movement platform that meets your technical and organizational needs.

Reading this guide will help you understand:

The enterprise challenges and priorities for moving data
Why MQTT is the de facto standard and ideal protocol for IoT
The MQTT platform landscape today
How to lay the right foundation for a data driven enterprise

Get your copy of 2023 Buyer's Guide on MQTT Platform.

Send MQTT Messages in 10 Minutes Using HiveMQ Cloud Web Client

HiveMQ — Tue, 14 Feb 2023 23:00:00 +0000

** In 2022, we introduced a web client to help you easily connect to the HiveMQ Cloud broker. This blog will show you how to get started with MQTT project using this functionality and send messages in less than 10 minutes.**

The HiveMQ Web Client

An MQTT client is any device (from a microcontroller to a full-fledged server) that runs an MQTT library and connects to an MQTT broker over a network. The HiveMQ Cloud web client is a ready-to-use client within the HiveMQ Cloud user interface that helps you connect to your devices easily with the HiveMQ Cloud Broker.

How to Access the Web Client?

Once you log-in to your HiveMQ cloud and select the cluster of your choice, you can view the Web Client option on the top menu bar, as seen in the image below. You can access it by simply clicking on the navigation item.

Connecting to MQTT Broker / Cluster

To securely connect to the client, you must use your credentials, i.e., a username and a password. These credentials are used to allows MQTT clients access to your MQTT broker. Manage your credentials inside the HiveMQ Cloud console under the, “Access Management.”

If you already have credentials, simply enter them and click on the “Connect Client” button.

If you don’t have credentials, no worries; you can create your credentials in a single click using the “Connect With Generated Credentials” button. Once you click it, you will get a pop-up sharing the credentials. Copy and save this for later use.

Clicking on the create and connect button, automatically creates credentials and connects the client to the broker.

Once connected, you will see “Web-Client Connected” displayed in green below the connection setting, as seen in the image below.

Send Your First Message

Now that you have successfully connected the web client to the HiveMQ Cloud broker, you are ready to send your first message.

First, subscribe to a topic you want to receive messages from. You can use a specific topic name as displayed in the screenshot, or you can subscribe to all topics, using the wildcard “#”.

You can learn more about MQTT Topics, wildcards, and some best practices here.

Next, select the quality of service (QoS) you want to receive messages with. If you do not have a specific use case where you need a higher QoS, you can keep the default QoS 0 setting.

As you can see in the image, we create a topic subscription:

my/test/topic

Once we hit the subscribe button, we can view the subscribed topic below as a list.

To receive messages, we need to publish a message with this topic. Make sure to use a topic that matches the subscription you defined earlier.

my/test/topic

Your message can contain any information you like. In our case, we wrote a simple “Hello,” as seen in the image below. But you can also send a JSON payload, or any other format that suits your needs.

Once complete, press the publish button. The message will be published to any client connected to your broker that has subscribed to this topic.

You can view this message inside the web client with the corresponding topic, QoS, and timestamp below the publish button.

You can also connect from any other device to your HiveMQ Cloud cluster using the same credentials and use it to subscribe to topics from the Web Client. In our case, we can use MQTT CLI (Command Line Interface) to receive messages sent using GUI. To learn more about MQTT CLI, please check out HiveMQ Cloud getting started with MQTT CLI section.

If you’ve followed these steps, you should now be able to connect your web client to your HiveMQ Cloud cluster and use it to publish and receive messages.

Get Started

To access the HiveMQ Cloud Web Client, all you need to do is sign up for HiveMQ Cloud.

Sign-up for HiveMQ Cloud

Were you able to successful follow these steps to connect your web client to your HiveMQ Cloud cluster? Did you have any problems along the way? Did you make/take any alternate steps?
Let us know your thoughts in the comments.

How to Stream Data Between HiveMQ Cloud and Apache Kafka for Free

HiveMQ — Thu, 09 Feb 2023 11:07:00 +0000

According to a study published by Statista, IoT devices will produce 79 zettabytes of data in 2025, which will be a 483% increase from 2019. To put this number into perspective, if we store this information in smartphones with a storage of 128 GB each, we would need 617.1875 billion smartphones. Yet, without further processing, this data is worth almost nothing. Only by transforming and analyzing this data do you unlock the immense added value promised by the Internet of Things (IoT).

A common question is how do you actually process the data collected from IoT devices. There are several ways to do this, but one of the most compelling is using MQTT protocol to send IoT data via Apache Kafka for further processing in a system of your choice.

MQTT and Apache Kafka are often used together to enhance the functionality of IoT and Machine-to-Machine communications. You commonly see them married in the following use cases:

Data collection: MQTT is used to collect data from IoT devices and publish it to a Kafka broker, where it is processed, analyzed, and stored for future use.
Real-time processing: Using MQTT and Kafka, organizations build real-time data processing pipelines that handle large amounts of incoming data from IoT devices.

The easiest way to process data from your IoT devices to your Kafka service is our newly introduced Kafka integration with HiveMQ Cloud.

In this blog, we will show you the key capabilities of the Kafka-HiveMQ Cloud integration, how you can use it to stream your data, and walk you through how you can set it up.

The HiveMQ Cloud Kafka Integration

Before we jump into the step-by-step instructions, let’s look at the benefits of the Kafka-HiveMQ Cloud integration. This simple configuration enables you to stream your data efficiently between your HiveMQ Cloud broker and your Kafka cluster for bidirectional message exchange without ongoing operational burden.

There are five (5) easy steps to ingest data from your IoT devices with the Apache Kafka service of your choice. These can be broadly divided into

Connection Configuration parameters
Topic mapping parameters

The connection configuration parameters help establish a secure connection between HiveMQ Cloud and your Apache Kafka cluster. The topic mappings let you set up the bidirectional data flow between your MQTT cluster and Apache Kafka.

But first you must find the Kafka extension in the “Integrations” tab inside your HiveMQ Cloud cluster. This integration is available with HiveMQ Cloud.

Note: if you are using the free version of HiveMQ Cloud for the first time to follow these instructions, you can start without adding any payment information.

Now you are ready to dive into the five steps:

1.Connect HiveMQ Cloud with the Kafka service of your choice: To connect, you need a list of bootstrap servers for your Kafka cluster so the integration can fetch the initial metadata about your Kafka cluster.

2.Secure the connection: Now you need to add your Kafka credentials. This helps ensure there is a secure connection between HiveMQ Cloud and Kafka.

We offer two different SASL mechanisms for connection security.

3.Send data from HiveMQ to Kafka: Once you set up and secure the connection, you can choose what data to forward from your IoT devices. This requires mapping topics from HiveMQ Cloud to your Kafka cluster. The source topic is the MQTT topic you want to send from your HiveMQ cluster. The destination topics are the Kafka topic receiving the messages that your HiveMQ cluster sent.

4.Establish bidirectional communication: For bidirectional communication between Kafka and HiveMQ, you can configure the Kafka cluster to HiveMQ Cloud similarly as you define the topic mapping from HiveMQ Cloud to your Kafka cluster. In this case, the source topic represents the Kafka topic from which the integration should read messages. These messages are then published with the defined destination topic on your HiveMQ Cloud MQTT broker cluster.

5.Enable the configuration: You can start the data flow between the HiveMQ cloud cluster and your Kafka cluster by selecting the “enable” button.

If you’ve followed these five steps, you should now be able to employ Apache Kafka with HiveMQ Cloud to use data from your IoT devices for bidirectional communication.

Get Started

To access the Kafka-HiveMQ Cloud functionality for free, all you need to do is sign up.

Sign-up for HiveMQ Cloud

The integration is a lightweight version of our HiveMQ Enterprise Extension for Kafka and offers to solve frequently requested use cases. If you are still missing functionality, don’t hesitate to reach out to us. We are always keen on direct user feedback.

MQTT Broker Comparison – Open Source Vs. Commercial Vs. Cloud-managed Vs. General Purpose

HiveMQ — Fri, 19 Aug 2022 12:17:00 +0000

MQTT brokers help implement the publish-subscribe communication model between devices and applications. The MQTT broker also helps implement rules and filters that help make the communications efficient and secure.

Where MQTT Brokers are Used

The most common use cases for MQTT and MQTT brokers are in IoT applications.
In fact, MQTT is the de facto standard for IoT use cases. It’s deployed in bandwidth and resource-challenged environments where the clients have to be lightweight.

Thanks to the efficient nature of the protocol, MQTT is deployed across a variety of verticals and use cases. For example, it helps car-sharing app ShareNow to give its users instant access to their cars, it helps Netflix certify devices that can use its software and it helps Matternet monitor autonomous drones delivering medical samples.

Types of MQTT Broker

The MQTT specification lays out the functionality expected from an MQTT-based deployment, and that offers a common definition that communities and businesses can then use for building their applications.

Currently, an MQTT broker is available in the following variants:

Types of MQTT Brokers	Examples
Open Source	HiveMQ CE, Mosquitto
Commercial	HiveMQ Professional and Enterprise, EMQ, VerneMQ
Cloud (Managed)	HiveMQ Cloud, CloudMQTT, AWS IoT Core, Azure IoT Hub
General-purpose brokers with MQTT support	Solace PubSub+, IBM MQ, RabbitMQ, ActiveMQ

Choosing the Best MQTT Broker

An effective way to evaluate software technology is through Architectural Requirements (a.k.a. Non-Functional Requirements). An MQTT broker comparison based on these architectural requirements should give you insight into how to find the best MQTT broker for your needs

Note: This is a category related view and HiveMQ’s Enterprise MQTT Broker is not the focus of this table

Common Challenges by variants
	Open Source	Commercial	Cloud-Managed	General Purpose
Scalability	Limited scalabilty	Do not scale to millions of devices and messages	Require support tickets to add capacity	Unable to scale linearly and require massive step upgrades
Security	Limited options	Lack support for latest cipher suites for encryption	Lack flexibility e.g. turning off TLS for private networks to save compute and bandwidth	Miss advanced security features like plug-ins and chaining of authentication / authorization logic
Resilience	Cannot cluster for higher availability	Missing plugins for DB integration	Lack reason codes and other core MQTT features which sacrifices resolution times	- Master-Slave architecure create long failover time - Miss key features like Retained Messages that hurts recovery times
Agility	Hard to manage when coded in difficult libraries like Erlang	Require restarting application when adding nodes		Poor MQTT compliance makes interwork with systems unpredictable
Observability		Very few meaningful metrics available	- can’t query individual endpoints - action/hops inside the cloud are a black box	Force a stack of closed management systems that hamper collaboration between systems
Availability	No overload protection from overactive publishers	Persist on memory and not on disk causing data loss in many scenarios

These NFRs should form the foundation of an MQTT broker comparison and it will help to have deeper understanding of each of the architectural requirements:

Scalability

What to look for	Why
Native support for the publish-subscribe pattern	Efficiency of Fan-in/Fan-out pattern helps avoid spaghetti architecture and its complexity
Linear scalability	Helps avoid abrupt infrastructure costs when handing incremental growth
High number of topics and concurrent connections	Helps teams prioritize the business logic over managing lower-level components
Grow and shrink cluster size at runtime without losing data	Keeps availability and uptime commitments, whether scaling up or down

Resilience

What to look for	Why
Fault tolerance at multiple levels (broker, cluster, cloud)	IoT environments are prone to network outages and disruptions
Masterless cluster architecture	Master/Slave architecture suffers from long recovery times which hurts application availability and performance

Agility

What to look for	Why
Variety of deployment options - on-premise, cloud, fully-managed	Helps right-size your deployment for different use cases while operating under the same technical principles
Easy Maintainability through standards-based approach	Ease of development that helps accelerate time to market
Testability	For quality and performance assurance
Support for multi-cloud strategy	Helps avoid vendor lock-in and brings in the best features from multiple platforms
Tested and packaged extensions for common enterprise system	Complex integration like Apache Kafka etc. can be very time-consuming to build and maintain in-house
Expertise to certify extensions for enterprise use	During deployment and support issues, a vendor-certified extension is one less problem for the enterprise

Availability

What to Look for	Why
Cluster Overload Protection	Reduces the rate of incoming messages and connection requests from publishing clients that risk overloading a cluster
Built-in support for features like Retained Messages	IRL environments, a client needs a new/last state to be productive

Usability

What to Look for	Why
REST API	For programmatic access to the broker
K8s Operator	Your DevOps can easily orchestrate, automate, and manage the lifecycle of multiple HiveMQ cluster deployments within Kubernetes (platform agnostic)
Wide support of libraries	Helps your developers spend less time learning new coding languages and constructs

> To know what to look out for in regards to Security, Observability, Extensibility, and more, read MQTT Broker Comparison – Which is the Best for Your IoT Application?

Choosing the Best MQTT Broker

From an architectural perspective, it’s clear that enterprises should choose an MQTT broker that scales without compromising on the security and resilience of the application. The ability to tweak the parameters of the broker and integrate with enterprise systems like Kafka can be very powerful for a business.

While resilience, security, flexibility, and scalability are key, it’s important that the MQTT broker you choose is easy to use and manage - manually and programmatically.

HiveMQ Enterprise MQTT Broker has brought innovative features to mature businesses for their mission-critical applications. HiveMQ has 100% compliance to the MQTT 3.1.1 and 5 standards while offering highly-specialized professional services and 24x7 support to 150+ IoT customers across the globe.

See how HiveMQ Enterprise MQTT Broker stacks up against your enterprise criteria for deployment. Contact us today.

MQTT: Retained Messages | Part 8

HiveMQ — Wed, 06 Jul 2022 08:05:11 +0000

In MQTT, the client that publishes a message has no guarantee that a subscribing client actually receives the message. The publishing client can only make sure that the message gets delivered safely to the broker. Basically, the same is true for a subscribing client. The client that connects and subscribes to topics has no guarantee on when the publishing client will publish a message in one of their topics of interest. It can take a few seconds, minutes, or hours for the publisher to send a new message in one of the subscribed topics. Until the next message is published, the subscribing client is totally in the dark about the current status of the topic. This situation is where retained messages come into play.

Retained Messages

A retained message is a normal MQTT message with the retained flag set to true. The broker stores the last retained message and the corresponding QoS for that topic. Each client that subscribes to a topic pattern that matches the topic of the retained message receives the retained message immediately after they subscribe. The broker stores only one retained message per topic.

If the subscribing client includes wildcards in the topic pattern they subscribe to, it receives a retained message even if the topic of the retained message is not an exact match. Here’s an example: Client A publishes a retained message to myhome/livingroom/temperature. Sometime later, client B subscribes to myhome/#. Client B receives the myhome/livingroom/temperature retained message directly after subscribing to myhome/#. Client B (the subscribing client) can see that the message is a retained message because the broker sends retained messages with the retained flag set to true. The client can decide how it wants to process the retained messages.

Retained messages help newly-subscribed clients get a status update immediately after they subscribe to a topic. The retained message eliminates the wait for the publishing clients to send the next update.

In other words, a retained message on a topic is the last known good value. The retained message doesn’t have to be the last value, but it must be the last message with the retained flag set to true.

It is important to understand that a retained message has nothing to do with persistent sessions. Once a retained message is stored by the broker, there’s only one way to remove it. Keep reading to find out how.

To learn how to send or delete a retained MQTT message, read this article and watch this video.

Get your copy of MQTT Essentials eBook to understand the protocol in detail without you having to read the entire specification.

MQTT: Persistent Session and Queuing Messages | Part 7

HiveMQ — Wed, 06 Jul 2022 07:42:01 +0000

n this post we talk about persistent sessions and message queueing in MQTT.

Persistent Session

To receive messages from an MQTT broker, a client connects to the broker and creates subscriptions to the topics in which it is interested. If the connection between the client and broker is interrupted during a non-persistent session, these topics are lost and the client needs to subscribe again on reconnect.

Re-subscribing every time the connection is interrupted is a burden for constrained clients with limited resources. To avoid this problem, the client can request a persistent session when it connects to the broker. Persistent sessions save all information that is relevant for the client on the broker. The clientId that the client provides when it establishes connection to the broker identifies the session.

What’s stored in a persistent session?

In a persistent session, the broker stores the following information (even if the client is offline). When the client reconnects the information is available immediately.

Existence of a session (even if there are no subscriptions).
All the subscriptions of the client.
All messages in a Quality of Service (QoS) 1 or 2 flow that the client has not yet confirmed.
All new QoS 1 or 2 messages that the client missed while offline.
All QoS 2 messages received from the client that are not yet completely acknowledged.

How do you start or end a persistent session?

When the client connects to the broker, it can request a persistent session. The client uses a cleanSession flag to tell the broker what kind of session it needs:

When the clean session flag is set to true, the client does not want a persistent session. If the client disconnects for any reason, all information and messages that are queued from a previous persistent session are lost.
When the clean session flag is set to false, the broker creates a persistent session for the client. All information and messages are preserved until the next time that the client requests a clean session. If the clean session flag is set to false and the broker already has a session available for the client, it uses the existing session and delivers previously queued messages to the client.

To understand in-depth on how an MQTT client knows if a session is already stored and to know some of the best practices while using persistent sessions, read this article.

Watch this video to visually understand persistent sessions in MQTT.

Get your copy of MQTT Essentials eBook to understand the protocol in detail without you having to read the entire specification.

MQTT: Quality of Service (QoS) Levels | Part 6

HiveMQ — Wed, 06 Jul 2022 07:30:38 +0000

In this post, we explain the different Quality of Service levels in MQTT.

What is Quality of Service (QoS)?

The Quality of Service (QoS) level is an agreement between the sender of a message and the receiver of a message that defines the guarantee of delivery for a specific message. There are 3 QoS levels in MQTT:

At most once (0)
At least once (1)
Exactly once (2).
When you talk about QoS in MQTT, you need to consider the two sides of message delivery:

Message delivery form the publishing client to the broker.
Message delivery from the broker to the subscribing client.

Why is Quality of Service (QoS) important?

QoS is a key feature of the MQTT protocol. QoS gives the client the power to choose a level of service that matches its network reliability and application logic. Because MQTT manages the re-transmission of messages and guarantees delivery (even when the underlying transport is not reliable), QoS makes communication in unreliable networks a lot easier.

There are 3 levels of QoS in MQTT

QoS 0 - at most once
QoS 1 - at least once
QoS 2 - exactly once

To under these QoS levels in detail, read this article or watch the below video.

Get your copy of MQTT Essentials eBook to understand the protocol in detail without you having to read the entire specification.

MQTT: Topics, Wildcards, & Best Practices | Part 5

HiveMQ — Wed, 06 Jul 2022 07:03:35 +0000

In this post, we focus on MQTT topics, wildcards, and best practices.

MQTT Topics

In MQTT, the word topic refers to an UTF-8 string that the broker uses to filter messages for each connected client. The topic consists of one or more topic levels. Each topic level is separated by a forward slash (topic level separator).

In comparison to a message queue, MQTT topics are very lightweight. The client does not need to create the desired topic before they publish or subscribe to it. The broker accepts each valid topic without any prior initialization.

Here are some examples of topics:

myhome/groundfloor/livingroom/temperature
USA/California/San Francisco/Silicon Valley
5ff4a2ce-e485-40f4-826c-b1a5d81be9b6/status
Germany/Bavaria/car/2382340923453/latitude

Note that each topic must contain at least 1 character and that the topic string permits empty spaces. Topics are case-sensitive. For example, myhome/temperature and MyHome/Temperature are two different topics. Additionally, the forward slash alone is a valid topic.

MQTT Wildcards

When a client subscribes to a topic, it can subscribe to the exact topic of a published message or it can use wildcards to subscribe to multiple topics simultaneously. A wildcard can only be used to subscribe to topics, not to publish a message. There are two different kinds of wildcards: single-level and multi-level.

Single Level: +

As the name suggests, a single-level wildcard replaces one topic level. The plus symbol represents a single-level wildcard in a topic.

Any topic matches a topic with single-level wildcard if it contains an arbitrary string instead of the wildcard. For example a subscription to myhome/groundfloor/+/temperature can produce the following results:

Multi Level:

The multi-level wildcard covers many topic levels. The hash symbol represents the multi-level wild card in the topic. For the broker to determine which topics match, the multi-level wildcard must be placed as the last character in the topic and preceded by a forward slash.

When a client subscribes to a topic with a multi-level wildcard, it receives all messages of a topic that begins with the pattern before the wildcard character, no matter how long or deep the topic is. If you specify only the multi-level wildcard as a topic (#), you receive all messages that are sent to the MQTT broker. If you expect high throughput, subscription with a multi-level wildcard alone is an anti-pattern (see the best practices below).

Topics beginning with $

Generally, you can name your MQTT topics as you wish. However, there is one exception: Topics that start with a $ symbol have a different purpose. These topics are not part of the subscription when you subscribe to the multi-level wildcard as a topic (#). The $-symbol topics are reserved for internal statistics of the MQTT broker. Clients cannot publish messages to these topics. At the moment, there is no official standardization for such topics. Commonly, $SYS/ is used for all the following information, but broker implementations varies. One suggestion for $SYS-topics is in the MQTT GitHub wiki. Here are some examples:

$SYS/broker/clients/connected
$SYS/broker/clients/disconnected
$SYS/broker/clients/total
$SYS/broker/messages/sent
$SYS/broker/uptime

Summary

These are the basics of MQTT message topics. As you can see, MQTT topics are dynamic and provide great flexibility. When you use wildcards in real-world applications, there are some challenges you should be aware of. We have collected the best practices that we have learned from working extensively with MQTT in various projects and are always open to suggestions or a discussion about these practices.

MQTT Best Practices When Using Wildcards

Never use a leading forward slash
Never use spaces in a topic
Keep the MQTT topic short and concise
Use only ASCII characters, avoid non printable characters
Embed a unique identifier or the Client Id into the topic
Don’t subscribe to #
Don’t forget extensibility
Use specific topics, not general ones

Read this article to know the details of each best practices when using MQTT Wildcards.

Watch this video to visually understand MQTT topics and wildcards.

Click here to read the original post.

Get your copy of MQTT Essentials eBook to understand the protocol in detail without you having to read the entire specification.

MQTT: How to Publish, Subscribe & Unsubscribe | Part 4

HiveMQ — Wed, 06 Jul 2022 06:28:07 +0000

Earlier in this series, we covered the basics of the publish/subscribe model. In this post we delve into the specifics of publish/subscribe in the MQTT protocol. If you haven’t read about the basics of the publish/subscribe pattern yet, we strongly encourage you to read that post first.

Publish

An MQTT client can publish messages as soon as it connects to a broker. MQTT utilizes topic-based filtering of the messages on the broker. Each message must contain a topic that the broker can use to forward the message to interested clients. Typically, each message has a payload which contains the data to transmit in byte format. MQTT is data-agnostic. The use case of the client determines how the payload is structured. The sending client (publisher) decides whether it wants to send binary data, text data, or even full-fledged XML or JSON.

A PUBLISH message in MQTT has several attributes that we want to discuss in detail:

Packet Identifier The packet identifier uniquely identifies a message as it flows between the client and broker. The packet identifier is only relevant for QoS levels greater than zero. The client library and/or the broker is responsible for setting this internal MQTT identifier.

Topic Name The topic name is a simple string that is hierarchically structured with forward slashes as delimiters. For example, “myhome/livingroom/temperature” or “Germany/Munich/Octoberfest/people”. For details on topics, see part 5 of MQTT Essentials.

QoS This number indicates the Quality of Service Level (QoS) of the message. There are three levels: 0, 1, and 2. The service level determines what kind of guarantee a message has for reaching the intended recipient (client or broker). For details on QoS, see part 6 of MQTT Essentials.

Retain Flag This flag defines whether the message is saved by the broker as the last known good value for a specified topic. When a new client subscribes to a topic, they receive the last message that is retained on that topic. For details on retained messages, see part 8 of MQTT Essentials.

Payload This is the actual content of the message. MQTT is data-agnostic. It is possible to send images, text in any encoding, encrypted data, and virtually every data in binary.

DUP flag The flag indicates that the message is a duplicate and was resent because the intended recipient (client or broker) did not acknowledge the original message. This is only relevant for QoS greater than 0. Usually, the resend/duplicate mechanism is handled by the MQTT client library or the broker as an implementation detail. For more information, part 6 of MQTT Essentials.

When a client sends a message to an MQTT broker for publication, the broker reads the message, acknowledges the message (according to the QoS Level), and processes the message. Processing by the broker includes determining which clients have subscribed to the topic and sending the message to them.

The client that initially publishes the message is only concerned about delivering the PUBLISH message to the broker. Once the broker receives the PUBLISH message, it is the responsibility of the broker to deliver the message to all subscribers. The publishing client does not get any feedback about whether anyone is interested in the published message or how many clients received the message from the broker.

Publishing a message doesn’t make sense if no one ever receives it. In other words, if there are no clients to subscribe to the topics of the messages. To receive messages on topics of interest, the client sends a SUBSCRIBE message to the MQTT broker. This subscribe message is very simple, it contains a unique packet identifier and a list of subscriptions.

Packet Identifier The packet identifier uniquely identifies a message as it flows between the client and broker. The client library and/or the broker is responsible for setting this internal MQTT identifier.

List of Subscriptions A SUBSCRIBE message can contain multiple subscriptions for a client. Each subscription is made up of a topic and a QoS level. The topic in the subscribe message can contain wildcards that make it possible to subscribe to a topic pattern rather than a specific topic. If there are overlapping subscriptions for one client, the broker delivers the message that has the highest QoS level for that topic.

Suback

To confirm each subscription, the broker sends a SUBACK acknowledgement message to the client. This message contains the packet identifier of the original Subscribe message (to clearly identify the message) and a list of return codes.

Packet Identifier The packet identifier is a unique identifier used to identify a message. It is the same as in the SUBSCRIBE message.

**Return Code **The broker sends one return code for each topic/QoS-pair that it receives in the SUBSCRIBE message. For example, if the SUBSCRIBE message has five subscriptions, the SUBACK message contains five return codes. The return code acknowledges each topic and shows the QoS level that is granted by the broker. If the broker refuses a subscription, the SUBACK message contains a failure return code for that specific topic. For example, if the client has insufficient permission to subscribe to the topic or the topic is malformed.

After a client successfully sends the SUBSCRIBE message and receives the SUBACK message, it gets every published message that matches a topic in the subscriptions that the SUBSCRIBE message contained.

Unsubscribe

The counterpart of the SUBSCRIBE message is the UNSUBSCRIBE message. This message deletes existing subscriptions of a client on the broker. The UNSUBSCRIBE message is similar to the SUBSCRIBE message and has a packet identifier and a list of topics.

List of Topic The list of topics can contain multiple topics from which the client wants to unsubscribe. It is only necessary to send the topic (without QoS). The broker unsubscribes the topic, regardless of the QoS level with which it was originally subscribed.

Unsuback

To confirm the unsubscribe, the broker sends an UNSUBACK acknowledgement message to the client. This message contains only the packet identifier of the original UNSUBSCRIBE message (to clearly identify the message).

Packet Identifier The packet identifier uniquely identifies the message. As already mentioned, this is the same packet identifier that is in the UNSUBSCRIBE message.

After receiving the UNSUBACK from the broker, the client can assume that the subscriptions in the UNSUBSCRIBE message are deleted.

Watch this video to visually understand how to publish, subscribe & unsubscribe an MQTT message.

Click here to read the original post.

Get your copy of MQTT Essentials eBook to understand the protocol in detail without you having to read the entire specification.