There has been a steady progression of demos showing an LLM attached to a sensor.
Send a temperature value to a chatbot and ask it what it means.
It's interesting.
But is it useful?
The engineering challenge of adding AI capabilities to existing IoT systems - including noisy telemetry, thousands of devices, workflow and operational data - is not as simple as plugging an LLM at the end of the pipeline.
Let's explore what a useful architecture might look like.
Start With the Data Pipeline
A basic IoT architecture might resemble:
- Device
- Sensor
- Gateway
- MQTT / Network
- Data Platform
- Application
Adding an AI capability to this architecture could involve attaching an LLM to the data platform output.
In practice, this is rarely a useful architecture.
The model would need to understand the context of a temperature value of 82°C - which could mean entirely different things based on the equipment, environment, operating mode, or position of the sensor.
A more useful architecture is something like:
- Sensor Data
- Validation
- Context Enrichment
- Rules / ML Analysis
- LLM Interface
Human or Application
The LLM is an interface to information, rather than an analytical engine.
LLMs Are Not Always the Right Tool
This leads us to the next point: statistical analysis, rules-based analysis, or traditional ML techniques may be more appropriate for analyzing high-frequency telemetry than a language model.
An LLM is a valuable tool when the problem involves interaction.
For instance, someone might ask "What were the anomalous events of the last shift?".
The application could return relevant telemetry, maintenance records, operational context, and the results of any rules- or ML-based analysis before passing this information to the LLM to generate a succinct summary.
The LLM is used to interrogate a body of information, rather than analyze raw telemetry.
Give the Model Context
Let's say we have an asset with the following telemetry and context:
Asset: Press-07
Temperature 82°C
Vibration: Elevated
Speed: 1,450 RPM
Utilization 91%
Last maintenance: 37 days ago
Anomalies: 3
A good application of AI could be to combine these signals and present them in a way that is useful to an operator.
The creation of this summary is not the important part.
The important part is the context assembly that happens before the prompt is created for the model to consume.
This usually takes the form of APIs, data services, retrieval systems, metadata stores, and rules - all of which wrap around the model to create a useful application.
Where LLMs Can Add Value
There are several cases where a language model can serve as a valuable interface to an IoT system.
1. Operational Summaries
Rather than having an operator consult a half-dozen dashboards, an application could generate a summary of notable events.
Telemetry
Event Detection
Context
LLM
Shift Summary
This could allow operators to more easily consume the information that is important to them.
2. Natural-Language Investigation
An operator could ask something like "What changed on Line 3 today?".
By parsing this request, the application could retrieve relevant events and data from the various systems involved and provide a summary of what happened.
This could be an effective way to allow operators to investigate unusual events.
3. Maintenance Assistance
An application could serve as an intermediary for a technician and the various systems that provide context for a piece of equipment.
Rather than the technician having to sift through different data sources, they can ask questions of the application.
Again, it is important that the application retrieve and display relevant information rather than generate text as if it were factual.
What About Automations?
This leads us to the topic of - well, automations.
An inference from an LLM is not necessarily an order.
A more appropriate architecture would be:
- IoT Event
- Detection
- Context
- AI Recommendation
- Confidence / Rules Check
- Human Review
- Action
Automating decisions based on AI inferences is appropriate for some applications.
When the stakes are high - lives, production, security, compliance, or other factors - double-checking recommendations and requiring human validation may be appropriate.
The Difference Between a Demo and a System
An LLM can be attached to a MQTT topic and produce interesting results on a weekend project.
A production application has to be able to handle edge cases.
- What if a sensor signal is invalid or missing?
- How are identity and access managed?
- How is the model validated and audited?
- How are its decisions logged and explained?
- How does it perform when it is unavailable?
- Who can see what information?
- How do end-users correct decisions the model has made?
These are architectural concerns, rather than prompt engineering.
The Real Opportunity
The opportunity with AI and IoT is not likely to involve an LLM querying a sensor stream directly.
A more realistic application might have layers: the IoT devices and sensors provide signals, the data infrastructure provides context, traditional ML and rules identify patterns and anomalies, and an LLM provides a conversational interface to the results.
Applications tie these capabilities together and expose them to an operator.
Asking "Can we connect an LLM to our IoT data?" is an interesting question to ask on a Monday morning.
A more interesting question to ask would be "What can we make easier to understand?".
That is what Aperture Venture Studio is doing with its focus area of Operational and Physical-World AI: building ventures around production AI and IoT systems.
Top comments (0)