Hackathon disclosure: This article was created for the purposes of entering the All Things Agentic Hackathon.
When I started building Plant Debugger, the idea was much simpler than the final product.
An anomaly happens. An AI agent diagnoses it. The agent changes a control setting. The plant recovers. Done.
It was basically a self-healing plant.
Having worked in production engineering, I know firsthand that reality is very different. In almost all operating facilities, full automation is virtually non-existent. At best, plants have retrofitted a few sensors to monitor equipment data on a central dashboard, while local gauges remain analog and bypass valves must be operated by hand.
If a plant were truly 100% automated with remote actuators on every line, my view is that it should be operated by deterministic control logic (PLCs/DCS) and advanced mathematical/physics models (such as Model Predictive Control) rather than LLMs.
However, in the real world of brownfield operations, useful telemetry detects an anomaly, but the critical observations and manual adjustments required to troubleshoot it remain outside the API boundary. The troubleshooting "last meter" has no API.
That realization made my first version of Plant Debugger look impressive on paper — but less useful in reality.
The more I tried to make the prototype feel like something that could exist in an ordinary industrial site, the more I realized I was quietly assuming away the hardest part.
Real plants are not software systems with an API on every object.
A useful sensor may exist remotely, but an important gauge may still be local. A bypass valve may still be manual. An experienced operator may know that one side of a machine is inaccessible while another piece of equipment is running. The interesting part of troubleshooting often begins exactly where remote automation ends.
That changed the project.
The question was no longer:
How can an agent automatically repair a plant?
It became:
What should an agent do when the next useful observation or action exists only in the physical world?
That question led to the Plant Debugger I ended up submitting.
The first version was too clean
My first instinct was to make the plant behave like a software environment.
If the agent suspected a restriction, it could open a bypass. If it suspected another cause, it could change a control variable. Then it could watch telemetry and decide whether the fix worked.
Technically, that gives you a satisfying closed loop:
observe → reason → act → verify
That works well in a software-like environment where every component exposes a reliable control interface. But it creates two fatal flaws in a brownfield plant:
First, it silently assumes instrumentation and actuation that simply do not exist.
Second, if a person is introduced only as an approval step — “AI recommends, human clicks approve” — the result is not much better. The human becomes a passive gate in front of an otherwise digital workflow.
But it also creates a subtle fiction. It suggests that the agent has remote access to things that, in many brownfield plants, are still physical.
I did not want to win the demo by pretending every valve had an actuator and every field observation had already been digitized.
So I started removing capabilities instead of adding them.
I made one important differential-pressure indication local-only. I kept the bypass valve manual. I treated some practical site knowledge as something an operator had to contribute explicitly.
Oddly, the system became more interesting as it became less automated.
The missing APIs created a real reason for an AI agent and a field operator to collaborate.
I also did not want the human to become an approval button
There is an easy way to add a human to an agentic workflow:
AI recommends → human approves → system executes.
That can be useful for governance, but it was not the interaction I wanted to demonstrate.
The operator in Plant Debugger is not there to judge whether the AI's diagnosis is correct. The operator contributes something the software genuinely cannot obtain remotely: a field image, a manual action, a visible final state, or a stable piece of site-specific knowledge.
The agent, in turn, is responsible for deciding which of those contributions is useful next.
That distinction became the core of the product.
The collaboration is reciprocal:
- the agent decides what it needs from the field;
- the operator provides the physical observation or execution;
- the plant response comes back;
- the next agent decision changes because of that result.
This was much more compelling to me than putting an “Approve” button in front of an otherwise digital workflow.
The hardest design question was: is this actually an agent?
At one point, Plant Debugger was technically using an LLM inside a workflow, but I was not fully satisfied that the workflow itself was agentic.
If the sequence is effectively fixed — inspect this gauge, move this valve, verify recovery — then the model may be reasoning, but the application is still doing most of the orchestration.
I considered making the demo look more agentic by forcing an initial experiment to fail and then showing a re-plan.
I rejected that idea.
A deliberately scripted mistake would make the video more dramatic, but it would also weaken the point. The “failure” would exist because I wrote it into the demo, not because the agent chose an experiment and learned from the result.
So I moved the choice of the next field capability into the ADK workflow itself.
The agent sees the evidence available at that moment and decides whether it needs another observation, a bounded physical experiment, or escalation. If an experiment does not recover the plant, the resulting telemetry becomes new evidence and the same workflow continues.
That was the moment the prototype started to feel less like “an LLM in a flow” and more like the system I had originally imagined.
A small demo was better than a realistic digital twin
I spent time considering whether the project needed a more elaborate plant model, more equipment, more failure modes, or a richer digital twin.
I eventually cut most of that.
The final demo uses a deliberately small topology: a pump, a filter/strainer, a manual bypass around that filter, and a downstream flow measurement.
That is enough to create a meaningful diagnostic story:
- remote telemetry says something is wrong;
- one useful observation exists only at the equipment;
- one useful experiment requires a human to physically act;
- the downstream response tells the agent whether the experiment helped.
The simplicity mattered because I wanted a viewer to understand the causal story in seconds.
A more realistic simulator would have been technically impressive, but it would have made the agent's behavior harder to read.
The plant simulator therefore exists mainly to make one thing reproducible: the physical world can answer back to the agent.
The demo itself changed the product
Building the video exposed weaknesses that were less obvious in code.
For example, an early UI could show that “something happened,” but not clearly why the next task had changed. That made the system look more scripted than it really was.
So the Commander evolved toward a simple causal presentation:
new plant evidence → diagnosis update → next field task
I also had to think carefully about what counted as evidence.
A manual valve reaching its final position should not be assumed just because the instruction said “open it.” If the final state matters and there is no remote position sensor, the operator should return something visible.
Likewise, an image should not merely decorate the interface. It should be tied to the task that requested it and actually change the next decision.
These sound like small UI choices, but they changed the product philosophy. I became much more interested in making the agent's behavior inspectable than in making it look magically intelligent.
Some ideas were useful, but not important enough to become the headline
One example was persistent site knowledge.
I liked the idea that an experienced operator could teach the system something stable — for example, that a particular maintenance aisle should be used while another side is restricted — and that a later work order would reflect it.
That feature works well as a demonstration of adaptation.
For a while, though, it started taking too much attention. The project risked becoming a story about “memory” rather than a story about an agent working across the digital/physical boundary.
So I deliberately moved it back into a supporting role.
That does not make it incidental to the collaboration story: the live physical loop demonstrates adaptation within an incident, while Site Knowledge demonstrates adaptation across incidents, because an explicit human contribution can change a later work order.
That was a recurring pattern during development: many features were good ideas, but the strongest version of the project came from deciding what not to make the main idea.
What survived all of those cuts
The final prototype is much simpler to describe than some of the intermediate versions.
Google ADK owns the troubleshooting loop and can pause when the next useful result must come from the field. Gemini interprets telemetry and images. The operator uses a mobile-friendly work-order UI to return field evidence or confirm a bounded manual action. The workflow resumes from that result. Deterministic code constrains actions and decides whether recovery is actually sustained.
Google Cloud provides the runtime and durable evidence behind the demo:
- Cloud Run for the public Commander & Operator applications and private plant simulator;
- Google ADK for dynamic resumable workflow orchestration;
- Gemini on Vertex AI for multimodal visual reading and causal planning;
- Firestore for dual-collection state persistence (Field Ops & ADK workflow sessions);
- Cloud Storage for private field-image evidence with signed URLs and SHA-256 integrity;
- Secret Manager, Pub/Sub, and Cloud Logging for OIDC-authenticated event delivery and operational audit logs (see Pub/Sub Live Verification Video).
The important part of the interaction pattern remains:
Field evidence changes the next agent decision.
That sentence became a useful test for almost every design choice I made near the end.
If a feature did not make that loop clearer, safer, or more credible, it probably did not belong in the final demo.
What I learned from building it
The biggest lesson was that “more autonomous” is not always “more useful.”
At the beginning, I thought the strongest industrial agent would be the one that could do the most without a person.
By the end, I thought the more interesting system was the one that understood when it needed a person — not as an approver, but as a partner with access to the physical world.
A second lesson was that agentic behavior is easier to believe when the system is allowed to encounter uncertainty. I did not want the model to magically know the hidden fault, and I did not want the demo path to be a disguised script. The physical observations and plant response needed to matter.
The third lesson was about scope. I considered richer simulation, more automation, more memory, more visualizations, and more elaborate architecture. Most of those ideas were individually reasonable. The project improved when I kept only the pieces that strengthened the central interaction.
Plant Debugger started as a self-healing plant demo.
It ended up becoming an experiment in something I find more realistic:
An agent that knows where its APIs stop, asks the field for the right thing, and continues reasoning when reality answers back.
That is the “last meter” I wanted to explore.
Project links
- GitHub Repository: https://github.com/marcosanyo/PlantDebugger.git
- Main Demo Video (YouTube): https://youtu.be/InbLYOjIL8s
- Google Cloud Pub/Sub & OIDC Verification Video (YouTube): https://youtu.be/LHEr9a6qLCM
- Hackathon: All Things Agentic Hackathon
Top comments (0)