Every building automation project starts with the same paperwork problem: pages of technical drawings showing hundreds of individual input and output connections that have to be read, classified and transcribed by hand before any controller gets configured.
We built a tool that automates the repetitive parts of that job for a client who installs and configures the Honeywell, Priva and ISMA controllers that run HVAC, climate control and lighting in commercial buildings. Here is what worked.
The task
Every project begins with a technical drawing: a multi-page Visio document showing every sensor, actuator and control signal, marked with arrows and shapes that indicate its type and direction.
Before any controller can be selected or wired, an engineer works through that drawing point by point. For each connection they identify whether it is digital or analog, input or output, which device it belongs to and what it is called, and they transcribe all of it into a structured IO list in the correct order.
For an experienced engineer that is one to two hours of careful, repetitive reading per document, and a single missed or misread symbol can carry through to a wiring mistake on site.
It is valuable to get right, tedious to do by hand, and it used to be hard to hand to software, because it depends on reading a visual convention, not just text.
First attempt: just ask
The client had already tried the obvious thing: pointing ChatGPT straight at a drawing and asking it to extract the IO list. It came out at around 50% accuracy or worse, nowhere near something an engineer could rely on without re-checking every single line anyway.
What we changed
We did not build a fully automatic black box. No model hits 100% on documents that mix vendor-specific symbol conventions, project-specific naming and drawings that have quietly evolved their own local dialect over years of projects. So we designed a human-in-the-loop tool: the AI does the reading, an engineer does the final check.
The accuracy gain came from the workflow around the model. In short:
1. Give the model two views of every page. A drawing is a diagram, and the connection between a symbol and the wire it sits on carries meaning that plain text extraction throws away. So each page goes in twice: the raw text layer, which is precise about labels, and a rendered image of the same page, which preserves which symbol is attached to which peripheral. We also wrote our own preprocessing pipeline that extracts the exact position of every element on the page, so the model reads the drawing the way a person's eye would (left to right, top to bottom) and not in whatever order the PDF happens to store it internally.
2. Ask for a structured, ordered result. The model returns a clean, ordered IO list in a fixed structure, ready for an engineer to check instead of building from scratch.
3. Teach the team's conventions with examples. We built a small library of few-shot examples from the client's own historical projects (heat pump installations, server room ventilation, HVAC systems), so the model learned the naming conventions and peripheral pairings this particular team actually uses, not a textbook version. Each page is also read against its own legend.
4. Keep a human checkpoint. Every extracted list goes through a review interface with the original drawing on one side and the AI's structured reading on the other, so an engineer can check and correct type, direction, naming or ordering in seconds, instead of re-deriving it from scratch.
5. Finish the job. Once an engineer has signed off, the reviewed list is assembled straight into a finished document for the field team. There is no extra formatting pass and no copy-pasting between tools.
The model choice
At the time we built this on Google's Gemini, because it was recognized as the strongest at genuinely multimodal reasoning: reading a rendered drawing and its underlying text together in one pass, instead of stitching separate tools together. It was also cost-effective enough to run across a full multi-page drawing set without turning AI spend into a line item on every project. Models change quickly, so treat that as a snapshot of the time, not a recommendation.
The result
A task that used to take an experienced engineer one to two hours per drawing now takes a few minutes of double-checking. The tool is in daily production use on real projects.
The client's engineering lead put it this way:
We used to set aside a couple of hours for a drawing like this. Now I skim the output, fix a couple of things, and I'm done before my coffee's cold.
What I would take away
- The gain came from the workflow, not from a cleverer prompt. Structuring what goes in, structuring what comes out, and putting a human checkpoint before anything is trusted is what took the result from about 50% to over 95%.
- Design for review, not for 100%. A fast review step is what makes the output safe to use.
- Use the team's own examples. They are how the model learned this team's naming conventions and pairings.
- These numbers come from one client's drawings. Yours will differ, so measure on your own documents before trusting any percentage, ours included.
If you want more detail on the project, there is a longer write-up here: the IO List Generator case study.
We are Nebulae, a Belgian software development company. If part of your operation still depends on someone reading, transcribing or double-checking something by hand, there is a good chance the engineering to change that already exists.
Top comments (0)