DEV Community

Cover image for How to run a fair trial between AI visual inspection tools
Dev Station
Dev Station

Posted on Edited on

How to run a fair trial between AI visual inspection tools

Four groups of AI visual inspection tool, split by who owns the model

Vendor trials for visual inspection usually end with a demo on the vendor's sample parts and a single accuracy number. That number says little about your line. If you are the engineer asked to pick a tool, the trial design decides more than any feature table.

We compared the four tool groups (smart cameras, deep learning software on your own cameras, train your own platforms, turnkey systems) in Best AI Visual Inspection Tools for Manufacturing. This post covers the trial itself.

Pick the group before the product

Decide who should own the trained model first. Then trial two or three products inside that group, because a smart camera against a turnkey service compares two different contracts and the result tells you nothing about either.

Build a private test set

  • [ ] Good parts plus every defect class you care about
  • [ ] Images from more than one shift and more than one batch
  • [ ] Grades from your own quality team
  • [ ] Held back from every vendor until scoring day

Give every vendor the same training set

Same images, same labels, same time limit. Any difference in the result then comes from the tool.

Score per defect class

Score on the private set only, and report each class separately:

recall              = defects_caught / all_real_defects
precision           = defects_caught / all_parts_flagged
false_calls_per_100 = good_parts_flagged / parts_inspected * 100
Enter fullscreen mode Exit fullscreen mode

Operators live with the last figure. Put it on the scorecard next to recall, because a tool your operators keep overruling will not stay switched on for long.

Simulate a product change

Introduce a new variant midway through the trial and time how long each tool takes to handle it. Product change is where the four groups separate, and most trials never test it. A smart camera may need the vendor representative to reconfigure the program, software on your own cameras moves as fast as your vision engineer, and a turnkey system moves on the vendor schedule.

Ask where the result goes

A pass or fail that stays inside the camera cannot answer an audit question. Ask how images, decisions and operator overrides reach your quality record or MES. Ask too who owns the labelled images if the contract ends.

Dev Station builds that records and integration layer around inspection tools for manufacturers in the US and UK. The full breakdown of each tool group, with named products, is in the original guide on dev-station.tech.

Top comments (0)