ModelRift ran a controlled eval this week: six Claude Opus 5 subagents, three printable parts, CadQuery on one side, OpenSCAD on the other, every final STL checked by a parser that reads the file directly instead of trusting the tool that made it.
Their words: "Renders caught nothing that mattered."
The OpenSCAD side ran a render-inspect-fix QA loop. The agent looked at the rendered images, decided if the part looked right, and iterated. That loop did the job the tools' marketing says a visual reviewer does: eyeballing the output for silent problems. It found zero.
Meanwhile the file-level parser, reading triangle count, bounding box, watertightness, non-manifold edges, flipped faces and connected components, flagged silent wrong geometry in 5 of 11 OpenSCAD runs and 4 of 11 CadQuery. T2, the two-part snap-fit enclosure that had to actually fit, racked up all of it: 5 silent-wrong runs total across both tools.
That's the whole point, and it generalizes past CAD. A render is a rendering of the object, not the object. What the agent committed to disk is the thing that gets printed, deployed, or merged. If your QA loop checks the picture and ships the file, you're reviewing the wrong artifact.
The same failure shows up in code review when the review reads the diff instead of what the build does with it, and in benchmarks that score the agent's trace instead of the artifact it produced. "It looked fine in the UI" is not verification. It's a vibe check.
What made this eval useful is exactly what most comparisons skip: nobody took the agent's word for anything. The verdict came from a parser with no loyalty to either tool. If your eval of an AI code generator isn't reading the final artifact with an independent checker, you're measuring the render, not the result.
Top comments (0)