DEV Community

Cover image for Pretty Colours, Wrong Answers: 9 Simulation Mistakes an AI Agent Made (2 Passed Every Gate)
NovaSolver
NovaSolver

Posted on

Pretty Colours, Wrong Answers: 9 Simulation Mistakes an AI Agent Made (2 Passed Every Gate)

Animated bar chart for a plate with a hole. The agent asked for 5, 10 and 15 mm holes; Heywood's Kt is 2.729, 2.512 and 2.343. The first run returned 2.541 for all three. The re-run returned 2.720, 2.541 and 2.377

A simulation that is set up wrong still converges. It still renders a smooth stress plot with a colour bar. Nothing on the screen looks wrong.

In November 2026, an AI agent (Claude Code) drove NovaSolver Studio, an open-source FEM/CFD app, through its MCP server. It wrote Python MCP client scripts, STEP geometry and reference calculations, and every call and reply went into a log: 3,355 calls, 187 solves started, 136 accepted comparisons (132 external references and 4 other Studio meshes; median difference 0.13%).

Getting there involved 24 groups of run-side problems. Here are nine that matter, in rising order of how dangerous they were. Then a chart of what caught them.

1. Asking for a picture before there was anything to draw

The agent loaded a double-walled glass mug and called render_view straight away. The reply said there were no results yet. It set the conditions, solved, and rendered again.

Harmless. It is on the list because it is the pattern behind the others: the agent assumed a state the case was not in.

2. Guessing an enum

import_geometry(physics="electric") (the value is "electrostatic") and render_view(view="side"). Both parameters are Literal types in Studio's tool signatures, so the MCP layer's argument validation rejected them before any Studio code ran.

Also harmless, because the schema was strict. A free-form string would have been a silent default.

3. A default that made the problem unsolvable

Eight DC current cases (a PCB trace corner and a contact spot) came back RED from validate_case. Joule heating was on by default, with no heat-removal boundary, so positive heating could not satisfy steady-state heat balance. Studio's solve path refuses RED cases, so nothing ran. The agent switched physics.joule_heating off and all eight solved.

4. GCI of 1e-10%, and still a failed mesh test

Four aluminium chime rods. The reported GCI was around 1e-10%, and the gate still failed all four: adjacent element counts did not grow sufficiently (refined=False). Two spinning disks failed for the same reason with GCI at 0.0004% or less.

A tiny change between meshes can mean the mesh is fine. It can also mean all three meshes share the same error. The agent added refinement on the rod sides (2.5 mm) and on the disk holes, and all passed.

5. GCI under 5%, and still a fail

A copper heat spreader, six thicknesses. On the default mesh, three of the six failed. The temperature kept rising by about the same step at each refinement, below Studio’s p ≥ 1 criterion for its asymptotic flag; this implementation criterion is distinct from a general definition of convergence. The GCI estimates were 4.0, 6.1 and 4.3%: two of the three were under the 5% mark.

With 0.5 mm elements on the heat source face, all six passed (GCI 0.53% or less) and matched the exact series within 0.05%.

6. Changing the excitation and contact mesh

For a van der Pauw sheet-resistance study, the agent drove contact A with 1 A of current. Studio judged mesh convergence on the dissipated power. With small contacts, the power kept rising as the mesh refined, and two of eight cases failed.

It switched to 1 V / 0 V voltage contacts. Then two cases still failed: 0.28 mm contacts were not resolved on the default mesh. Splitting the electrode faces at a tenth of the contact size fixed it, and all eight passed.

At fixed current, power is proportional to effective resistance (P/I²); using it was not itself an error. The change also replaced uniform current-density contacts with equipotential voltage faces, and contact refinement was still needed.

7. A reference check that failed for a good reason

A 50 mm steel column. Studio's built-in Euler check failed: the buckling load was 2.26% below Euler, outside its 2% tolerance. The agent did not just re-mesh until it passed. It compared with Euler including shear deformation and got +0.25%, so the value was kept with that explanation.

The same column's buckling load, 164.8 kN, is 7.0 times the 23.5 kN that yields it. The elastic buckling load exceeds the nominal full-section yield load, so it cannot serve as a load capacity under the elastic assumption; the actual failure sequence was not simulated.

8. The shelf that was "resting" but clamped

Bar chart of maximum deflection for an 800 x 250 x 18 mm shelf board with 30 kg of books: beam formula simply supported 1.467 mm; first run with a 10 mm underside strip constrained in z 0.346 mm (GCI 0.99%, PASS); re-run with ends held in z only 1.487 mm; beam formula both ends fixed 0.293 mm; Studio both ends fixed 0.289 mm

The plan: a shelf board resting on two supports. The setup: constrain a 10 mm strip on the underside at each end in z only.

Preventing deflection across the finite-width strip also strongly restrained bending rotation. So the "resting" board behaved almost like a clamped one: 0.346 mm maximum deflection, nearer the clamped value (0.29 mm) and about a quarter of the simply supported beam formula's 1.467 mm.

The mesh gate passed, GCI 0.99%. The gate estimated discretisation uncertainty for the selected quantity in an unintended setup.

The agent caught it by comparing with the beam formula. Holding the end faces in z only, so they can rotate, gave 1.487 mm: 1.4% above the beam formula.

9. Three different holes, one number

The most instructive one is in the GIF at the top.

On a plate-with-a-hole template, the agent attempted four updates combining the hole diameter and a mesh refinement on the hole; three cases have recorded results, and one started job has no recorded completion. The refinement field cannot be edited on a template, so the server rejected the whole change. The driver logged the reply as text and carried on:

# the first run, reconstructed from the log: the reply was a rejection, and nothing checked it
await d.call("update_case", case_id=cid, expected_revision=1,
             changes={"geometry.params.hole": f"{D} mm", "mesh.refine.hole": f"{D / 20:g} mm"})
await d.call("validate_case", case_id=cid)          # GREEN
await d.wait(await d.call("check_convergence", case_id=cid))
Enter fullscreen mode Exit fullscreen mode

validate_case was GREEN, because the default case is valid. The holes meant to be 5, 10 and 15 mm were all solved at 10 mm. All passed the template's Kt reference check; the saved replies do not record a GCI verdict. Kt came back as 2.5411 three times.

The agent noticed only because the numbers were identical. The fixed script removes the refinement and asserts on the revision number in the reply, so a rejected change stops the run:

r = await d.call("update_case", case_id=cid, expected_revision=1,
                 changes={"geometry.params.hole": f"{D} mm"})
assert r.startswith("rev 2") or (D == 10 and r.startswith("rev 1")), r
Enter fullscreen mode Exit fullscreen mode

With a single case there would have been nothing to compare, and nothing to notice.

What caught them

Bar chart of who caught the 24 groups of run-side problems: Studio mesh or reference gate 9, Studio memory check or failed job 5, Studio reply 2, MCP argument schema 2, Studio validate_case 1, the agent comparing with a formula 2, reading the logs afterwards 1, driver script assert 1, fact-check of the article 1

Across all 24 groups, the server caught 19: the mesh and reference gates (9), memory checks and failed jobs (5), error replies (2), the argument schema (2) and validate_case (1).

The other five were caught outside the server:

  • The agent comparing with a hand formula (the shelf, and the plate's identical values)
  • An assert in the driver script
  • Reading the logs afterwards (four of five pipe diameters had only a single solve, no convergence study, so the article states that only the 5 mm pipe was checked)
  • A fact-check of the article, which found that Studio's gate message says "do not converge monotonically" even for sequences that are monotonic but not in the asymptotic range

Look at which ones the server could not catch. Both are types of setup mistake that left a valid but unintended model. No solver can know intent.

The checklist we ended up with

  • Every write is asserted. Check the reply's revision number or isError, and stop on a rejection
  • Every result has a hand number next to it. A beam formula, a series, an order-of-magnitude estimate. The shelf and the plate were both caught this way
  • Identical outputs for different inputs are a bug until proven otherwise
  • A tiny GCI is not a pass by itself. Converged levels, sufficient element-count growth and a settled sequence are also required; the asymptotic flag is only one route
  • Choose the excitation and contact mesh together. Switching to equipotential voltage faces still required contact refinement in the van der Pauw case
  • Read the setup summary. validate_case returns one in plain words: which face carries what load, which way, what holds the part. It exists because no check can tell a load on the wrong face

The colour map is the last thing to look at, not the first.

NovaSolver Studio is free and open source (GPL-3.0). It solves structural, thermal, flow and electrical problems in one app and can be driven from MCP clients.

What is the most convincing wrong result you have seen from a simulation or any other tool, and what was the one check that would have caught it?

Top comments (0)