Seven Real Failures of an LLM Agent Operating a CAD Kernel (and How the Architecture Contained Them)

DryDock — Mon, 20 Jul 2026 13:04:43 +0000

I built a system where an LLM talks to a customer about a silicone casting mold, and a
deterministic geometry kernel — OpenCASCADE, three decades of production C++ — does the
actual mass-solving, boolean surgery, and part-splitting. The LLM never touches the kernel
directly. It fills out a strictly validated contract; the kernel either produces a mold or
explains, with numbers, why it can't.

That split is the whole idea, and it's not a new one — it's a pragmatic instance of what the
neuro-symbolic AI literature calls Type 6: a neural system that produces a structured
artifact, and a symbolic engine that executes or verifies it. What I want to write about here
isn't the idea. It's what happened when I actually built it: concrete failures, in a finished
system with a green test suite, that the architecture caught — and, elsewhere in the build,
at least one it didn't, which turned out to be the most instructive of all. This essay walks
through five of the failures. The rest, including the one the architecture didn't catch, are
in the longer playbook linked at the end.

If you're building an agent over any domain where "close enough" isn't good enough — CAD,
finance, compliance, infra — these are the shapes your failures will take too, even though
the domain will look nothing like mine.

1. The kernel that said "done" and returned nothing

The operation that turns a solid block into a mold is a boolean subtraction: block − cavity.
I called OpenCASCADE, checked IsDone(), and moved on — which is what the API's shape
invites you to do, and what most wrappers do.

Two things I found the hard way: in the binding I'm using, the cut operation exposes only
IsDone() — no error report at all — and it can return IsDone() == True while handing you
an empty shape or a zero-volume solid. Separately, an offset operation (used to build a thin
outer shell) returned IsDone() == True while producing a shell of volume zero — no exception,
no negative flag, just success and emptiness.

A three-decade-old kernel, and it lies about success in exactly the way you'd assume it
couldn't.

The fix is a stance, not a patch: success is never declared by the kernel — it's declared by
postconditions the wrapper computes itself. Because the pipeline constructs the block around
the cavity, I know the exact expected volume by conservation (v_block − v_cavity), so I can
check it arithmetically, not heuristically:

expected = v_block - v_cavity     # exact, by construction
rel = abs(v - expected) / expected
if rel > 5e-3:
    return f"volume {v:.3f} mm3 deviates {rel:.3%} from expected {expected:.3f} mm3"

A kernel "success" that fails this check is recorded as a failed attempt and the escalation
ladder (below) advances. The LLM never sees the lie — it only sees a job that hasn't succeeded
yet.

2. The hollow solid that was never one shell

Sometimes the exact boolean path is too fragile for a given geometry, so I fall back to a mesh
boolean library and rebuild the result as a stitched solid: build a face per triangle, sew them,
construct a solid from the sewn shells.

My first version did this:

solid = BRepBuilderAPI_MakeSolid(TopoDS.Shell_s(sewed)).Solid()   # wrong

One closed surface, one solid — a mental model that's correct for the shapes you doodle while
writing the function, and wrong for the shape this function exists to handle. The result of
"block minus a fully contained cavity" is a hollow solid, and a hollow solid's boundary is
at least two closed shells — the outer wall and the wall of the interior void. That's not an
edge case; it's the normal case, because a mold is precisely a block with a cavity in it. The
naive version silently drops the cavity wall from the solid's description, and the kernel
doesn't warn you — it just hands back a solid that's topologically incomplete, and the
brokenness surfaces three steps later as a volume that makes no sense.

Representation crossings — translating one shape of data into another — are where systems
quietly rot, because each side's documentation describes its own world, never the shape of
the traffic between them. The fix was to stop assuming cardinality and observe it: iterate
over every shell the sewing step actually returns, not "the" shell.

3. The retry ladder that testifies

Topological booleans fail — not as a bug, but intrinsically: coincident faces and tolerance
mismatches put exact boolean operations on a knife's edge. The standard trick is "fuzzy"
tolerance: let the kernel treat near-coincident geometry as coincident, and escalate the fuzz
if the exact attempt fails.

The naive version of this is a while loop that jitters a parameter until something passes,
with a log line somewhere. When a job ships, nobody can say what numerical regime produced it;
when a job fails, the error says "boolean failed" and the forensics start from zero.

So instead, the escalation ladder is data, not control flow. Every rung — tolerance tried,
outcome, timing — is a typed record in the result, not a log line:

class BooleanAttempt:
    fuzzy_tolerance: float
    outcome: Literal["ok", "kernel_error", "postcondition_failed", "mesh_fallback"]
    detail: str | None
    ms: float

On success with escalation, the job still emits a warning — needing extra fuzz is a geometry
smell worth surfacing, not burying. On exhaustion, the complete attempt list travels inside
the error, so the failure report tells you which tolerances were tried, which failed in the
kernel versus which failed a postcondition, and by how much — an engineer, or the orchestrating
LLM, can tell "this geometry is fundamentally degenerate" from "we were 0.6% off at every rung"
without re-running anything.

4. `EXPORT_FAILED` or `OUTPUTS_UNSATISFIABLE`? Why the answer is architecture

A client can ask for specific physical parts by name. During export, I hit a question that
sounds like naming trivia: if the spec asks for a rigid outer shell but the job doesn't have
one enabled, what error is that?

EXPORT_FAILED already existed for "couldn't write an artifact" — and it's in the small set of
codes that map to status: "failed", meaning infrastructure trouble, retry with the same
input. But the missing-part case isn't infrastructure trouble. Given the same spec, it's
deterministic — no retry conjures a shell from a job that never enabled one.

Here's why the distinction is load-bearing rather than pedantic: the agent's protocol is to
apologize and retry on failed, and to renegotiate the spec on rejected. Ship the missing-
part case as failed, and the agent retries forever against a wall, eventually surfacing "a
technical problem on our side" for what's actually a one-field spec inconsistency it could have
fixed in one turn. The fix was a new code, OUTPUTS_UNSATISFIABLE, outside the failed set,
carrying both sides of the story in its context — requested parts, produced parts, why —
so the agent can propose the exact correction instead of retrying blind.

The general test: given identical input, does this failure reproduce? Deterministic means the
input must change. Non-deterministic means the input is fine and the infrastructure isn't. Every
exact domain has tempting misclassifications on this line — "insufficient funds" is not a
payment-gateway outage — and getting it wrong either burns retries on the impossible or makes
users redesign around a hiccup.

5. The fallback that changed nothing downstream

When the fuzzy-tolerance ladder above is exhausted, the system falls back to a completely
different engine: tessellate both solids, difference them as meshes, then reconstruct a
stitched OCCT solid from the result triangles (case #2). Hundreds of lines of downstream
pipeline — plane search, part splitting, registration keys, channel drilling — are 100%
native to the exact-geometry representation.

The tempting design is to let the alternate representation flow forward: teach every
downstream stage to also handle meshes. That doubles the surface every future stage has to
validate, and runs your worst geometry through your least-exercised code.

Instead, the fallback converts back at the seam: the reconstructed solid is wrapped to expose
the exact same interface the primary path produces — same methods, same shape — so nothing
downstream changes by a single line, and it's gated by the same postconditions as the primary
path (case #1). The one thing that isn't erased is the semantic difference: a tessellated
result is faceted geometry, fine for a 3D-printable format, a lie if exported as exact CAD data
— so the system statically refuses that combination in the contract, and dynamically reports
"this needed the approximate path" when it happens. The interface is preserved for the code;
the degradation is preserved as information for the human.

What this doesn't cover

Five failures, and the pattern that contained all of them, isn't the whole story. Two more
case files from the same build didn't make it into this essay — including the single most
instructive failure in the whole project: a one-line sign bug that survived every test in the
suite because the default test direction happened to make the bug invisible, and that neither
the contract nor the postconditions could have caught, for reasons that turn out to generalize
past geometry entirely. The other missing case shows what happens when every individual field
in a spec validates and the spec as a whole is still physically impossible.

Both of those, plus the architecture that ties all seven together — contract
boundary, error taxonomy, structured evidence trail — are in the longer
playbook. The system itself, mold-compiler, is real and running.

→ Read the full playbook: https://drydockdev.gumroad.com/l/uzsubs

DEV Community: DryDock