DEV Community

IdeaSniper
IdeaSniper

Posted on

Your agent passed 10,000 evals. It can still deadlock.

If your agent workflow has 20 routing variables — retry counters, confidence thresholds, tool-success flags, budget checks — the guards on its edges combine in 2^20 = 1,048,576 ways. Run 10,000 evals and you have sampled under 1% of them (0.95%, to be exact). The other 99% is where the stalls live.

A green eval suite proves the workflow can work. It says nothing about the combinations you didn't sample — including the ones where every outgoing edge of some node is closed at the same time and the agent has nowhere to go.

The failure mode

This is an illustrative example, but the shape is common. Take a research agent with a critic loop: a planner calls a tool executor, a critic node checks the result, and two edges leave the critic. One loops back for another attempt while retry_count < 3. The other exits — and whoever wrote it copy-pasted the same guard, so the exit also requires retry_count < 3.

Every node is correct in isolation. Every eval passes, because the evals never produce the state where the critic is reached with the counter already at 3. In production that state shows up on a Tuesday. Both edges are closed. The agent does not crash and does not error — it sits in the loop burning tokens and wall-clock time until something external kills it.

The bug is not in any node. It is in the combination of two guards, and combinations are exactly what sampled tests are worst at covering.

Tracing does not save you either. A tracing tool will show you the stalled run in detail — after it has burned the tokens, hit the timeout, and possibly retried against a paid API a few hundred times. Observability explains stalls that already happened. It cannot tell you whether a stall is possible in a graph you have not deployed yet. That check has to happen before the run, on the graph itself.

Why sampling can't rule this out

Evals sample paths in proportion to how likely your test data makes them. Trap states live in the corners your data rarely produces: the counter at its max and the tool failing and the confidence score landing under the threshold, all on the same run.

More evals do not fix the math. 100,000 samples still leave about 90% of a 20-variable space unexamined, and every variable you add doubles the space while your eval budget stays flat. "Can this graph stall?" is not a testing question. It is a counting question: out of all assignments to the guard conditions, how many leave the workflow stuck?

Count the traps instead

That is what I built SharpLogic to do. It takes a workflow graph — nodes, edges, and the guard condition on each edge, in a LangGraph-style JSON export — and checks it before the agent runs:

  1. Compile. Each distinct guard condition becomes a boolean variable, and the graph is encoded into CNF as a trap formula: an assignment satisfies it exactly when the workflow stalls under that assignment — a reachable node with every outgoing edge disabled, a dead end that is not a terminal node, or a cycle whose exits are all closed.
  2. Count. A zero-dependency Rust #k-SAT counter counts the satisfying assignments exactly. No sampling. The count is the number of condition combinations under which your workflow stalls, and the solver returns witnesses — concrete assignments that produce a stall, so you can see which node gets stuck and why.
  3. Structural pass. In parallel, a graph pass (Tarjan's SCC algorithm and friends) reports dead ends and cycles with no exit at all, which are broken regardless of what the guards say.

I verified the counter against brute-force enumeration on small graphs, where checking every assignment by hand is feasible. The counts match.

Because the output is a number and not a vibe, it can sit in a build pipeline like any other check: edit the graph, recount, and if the trap count moved off zero, look at the witness before you ship.

A worked example

Here is the critic graph from above, reduced to its skeleton:

{
  "nodes": ["Start", "Tool_Executor", "Critic", "End"],
  "edges": [
    { "from": "Start", "to": "Tool_Executor" },
    { "from": "Tool_Executor", "to": "Critic" },
    { "from": "Critic", "to": "Tool_Executor", "guard": "retry_count < 3" },
    { "from": "Critic", "to": "End", "guard": "retry_count < 3" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

One guard condition means two total assignments. The live endpoint returns:

total assignments: 2
trap_count: 1
witness: Deadlock at node 'Critic': reachable with every outgoing edge disabled
solve time: ~148µs
Enter fullscreen mode Exit fullscreen mode

Half of this graph's condition space is a stall. Now fix the exit guard so the two branches are complementary:

{ "from": "Critic", "to": "End", "guard": "retry_count >= 3" }
Enter fullscreen mode Exit fullscreen mode

Recount: trap_count: 0. One line, and the stall is not patched around — it no longer exists in the graph.

Honest limits

  • 64 variables, locally. The counter caps at 64 guard variables. Most workflow graphs I have tried sit far below that, but a graph with hundreds of distinct conditions is out of scope today.
  • Exact counting is worst-case exponential. A dense random 64-variable formula takes ~6.7s. Real workflow graphs are sparse and solve in microseconds to sub-millisecond; the slowest case I have measured, a strongly coupled 44–50 variable graph, took 260–290µs. Dense adversarial formulas are not the use case and will be slow.
  • It analyzes the graph you give it. SharpLogic counts stall assignments for the nodes, edges, and guards in your export. If the export does not match what your runtime actually does, the count describes the export.
  • It does not prove anything about ETH. SharpLogic was inspired by OpenAI's October 2026 post on the exponential time hypothesis. That paper is still under expert review, and this tool neither proves it nor implements its algorithm — the engine is a conventional exact DPLL-style counter with component decomposition and memoization. The engineering stands on its own either way.

Try it

No signup, no install: paste a graph JSON into the browser trial and get the trap count and witnesses back — sharplogic.vercel.app/#try. The demo graph answers in ~150µs.

For local use:

curl -fsSL sharplogic.vercel.app/install.py | python3 -
Enter fullscreen mode Exit fullscreen mode

Code, issues, and the brute-force verification tests: github.com/reacherwu/SharpLogic.

One question

Which format do you define your agent workflows in — LangGraph JSON exports, graphs defined in code, a visual builder, something else entirely? That answer decides which importer I build next, so I would genuinely like to know.

Top comments (1)

Collapse
 
salparvez profile image
Sal Parvez | ML Systems RI •

The copy-pasted exit guard is such a real bug. Every node is fine on its own and the combination is what traps you. Construction schedules jam the same way, with two trades each waiting on the other to finish. Checking the graph before it runs, instead of reading traces afterward, is the right order.