What happens when enterprise requirements - human approval gates, audit
trails, structured output - hit three agent frameworks? The first article
measured how Strands, LangGraph, and CrewAI differ on a plain task. This one
measures what happens when the task grows up: 45 more runs, same recorder
proxy, same model, same tools.
The headline finding: the frameworks fail differently. Strands - the
model-driven one - finished with an empty output three times while exiting
- LangGraph and CrewAI never did.
https://github.com/sunnydachs/agent-framework-showdown
(Read the previous article here. here.)
Why the enterprise angle
Two walls keep coming up in developer communities when agents touch regulated
work:
- Audit: "why did the agent make this decision" must be answerable, or the workflow can't enter regulated territory
- Approval: a human must be able to stop an agent before a destructive action (data mutation, sending, publishing)
The EU AI Act makes automatic logging and retention a legal obligation for
high-risk AI systems. So past the "working demo", these are the two
validations that matter. I ran them.
Experiment 1: human approval gates (18 runs)
The task: write a news digest, ask a human to approve, and only publish if
approved. Publishing is a simulated destructive action that must never fire
before approval.
The three frameworks implement the gate differently:
-
LangGraph:
interrupt()suspends the whole graph;Command(resume=...)continues it after the decision -
CrewAI:
Task(human_input=True)- the crew pauses for console feedback after the task completes - Strands: prompt-only ("ask the reviewer before publishing") - the model-driven loop is the design
Results:
LangGraph suspended 6/6 runs and resumed in 0.0s - the checkpointer
restores state without re-executing. Reject routes away from the publish node
via an edge condition, so the gate is enforced by the graph's structure, not
by the model's good behavior.
Strands respected the order too: ask -> check -> publish 3/3, and never
published on reject. But one run called publish_article twice with the
identical draft. The prompt was followed; the model just double-fired the
destructive action. In a model-driven design, a second execution after
approval is a real risk.
CrewAI is the simplest shape: 1 call for approve, 2 for reject (the
feedback re-runs the task). One operational gotcha measured along the way:
returning the same rejection on every prompt spins an infinite loop - 131 LLM
calls with the prompt growing from 240 to 6,561 tokens. CrewAI re-runs and
re-prompts on every non-empty feedback, so the repetition policy is the
caller's responsibility.
Experiment 2: audit-trail reconstruction (36 runs analyzed)
The regulated question is "why did the agent decide this". Because every run
goes through the same recorder proxy, the traces have one shape - so I scored
whether an auditor can recover the seven audit-relevant facts (decision
rationale, tool call order, tool arguments, model identity, and more) from
each framework's traces:
| Framework | Rationale | Tool order | Args | Silent failures |
|---|---|---|---|---|
| Strands | 100% | 100% | 100% | 2 |
| LangGraph | 100% | 0% | 0% | 0 |
| CrewAI | 100% | 50% | 50% | 0 |
Strands is model-driven, so everything the model saw and reasoned about stays
in the trace - the strongest audit story of the three. The flip side is
exactly those 2 silent failures.
LangGraph's 0% is not a defect: its tool calls live in code, not on the wire.
Read the code and you know the order; read only the trace and you don't. That
is the real audit-design trade-off: where the evidence lives changes with
the framework's control-flow model.
Experiment 3: structured output (9 runs)
Output the digest as strict JSON with exactly 4 keys (summary,
word_count, topics, publish_ready). All three frameworks hit 100%
compliance, and word_count matched the actual summary length in every run -
putting the count inside the schema makes the model's self-verification
effective. Strands ran a validate loop averaging 2 calls (4 revisions in one
run).
The most important finding: they fail differently
Across all 45 runs, only Strands finished with an empty output three
times. The model built the complete result, handed it to the validation tool,
and then emitted nothing as the final answer. The run exits 0 - it looks
successful. You only catch it by reading the trace.
LangGraph and CrewAI: zero. In a pipeline design, the output node IS the
deliverable, so an empty answer is structurally hard to produce.
For enterprise use, this is the scariest class of failure: not an error that
stops the run, but a success-shaped empty result that breaks everything
downstream. (The empty outputs were recoverable - the model passes its full
result to the tool as an argument, so the trace holds it.)
Honest limitations
One model, 3 runs per cell - directional, not a definitive ranking. The human
is scripted; no real UI or notification flow. The destructive action is
simulated - though whether the gate held is read directly from the recorded
traffic, which is the part that's solid.
Reproduce it
Everything is open. The repo README has the commands for all five experiments
(72 recorded runs total, one proxy in front of every framework - that's the
whole foundation):
https://github.com/sunnydachs/agent-framework-showdown
This is a personal OSS project - no warranty. Use at your own risk, and
issues are welcome.
Cover image: generated with a local flux-schnell pipeline.
Top comments (3)
The finding that LangGraph's
interrupt()enforces the gate structurally while Strands leaves it to the model is the whole ballgame for regulated work. We run agents that can publish and delete, and the rule we converged on is: any destructive action has to route through an edge/state the model cannot talk its way past. Prompt-only gates fail exactly when you most need them — the run that's confident and wrong is the one that "asks" and then publishes anyway. The "empty output while exiting 0" from Strands is a great catch and lines up with what I see: model-driven loops fail silently, graph-driven ones fail loudly, and loud failures are cheaper. Two things I'd love to see in a follow-up: (1) resume latency after a crash mid-interrupt, not just a clean pause — the checkpointer restoring in 0.0s is impressive but the real test is process death; (2) whether the audit trail survives a retry without double-counting the approval. Nice methodology keeping the recorder proxy and model fixed across runs.The double-firing publish in Strands is the failure that bites hardest once actions touch an external API. We ran into the exact same pattern: prompt instructions said to execute once after approval, but the model-driven loop emitted two consecutive tool calls with identical payloads. You can tell the model not to double-fire, but the only reliable guard is an idempotency key at the execution boundary, either derived from the draft hash or passed as a run-scoped token to the downstream API.
The empty output exiting 0 is just as sneaky. Runners checking only exit status treat a polite handover to a validator tool as a complete pass. We ended up gating the runner exit on non-empty payload length and tracked file hashes rather than exit code alone.
Using one recorder proxy across all three frameworks makes this comparison much more informative than feature checklists. The duplicate publish call in Strands is especially important: an approval gate can hold while the side effect is still unsafe unless the execution layer deduplicates it. I’d be interested in a follow-up that scores approval, idempotency, and audit reconstruction together with the same operation ID. For LangGraph, was the missing tool order/argument data absent from the raw provider traffic too, or only from the framework-level trace surface?