We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model — and one task failed 20 times in a row.
The LangChain vs LangGraph question comes up every time someone starts a new agent project. We had a slightly different version of it: once you've chosen LangGraph as your runtime, does picking the library around it actually change production outcomes?
So we ran a controlled benchmark to find out.
The setup
LangGraph 1.2.9 against Pydantic AI 2.13.0. Four tasks, designed to test realistic agent work: order processing with tool calls, shipping quote calculation, refund eligibility with date logic, and inventory reorder decisions. All tasks shared the same tool implementations and the same scorer — the only variable was which library orchestrated the calls.
We used gpt-4o, temperature 0, no parallel tool calls. The harness is public at github.com/benchclawio/harness. Every run was logged, nothing was cherry-picked, the raw JSONL is published.
160 task executions total. Here's what happened.
The frameworks tied
LangGraph: 100% correctness across all four tasks.
Pydantic AI: 100% correctness across all four tasks.
Not close — exact tie. Zero correctness gap on any task.
There was a performance gap: LangGraph was roughly 1.4–1.8 seconds faster per task on median wall time. The cause is Pydantic AI's async-to-sync bridge — when you call it synchronously, it spins an event loop internally, and that overhead is real and consistent. If you're building a latency SLO, that matters. If you're evaluating correctness first, the frameworks are identical with gpt-4o.
This result is unsatisfying in a useful way. It tells you that the framework abstraction is not where correctness variation lives — at least on these tasks, at this model, at this scale.
So we ran the same tasks again. Same harness, same task suite. Different model.
The model didn't tie
We substituted gpt-4o-mini for gpt-4o. Everything else held constant.
Overall correctness dropped to 75%. The framework split: same.
One task failed completely, 20 for 20 runs: refund-policy-minimal-tools. The task requires computing whether a return is within the 18-day return window. gpt-4o-mini calculated 19 days inclusive where the answer is 18 days exclusive — a date-arithmetic edge case. It failed this identically on every single run, regardless of which framework was handling the tool calls.
The other three tasks: 100%. The failure wasn't variance — it was systematic. And it was entirely a model property, not a framework property.
What this means for the framework debate
We had to decide what to compare LangGraph against before we started. Our dependency analysis (benchclaw.io/langchain-vs-langgraph/) found that LangChain 1.3.14 now declares LangGraph as an unconditional dependency — installing LangChain installs LangGraph. The reverse isn't true. So "LangChain vs LangGraph" is less of a choice than it used to be; the more real decision is whether to write in Pydantic AI or in LangGraph primitives once you're in the LangGraph runtime.
Even reframed that way, our benchmark says the framework abstraction doesn't move correctness outcomes on structured tasks at gpt-4o.
What did move outcomes was the model. By a lot, on a task that sounds easy.
The practical implication: if your task suite involves any temporal reasoning — eligibility windows, expiry dates, SLA calculations — test both your intended model and a cheaper fallback explicitly before you ship. "It works in testing" is not enough if testing only used gpt-4o and production routes some traffic to gpt-4o-mini or a smaller model.
The finding we didn't expect
Before we ran the model comparison, we expected the framework difference to be small. We expected correctness to hold. What we didn't anticipate was how systematic the failure mode was.
gpt-4o-mini didn't get the date arithmetic right on some runs and wrong on others. It got it wrong 20 out of 20 times, by exactly the same margin. That's not a probabilistic sampling failure — that's a knowledge gap baked into the model at this temperature and this task phrasing.
The implication is that per-task model validation matters more than framework benchmarking for production agent decisions. You can swap LangGraph for Pydantic AI or back again in an afternoon. You can't patch a model's arithmetic mid-deployment.
The data
Full benchmark with methodology, raw JSONL, and harness: benchclaw.io/langgraph-vs-pydantic-ai-benchmark/
Harness repo (Apache 2.0, citable): github.com/benchclawio/harness
If you want to reproduce it, the harness runs offline on any machine with Python and an OpenAI key.
LangGraph 1.2.9, pydantic-ai-slim[openai] 2.13.0, gpt-4o (temperature 0) and gpt-4o-mini (temperature 0). Benchmarked 2026-07-24. All runs logged; nothing omitted.
Top comments (0)