737x faster LangGraph checkpoints, and the case where Rust lost

Run a LangGraph agent long enough and the model call stops being your bottleneck.
The plumbing takes over. Every step, the graph serializes its state to a
checkpoint so you can resume, replay, or recover. LangGraph does that with
Python's deepcopy. For a small dict that is fine. For a 250KB agent state with
nested messages, tool outputs, and accumulated context, deepcopy is brutally
slow, and you pay it on every single step of a long run.

So I built fast-langgraph: a set of Rust accelerators for the hot paths in
LangGraph, packaged as drop-in components that keep full API compatibility.

Lead with the numbers, including the ones that hurt

Here is what the Rust paths actually buy you, measured against the Python
equivalents:

Operation	Speedup	Where
Complex checkpoint (250KB)	737x faster than deepcopy	Large agent state
Complex checkpoint (35KB)	178x faster	Medium state
Sustained state updates	13-46x	Long-running graphs, many steps
LLM response caching	10x at 90% hit rate	Repeated prompts, RAG
End-to-end graph execution	2-3x	Production workloads with checkpointing

And the automatic mode, the one that needs zero code changes, lands around
2.8x for a typical invocation.

Now the honest part. These are not "Rust is faster at everything" numbers. The
checkpoint speedup scales with state size. It is a serialization story. For a
small, flat dict, Python's built-in dict is implemented in C and already fast.
Rust does not win there, and the README says so plainly. The 737x is a large
complex-state number, not a headline you get on a toy graph.

The core idea: reimplement the critical paths, keep the API

LangGraph is good. I did not want to fork it or replace it. I wanted to swap out
the three operations that dominate a real workload:

1. Checkpoint serialization. deepcopy on complex nested state is the single biggest cost in a long run. Rust does a structured serialize instead.
2. State management at scale. High-frequency updates accumulate overhead. A Rust merge path handles append-heavy state.
3. Repeated LLM calls. Identical prompts waste real API money. A cache in front of the model kills the redundant calls.

Everything else stays Python. The Rust code is behind a compatible surface.

Two ways to turn it on

The easiest is automatic acceleration. One flag, or one function call, and your
existing graph runs faster with no code changes:

# At the top of your application
import fast_langgraph
fast_langgraph.shim.patch_langgraph()

# Rest of your code unchanged - runs 2-3x faster
from langgraph.graph import StateGraph
# ...

Or set FAST_LANGGRAPH_AUTO_PATCH=1 in the environment, which is the path I
prefer in production because it needs zero code diff.

For the biggest wins you reach for the components directly. The checkpointer is a
drop-in replacement:

from fast_langgraph import RustSQLiteCheckpointer

# 5-6x faster than the default checkpointer
checkpointer = RustSQLiteCheckpointer("state.db")
graph = graph.compile(checkpointer=checkpointer)

The LLM cache is a decorator, and it reports its own hit rate so you can prove the
win instead of guessing:

from fast_langgraph import cached

@cached(max_size=1000)
def call_llm(prompt):
    return llm.invoke(prompt)

# First call: hits the API (~500ms)
response = call_llm("What is LangGraph?")

# Second identical call: returns from cache (~0.01ms)
response = call_llm("What is LangGraph?")

print(call_llm.cache_stats())
# {'hits': 1, 'misses': 1, 'size': 1}

That cache is where the 10x number comes from, and only at a high hit rate. If
your prompts are all unique, the cache does nothing. It is a RAG-and-retry
optimization, not magic.

How it works under the hood

The design constraint that mattered most: nobody rewrites their agent to try an
accelerator. So the whole thing is built as compatible shims over Rust
implementations.

The shim patches LangGraph's hot functions at import time. apply_writes, the
channel batch update, gets a Rust version. The executor caching path reuses a
ThreadPoolExecutor across invocations instead of rebuilding it. Neither of those
is a huge single win (1.2x and 2.3x respectively), but they compose into that
~2.8x automatic number because they sit on the path every invocation walks.

The manual components are where the large numbers live. RustSQLiteCheckpointer
replaces the serialize-and-persist path. langgraph_state_update does the state
merge in Rust with explicit append keys:

from fast_langgraph import langgraph_state_update

new_state = langgraph_state_update(
    current_state,
    {"messages": [new_message]},
    append_keys=["messages"]
)

That merge path is the 13-46x sustained-update number for long graphs that append
to messages hundreds of times.

There is also a profiler, so you can find your own bottleneck before you port
anything:

from fast_langgraph.profiler import GraphProfiler

profiler = GraphProfiler()
with profiler.profile_run():
    result = graph.invoke(input_data)
profiler.print_report()

Profile first. The wins are in specific hot paths, not "the code."

Where this does NOT fit

I would rather tell you where it loses than have you find out in production.

• Small, simple state. If your agent state is a flat dict of a few keys, Python's C-implemented dict already wins. The Rust checkpoint path pays off when state is large and nested, and the speedup scales with size. On a toy graph you will see close to nothing.
• All-unique prompts. The @cached decorator is a 10x win at a 90% hit rate. If every prompt is different, the cache is dead weight. It is for repeated prompts and RAG, not open-ended chat with high entropy inputs.
• You are not checkpointing. Half the value is faster persistence. If your graph runs once and exits without saving state, you skip the operation that benefits most, and you are left with the smaller 2-3x end-to-end range.
• You need a specific LangGraph internal that is not patched. The shim covers the hot paths, not the entire surface. Automatic mode is conservative for a reason, and it falls back to Python where it is not confident.

If your agents are short and cheap, this is not for you. If you run long,
stateful, checkpoint-heavy graphs in production, the serialization path is where
your time actually goes.

Takeaways

• The bottleneck in a long agent run is usually the plumbing, not the model. deepcopy on big state is a real tax you pay every step.
• Big speedups here are a data-and-serialization story, not a language story. 737x is a 250KB-state number and it scales down with your state.
• Measure the small cases. Rust loses to C-backed Python dicts on flat state, and pretending otherwise gets you caught.
• Make it a drop-in with automatic fallback, or it never gets adopted. One import, or one env var, is the whole onboarding.

Works with any LangGraph version, Python 3.9+. Code, the full benchmark
breakdown, and the architecture notes are here:
https://github.com/neul-labs/fast-langgraph

If you run stateful LangGraph agents at volume, I would love to know how big your
checkpoint state actually gets. Run the profiler, tell me your hot path. Kick the
tyres, issues welcome.

Comments

[Xin & EQ]

The thread connecting your comment on my article and this build: rule retirement is a cache eviction problem, and LLM response caching is also a cache eviction problem. Same systems thinking, different layers. Both times the insight is that the infrastructure is invisible until it hurts, and nobody instruments it until then.
What makes the 737x credible is the "where this does NOT fit" section. "Rust loses to C-backed Python dicts on flat state" is the kind of honest limitation that makes the big number believable. If you claimed 737x everywhere, the whole thing would read as marketing.
The @cached hit rate reporting ({'hits': 1, 'misses': 1, 'size': 1}) is the same pattern as my rule hit/miss tracking - track what fires, track what doesn't, evict the dead weight. The difference is you built automated eviction logic; I'm still doing it by hand. There's a lesson there about what "manual review" actually costs you at scale.