Originally published on AIdeazz — cross-posted here with canonical link.
Our first production LangGraph agent, a multi-step content generation pipeline, silently discarded 80% of its work for three weeks. The problem wasn't the LLM, the prompt, or the routing logic. It was a state schema mismatch during checkpointing, a bug that manifested as successful writes but failed reads, effectively deleting every intermediate step. We shipped it anyway, because the alternative was no agent at all.
This wasn't the only checkpointing disaster. Before we landed on a stable pattern for LangGraph stateful agents on Oracle Cloud, we went through three major rewrites. Each one taught us a hard lesson about the gap between local development and production stability, especially when dealing with persistent state in multi-agent systems.
The Silent State Schema Mismatch: A $1,200/month Lesson
The agent's job was to take a user request, break it down, generate content drafts, review them, and then publish. Each step was a node in a LangGraph, and the state was a TypedDict holding various strings, lists, and a status enum. We used SqliteSaver for local development, then migrated to PostgresSaver on Oracle Cloud. The PostgresSaver uses a jsonb column to store the state.
The initial AgentState looked like this:
class AgentState(TypedDict):
request_id: str
user_query: str
drafts: List[str]
current_step: str
status: Literal["pending", "generating", "reviewing", "publishing", "completed", "failed"]
When we added a new field, review_comments: List[str], we updated the TypedDict. Locally, this worked fine. In production, the agent would start, generate the first draft, save the state, and then fail on the next step. The error message was generic: KeyError: 'review_comments'.
The root cause: PostgresSaver's jsonb column doesn't enforce a schema. When we updated the TypedDict in our application code, existing checkpoints in the database still had the old schema. LangGraph's SqliteSaver is more forgiving, often defaulting missing keys to None or an empty list. PostgresSaver, however, would deserialize the jsonb into a dictionary, and when the Python code tried to access state["review_comments"], it wasn't there.
We lost about 200 agent runs, each costing roughly $6 in Groq/Claude API calls, before we traced it back. The solution was not just to update the TypedDict, but to explicitly handle schema migrations for existing checkpoints. We wrote a one-off script to iterate through all existing jsonb states in the langchain_checkpoint table, add the missing review_comments key with an empty list, and update the records.
Lesson 1: TypedDict is a compile-time hint, not a runtime schema enforcement for jsonb. Treat jsonb state as a mutable schema that requires explicit migration or robust default handling on read.
Checkpoint Corruption: The Race Condition That Froze Agents
After the schema mismatch, we started seeing agents freeze mid-run. No errors, just stalled. A manual inspection of the langchain_checkpoint table showed the thread_ts (timestamp) for these stalled agents was stuck, and the checkpoint jsonb column was often malformed or truncated.
Our agents run on Oracle Container Engine for Kubernetes (OKE), with multiple replicas for high availability. Each replica could potentially try to update the same checkpoint. While PostgresSaver uses FOR UPDATE to lock rows during a write, we found a subtle race condition. If two replicas tried to load the same checkpoint, modify it, and then save it almost simultaneously, one might overwrite the other's changes, or in rare cases, corrupt the jsonb data if the database transaction wasn't fully committed before another read began. This was exacerbated by network latency to the Oracle Autonomous Database.
The PostgresSaver's default thread_id is a simple string. We needed a more robust locking mechanism. We implemented a custom PostgresSaver subclass that added an optimistic locking field, version_id, to the langchain_checkpoint table.
ALTER TABLE langchain_checkpoint ADD COLUMN version_id INT DEFAULT 1;
When an agent loads a checkpoint, it also loads its version_id. When it saves, it increments version_id and includes WHERE version_id = :old_version_id in the UPDATE query. If no rows are updated, it means another process modified the checkpoint, and our agent retries the entire step (load, process, save) with a backoff.
This pattern drastically reduced checkpoint corruption and agent freezes. It added complexity, but it was necessary for true multi-replica stability.
Lesson 2: PostgresSaver's default locking is sufficient for single-process, single-thread access. For distributed, multi-replica deployments, implement optimistic locking or a distributed lock manager.
The One Pattern That Finally Clicked: Atomic State Transitions
Even with schema migration and optimistic locking, our agents still occasionally got into invalid states. For example, an agent might transition from generating to reviewing, but fail to save the review_comments in the same transaction. If the system crashed between these two operations, the agent was in an inconsistent state: status was reviewing, but review_comments was empty.
The core issue was that LangGraph's state updates, while atomic for the entire state object, didn't enforce atomicity of logical transitions. We needed to ensure that a state change (e.g., status update) and its associated data changes (e.g., adding review_comments) happened as a single, indivisible operation from the agent's perspective.
We refactored our AgentState and agent nodes to embrace a "command-pattern" like approach. Instead of directly modifying AgentState fields within a node, each node would produce a list of "state update commands." These commands would then be applied atomically at the end of the node's execution, before checkpointing.
Example AgentState with a command queue:
class AgentState(TypedDict):
request_id: str
user_query: str
drafts: List[str]
current_step: str
status: Literal["pending", "generating", "reviewing", "publishing", "completed", "failed"]
review_comments: List[str]
# New: a queue of pending state updates
pending_updates: List[Tuple[str, Any]] # (field_name, new_value)
Each node would append to pending_updates. A dedicated "state transition" node (or a custom checkpoint_saver hook) would then apply these updates and clear the queue.
def generate_draft_node(state: AgentState):
# ... generate draft ...
new_draft = "Generated content"
return {
"pending_updates": state.get("pending_updates", []) + [
("drafts", state.get("drafts", []) + [new_draft]),
("status", "reviewing")
]
}
def apply_updates_node(state: AgentState):
updates = state.get("pending_updates", [])
new_state = state.copy()
for field, value in updates:
new_state[field] = value
new_state["pending_updates"] = [] # Clear the queue
return new_state
This pattern ensures that all logical changes for a step are bundled and applied together. If the agent crashes before apply_updates_node runs, the pending_updates are still there on reload, and the agent can retry applying them. If it crashes during apply_updates_node, the optimistic locking ensures consistency on retry.
This approach made our LangGraph stateful agents significantly more robust. We now run multiple such agents on Oracle Cloud, routing between Groq for speed-critical tasks and Claude for complex reasoning, all orchestrated via Telegram and WhatsApp. The infrastructure is lean: OKE, Oracle Autonomous Database, and a custom API gateway. Zero VC funding, just relentless iteration on stability.
Frequently Asked Questions
Q: Why not use a message queue (e.g., Kafka) for state transitions instead of database checkpointing?
A: For our use case, the primary state is conversational and long-lived, requiring persistent storage and direct access for UI/debugging. A message queue would add significant complexity for state reconstruction and querying, which we wanted to avoid with a lean team.
Q: How do you handle LLM API rate limits and retries within LangGraph nodes?
A: Each LLM call within a node is wrapped in a custom retry decorator with exponential backoff. We also implement a global token-bucket rate limiter at the API gateway level to prevent exceeding provider limits, routing requests to different LLMs (Groq, Claude) based on current load and token cost.
Q: What's the cost of running these agents on Oracle Cloud?
A: Our core infrastructure (OKE, Autonomous Database, Load Balancer) costs approximately $150/month. LLM API costs vary, but average $800-$1,500/month depending on usage. Total operational cost for a production agent system is under $2,000/month, excluding development time.
Q: How do you debug a stalled or corrupted agent in production?
A: We have a custom dashboard that queries the langchain_checkpoint table, displaying thread_id, status, current_step, and thread_ts. For stalled agents, we can manually inspect the jsonb state, and if necessary, use a custom script to reset the status or inject a pending_updates command to force a specific transition.
Q: What if the jsonb column size becomes an issue for very long conversations?
A: We cap the history stored in the jsonb state to the last N turns (typically 10-20). Older conversation history is offloaded to a separate conversation_history table, linked by request_id, to keep the checkpoint size manageable and improve read/write performance.
Top comments (0)