LangChain for LLM Apps: A Working Engineer's Notes

I've shipped enough LangChain code in the last two years to have strong opinions about which parts I keep and which I quietly replace. This post is the condensed version of the notes I hand to engineers joining an LLM project mid-flight: what LangChain actually gives you, where it hurts in production, and the specific patterns I use to keep agent and RAG systems boring and reliable. There's a free PDF at the end with the checklist I use before any LangChain app touches real traffic.

What LangChain is actually good for (and where I stop using it)

LangChain is a set of abstractions over LLM calls, prompts, retrievers, tools, and agent loops. It's genuinely useful when you want a common interface across Anthropic, OpenAI, Bedrock, and local Ollama models, or when you want to prototype a RAG pipeline in an afternoon without writing every glue layer yourself. LCEL (the pipe syntax) makes small chains readable and streamable.

Where I stop using it: the moment a chain becomes business-critical, I extract the actual prompt and the actual API call. The framework is great for the first 80% and painful for the last 20%. Debugging a five-layer chain when latency spikes at 2 AM is not something you want to do through three levels of Runnable wrappers.

My rule of thumb after building content pipelines, RAG systems, and multi-agent workers:

Use case	LangChain?
Prototyping a RAG PoC	Yes
Cross-provider model swapping	Yes
Simple sequential chain with retries	Yes, LCEL is fine
Agent with 3+ tools, long horizon	LangGraph, not classic agents
High-throughput production endpoint	Raw SDK + your own orchestrator
Complex evals and observability	LangSmith or Braintrust, not homegrown

The strongest signal a chain has outgrown LangChain: you find yourself reading the framework source to understand why a callback fired twice.

Chains: keep them flat, keep them typed

The single biggest quality-of-life win I've had with LangChain is treating chains as functions with typed inputs and outputs, not as clever DSLs. LCEL lets you compose, but composition without types is where prompts silently break in production.

A pattern I use in almost every project:

from pydantic import BaseModel
from langchain_core.prompts import ChatPromptTemplate
from langchain_anthropic import ChatAnthropic

class BriefInput(BaseModel):
    topic: str
    audience: str
    max_words: int

class BriefOutput(BaseModel):
    title: str
    outline: list[str]
    hook: str

model = ChatAnthropic(model="claude-sonnet-4", temperature=0.2)
prompt = ChatPromptTemplate.from_messages([
    ("system", "You write B2B briefs. Respond as JSON matching the schema."),
    ("user", "Topic: {topic}\nAudience: {audience}\nMax words: {max_words}")
])

chain = prompt | model.with_structured_output(BriefOutput)

Three things this buys me:

1. Contract stability. Downstream code sees BriefOutput, not a string. When a prompt drift breaks output, it fails at the boundary, not deep in a template.
2. Testability. I can mock the model and unit test the schema binding without hitting the API.
3. Provider portability. with_structured_output works across Anthropic and OpenAI. Swapping models is a one-line change for A/B tests on cost.

Avoid the temptation to chain more than three or four steps in LCEL. If you need branching, retries with different prompts, or human-in-the-loop, you're in LangGraph territory or you should be writing plain Python with the SDK.

RAG: the retriever is 80% of the quality

Every LangChain RAG tutorial spends 90% of the code on the LLM call and 10% on retrieval. In production, that ratio is inverted. I've had projects where switching from naive cosine similarity to hybrid search (BM25 + dense + reciprocal rank fusion) moved answer accuracy from around 62% to over 85% on the same eval set, without touching the prompt.

Here's the retrieval stack I default to on client work, mostly on Postgres with pgvector:

• Chunking: semantic chunking with a 512-token target, 64-token overlap. Section-aware where source docs have real structure (Markdown, HTML). Never fixed-size splits on prose.
• Embeddings: text-embedding-3-small for cost, text-embedding-3-large when the domain has heavy jargon. Store the model name in the row; you will re-embed.
• Hybrid search: dense (pgvector) + sparse (Postgres full-text or BM25). Fuse with RRF, not weighted averages. RRF is more robust when scores are on different scales.
• Reranking: Cohere rerank or a small cross-encoder for the top 50. Cuts the context by 5x and improves faithfulness.
• Query rewriting: decompose multi-hop questions before retrieval. LangChain's MultiQueryRetriever is fine here, but log the rewritten queries. You'll find bugs.

The specific LangChain pieces I keep for RAG: VectorStoreRetriever for the interface, EnsembleRetriever for RRF, document loaders for the boring formats. I replace ConversationalRetrievalChain with my own orchestrator every time. It hides too much.

The biggest RAG gotcha nobody warns you about: eval before optimization. Build a set of 30-50 real questions with expected sources, and score every retriever change against it. Without evals, RAG tuning is astrology. I run these before every deploy, and I keep a small dashboard of retrieval precision@k over time. When it drops, I know before users do.

Agents: LangGraph or nothing

Classic LangChain agents (AgentExecutor, ReAct with tool loops) are fine for demos and dangerous in production. They hide the state machine, they're hard to resume, and when a tool call fails you get a stack trace that reads like a novel.

For anything real, I use LangGraph. It makes the state explicit, it supports checkpointing, and it plays nicely with human-in-the-loop patterns that clients actually want.

A minimal LangGraph agent pattern I use for content and research workflows:

from langgraph.graph import StateGraph, END
from typing import TypedDict, Annotated
import operator

class AgentState(TypedDict):
    query: str
    plan: list[str]
    findings: Annotated[list[dict], operator.add]
    draft: str
    approved: bool

graph = StateGraph(AgentState)
graph.add_node("plan", plan_node)
graph.add_node("research", research_node)
graph.add_node("write", write_node)
graph.add_node("review", review_node)

graph.set_entry_point("plan")
graph.add_edge("plan", "research")
graph.add_edge("research", "write")
graph.add_conditional_edges(
    "write",
    lambda s: "review" if not s["approved"] else END,
    {"review": "review", END: END}
)
graph.add_edge("review", "write")

What this buys me in production:

• Checkpointing. State persists to Postgres between steps. If the process dies, I resume from the last node, not from scratch. On long research agents this saves real money.
• Observability. Every node transition is a log line. I can replay a failed run node-by-node in a notebook.
• Interruptibility. Human approval before publish is a first-class primitive, not a hack.

The rule I give teams: if your agent needs more than three tools and one loop, use LangGraph. If it needs more than seven tools, split it into two agents with a coordinator. Single agents with 10+ tools become non-deterministic garbage, no matter which framework you use.

Production gotchas I've hit and how I fix them

These are the ones that cost me real hours on real projects.

1. Token budgeting is your job, not the framework's. LangChain will happily stuff 30 documents into a prompt and let the model truncate silently. Always count tokens before the call. I keep a tiktoken-based helper that logs a warning above 70% of context and hard-fails above 90%.

2. Streaming callbacks fire in weird orders. If you're building a UI that streams tokens, don't rely on callback ordering for state changes. Emit structured events yourself alongside the token stream. I use server-sent events with typed payloads: {"type": "token", ...}, {"type": "tool_start", ...}, etc.

3. Retries need jitter and per-model logic. LangChain's built-in retries are naive. Anthropic and OpenAI have different rate limit semantics, and their overloaded errors need different backoff. I wrap the model with a custom retry layer using tenacity with jittered exponential backoff and separate policies per provider.

4. Caching is off by default and it should not be. For any deterministic prompt (temperature 0, same context), enable SQLiteCache or Redis-backed cache. On one content pipeline this cut API spend by around 40% just from repeated evaluation runs during development.

5. astream_events v2 is what you actually want. If you're building streaming UIs with LangGraph, use astream_events(version="v2") and filter by event type. The older streaming APIs mix intermediate and final outputs in confusing ways.

6. Version pins matter more than in most frameworks. LangChain moves fast and has broken minor-version compatibility more than once. Pin langchain, langchain-core, and every provider package to exact versions. Update deliberately, not on pip install -U.

7. Prompts belong in files, not string literals. Store prompts as versioned files (I use plain .md with frontmatter for metadata). Load them at startup. This makes prompt diffs reviewable in PRs and lets non-engineers propose changes without touching Python.

What I'd do on a new LangChain project today

If a client hired me tomorrow to build an LLM app with LangChain, this is the shortlist I'd follow:

1. Start with LCEL for chains, LangGraph for anything with state or tools. Skip classic agents entirely.
2. Build the eval set on day one. 30-50 real inputs with expected outputs, scored automatically. This is the single highest-ROI thing you can do.
3. Type your I/O with Pydantic at every LLM boundary. Structured output is the difference between a demo and a system.
4. Use LangSmith or an equivalent from the first line of code. Retroactive observability is painful; up-front tracing is free.
5. Own your retrieval stack. Use LangChain retrievers as an interface, but understand every step: chunking, embedding, hybrid, rerank.
6. Budget tokens explicitly. Log context size on every call. Set hard limits.
7. Cache aggressively in dev, selectively in prod. Deterministic prompts should never re-run.
8. Plan the exit. Write code that could be ported off LangChain in a day. That mostly means keeping business logic out of chains.

The uncomfortable truth: LangChain is a scaffolding, not an architecture. The teams shipping reliable LLM apps I've seen up close all treat it that way. They use it where it saves time and replace it where it costs time.

Free PDF: my LangChain production checklist

I've packaged the checklist I use before shipping any LangChain app (retriever eval, token budgeting, retry policy, observability, prompt versioning, the whole list) as a one-page PDF. If you want it, drop me a line via lazar-milicevic.com/#contact and I'll send it over, no list, no funnel.

If you're deeper into building LLM apps in production, I've written more on running AI PoCs that ship and how I scope this kind of work. Happy to talk shop if you're building something real.