From LangChain Demos to a Production-Ready FastAPI Backend

Why LangChain Needs a Proper Backend Architecture

Most LangChain examples stop where real backend work actually begins.

Many AI examples live in notebooks, scripts, or Streamlit demos, but they quickly break down once they need to run inside a production backend system. As soon as AI becomes part of an API, it must follow the same rules as any other backend component. Inputs and outputs must be well defined, dependencies need to be explicit, and the overall structure must allow change without rewriting everything.

This article addresses exactly this starting point.

We will establish a clean and maintainable FastAPI endpoint that integrates LangChain in a backend friendly way. The goal is to create a solid architectural foundation that can be extended step by step. At this stage, the implementation is intentionally kept simple. Later articles will gradually introduce more advanced LLM and agent capabilities on top of this baseline.

The focus here is not on showcasing LangChain features. Instead, it is about defining a clear and robust endpoint architecture that remains understandable, testable, and scalable as complexity increases.

Thinking of AI as a Backend Component

Before looking at code, it is important to align on how AI should be treated inside a backend system. The goal is not to expose an LLM directly, but to embed AI logic behind a stable and predictable API.

A backend ready AI endpoint should provide the following guarantees:

• Clear request and response contracts
• Explicit orchestration of dependencies
• Encapsulation of AI logic away from HTTP concerns
• Predictable outputs that can be validated and consumed by other systems

FastAPI fits naturally into this model because it already enforces structure through Pydantic models and dependency injection. This makes it possible to integrate LangChain without special cases or ad hoc glue code.

Defining the Contract with Pydantic

The first building block is a strict API contract. Input and output are defined explicitly using Pydantic models.

# RequestModel
class InsightQuery(BaseModel):
    question: str
    context: str

# ResponseModel
class Insight(BaseModel):
    title: str
    summary: str
    confidence: float

    @field_validator("confidence")
    @classmethod
    def clamp_confidence(cls, v):
        if v is None:
            return 0.0
        if v < 0:
            return 0.0
        if v > 1:
            return 1.0
        return float(v)

This contract ensures that the API remains predictable regardless of how the underlying AI logic evolves. The confidence validator also demonstrates an important principle. Even if AI produces imperfect values, the backend enforces consistency before returning a response. Without it, LLM output quickly becomes unpredictable and hard to integrate into real systems.

Injecting the LLM via FastAPI Depends

Instead of creating the LLM directly inside the endpoint or inside the chain, it is injected using FastAPI dependencies.

# FastAPI Endpoint definition
@router.post(path="/query", response_model=Insight)
def create_insight(
        request: InsightQuery,
        settings: Settings = Depends(get_settings),
        llm: BaseChatModel = Depends(init_openai_chat_model)
):

The language model itself is initialized in a separate dependency function.

def init_openai_chat_model(settings: Settings = Depends(get_settings)):
    """
    Initializes and returns the LangChain OpenAI chat model
    """
    return ChatOpenAI(
        model=settings.openai_model.model_name,
        temperature=settings.openai_model.temperature,
        api_key=settings.openai_model.api_key,
    )

This approach has several advantages. The endpoint stays focused on orchestration, configuration is centralized, and the LLM can be replaced or mocked easily during testing. From FastAPI’s perspective, the language model is just another dependency, no different from a database session or a service client.

Encapsulating the LangChain Logic

The LangChain logic itself is encapsulated in a dedicated function. The endpoint does not need to know how the chain is built or executed.

def run_insight_chain(prompt_messages: ChatModelPrompt, llm: BaseChatModel, question: str, context: str) -> Insight:
    """
    Builds and runs the LangChain insight chain
    """
    prompt_template = ChatPromptTemplate([
        ("system", prompt_messages.system),
        ("human", prompt_messages.human)
    ])

    parser = PydanticOutputParser(pydantic_object=Insight)

    chain = prompt_template | llm | parser

    response = chain.invoke({
        "format_instruction": parser.get_format_instructions(),
        "question": question,
        "context": context
    })

    return response

This design cleanly separates concerns. Prompt construction, model execution, and output parsing live in one place. The rest of the application only deals with inputs and outputs.

Orchestrating Everything in the FastAPI Endpoint

The endpoint now becomes a thin orchestration layer.

@router.post(path="/query", response_model=Insight)
def create_insight(
        request: InsightQuery,
        settings: Settings = Depends(get_settings),
        llm: BaseChatModel = Depends(init_openai_chat_model)
):
    """
    Post Insights Endpoint: Creates a new insight to a given context and related question.
    """

    prompt_messages = load_prompt_messages(
        settings.prompt.insight_path,
        settings.prompt.insight_version
    )

    response = run_insight_chain(
        prompt_messages,
        llm,
        request.question,
        request.context
    )

    return response

The endpoint coordinates configuration, prompt loading, and chain execution without embedding business logic. This keeps the API readable and makes future extensions straightforward.

Why this Structure Scales

Even though the example is simple, the structure is intentionally forward compatible. Retrieval can later be added as another dependency. Agent logic can replace the chain function without touching the endpoint contract. State handling and error management can be layered on top without rewriting the core flow.

Most importantly, AI is treated as a backend concern, not a special case. It follows the same architectural rules as any other component in a production system.

Final Thoughts

This article shows the difference between experimenting with AI and operating it as part of a backend system. It establishes the first building block of a production oriented AI backend. From here, adding retrieval, memory, or agents becomes an architectural decision instead of a refactor.

💻 Code on GitHub: hamluk/fastapi-ai-backend/part-2