Implementing A/B Testing for Prompts in LangChain

Prompt engineering requires constant iteration. A common pattern is to have a "Production Prompt" and a "Challenger Prompt". Hardcoding these checks into your application logic ("If A use Prompt 1, else Prompt 2") creates technical debt.

In this demo, we use configurable_alternatives at the Prompt Layer to cleanly manage multiple persona definitions.

The Architecture

The Prompt Template becomes a dynamic node in our graph. The configuration tells it which internal template string to render.

The Code Implementation

# 1. Define Templates
concise = PromptTemplate.from_template("Be brief: {q}")
verbose = PromptTemplate.from_template("Be detailed: {q}")

# 2. Create Switchable Prompt
prompt = concise.configurable_alternatives(
    ConfigurableField(id="prompt_type"),
    default_key="concise",
    verbose=verbose
)

# 3. Run it
# Uses "verbose" template automatically
chain.invoke({"q": "Hi"}, config={"configurable": {"prompt_type": "verbose"}})

Benefits

• Clean Code: Your main chain logic chain = prompt | llm never changes, ensuring you are testing only the prompt difference.
• Scalability: You can add 10 different variants (Pirate, Lawyer, ELI5) without rewriting flow control logic.
• Production Ready: You can deploy this and use feature flags to route 10% of users to the new prompt to verify performance.

Checkout the Github repo here: github.com/harishkotra/langchain-ollama-cookbook/tree/main/03_prompt_variant_switcher