Bhargav Bhatt

Posted on Oct 25 AI-assisted

Migration Guide: Dialogflow CX to LlamaIndex RAG Agent (OpenAI Responses API)

#architecture #llm #openai #rag

The Architectural Shift

Dialogflow CX is a deterministic finite state machine: Flows → Pages → Routes → Intents. You author every path. The 2025–2026 rebrand to "Conversational Agents" on Vertex AI acknowledges this is giving way to an LLM instruction-following model (Root Agent → Sub-Agents → Tools), but the classic flow engine still underpins the runtime.

LlamaIndex + OpenAI Responses API is the opposite: retrieval-augmented generation with no pre-authored paths. You feed it documents, and the model decides what to retrieve and how to respond. The OpenAIResponses class (not the legacy OpenAI class) is the current integration point — it uses the Responses API code path, not chat/completions.

What Breaks

1. Intents become retrieval queries

In CX, you have 50–200 intents with training phrases. Each intent maps to a fulfillment. In the RAG agent, there are no intents. The user's message is embedded, matched against your document corpus, and the top-k chunks are stuffed into the prompt.

What this means in practice: Your "What's your refund policy?" intent with 12 training phrases becomes a paragraph in a knowledge-base document. If your document is poorly chunked or the embedding model doesn't capture the semantic distance between "refund" and "money back," the retrieval misses. You lose the explicit mapping.

2. Context parameters become memory management

CX passes context-parameters (e.g., order_id, customer_email) between pages deterministically. In the LlamaIndex agent, conversation state lives in a ChatMemoryBuffer (default token limit: 3000). There's no schema enforcement. If the model "forgets" the order ID after three turns, your downstream tool call fails silently.

Fix: Use structured state (Pydantic models) alongside the memory buffer, or switch to a FunctionAgent with explicit tool parameters that the model must fill.

3. Webhooks become tool calls

CX webhooks are HTTP POSTs to your backend with a defined request/response schema. In LlamaIndex, you define Python functions as tools:

from llama_index.core.tools import FunctionTool

def check_order_status(order_id: str) -> str:
    """Look up order status by ID."""
    # your backend call
    return f"Order {order_id}: Shipped"

order_tool = FunctionTool.from_defaults(fn=check_order_status)

The model decides when to call it. You lose the deterministic trigger. A user saying "where's my stuff?" might or might not trigger the tool depending on what's in the retrieval context.

4. No-match handling is now a hallucination risk

CX has sys.no-match-default — a clean fallback. The RAG agent has no equivalent. If retrieval returns irrelevant chunks, the model will often confidently answer from the wrong document rather than say "I don't know." This is the single biggest behavioral regression.

5. The OpenAI SDK version cliff

As of 2026, openai>=3.0,<4 is the required pairing with llama-index>=0.14.24. The Responses API is the default code path. If your existing code uses openai.ChatCompletion.create() or the v2 client, it won't work. The OpenAIResponses class handles the new API surface (built-in tools, structured prediction via responses.parse, streaming, async).

What You Gain

Capability	CX	LlamaIndex RAG
Open-domain questions	"I don't understand"	Answers from your corpus
New knowledge	Re-author intent + training phrases	Drop a PDF, re-index
Multi-turn reasoning	Manual context parameter threading	Automatic (with memory buffer)
Tool use	Webhook per intent	Model decides when to call
Iteration speed	Console + deploy per change	Edit a document, re-query
Cost model	Per session (Google)	Per token (OpenAI) + infra

The biggest gain: your knowledge base is now a data problem, not a conversation-design problem. Adding a new policy means uploading a document, not authoring 8 intents with 40 training phrases.

The Testing Gap

This is where the migration is genuinely hard. Here's what you lose and what you need to build:

What CX gave you for free

Test cases per intent (you could attach 5 test utterances per intent and get a pass/fail in the console)
Deterministic replay (same input → same path, every time)
Structured failure modes (no-match, no-input, invalid-parameter are distinct events you can alert on)

What you must build

1. Retrieval quality evals (the new "intent accuracy")

Your test cases are now: given query Q, does retrieval return the correct document chunk? Use Ragas or DeepEval:

from ragas.metrics import faithfulness, answer_relevancy, context_precision
from ragas import evaluate
from datasets import Dataset

eval_data = Dataset.from_dict({
    "question": ["What's the refund window?", ...],
    "answer": [agent_response, ...],
    "contexts": [retrieved_chunks, ...],
    "ground_truth": ["30 days from delivery", ...],
})

result = evaluate(dataset=eval_data, metrics=[faithfulness, context_precision])

2. Hallucination detection (replaces no-match)

You need an LLM-as-judge that scores: is this answer grounded in the retrieved context, or is the model inventing? Gate deploys on faithfulness > 0.85.

3. Tool-call regression (replaces webhook integration tests)

For each tool, write a test that asserts: given a query that should trigger the tool, does the agent actually call it with the right parameters? This is non-deterministic, so run N=10 and require ≥9/10.

4. Multi-turn state tests

CX context parameters were testable because they were explicit. Now you need conversation-level tests: does the agent retain the order_id across 4 turns before calling the status tool? DeepEval's KnowledgeRetention and ConversationCompleteness metrics cover this.

5. The gap no tool fills yet

There's no equivalent of CX's "simulate 1000 sessions and report intent distribution." You need to build a simulation flywheel: generate synthetic multi-turn conversations (using a different model as the "user"), run them through your agent, and score the traces. DoorDash reportedly reduced hallucination ~90% this way.

Concrete Skeleton

# agent.py
import os
from llama_index.core import VectorStoreIndex, Settings, SimpleDirectoryReader
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAIResponses
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.memory import ChatMemoryBuffer

# --- Configuration ---
Settings.llm = OpenAIResponses(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# --- Ingestion (replaces intent authoring) ---
documents = SimpleDirectoryReader("./knowledge_base", recursive=True).load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=4)

# --- Tools (replaces webhooks) ---
def lookup_order(order_id: str) -> str:
    """Look up order status by order ID."""
    ...

def check_eligibility(customer_email: str) -> str:
    """Check refund eligibility for a customer."""
    ...

from llama_index.core.tools import FunctionTool
tools = [
    FunctionTool.from_defaults(fn=lookup_order),
    FunctionTool.from_defaults(fn=check_eligibility),
    FunctionTool.from_defaults(
        fn=lambda q: str(query_engine.query(q)),
        name="search_knowledge_base",
        description="Search internal documentation for policy answers.",
    ),
]

# --- Agent (replaces the entire flow) ---
agent = FunctionAgent(
    tools=tools,
    llm=Settings.llm,
    system_prompt=(
        "You are a customer support agent. Use the knowledge base for policy "
        "questions. Use tools for account-specific lookups. If you cannot "
        "find an answer in the knowledge base, say so explicitly — do not "
        "guess."
    ),
)

The system prompt line "do not guess" is your no-match handler. It's weaker than CX's sys.no-match-default, but it's the starting point.

Migration Sequence

Export your CX knowledge — pull all fulfillment text, intent descriptions, and webhook response schemas into a document corpus.
Build the index — chunk with SentenceSplitter(chunk_size=512, chunk_overlap=50). Validate retrieval on 50 known query→answer pairs before wiring the agent.
Port webhooks to tools — one FunctionTool per webhook. Keep the same backend endpoints.
Replace the flow with a single agent — start with one FunctionAgent. Don't build a multi-agent hierarchy until the single agent passes evals.
Build the eval suite before you cut traffic — 200 test cases minimum. Gate on faithfulness, context precision, and tool-call accuracy.
Run in shadow mode — route 10% of live traffic to the RAG agent, log responses, compare against CX. Iterate for 2 weeks.
Cut over — flip the traffic split. Keep CX running for 30 days as a rollback.

The version pin: llama-index>=0.14.24, openai>=3.0,<4. Both default to the Responses API. Don't use the legacy OpenAI class — it's the chat/completions path and will drift.

DEV Community