If you've been handed "add AI chat to the product" as a ticket, you've probably noticed the ticket undersells the actual scope. This isn't a widget install, even when it looks like one at first. It's a set of architecture decisions about data access, action boundaries, and where the model is allowed to touch production systems, and getting those wrong is how a launch turns into a public walk-back a year later.
This is the technical breakdown of what actually goes into a conversational AI integration, framed the way you'd scope any system with a probabilistic component in it. If you're doing this scoping without in-house LLM experience yet, this is also exactly the kind of architecture review a team offering AI app development services gets asked to run before a project kicks off, because the failure modes here aren't obvious until you've shipped a few of these.
Chatbot, generative model, or agent, pick the right target before you architect anything
These three get conflated constantly, and the difference changes your whole stack:
Rule-based chatbot: input -> match pattern -> scripted response
Generative chatbot: input -> LLM -> natural language response (no actions)
Conversational agent: input -> LLM -> tool call(s) -> API/DB write -> response
Most teams searching for "conversational AI integration" actually need the third pattern, even if they start by building the second. If your actual goal is task completion (checking order status, updating a record, issuing a refund), building a generative-only chatbot first means a rebuild later, not an incremental upgrade.
RAG isn't optional, and it's not a weekend add-on
Skipping retrieval-augmented generation is the single most common reason a launched integration underperforms. Without it, the model answers from training data and will confidently invent your return policy, your pricing, or your product details. A proper RAG pipeline needs:
Ingestion pipeline:
documents/CRM/product data -> chunking -> embedding -> vector store
Query time:
user question -> embed query -> retrieve top-k chunks -> inject into context -> generate response
Treat this as its own project, not a bullet point. Chunking strategy tuned to your actual document structure, an ingestion pipeline that runs on updates (not just once at launch), and retrieval quality tuning against real user queries are all separate engineering tasks. Skipping any of them is how you end up with a technically grounded model that still gives wrong answers because retrieval quality was never actually validated.
Four integration paths, and what each one actually costs you architecturally
No-code widget. Fast to ship, essentially a chat window on top of a single hosted model with basic config. Fine for FAQ-style support on a marketing site. Can't reach your backend meaningfully.
Direct LLM API integration. You call OpenAI or Anthropic directly from your backend, controlling exactly what data enters the context window and what tool calls the model can trigger. Most control, most engineering effort, typically the lowest per-conversation cost once you're past early-stage volume.
Enterprise platform. Bundles model access, multi-channel deployment, and compliance tooling. Right call for high-volume, multi-channel support orgs where building that infrastructure yourself isn't a good use of engineering time. Comes with real licensing costs and less architectural flexibility.
Custom-built assistant. Wired directly into your CRM, ticketing system, and internal tools via workflow automation, completing tasks end to end rather than describing them. This is the pattern Klarna used: authenticated access to purchase and payment data before the first message. Longest build time, and it needs a maintenance plan from day one, not as an afterthought.
Action boundaries need to be a design decision, not a default
Here's the architectural lesson from Klarna's well-known course correction: the platform itself wasn't the failure. The failure was scoping autonomy broadly across every query type instead of defining, explicitly, which tasks the model handles alone and which require human confirmation.
Action boundary map (example):
Autonomous: check order status, answer FAQ, look up account info
Human-confirmed: issue refund, cancel subscription, change permissions
Always escalate: disputes, hardship cases, anything outside defined scope
Map this before you write a single prompt. Every autonomous action needs a defined API endpoint with proper authentication behind it; this is backend engineering work, not prompt engineering, and it's the part that gets skipped under deadline pressure most often.
Security checklist that's easy to skip and expensive to skip
- Rate limiting on the chat endpoint to prevent abuse and runaway API costs
- Input sanitization specifically against prompt injection; test this adversarially before launch, not after
- Explicit rules for what data enters the model's context window per request; don't pass more than the task needs
- Session and memory storage design that accounts for data retention policy; this is a privacy decision, not just a technical one
- A human-review mode for the first weeks post-launch, checking a sample of responses before they reach production traffic unsupervised Klarna reportedly ran its assistant in human-review mode for weeks before full launch. That's a cheap step to include in your rollout plan and an expensive one to skip.
What this actually costs to build
For scoping purposes: a no-code widget runs $0 to $500 a month, mostly API usage. A direct API integration for a focused MVP typically runs $5,000 to $20,000 in build cost plus ongoing usage. A custom assistant with RAG and real action-taking capability runs $20,000 to $80,000 or more depending on how many systems it integrates with. Budget an additional 15 to 20% of build cost annually for maintenance, prompt updates, model version changes, and keeping your knowledge base current, since none of that stops being necessary after launch.
The takeaway
Conversational AI integration is a data and workflow architecture problem wearing a chat UI. Pick the right target (chatbot vs agent) before you build anything, treat RAG as core infrastructure rather than a checkbox, map action boundaries explicitly before writing prompts, and build your security and monitoring plan into the initial scope instead of discovering you need it after launch.
For the full breakdown of integration approaches, costs, and a closer technical look at what went wrong at Klarna, this AI integration guide is worth reading alongside your own architecture doc.
Top comments (0)