Adding an AI assistant to an existing application is not primarily a chat-interface problem. It is an application architecture problem.
The interface may be a small panel or a conversational screen, but the assistant still needs to authenticate users, retrieve the right information, respect existing permissions, call approved tools and fail safely when it cannot complete a task.
This tutorial walks through a practical architecture for integrating a large language model (LLM) into an existing web application. It covers authentication, retrieval-augmented generation (RAG), tool permissions, logging, evaluation, privacy and human handoff.
The examples are architectural illustrations, not a complete, production-ready implementation. Adapt them to your framework, identity provider, data model and chosen model API.
- Start with the application boundary
Consider a hypothetical business application that lets staff search internal procedures and ask questions about customer-support workflows.
The assistant needs to answer questions using approved documents. Later, the product team may want it to retrieve a customer's order or initiate a support action.
These are different levels of capability. Reading a document is not the same as changing a record.
A useful starting architecture is:
User interface
|
v
Existing application backend
|
+--> Authentication and authorisation
|
+--> Input validation and request limits
|
+--> Retrieval from approved data
|
+--> LLM API adapter
|
+--> Permission-checked tool execution
|
+--> Logging, evaluation and monitoring
Keep the model behind your application backend rather than exposing a privileged API credential in browser code. The backend should be responsible for deciding what context the model receives and which operations the current user may perform.
This boundary also gives you a place to enforce rate limits, redact sensitive fields, handle provider errors and apply application-specific policies.
Design principle: the model can propose an action; trusted application code must decide whether that action is permitted.
- Reuse authentication, but enforce authorisation separately
An existing application usually already has a user identity system. Reuse it rather than inventing a parallel login mechanism for the assistant.
However, a logged-in user is not automatically entitled to access every record the model could retrieve.
For each request:
Authenticate the user using the application's established mechanism.
Resolve their current roles, organisation and relevant access scope.
Apply authorisation checks to every requested resource or operation.
Pass only the permitted context to the model.
Recheck permissions when executing any consequential action.
For example, suppose a staff member asks, “Summarise the latest support case for customer X.”
The backend should first determine whether that staff member may access the relevant case. It should not retrieve every matching case and rely on the model to select an appropriate one.
The same rule applies to retrieval-augmented generation. Filtering search results by tenant or access scope must happen before those results become model context.
Do not treat a system prompt such as “Never reveal confidential information” as an access-control mechanism. Instructions can guide model behaviour, but they cannot replace server-side authorisation.
- Add retrieval-augmented generation where it solves a real problem
RAG is useful when answers need information from an organisation's documents or data that is not reliably available in the model's built-in knowledge.
A typical retrieval flow looks like this:
Question
|
v
Validate request and resolve user scope
|
v
Retrieve authorised, relevant content
|
v
Apply relevance and context limits
|
v
Build model input with source references
|
v
Generate answer
|
v
Return answer with appropriate citations
The retrieval layer might use keyword search, vector similarity or a combination of both. Choose based on the nature of the content, query patterns, latency requirements and the quality of the results—not simply because vector databases are fashionable.
A few implementation details matter:
Preserve access boundaries. Indexing documents does not remove the need to enforce access rules at query time.
Keep source metadata. Store identifiers and other metadata needed to trace a retrieved passage back to its source.
Control context size. More retrieved text is not necessarily better. Irrelevant passages can distract the model and increase processing cost.
Handle missing evidence. If retrieval produces no suitable material, the assistant should say it cannot find enough information rather than inventing an answer.
Plan for document changes. Define how updates, deletions and permission changes propagate to the index.
A useful test is whether the assistant can distinguish between “the source says this”, “the source does not cover this” and “the sources disagree”.
If those cases are indistinguishable in your interface, users may place more confidence in the answer than the evidence supports.
- Treat tools as privileged application operations
An assistant becomes more consequential when it can call tools: for example, fetching an order, creating a support ticket or updating a customer record.
Do not give the model unrestricted access to your database or a general-purpose execution environment. Expose narrow, explicitly defined operations through your backend.
For instance, a tool definition might describe an operation like this:
{
"name": "get_order_status",
"description": "Retrieve the status of an order the current user may access",
"parameters": {
"order_id": "string"
}
}
This is an illustrative schema, not a complete provider-specific tool definition.
When the model requests this operation, the backend should:
Validate the tool name and argument structure.
Reject unexpected fields and malformed identifiers.
Resolve the current user's permissions independently of the model.
Execute a narrowly scoped query.
Return only the fields required for the task.
Record the outcome using an appropriate audit policy.
Never assume that a tool call is authorised because the model requested it.
For a read-only operation, a successful permission check may be sufficient. For an action that changes data, sends a message or triggers a financial or operational commitment, you may need explicit user confirmation, idempotency controls and a way to reconcile uncertain outcomes.
An important distinction: validating an order ID confirms that the input has the expected shape. It does not establish that the user is allowed to access that order.
- Make failure handling part of the design
LLM APIs and surrounding services can fail in different ways. The application should distinguish between at least these cases:
Failure Handling Guidelines
Authentication Failure:
Reject the request before retrieving any data.No Useful Retrieval Evidence:
Explain the limitation or ask the user for clarification.Model API Timeout:
Retry within a defined limit when appropriate, then return a clear error message.Malformed Model Output:
Validate the output before use. Reject invalid responses or safely recover without executing them blindly.Tool Permission Denied:
Do not execute the operation. Return an appropriate error message.Uncertain Write Operation:
Verify the operation's outcome before attempting another write to avoid duplicate or unintended changes.
Avoid retry loops that multiply cost or repeat side effects. Use request timeouts, bounded retries and idempotency keys where the underlying operation supports them.
Streaming responses introduce another design choice. Partial text can improve perceived responsiveness, but it does not mean the complete answer has been validated. Do not stream privileged actions into execution before the relevant checks have passed.
- Log enough to investigate, not everything by default
When an assistant gives an incorrect answer or attempts an unexpected tool call, developers need to understand what happened.
Useful operational fields can include:
A request or correlation ID.
Model and application versions.
Retrieval source identifiers and relevance information.
Tool names, validation outcomes and execution status.
Latency, token usage and provider error categories.
Evaluation results or user feedback, where available.
Avoid automatically storing every full prompt, document passage and response indefinitely. Conversation data may contain personal, confidential or commercially sensitive information.
Define retention periods, access controls and redaction rules. Restrict access to diagnostic logs, and document which data is sent to external model providers. Where the risk warrants it, test whether sensitive data can be excluded or masked before model submission.
Logs should help you reproduce a failure without becoming another uncontrolled repository of customer information.
- Evaluate the assistant as a software feature
A successful API response does not prove that an assistant is useful, accurate or safe.
Build an evaluation set from representative tasks and known edge cases before expanding the rollout. Include examples such as:
Questions directly answered by approved documentation.
Questions with incomplete or conflicting evidence.
Requests for records outside the user's permissions.
Prompt-injection attempts embedded in retrieved content.
Invalid tool arguments and repeated tool calls.
Requests that should be escalated to a person.
Measure separate outcomes rather than relying on one overall score. Useful dimensions include answer correctness, evidence support, retrieval relevance, permission enforcement, tool-call accuracy, latency and cost.
For consequential actions, test the full workflow—not just the model's proposed tool call. A correct-looking proposal can still become an unsafe operation if the backend validates it incorrectly.
Keep a regression set and rerun it when changing the model, prompts, retrieval pipeline, tools or permission logic. Human review remains important for ambiguous cases and high-impact workflows.
- Design the human handoff deliberately
A production assistant needs a clear path for questions it cannot answer or tasks it should not complete.
A handoff can be triggered when evidence is insufficient, a tool fails repeatedly, a request falls outside the assistant's scope or a user explicitly asks for a person.
A useful handoff should preserve the minimum context needed to continue the task: the user's original request, relevant authorised sources, actions already attempted and the reason for escalation.
Do not tell the user that an action has completed unless the application has confirmed it. If the outcome is uncertain, say so and provide a route to check the status.
This is especially important when integrating with existing ticketing, ordering or account-management workflows. The assistant should complement the application's existing operational controls rather than quietly bypassing them.
- Introduce the capability in stages
A practical rollout can follow four stages:
Stage 1 — Read-only answers. Retrieve approved content and provide source references. Measure answer quality and failure patterns.
Stage 2 — Controlled tools. Add a small number of narrowly scoped operations with server-side permission checks and audit records.
Stage 3 — Confirmed actions. Introduce explicit confirmation for consequential changes, alongside idempotency and recovery handling where appropriate.
Stage 4 — Broader rollout. Expand access only after reviewing evaluation results, operational failures, data-handling requirements and support capacity.
These are planning stages, not a guarantee that a particular schedule or implementation will work for every application. The appropriate sequence depends on the system's risks and existing architecture.
Final checklist
Before shipping an AI assistant integration, ask:
Does every request use the application's established identity and permission model?
Is retrieved content restricted to resources the current user may access?
Are tools narrowly scoped, validated and independently authorised?
Can the assistant acknowledge missing or conflicting evidence?
Are timeouts, retries and uncertain write outcomes handled safely?
Are logs useful for diagnosis without collecting unnecessary sensitive data?
Have evaluation tests covered normal use, adversarial inputs and permission failures?
Is there a clear human handoff and a way to verify consequential actions?
Integrating an AI assistant is less about adding a clever conversation layer and more about deciding what the application will allow that layer to know and do.
Start with one useful workflow, keep the authority in trusted application code, and expand only when the evidence shows that the integration is behaving as intended.
Discussion: If you are adding an LLM to an existing product, which boundary has required the most thought so far: retrieval permissions, tool execution, evaluation or data privacy?
Top comments (0)