A production AI system is rarely just:
User → API → LLM → Response
That architecture works well in a demo.
Production is different.
Once an AI system has to work with private data, enterprise systems, user permissions, unreliable model outputs, latency constraints, cost limits, and changing requirements, the model becomes only one component of a much larger system.
At SotaTek, we approach AI development from this broader systems perspective: the model is only one layer of the product, while data, application logic, integrations, security, evaluation, infrastructure, and observability determine whether that capability can actually work in production.
A more realistic architecture looks like this:
User / Business Workflow
↓
Application
↓
Authentication / Authorization
↓
AI Orchestration Layer
↙ ↘
Retrieval Tools
↓ ↓
Knowledge Enterprise
Sources Systems
\ /
\ /
Model
↓
Guardrails / Validation
↓
Response
↓
Logging / Evaluation
↓
Monitoring / Feedback
The interesting engineering problems are usually somewhere between those boxes.
**Start With the Workflow, Not the Model
**One of the easiest mistakes in AI development is starting with:
Which model should we use?
A better starting point is:
What changes in the business workflow if the AI system works exactly as intended?
Consider an internal support assistant.
A naive implementation might look like:
response = llm.generate(user_question)
But the real requirement may be:
User
↓
Ask question
↓
Check permissions
↓
Find relevant company information
↓
Generate answer
↓
Validate response
↓
Log interaction
↓
Return answer
The model is only responsible for part of that workflow.
This distinction becomes important as soon as the application has real users and real data.
**The Application Layer Owns the Business Logic
**A common anti-pattern is putting too much responsibility inside the prompt.
For example:
SYSTEM_PROMPT = """You are an enterprise assistant.Only access information the user is allowed to see.Never modify financial records.Never approve a transaction.Always follow company policy."""
That sounds reasonable, but prompts are not an authorization system.
Permissions should be enforced by application code.
async def get_customer_data(user, customer_id): if not user.can_access("customer", customer_id): raise PermissionError("Access denied") return await customer_repository.get(customer_id)
The LLM can decide what information it needs.
The application decides whether it is allowed to access that information.
That boundary matters.
LLM
│
"I need customer data"
↓
Tool / API Layer
↓
Authorization Check
↓
Enterprise API
This also makes the system easier to audit and test.
RAG Is an Application Architecture, Not Just a Vector Database
For enterprise knowledge systems, retrieval-augmented generation is useful when information changes frequently or is private to the organization.
A simplified pipeline looks like:
Document
↓
Chunking
↓
Embedding
↓
Vector Database
↓
User Query
↓
Query Embedding
↓
Similarity Search
↓
Retrieved Context
↓
LLM
↓
Answer
A minimal implementation might look like:
async def answer_question(question: str): query_vector = await embed(question) documents = await vector_db.search( vector=query_vector, top_k=5 ) context = "\n\n".join( document.content for document in documents ) return await llm.generate( system=SYSTEM_PROMPT, user=question, context=context )
But this is still missing an important part:
authorization-aware retrieval.
Suppose the vector database contains:
Public documents
Internal documents
Finance documents
HR documents
Executive documents
A semantic search returning the most relevant document does not mean the user should see it.
The retrieval layer may therefore need metadata filtering:
documents = await vector_db.search( vector=query_vector, top_k=5, filters={ "tenant_id": user.tenant_id, "allowed_roles": { "$in": user.roles } })
The retrieval system is now part of the security boundary.
AI Agents Add a Different Class of Engineering Problems
A chatbot mostly generates responses.
An agent can take actions.
For example:
User
↓
Agent
├── search_customer()
├── get_invoice()
├── create_ticket()
├── update_order()
└── escalate_to_human()
A simplified tool-calling loop might look like:
async def run_agent(user, request): response = await llm.generate( messages=[request], tools=AVAILABLE_TOOLS ) if not response.tool_calls: return response.text for call in response.tool_calls: tool = get_tool(call.name) if not user.can_execute(tool.permission): raise PermissionError( f"Not allowed: {call.name}" ) result = await tool.execute( **call.arguments ) await audit_log.write( user=user.id, tool=call.name, arguments=call.arguments ) return await llm.generate( messages=[request, result], tools=AVAILABLE_TOOLS )
Now the architecture needs more than an LLM.
It needs:
- permissions
- tool contracts
- state management
- audit logs
- approval rules
- retries
- idempotency
- human escalation
- failure handling This is where an AI agent starts looking much more like a distributed application than a chatbot. Tool Calls Need Idempotency Consider an agent that can create an order. The model calls: { "tool": "create_order", "arguments": { "customer_id": "123", "amount": 500 } }
What happens if:
- The API succeeds.
- The network times out.
- The model retries the tool call. Without idempotency, you might create two orders. A safer API design is: POST /orders Idempotency-Key: agent-run-8f32...
The backend can then ensure that repeated requests with the same key do not create duplicate side effects.
This is a good example of why AI engineering overlaps heavily with conventional backend engineering.
The model can decide what it wants to do.
The backend still has to guarantee what actually happens.
Model Selection Is a System Trade-off
There is rarely a universally correct model.
The architecture may use:
AI Request
↓
Routing Layer
↙ ↓ ↘
Small Large Vision
Model Model Model
A simple routing strategy could look like:
def select_model(task): if task.type == "classification": return SMALL_MODEL if task.requires_reasoning: return LARGE_MODEL if task.requires_vision: return VISION_MODEL return DEFAULT_MODEL
The decision should consider more than benchmark quality.
Typical variables include:
Accuracy
Latency
Cost
Context length
Privacy
Throughput
Availability
Deployment model
Vendor dependency
Maintenance complexity
For some workloads, a smaller model with predictable latency can be more practical than a larger model that produces slightly better answers but costs significantly more.
For others, additional reasoning capability may justify the cost.
The architecture should make this trade-off measurable rather than ideological.
Evaluation Is Part of the Product
One of the biggest differences between a demo and a production AI system is evaluation.
Traditional software often has deterministic tests:
assert calculate_total(100, 10) == 110
AI systems are more complicated.
A useful evaluation pipeline might look like:
Test Dataset
↓
Model
↓
Generated Output
↓
Evaluation
┌───┼───────────┐
↓ ↓ ↓
Quality Grounding Latency
↓ ↓ ↓
Evaluation Report
For a RAG system, you might track:
metrics = { "retrieval_precision": ..., "answer_groundedness": ..., "answer_relevance": ..., "hallucination_rate": ..., "latency_p95": ..., "cost_per_request": ...}
The important part is defining what “good enough” means before production.
Otherwise, teams can discover that the system technically works but nobody can agree whether it works well.
Observability Needs to Cover More Than API Errors
A normal backend might monitor:
HTTP 5xx
CPU
Memory
Latency
Request volume
An AI system needs additional signals.
For example:
Request
↓
Prompt
↓
Retrieval
↓
Model
↓
Tool Calls
↓
Response
Useful telemetry can include:
Model latency
Time to first token
Total tokens
Input tokens
Output tokens
Retrieval latency
Number of retrieved documents
Tool execution time
Tool failures
Guardrail violations
User feedback
Cost per request
A production trace might therefore look like:
request_id: 7f82...
API latency: 1.2s
retrieval latency: 180ms
LLM latency: 820ms
tokens: 2,431
tools called: 2
retrieved docs: 5
estimated cost: $0.014
Now you can investigate why an AI request became slow or expensive instead of treating the model as a black box.
Guardrails Should Sit Around the Model
Guardrails are often described as a single safety layer.
In production, they can exist at several points:
User Input
↓
Input Validation
↓
Authorization
↓
Retrieval
↓
LLM
↓
Output Validation
↓
Tool Authorization
↓
Final Response
For example:
if contains_sensitive_request(user_input): return "This request cannot be processed."response = await llm.generate(...)if not passes_output_policy(response): return "The generated response requires review."
For systems that can execute actions, the tool layer should also enforce permissions independently of the model.
The model should never be the final authority for security-sensitive operations.
Deployment Changes the Architecture
The same AI application can require very different infrastructure depending on the environment.
AI Application
↓
┌──────────┼──────────┐
↓ ↓ ↓
Cloud Private Edge
↓ ↓ ↓
Managed On-prem Local
APIs Models Models
The choice depends on factors such as:
- latency
- data sensitivity
- connectivity
- GPU availability
- operational requirements
- infrastructure cost
- compliance requirements
- expected traffic For example, an industrial computer vision system operating inside a factory may have very different requirements from a cloud-based enterprise chatbot. The model is only one part of the deployment decision. The Production Architecture Putting the pieces together: ┌──────────────────┐ │ Users │ └────────┬─────────┘ ↓ ┌──────────────────┐ │ Application/API │ └────────┬─────────┘ ↓ ┌────────────────────────┐ │ Authentication / AuthZ │ └────────────┬───────────┘ ↓ ┌────────────────────────┐ │ AI Orchestration │ └───────┬────────┬────────┘ ↓ ↓ Retrieval Tools ↓ ↓ Knowledge Enterprise Base Systems \ / \ / ↓ ↓ ┌───────────┐ │ Model │ └─────┬─────┘ ↓ ┌──────────────────┐ │ Guardrails / Eval│ └────────┬─────────┘ ↓ Final Output ↓ ┌────────────────────────┐ │ Logging / Monitoring │ │ Cost / Quality / SLOs │ └────────────────────────┘
This is why “adding an LLM” and “building an AI system” are two very different engineering tasks.
If you want the broader view of how these components fit into an AI development engagement, from use-case discovery and readiness assessment through PoC, production engineering, deployment, and ongoing optimization, see our AI Development Company guide.
A Practical AI Development Lifecycle
A production project can be reduced to a fairly simple sequence:
Discovery
↓
Feasibility
↓
Data / AI Readiness
↓
Architecture
↓
PoC
↓
Evaluation
↓
Production Engineering
↓
Integration
↓
Deployment
↓
Monitoring
↓
Continuous Optimization
The important part is not to jump directly from:
Idea → Production
without validating the intermediate assumptions.
A PoC should answer questions such as:
Can the data support the use case?
Is the model accurate enough?
Is retrieval good enough?
Is latency acceptable?
Is the cost acceptable?
Can the system integrate with existing workflows?
What security risks exist?
What happens when the model is wrong?
If those questions cannot be answered, scaling the infrastructure will not solve the underlying problem.
For a deeper look at how these components come together across the AI development lifecycle, from use-case discovery and PoC validation to production deployment and optimization, we’ve covered the broader process in our AI Development Company guide.
The Engineering Boundary That Matters Most
The most useful mental model is probably this:
AI Model
│
┌────────┴────────┐
│ │
What to generate What to do
│ │
↓ ↓
Application Tool Layer
Logic Permissions
RAG Validation
Context Audit Logs
Evaluation Transactions
│ │
└────────┬────────┘
↓
Production
The model provides intelligence.
The surrounding software provides reliability.
That distinction becomes increasingly important as AI systems move from demos into business-critical workflows.
Final Thoughts
The hardest part of AI development is often not getting a model to produce an impressive output.
It is turning that capability into a system that can operate reliably with real data, real users, real permissions, real infrastructure, and real business constraints.
That means treating AI development as a systems engineering problem:
Model
+
Data
+
Application
+
Integration
+
Security
+
Evaluation
+
Infrastructure
+
Observability
Production AI System
At SotaTek, this broader system perspective is particularly important when moving from PoC validation into production AI applications, enterprise integrations, and AI operations.
The interesting question is no longer:
Which AI model should we use?
It is:
What system needs to exist around the model for this capability to work reliably in production?
Top comments (0)