The first version of an LLM feature can feel almost suspiciously easy.
User input
↓
Prompt
↓
LLM
↓
Response
A few API calls later, you have a convincing demo.
Then real users arrive.
Someone pastes a 20-page document into a field you expected to contain three paragraphs. A response takes eight seconds instead of two. The model returns perfectly reasonable prose where your application expected JSON. A retry runs twice. A seemingly harmless prompt change improves common cases while quietly making several edge cases worse.
That is when LLM feature production stops looking like prompt engineering and starts looking like production engineering.
The model call is often the simplest part.
Everything around it is what makes the feature dependable.
1. "It Works" Is Harder to Define
Traditional application logic often has deterministic expectations.
def calculate_total(price, quantity):
return price * quantity
If the input is:
10 × 4
the expected output is:
40
Testing is straightforward.
Now consider:
Summarize this support ticket and identify the customer's primary issue.
There may be several acceptable answers.
That changes how quality has to be measured.
Instead of checking only:
actual == expected
you may need to ask:
Was the main issue identified?
Were important details preserved?
Did the answer invent information?
Was the response concise?
Did it follow the required format?
A production LLM feature therefore needs evaluation criteria, not only traditional unit tests.
2. Build Evals Before Continuously Editing the Prompt
Prompt editing becomes addictive.
One poor response appears, so you add:
IMPORTANT: Never do X.
Another edge case appears:
VERY IMPORTANT: Also avoid Y.
Eventually you end up with:
prompt-final-v14-revised-final-2
but nobody can confidently say whether version 14 is actually better than version 8.
A healthier workflow is:
Collect representative examples
↓
Define good behavior
↓
Create an evaluation set
↓
Run current prompt/model
↓
Change one thing
↓
Run evaluations again
Include more than happy paths.
Your dataset should contain:
- Normal inputs
- Very short inputs
- Large inputs
- Ambiguous requests
- Missing information
- Unexpected formatting
- Adversarial input
- Cases where the model should abstain
- Cases requiring human review
At this point, the work starts looking less like prompt experimentation and more like designing an LLM production architecture with explicit evaluation, guardrail, latency, cost, and observability requirements.
Once model changes become measurable, development gets much less mysterious.
3. Free-Form Text Is Dangerous When Software Consumes It
Humans can understand:
This looks like a fairly urgent billing problem.
Your application may expect:
{
"priority": "high",
"category": "billing",
"requires_human_review": true
}
Both responses mean roughly the same thing.
Only one is safe for downstream software.
If another system consumes the model output, define a contract.
For example:
{
"priority": "low | medium | high",
"category": "string",
"requires_human_review": "boolean"
}
Then validate the response.
The pattern becomes:
Model
↓
Schema validation
↓
Business validation
↓
Application
not:
Model
↓
Hope the parser survives
A useful rule is:
If software consumes an LLM response, treat that response like an API payload.
4. Latency Isn't One Number
During development you might see:
Response time: 1.6 seconds
and think the problem is solved.
Production looks more like:
1.3s
1.5s
1.8s
2.0s
6.4s
1.7s
8.9s
The user experiences the slow request too.
So measure more than an average.
Useful metrics include:
p50 latency
p95 latency
p99 latency
timeout rate
time to first output
total generation time
Then examine where that latency comes from:
Input processing
↓
Retrieval
↓
Model request
↓
Generation
↓
Output validation
↓
UI
Streaming can improve perceived responsiveness for user-facing generation.
It does not remove the need to understand the actual end-to-end latency.
5. Retries Need More Thought Than Expected
A simple retry looks harmless:
try:
call_model()
except TimeoutError:
call_model_again()
But what if the first request actually completed and only the response was lost?
Now you may have:
Two model requests
Two charges
Two tool calls
Two workflow actions
This becomes much more serious when the LLM can trigger:
Send email
Create ticket
Update CRM
Modify database
Run workflow
Before automatically retrying, ask:
Is the operation idempotent?
Can the first operation still complete?
Could repeating it create side effects?
Which errors should actually be retried?
What backoff strategy should we use?
Text generation is one thing.
LLM-driven actions require much stronger guarantees.
6. Prompts Eventually Become Application Code
At the beginning, your prompt may be:
prompt = """
Summarize this document.
"""
A few releases later it contains:
Business rules
Formatting rules
Examples
Tool instructions
Tone requirements
Safety behavior
Fallback behavior
Edge cases
At that point, the prompt is part of your application.
Treat it accordingly.
Track:
Prompt version
Model version
Configuration
Deployment date
Eval results
Reason for change
Instead of silently changing production instructions, think in versions:
customer-summary/v12
Then when somebody reports an incorrect output, you can answer:
Which prompt and model generated this?
That makes debugging substantially easier.
7. Treat the Model Like a Dependency
We pin:
Node packages
Python packages
Container images
Database versions
Yet it is easy to treat the model as:
whatever-is-latest
That creates unnecessary uncertainty.
A better workflow is:
Current model
↓
Candidate model
↓
Run eval suite
↓
Compare quality
↓
Compare latency
↓
Compare cost
↓
Small rollout
↓
Production
Model changes should be treated like software dependency changes.
Test them.
Measure them.
Roll them out intentionally.
8. Context Has a Real Cost
LLM systems tend to accumulate context.
System instructions
+
Conversation history
+
User profile
+
Retrieved documentation
+
Previous support cases
+
Current request
Adding more information can feel safer.
But larger context can increase:
Latency
Cost
Noise
Retrieval complexity
Irrelevant information
A better question is:
What is the minimum context needed to solve this task reliably?
Instead of:
Entire customer history
retrieve:
Relevant customer history
Instead of:
Entire documentation corpus
retrieve:
Relevant passages
Context should be selected deliberately, not accumulated indefinitely.
9. RAG Doesn't Automatically Make Answers Correct
Retrieval-Augmented Generation sounds simple:
Retrieve correct information
↓
Give it to the model
↓
Receive correct answer
But the real pipeline has more failure points:
Poor query
↓
Wrong retrieval
↓
Weak ranking
↓
Missing context
↓
Model ignores useful context
↓
Unsupported answer
This is why retrieval should be evaluated separately from generation.
Ask:
Did we retrieve the right document?
Did we retrieve the right passage?
Was it ranked highly enough?
Did the model use the evidence?
Were important claims supported?
Otherwise, two completely different failures appear identical to the user:
"The AI was wrong."
10. Cost Needs Feature-Level Visibility
An API call looks inexpensive when tested manually.
Then production multiplies it:
One request
×
thousands of users
×
many actions
×
large context
×
long responses
×
retries
Track cost where it is useful.
For example:
Cost per request
Cost per successful task
Cost per user
Cost per product feature
Input tokens
Output tokens
Retry rate
Model used
That lets you answer:
Which workflow is actually expensive?
Sometimes the answer is a smaller model.
Sometimes it is better retrieval.
Sometimes it is caching.
Sometimes it is reducing context.
And sometimes the best optimization is:
This operation doesn't need an LLM at all.
11. Keep Deterministic Work Deterministic
An LLM can produce a plausible answer to almost anything.
That doesn't mean it should own every decision.
Let normal application logic handle things like:
Permissions
Authentication
Arithmetic
Database constraints
Schema validation
Rate limits
Known business rules
Use the model where ambiguity actually exists:
Summarization
Classification
Extraction
Natural-language understanding
Generation
Reasoning over uncertain input
A healthy architecture is usually:
Deterministic software
+
LLM where probabilistic behavior adds value
not:
Model decides everything
The fewer hard guarantees you delegate to probabilistic output, the easier the product becomes to operate.
12. Observability Creates a Privacy Problem
To debug an LLM feature, you naturally want:
Input
Prompt
Response
Model
Latency
Token count
Errors
Tool calls
But those prompts can contain:
Customer information
Internal documents
Personal data
Commercial information
Credentials accidentally pasted by users
So "log everything" is not automatically good observability.
Define:
What can be stored?
What needs redaction?
How long is it retained?
Who can access it?
Which logs contain model input?
Which logs contain model output?
Privacy and observability need to be designed together.
13. Tool Use Changes the Security Model
A model that generates text has limited power.
A model connected to application tools may be able to:
Search private records
Modify data
Send messages
Call APIs
Create transactions
Trigger workflows
That means the model's suggestion cannot automatically become authorization.
A safer design looks like:
User request
↓
Model proposes action
↓
Validate parameters
↓
Verify user authorization
↓
Apply deterministic business rules
↓
Require confirmation when appropriate
↓
Execute
Keep two ideas separate:
The model believes this action is appropriate.
and:
The application allows this user to perform the action.
They are not the same thing.
14. Design What Happens When the Model Fails
Most prototypes focus entirely on:
Successful response
Production also needs:
Timeout
Invalid output
No useful context
Provider outage
Rate limit
Low-confidence result
Refusal
Tool failure
The product still needs to behave sensibly.
Depending on the workflow, that might mean:
Retry safely
Show retrieved sources
Fall back to search
Use deterministic behavior
Ask for clarification
Route to a human
Allow manual completion
One of the best lessons from LLM feature production is:
The model does not have to succeed every time for the product to handle every outcome well.
15. Roll Out Prompt and Model Changes Like Software Releases
A changed prompt is a production change.
So is:
New model
New RAG strategy
New tool description
New temperature/configuration
New system instruction
Use a controlled workflow:
Change
↓
Offline evaluation
↓
Staging
↓
Limited production rollout
↓
Observe
↓
Expand
Useful operational metrics might include:
Task success
Human correction rate
Regeneration rate
Fallback rate
Latency
Cost
Tool failures
User abandonment
Traditional monitoring asks:
Is the service online?
LLM monitoring must also ask:
Is the feature still producing useful results?
Those are different questions.
16. The Prototype Eventually Becomes an AI System
The first implementation may look like:
UI
↓
LLM
↓
Answer
A real production system starts looking more like:
User Input
↓
Input Validation
↓
Context / Retrieval
↓
Prompt Version
↓
Model
↓
Output Validation
↓
┌─────────┴─────────┐
↓ ↓
Valid Output Failure Path
↓ ↓
User / Tool Safe Fallback
And around that core:
Evaluations
Observability
Safety checks
Prompt versions
Model versions
Latency monitoring
Cost tracking
Privacy controls
Once a prototype reaches this stage, the work resembles full-cycle Generative AI Development Services: integrating models into applications, deploying them reliably, monitoring performance, handling model updates, and maintaining the surrounding production system.
That is a useful mental shift.
The goal isn't simply to put an LLM behind an API endpoint.
The goal is to build a dependable product capability around it.
A Practical LLM Production Checklist
Before shipping, I would want clear answers to these:
[ ] What does a good answer mean?
[ ] Do we have representative eval cases?
[ ] Are prompt changes versioned?
[ ] Is the active model/version tracked?
[ ] Does machine-consumed output have a schema?
[ ] Do we know p50/p95/p99 latency?
[ ] What happens after a timeout?
[ ] Are retries safe?
[ ] What context is actually necessary?
[ ] Is retrieval evaluated separately?
[ ] Do we know cost per successful task?
[ ] What user information reaches the model?
[ ] What information goes into logs?
[ ] Are tool actions independently authorized?
[ ] What happens when the model cannot complete the task?
[ ] Can prompt or model changes be rolled back?
[ ] Are quality metrics monitored after deployment?
If several of those questions don't have answers, you may have an excellent prototype.
You probably don't have a finished production feature yet.
The Biggest Lesson
The surprising thing about LLM feature production is how quickly the LLM becomes only one component of the system.
The real production problem is:
Model quality
+
Evaluation
+
Latency
+
Reliability
+
Structured output
+
Retrieval
+
Context management
+
Cost
+
Privacy
+
Security
+
Observability
+
Fallback UX
A good prompt matters.
A capable model matters.
Neither one replaces engineering.
The strongest LLM features eventually stop feeling like impressive AI demos.
They simply feel like reliable product features that happen to have an LLM somewhere underneath.
That is probably the point when the feature is actually ready to ship.
Top comments (0)