Moving from a hosted GPT API to a self-hosted model sounded straightforward.
The application already had one clean interface:
Prompt
↓
LLM API
↓
Response
So the migration plan looked simple:
Replace API endpoint
↓
Change model name
↓
Deploy
The endpoint worked.
The model answered.
And then almost everything around it started behaving differently.
The lesson from our self hosted LLM migration was simple:
Replacing the model was easy. Replacing the behavior we had built around the model was not.
Note: This is a representative engineering post-mortem. The failures and architecture below illustrate common migration problems rather than a specific customer deployment.
Why We Wanted to Self-Host
The reasons were reasonable.
We wanted more control over:
- Model selection
- Infrastructure
- Deployment location
- Data handling
- Inference configuration
- Capacity planning
- Model upgrades
We also wanted the ability to optimize around our own workload instead of treating inference as an external black box.
So we introduced a self-hosted inference server behind the existing application.
Conceptually, the architecture changed from:
Before
Application
↓
Hosted GPT API
to:
After
Application
↓
LLM Gateway
↓
Self-Hosted Model Server
↓
GPU Infrastructure
The API looked familiar.
That created false confidence.
Break #1: API Compatibility Was Not Behavior Compatibility
One of our first assumptions was:
If the server supports an OpenAI-compatible API, our application should behave the same.
At the HTTP level, that was mostly true.
For example, servers such as vLLM expose OpenAI-compatible Chat Completions, Responses, embeddings, and other endpoints.
But receiving a familiar response structure such as:
{
"role": "assistant",
"content": "..."
}
does not mean two models will interpret the same conversation identically.
We saw changes in:
- Instruction following
- Response length
- Formatting
- Refusal behavior
- JSON reliability
- Tool selection
- Edge-case reasoning
The integration compiled.
The product behavior changed.
That was our first major mistake:
We tested protocol compatibility before behavioral compatibility.
Break #2: Our Prompts Were More Model-Specific Than We Thought
We had system prompts that had been tuned over months.
For example:
Return a concise answer.
Do not include commentary.
Use the supplied context only.
Return JSON matching the requested format.
They worked consistently enough with the previous model that we started treating them as application logic.
Then we moved models.
Some responses became longer.
Some ignored preferred formatting.
Some interpreted instructions differently.
Others needed clearer examples.
The prompts were not really portable.
They were part of the model integration.
Then We Discovered Chat Templates
With many self-hosted chat models, the messages sent by the application eventually need to become a token sequence such as:
<system>
You are...
<user>
...
<assistant>
But different models may expect different special tokens and formatting.
Hugging Face explicitly recommends using the chat template associated with the model because formatting that does not match training can hurt performance.
That meant our architecture was actually:
Application messages
↓
Chat template
↓
Tokenizer
↓
Model
not simply:
Application messages
↓
Model
A wrong template could produce valid inference with worse behavior.
That is a particularly dangerous failure because nothing crashes.
The model just becomes less reliable.
Break #3: Structured Output Became Our Problem
Several application workflows expected objects like:
{
"category": "billing",
"priority": "high",
"requires_human": true
}
With the hosted implementation, we relied heavily on schema-constrained responses.
OpenAI supports Structured Outputs and strict function schemas designed to make responses conform to supplied schemas.
Our first self-hosted implementation mostly relied on prompting:
Respond with valid JSON only.
It worked.
Until it did not.
We received responses such as:
Sure! Here's the JSON:
followed by JSON.
Or:
{
"priority": "HIGH"
}
when the schema expected:
high
Occasionally, we also received incomplete objects.
The migration exposed an architectural mistake:
Parsing model output is not validation.
We Added Schema Enforcement
Our gateway eventually owned structured-output validation.
Conceptually:
Model output
↓
Schema validation
↓
Valid?
┌──┴──┐
Yes No
↓ ↓
Return Retry / Fail safely
Modern inference servers can also support constrained or structured generation.
vLLM, for example, supports JSON Schema, regex, grammar, and other structured-output constraints.
That helped enormously.
But it also reinforced the larger lesson:
The hosted platform had been providing more than tokens. It had been providing behavioral infrastructure.
Break #4: Tool Calling Changed
Our application used tools for actions such as:
search_documents()
lookup_customer()
create_ticket()
get_order_status()
Before migration, we had logic built around a fairly predictable tool-call structure.
After migration, three things changed.
Sometimes the new model:
- Answered directly instead of calling the tool
- Chose the wrong tool
- Produced arguments that were technically valid but semantically poor
Tool calling is partly an inference-server feature, but it is also heavily model-dependent.
For example, vLLM requires model-specific tool parsers and configurations for automatic tool selection in many setups.
So this:
tools=[...]
was not enough.
We had to test:
Should the model call a tool?
Which tool should it choose?
Are the arguments correct?
What happens after the tool output returns?
Can it handle multiple tool steps?
Tool-use quality became its own evaluation suite.
Break #5: Latency Looked Great Until Concurrency Arrived
Our first benchmark was encouraging.
One request:
User
↓
Model
↓
Fast response
Then production-like traffic arrived.
Now it looked like:
Request ─┐
Request ─┤
Request ─┤
Request ─┼──→ GPU
Request ─┤
Request ─┤
Request ─┘
Suddenly we were debugging:
- Queue time
- Batching
- Token throughput
- Long prompts
- Long generations
- Memory pressure
- Concurrent requests
Single-request latency had told us almost nothing about production capacity.
The important metrics became:
Time to First Token
Tokens per Second
Queue Time
Requests per Second
P95 Latency
P99 Latency
A self-hosted model is not just a model.
It is a serving system.
Break #6: Context Length Became a Capacity Problem
Our application frequently supplied:
- System instructions
- Conversation history
- Retrieved documents
- Tool definitions
- User input
A request could become large quickly.
Under a hosted API, we mainly thought about context as a model limit and usage cost.
Self-hosting added another concern:
Memory capacity.
Longer contexts meant more inference memory and lower effective concurrency.
So a request that worked perfectly alone could reduce throughput when many users arrived together.
We started treating context as an infrastructure resource.
Instead of automatically sending:
Entire conversation
+
10 retrieved documents
+
All tool definitions
we became much stricter about:
- Document selection
- History trimming
- Prompt size
- Tool exposure
- Output limits
Prompt optimization became capacity optimization.
Break #7: Retries Became More Dangerous
Our hosted integration had retry logic.
Something failed?
Retry.
That behavior became dangerous when infrastructure was already overloaded.
Consider:
Server overloaded
↓
Request times out
↓
Client retries
↓
More load
↓
More timeouts
↓
More retries
Now we had created a retry storm.
We changed the gateway to use:
- Bounded retries
- Backoff
- Timeouts
- Concurrency limits
- Overload handling
Inference failure had become distributed-systems failure.
Break #8: We Became the Operations Team
Before:
POST request
↓
Receive result
After:
Model weights
GPU drivers
Inference server
Container images
Autoscaling
Networking
Load balancing
Monitoring
Logging
Model versions
Security
Capacity planning
Failure recovery
None of those are reasons not to self-host.
They are simply costs that need to exist in the migration plan.
We had moved responsibility across a boundary.
The managed API provider was no longer operating inference for us.
Self-hosting also turns model deployment into a cloud infrastructure and scaling problem, because GPU capacity, monitoring, networking, and failure recovery now belong to your team.
The Most Important Fix: Put a Gateway in Front
Initially, parts of the application called the inference server directly.
That made every model difference leak into product code.
We replaced that design with an internal LLM gateway.
Application
↓
LLM Gateway
↓
┌───────────────┐
│ Hosted Model │
│ Self-Hosted A │
│ Self-Hosted B │
└───────────────┘
The gateway owned:
- Request normalization
- Prompt versions
- Timeouts
- Retries
- Structured-output validation
- Tool schemas
- Logging
- Metrics
- Model routing
Application code stopped caring which inference backend produced the answer.
A stable gateway also makes enterprise AI development easier because applications can switch models, validate outputs, monitor behavior, and scale inference without coupling every workflow to one provider.
That made the migration much safer.
We Also Stopped Doing Big-Bang Model Swaps
Our first approach was basically:
GPT
↓
New model
Later migrations used:
Current model
↓
Shadow traffic
↓
Evaluation
↓
Small percentage
↓
Compare
↓
Increase traffic
↓
Full migration
The new model initially received traffic without controlling the user-visible response.
We compared:
- Answer quality
- JSON validity
- Tool selection
- Latency
- Token usage
- Failure rate
That surfaced problems before customers saw them.
Our Evaluation Suite Became the Real Migration Plan
The most useful artifact was not the deployment script.
It was our evaluation dataset.
We collected representative requests for:
Simple Q&A
RAG
Extraction
Classification
Tool calling
Long context
Ambiguous instructions
Bad input
Edge cases
For every candidate model, we tested the same workload.
Conceptually:
| Test | Existing Model | Self-Hosted |
|---|---|---|
| JSON schema compliance | PASS | PASS |
| Tool selection | PASS | FAIL |
| RAG answer quality | PASS | PASS |
| Long-context request | PASS | SLOW |
| Classification | PASS | PASS |
Now model selection was based on product behavior.
Not leaderboard scores.
What Broke
Looking back, the failures fell into a few categories.
Prompt Behavior
Prompts that worked well with one model needed adjustment.
Formatting
Structured output required stronger enforcement.
Tool Use
Tool selection and arguments needed fresh evaluation.
Performance
Single-request benchmarks hid concurrency problems.
Context
Large prompts reduced serving capacity.
Reliability
Retries, overload, and failover became our responsibility.
Operations
Inference infrastructure became part of the product.
What Worked
Several changes made the migration much safer.
A Stable Internal API
Application code talked to our gateway, not directly to a specific model.
Model-Specific Prompting
We stopped pretending every model needed identical prompts.
Schema Validation
Structured outputs were validated rather than trusted.
Production-Like Load Tests
We tested concurrent traffic instead of one request at a time.
Shadow Evaluation
New models were compared before receiving full traffic.
Versioned Models and Prompts
Every production request could be traced to:
Model version
Prompt version
Gateway version
Inference configuration
That made regressions much easier to investigate.
The Architecture We Ended Up With
The final system looked closer to this:
Application
│
▼
LLM Gateway
│
┌───────────────┼───────────────┐
▼ ▼ ▼
Prompt Manager Validation Observability
│ │ │
└───────────────┼───────────────┘
▼
Model Router
┌────┴────┐
▼ ▼
Hosted Self-Hosted
Model Models
The key abstraction was no longer:
callModel()
It was:
executeLLMWorkflow()
because production AI applications depend on much more than generation.
What We Would Do Differently
For the next self hosted LLM migration, we would use this order.
1. Build Evaluations First
Build the evaluation suite before choosing the replacement model.
2. Inventory Every Model-Dependent Feature
Include:
- Structured output
- Tools
- RAG
- Vision, if applicable
- Context length
- Streaming
3. Introduce an LLM Gateway
Put the gateway in place before changing providers or models.
4. Validate the Chat Template
Make sure the model receives messages in the format it was trained to understand.
5. Load-Test Realistic Prompts
Do not rely only on tiny benchmark requests.
6. Test Tool Behavior Separately
API support does not prove that the model uses tools correctly.
7. Run Shadow Traffic
Compare production workloads before switching user-visible responses.
8. Plan Operations Before Launch
GPU capacity, monitoring, failure handling, scaling, and upgrades belong in the original migration scope.
The Biggest Lesson
We originally thought we were replacing this:
GPT
with this:
Self-hosted model
What we were really replacing was:
Model
+
Serving infrastructure
+
Structured output behavior
+
Tool behavior
+
Scaling
+
Reliability
+
Operational responsibility
That is a much larger migration.
And it explains why an application can successfully return its first self-hosted response while still being nowhere near production-ready.
Final Takeaway
A self hosted LLM migration should not begin with:
Which open model looks closest to our current GPT model?
Start with:
What behavior does our application depend on?
How will we measure that behavior?
Which parts are currently handled by the provider?
Which of those responsibilities are moving to us?
Then migrate in layers:
Inventory dependencies
↓
Build evaluations
↓
Add gateway
↓
Deploy model
↓
Validate outputs
↓
Load test
↓
Shadow traffic
↓
Gradual rollout
↓
Monitor
Self-hosting gave us more control.
But that control came with responsibility.
The real migration was not from one model to another.
It was from consuming inference as a service to operating inference as a production system.
Top comments (0)