When teams start building with AI, someone eventually raises the question of where the data actually goes. That concern is legitimate — but it often short-circuits into a conclusion before anyone has read the actual requirement driving it.
There are three real deployment options: a standard hosted API, an enterprise-tier hosted service with contractual data protections, and open-weight models running on your own infrastructure. Many projects land on option three without seriously evaluating option two. That skip is expensive.
What "Must Self-Host" Usually Means
Most mandates to keep AI inference local trace back to one of four sources: sector regulations, contract clauses with clients or vendors, internal security policies written before enterprise AI tiers existed, or security team guidelines applied conservatively.
The critical question is what the requirement actually constrains. There's a meaningful difference between a rule that governs data properties — who can use data for training, how long it's retained, what access controls exist — and one that requires physical custody of the model. Those two things sound related but lead to completely different architectures.
Get the specific language in writing before making any infrastructure decisions. "We can't send data to third parties" is a starting point, not a specification. Find out who approved the policy, what evidence would satisfy it, and whether it applies to all data or only specific categories.
The Enterprise Tier Most Teams Skip
Hosted providers offer business and enterprise plans that include terms many teams assume require self-hosting: no use of customer content for training, configurable or zero data retention, regional processing, tenant isolation, audit logging, and data processing agreements you can show to legal or compliance reviewers.
The difference between enterprise contract terms and standard terms is the difference between an enforceable promise and a best-effort practice. If a provider's enterprise agreement says they won't train on your data and they do anyway, you have contractual remedies. That's meaningfully different from relying on a public-facing privacy policy that can change.
Before pricing out GPU infrastructure, request written terms for the exact product your team would use. Verify what they say about training restrictions, retention windows, breach notification timelines, and deletion on request. Then get sign-off from whoever owns the compliance decision. This step often closes the self-hosting question entirely.
When Self-Hosting Actually Makes Sense
Some requirements genuinely can't be satisfied by any hosted service — specific regulatory frameworks, sensitive data categories, or client contracts with strict custody language. When that's the case, the economics deserve a realistic look before committing.
Hardware is rarely the largest cost. The fuller picture includes:
- Infrastructure for peak load, not average load — concurrent request spikes determine your sizing, not daily averages
- Non-production environments for testing and development
- Redundancy for availability requirements
- Engineering hours for ongoing maintenance, patching, and model evaluation cycles
- On-call coverage for inference failures
- Quality gaps between open-weight and frontier models, measured on your actual tasks, converted to human review costs
Build your cost model from real measurements. Sample a week of actual requests, count tokens, identify daily peaks, and project what 3x growth looks like. Then compare that number to the enterprise tier price, not to a rough internal estimate.
Hybrid Routing as the Middle Path
Many systems don't require uniform treatment of all data. Only certain fields, records, or request types fall under the actual mandate. In those cases, a hybrid approach routes only the constrained slice to a locally-hosted model while everything else goes through a hosted API.
Practical patterns include keeping embeddings local while running generation through a hosted service, using redaction at the boundary to strip sensitive fields before they leave your infrastructure, or using a local model as a preliminary classifier that routes requests based on content type.
The routing logic in a hybrid system becomes a security control in its own right. Test it against edge cases, log every routing decision, and configure it to fail closed — uncertain requests stay local rather than defaulting to the hosted path.
A Practical Decision Order
The sequence matters. Getting drawn into GPU pricing before confirming what the requirement actually says leads to expensive commitments that may turn out to be unnecessary.
A useful order:
- Get the requirement in writing and classify what it actually constrains
- Evaluate whether a hosted enterprise agreement satisfies it
- Measure real volume, latency targets, and peak concurrency before sizing anything
- Cost the self-hosted option end-to-end, including engineering and on-call time
- If neither fully fits, design the hybrid split around the specific data categories under constraint
Most mandates that initially seem to require self-hosting are satisfied by a hosted enterprise agreement once the terms are reviewed carefully. The ones that remain after that review usually represent genuine regulatory or contractual requirements — and at that point, the infrastructure complexity is justified.
This guide originally appeared on agentpalisade.com. Agent Palisade helps small and mid-sized businesses put AI to work inside the tools they already use — practical automation, internal assistants, and AI security reviews. Book a free 30-minute call.
Top comments (0)