Self-hosting AI gets pitched as either trivial (just run Ollama) or impossible (you need a data center). Both miss the same thing: a working stack is five layers, and the model is only one of them. Here is how the layers fit and why skipping one is what usually breaks a self-hosted setup.
Inference: The Layer Everyone Starts With
The inference layer runs the model itself. Tools like Ollama make this the easy part now, a single command on a mid-range GPU. Because it is easy, people stop here and assume the rest will follow. It does not. Inference alone gives you a chat box, not a system that does work.
Embeddings And Memory: Where Real Use Cases Live
Embeddings turn your documents into vectors, and a vector store lets the system search them by meaning. Memory is what lets the stack remember past runs instead of starting cold every time. These two layers move you from answering questions to working with your actual data, and they are the layers most first attempts leave out.
Tools And Orchestration: Turning Answers Into Actions
The tool layer lets the model call real functions, hit an API, run a query, write a file. Orchestration is the layer that decides the order, handles handoffs, and keeps multi-step work on track. Without these, you have a smart text generator. With them, you have something that completes tasks.
The Cost Question
The reason to bother is control and cost. Every byte stays on hardware you own, and for steady high-volume workloads a self-hosted stack tends to break even against cloud APIs within three to six months, on hardware that can start around a single mid-range GPU.
Start Small, Add Layers
Stand up inference first, confirm it works, then add embeddings and memory, then tools and orchestration, one verified layer at a time. The full breakdown of each layer and what to run for it is here: https://www.autolearningagents.com/self-hosted-ai-stack/
Top comments (0)