Running a large language model on your own hardware used to be a research project. You needed to understand quantization, manage GPU memory by hand, and write inference code against a low level library. Ollama changed the shape of that work. It packages models, serves them behind a simple HTTP API, and handles the details of loading, caching, and unloading on one machine. The interesting question is no longer whether you can run a model locally. It is whether you should run it in production, and what production means when the model is a process on a server you own rather than an API key for someone else's service.
The reasons to run locally are concrete. The first is privacy. Prompts and documents never leave your infrastructure, which matters for customer data, internal documents, and regulated industries where sending text to a third party API is not an option. The second is cost at steady volume. A hosted API charges per token, and the bill grows with every request. A local GPU has a fixed cost, and once you have paid for the hardware, a thousand extra requests cost almost nothing except electricity. The third is latency and availability. A local model answers in tens of milliseconds of network time, because the network is a local socket, and it keeps working when the internet connection does not. The fourth is control. You choose the model, the version, and the quantization, and no provider update changes your behavior overnight.
The reasons not to run locally are equally concrete. A hosted API gives you the largest and most capable models, updated constantly, with no hardware to buy or maintain. A local setup gives you smaller models, which are less capable at reasoning, long context, and edge cases. A local GPU is a single point of failure that you must monitor, cool, and replace. A hosted API scales to a spike in traffic without you doing anything, while a local server saturates and queues. The honest framing is that local models are right for high volume, well understood tasks where privacy or cost matters, and hosted APIs are right for tasks where capability matters most. Many production systems use both, with the local model handling the routine work and the hosted model handling the hard cases.
Model selection starts with quantization, which is the tradeoff that determines everything else. A model is a set of weights, and quantization stores those weights with fewer bits. A 7 billion parameter model at 16 bit precision needs about 14 gigabytes of memory. The same model quantized to 4 bits needs about 4 gigabytes, and it runs on a consumer GPU or even a good CPU. The cost is a small loss in quality, which is acceptable for many tasks and unacceptable for some. Ollama models are distributed pre quantized, with names that encode the size and the quantization, so picking a model is really picking a point on the quality versus resource curve. Start with a small model, measure it on your real prompts, and only move to a larger one when the small one fails in a way you can describe.
Getting started is deliberately simple. Install Ollama, pull a model, and it is serving on localhost. The commands below are the whole setup, and the simplicity is part of the appeal: there is no cluster to configure and no deployment pipeline to build for the first version.
ollama pull llama3.1:8b
ollama run llama3.1:8b "Explain what a database index is in two sentences."
curl http://localhost:11434/api/tags
The API is the interface your application will actually use, and it is worth learning directly rather than through a wrapper. The generate endpoint takes a prompt and returns a response, and the chat endpoint takes a list of messages with roles, which is the right shape for a conversational feature. Both endpoints support streaming, which is important for user experience: a response that streams token by token feels fast even when the total time is the same, because the user sees the first words immediately.
curl http://localhost:11434/api/chat -d '{
"model": "llama3.1:8b",
"messages": [
{ "role": "system", "content": "You are a helpful support assistant. Answer briefly." },
{ "role": "user", "content": "How do I reset my password?" }
],
"stream": false
}'
Streaming is a separate concern from generation, and the application code should handle it explicitly. The code below calls the chat endpoint from Go, reads the streamed response line by line, and prints each token as it arrives. The important detail is that the response body is a stream of JSON objects, one per token, and the client must read it incrementally rather than waiting for the whole body.
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"net/http"
)
type chatRequest struct {
Model string `json:"model"`
Messages []chatMsg `json:"messages"`
Stream bool `json:"stream"`
}
type chatMsg struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatChunk struct {
Message chatMsg `json:"message"`
Done bool `json:"done"`
}
func main() {
reqBody, _ := json.Marshal(chatRequest{
Model: "llama3.1:8b",
Stream: true,
Messages: []chatMsg{
{Role: "user", Content: "Explain connection pooling in two sentences."},
},
})
resp, err := http.Post("http://localhost:11434/api/chat",
"application/json", bytes.NewReader(reqBody))
if err != nil {
panic(err)
}
defer resp.Body.Close()
scanner := bufio.NewScanner(resp.Body)
scanner.Buffer(make([]byte, 1024*1024), 1024*1024)
for scanner.Scan() {
var chunk chatChunk
if err := json.Unmarshal(scanner.Bytes(), &chunk); err != nil {
continue
}
fmt.Print(chunk.Message.Content)
if chunk.Done {
break
}
}
fmt.Println()
}
The scanner buffer line deserves attention. The default scanner buffer is small, and a long token or a long line can exceed it, which silently stops the stream. Setting an explicit buffer is the kind of detail that separates a demo from a production client, and it is exactly the kind of detail that only shows up under real load.
Production operation starts with the questions that have nothing to do with the model. How many requests can one server handle at once? Ollama loads a model into memory, and concurrent requests share it, but each request needs its own context window, which consumes memory and compute. The number of parallel requests is limited by GPU memory and by the context size you configure. A server that handles ten concurrent short prompts may handle only two concurrent long ones. Measure the concurrency on your hardware with your prompts, and set a limit in your application or your load balancer so that the eleventh request waits in a queue with a clear timeout instead of degrading all ten.
Health checking and monitoring are the next layer. Ollama exposes the list of running models and the version endpoint, and your health check should verify that the model you need is actually loaded, not just that the process is alive. A process that is running but has evicted your model will answer every request with a slow cold load, and a health check that only checks the port will miss it. Metrics to track are time to first token, total generation time, tokens per second, queue depth, and GPU memory use. Time to first token is the metric your users feel, because it determines how long they wait before anything appears on screen.
Security is often an afterthought in local setups, and it should not be. Ollama binds to localhost by default, and that default is correct. The moment you expose it on a network, anyone who can reach it can use your GPU, read your prompts, and run any model you have installed. If a service on another host needs access, put a reverse proxy in front of it with authentication, and never expose the Ollama port directly to the internet. The same rule applies to any internal tool: the API has no authentication of its own, so the network boundary and the proxy are the security.
Retrieval augmented generation, usually called RAG, is the pattern that makes a local model useful for your own data. The model itself does not know your documentation, your tickets, or your product. A RAG system stores embeddings of your documents in a vector store, retrieves the relevant chunks for each question, and includes them in the prompt. Ollama serves embedding models as well as chat models, so the same server can produce the embeddings and answer the questions. RAG quality depends far more on retrieval than on the model: if the retrieved chunks are wrong, even a perfect model gives a wrong answer. Spend effort on chunking, the retrieval query, and evaluation, not on chasing a slightly larger model.
Evaluation is the discipline that keeps a local model honest. A hosted model comes with a reputation, but a local model comes with a choice you made, and that choice needs evidence. Build a set of prompts that represent your real workload, with expected answers or at least expected properties, and run them against every model and every quantization you consider. Check for factual accuracy, for format compliance, and for the failure modes that matter in your product, such as inventing a link or a function that does not exist. Re run the evaluation when you change the model, the prompt, or the retrieval step, because each of those changes can move the results.
The deployment shape that works for most teams is simple: one Ollama server per GPU machine, behind a reverse proxy, with your application calling it over HTTP. If you need more capacity, add more machines and load balance across them, and keep the model list identical on each one so any server can answer any request. If you need high availability, run at least two servers, because a single GPU machine is a single point of failure with a real mean time to repair. Kubernetes can manage this, but it is not required. A plain server with systemd, monitoring, and a runbook is a legitimate production setup, and it is simpler to debug than a GPU operator in a cluster you do not fully understand.
The decision to run local models is ultimately about control versus capability. You gain privacy, predictable cost, and independence from a provider. You accept smaller models and hardware to operate. For summarization, classification, extraction, and question answering over your own documents, a local model is often more than good enough, and the privacy and cost advantages are decisive.
Top comments (0)