DEV Community

Ayi NEDJIMI
Ayi NEDJIMI

Posted on

Ollama vs LM Studio vs vLLM: Running Local LLMs in Production

#devops #python #ai #llm

Running a language model locally sounds simple until you try to do it at scale. You have GPU servers sitting idle, latency requirements your cloud API cannot meet, or data you simply cannot send outside your network perimeter — and suddenly the choice of runtime matters enormously. This article cuts through the noise and tells you exactly when to pick each option.

The Three Contenders

Ollama is a single-binary server that wraps llama.cpp. It speaks a REST API nearly identical to OpenAI's, handles model downloads, and runs on Mac, Linux, and Windows. Setup takes under five minutes. The trade-off: it is designed for one request at a time and has limited batching support beyond that.

LM Studio is primarily a desktop GUI — useful for evaluation and rapid testing, but not production software. It ships a local server mode, but it is not built for concurrent load, automated deployment, or headless operation. Cross it off the production list immediately.

vLLM is a proper inference server built with production in mind. It implements PagedAttention for memory efficiency, supports continuous batching, runs multi-GPU setups with tensor parallelism, and exposes an OpenAI-compatible API. The cost: Python dependencies, CUDA, and a steeper setup curve.

Setting Up Ollama for a Development Workflow

Ollama is the right choice when you need a fast local loop: pull a model, call it, iterate. Here is a minimal Python client using its REST API:

import requests

def chat(prompt: str, model: str = "llama3.2") -> str:
    response = requests.post(
        "http://localhost:11434/api/chat",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": False,
        },
        timeout=60,
    )
    response.raise_for_status()
    return response.json()["message"]["content"]

if __name__ == "__main__":
    print(chat("Explain PagedAttention in two sentences."))
Enter fullscreen mode Exit fullscreen mode

Start Ollama with ollama serve, pull a model with ollama pull llama3.2, and this works immediately. No GPU required — it falls back to CPU with quantized models (Q4_K_M cuts an 8B model to roughly 5 GB).

The OpenAI-compatible endpoint at /v1/chat/completions means you can swap in any existing OpenAI client with just a base URL change:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "What is entropy in cryptography?"}],
)
print(resp.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

This portability is Ollama's biggest asset. You develop against it locally, switch to a production endpoint in staging, and no application code changes.

Deploying vLLM for Real Throughput

When you need to serve multiple concurrent users — anything beyond a personal assistant or internal CLI tool — Ollama's single-request model becomes the bottleneck. vLLM handles this with continuous batching: it fills GPU memory intelligently across in-flight requests rather than processing them one at a time.

A minimal Docker deployment on a single NVIDIA GPU:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dtype auto \
  --max-model-len 4096
Enter fullscreen mode Exit fullscreen mode

This spins up an OpenAI-compatible server at port 8000. Same client code as above, different base URL. For multi-GPU setups add --tensor-parallel-size 2 (or 4, 8). For quantization use --quantization awq or --quantization gptq to fit larger models in available VRAM.

What vLLM does not do: it does not manage model storage, does not auto-download models in a friendly way (you need HuggingFace tokens and the correct environment variables), and the configuration surface is large. It also requires CUDA — no CPU fallback, no macOS support.

How to Choose

Here is the honest decision matrix:

Scenario Pick
Local dev, single user Ollama
Evaluation / demo LM Studio or Ollama
Production, multi-user vLLM
No GPU available Ollama (CPU + quantization)
Compliance / air-gapped network Either (both work fully offline)
Windows server Ollama (vLLM is Linux/CUDA only)
Multi-GPU scaling vLLM

The hidden cost of vLLM is operational: you need someone who understands CUDA out-of-memory errors, tensor parallelism misconfiguration, and the HuggingFace model cache. For a team of two running an internal tool, Ollama on a single A10 handles a surprising amount of concurrent load — especially with a simple Redis queue in front of it.

For teams that do need vLLM's throughput, the combination that works well in practice is Ollama for dev/test environments and vLLM behind a reverse proxy in production. Application code never changes because both speak the same API contract.

Security Considerations

Running a language model locally does not automatically make it more secure. You are now responsible for the full stack:

  • Network exposure: Ollama binds to 0.0.0.0:11434 by default. Restrict it to 127.0.0.1 or put it behind a reverse proxy with authentication before any server that touches your internal network.
  • Model provenance: Models from HuggingFace or Ollama's registry are not reviewed for malicious weights by default. Verify checksums, know what you are running, and pin model versions in production.
  • Prompt injection: Local models are just as vulnerable to prompt injection as hosted ones. If your application feeds user-controlled input into the model, apply strict input validation and treat all model output as untrusted before acting on it.

If you are deploying in a regulated environment, our security hardening checklists cover AI/ML workloads alongside the usual network, container, and access control requirements.

The Takeaway

LM Studio is a tool for humans, not servers. Ollama is the right choice for development workflows, low-concurrency internal tools, and environments without a discrete GPU. vLLM is what you reach for when you need real throughput, multi-GPU scaling, or memory-efficient serving of larger models.

The OpenAI-compatible API shared by both Ollama and vLLM means the choice is an infrastructure decision, not an application code decision. Start with Ollama, instrument your concurrency, and migrate to vLLM when your GPU utilization numbers or p95 latency justify the operational overhead. Do not over-engineer before you have the traffic.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Top comments (0)