Originally published on AI Tech Connect.
What you need to know Self-hosting is a utilisation bet, not a price comparison. A rented GPU costs the same whether it serves one request a minute or two hundred. Self-hosting only beats an API when sustained load keeps that GPU busy — or when data residency leaves you no choice. vLLM is the sensible default engine. Its PagedAttention and continuous batching deliver an order-of-magnitude throughput gain over naive serving, it exposes an OpenAI-compatible API, and it installs in roughly three commands. Size VRAM from first principles. Weights are parameters multiplied by bytes per parameter; the KV cache is the variable that bites under concurrency. Plan for both, and never fill a card past about 85 to 90 percent. FP8 is the default quantisation; INT4 is a fitting tool. FP8 on H100-class…
Top comments (0)