Compare leading open-weight models, licenses, benchmarks, and deployment paths for self-hosted AI stacks.
By 2026, the landscape of open-weight language models has stabilized around a small number of dominant architectures. Llama, Mistral, and Qwen represent the three largest ecosystems. Teams building self-hosted AI stacks now face a simpler but still consequential choice: which model balances capability, cost, license clarity, and deployment ease for your specific workload. This guide compares these families on concrete dimensions: benchmark performance, inference cost, hardware requirements, and production considerations.
Why this matters now
Open-weight models have crossed a usability threshold. Llama 3.1 (405B) and Mistral Large outperform GPT-3.5 on standard benchmarks and rival GPT-4 on many tasks. The cost curve has inverted: self-hosting a 70B model on cloud infrastructure (AWS, GCP, Azure) now costs less per token than API access for consistent, high-volume inference. Equally important, commercial restrictions have clarified. Meta's Llama Community License, Mistral's Apache 2.0, and Alibaba's Qwen Apache license all permit commercial use without royalties.
The practical upside is real for teams with non-trivial inference volume (10M tokens/month or higher). A startup running a chatbot, classification pipeline, or content moderation system can reduce per-token costs by 60 to 80 percent compared to GPT-4 API pricing, while retaining full control over data and model behavior. The downside is operational: hosting, quantization, fine-tuning, and serving infrastructure now become your responsibility. This tradeoff has made open-source model selection a strategic, not just technical, decision.
Llama: Scale and stability
Photo by Leandro Rossi on Pexels.
Llama 3.1 remains the reference model for open-weight LLMs in 2026, with a 405B variant that matches GPT-4 Turbo on many benchmarks and an 8B version competitive with models triple its size.
Meta released Llama 3.1 (8B, 70B, 405B) in mid-2024 and has committed to regular updates. The 8B model achieves ~95 percent of 70B capability on most tasks, making it the de facto standard for edge deployment and cost-conscious teams. The 70B model is the "sweet spot" for production: strong enough for reasoning, code generation, and multi-turn dialogue, yet fits on dual A100 GPUs with int8 quantization. The 405B variant trains on 15.6 trillion tokens and competes directly with proprietary frontier models, though inference cost at this scale exceeds most open alternatives.
Llama benchmarks (MMLU, HumanEval, GSM8K):
Llama 3.1 8B: 66.7% (MMLU), 62.2% (HumanEval)
Llama 3.1 70B: 85.2% (MMLU), 81.7% (HumanEval)
Llama 3.1 405B: 87.6% (MMLU), 87.3% (HumanEval)
License: Llama Community License (permissive for commercial use, no royalties).
Deployment: Llama models are optimized for vLLM, which is the fastest open inference server on AWS and GCP. A single A100 40GB GPU serves Llama 70B int8 at 20-40 tokens/second depending on batch size. For cost-cutting, int4 quantization (GGUF or GPTQ) reduces VRAM by half with minimal quality loss (typically 1-2 percent on benchmarks, hard to detect in production). Llama 3.1 8B runs on a T4 GPU (16 GB) with int4 quantization, opening access to cost-conscious labs.
Mistral: Efficiency and speed
Mistral has prioritized model efficiency and inference speed over raw parameter count, resulting in smaller, faster models that often match or exceed larger competitors.
Mistral 7B (2023) proved that 7B parameters could compete with 13B models from other vendors when trained on curated data and optimized for inference. Mistral has since released Mistral 8x7B (Mixture of Experts), which offers 7B parameters but activates only 2.6B per token, and Mistral Large, which targets GPT-4 performance in a lean package. The 8x7B model is particularly interesting for teams with high throughput needs: it's fast, memory-efficient, and the MoE architecture reduces inference latency.
Mistral benchmarks:
Mistral 7B: 64.2% (MMLU), 47.4% (HumanEval)
Mistral 8x7B: 80.2% (MMLU), 74.4% (HumanEval)
Mistral Large: 84.0% (MMLU), 85.2% (HumanEval)
License: Apache 2.0 (unrestricted commercial use).
Deployment: Mistral models are optimized for TensorRT-LLM and are the fastest models to serve on NVIDIA hardware. Mistral 7B achieves 50+ tokens/second on a single A10G GPU. Mistral 8x7B, despite routing overhead, is nearly as fast and fits on dual T4s (16 GB each) with int4 quantization. Mistral has published official GGUF quantizations, making CPU inference practical for low-traffic services. A key advantage: Mistral models have smaller context windows (32K tokens for Mistral Large, versus 100K+ for newer Llamas), which lowers memory pressure and inference time for typical chat and RAG tasks.
Qwen: Multilingual and Chinese-first
Photo by Dr Photographer on Pexels.
Qwen (Alibaba) has built the strongest open-weight model for Chinese and multilingual tasks, with Qwen 2.5 now competitive on English benchmarks and code.
Qwen 2 and its successor Qwen 2.5 excel on Chinese, math, and code. The 72B variant outperforms Llama 3.1 70B on Chinese MMLU and Chinese coding tasks, a critical advantage for teams operating in Mandarin or serving multilingual audiences. Qwen's alignment for instruction-following is strong; the models are less prone to refusals than Llama and produce longer, more detailed responses. For non-English-speaking teams, Qwen should be the default consideration.
Qwen benchmarks (Qwen 2.5 72B):
English MMLU: 84.9%
Chinese MMLU (CMMLU): 88.1%
HumanEval: 83.6%
MATH (competition problems): 57.3%
License: Apache 2.0 (unrestricted).
Deployment: Qwen 2.5 models have long context windows (128K tokens) by default, which is ideal for RAG and document summarization but increases memory usage. A Qwen 72B model in int8 requires 144 GB VRAM, comparable to Llama 70B. However, context compression techniques (attention outliers, kv-cache quantization) reduce this by 20-30 percent with minimal accuracy loss. Qwen's official quantization toolchain (AutoGPTQ, GPTQ-Int4) is mature; int4 quantization is stable and widely tested by the Qwen community. For single-GPU setups, Qwen 32B (int4) is a strong alternative to Llama 70B on 40 GB hardware.
Practical sizing and hardware recommendations
Choosing a model tier depends on latency, throughput, and total cost. This matrix covers typical production scenarios.
Single GPU (24-40 GB VRAM): Llama 3.1 8B (full precision on A10G, A30) or Mistral 7B (full precision). Both achieve <100 ms latency for single-user requests. For throughput, quantize to int4 and expect 2x speedup. If you need more capability, Llama 3.1 70B int4 fits on a 40 GB A100, with latency of 300-500 ms per request in batch mode.
Dual GPU (40-80 GB VRAM): Llama 3.1 70B int8 (A100 40GB x2 or A6000 48GB x2). This is the standard for production RAG and chat systems. Expect 30-40 tokens/second in batch mode, or <200 ms latency for single requests. Mistral 8x7B on the same hardware is significantly faster (50-60 tokens/second) due to mixture-of-experts routing.
Multi-GPU cluster (80+ GB VRAM): Llama 3.1 405B or Qwen 2.5 72B (full precision). Llama 405B requires tensor parallelism across 4-8 GPUs depending on batch size and acceptable latency. This tier is cost-effective only for high-throughput services (100M+ tokens/day) or latency-insensitive batch processing.
CPU-only or edge devices: Mistral 7B or Llama 3.1 8B quantized to GGUF int4 (Ollama, llama.cpp). Expect 5-10 tokens/second on modern CPUs (2024 Xeon, AMD EPYC). This is viable for chatbots, autocomplete, and content filtering where sub-second latency is not required.
License, compliance, and commercial use
All three model families permit commercial deployment, but fine-print matters for high-stakes use cases.
Llama 2 and 3 use the Llama Community License, which allows commercial use, redistribution, and fine-tuning without royalties. The license does prohibit use to develop competing large language models, a restriction that rarely matters in practice. Meta has been clear that Llama models are free to use and modify.
Mistral and Qwen both use Apache 2.0, which is unrestricted. You can modify, sell, and relicense derived works. There are no "no competition" clauses.
In practice, all three are safe for commercial chatbots, content moderation, and data processing. If your use case is building a competitive LLM (training a model on a corpus and releasing it), Mistral and Qwen are slightly less ambiguous, though Meta's enforcement history suggests Llama is also fine in this context. Consult legal counsel before production deployment if you operate in regulated industries (finance, healthcare, legal).
Quantization, inference, and total cost of ownership
Quantization is not optional for cost-conscious production; it is the standard.
Int8 quantization (e.g., AutoAWQ, vLLM's FP8) reduces model size by 50 percent with negligible accuracy loss (<1 percent on benchmarks). It is reversible and widely supported. Int4 quantization (GPTQ, GGUF) cuts memory by 75 percent and often adds a small latency boost due to reduced memory bandwidth. Quality loss is minimal for most applications; on summarization, classification, and Q&A, int4 is invisible. On creative writing and reasoning tasks, you might notice a 2-3 percent drop in human preference scores.
Monthly inference cost for a typical production workload (1M tokens/day, one year commitment on AWS):
Llama 3.1 8B on g4dn.xlarge (A10G, $0.80/hr): $580/month
Llama 3.1 70B on p3.8xlarge (V100 x4, $24.48/hr): $18,000/month (or split across tenants)
OpenAI GPT-4 (API): $60,000+/month (at $0.06 per 1K tokens)
This math changes dramatically at scale. A 10M token/day workload on proprietary APIs costs $600k/year; the same on Llama 70B costs under $200k/year in cloud infrastructure plus operational overhead (monitoring, updates, security). The ROI breakeven for self-hosting is typically 5-10M tokens/day.
When open-source LLMs fail (and when to use APIs)
Open-weight models are not a universal replacement for commercial APIs. Understand the gaps.
Reasoning and planning: On complex, multi-step reasoning tasks (e.g., long-horizon planning, formal logic, scientific synthesis), GPT-4 and Claude 3.5 still lead by a visible margin. Llama 3.1 405B closes the gap on many tasks, but inference cost at that scale often exceeds API pricing. If your application is reasoning-heavy, test thoroughly before committing to self-hosted inference.
Rare or specialized knowledge: Open models are trained on publicly available data with a knowledge cutoff (April 2024 for Llama 3.1, earlier for most others). If your use case requires real-time information, recent events, or proprietary databases, pair any open model with retrieval-augmented generation (RAG). For knowledge-dependent tasks, RAG on an open model often outperforms an API without RAG, but setup and monitoring are more complex.
Safety and moderation: Commercial APIs include additional safety training, adversarial testing, and compliance scaffolding. Open models can produce harmful content when pushed. If you serve public users, budget for safety fine-tuning, red-teaming, and monitoring. Llama 3.1 is aligned better than older Llama 2, but not on par with commercial models.
Latency-critical applications: Commercial APIs guarantee SLA latency (e.g., <500 ms). Self-hosted inference adds operational complexity: autoscaling, failover, queue depth. For sub-100 ms latency requirements, cloud inference services (AWS Bedrock, GCP Vertex AI) or edge deployment (ONNX Runtime, TensorRT) may be more practical than managing GPU infrastructure.
Low-cost, low-volume workloads: If you need <1M tokens/month, API pricing is cheaper than the capital and operational cost of a GPU. The breakeven is roughly 5-10M tokens/month depending on region and instance type.
Deployment architectures and getting started
Three patterns dominate production deployments in 2026.
Managed inference on cloud VMs: Rent a g4dn.xlarge or p3.8xlarge on AWS EC2, GCP Compute Engine, or Azure. Run vLLM or TensorRT-LLM as a Docker container. Expose an OpenAI-compatible API endpoint (http://localhost:8000/v1/chat/completions). This approach gives you full control and pricing transparency, with minimal DevOps overhead. Cost is predictable, scaling is manual (resize the instance or add replicas behind a load balancer). Recommended for teams with <100M tokens/month.
Serverless or batch inference: Use Lambda, Cloud Run, or Batch for infrequent requests or batch jobs. Models are loaded cold on each invocation, so latency is high (5-30 seconds for first request), but you pay only for execution time. Cost-effective for one-off summarization, classification, or cron jobs. Not suitable for real-time chat or APIs.
On-premise or edge deployment: Run a quantized model (GGUF, ONNX) locally using llama.cpp, Ollama, or ONNX Runtime. No cloud costs, full data privacy, but limited capability (typically 8B or 13B models). Suitable for internal tools, local development, and embedded devices. Throughput is low (5-20 tokens/second on CPU), so batch processing and asynchronous architecture are required.
For teams just starting, the fastest path is: (1) Download a GGUF-quantized model (e.g., Mistral 7B or Llama 3.1 8B from Hugging Face). (2) Install Ollama and run it locally to validate behavior and latency. (3) If you need production-grade throughput, rent a g4dn.xlarge on AWS, deploy vLLM with the model, and expose an API. Total time to first inference: <30 minutes. Total time to production: 2-4 weeks including monitoring, logging, and safety testing.
Model selection decision tree
Use this framework to narrow the choice.
Do you need multilingual or Chinese capability? If yes, choose Qwen. Otherwise, continue.
Is latency critical (sub-200 ms)? If yes, prefer Mistral 7B or 8x7B (faster inference, smaller context window). Llama 3.1 8B is also good. Otherwise, continue.
Do you need cutting-edge reasoning or math? If yes, Llama 3.1 70B or 405B. Qwen 2.5 72B is also strong on math. Otherwise, continue.
What is your total inference volume per month? <1M tokens: use an API (ChatGPT, Claude). 1-10M tokens: Llama 3.1 8B on a single GPU. 10-50M tokens: Llama 3.1 70B on dual GPUs. 50M+: Consider Qwen 72B, Mistral 8x7B, or split across multiple instances.
Staying current
Open-weight model releases are now quarterly. Llama 3.2 (announced for late 2024) includes vision capabilities. Mistral and Qwen are on similar cadences. For long-term projects, plan for model upgrades every 6-12 months. The good news: new models are backward-compatible in API, so switching is a matter of redeploying weights and re-benchmarking.
Follow model releases via Hugging Face (huggingface.co/models), arxiv (arxiv.org/list/cs.CL), and vendor announcements. Benchmark new models on your specific use case before upgrading; published benchmarks often do not predict performance on custom tasks.
Production teams should freeze model versions for consistency. If you deploy Llama 3.1 70B today, do not auto-upgrade to Llama 3.2 without testing. Version pinning is standard practice and prevents silent degradation.
The decision to self-host an open-weight model is now simpler than it was in 2023: Llama, Mistral, and Qwen are all mature, well-documented, and commercially viable. The question is not whether to use them, but when the operational cost of self-hosting falls below the API alternative for your specific workload. For teams with consistent inference volume and data-privacy requirements, that crossover point is here. For others, APIs remain the right tool. Start with a proof-of-concept on a single GPU, measure real-world latency and quality on your data, and scale from there.
This article was originally published on AI Glimpse.
Top comments (0)