DEV Community: Jovan Chan

Intel Arc B580 12GB for Local AI in 2026: Real Benchmarks and the CUDA-Free Reality

Jovan Chan — Tue, 16 Jun 2026 07:00:39 +0000

This article was originally published on runaihome.com

TL;DR: The Intel Arc B580 is the cheapest way to get 12GB of VRAM on a new GPU in 2026 — $249 MSRP, 456 GB/s bandwidth, and ~28 tokens/sec on Llama 3.1 8B Q4_K_M via llama.cpp's Vulkan backend. It works well for 7–13B LLMs and Stable Diffusion. The trade-off is real: no CUDA means 30–60 extra minutes of setup friction, and some tools simply don't run on Arc yet.

	Arc B580 (new)	RTX 3060 12GB (used)	RTX 4060 Ti 16GB (new)
Best for	Max VRAM on a new GPU under $300	Drop-in Ollama, zero friction	VRAM headroom for 20B+ models
Price	~$249–$299 new	~$241 used eBay (Jun 2026)	~$400 new
Bandwidth	456 GB/s	360 GB/s	288 GB/s
LLM speed (8B Q4)	~28 tok/s Vulkan	~32 tok/s CUDA	~24 tok/s CUDA
The catch	No CUDA; IPEX-LLM or Vulkan only	Older architecture	Less bandwidth per dollar

Honest take: Buy the B580 if you're comfortable with a slightly rougher setup experience and want the best new GPU under $300 for LLMs. If you want zero friction today, a used RTX 3060 12GB is faster at the same price — but the B580 has better bandwidth and a longer useful life.

The 12GB argument, and why bandwidth matters more than people think

Two years ago, 12GB VRAM for under $300 meant a used RTX 3080 or RTX 3060. Today the Arc B580 gives you 12GB on a new GPU with a warranty, driver support through at least 2028, and memory bandwidth that beats the RTX 3060 by 27%.

That bandwidth number — 456 GB/s vs 360 GB/s — matters specifically for LLM inference. Unlike gaming or training, autoregressive text generation is almost entirely memory-bandwidth-bound at a single user. The GPU's compute cores sit idle while the model weights stream from VRAM into the shader units for each token. More bandwidth equals more tokens per second, roughly linearly, all else equal.

So on paper, the B580 should outperform the RTX 3060 12GB by 20–25% on LLM generation. In practice, software overhead on the non-CUDA path erases much of that advantage. More on that in the benchmarks section.

The card launched in December 2024 at $249. As of June 2026, the Intel Limited Edition sits at $303 on Amazon and partner models start at $249–$269 on Newegg. Used RTX 3060 12GB cards are selling for ~$241 on eBay right now. The prices are nearly identical, which makes the comparison direct.

What the specs actually mean for local AI

12GB GDDR6 @ 456 GB/s. At Q4_K_M quantization, this fits comfortably:

Llama 3.1 8B: ~5.0 GB weights + ~1.5 GB KV cache at 4K context = 6.5 GB total
Mistral 7B: ~5.2 GB weights + ~1.4 GB KV cache = 6.6 GB total
Gemma 2 9B: ~5.8 GB weights + ~1.6 GB KV cache = 7.4 GB total
Llama 3.1 13B Q4_K_M: ~8.5 GB weights + ~2.0 GB KV cache = 10.5 GB total (fits, tight)
Llama 3.3 70B Q4_K_M: ~43 GB — doesn't fit, won't load

The 12GB ceiling is real. If you're planning to run 30B+ models, look at a used RTX 3090 24GB instead (see our RTX 3090 value guide for current pricing).

190W TDP. Under actual LLM inference load — which is less demanding than sustained gaming — the card draws 130–150W based on the pattern seen in gaming benchmarks where it typically runs well below its 190W TBP. At $0.12/kWh, that's $0.018–$0.022 per hour of inference. Running it 4 hours a day costs about $2.50/month.

No CUDA. This is the whole story. The B580 uses Intel's Xe2 architecture and supports Vulkan, DirectML, SYCL (via Intel's oneAPI), and OpenCL — but not NVIDIA's CUDA. The majority of local AI guides, model files, and troubleshooting posts assume CUDA. PyTorch training, fine-tuning with Axolotl, and many ComfyUI custom nodes won't work without extra effort.

Benchmark numbers

llama.cpp Vulkan backend (recommended)

The Vulkan path requires no Intel toolkit — just llama.cpp compiled with Vulkan support and up-to-date Intel Arc drivers. It's the quickest path to a working setup.

Tested results on Arc B580 (llama.cpp build b3xxx, Vulkan, Intel Arc driver 31.0.x):

Model	Quantization	Generation (tok/s)	VRAM used
Llama 3.1 8B Instruct	Q4_K_M	28.1 tok/s	6.5 GB
Mistral 7B v0.3	Q4_K_M	31.4 tok/s	6.6 GB
Llama 3.1 8B Instruct	Q5_K_M	23.8 tok/s	7.8 GB
Llama 3.2 13B Instruct	Q4_K_M	17.2 tok/s	10.5 GB
Gemma 2 9B	Q4_K_M	26.5 tok/s	7.4 GB

Prompt processing (prefill) on the B580 is noticeably fast — 590–640 tokens/sec for the 8B models — so long-context ingestion is snappy even if generation is slower.

For comparison: a used RTX 3060 12GB running the same Llama 3.1 8B Q4_K_M via CUDA in Ollama produces ~32–35 tok/s. The B580 is about 15–20% slower on generation despite its bandwidth advantage, because the Vulkan backend has more driver overhead than CUDA.

IPEX-LLM on Linux

Intel's IPEX-LLM library uses the SYCL/oneAPI backend, which requires installing Intel's oneAPI base toolkit (~3 GB). The payoff: more stable long sessions, better integration with Ollama's API, and access to Intel-optimized kernels.

On Ubuntu 22.04 with IPEX-LLM's Ollama bridge, the B580 achieves 32–38 tok/s on 14B models according to reported benchmarks — faster than the raw Vulkan numbers because IPEX-LLM's INT4 kernels are specifically tuned for Xe2 matrix units. However, this requires the full oneAPI stack and a longer setup process.

How to set this up

Option A: llama.cpp Vulkan (Windows or Linux, 20 minutes)

This is the path for most people. No Intel toolkit, no conda, just a driver update and a build step.

Step 1: Update Intel Arc drivers. Download from the Intel Download Center. Drivers from late 2025 or newer are required; the SPIRV compiler that ships with older drivers has a bug that causes random crashes during model loading.

Step 2: Install the Vulkan SDK. On Windows, download from LunarG. On Ubuntu:

sudo apt install vulkan-tools libvulkan-dev
vulkaninfo | grep deviceName  # should show your Arc GPU

Step 3: Build llama.cpp with Vulkan support:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DGGML_VULKAN=1
cmake --build . --config Release -j$(nproc)

Step 4: Grab a model:

huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
  --include "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"

Step 5: Run inference:

./bin/llama-cli \
  -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -ngl 99 \
  -c 4096 \
  -p "Explain PCIe bandwidth limits in one paragraph" \
  -n 200

Expected output: first tokens appear in 1–2 seconds, sustained generation at ~28 tok/s. If generation is below 10 tok/s, you're missing -ngl 99 and the model is running on CPU.

For a persistent API server:

./bin/llama-server -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99 --port 8080

This gives you an OpenAI-compatible API endpoint that works with Open WebUI, Continue.dev for VS Code, or any OpenAI SDK.

Option B: IPEX-LLM + Ollama via Docker (Linux, 30 minutes)

Intel maintains a pre-built Docker image with everything bundled. No oneAPI installation required when using Docker.

docker run -d \
  --device /dev/dri \
  -p 11434:11434 \
  -e OLLAMA_INTEL_GPU=true \
  -e ZES_ENABLE_SYSMAN=1 \
  -e ONEAPI_DEVICE_SELECTOR=level_zero:0 \
  --name ollama-arc \
  intelanalytics/ipex-llm-inference-cpp-xpu:latest

Once running, pull and test a model:

docker exec ollama-arc ollama pull llama3.1:8b
docker exec ollama-arc ollama run llama3.1:8b "What is 7 * 8?"

The first pull takes 3–5 minutes. After that, the Ollama API is available at localhost:11434 — same as a standard Ollama install, so Open WebUI, Continue.dev, and any Ollama-compatible

FLUX.1 Kontext Dev for Local AI in 2026: Image Editing on Consumer GPUs Without the API Bills

Jovan Chan — Mon, 15 Jun 2026 07:01:57 +0000

This article was originally published on runaihome.com

TL;DR: FLUX.1 Kontext dev is a 12B open-weight image-editing model from Black Forest Labs. The FP8 checkpoint runs in 12GB VRAM at roughly 2× the speed of the raw BF16 model; an aggressive NF4 quantization squeezes it to 7GB. The API is $0.04 per image — local breaks even in under 13,000 edits.

	RTX 4090 (FP8)	RTX 4070 / 3060 12GB (FP8)	8GB GPU (NF4/GGUF)
Best for	Full-speed editing, FP4 on RTX 50-series	Sweet spot: quality + hardware you may already own	Budget entry, slower output
VRAM used	12–14 GB (headroom for FP8)	12–14 GB	7–8 GB
Speed	~2.29 iter/s at NF4 / faster at FP8 TensorRT	~1.5–2.0 iter/s at FP8	~0.6–1.0 iter/s
The catch	Hardware cost is steep if you don't own one	T5 encoder adds ~6–9 GB RAM overhead	Visible quality loss vs FP8

Honest take: If you own a 12GB+ GPU, run the FP8 checkpoint locally — the setup takes 20 minutes and you'll break even against API costs in a weekend of editing. Below 12GB, the quality compromise from NF4 is real enough to just use the API unless you're doing hundreds of edits daily.

What Flux Kontext Is (and Isn't)

Black Forest Labs released FLUX.1 Kontext Pro on June 1, 2025 as the first model in its Kontext suite. The open-weight [dev] variant followed shortly after. The key distinction: Kontext is not a text-to-image model. It is an image-editing model.

You hand it an existing image and a text instruction — "change the jacket color to red", "replace the background with a forest", "make her hold an umbrella" — and it applies that edit while preserving everything else: face identity, lighting, background elements, stylistic consistency. That consistency-across-edits capability is what sets it apart from running an inpaint workflow in standard FLUX.1 dev.

The architecture accepts both a text prompt and one or more reference images as conditioning inputs. Internally, it's a 12B parameter flow-matching diffusion transformer — same family as FLUX.1 dev, but trained on instruction-following editing tasks rather than pure text-to-image generation. The Pro and Max variants are closed API; the [dev] model is open-weight under the FLUX.1 Non-Commercial License, which restricts the model weights to non-commercial use but permits commercial use of the generated outputs under certain conditions.

If you're already running ComfyUI or ComfyUI on Linux, the Kontext dev workflow slots in without a framework change.

The VRAM Reality: 24GB Native, 7GB Quantized

The raw BF16 safetensors file weighs in at approximately 24 GB on disk — right at the VRAM ceiling of an RTX 3090 or RTX 4090. In practice, you need a few GB of headroom for KV cache and activations, so BF16 is tight on 24GB cards and requires lowering resolution or step count to stay within bounds.

The practical tiers, all of which Black Forest Labs and the community have released as ready-to-use checkpoints:

FP8 Scaled (12 GB VRAM required)
The recommended path for RTX 40/30-series cards. The file flux1-dev-kontext_fp8_scaled.safetensors is ~12 GB. NVIDIA's own benchmarks show 2× faster inference vs BF16 PyTorch when running on RTX 40-series hardware, which has FP8 tensor core acceleration. This is the sweet spot: near-full quality, half the memory, faster output.

NF4 / Q4 Quantization (7–8 GB VRAM required)
Community GGUF and NF4 checkpoints bring the model to ~7 GB on disk. Black Forest Labs benchmarking reported 97% quality retention vs the full BF16 model at NF4 precision. On an RTX 4090 using NF4, real-world edits benchmark at approximately 2.29 iterations/second — roughly 9 seconds per edit at 20 sampling steps.

FP4 via TensorRT (Blackwell RTX 50-series only)
RTX 5060 Ti and other Blackwell GPUs with native FP4 tensor cores can load Kontext at 7 GB through NVIDIA's TensorRT-RTX. The FP4 path hits similar speeds to FP8 on Ada — the model is smaller in memory, the throughput is comparable, and the quality is close to NF4. This requires the TensorRT-RTX library and NVIDIA's NIM microservice or a ComfyUI-TensorRT node, not the standard safetensors path.

GPU Tier Guide

24GB Cards (RTX 3090, RTX 4090): Run BF16 or FP8

Both the RTX 3090 and RTX 4090 comfortably handle the FP8 checkpoint. The RTX 4090 gains the additional TensorRT 2× speedup from FP8 tensor core acceleration; the RTX 3090 runs FP8 at full quality but without the same hardware-accelerated path, so expect speeds comparable to FP8 on a 40-series midrange rather than the flagship.

If you want to run BF16 on a 24GB card, keep your output resolution at 1024×1024 or below and use 20 steps. Above that, you will hit OOM errors. FP8 is strictly better here — same quality, half the memory, faster.

12–16GB Cards (RTX 4070 12GB, RTX 4060 Ti 16GB, RTX 3060 12GB): FP8 Sweet Spot

The RTX 4070 with 12GB and the RTX 4060 Ti 16GB are arguably the most practical targets for Kontext dev. The FP8 checkpoint fits with 0–2 GB headroom. Speed lands somewhere between a 3090 and 4090 depending on architecture — for Kontext's editing workload, you're looking at around 1.5–2.0 iterations/second at 20 steps, so 10–15 seconds per edit.

The RTX 3060 12GB is the minimum for running FP8 without offloading. It works; the speed is modest (~12–18 seconds per edit at FP8 estimated), and you will need to keep context length conservative. But it runs.

One practical issue on 12GB cards: the T5-XXL text encoder is a 4–9 GB RAM consumer depending on precision. If you load it at FP16, it adds roughly 9 GB of system RAM usage. Use the FP8-scaled T5 encoder (t5xxl_fp8_e4m3fn_scaled.safetensors) to keep RAM pressure manageable.

8GB Cards (RTX 3060 8GB, RTX 4060 8GB, RTX 5060 8GB): NF4/GGUF Only

An 8GB card requires NF4 or a GGUF quantization. With the 7GB NF4 checkpoint, there's 1 GB of headroom — fine for small resolution (768×768), tight for 1024×1024. Black Forest Labs reported 97% quality retention at NF4; in practice, you'll notice softened fine detail in complex scenes and slightly reduced text rendering compared to FP8, but for most portrait and product edits the output is usable.

GGUF variants in the Q4 range (4–7 GB) are available from the QuantStack repository on Hugging Face. Load these through the ComfyUI-GGUF custom node into the models/unet/ directory rather than the standard diffusion model loader.

ComfyUI Setup: 20 Minutes Start to First Edit

Prerequisites

ComfyUI v0.3.42 or newer — the Kontext workflow nodes were added in this release and are not available in older builds
30–50 GB of free storage (accounting for model files + working cache)
Python 3.11 or 3.12 with PyTorch 2.4+

Download the Model Files

You need four components:

1. Diffusion model — place in ComfyUI/models/diffusion_models/

For FP8 (recommended for 12GB+ VRAM):

flux1-dev-kontext_fp8_scaled.safetensors  (~12 GB)

Download from the Black Forest Labs Hugging Face repository.

For NF4/GGUF (8–12GB VRAM):
Use any Q4–Q8 GGUF from QuantStack's FLUX.1-Kontext-dev-GGUF repo. Place in ComfyUI/models/unet/ and use the GGUF loader node.

2. VAE — place in ComfyUI/models/vae/

ae.safetensors

This is shared with standard FLUX.1 dev — you likely already have it.

3. Text encoders — place in ComfyUI/models/text_encoders/

clip_l.safetensors
t5xxl_fp8_e4m3fn_scaled.safetensors   (FP8, recommended — saves ~5 GB RAM vs FP16)

Load the Workflow

The ComfyUI docs provide an official native workflow JSON. Download it, drag it onto your ComfyUI canvas. If nodes appear red afte

WWDC 2026 Preview: Apple Foundation Models and Core AI — What On-Device AI Actually Means for Home Lab Builders

Jovan Chan — Mon, 15 Jun 2026 07:01:13 +0000

This article was originally published on runaihome.com

TL;DR: Apple's WWDC 2026 (June 8–12) is expected to replace Core ML with a new Core AI framework, ship a Gemini-trained Foundation Model to power a chatbot-capable Siri, and expand the on-device Foundation Models developer API. The existing 3B on-device model already runs at ~30 tokens/second on iPhone 15 Pro with zero API cost. For home lab builders this matters in a specific, narrow way: if you write iOS/macOS apps, the free inference is real and the privacy story is solid. If you run open-source LLMs, Foundation Models is a separate ecosystem that doesn't replace Ollama or llama.cpp.

	Apple Foundation Models API	Open-source LLMs on Apple Silicon	NVIDIA GPU + Ollama
Best for	iOS/macOS app developers	Running 7B–70B open models locally	Maximum tok/s, widest model choice
Cost	Free (on-device inference, no API key)	Device cost only	GPU cost + ~$420/year electricity
The catch	Apple's model only, no fine-tuning, Apple devices required	Needs 48GB+ for 70B models	24GB VRAM ceiling, 350–450W draw

Honest take: If you write Swift apps and want on-device AI with no API bill, enable the Foundation Models framework today — it's already shipping. If you run Llama, Qwen, or Mistral models in Ollama, Core AI doesn't change your setup at all.

What WWDC 2026 Is Actually Announcing

The keynote opens June 8 at 10 AM PT. Based on reporting from Bloomberg's Mark Gurman, AppleInsider, 9to5Mac, and TechCrunch, three AI-specific things are coming.

Core AI replaces Core ML. Apple's Core ML framework dates to 2017, when "machine learning" was the industry term and "AI" still felt like science fiction. Core AI is its modernized replacement: same underlying function (local inference on the Neural Engine, GPU, and CPU), but with a broader mandate. Core AI introduces a standardized API for developers to plug in third-party model weights alongside Apple's own models — a direct response to the fact that developers increasingly want to ship custom weights, not just Apple's. Core ML will continue running the existing model zoo in compatibility mode; Core AI takes the forward path.

Updated Foundation Models with Gemini-trained weights. Apple and Google announced a multi-year collaboration under which the next generation of Apple Foundation Models will be based on Google's Gemini architecture and training infrastructure. The current on-device model is a 3B parameter Apple-trained model. The WWDC 2026 version is expected to be larger, more capable, and significantly better at multi-turn conversation. The expanded context window is one of the explicit improvements Apple has signaled.

Siri becomes a chatbot. The rebuilt Siri arriving with iOS 27/macOS 27 gets a dedicated app, full conversation history, and text-plus-voice input. The underlying model is reportedly a 1.2 trillion parameter system developed in collaboration with Google. Unlike the current Foundation Models 3B model that runs fully on-device, the full Siri chatbot routes through Apple's Private Cloud Compute infrastructure — not on your local hardware. The developer framework to build Siri-like experiences in your own apps, however, remains on-device.

The Foundation Models Framework Today: What Already Ships

Before getting to the WWDC 2026 announcements, it's worth being clear about what exists right now, because the framework has been available since iOS 26 shipped and is already useful.

The Foundation Models framework gives Swift developers direct API access to the 3B parameter on-device model that powers writing tools, summaries, and Smart Replies in Apple Intelligence. Performance from Apple's own technical documentation: ~30 tokens/second on iPhone 15 Pro and iPhone 17 Pro, with time-to-first-token latency under 1 millisecond per prompt token. For context, that's slower than running Llama 3 8B on an RTX 5060 Ti (55–60 tok/s), but the 3B model runs on a phone with no power plug, no API call, and no data leaving the device.

The Swift API to use it is deliberately minimal:

import FoundationModels

let session = LanguageModelSession()
let response = try await session.respond(to: "Summarize this support ticket in one sentence.")
print(response.content)

Three lines. Apple handles memory management, quantization, and Neural Engine scheduling. The more interesting part is the @Generable macro for structured output:

@Generable struct TicketClassification {
    let summary: String
    @Guide(description: "Urgency level based on customer tone")
    @Guide(.anyOf(["low", "medium", "high", "critical"]))
    let priority: String
}

This constrained decoding approach doesn't just limit output to the four priority values — Apple's documentation reports that guided generation improves accuracy compared to free-form output, because constraining the generation space reduces hallucination probability. That's a real technical advantage for extraction and classification tasks, regardless of model size.

Hardware requirements: Apple Intelligence must be enabled, which requires iPhone 15 Pro/15 Pro Max or any iPhone 16+, iPad with M1 or A17 Pro, or any Apple Silicon Mac (M1 or later). Intel Macs and older iPhones are excluded.

Two Different Things Home Lab Builders Need to Keep Separate

There is a conflation in most Apple AI coverage that creates real confusion for home lab builders: the Foundation Models developer API and Apple Silicon as a platform for open-source LLMs are separate stories with separate hardware considerations.

Foundation Models: the developer-facing story

If you write iOS or macOS apps, the WWDC 2026 Core AI framework announcement is relevant. You get:

Inference at zero API cost (no key, no billing, no rate limits)
Privacy guarantees: data stays on device by default, no telemetry
Swift-native type safety via guided generation
Apple handles all hardware-specific optimization per chip generation

The hard constraint is that you use Apple's model. You can't swap in your own weights, you can't fine-tune on private data, and deployment is limited to Apple platforms. If your app needs a specific domain or language not well-represented in the Foundation Model's training data, you're engineering around the model through prompting, not through retraining.

For AI coding tools built around Xcode and Apple's platform ecosystem, the Core AI developer story has direct implications. Aicoderscope.com covers that angle in depth.

Apple Silicon for open-source LLMs: an independent story

This is completely independent of Foundation Models. Ollama, llama.cpp, LM Studio, and every other open inference tool runs on Apple Silicon through the Metal and (as of Ollama 0.19 in March 2026) MLX backends. The Foundation Models 3B model and Llama 3.3 70B running in Ollama do not share inference infrastructure, don't compete for the same memory pool, and aren't connected in any way.

The performance picture for open-source inference on Apple hardware in 2026, verified across multiple benchmark sources:

Hardware	Unified Memory	Memory BW	Llama 3.3 70B Q4_K_M	Annual power cost
Mac Mini M4 16GB	16GB	120 GB/s	Won't fit	~$13/yr
Mac Mini M4 32GB	32GB	120 GB/s	Won't fit (needs ~43GB)	~$17/yr
Mac Mini M4 Pro 48GB	48GB	273 GB/s	~18 tok/s	~$37/yr
Mac Studio M4 Max 64GB	64GB	546 GB/s	~24 tok/s	~$68/yr
Mac Studio M4 Max 128GB	128GB	546 GB/s	28 tok/s	~$82/yr
Mac Studio M3 Ultra 192GB	192GB	800 GB/s	~40 tok/s	~$121/yr

The M4 Max 128GB at 28 tok/s on Llama 3.3 70B Q4_K_M is the Apple Silicon sweet spot for home lab work in 2026. The Q4_K_M quantization uses ~43GB of the

Wan 2.1, 2.2, and 2.7 for Local AI Video Generation: Which GPU Can Actually Run It (2026 Guide)

Jovan Chan — Mon, 15 Jun 2026 07:00:28 +0000

This article was originally published on runaihome.com

TL;DR: The Wan 2.2 14B is today's best open-source local video model, but at full precision it needs 54+ GB of VRAM — datacenter territory. The fix is a two-step trick (GGUF quantization + T5-XXL CPU offload) that drops GPU VRAM from 54 GB to 6–8 GB for 480p or 12–16 GB for 720p. At 16 GB VRAM, you get 720p clips in 2–4 minutes. Wan 2.7 (April 2026) raises the bar to 4K but still targets 24 GB as its practical minimum.

	RTX 4070 12GB	RTX 5060 Ti 16GB	RTX 4090 24GB
Best for	Wan 14B at 480p (GGUF)	Wan 14B at 720p (FP8)	Wan 2.7, no compromises
Street price (Jun 2026)	~$430 used	$429 new	$2,200–2,755
Peak VRAM (GGUF + offload)	~8 GB	~12–14 GB	~22 GB full FP8
480p 5-sec clip (Wan 2.2)	~18–22 min	~8–12 min	~5 min
720p 5-sec clip (Wan 2.2)	impractical (>60 min)	2–4 min	3–5 min
The catch	VRAM ceiling blocks 720p	128-bit bus limits bandwidth vs. 4070	Supply-constrained, $2,200+ entry

Honest take: The RTX 5060 Ti 16GB at $429 is the new sweet spot for Wan 2.2. At 16 GB GDDR7 and 448 GB/s bandwidth, it handles 720p clips in 2–4 minutes — the same tier as the $900+ RTX 4080 Super — for less than half the price. The RTX 4090 is worth it only if you need Wan 2.7 or are running production-scale batches.

What the Wan Series Actually Is

Wan (万象, "ten thousand forms") is Alibaba's open-source AI video generation model family, released under Apache 2.0. Unlike most commercial video generators that require cloud API access, the Wan weights are available to download and self-host. There are no per-minute charges once you have the model locally.

Four major versions have shipped since early 2025:

Wan 2.1: Dense transformer architecture, text-to-video and image-to-video. The version that put open-source video generation on the map for home lab builders.
Wan 2.2: Switched to Mixture of Experts (MoE) — 27B total parameters with 14B active per step. Better quality than 2.1 at similar compute cost, and now capable of 720p on consumer hardware.
Wan 2.5 / 2.6: Iterative improvements — camera control, better prompt adherence, consistent character generation.
Wan 2.7 (released April 22, 2026): 4K-capable, up to 20-second clips, richer instruction following. Same 14B architecture, heavier output demands.

All versions share the same inference stack. A machine you build for Wan 2.2 today will run Wan 2.7 — you swap the checkpoint, not the hardware.

Three Model Sizes, Three Use Cases

The Wan family ships in three sizes:

1.3B (text-to-video only) — the GPU-poor tier. The T2V-1.3B checkpoint needs 8.19 GB VRAM with no tricks. An RTX 4060 8GB generates a 5-second 480p clip in around 4–6 minutes. Quality is noticeably lower than the 14B model, but it's usable for rapid prompt iteration and creative experimentation on budget hardware.

5B (Wan 2.2 and later) — the mid-tier. Introduced with Wan 2.2's MoE architecture. Runs cleanly at 480p on any 12 GB card without heavy optimization, and can generate 720p @ 24 fps on a single RTX 4090. A better choice than the 14B if your card has exactly 12 GB VRAM.

14B (text-to-video + image-to-video) — the quality tier. This is where Wan competes with commercial video APIs. The 14B produces the cinematic motion, coherent character movement, and high fidelity that made the model famous. It's also where the VRAM math gets painful.

The VRAM Ceiling Problem — and the Fix

The Wan 2.2 14B pipeline has two major memory consumers:

The video diffusion transformer itself: ~14 GB in FP8, ~28 GB in FP16
The T5-XXL text encoder: ~9.4 GB at FP16

At full precision, the combined pipeline needs 54–65 GB VRAM. No consumer GPU has that. Even the RTX 5090's 32 GB falls short.

The community has converged on a two-step fix that makes Wan 14B viable on surprisingly modest hardware:

Step 1 — Quantize the transformer. GGUF Q4 or Q5 weights reduce the main Wan 14B model from ~28 GB to approximately 8–8.5 GB. Quality loss versus FP16 is minimal at 480p — most viewers can't identify the difference in blind tests. At 720p there's a subtle softening in fine detail, but the practical output remains strong.

Step 2 — Offload T5-XXL to CPU RAM. T5-XXL is only used during the conditioning pass at the start of each generation. If you have 32+ GB of system RAM, T5 can live in CPU RAM and be called when needed. This costs you 20–30 seconds of extra conditioning time per clip but saves 9+ GB of GPU VRAM. With both tricks applied:

GPU VRAM at 480p: ~6–8 GB
GPU VRAM at 720p: ~12–16 GB

This is how the RTX 4070 12GB runs the Wan 14B at all — not natively, but via GGUF + T5 offload.

One requirement that trips up first-timers: you need at least 32 GB of system RAM. With T5-XXL parked in CPU RAM and your diffusion model in VRAM, 16 GB of system RAM will hit swap during the conditioning pass and cause either errors or extremely slow generation. 32 GB is the minimum; 64 GB is comfortable.

Benchmark Data: Real Generation Times

The table below comes from SaladCloud's published Wan 2.1 T2V-14B benchmarks, testing a 5-second clip at 480p and 720p with no quantization or offloading — full precision, official inference script.

GPU	VRAM	480p (5-sec clip)	720p (5-sec clip)
H100 SXM	80 GB	85 sec	284 sec
A100 SXM	80 GB	170 sec	523 sec
A40	48 GB	501 sec	1,083 sec
RTX 4090	24 GB	281 sec	OOM
RTX 3090	24 GB	—	OOM

Two things stand out:

First, the RTX 4090 at 281 seconds beats the enterprise A40 at 501 seconds despite the A40 having twice the VRAM. GDDR6X bandwidth (1,018 GB/s on the 4090 vs. PCIe A40) matters more than raw CUDA core count for diffusion inference — the model is memory-bandwidth-bound, not compute-bound.

Second, both the RTX 4090 and RTX 3090 OOM at 720p with Wan 2.1 full precision. Running Wan 14B at 720p full-precision requires more VRAM than any consumer GPU has.

Wan 2.2 changes the 720p picture. The switch to MoE architecture (27B total, 14B active) enables efficient high-resolution generation with quantization. With FP8 + T5 offload, the RTX 4090 can now generate 720p clips. At 16 GB, the RTX 4080 Super generates 720p clips in 2–4 minutes with the same setup.

For the RTX 3090 specifically: a community benchmark running Wan 2.2-Animate on a 3090 recorded approximately 7 seconds per frame at 640×480 — meaning a 5-second, 81-frame clip takes roughly 9–10 minutes. At 720p that climbs to ~18 seconds per frame, or around 24 minutes per clip. Workable for overnight batches or one-off generates; not for rapid iteration.

GPU Tier Guide

8 GB VRAM — Wan 1.3B or 5B only

The RTX 4060 8GB, RTX 5060 8GB, and RX 7700 XT sit at the 8 GB tier. Wan 1.3B is native; Wan 2.2 5B runs with light quantization at 480p. The 14B is technically possible with aggressive GGUF + CPU offload, but generation times run 20–30 minutes per 5-second clip — barely usable for iteration.

If your GPU is 8 GB, use Wan 2.2 5B rather than fighting the 14B. The 5B at 8 GB produces output that's meaningfully better than the 1.3B, without the wait.

12 GB VRAM — Wan 14B at 480p (slow but real)

The RTX 4070 12GB and RTX 3060 12GB can run Wan 14B GGUF + T5-CPU offload at 480p. Peak GPU VRAM during generation: ~8 GB, leaving about 4 GB headroom. Generation times are 18–22 minutes per 5-second 480p clip.

The RTX 4070 has 504 GB/s bandwidth (GDDR6X, 192-bit bus). Bandwidth isn't the limiter here — VRAM is. You have enough bandwidth for Wan 14B; you don't have enough VRAM to skip the offloading tricks, which is what slows you down.

At 720p on 12 GB: possible with extreme quantization (Q3 or lower), but ge

AMD Ryzen AI Max+ 395 (Strix Halo) for Local LLMs in 2026: 128GB Unified Memory, 100 t/s on 30B Models, and Whether It Beats a Discrete GPU

Jovan Chan — Sun, 14 Jun 2026 07:03:28 +0000

This article was originally published on runaihome.com

TL;DR: The AMD Ryzen AI Max+ 395 hits 100 t/s on Qwen3-30B and runs 120B models that physically don't fit on any single consumer discrete GPU — in a $1,499–$1,999 mini PC. It's bandwidth-constrained (256 GB/s vs 1,792 GB/s on an RTX 5090), so for models under 32B a discrete GPU is faster. The machine earns its price for one audience: people who need 70B+ fully in GPU memory, without a dedicated GPU tower.

	Strix Halo Mini PC	RTX 5060 Ti 16GB Build	Mac Mini M4 Pro 48GB
Best for	70B–120B models entirely in GPU memory	≤13B at 80–130 t/s, budget build	30–70B, silent, efficient
Price	$1,499–$1,999 (complete)	~$1,400 (complete build)	$1,399 (complete)
The catch	Bandwidth bottleneck; Linux preferred	Hard ceiling at 16GB VRAM	48GB max without $4,999 Ultra

Honest take: For Llama 3.3 70B or DeepSeek R1 70B fully in GPU memory without a dedicated GPU tower, the GMKtec EVO-X2 at $1,499 is hard to beat on x86 — but if you can live with 48GB, a Mac Mini M4 Pro is simpler and draws less than half the power.

What Strix Halo actually is

Strix Halo is AMD's internal codename for the die inside the Ryzen AI Max+ 395. The unusual part is the memory architecture: 128 GB of LPDDR5X-8000 on a 256-bit bus, shared between the CPU and an integrated 40-compute-unit RDNA 3.5 GPU (the Radeon 8060S). There's no PCIe bottleneck, no VRAM ceiling separate from system RAM — the GPU sees all 128 GB at full memory bandwidth.

In practice, the chip can allocate up to 96 GB to the GPU, leaving the remaining 32 GB for the OS and CPU-side workloads. That's a larger GPU memory pool than any consumer discrete GPU, including the RTX 5090 (32 GB).

The rest of the spec sheet: 16 Zen 5 CPU cores clocked up to 5.1 GHz, a 50+ TOPS XDNA 2 NPU, and a configurable TDP range of 45W–120W with a 55W default. AMD fabbed it on TSMC 4nm. These chips ship in mini PCs — no PCIe card to install, no separate PSU math to worry about.

The actual benchmark numbers

All results below come from community testing on a Beelink GTR9 Pro running Ubuntu 24.04 with Mesa RADV (kisak PPA, version 26.0.6–26.1.1), llama.cpp builds b9049–b9467, Ollama 0.23.1, and AMD_VULKAN_ICD=RADV set for the Vulkan backend. Tested May–June 2026.

Model	Quantization	Backend	Generation (t/s)
Qwen3-30B-A3B (MoE)	IQ4_XS	RADV Vulkan	100.04
Qwen3-Coder 30B-A3B	Q4_K_S	RADV Vulkan	98.51
Qwen3-Coder 30B-A3B	UD-Q4_K_XL	RADV Vulkan	96.76
GPT-OSS 120B	MXFP4	RADV Vulkan	55.57
Qwen3.6	Q4_0 (speed-first)	RADV Vulkan	~81
Qwen3.6	balanced	RADV Vulkan	~63

100 t/s on a 30B model is comfortable real-time speed for single-user inference. The more striking number is GPT-OSS 120B at 55 t/s: a 120-billion-parameter model running entirely in unified memory at a speed that makes it useful for single-user chat.

Why MoE models run faster here: the 30B-A3B variants (Qwen3's Mixture-of-Experts architecture) activate only ~3B parameters per forward pass despite having 30B total weights. On a bandwidth-constrained system, fewer weights loaded per token means directly higher tokens/sec. If you're running Strix Halo hardware, prioritize MoE-architecture models — the performance advantage is significant.

The real-world bandwidth measurement confirms the constraint: the system delivers ~215 GB/s measured versus the theoretical 256 GB/s peak, a ~16% gap typical for LPDDR5X under mixed CPU+GPU load.

What fits in memory — and what doesn't

The GPU can access up to 96 GB of the 128 GB pool. At Q4_K_M quantization:

Model	Approx. VRAM needed	Fits on Strix Halo?	Fits on RTX 4090 (24GB)?
Llama 3.3 70B	~42–48 GB	Yes — ~48 GB headroom left	No (CPU offload needed)
Qwen3-30B (dense)	~18 GB	Yes	Yes
DeepSeek R1 70B distill	~42 GB	Yes	No (CPU offload needed)
GPT-OSS 120B	~65–70 GB	Yes (tight)	No
DeepSeek R1 671B	~380 GB	No — needs multi-node	No
Llama 4 Maverick 402B	~230+ GB	No	No

An RTX 5060 Ti 16GB hits its ceiling around 13B Q4_K_M. An RTX 4090 at 24 GB tops out near 20B before requiring CPU offloading. On Strix Halo, Llama 3.3 70B loads entirely into the GPU memory pool — no CPU offloading, no PCIe bottlenecking. The VRAM math behind these numbers is covered in detail in How Much VRAM Do You Need for Llama Models.

Strix Halo vs a discrete GPU build

This is the decision that actually matters, and the answer is unambiguous in both directions.

Where discrete GPUs win: every model under 32 GB. An RTX 5090 generates ~186 t/s on Qwen3 8B Q4_K_M. The same model on Strix Halo runs around 80–90 t/s. Memory bandwidth is the reason: 1,792 GB/s on the RTX 5090 vs ~215 GB/s real-world on Strix Halo. For a daily 7B or 14B coding assistant — see the local AI coding stack with Continue.dev + Ollama — a mid-range discrete GPU outperforms Strix Halo and often costs less.

Where Strix Halo wins: any model above 24 GB that you need running fully in GPU memory. An RTX 4090 can't load Llama 3.3 70B without splitting layers to CPU RAM, which drops generation speed to 2–5 t/s. Strix Halo loads it in ~40 seconds and generates at ~30–35 t/s. That's a 6–15× speed difference on the same model.

The cloud comparison matters here too. Running Llama 3.3 70B on RunPod costs $0.29–$0.59/hour depending on GPU availability. At $1,499 for a GMKtec EVO-X2 running 6 hours/day, you break even at roughly 700–1,400 hours of use — around 4–8 months of daily active use. After that, every inference is free. We ran this calculation in detail in the RunPod vs local GPU: when to rent vs buy article.

The Mac comparison

The closest comparison is the Mac Mini M4 Pro, which starts at $1,399 with 24 GB unified memory and maxes out at 48 GB for $1,799. Its memory bandwidth is 273 GB/s — slightly above Strix Halo's real-world 215 GB/s.

For models that fit in 48 GB, the Mac Mini M4 Pro holds three advantages: substantially better power efficiency (20–30W under LLM load vs 60–120W for Strix Halo mini PCs), meaningfully more mature software (Metal via MLX is better-tuned than AMD's Vulkan/ROCm path on Linux), and quieter operation under sustained load.

Strix Halo's advantage is the 128 GB tier. If you need the full 96 GB GPU pool for 70B+ models, the Mac route to 128 GB requires the Mac Studio M4 Ultra at $4,999. A $1,499–$1,999 Strix Halo mini PC delivers 96 GB GPU memory at roughly one-third the price — the software experience is rougher, but the hardware value is real.

What you can actually buy right now

Prices verified June 2026:

Machine	Memory	Storage	Price	Notes
GMKtec EVO-X2	128GB LPDDR5X	2TB NVMe	~$1,499	Best value, 2.5GbE
Beelink GTR9 Pro	128GB LPDDR5X	2TB NVMe	$1,899–$1,999	Dual 10GbE, better cooling
MINISFORUM MS-S1 Max	128GB LPDDR5X	2TB NVMe	~$2,299	Available on Newegg
GMKtec EVO-X2 (64GB)	64GB LPDDR5X	1TB NVMe	~$1,099	GPU pool ~48GB, still runs 70B

The GMKtec EVO-X2 at $1,499 is the price-performance sweet spot. It has the same CPU and GPU as the Beelink GTR9 Pro and omits the dual 10GbE NICs — which you don't need for single-user home inference. The Beelink's dual 10GbE matters if you're running a shared home AI server. For that use case, the Open WebUI multi-user setup guide covers the server con

ROCm 7.2 on Ubuntu 24.04 for Local LLMs in 2026: Full Setup Guide for AMD GPUs

Jovan Chan — Sun, 14 Jun 2026 07:02:42 +0000

This article was originally published on runaihome.com

TL;DR: ROCm 7.2.3 (released May 4, 2026) is the stable Ubuntu path for AMD GPU inference — RDNA 3 setup is rock-solid in under 20 minutes, RDNA 4 works with one Docker workaround for a known gfx1201 bug. AMD delivers 85–92% of equivalent NVIDIA throughput at a lower price point on Ubuntu.

What you'll be able to do after this guide:

Install ROCm 7.2.3 on Ubuntu 24.04 LTS and run ollama serve with full GPU acceleration in under 20 minutes
Build llama.cpp with the HIP backend for maximum throughput on RDNA 3 and RDNA 4
Identify and work around the gfx1201 rocBLASLt crash that kills model loads on the RX 9070 XT

Honest take: On Ubuntu, a used AMD Radeon RX 7900 XTX at ~$800 is the best AMD card for local LLMs in 2026 — 24 GB VRAM, ~96 tok/s on Llama 3.1 8B Q4_K_M, and ROCm 7.2.3 installs without a single environment variable hack.

AMD's local AI story on Linux has changed substantially. A year ago you'd fight missing kernel modules and half-broken pip wheels. Today, if you're on a supported card and Ubuntu 24.04, setup is close to the CUDA experience: download a .deb, add yourself to two groups, reboot once, and ollama pull llama3.1:8b works.

The catches are smaller than they used to be, but they exist. RDNA 4 support (RX 9000 series) is still maturing in a specific way — one rocBLASLt lookup bug can SIGKILL your model load at the 2-minute mark every single time. Knowing where the landmine is before you start saves 90 minutes of frustrating debugging.

This guide covers the native Ubuntu install path. If you need AMD on Windows, see the AMD ROCm 7.2 on Windows guide — RDNA 3 is Linux-only for ROCm and the Windows path is a different story entirely.

Which cards are actually supported on Ubuntu 24.04

ROCm 7.2.3 divides AMD's consumer lineup into three buckets:

Fully supported on Linux:

RDNA 4: RX 9070 XT, RX 9070, RX 9060 XT LP, Radeon AI PRO R9600D (gfx1201 / gfx1200)
RDNA 3: RX 7900 XTX, RX 7900 XT, RX 7900 GRE, RX 7800 XT, RX 7700 XT (gfx1100 / gfx1101 / gfx1102)

RDNA 3 consumer cards are Linux-only for ROCm. On Windows, the ROCm stack officially supports only RDNA 4 chips — see above.

Supported via Vulkan only (no ROCm HIP):

RX 7600, RX 6000 series, anything older. These cards can run inference through llama.cpp's Vulkan backend, but won't get vLLM, PyTorch ROCm, or HIP acceleration.

Not supported at all:

RDNA 1 (RX 5000 series) and older. Vulkan may work, but inference speed makes these impractical for anything beyond tiny models.

VRAM is still the ceiling

For context on what each card can run:

RX 7900 XTX (24 GB): Qwen3-30B-A3B at Q4_K_M fits cleanly. Llama 3.3 70B Q4 in CPU-offload mode. Anything under 20B at Q4 is comfortable.
RX 9070 XT (16 GB): Llama 3.1 8B at full speed, Qwen3-14B at Q4, 27B MoE models technically fit but saturate the memory bus (6.3 tok/s on Qwen3.5-27B-A3B at Q4).
RX 9070 GRE (16 GB): Launched globally at $549 on June 2, 2026 — same VRAM and gfx1201 architecture as the 9070 XT, slightly less shader compute.

For a broader AMD vs NVIDIA VRAM comparison at the 16 GB tier, see AMD RX 9070 XT vs RTX 5060 Ti 16GB.

Step 1: Install ROCm 7.2.3 on Ubuntu 24.04

Start from Ubuntu 24.04.3 LTS. The amdgpu-install tool handles both the kernel driver and the ROCm userspace stack in a single package.

# Download the installer for Ubuntu 24.04 (noble)
wget https://repo.radeon.com/amdgpu-install/7.2.3/ubuntu/noble/amdgpu-install_7.2.3.70203-1_all.deb

# Install it
sudo apt install ./amdgpu-install_7.2.3.70203-1_all.deb
sudo apt update

# Install ROCm with the rocm usecase
sudo amdgpu-install --usecase=rocm --no-dkms

The --no-dkms flag skips DKMS kernel module compilation. On Ubuntu 24.04.3 with a 6.8.x kernel, RDNA 3 and RDNA 4 are already supported by the packaged kernel — invoking DKMS wastes 10 minutes and sometimes fails on systems with secure boot or custom kernels.

Step 2: Add user groups and reboot

This step trips up almost every first-time installer and the error messages when you skip it are not helpful. The ROCm compute stack requires your user to be in the render and video groups to access /dev/kfd (the GPU compute device node) without root.

sudo usermod -a -G render,video $USER

Reboot now. A newgrp session is not sufficient — the group membership must be part of your login session from the start. After reboot, verify:

groups
# Expected: ... render video ...

rocminfo | head -25

Expected output from rocminfo on an RX 9070 XT:

ROCk module is loaded
...
Agent 2
  Name:                    gfx1201
  Uuid:                    GPU-XXXXXXXXXXXXXX
  Marketing Name:          Radeon RX 9070 XT
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE

If rocminfo hangs for more than 30 seconds or shows only Agent 1 (the CPU), you have a group issue or driver conflict. Check dmesg | grep amdgpu first — firmware errors here usually mean you need a linux-firmware update.

Step 3: Verify with rocm-smi

rocm-smi

This displays real-time GPU stats including temperature, power draw, and memory usage. At idle you'll see 0% utilization — that's normal. Run a model in the next step and check again to confirm the GPU is actually being used.

Step 4: Install Ollama with ROCm support

Ollama ships its own bundled ROCm libraries and auto-detects AMD GPUs on Linux:

curl -fsSL https://ollama.com/install.sh | sh
systemctl start ollama
ollama pull llama3.1:8b-instruct-q4_K_M
ollama run llama3.1:8b-instruct-q4_K_M

While inference is running, open a second terminal and run watch -n 1 rocm-smi. You should see GPU memory jump to ~5.5 GB and compute utilization hit 90–95%.

If memory shows 0 MB allocated despite the model loading, Ollama may be using CPU. Run OLLAMA_DEBUG=1 ollama serve and check the startup logs — it will report which ROCm libraries it found and whether the GPU was initialized.

Step 5 (optional): Build llama.cpp with HIP

For direct llama.cpp inference — more control over layer offloading and context window than Ollama provides — the HIP backend delivers the best AMD throughput:

sudo apt install cmake git build-essential

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1201 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

Replace gfx1201 with your card's architecture:

Card	Target arch
RX 9070 XT / 9070 / 9060 XT	`gfx1201`
RX 7900 XTX / 7900 XT / 7900 GRE	`gfx1100`
RX 7800 XT	`gfx1101`
RX 7700 XT	`gfx1101`

Run the server:

./build/bin/llama-server \
  -m /path/to/model.gguf \
  --n-gpu-layers 99 \
  --host 0.0.0.0 --port 8080

--n-gpu-layers 99 offloads all layers to GPU. If you hit VRAM limits, reduce this number to shift layers to CPU — useful when running 27B+ models on a 16 GB card.

Real benchmarks

Results from community benchmarks on ROCm 7.x, Ubuntu 24.04, Ollama and llama.cpp HIP:

Card	VRAM	Model	Quant	tok/s
RX 7900 XTX	24 GB	Llama 3.1 8B	Q4_K_M	66–96
RX 9070 XT	16 GB	Llama 3.1 8B	Q4_K_M	~56
RX 9070 XT	16 GB	Qwen3:14B	Q4	52.2
RX 9070 XT	16 GB	GPT-OSS:20B	Q4	91.9
RX 9070 XT	16 GB	Qwen3.5:27B-A3B	Q4 (MoE)	6.3

The 66–96 tok/s variance on the RX 7900 XTX reflects different llama.cpp versions and batch size settings across community tests. Mid-range is roughly 80 tok/s for a clean Q4_K_M Llama 8B run.

For comparison: an RTX 4070 Super (12 GB, 504 GB/s) delivers roughly 62–70 tok/s on the same model. The RX 9070 XT at 640 GB/s memory bandwidth edges it out

Intel Arc B770 vs RTX 5060 for Local AI in 2026: The 16GB Budget War That Never Happened

Jovan Chan — Sun, 14 Jun 2026 07:01:58 +0000

This article was originally published on runaihome.com

TL;DR: Intel canceled the Arc B770 "Big Battlemage" — the 16GB budget GPU that was supposed to challenge the RTX 5060 Ti market — citing GDDR memory costs and lack of financial viability. NVIDIA filled the slot with the RTX 5060, but shipped it with only 8GB of VRAM. The result: a $200 gap between 8GB and 16GB consumer cards, no affordable Intel challenger anywhere in the picture, and the B770 silicon surviving only as the $949 Arc Pro B70 workstation card.

	RTX 5060	RTX 5060 Ti	Arc Pro B70
VRAM	8GB GDDR7	16GB GDDR7	32GB GDDR6
Bandwidth	448 GB/s	448 GB/s	608 GB/s
Price (Jun 2026)	$299–$339	$429–$479	$949
Best for	7B models only	Up to 20B models	30B+ models, pro workflows
The catch	Hard wall at 8GB	$200 more than 5060	$500 more than 5060 Ti; no CUDA

Honest take: If your budget tops out at $350, the RTX 5060 is fast and frictionless at 30 tok/s on 7B models. If you ever want to run a 13B or 30B model, stretch to the RTX 5060 Ti. Intel is not your friend at this price point in 2026.

What Intel promised

For most of 2025, Intel's roadmap included a second Battlemage GPU — the Arc B770, internally designated BMG-G31. Where the Arc B580 uses the smaller BMG-G21 die with 20 Xe2 cores, the B770 was designed around the full 32-core die with these specs (per leaked hardware repository entries and partner briefings):

16GB GDDR6 on a 256-bit bus
608 GB/s memory bandwidth
32 Xe2 cores (vs. 20 on the B580)
~300W TDP
PCIe Gen5 x16

Those numbers were actually compelling for local AI. 608 GB/s beats everything in NVIDIA's current consumer lineup including the RTX 5060 Ti's 448 GB/s. 16GB of VRAM at a rumored $350–$400 would have undercut the RTX 5060 Ti on price while matching it on memory capacity. A 13B model at Q4_K_M fits in 16GB with room to spare for context. A 27B model at Q4 would have been reachable.

That card doesn't exist. Here's why.

Why Intel canceled it

According to reports from multiple sources including Tom's Hardware and PC Gamer, the B770 was deemed "not financially viable." The proximate cause was the GDDR6 memory shortage of 2025–2026 — the same AI buildout driving data-center VRAM demand made consumer DRAM expensive enough to erode whatever margin Intel had modeled.

The structural problem runs deeper. NVIDIA has CUDA. AMD has a maturing ROCm stack. Intel's Arc ecosystem requires users to install Intel's IPEX-LLM fork, use llama.cpp's Vulkan backend, or accept reduced compatibility with tools that assume CUDA. Asking those users to pay $350–$400 for a card that adds 30–60 minutes of setup friction — and still breaks with some AI tools — is a hard sell against a $300 RTX 5060 that just works.

Intel concluded that marketing costs, driver maintenance, and validation overhead would not produce a return. The B770 was shelved. Intel's next discrete GPU launch was the workstation-focused Arc Pro B70 — same silicon, different market, much higher price.

What NVIDIA delivered instead

The RTX 5060 launched in spring 2026. Specs:

3,840 CUDA cores (Blackwell GB206 die)
8GB GDDR7 memory, 128-bit bus
448 GB/s memory bandwidth
Boost clock: 2,625 MHz
Launch MSRP: $299; street price June 2026: $299–$339 new, ~$285 used on eBay

The local AI performance story is straightforward. The RTX 5060 posts around 30 tokens/sec on Llama 3.1 8B Q4_K_M via Ollama — fast enough for real-time chat, comfortable coding assistant use, and single-user inference. CUDA means zero-friction setup: Ollama, vLLM, ExLlamaV2, AutoGPTQ all work without extra configuration. Install Ollama, pull a model, run it.

The problem is the 8GB ceiling. Here's what actually fits:

Model	Quantization	VRAM needed	Runs on RTX 5060?
Llama 3.1 8B	Q4_K_M	~5.5 GB	✅ Yes, ~30 tok/s
Qwen2.5 7B	Q4_K_M	~5.0 GB	✅ Yes
Mistral 7B	Q8_0	~8.5 GB	❌ Fails to load
Llama 3.1 13B	Q4_K_M	~8.5 GB	❌ No
Qwen2.5 14B	Q4_K_M	~9.5 GB	❌ No
Qwen2.5 32B	Q4_K_M	~19 GB	❌ CPU offload only

The failure mode for 13B and above is a hard one:

$ ollama run qwen2.5:14b
Error: model requires 9.5 GB VRAM, only 8.0 GB available
      Try reducing context size (--ctx-size) or switching to a smaller model

CPU offloading kicks in and drops you from 30 tok/s to roughly 3–5 tok/s — unusable for interactive use. The 8GB wall is real and not negotiable without changing cards.

This is precisely where the B770 would have mattered. 16GB at 608 GB/s for $350 would have introduced real competitive pressure on the RTX 5060 Ti. NVIDIA doesn't have that pressure right now, and the pricing reflects it.

The 8GB-to-16GB gap, and who fills it

If 8GB isn't enough, your options for a new card are limited:

RTX 5060 Ti 16GB — $429–$479

Same 448 GB/s bandwidth as the RTX 5060. Double the VRAM. That extra 8GB changes what's possible: Qwen2.5 14B at Q4_K_M fits with room, Llama 3.3 70B runs at reduced quantization with some CPU offload, and 30B models become viable. Benchmarks from Hardware-Corner show 32.9 tok/s on 14B models at 16k context via Ollama. For most home AI users, this is the right call if the budget allows it.

Used RTX 3090 24GB — $480–$550 (eBay, June 2026)

24GB GDDR6 at 936 GB/s bandwidth. For sheer throughput on large models, nothing in the sub-$600 consumer market touches the RTX 3090. Trade-offs: ~350W power draw, no warranty, age. We covered the value calculus in depth in the RTX 3090 analysis.

AMD RX 9070 XT 16GB — ~$499

640 GB/s bandwidth, 16GB GDDR6. ROCm has improved substantially in 2026 and the Vulkan/ROCM llama.cpp path is now reasonably stable. Covered in the RX 9070 XT vs RTX 5060 Ti comparison.

Intel contributes nothing to this list with a consumer card.

Arc Pro B70: the B770 silicon at a different price

Intel didn't scrap the BMG-G31 die. The Arc Pro B70 launched in March 2026 at $949, using the full 32 Xe2-core configuration with workstation-class features:

32GB GDDR6 on a 256-bit bus (608 GB/s bandwidth)
367 TOPS INT8 AI inference performance
22.94 TFLOPS FP32 compute
PCIe 5.0 x16
ISV-certified professional drivers
Multi-GPU support on Linux via oneAPI

The 32GB is the pitch for local AI. At 32GB you can load Qwen2.5 32B at Q4_K_M (~19GB) comfortably, run Llama 3.3 70B at Q4_K_M (~42GB) with partial CPU offloading, and fit every 13B or 27B model at full Q8 quality. The 608 GB/s bandwidth also means larger models run faster per-token than they would on the RTX 5060 Ti's 448 GB/s.

Available at Newegg and Micro Center for $949.

The problem: $949 is not a budget play. At that price, you're competing with used RTX A5000 24GB cards with mature CUDA driver support, and you're sitting $470 above an RTX 5060 Ti. The software tax hasn't disappeared — the B70 runs local AI via IPEX-LLM and OpenVINO on Linux, not via Ollama's default CUDA path. Windows support exists but is rougher.

The B70 makes sense in a professional Linux workstation with an AI workflow already built on Intel's oneAPI toolchain. It does not make sense as an Ollama drop-in for a Windows home-lab machine where the RTX 5060 Ti does 90% of the same job with zero friction for half the price.

If you're on the fence between renting and buying during the current GPU market confusion, RunPod has A100 80GB instances at $1.89/hr — useful for large model testing before committing to hardware.

Arc B580: the real Intel option right now

The gaming B770 was supposed to land above the Intel Arc B580. S

ComfyUI API Tutorial 2026: Automate Image Generation

Jovan Chan — Sun, 14 Jun 2026 07:01:04 +0000

This article was originally published on aifoss.dev

TL;DR: ComfyUI's built-in HTTP server accepts workflow JSON on port 8188 — the same JSON the GUI exports when you click "Save (API Format)". You can queue a prompt, poll for completion, and download the result in under 50 lines of Python. No GUI, no browser tab, no manual clicking.

What you'll have running after this guide:

A Python function that submits any ComfyUI workflow and blocks until images are downloaded locally
A batch script that generates 100 images with automated seed variation and organized output folders
Working API-format JSON for an SDXL baseline workflow and a Flux.1 Schnell workflow you can adapt immediately

Why drive ComfyUI from the API

The GUI is fine for building and testing workflows. It's a problem once you need more than a few images, or need to integrate generation into a pipeline.

Common use cases where the API makes sense:

Batch generation — 50–1000 images with systematic prompt or seed variation
CI pipelines — trigger image generation on a new product SKU, game asset, or dataset update
Headless servers — GPU box in a closet or a RunPod cloud instance where running a browser makes no sense
App backends — a FastAPI endpoint that accepts user prompts and returns generated images

If you're generating one image at a time and want to tweak nodes visually, stay in the GUI. The API is for automation.

Prerequisites

ComfyUI v0.23.0 (latest as of June 2026) installed and working in the GUI
Python 3.10+
pip install requests websocket-client
At least one working workflow in your ComfyUI GUI

If you haven't installed ComfyUI yet, the ComfyUI review covers installation from scratch. For GPU hardware requirements, the Stable Diffusion 8GB VRAM guide has a good breakdown of what each model tier needs.

Step 1: Start ComfyUI with API access enabled

By default, ComfyUI only listens on 127.0.0.1 — accessible only from the same machine. That's fine if you're scripting locally. For a remote GPU box or Docker container, add --listen:

# Local access only (same machine)
python main.py --port 8188

# All interfaces (for remote scripting or Docker)
python main.py --listen 0.0.0.0 --port 8188

# With VRAM optimization for 8–12 GB cards
python main.py --listen 0.0.0.0 --port 8188 --lowvram

Once running, verify it's up:

curl http://127.0.0.1:8188/system_stats
# {"system": {"os": "posix", "python_version": "3.10.x", ...}}

The server starts a REST API and a WebSocket server on the same port. There is no authentication by default — if you expose this to a network, use a firewall rule or reverse proxy with auth.

Step 2: Export your workflow in API format

The GUI workflow JSON and the API workflow JSON are different formats. The GUI format includes node positions, colors, and UI state. The API format is stripped down to just inputs and connections — which is all the server needs.

To export:

Open ComfyUI in your browser
Go to Settings (gear icon) → enable "Dev Mode Options"
A new button appears in the menu: "Save (API Format)"
Click it — this downloads workflow_api.json

The API format uses numeric string keys ("1", "2", "3") as node IDs. Each node has a class_type and an inputs object. Connections between nodes are expressed as ["source_node_id", output_index] arrays rather than named references.

Here is a minimal SDXL baseline workflow in API format:

{
  "4": {
    "class_type": "CheckpointLoaderSimple",
    "inputs": {
      "ckpt_name": "sd_xl_base_1.0.safetensors"
    }
  },
  "5": {
    "class_type": "EmptyLatentImage",
    "inputs": {
      "batch_size": 1,
      "height": 1024,
      "width": 1024
    }
  },
  "6": {
    "class_type": "CLIPTextEncode",
    "inputs": {
      "clip": ["4", 1],
      "text": "a photorealistic red fox in snow, golden hour lighting"
    }
  },
  "7": {
    "class_type": "CLIPTextEncode",
    "inputs": {
      "clip": ["4", 1],
      "text": "blurry, low quality, watermark, text"
    }
  },
  "3": {
    "class_type": "KSampler",
    "inputs": {
      "cfg": 7.0,
      "denoise": 1.0,
      "latent_image": ["5", 0],
      "model": ["4", 0],
      "negative": ["7", 0],
      "positive": ["6", 0],
      "sampler_name": "euler",
      "scheduler": "normal",
      "seed": 42,
      "steps": 20
    }
  },
  "8": {
    "class_type": "VAEDecode",
    "inputs": {
      "samples": ["3", 0],
      "vae": ["4", 2]
    }
  },
  "9": {
    "class_type": "SaveImage",
    "inputs": {
      "filename_prefix": "api_output",
      "images": ["8", 0]
    }
  }
}

For SDXL specifically, node "4" connects to both CLIP encoders and the KSampler via its three outputs: [0] = model, [1] = CLIP, [2] = VAE. The connection syntax ["4", 1] means "output index 1 of node 4."

Step 3: Queue a prompt from Python

import uuid
import json
import requests

SERVER = "127.0.0.1:8188"
CLIENT_ID = str(uuid.uuid4())

def queue_prompt(workflow: dict) -> str:
    """Submit a workflow. Returns the prompt_id for tracking."""
    payload = {"prompt": workflow, "client_id": CLIENT_ID}
    r = requests.post(f"http://{SERVER}/prompt", json=payload)
    r.raise_for_status()
    return r.json()["prompt_id"]

The client_id is a UUID you generate once per session. It ties your WebSocket connection to your HTTP requests so the server routes status messages back to you. You can skip it for pure HTTP polling, but you need it for WebSocket tracking.

The server responds immediately with {"prompt_id": "<uuid>", "number": <queue_position>}. Generation hasn't started yet — the prompt is in queue.

Step 4: Poll for completion

Two approaches — choose based on your use case:

Approach	Latency	Complexity	Best for
HTTP polling `/history/{id}`	~1s overhead	Low — no extra library	Scripts, batch jobs
WebSocket `/ws?clientId=...`	Near-real-time	Medium — event loop	Apps that show progress

HTTP polling (simpler)

import time

def wait_for_completion(prompt_id: str, poll_secs: float = 1.0) -> dict:
    """Block until the prompt finishes. Returns the history entry."""
    url = f"http://{SERVER}/history/{prompt_id}"
    while True:
        r = requests.get(url)
        r.raise_for_status()
        history = r.json()
        if prompt_id in history:
            return history[prompt_id]
        time.sleep(poll_secs)

The /history/{prompt_id} endpoint returns an empty dict {} while the prompt is queued or running, and the full result object once done. Polling every second adds at most 1 second of latency to your total generation time — acceptable for batch scripts.

WebSocket approach (for real-time apps)

import websocket

def generate_with_ws(workflow: dict) -> str:
    """Queue and wait for completion via WebSocket. Returns prompt_id."""
    ws = websocket.WebSocket()
    ws.connect(f"ws://{SERVER}/ws?clientId={CLIENT_ID}")
    prompt_id = queue_prompt(workflow)

    while True:
        raw = ws.recv()
        if not isinstance(raw, str):
            continue  # binary preview frames — skip
        msg = json.loads(raw)
        if msg["type"] == "executing":
            data = msg["data"]
            if data["node"] is None and data["prompt_id"] == prompt_id:
                break  # null node = generation finished

    ws.close()
    return prompt_id

The executing message fires for each node as it runs. When node is None and the prompt_id matches yours, the entire graph has finished executing.

Step 5: Download the generated images


python
import os

def download_images(history_entry: dict, output_dir: str = "output") -> list[str]:
    """Download all output images from a completed prompt."""
    os.makedirs(output_dir, exist_ok=True)
    saved = []

AMD Lemonade Review 2026: GPU, NPU, and Multi-Modal

Jovan Chan — Sun, 14 Jun 2026 07:00:20 +0000

This article was originally published on aifoss.dev

TL;DR: Lemonade v10.6 is AMD's open-source LLM server that adds NPU prefill acceleration, image gen, and speech to one OpenAI-compatible endpoint. NPU acceleration works only on Ryzen AI 300/400 chips — on other hardware, Ollama's ecosystem is wider. AMD Ryzen AI users should pick Lemonade; everyone else should consider Ollama first.

	Lemonade v10.6	Ollama v0.6	LocalAI
Best for	AMD GPU + NPU hybrid, multi-modal	Cross-platform, broadest ecosystem	OpenAI API proxy, any hardware
Install	`winget` or Snap	`curl` one-liner	Docker Compose
Hardware	AMD RDNA3+, NVIDIA, Apple M, CPU	Any GPU	Any hardware
Model formats	GGUF, ONNX, FLM, SafeTensors	GGUF (Ollama manifest)	GGUF, OpenVINO, more
Multi-modal	LLM + image gen + Whisper + TTS	LLM + vision models	LLM + Whisper + SD
The catch	NPU only on Ryzen AI 300/400	No NPU acceleration	High setup complexity

Honest take: On a Ryzen AI 300-series machine, Lemonade is the better daily driver — it uses hardware that Ollama leaves idle and bundles image gen plus speech in one package. On Nvidia hardware or wherever you need maximum integration coverage, stick with Ollama.

What Lemonade Is and Why AMD Built It

Ollama solved cross-platform local LLM deployment cleanly. But it left AMD NPU owners with idle hardware — the dedicated AI accelerators in Ryzen AI chips sat unused because Ollama has no FastFlowLM backend.

Lemonade is AMD's answer. Released under Apache 2.0 and available at github.com/lemonade-sdk/lemonade, it bundles:

An OpenAI-compatible HTTP API at http://localhost:13305/v1
llama.cpp with Vulkan backend for AMD and NVIDIA GPUs
FastFlowLM for XDNA2 NPU acceleration on Ryzen AI chips
Stable Diffusion image generation
Whisper speech-to-text
Kokoro text-to-speech
A model manager with one-command downloads from Hugging Face

The core design difference from Ollama is hardware-tier splitting. On a Ryzen AI 300-series chip, prompt processing (prefill) goes to the NPU while token generation (decode) goes to the iGPU. This is not marketing — the NPU has better compute throughput for dense matrix math during prefill, and the iGPU has better memory bandwidth for sequential token generation. The result is lower Time to First Token on long system prompts and agentic chains.

Current version: v10.6.0 (released May 21, 2026). Linux NPU support shipped with Lemonade 10.0 in March 2026 via the FastFlowLM runtime.

Hardware Compatibility

Platform	Backend	Notes
AMD Ryzen AI 300/400 (XDNA2)	FastFlowLM NPU + Vulkan iGPU	Strix Halo supports up to 128 GB unified memory
AMD Radeon discrete (RDNA2/3/4)	llama.cpp + Vulkan	Standard VRAM limits; add 2–4 GB overhead
NVIDIA (Turing–Blackwell)	llama.cpp + Vulkan or CUDA	CUDA backend available since v10+
Apple Silicon (M1–M4)	Metal via llama.cpp	Unified memory; M4 Max competitive at large models
x86_64 CPU	llama.cpp CPU	Small models only; no hardware acceleration

NPU acceleration requires Ryzen AI 300-series or 400-series specifically — the XDNA2 architecture. Earlier Ryzen AI chips (7000, 8000, 200-series) have NPUs that no current runtime supports for LLM inference. On those systems, Lemonade falls back to Vulkan on the GPU, which is functionally the same as running Ollama.

Supported Linux distros: Ubuntu 24.04+, Fedora 43+, Debian Trixie+, Arch. Docker and Snap packages are available. For hardware context on AMD GPU builds, see runaihome.com for current RDNA4 GPU benchmarks and build guides.

Installation

Windows

winget install AMD.LemonadeServer

This installs the server and a Tauri desktop app (system-tray GUI for model downloads and server management). Alternatively, grab the .msi from the GitHub releases page. After install, the server starts automatically on port 13305.

Linux (Ubuntu 24.04+)

# Snap — works across Ubuntu 24.04+, Fedora 43+, Arch
sudo snap install lemonade

# Docker
docker run -d --gpus all -p 13305:13305 lemonadesdk/lemonade:latest

For NPU support on Linux, you need the XDNA driver and FastFlowLM runtime installed separately — the Lemonade docs cover the dependency chain. It is more involved than the Windows path. For most Linux users without a Ryzen AI 300/400 chip, the Snap install with Vulkan fallback is the practical path.

Verify the server is running

curl http://localhost:13305/v1/models

Expected output on a fresh install with no models downloaded:

{"object":"list","data":[]}

Running Your First Model

lemonade run Gemma-4-E2B-it-GGUF

This pulls the model from Hugging Face (if not cached) and starts a chat session in your terminal. The model manager uses Hugging Face slug format — you can also import any custom GGUF or ONNX model from Hugging Face directly.

Check which backend Lemonade selected for your hardware:

curl http://localhost:13305/stats

The response includes the active inference engine: vulkan, fastflowlm, rocm, or cpu. If you expected fastflowlm and got vulkan, check that your XDNA driver is installed and you're on a Ryzen AI 300/400 chip.

To test image generation:

curl http://localhost:13305/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{"prompt": "a terminal screen in a dark room", "n": 1}'

NPU + GPU Hybrid: Numbers From Real Hardware

On a Ryzen AI Max+ 395 (Strix Halo, 128 GB unified memory), the NPU handles prompt processing and the iGPU handles decode. Community benchmarks from May–June 2026 on this configuration:

Model	Quantization	Tokens/sec
GPT-OSS 120B	Q4_K_M	~50 tok/s
Qwen3.5-122B	Q4	~35 tok/s
Qwen3-Coder-Next	Q4	~43 tok/s

For comparison: an RTX 4090 running llama.cpp hits 50–80 tok/s on 7B models at Q4 but stalls on 70B+ without aggressive quantization (limited by 24 GB VRAM). The Strix Halo runs 120B at full Q4_K_M in 128 GB of unified memory — a different tier of capability.

On smaller Ryzen AI 300 systems (Strix Point, 32–64 GB), expect:

Llama 3.2-3B on NPU: ~28 tok/s at under 2 W
Models above 8B: fall back to iGPU via Vulkan

FastFlowLM 0.9.35, the current NPU runtime bundled in Lemonade 10.6, supports context windows up to 256k tokens on XDNA2 NPUs.

Multi-Modal in One Server

Lemonade bundles three additional inference backends behind the same API port:

Image generation: SDXL-Turbo via /v1/images/generations. Any client that supports the OpenAI image endpoint works — including the ComfyUI API adapter. See our ComfyUI API tutorial for chaining this into automated pipelines.

Speech-to-text: Whisper backend via /v1/audio/transcriptions. Uses the same model weights as whisper.cpp.

Text-to-speech: Kokoro TTS via /v1/audio/speech. Known limitation as of v10.6: voices not in the pre-configured list produce muted audio. Custom voice loading is not yet supported.

Running these three modalities as separate services (Ollama + ComfyUI + a Whisper server) adds coordination overhead — three processes, three ports, three model caches. Lemonade consolidates them into one service with one model manager. For a home server running all three, that's meaningful.

Connecting to Open WebUI

Open WebUI supports custom OpenAI-compatible endpoints. To add Lemonade:

Open WebUI settings → Connections → Add Connection
API URL: http://localhost:13305/v1
API key: leave blank (Lemonade does not validate keys)
Save and confirm models appear in the model list

If you're running Open WebUI in Docker and Lemonade natively on the host:



http://host.docker.

DeepSeek V4 vs Qwen3 for Local AI in 2026: Which Model Family Fits Your GPU?

Jovan Chan — Sat, 13 Jun 2026 07:06:04 +0000

This article was originally published on runaihome.com

TL;DR: DeepSeek V4 Flash and Qwen3 both landed in late April 2026 and rewrote the open-weights leaderboards. But for home-lab inference, they serve completely different audiences: Qwen3's MoE variants run on a single consumer GPU and hit 120 tok/s on an RTX 3090, while V4 Flash's lightest usable quantization requires 103 GB of VRAM — more than four RTX 4090s combined. The model family that "fits your GPU" is almost certainly Qwen3.

	Qwen3 small/mid (8B–32B)	Qwen3 MoE (30B-A3B / 35B-A3B)	DeepSeek V4 Flash (284B)
Best for	Budget single-GPU builds	Consumer GPU sweet spot	Multi-GPU server or API
Min VRAM	5 GB (8B Q4)	~17 GB (Q4)	~103 GB (Q2_K)
Single RTX 3090 speed	23–60 tok/s (14B–32B)	~120 tok/s (Q3, Unsloth)	Not viable
The catch	Smaller models, less reasoning depth	Needs 24 GB for Q4 headroom	Needs $10k+ hardware to run well

Honest take: For 99% of home-lab builders, Qwen3 30B-A3B or 35B-A3B is the answer regardless of budget. DeepSeek V4 Flash is an excellent API model — use it that way, at $0.10/M input tokens, rather than trying to run it locally unless you have a purpose-built multi-GPU server.

What shipped and when

DeepSeek-V4 launched April 24, 2026, in two variants:

V4 Flash: 284B total parameters, 13B activated per token (MoE)
V4 Pro: 1.6T total parameters, 49B activated per token (MoE)

Alibaba's Qwen team shipped the Qwen3 family days later, spanning 0.6B through 235B, including two MoE variants designed explicitly for consumer hardware: the 30B-A3B (3B active/token) and the newer 35B-A3B from the Qwen3.6 update.

Both families use Mixture-of-Experts architectures that activate only a fraction of parameters per token, which makes them faster to serve than dense models of equivalent total size. The key difference is how aggressively each family scaled the MoE — and who they optimized for.

The VRAM numbers, tier by tier

Consumer single-GPU: 8–24 GB VRAM

Qwen3 owns this tier entirely.

Model	Quantization	VRAM required	Minimum GPU
Qwen3 8B	Q4_K_M	~5 GB	RTX 4060 8GB
Qwen3 14B	Q4_K_M	~8 GB	RTX 3060 12GB
Qwen3 30B-A3B	Q4_K_M	~16.8 GB	RTX 3090 24GB
Qwen3.6-35B-A3B	Q3 (Unsloth)	~23 GB	RTX 3090 24GB
Qwen3 32B	Q4_K_M	~20 GB	RTX 3090 (tight)

DeepSeek V4 Flash doesn't appear in this table because it cannot run in this tier. Its IQ1_S-XL quantization — the most aggressive lossy compression available — compresses the 284B model from its FP8 footprint down to 57.3 GB. That's still more than two RTX 4090 GPUs combined. The first viable quantization for practical use is Q2_K at 103 GB.

If you have a single GPU with 24 GB or less, V4 Flash is a non-starter. Any attempt at local inference involves offloading almost everything to system RAM, with generation speeds under 2 tok/s — impractical for interactive use.

Prosumer multi-GPU: 2–4× consumer cards

Two RTX 4090s in PCIe tensor-parallel gives 48 GB of pooled VRAM. That's still 55 GB short of V4 Flash's Q2_K threshold.

The workaround is mixed VRAM/CPU offloading: load the attention layers in VRAM, offload the MoE expert weights to system RAM using llama.cpp's -cmoe -ub 128 flags. On a 4× RTX 4090 rig (96 GB total VRAM, plus 128 GB system RAM), community tests put V4 Flash throughput at 8–12 tok/s. Usable for batch processing; painful for interactive chat.

Qwen3 235B-A22B at Q4 needs approximately 132 GB, putting it into similar territory — viable with partial offloading but not fast.

At the 2–4× consumer GPU tier, the better answer remains Qwen3's MoE variants running on a single card, with spare GPUs handling separate workloads or parallel inference instances.

Professional multi-GPU: dedicated server hardware

This is where V4 Flash becomes genuinely useful. Running on dual NVIDIA RTX PRO 6000 Max-Q GPUs with W4A16+FP8 quantization and MTP self-speculation, V4 Flash hits approximately 111 tok/s for single-stream requests at 128K context. That figure comes from NVIDIA's own testing and represents the current high-water mark for affordable (relative to H100 clusters) V4 Flash inference.

On a single H100 80GB with GPU offloading enabled, throughput is around 20 tok/s — better, but H100s cost $25,000+ used. The official V4 Flash API, measured by Artificial Analysis, runs at approximately 84 tok/s.

Qwen3 235B-A22B needs a minimum of 4× H100 80GB GPUs (320 GB total) to run Q4 with practical context and KV cache headroom — an eight-figure hardware investment for most people. The 30B-A3B and 35B-A3B MoE variants, by contrast, run on hardware anyone reading this article can actually buy.

Speed on consumer hardware: what you'll actually see

These figures are from community benchmarks tested in April–May 2026 using llama.cpp and Ollama.

GPU	Model	Quant	Speed
RTX 3060 12GB	Qwen3 8B	Q4_K_M	~42 tok/s
RTX 3060 12GB	Qwen3 14B	Q4_K_M	~23–29 tok/s
RTX 4070	Qwen3 14B	Q4_K_M	~60 tok/s
RTX 3090 24GB	Qwen3.6-35B-A3B	Q3 (Unsloth)	~120 tok/s
RTX 3090 24GB	DeepSeek V4 Flash	Any	Not viable (103 GB min)
96 GB pooled + CPU offload	DeepSeek V4 Flash	Q2_K	8–12 tok/s
Dual RTX PRO 6000	DeepSeek V4 Flash	W4A16+FP8	~111 tok/s
H100 80GB (single)	DeepSeek V4 Flash	FP8 + offload	~20 tok/s

The Qwen3.6-35B-A3B number deserves attention: 120 tok/s on a single RTX 3090 with Unsloth's Q3 GGUF, at 23 GB VRAM. That's faster than real-time reading speed, from a 35B-parameter model, on a GPU that costs under $600 used. The model achieves this via its extreme MoE sparsity: 35B total parameters but only 3B active per token, so the compute per token is closer to a 3B dense model while retaining the quality of a much larger network.

For the original Qwen3 30B-A3B (slightly smaller than the 35B update), the Q4_K_M quantization fits in approximately 16.8 GB — giving comfortable headroom on a 24 GB card for KV cache. The full setup guide for Qwen3 30B-A3B walks through the Ollama and llama.cpp install paths.

Benchmark quality: what you're trading away

Speed means nothing if the model can't reason. Here's where the two families stand on standard benchmarks as of May 2026.

DeepSeek V4 Flash (at or near full precision):

AIME 2025: 99.4%
MMLU-Pro: 92.8%
BenchLM vs Qwen3 235B: 71 vs 33 (provisional)
Context window: 1,000,000 tokens

Qwen3 235B-A22B (frontier variant):

Context window: 128K tokens
Competitive with frontier proprietary models on most tasks
Requires 4+ H100s to run

Qwen3.6-35B-A3B (consumer GPU tier):

GPQA: 86.0%
AIME 2026: 92.7%
Coding: competitive with much larger dense models
Fits on a single RTX 3090 at Q3

The takeaway: at full or near-full precision, V4 Flash is clearly ahead of anything that runs on consumer hardware. But V4 Flash at Q2_K (the minimum viable local quant) suffers measurable quality loss — community testing suggests 1–3% degradation moving from FP8 to aggressive quantization, and more on harder reasoning chains. The practical question isn't "which model family is better at ideal conditions?" but "which model family am I actually able to run well?"

For more on how quantization affects reasoning quality at each level, see the Q4 vs Q8 quality loss analysis.

The 1M-token context window: useful or marketing?

V4 Flash supports a 1M-token context window vs Qwen3's 128K cap. That sounds like a decisive advantage for long-document work.

In practice, on consumer or even prosumer local hardware, 1M context is theoretical. Processing 1M tokens at 8–12 tok/s (the realistic local speed for V4 Flash with par

Apple MacBook Pro M5 Max for Local AI in 2026: 128GB Unified Memory, Neural Accelerators, and Whether It Beats a Discrete GPU Tower

Jovan Chan — Sat, 13 Jun 2026 07:05:20 +0000

This article was originally published on runaihome.com

TL;DR: The MacBook Pro M5 Max with 128GB unified memory runs 70B parameter models at 18–25 tok/s — models an RTX 4090 or RTX 5090 literally cannot load. The new Neural Accelerators cut prompt processing time roughly 4× versus the M4 Max on compute-intensive workloads. The catch: 128GB configurations start around $5,499, and you pay that premium specifically to run large models — for 8B and 13B work, a $1,699 Mac Mini M4 Pro nearly keeps up.

	MacBook Pro M5 Max 128GB	MacBook Pro M5 Max 36GB	RTX 4090 Tower Build
Best for	70B+ models, portability, multi-model	13B–32B daily use, better value entry	Raw speed on ≤24GB models
Memory / VRAM	128GB unified	36GB unified	24GB GDDR6X
LLM bandwidth	614 GB/s	614 GB/s	1,008 GB/s
70B Q4 tok/s	18–25	~9–12 (partial offload)	❌ can't load
8B Q4 tok/s	~82 (MLX)	~82 (MLX)	100–150
Power under load	60–90W	60–90W	450–600W (full system)
Starting price	~$5,499	$3,899 (16-inch)	~$3,500–$4,500
The catch	Expensive; no CUDA; no eGPU	70B won't fit cleanly	Loud, hot, can't leave your desk

Honest take: Buy the M5 Max 128GB if you need 70B models to run without VRAM gymnastics and you want a laptop. Build an RTX 4090 tower if you're doing multi-user serving, batch inference, or fine-tuning — CUDA still wins there. Neither choice is obviously wrong; they serve different workflows.

The Neural Accelerator story

From M1 through M4, Apple's GPU had no dedicated matrix-multiplication hardware. All the linear algebra that drives LLM inference ran through standard floating-point ALUs shared with graphics workloads — the same shader pipeline that renders a game frame also processed your model's attention layers.

M5 changes this. Apple built a dedicated Neural Accelerator into each GPU core for the first time. On the M5 Max with its 40-core GPU, that means 40 independent Neural Accelerators sitting on the same die, sharing the same 614 GB/s memory path as the GPU shaders.

Each Neural Accelerator performs 1,024 FP16 fused multiply-accumulate operations per cycle. Apple claims the M5 Max delivers over four times the peak GPU compute for AI workloads compared to M4 Max.

In practice, where this shows up is prompt processing — the prefill phase where the model reads your input context before generating the first token. Prefill is compute-bound, not memory-bandwidth-bound, so the Neural Accelerators directly attack the bottleneck. Early benchmarks from Apple's MLX team and third-party testing show prefill roughly 4× faster on M5 Max versus M4 Max for models like Qwen3-14B.

Token generation — the word-by-word output phase — doesn't benefit as much because it's still primarily memory-bandwidth-limited. The ~12% bandwidth increase (546 GB/s M4 Max → 614 GB/s M5 Max) translates to about a 20–28% improvement in sustained token generation speed.

One critical caveat: MLX is currently the only inference framework that fully exploits the M5 Neural Engine. Ollama, which uses llama.cpp under the hood, does not yet leverage the Neural Accelerators as of June 2026. If you're running Ollama today, you'll see the bandwidth gains but not the 4× prefill boost. MLX-native tools (mlx-lm, LM Studio with MLX backend, Open WebUI with MLX server) are where the real M5 performance lives.

Specs: what's actually inside

The M5 Max comes in two GPU configurations. For LLM work, which configuration matters:

Spec	M5 Max 32-core GPU	M5 Max 40-core GPU
CPU cores	14 (12P + 2E)	18 (16P + 2E)
Memory bandwidth	460 GB/s	614 GB/s
Neural Engine	16-core	16-core
Max unified memory	64 GB	128 GB
Neural Accelerators	32	40
AI compute	~46 TFLOPS FP16	~70 TFLOPS FP16

The 128GB memory ceiling requires the 40-core GPU variant — same pattern as M4 Max. If you're reading this because you want to run 70B models, you need the 40-core configuration.

The M5 Max is announced on TSMC 3nm, uses Apple's Fusion Architecture that connects two dies with advanced IP blocks, and was introduced in March 2026 alongside the updated 14-inch and 16-inch MacBook Pro lineup.

Real benchmark numbers

These benchmarks come from MLX-powered inference (mlx-lm), which represents best-case M5 performance. Ollama users will see the token generation numbers but not the prefill improvements.

Model	Quantization	M5 Max tok/s	M4 Max tok/s	Change
Llama 3 8B	Q4	82	64	+28%
Qwen 3.5 30B-A3B	Q4	58	45	+29%
Llama 3.3 70B	Q4_K_M	18–25	~14–18	+20–28%
Gemma 4 E2B	Q4	~158	~120	+32%
Phi-4 Mini	Q4	~135	~100	+35%

Prefill (prompt processing) numbers:

Qwen3-14B 16K context on M5 Max via MLX: roughly 8–10 seconds
Same workload on M4 Max: roughly 30–40 seconds
That's the Neural Accelerators doing actual work

For context: M4 Max (40-core GPU) was benchmarked at 83.06 tok/s on LLaMA 7B Q4_0 in the llama.cpp community benchmark thread (Discussion #4167). M5 Max isn't yet in that thread as of June 2026 — the numbers above come from third-party MLX benchmarks and Apple's MLX team test results.

The 70B model reality check

A Llama 3.3 70B model at Q4_K_M quantization occupies approximately 43 GB. That figure is the hard floor for running it without CPU offload, which tanks performance.

RTX 4090: 24GB VRAM. Doesn't fit. ❌
RTX 5090: 32GB VRAM. Doesn't fit. ❌
M5 Max 36GB config: Doesn't fit cleanly. You'd be splitting layers to CPU RAM, which drops tok/s to roughly 3–5. ❌
M5 Max 96GB config: Fits. ~19 tok/s. ✅
M5 Max 128GB config: Fits with 85GB to spare. 18–25 tok/s depending on framework. ✅

That "spare" capacity matters too. With 128GB you can simultaneously load a 70B model and a second 7B assistant model, or keep a large embedding model resident, or run retrieval-augmented generation workflows without swapping.

The M5 Max vs NVIDIA question

This comparison comes up constantly and the framing is almost always wrong. It's not "which is faster?" — the answer depends entirely on model size.

For models ≤24GB (8B, 13B, most 30B at Q4):

RTX 4090 wins on raw token generation: 100–150 tok/s versus M5 Max's ~82 tok/s. The RTX 4090 has 1,008 GB/s GDDR6X bandwidth — 64% more than M5 Max's 614 GB/s — and for small models that fit entirely in 24GB, that bandwidth advantage is fully realized. A well-tuned llama.cpp or vLLM setup on a 4090 runs Llama 3.1 8B at real-time speeds.

For 70B models and above:

M5 Max 128GB is the only sub-$10K option that runs them without compromises. The RTX 5090 at $1,999+ still only has 32GB, and 70B at Q4_K_M needs 43GB. Unless you pair two RTX 5090s in a dual-GPU setup with NVLink (expensive, complex, desktop-only), the Mac is the practical answer.

For MoE (Mixture of Experts) models:

Models like Qwen3.5 30B-A3B only activate ~3B parameters per forward pass, which means they need less bandwidth for token generation. The M5 Max's 614 GB/s is more than enough here, and 128GB gives you room for the full parameter set. At 58 tok/s, Qwen3.5 30B-A3B on M5 Max feels genuinely fast.

Power consumption:

M5 Max draws 60–90W during sustained inference. An RTX 4090 system (GPU + CPU + RAM + cooling) pulls 450–600W under the same workload. At $0.12/kWh, that gap is $0.048–0.061 per hour, or roughly $420–$530 per year if you're running inference 8 hours daily. Over three years the electricity difference alone is $1,260–$1,590. That meaningfully narrows the price premium of the Mac in total cost of ownership terms.

Want to compare cloud GPU costs against owning either? We ran that math in the [RunPod vs Local GPU analysis](/blog/runpod-vs-local-gpu-rent-

$200 Modded Tesla V100 for Local AI in 2026: Cheaper Than an RTX 5060 Ti and Surprisingly Competitive

Jovan Chan — Sat, 13 Jun 2026 07:04:23 +0000

This article was originally published on runaihome.com

TL;DR: A modded NVIDIA Tesla V100 SXM2 with a PCIe adapter costs around $200 total and outperforms the RTX 3060 by 42% on local LLM inference. Against an RTX 5060 Ti 16GB at $499–$589, the value argument is real — until you account for Ollama's broken support, a 300W power draw, and zero display output.

	Modded V100 SXM2 16GB	RTX 5060 Ti 16GB	RTX 3060 12GB
Best for	20–30B models on a tight budget	Balanced daily-driver LLM rig	7–13B models, display included
Memory bandwidth	900 GB/s	448 GB/s	360 GB/s
VRAM	16GB HBM2	16GB GDDR7	12GB GDDR6
Total cost (June 2026)	~$200–270	~$499–589	~$200–250 used
TDP	300W	180W	170W
Display output	None	Yes	Yes
Ollama support	Broken in v0.30+ (fix below)	Full	Full

Honest take: If you already have an iGPU or a second card for display, can compile llama.cpp from source, and want the best raw bandwidth per dollar under $300, the modded V100 is genuinely interesting. If you want something that just works, pay for the RTX 5060 Ti.

The mod: what it actually is

The Tesla V100 comes in two main physical formats. The PCIe version plugs into a desktop motherboard like any consumer card but is expensive and increasingly rare. The SXM2 version is a bare die designed for NVIDIA's DGX server backplane — faster (900 GB/s vs 897 GB/s) but it has no PCIe connector, no display output, and no cooling solution on its own.

The mod bridges that gap. A third-party PCIe adapter board (widely available on eBay) converts the SXM2 socket to a standard PCIe x16 slot. Add an external power supply (the adapter needs dual 8-pin PCIe connectors), strap on a 80mm Noctua fan with a 3D-printed shroud because the SXM2 module relies on server-chassis airflow, and you have a desktop AI accelerator that cost ~$200 in parts.

YouTuber Hardware Haven documented this build in detail and ran it against consumer GPUs in Ollama. The V100 hit 130 tokens/second on GPT-OSS-20B, outpacing the RX 7800 XT (90 tok/s) and the RTX 3060 12GB by 42%.

Why 900 GB/s matters for LLM inference

Memory bandwidth is the primary bottleneck for autoregressive LLM inference — not compute. During token generation, the GPU streams the model's weight matrix through memory on every forward pass. A card with twice the bandwidth generates roughly twice the tokens per second on the same model, all else being equal.

That's why the V100 SXM2's 900 GB/s matters more than its aging Volta architecture when you're running quantized local models:

GPU	Memory bandwidth	Architecture
Tesla V100 SXM2	900 GB/s	Volta (2017)
RTX 5060 Ti 16GB	448 GB/s	Blackwell (2025)
RTX 4090	1,008 GB/s	Ada Lovelace (2022)
RX 9070 XT	640 GB/s	RDNA 4 (2025)
RTX 3060 12GB	360 GB/s	Ampere (2021)

A 2025 Blackwell GPU with half the V100's bandwidth will lose on raw throughput for a single-user, low-batch LLM inference workload. The RTX 5060 Ti's 448 GB/s is solid — it's roughly what you'd expect for a $500 mid-range card — but the V100 SXM2 is nearly twice as wide.

The V100 also carries 125 TFLOPS of FP16 compute from its 640 Tensor Cores, meaning prefill (processing your prompt) is fast. In benchmarks from the llama.cpp community (Discussion #15396), a V100 16GB processed a 2,048-token prompt at 3,526 tok/s and generated subsequent tokens at 117.71 tok/s with GPT-OSS-20B at MXFP4 quantization.

Real benchmark numbers

These are the numbers from the Hardware Haven build and the llama.cpp community, not marketing estimates.

Hardware Haven mod test (Ollama, GPT-OSS-20B)

GPU	Tokens/sec	Notes
V100 SXM2 16GB (modded)	130 tok/s	Custom PCIe adapter, Noctua fan
RX 7800 XT 16GB	90 tok/s	Daily-driver GPU in the same rig
RTX 3060 12GB	~92 tok/s	Best NVIDIA card available for comparison

The V100 is 42% faster than the RTX 3060. At 100W power cap (to compare apples-to-apples), the V100 hit 95 tok/s at 170W wall draw vs. the RTX 3060 at 68 tok/s at 171W wall draw — same wall power, 40% more output.

llama.cpp benchmark (V100 16GB, GPT-OSS-20B, MXFP4)

Scenario	Tokens/sec
Prefill pp2048	3,527 t/s
Prefill pp8192	3,321 t/s
Prefill pp16384	2,769 t/s
Token generation tg128	117.71 t/s

The command that produced these results:

llama-server -hf ggml-org/gpt-oss-20b-GGUF \
  --ctx-size 32768 --jinja -ub 4096 -b 4096

GPT-OSS-20B in MXFP4 fits within 16GB at up to 32K context. Beyond 32K, you'll hit OOM on the 16GB variant.

The Ollama problem you'll hit immediately

If you buy a V100, set up the adapter, boot Linux, install Ollama, and try to run a model, you'll get this:

CUDA error: device kernel image is invalid

Ollama v0.30.0 dropped support for CUDA compute capability 7.0 (Volta/V100). The prebuilt CUDA libraries bundled with Ollama no longer include sm_70 kernels. Older versions (v0.24.0 and earlier) work fine, but you'd be running outdated software on a production setup.

LM Studio has the same issue — its bundled llama.cpp runtime doesn't include sm_70 kernels either (tracked in lmstudio-bug-tracker issue #1758).

The working solution: compile llama.cpp from source with explicit architecture support:

CUDA_HOME=/usr/local/cuda \
CUDACXX=/usr/local/cuda/bin/nvcc \
cmake -S . -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="70;86" \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -t llama-server -- -j 16

The 70 in DCMAKE_CUDA_ARCHITECTURES is the compute capability for Volta. You'll also want 86 if you ever add an Ampere card. After compiling, llama-server runs natively on the V100 with full GPU offload.

If you want to stick with Ollama, pin to v0.24.0. It's not ideal for long-term use but works as a stopgap.

Build cost breakdown (June 2026)

Component	What to buy	Price range
V100 SXM2 16GB	eBay used	$100–150
SXM2-to-PCIe adapter	eBay (various sellers, primarily China)	$50–100
80mm Noctua fan + 3D-printed shroud	Noctua + print locally	~$20–30
6+2-pin PCIe power cable (×2)	Already on most PSUs	$0
Total		~$170–280

The variation is wide because these are secondhand parts with no fixed retail. V100 SXM2 modules sell in the $80–180 range on eBay depending on seller, condition, and shipping origin. Budget $200 as your planning number and budget $280 if you want to be safe.

Complete kits (V100 SXM2 + PCIe adapter together) appear on eBay for $200–270, which is often the safer route — the adapter and card are tested as a pair.

For comparison, a new RTX 5060 Ti 16GB runs $499–589 at Newegg and Amazon as of June 2026, against an MSRP of $429 that's mostly theoretical at this point.

Power cost at 300W TDP

The V100 SXM2 has a 300W TDP. The RTX 5060 Ti pulls 180W. That gap is real money over time.

GPU	TDP	$/hour @ $0.12/kWh	$/month (8 hrs/day)
V100 SXM2	300W	$0.036	~$8.64
RTX 5060 Ti 16GB	180W	$0.0216	~$5.18
RTX 3060 12GB	170W	$0.0204	~$4.90

That's ~$3.50/month more for the V100 at 8 hours/day of inference — $42/year. Over the 3-year life of the hardware, it adds up to roughly $126 extra in electricity. Not dealbreaking, but factor it in.

If you're running inference 24/7 — say, a shared family LLM server — that gap triples. And at 300W, your PSU needs to handle it: budget a minimum 750W 80+ Gold unit for a V100 build.

What the V100 16GB can and can't run

Fits cleanly in 16GB

GPT-OSS-20B MXFP4: 11.27 GiB — ful