DEV Community

Cover image for Everyone Is Benchmarking Edge0-145B on RTX 4090s — I Ran the 35B Tier on a $220 Used Mini PC
qianchuan meng (qianchuan)
qianchuan meng (qianchuan)

Posted on Edited on

Everyone Is Benchmarking Edge0-145B on RTX 4090s — I Ran the 35B Tier on a $220 Used Mini PC

All numbers in this post are my own measurements, not official data.
Hardware: Beelink SER (1st gen) mini PC / AMD Ryzen 5 5600H (6C/12T) / 16GB DDR4-3200 single-channel / Vega iGPU / 512G NVMe / Windows 11 x64.
Second-hand price for this exact class (Xianyu listings checked 2026-11-14): ¥1,550–1,590 ≈ $215–220 for a complete unit, ¥1,159 ≈ $160 barebones.

The setup

On Nov 12, Edge0 open-sourced Edge0-145B and Edge0-120B with very provocative numbers:

  • 145B params, 29GB peak memory, 80+ tokens/s decode
  • Runs on "a suitable laptop", no server needed
  • Coding, reasoning and agent capability "comparable to frontier cloud models"

Three days in, I went looking for real hands-on tests. There are only a handful, and they're all on RTX 4090/5090 or M2 Ultra (jansverre, goremy, etc.). Chinese platforms are nothing but press rewrites — zero real deployments. Even better: a Japanese blogger already debunked the headline memory claim — the advertised "29GB peak" counts active memory only; real footprint is 54GiB.

Meanwhile, if you're on AMD or an iGPU, this party was never for you: the 145B tier requires NVIDIA CUDA. Your ceiling in the Edge0 family is the 35B tier — the one the official docs say runs on plain CPU.

Cold water first: Edge0-35B is a September preview checkpoint. The model card itself admits weak agent capability, and 145B/120B are the main event. So why bother? Because between "runs on CPU" and "actually usable" sits an order-of-magnitude experience gap, and nobody has quantified it on cheap hardware.

So I did. Here are real numbers from a $220 used mini PC with single-channel RAM and no discrete GPU.

TL;DR: it runs. Warm-cache first token in 1.3s, decode 4.27 tok/s — fine for chat. But prefill is 11.5 tok/s, which makes it mathematically unusable as a coding-agent backend.

Deployment: 4 pitfalls the docs don't mention

Pit 1: HuggingFace is unreachable from China

huggingface.co times out here. The fix is the hf-mirror.com mirror — the model is 18.38GB across 4 shards (4-bit quant + LoRA + pre-router), ~45 minutes with 4 parallel downloads:

curl -L -C - -o model-00001-of-00004.safetensors \
  "https://hf-mirror.com/Edge0/Edge0-35B-A3B-preview/resolve/main/model-00001-of-00004.safetensors"
Enter fullscreen mode Exit fullscreen mode

Keep -C - for resume. Verify byte counts with stat -c%s — all 4 of my shards matched.

Pit 2: The release zip already contains everything

edge0-chat-0.2.0-windows-x64.zip (264MB) from GitHub Releases ships a prebuilt edge0-infer.exe — no compiling — plus the official chat app and the edge0-strata/ engine (for 145B, useless here).

Pit 3: The asset converter crashes on Windows

35B can't use raw HuggingFace weights; you convert them to GGUF assets (+22GB disk). The official converter fails immediately with:

error: [WinError 1314] 客户端没有所需的特权。
Enter fullscreen mode Exit fullscreen mode

Cause: it symlinks source weights into a staging directory, and non-admin Windows without Developer Mode can't create symlinks. Two fixes:

  1. Enable Developer Mode, or
  2. Patch the converter to fall back to a hard link (same volume, no privileges needed, semantically equivalent):
try:
    link.symlink_to(target)
except OSError:
    os.link(target, link)      # hardlink fallback
Enter fullscreen mode Exit fullscreen mode

After the patch: conversion finished in 661s, self-check passed all 703 tensors.

[self-check] edge0-35b.gguf: tensors=703 kv=39 OK
DONE 35b: gguf=20.26GiB lora=40.5MiB took 661.3s
Enter fullscreen mode Exit fullscreen mode

Pit 4: iGPU "Vulkan acceleration" is a negative optimization

The docs support --prefill-device vulkan. On their RTX 4070 SUPER reference machine it cuts time-to-first-token from 17s to 3.9s — tempting.

On my Vega iGPU it's worse across the board: prefill didn't improve (the iGPU is too weak), and decode dropped from 4.27 to 2.93 tok/s — the engine log shows 80 GPU↔CPU handoffs, each one a tax.

Lesson: iGPU users should just stay on CPU.

Bonus: why not just run vanilla Qwen3.6-35B-A3B via Ollama?

Fair question — Edge0-35B is Qwen3.6-35B-A3B plus a LoRA and a pre-router. Two reasons:

  1. Qwen3.6's Gated DeltaNet architecture has a decode-collapse bug on llama.cpp's AMD Vulkan backend — prefill runs 3100+ tok/s, then decode falls off a cliff to single digits. Known issue; my old iGPU would almost certainly hit it.
  2. Edge0's predicted routing (a trained pre-router prefetches the next token's experts from SSD while the current one computes) is a free ~50% decode speedup you don't get on the vanilla model.

On this machine, the Edge0 build is the better deal.

Benchmark: three configurations

Method: 253-token Chinese prompt, 10 warmup steps, 100 timed generation steps, official bench tool.

Config Decode Prefill TTFT Verdict
CPU 4 threads (default) 4.27 tok/s 11.5 tok/s 22.0s ✅ best
CPU 6 threads 4.20 tok/s 11.3 tok/s 22.5s no gain
Vulkan prefill + CPU decode 2.93 tok/s 11.0 tok/s 23.1s worse

The local chat UI I built after deployment — code rendering + metrics chips. This run: TTFT 8.4s, 130 tokens at 3.0 tok/s.

Three counter-intuitive findings:

Local chat UI running Edge0-35B — code rendering with metrics chips
1. More threads do nothing — the bottleneck is memory bandwidth. 6 threads scored the same as 4. Single-channel DDR4-3200 tops out at 25.6GB/s, and every decoded token streams the active experts' weights through memory. Cores can't help; they're all waiting on RAM.

2. Sparse MoE + predicted routing is why this survives on cheap hardware. 35B total params, ~3B active per token. The engine's pre-router predicts which experts the next token needs and prefetches them from SSD while the current token computes: 3,564 prefetches, only 2 late. This is the core of Edge0's arXiv paper (The Other Half of the Memory Wall).

3. Peak memory was 3.77GB. A 16GB machine never breaks a sweat — expert weights mostly live on SSD and page in on demand. The "phone-class memory" claim is real.

The soul-searching question: can it be a coding-agent backend?

No — and it's arithmetic, not opinion.

OpenCode's agent mode ships a system prompt of roughly 11,000 tokens per request (tool descriptions + role context). My prefill speed is 11.5 tok/s:

11,000 ÷ 11.5 ≈ 16 minutes
Enter fullscreen mode Exit fullscreen mode

That's your wait before the first token appears — before the model even starts "thinking".

The deeper issue: agent tools and SSD-streaming inference are architecturally mismatched. Agents are "huge system prompt + frequent multi-turn calls"; streaming engines shine at "small prompt + long output". Dual-channel RAM would double prefill to ~23 tok/s — 16 minutes becomes 8. Still unusable.

So who is this actually for?

Use case Experience Verdict
Local private Q&A (chat UI) 1.3s warm / ~6s cold first token ✅ sweet spot
Document analysis ≤3K tokens ~30s to first token ✅ acceptable
Long-form writing 4.3 tok/s, slower than reading ⚠️ painful
Coding-agent backend 16-minute first token ❌ mathematically dead

If you want the headline speeds, know which tier buys what:

  • 35B tier: no NVIDIA required. The official support matrix is macOS Metal/CPU, iOS, Android CPU, Linux CPU, Windows CPU/Vulkan — no NVIDIA anywhere. The bottleneck is memory bandwidth: my 4→6 thread test was flat, and official numbers scale with bandwidth — M5 16GB does 24.8 tok/s, Android flagships 6–9 tok/s.
  • 145B tier: NVIDIA is a hard gate. Linux/Windows + NVIDIA GPU + CUDA 13, compute capability ≥7.5 (RTX 20-series up) per the v0.2.0 release notes. There's no official "minimum spec", but the model card includes a constrained test run: 16 GiB VRAM + 20 GiB RAM budget, Q3_K, still 44.9 tok/s — the closest thing to a documented floor. On the speed table: RTX 4090 24GB does 91.7 tok/s, RTX 5090 D does 121.2. Also budget disk: 145B weights are 40.8 GiB (Q3_K) / 47.8 GiB (NVFP4).

One line: 35B runs on anything; 145B starts with buying an NVIDIA card.

The junk-reeconomy math

Item Cost
Beelink SER 5600H/16G/512G (used, 2026-11 market) ~$215
+16GB stick for dual-channel (optional upgrade) ~$28
Electricity (~60W × 4h/day) ~$1/month
Models, engine, tooling $0 (open source)
Total ~$245 for a fully offline, no-subscription, data-never-leaves local LLM terminal

For comparison: a new RTX 4090 is $1,600+. This budget doesn't buy "fast" — it buys "runs" and "private". And the remaining bottleneck (memory bandwidth) happens to be the cheapest upgrade in consumer hardware.

I still think this deployment proved something: Edge0's claimed floor is real. A GPU-less, single-channel, second-hand mini PC really can run a 35B MoE at interactive speeds. Three years ago that needed a server; today it fits in a shoebox.


What's the biggest model you've run on the worst hardware? Comments welcome.


Declarations: all benchmarks are personal measurements on the stated hardware, not official data; the tested checkpoint is Edge0-35B-A3B-preview. Edge0 name/trademarks belong to the Edge0 team (GitHub: Edge0-AI); this is an independent review — the UI logo in my screenshots is my own design. Inference framework: Apache-2.0; model weights: Edge0 Community Model License 1.0. References: official docs and the edge0-infer repo; patches shown inline.

Top comments (1)

Collapse
 
crtdrops profile image
Israel Negrete Lepe •

The thread test is the most useful line in here: 4 and 6 threads coming out flat says memory bandwidth, not cores, and that rules out half the tuning people would try first.

One thing I'd add: a 100-step bench is short enough to finish before a mini PC heats up. On an Intel N100 mini PC with a fan, running a vision model non-stop for 30 minutes, it reached 102 °C, the clock dropped from 2,790 to 2,514 MHz and throughput fell 3.9 %. A different workload from yours, but the same kind of box, and none of it showed in the first minutes.

Did you try one long generation, say 20 or 30 minutes, to see whether 4.27 tok/s holds once the 5600H is hot? Since the experts page in from the SSD, the drive's temperature might matter here too, though that part is only a guess.