DEV Community

Cover image for Qwen3.8-27B on 2 RTX 3090: My First Local Model I Actually Trust
Arsen Apostolov
Arsen Apostolov

Posted on AI-assisted

Qwen3.8-27B on 2 RTX 3090: My First Local Model I Actually Trust

#llm #ai #homelab #selfhosted

For years "local LLM" meant a toy — a chatbot you played with on weekends, too slow or too dumb to trust with real work. This month, for the first time, I stopped worrying. My assistant, Jarvis, runs its daily tasks on a 27B model on my own hardware. On the tasks I actually run, it feels Sonnet-like — on some, better. Tests continue.

What it does — not chat, work

  • Read ~500 Jira emails and built a live tracker of 20+ demands: status, UAT dates, declared FTE benefits, what's closed and why.
  • A compliance table for the EU AI Act's image-marking rules — provider vs. deployer, the metadata-vs-visible-label split, with dates. It caught a term the legal team was misusing.
  • A net-worth rollup from my own books, and a gold-position sizing question that came back as a plan: 5–10% of liquid, DCA'd over 2–3 tranches.
  • The homelab itself: benchmarking models, watching the boxes, writing this post.

The point isn't any single task. It's that I stopped being careful. With an API, every prompt is a small cost and a small privacy decision. Locally, it's just a prompt.

The setup

Qwen

        ┌────────────────────────────────┐
        │        ArDi (the hub)          │
        │  56-core · 64GB · 52 Docker    │
        │  Jarvis 2 core · Whisper ASR   │
        │  embeddings · homelab-monitor  │
        └───────────────┬────────────────┘
                        │  Tailscale
        ┌───────────────▼────────────────┐
        │       Vader (inference)        │
        │  vLLM · Qwen3.8-27B (262K ctx) │
        │  2× RTX 3090 · 48GB VRAM       │
        │  tensor-parallel · port 8010   │
        └────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Qwen3.8-27B is dense 27B, Apache 2.0, 262K context, vision + video. I picked a dense 27B over a bigger MoE on purpose: one model that fits in 48 GB and is fast, not a 2.4T monster that needs a cluster. Under load Vader draws ~685 W at 80 °C. It's a server, not a laptop.

If you're going to do this

Four things I'd tell my past self:

  1. VRAM is the budget, not FLOPs. A 27B dense model at Q8/Q4 fits two 24 GB cards with headroom. That's why I went dense: the math is simple — weights + KV cache ≤ 48 GB. A 2.4T MoE would need a cluster and a mortgage.
  2. Split the box. Inference on its own machine, everything else (the assistant core, ASR, embeddings, the monitor) on the hub, talking over Tailscale. When the GPU pegs at 100%, the rest of the house keeps working. One box doing everything is where "local AI" dies.
  3. vLLM over Ollama for serving. Ollama is fine for tinkering; vLLM with tensor parallelism is what makes 48 GB feel like one big GPU and what gets you real tokens/sec. Port 8010, OpenAI-compatible API — your existing clients just point at it.
  4. Measure the power, or you're guessing. "It's a 685 W box" only matters if you can attribute that draw to the process and price it. I built homelab-monitor for exactly that — VRAM and power per process, energy priced at your tariff. The whole hub ran 4.97 kWh this week: 1.30 BGN / 7 days. If you run more than one box, it's the thing I'd reach for first.

vLLM

The honest part

Tests continue. The hardest agentic chains still make me double-check. But for the daily 80%, it's reliable enough that I stopped caring. A local model you have to think about is a toy. One you don't is infrastructure.

Drafted with AI assistance. Hardware, cost and task data are from my own homelab, measured live.

Top comments (0)