DEV Community

Tech Workflow Hub
Tech Workflow Hub

Posted on

Running Inkling 2 Mini Locally & on vLLM: 26.4B Looped MoE, 1M Context & B200 PR #69104 Notes

Thinking Machines Lab released Inkling 2 Mini on November 20, 2026 under an Apache 2.0 license—built specifically as a multimodal (text + image) looped Mixture-of-Experts (MoE) model for laptops, MacBooks, and local GPU rigs.

Here is a practical engineering breakdown of its architecture, first-party API pricing, and early vLLM deployment notes.

1. Looped MoE Architecture & Specs (26.4B Total / ~3.4B–5.4B Active)

  • Physical vs. Active Footprint: 26.4B total parameters across 24 decoder layers, activating ~3.4B to 5.4B parameters per token depending on looping passes (the middle 12 layers execute in a 3-pass loop for 48 effective layer applications per token).
  • Context Window: Up to 1,000,000 (1M) tokens.
  • Quantization Formats: Official NVFP4 and MXFP8 support, plus community GGUF builds (via Unsloth) for local CPU/Apple Silicon inference.

2. Inference Notes: vLLM PR #69104 & B200 Driver Fix

Because looped MoE kernels hit sharp hardware edge cases at launch, check recent engine commits before deploying in production:

  • NVIDIA Blackwell (B200) Note: LLM PR #69104 specifically patches Inkling 2 Mini by dropping the ile-8 TRT-LLM-gen expert GEMM kernel to prevent an illegal memory access on driver 610.57.
  • Similarly, we are tracking early inference PRs across open ecosystems (see our DeepSeek V4.1 Flash vLLM PR #69107 tracker).

3. API Pricing vs. Self-Hosting

If you prefer calling the hosted Thinking Machines endpoint instead of local GGUF/vLLM serving:

  • Input: $0.15 / 1M tokens
  • Cached Hit: $0.015 / 1M tokens
  • Output: $0.50 / 1M tokens

For teams wiring local or API models into multi-step automation pipelines, see our Tech Workflow Hub hub for cost-control benchmarks.

Top comments (1)

Collapse
 
ahmetozel profile image
Ahmet Özel •

The looped MoE design—looping the middle 12 layers across three passes for 48 effective layers while keeping 3.4B to 5.4B active parameters—is an interesting trade-off between memory footprint and compute depth. For local deployment, shrinking parameter memory lets a 26.4B model fit where a dense 48-layer model never could.

Calling out the B200 driver 610.57 illegal memory access in vLLM PR 69104 is also a good reminder that day-zero kernel support is where novel architectures usually bite first. Curious how KV-cache memory scales on that 1M context window across the three looping passes.