DEV Community

Abhishek Khanra
Abhishek Khanra

Posted on

I Fixed Microsoft's BitNet C++ Kernels: How to Run 1-Bit LLMs on Pure CPU Without Compiling

#opensource #machinelearning #cpp #ai

Earlier this year, Microsoft Research published the BitNet b1.58 paper, demonstrating that 1-bit ternary models {-1, 0, +1} replace floating-point matrix multiplications with simple integer additions on CPU. On paper, this delivers 70–80% lower energy draw and extreme CPU inference throughput.

However, compiling Microsoft's official research repository (bitnet.cpp) has been broken for most developers:

  1. Hardcoded CMake compiler guards threw fatal errors under MSVC on Windows.
  2. Batch stride memory overwrites caused recurring segfaults on Apple Silicon (ARM64).
  3. The repo required recursive submodules, specific Clang versions, Python virtualenvs, or 5-10 GB Docker images.

The Engineering Fixes

I patched the native C++ engine bugs and submitted two pull requests upstream:

  • PR #642: Fixes ARM64 2B batch stride memory overwrites, FFN activation alignment (SiLU vs ReLU²), and tokenizer inference bugs.
  • PR #644: Adds native MSVC compiler support and CLI build targets on Windows.

Zero-Compile Single Binary: BitRun

To make running these models zero-friction on commodity hardware, I packaged BitRun: a standalone, statically linked runner with native SIMD kernels (NEON DotProd on Mac, AVX2 on Windows/Linux) and zero build dependencies.

Verified Benchmarks (100% CPU, 0% GPU)

  • Apple Silicon M4 Pro (12 Cores, ARM NEON):
    • Falcon3 1B Instruct: 137.5 t/s prompt | 98.6 t/s generation (1.9 GB RAM)
    • BitNet 70M: 8,758 t/s prompt | 1,355 t/s generation (<150 MB RAM)
  • Windows 10/11 x86_64 (Intel Core i5-12600K, AVX2):
    • Falcon3 1B Instruct: 40–45 t/s generation
  • Linux x86_64 (WSL2 / Ubuntu, AVX2):
    • Falcon3 1B Instruct: ~130 ms TTFT | 40–45 t/s generation

Realistic Scope

These 1-bit checkpoints (Falcon3-1B, BitNet-2.4B) are low-power research prototypes for edge and CPU inference, not replacements for cloud frontier APIs. Their primary purpose is testing ternary integer execution on low-wattage devices.

Quickstart (One-Line Install)

macOS & Linux: 

curl -fsSL https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Windows (PowerShell): 

irm https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1 | iex
Enter fullscreen mode Exit fullscreen mode

Run in Terminal: 

bitrun pull falcon3-1b
bitrun run falcon3-1b
Enter fullscreen mode Exit fullscreen mode

Serve via OpenAI-Compatible REST API: 

bitrun serve falcon3-1b --port 11435
Enter fullscreen mode Exit fullscreen mode

Top comments (0)