If you have ever tried running local language models on your laptop, you know the exact problem: your battery percentage drops before your eyes, and your terminal dumps an unformatted wall of text while you guess how much power your CPU is actually consuming.
When Microsoft Research published the BitNet b1.58 paper, the core concept sounded like the future of edge computing: train language models from scratch using ternary weights {-1, 0, +1} so you can eliminate floating-point matrix multiplications entirely. During inference, your CPU computes tensor operations using simple integer additions.
That promised 70% to 80% lower energy draw and extreme CPU speeds on ordinary laptops.
When I released the initial alpha of BitRun, thousands of developers tested it and gave me direct, unfiltered feedback:
- You wanted to see real electricity numbers: active Watts and Joules per token alongside generation speed.
- You wanted models larger than 2 billion parameters.
- You wanted an interactive interface that felt like a command center instead of a basic terminal scroll.
So over the last couple of days, we went back to the workbench and reworked the entire stack. Here is v0.2.0-beta.
The BitRun Cockpit: A real-time command center in your terminal
The centerpiece of this release is what I call BitRun Cockpit. Running bitrun cockpit bonsai-1.7b launches a dual-panel terminal interface designed specifically for low-power inference:
- The Left Panel: Interactive chat with character-by-character token streaming over local server-sent events. It includes independent scrolling so you can review earlier parts of long conversations without interrupting output.
-
The Right Panel: A live telemetry dashboard monitoring active hardware metrics:
- Generation and prompt speed in tokens per second
- Time-to-first-token in milliseconds
- CPU Performance-core and Efficiency-core load percentages
- Instantaneous CPU package power in Watts
- Cumulative energy consumed per token in Joules
- Estimated battery discharge rate
All background runtime logs are automatically piped to a dedicated file (~/.bitrun/engine.log), keeping the terminal interface crisp without screen tearing.
Scaling 1-bit computing to 8 billion parameters on ordinary laptops
A major question community members asked was whether 1-bit computing breaks down at scale.
In this release, BitRun runs PrismML's 8-billion parameter bonsai-8b entirely in CPU memory using around 2.08 GB of RAM. No dedicated GPU, no cloud API, and zero compilation required.
Because 1.58-bit models replace floating-point matrix multiplications with integer additions, the memory footprint drops by roughly 80% compared to 16-bit baselines, allowing full-scale reasoning models to live comfortably on standard laptops.
Rebuilding the engine from the ground up
To make this release possible, we rebuilt the underlying inference runtime:
1. Decoupled from Legacy Submodules
Rather than maintaining patches on top of Microsoft's frozen research submodule, we rebased our native engine on modern upstream llama.cpp architecture. This sovereign engine branch gives us complete control over SIMD optimization and tensor layouts.
2. Resolving the 8B Tensor Offset Crash
Early attempts to load 8B ternary checkpoints crashed due to a GGUF Type 42 tensor collision. We implemented dynamic legacy type remapping in the reader, enabling 8-billion parameter weights to load without memory faults.
3. Matrix Memory Bounds and Stride Safety
In packed 2-bit kernels, stride mismatches previously caused memory corruption past 32 context tokens. We standardized row stride calculations (.blck_size = 4, .type_size = sizeof(int8_t)) globally across the tensor graph, ensuring stable multi-turn conversations.
4. Architecture-Specific SIMD Acceleration
Integrated ARM NEON dot-product kernels and squared ReLU activations on Apple Silicon, paired with AVX2 fallbacks for Windows and Linux PCs.
What the numbers look like on actual hardware
Every metric below was recorded on physical hardware running with GPU offloading completely disabled:
Apple Silicon (MacBook Pro, M4 Pro CPU, 12 Cores, 24 GB RAM)
| Model | Parameters | Format | Disk Size | Generation Speed | Active Power | Energy / Token |
|---|---|---|---|---|---|---|
| bonsai-1.7b | 1.7B |
i2_s Ternary |
~237 MB | 72.8 tokens/sec | 11.7 Watts | 0.316 Joules |
| falcon3-1b | 1.0B |
i2_s Packed |
~1.30 GB | 73.2 tokens/sec | 11.7 Watts | 0.345 Joules |
| bitnet-2b | 2.4B |
i2_s Ternary |
~1.13 GB | 31.6 tokens/sec | 11.7 Watts | 0.484 Joules |
| bonsai-8b | 8.0B |
i2_s Ternary |
~2.08 GB | 21.0 tokens/sec | 11.7 Watts | 0.886 Joules |
| bitnet-70m | 70M | Q8_0 |
~79 MB | 1,619.8 tokens/sec | 11.7 Watts | 0.043 Joules |
Windows 10 Native (Intel Core i5-12600K, MSVC AVX2, 32 GB RAM)
| Model | Parameters | Format | Disk Size | Generation Speed | Active Power | Energy / Token |
|---|---|---|---|---|---|---|
| bonsai-1.7b | 1.7B |
i2_s Ternary |
~237 MB | 85.6 tokens/sec | 22.6 Watts | 0.264 Joules |
| falcon3-1b | 1.0B |
i2_s Packed |
~1.30 GB | 11.1 tokens/sec | 22.6 Watts | 2.036 Joules |
| bitnet-2b | 2.4B |
i2_s Ternary |
~1.13 GB | 6.7 tokens/sec | 22.6 Watts | 3.373 Joules |
| bonsai-8b | 8.0B |
i2_s Ternary |
~2.08 GB | 4.0 tokens/sec | 22.6 Watts | 5.650 Joules |
| bitnet-70m | 70M | Q8_0 |
~79 MB | 597.8 tokens/sec | 22.6 Watts | 0.038 Joules |
Linux WSL2 (Intel Core i5-12600K, GCC AVX2, 2 vCPUs)
| Model | Parameters | Format | Disk Size | Generation Speed | Active Power | Energy / Token |
|---|---|---|---|---|---|---|
| bonsai-1.7b | 1.7B |
i2_s Ternary |
~237 MB | 16.8 tokens/sec | 8.5 Watts | 0.505 Joules |
| falcon3-1b | 1.0B |
i2_s Packed |
~1.30 GB | 6.7 tokens/sec | 8.5 Watts | 1.261 Joules |
| bitnet-2b | 2.4B |
i2_s Ternary |
~1.13 GB | 4.4 tokens/sec | 8.5 Watts | 1.938 Joules |
| bonsai-8b | 8.0B |
i2_s Ternary |
~2.08 GB | 0.5 tokens/sec | 8.5 Watts | 16.058 Joules |
| bitnet-70m | 70M | Q8_0 |
~79 MB | 134.1 tokens/sec | 8.5 Watts | 0.063 Joules |
Note on architectural throughput: Apple Silicon benefits from dedicated ARM NEON dot-product intrinsics in arch/arm/quants.c. On x86_64, bonsai-8b and packed formats currently route through the portable generic loop in arch-fallback.h, which keeps execution stable across older CPUs until dedicated AVX2/AVX-512 kernels are added.
How you can spin it up right now (zero compile)
Pre-compiled standalone binaries are available for macOS (Apple Silicon), Windows (x86_64), and Linux. You do not need CMake, Python virtualenvs, or Docker:
On macOS or Linux:
curl -fsSL https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh | sh
On Windows (PowerShell):
irm https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1 | iex
Launch the Cockpit:
bitrun cockpit bonsai-1.7b
Run an OpenAI-compatible REST server on localhost:8080:
bitrun serve bonsai-1.7b
Point Cursor, Continue.dev, or Open WebUI directly to your local endpoint without cloud dependencies.
Open Source Links
All code and pre-compiled release binaries are open source under the MIT license:
- GitHub Repository: https://github.com/IshekKhal/bitrun
- Release Assets (v0.2.0-beta): v0.2.0-beta on GitHub
Try BitRun on your machine and tell me what energy metrics your CPU reports!
Top comments (0)