DEV Community: Hocine Benkelaya

nbxtensor: a tensor library for inference that doesn't need PyTorch

Hocine Benkelaya — Sat, 07 Nov 2026 22:49:21 +0000

If you want to run a modern model on a GPU from Python, the default answer is "install PyTorch". That is a fine answer for training. For inference it brings a lot along: a large install, a CUDA-first view of the world, and a runtime whose memory behaviour you discover rather than decide.

nbxtensor is NeuroBrix's answer for inference: a tensor library with Triton kernels, planned memory and certified tuning — and no PyTorch.

pip install nbxtensor

It runs wherever Triton runs: NVIDIA, AMD, Intel and Apple GPUs, on Linux, Windows and macOS.

What "no PyTorch" means here

Not "PyTorch is optional". nbxtensor never imports torch — not as a dependency, not as an optional extra, not in a fallback branch. Its kernels, its tests and the models NeuroBrix runs on it all work in an environment where torch is not installed.

The practical effect shows up in the NeuroBrix engine that is built on it: measured on a clean machine, the light install (the Triton engine on nbxtensor) is 1.48 GB; the same engine with PyTorch added is 6.82 GB.

If you do live in a PyTorch codebase, exchange goes through DLPack, so you are not locked in either direction.

A first kernel

This is the example from the README, executed by the test suite on every change:

import numpy as np
from nbxtensor.tensor import NBXTensor
from nbxtensor.ops.linalg_wrappers import mm

rng = np.random.default_rng(0)
a = rng.standard_normal((256, 512)).astype(np.float16)
b = rng.standard_normal((512, 128)).astype(np.float16)

c = mm(NBXTensor.from_numpy(a), NBXTensor.from_numpy(b))   # a Triton kernel on the GPU, no torch

ref = a.astype(np.float32) @ b.astype(np.float32)
assert c.shape == (256, 128)
assert np.allclose(c.numpy().astype(np.float32), ref, rtol=2e-2, atol=2e-1)

NBXTensor holds device memory, a shape and a dtype. The operation dispatches to a Triton kernel. There is nothing else in the path.

Four parts

1. Tensors. Device memory, shapes, strides, views, dtypes and their promotion rules — the plumbing you need to run a graph, without a framework underneath.

2. Triton kernels. Matrix products, attention, convolutions, normalisations, elementwise and reductions, written in Triton so that one source serves every GPU vendor Triton supports. A device is described by a file of facts — tile sizes, shared memory, the matrix unit — not by code.

3. Planned memory. Allocators per device behind one interface, and a planner that sizes buffers before the first run. Every kernel declares what it costs (its outputs, workspace and scratch), so the planner can price a model before running it. In NeuroBrix, that planner is Prism: it decides from your hardware profile whether a model fits on one GPU, is split across several, is streamed, or has its largest operations tiled.

4. Certified tuning. A Triton kernel has settings — block sizes, warps, stages — and the right ones depend on the GPU and the shape. The usual approach is to autotune on the user's machine at first run. nbxtensor instead serves settings that were certified once per GPU class: each candidate is checked against a float64 oracle, the ones that pass are timed, and the winner is stored with its proof. 18,552 settings are certified for Apple GPUs today.

If your GPU class has no certificate yet, the missing settings are tuned once on your machine and kept; you can then certify them and share them by pull request, so the next person with your GPU never tunes at all.

Fast where it counts

Two examples of what that buys:

In NeuroBrix's global benchmark of 2026-11-05 on V100 16 GB GPUs, the Triton engine on nbxtensor was faster than the PyTorch engine on 18 of 49 models, up to 5x on Janus-Pro-7B (212.6 s down to 41.8 s).
On Volta, Triton compiles matrix products to FMA, so the V100's tensor cores sit idle. nbxtensor brings them back with hand-written m8n8k4 kernels, chosen per operation by measurement rather than by rule.

Apple Silicon

PyPI publishes no Triton for macOS. nbxtensor installs NeuroBrix's own Triton build for Apple GPUs (NeuroBrix/triton-ext) in one command:

pip install nbxtensor
python -m nbxtensor install-metal
python -m nbxtensor doctor     # -> backend in force: metal

doctor works on every platform: it names the backend in force and, when a GPU runtime or a Triton backend is missing, the command that installs it.

Where it is used

nbxtensor is the foundation of the Triton engine of NeuroBrix, which runs 49 models in 9 families — language, code, vision-language, omni, speech recognition, speech understanding, text-to-speech, image, video and upscaling — from one .nbx container format and one entry point. Every NeuroBrix release runs all 49 models on both engines through that entry point and compares the outputs byte for byte with judged outputs.

Inference first: training is not in this release.

NeuroBrix 0.6: 49 AI models, one engine, no PyTorch

Hocine Benkelaya — Sat, 07 Nov 2026 22:48:45 +0000

Most inference stacks are a collection of pipelines: one for language models, another for speech, another for image generation, another for video, each with its own loader, its own memory habits and its own list of supported GPUs. NeuroBrix takes the opposite route. One engine runs every model. Version 0.6 is its first stable release.

pip install neurobrix

It runs on Linux, Windows and macOS, on NVIDIA, AMD, Intel and Apple GPUs — wherever Triton runs.

49 models, 9 families, one entry point

NeuroBrix 0.6 runs 49 models across 9 families: language and code models (Qwen3-30B-A3B, Qwen3-Coder, DeepSeek-Coder-V2-Lite, DeepSeek-MoE), vision-language and omni models (GLM-4.1V, Qwen3-VL, Qwen3-Omni, Janus-Pro, MiniCPM-o), speech recognition (Whisper, Parakeet), speech understanding (Voxtral), text-to-speech (Kokoro, VibeVoice, Orpheus), image generation (Sana 1K and 4K, PixArt, Flex.1), video generation (Wan 2.1 and 2.2, CogVideoX, Mochi, Allegro, Open-Sora 2) and upscaling (Real-ESRGAN, HAT, SwinIR, Swin2SR).

They all go through the same command. The family is read from the model, so run asks for the inputs that family needs:

neurobrix import TinyLlama/TinyLlama-1.1B-Chat-v1.0
neurobrix run --model TinyLlama-1.1B-Chat-v1.0 --prompt "Explain recursion." --max-tokens 256

neurobrix run --model whisper-large-v3-turbo --audio recording.wav
neurobrix run --model Kokoro-82M --prompt "Hello from NeuroBrix." --output speech.wav
neurobrix run --model Flex.1-alpha --prompt "a lighthouse at dusk" --steps 20 --seed 42 --output lighthouse.png
neurobrix run --model CogVideoX-2b --prompt "a hot air balloon over mountains" --num-frames 17 --output balloon.mp4

The browsable list lives at neurobrix.es/models.

One container, no per-model code

Every model ships as a .nbx container: one file format that holds the model's graph and its weights. The engine reads the container and executes it. There is no if model == ... anywhere in the runtime, no model-specific pipeline to maintain, and adding a model does not mean writing a new code path. That is what lets one small team run 49 models through a single entry point.

Two engines in one package

NeuroBrix carries two execution engines that share the container, the plan and the flow, and share no compute code:

The Triton engine, built on nbxtensor, NeuroBrix's own tensor library. It does not use PyTorch at all — not at import, not at run time.
An optional PyTorch engine, for those who already have PyTorch: pip install "neurobrix[pytorch]".

By default NeuroBrix uses PyTorch if it is installed and the Triton engine otherwise. The difference shows on disk. Measured on a clean machine, the light install is 1.48 GB; with PyTorch it is 6.82 GB.

Prism: placement from facts, not guesses

Before anything runs, Prism, NeuroBrix's memory and placement planner, decides how the model sits on your hardware: on a single GPU, across several GPUs, streamed in pieces, or with its largest operations tiled. It works from a hardware profile — a file of facts about your devices — rather than from heuristics baked into code. When the machine is busy, it says so and tells you what it chose instead.

Speed

Speed comes from kernels that are measured, not assumed. In the global benchmark of 2026-11-05 on V100 16 GB GPUs, the Triton engine was faster than the PyTorch engine on 18 of the 49 models, up to 5x on Janus-Pro-7B (212.6 s down to 41.8 s).

Two pieces make that possible:

Certified kernel tuning. Every kernel setting is certified once per GPU class — checked against a float64 oracle, then timed — instead of autotuning on your machine. 18,552 settings are certified for Apple GPUs alone, and anyone can certify their own GPU and share it by pull request.
V100 tensor cores, back. Triton compiles matrix products to FMA on Volta, which leaves the tensor cores of a V100 idle. nbxtensor brings them back with hand-written m8n8k4 kernels, chosen per operation by measurement.

Validated the boring way

Every release runs all 49 models on both engines through the real entry point — the same neurobrix run you type. For 0.6.0 that was 196 of 196 cells on 4x V100. Outputs are compared byte for byte with outputs that were judged by eye (text read, images and videos looked at, audio transcribed), and runs are deterministic: the same inputs give identical outputs.

Apple Silicon

On a Mac, PyPI has no Triton build. NeuroBrix maintains its own Triton build for Apple GPUs, NeuroBrix/triton-ext, installed in one line:

pip install neurobrix
python -m nbxtensor install-metal
neurobrix doctor

Try it

Engine: github.com/NeuroBrix/neurobrix — Apache 2.0
Tensor library: github.com/NeuroBrix/nbxtensor
Models: neurobrix.es/models
Release notes: v0.6.2
Discussion: github.com/orgs/NeuroBrix/discussions/4

Found a model that misbehaves on your GPU? Open an issue — that is exactly the feedback this release is for.

— Hocine Benkelaya, NeuroBrix

DEV Community: Hocine Benkelaya

nbxtensor: a tensor library for inference that doesn't need PyTorch

What "no PyTorch" means here

A first kernel

Four parts

Fast where it counts

Apple Silicon

Where it is used

Links

NeuroBrix 0.6: 49 AI models, one engine, no PyTorch

49 models, 9 families, one entry point

One container, no per-model code

Two engines in one package

Prism: placement from facts, not guesses

Speed

Validated the boring way

Apple Silicon

Try it