DEV Community: BytePenny

What is dequantization?

BytePenny — Tue, 20 Oct 2026 00:00:00 +0000

A language model is mostly numbers: billions of weights. Stored at full precision, an 8B model takes about 16 GB, more than most laptops have. Quantization shrinks those numbers so the model fits. Dequantization turns them back into usable values when the model runs, and how an engine does that decides a large part of its speed.

Quantization: fewer bits per weight

A quantized file does not store each weight as a 16-bit number. It groups weights into small blocks, often 256 at a time, and stores:

one shared scale for the block (a normal number),
small integer codes for each weight, 2 to 8 bits each.

A weight is then roughly scale × code. Some formats add a per-block offset, or a second, smaller scale for every 16 or 32 weights, to follow the real values more closely.

Format (GGUF)	Bits per weight	8B model size
F16 (no quantization)	16	about 16 GB
Q8_0	8.5	about 8.5 GB
Q4_K_M	about 4.8	about 4.9 GB
IQ2_M	about 2.7	about 2.9 GB

The “I-quants” (IQ1 to IQ3) go further: instead of plain integers they store an index into a fixed table of good value patterns (a codebook), plus sign bits. That is how they reach 2 bits per weight and still produce readable text.

Dequantization: getting the numbers back

To compute with a weight, the engine has to know its value, so it reverses the packing: read the scale, read the code, multiply. That is dequantization.

Why bother? Because writing each token means reading nearly all the weights once, and memory speed is the limit (see CPU vs GPU). A model that is 4 times smaller is read about 4 times faster, as long as unpacking it doesn’t eat that gain back.

The price is a little accuracy (Q4_K_M is close to the original, 2-bit types lose more) and the extra work of unpacking.

Two ways to do it

Unpack everything at load time. Turn the whole model back into 16- or 32-bit floats once, then use ordinary fast matrix maths. Simple and fast, but the model needs several times its file size in RAM. A 2.6 GB Ministral 3B file needs about 7.5 GB this way.

Compute on the packed weights. Keep the file as it is and unpack each small block only at the moment it is multiplied. RAM use stays close to the file size, but the unpacking happens again for every token, so it has to be very cheap.

How Matricxon does it

Matricxon uses the second way by default (packed weights). The model file is memory-mapped, and native C kernels multiply the packed blocks directly:

The input is quantized too. The activation vector (the token’s current state) is turned into 8-bit integers, 256 at a time, with one scale each.
Integer dot products. Packed weights and the 8-bit input are multiplied as small integers with SIMD instructions that handle 16 bytes at once. Floats come back only once per block: weight scale × input scale × integer sum.
Unpack once, reuse many times. When reading a prompt, each weight row is unpacked once and reused for every prompt token while it is still in the CPU cache.
Built for your server. The C library compiles on first start with -march=native, so it uses whatever your CPU supports.

Every one of the 25 quantization types has a plain Python reference that unpacks a block step by step. The fast kernels, and the Numba versions used when no C compiler is available, are tested against it, so a speed-up can’t quietly change a result. If there is RAM to spare, MATRICXON_ENABLE_QUANTIZED_NATIVE_COMPUTE=false switches to unpacking everything at load time. On an NVIDIA GPU (experimental) weights are unpacked into VRAM by default, where the GPU’s float maths is fastest; MATRICXON_GPU_WEIGHT_MODE=packed keeps them compressed to save VRAM.

A recent example: making 2-bit models usable

Until this week, Matricxon’s I-quant kernels unpacked every weight to a float and multiplied one at a time. A 2-bit Llama 3.1 8B took over 3 minutes for its first word on an older laptop CPU (partly because the laptop was also short on RAM), then about 15 seconds per word.

The fix came from looking at the formats again: every I-quant weight is exactly a small integer scale × a small whole number from its codebook. So these types now use the same integer path as the 4-bit types, with nothing approximated. On one layer of that model:

IQ2_S, one 14336 × 4096 layer	Before	After
Writing one token	44.9 ms	9.6 ms
Reading a 16-token prompt	529 ms	about 57 ms

End to end, the first word now comes in 10 to 16 seconds and each further word in about 2.

Which quantization should you pick?

Q4_K_M is the usual best choice: about a third of the original size, close in quality, and fast to unpack.
Q8_0 if you have the RAM and want quality closest to the original.
IQ2 and IQ3 to fit a bigger model into little memory. They lose more quality and are slower to unpack on a CPU, so a smaller model in Q4_K_M is often the better trade.

pAIring shows each model’s expected memory use for the active engine before you download it, so you can see what fits your hardware.

Originally published at bytepenny.com.

Wolfram|Alpha as an MCP server: exact math for your AI

BytePenny — Thu, 15 Oct 2026 00:00:00 +0000

Ask a language model for 2¹⁰⁰, the integral of x² sin x, and it will answer confidently. Sometimes it is right. A language model doesn’t calculate; it predicts the next piece of text, and digits are just text to it. That works for words, but numbers need to be exact.

Wolfram|Alpha was built to do the opposite: compute answers instead of predicting them. Thanks to MCP (the Model Context Protocol), any AI app that supports MCP can now hand math and factual questions to Wolfram|Alpha and get back a computed result. This post explains what that is, what you gain, and where the limits are.

What Wolfram|Alpha is

Wolfram|Alpha is a computational knowledge engine from Wolfram Research, online since 2009. It is not a search engine: instead of finding pages that might contain the answer, it works the answer out. Underneath are two things:

The Wolfram Language (the language behind Mathematica), with decades of algorithms for algebra, calculus, statistics, number theory, geometry and much more.
The Wolfram Knowledgebase , a large, curated collection of data on chemistry, physics, astronomy, geography, finance, units and other fields.

It takes plain-language questions such as “solve x^3 - 2x + 1 = 0” or “distance from Earth to Mars now” and returns structured, computed results.

Wolfram|Alpha as an MCP server

An MCP server offers tools that a model can call. In February 2026, Wolfram made its technology available as a “foundation tool” for AI systems, including an official MCP service. Wolfram calls the idea computation-augmented generation (CAG): where RAG adds retrieved text to the prompt, CAG adds computed results.

The Wolfram Cloud MCP server lives at https://agenttools.wolfram.com/mcp. It uses the standard streamable HTTP transport, needs no API key, and is free for limited personal use. It offers three tools:

Tool	What it does	Good for
WolframAlpha	Sends a plain-language query to Wolfram	Alpha
WolframLanguageEvaluator	Runs Wolfram Language code in a cloud kernel (60-second default limit)	Multi-step calculations, custom formulas, plots
WolframContext	Semantic search over Wolfram’s documentation and knowledge	Helping the model write correct queries and code

All three are marked read-only. The cloud service is stateless: each call stands alone, with no saved variables between calls and no file uploads.

There are two other routes. Wolfram Local MCP runs on your own server with Mathematica, Wolfram|One or a licensed Wolfram Engine, and adds persistent sessions, local files and notebook tools. And several community MCP servers wrap the Wolfram|Alpha LLM API , which needs a free App ID from the Wolfram developer portal (a free non-commercial plan is available).

What a tool call looks like

A user asks: “What is the integral of x² sin(x), and what is it from 0 to π?”

The model sees the WolframAlpha tool and decides to use it.
It calls the tool with the query integrate x^2 sin(x) dx.
Wolfram|Alpha computes and returns the answer (this is the actual response):

# Indefinite integral
∫ x^2 sin(x) dx = 2 x sin(x) - (x^2 - 2) cos(x) + constant

# Definite integral
∫_0^π x^2 sin(x) dx = π^2 - 4 ≈ 5.8696

The model writes the reply around this exact result, and can add a link to the full Wolfram|Alpha page so the user can check it.

The model does the talking; Wolfram|Alpha does the math.

The math advantages

Language models break numbers into tokens, chunks of characters, and predict them one at a time. Long multiplication, carrying digits and keeping many steps exact are exactly where that fails. Handing the math to Wolfram|Alpha gives you:

Exact arithmetic. Big integers, fractions and high-precision decimals, computed rather than estimated. 2¹⁰⁰ is 1267650600228229401496703205376, every digit correct.
Symbolic math. Simplify, factor, expand, differentiate and integrate expressions, with exact forms like π² − 4 instead of a rounded decimal.
Equation solving. Polynomial, linear and differential equations and systems of equations, including complex roots and exact radicals.
Units and constants. Unit-aware calculations and conversions, plus physical constants such as the speed of light, without the model typing in values from memory.
Statistics and number theory. Means, regressions, distributions, primes, factorizations, modular arithmetic.
Plots. Graphs of functions and data, returned as images the app can show.
Results you can check. Every answer can come with a link to the full Wolfram|Alpha result, so a person can verify it.

Beyond math

Curated, up-to-date data. Chemical properties, planet positions, country statistics and exchange rates come from Wolfram’s knowledgebase, not from what the model happened to read during training.
Fewer hallucinated facts. When a number comes from a tool, the model has no need to invent one.
Small models with strong math. A compact model running on a CPU will never compute as well as Wolfram|Alpha, and it doesn’t have to. It only needs to recognize when to call the tool. This makes a small private model far more capable without bigger hardware. For more on that trade-off, see running LLMs on CPUs.

Limits and things to know

The question leaves your network. The query text goes to Wolfram’s cloud. Keep confidential details out of math queries, or use Wolfram Local MCP.
Internet access is required for the cloud service, and it is meant for small-scale use. For heavy or commercial use, look at Wolfram’s paid APIs.
The model must support tool calling, and has to choose to call the tool. A model that answers math on its own will still guess.
English queries work best. The model should translate the question for the tool and answer in the user’s language.
Code tools need care. WolframLanguageEvaluator runs code in Wolfram’s cloud. It is read-only for your systems, but some apps let you require approval for it.

Using it in pAIring

pAIring is an MCP host, so connecting Wolfram|Alpha takes a minute:

Open Admin settings > MCP servers and add a server named Wolfram with the URL https://agenttools.wolfram.com/mcp. No API key or headers are needed.
Raise the call timeout to about 60 seconds, since Wolfram Language evaluations can take that long.
Test the connection, then make sure MCP is on under Admin settings > System.
In a chat, turn tools on and ask a math question.

With a tool-capable model on Matricxon or llama.cpp, your chats stay on your own hardware, and only the math and fact queries go out to Wolfram.

Originally published at bytepenny.com.

CPU vs GPU

BytePenny — Thu, 15 Oct 2026 00:00:00 +0000

“You need a GPU for AI” is half true. A graphics card makes some parts of running a language model much faster, and others barely at all. Knowing which is which saves money, and tells you whether the computer you already have is enough.

Both are limited by memory, not maths

When a model writes a reply, every new token means reading nearly all of its weights once. So the speed of writing is set mostly by how fast the chip can read its memory:

Hardware	Memory speed (roughly)
Laptop or desktop, two DDR4/DDR5 channels	40 to 90 GB/s
Workstation or server, many channels	200 to 500 GB/s
Apple Silicon (unified memory)	100 to 800 GB/s
Consumer graphics card	300 to 1,000+ GB/s

That is why a GPU often writes 5 to 15 times faster than a desktop CPU with the same model. It is not that it thinks harder; it reads faster.

The catch: the model has to fit

A GPU is only fast while the whole model, plus the chat’s memory (the KV cache), fits in its own memory (VRAM). Consumer cards have 8 to 24 GB. A 4-bit 8B model (about 5 GB) fits easily. A 4-bit 70B model (about 40 GB) does not fit on any consumer card, but runs on a computer with 64 GB of ordinary RAM.

Splitting a model between the GPU and the CPU works, but helps less than you might expect: every token waits for the slowest part, which is the half that sits in system memory.

Where the GPU really shines

Reading the prompt. Before the first word of a reply, the model reads your whole message, and with a knowledge base, the documents found for it. This part is pure arithmetic, and a GPU can be 10 to 100 times faster. Long prompts are where a CPU feels slow.
Many users at once. A GPU can handle several conversations in the same pass over the weights. A CPU mostly takes turns.
Large models at speed. If you need a 30B to 70B model to answer quickly, you need serious GPU memory, or several cards.

Where the CPU holds its own

Capacity. 64 GB of RAM costs a fraction of 64 GB of VRAM. Large models fit; they are just slower.
Cost and what you already own. Every computer has a CPU. For one person, or a small team asking short questions, a well-chosen 3B to 8B model on a modern CPU is often fast enough.
Mixture-of-experts models. They read only a small part of their weights for each token, so they run far faster on a CPU than their size suggests.
Simplicity, power and noise. No drivers, no fans at full speed, lower power use, and it runs on any physical or virtual server.

Privacy is the same on both: a model on your own hardware keeps your data on your own hardware, whatever chip runs it.

How to choose

Your situation	A good fit
One user, short questions, a recent laptop	CPU, a 3B to 8B model at 4-bit
A small team, long documents or a knowledge base	A GPU with 12 to 24 GB, or a fast multi-channel CPU
Large models, but speed is not critical	CPU with plenty of RAM, or a mixture-of-experts model
Large models, fast, many users	One or more GPUs with enough memory for the whole model
Occasional need for a very large model	A cloud model for those cases only

What Matricxon and pAIring do

Matricxon runs on both CPUs and GPUs. On NVIDIA cards (MATRICXON_DEVICE=cuda, in BETA) weights are either unpacked for speed or kept compressed to save VRAM. On CPUs, a lot of fine-tuning went into making it work well: it multiplies compressed weights directly, keeps chats cached so a prompt is rarely read twice, and runs mixture-of-experts models efficiently. pAIring works with Matricxon, llama.cpp (with a GPU build it uses your GPU) and a growing list of cloud connectors, so you can start on the hardware you have, add a GPU server later, and switch engines with one setting.

The short answer: start with your CPU. Add a GPU when replies get too slow, usually because of long prompts, many users at once or a large model.

Originally published at bytepenny.com.