DEV Community: GINIGEN AI

The Inference Hardware You Already Own Might Train Models Too: A 325M Pretraining Run on an NPU

GINIGEN AI — Mon, 26 Oct 2026 05:53:38 +0000

Disclosure: the work described here is by VIDRAFT, an affiliate of Ginigen AI.

TL;DR

Edge and on-premise deployments usually split hardware into two worlds: GPUs to train, cheaper accelerators to serve. A new result from VIDRAFT narrows that split. They pretrained a 325M parameter language model from random initialization on the Rebellions ATOM-Max, a commercial inference NPU released in 2025, and trained the same model on a B200 GPU for comparison. On text neither model saw in training, the losses are 0.7993 (NPU) and 0.7994 (GPU).

Why should edge and on-prem teams care?

Most organizations running AI on premise already have inference accelerators in the rack. They buy them because they are cheaper per query and draw less power than training GPUs. When a model needs adapting to local data, the usual answer is to rent GPU time or ship data out.

If the inference hardware can train, that changes the economics and the data story. Adaptation and even pretraining can happen on hardware the team already owns, inside its own network.

What was actually shown?

Item	Value
Model	VKORE-325M-NPU-Base, 325.6M parameters
Start	random initialization, no pretrained weights
Data	5.0B byte tokens of FineWeb-Edu
Run	76,292 steps, about four days, no divergence
Hardware	Rebellions ATOM-Max NPU (2025)
Reference	the identical model on a B200 GPU

The two training curves overlap for the whole run, and both runs follow the same learning-rate schedule.

Is the NPU-trained model really as good?

On 1,024 windows from a FineWeb-Edu shard that was never in the training corpus:

Model	Loss	Bits per byte
Rebellions ATOM-Max NPU	0.7993	1.1531
B200 GPU reference	0.7994	1.1533

Window by window, the difference is -0.0001 with a standard error of 0.0007. That is a tie.

How far has this line come?

What is the catch?

Two honest limits. First, the hardware alone does not do this: the ATOM-Max and its standard software are built for inference, and training depends on VIDRAFT's own training stack and stability technique, which are under patent application. Buying the chip does not by itself enable training. Second, 325M on 5B tokens is a research-scale base model, not a production assistant. The result is about what the hardware class can do.

FAQ

Which NPU was used? The Rebellions ATOM-Max, a commercial inference NPU released in 2025.

Can I verify the numbers? Yes. Both weights and evaluate_unseen.py are in FINAL-Bench/VKORE-325M-NPU-Base.

Does this mean GPUs are unnecessary for training? No. It means inference NPUs can carry real training runs when the software exists. Scale, speed and cost per run are separate questions.

What does it mean for on-device AI? Models can be adapted close to where they run, on hardware already deployed, without moving data to a GPU cloud.

Chitos on the OWASP Benchmark: 77.8 Youden, on-prem

GINIGEN AI — Fri, 23 Oct 2026 09:23:19 +0000

The OWASP Benchmark is an official evaluation set that scores a security tool's accuracy across about 2,700 test cases as a Youden index from 0 to 100. Here is where Chitos lands, self-measured.

Configuration	Youden
Base code scanner alone	52.8
Chitos, online judgment engine (Claude Opus 5.5)	77.2
Chitos, on-prem judgment engine (Darwin-180B-RSI)	77.8

Reading the jump

The gap from 52.8 to 77 is the AI verification layer. It removes more than 60 percent of false positives while missing none of the real vulnerabilities. A Youden index weights sensitivity and specificity together, so this is not a precision trick that quietly drops true positives; it is cleaner findings at the same recall.

Why the on-prem row matters most

The online engine and the on-prem engine scored effectively the same. For regulated environments, that is the whole story: you can keep code and findings inside your network and still operate in the commercial upper range.

Chitos is non-destructive, autonomous in finding its own audit points, and refreshes vulnerability fingerprints daily. These numbers are self-measured, with cross-validation through the official scoring tool in progress.

Details: https://vidraft.net

0.1 Bits Per Weight Is Not 0.1 Bits Per Weight: Where the Bytes Go in Sub-2-Bit LLMs

GINIGEN AI — Wed, 21 Oct 2026 16:06:11 +0000

TL;DR

Recent coverage of Samsung Research's LittleBit repeats its headline: Llama2-13B at 0.1 bits per weight, under 1 GB. The paper (arXiv 2506.13771, NeurIPS 2025) supports the number, but the number is a property of the transformer blocks only. Three things follow when you account for the whole file:

The file is not 0.1 bits per parameter. The embedding table and lm_head stay in FP16. By our reconstruction they are about 80% of the 0.84 GB Llama2-13B file and about 83% of the 0.63 GB Llama2-7B file. File-level bits per parameter are 0.52 (13B) and 0.75 (7B), not 0.10.
The quality cost is real and was measured by the authors. At 0.1 BPW, WikiText-2 perplexity on Llama2-7B is 15.92 against 5.47 for FP16, about 2.9x. At 0.3 BPW several zero-shot tasks sit at chance level.
The 11.6x speedup is one MLP layer on an A100, from a custom kernel the authors say is not yet at the level of industry libraries. It is not a phone benchmark.

This post is arithmetic on published numbers and file sizes. We did not run LittleBit. Every figure is either quoted from the paper, read from Hugging Face metadata on 2026-10-22, or computed by us with the script shown below, and we say which.

What does "0.1 bits per weight" actually mean?

LittleBit replaces each linear layer's weight matrix W (shape d_out x d_in) with a low-rank factorization whose two factors are binarized to +1 or -1. Scales in FP16 compensate for the lost magnitude. The paper stores two parallel paths, a primary one and a residual one, and counts bits per layer like this (the paper's Appendix D, Eq. 16):

bits = 2r(d_out + d_in) + 32(d_out + d_in) + 32r
b    = bits / (d_out * d_in)          # effective bits per weight of that layer

The first term is the sign bits of the two factors on two paths. The second and third are the FP16 row, column and latent scales on two paths. To hit a target b, the authors solve for the latent rank r (Eq. 17):

def rank(b, d_out, d_in):
    return (b * d_out * d_in - 32 * (d_out + d_in)) / (2 * (d_out + d_in) + 32)

rank(0.10, 4096, 4096)    # 86.2   -> a rank-86 factorization of a 4096 x 4096 matrix
rank(0.10, 11008, 4096)   # 133.1  -> Llama-2 7B gate/up projection
rank(0.30, 4096, 4096)    # 290.6
rank(0.55, 4096, 4096)    # 546.1

So "0.1 bits per weight" for Llama-2 7B means each attention projection is approximated by two paths of rank about 86, which is 2.1% of the full 4096 rank. We checked that plugging the rounded rank back into Eq. 16 returns 0.0998. The paper's KV-cache table (Table 4) lists a latent rank of 192 for 0.1 BPW and 1,624 for 0.8 BPW, roughly twice our per-path values, which looks like the two paths' ranks added together. We did not confirm that reading with the authors.

This is also why the method is quantization-aware training and not a drop-in post-training step. The factors are initialized from an SVD (the paper calls it Dual-SVID) and then trained with a distillation loss, using WikiText-2 plus C4 partitions, sequence length 2048 and 5 epochs. The compute cost of that training is not stated in the parts of the paper we read.

Why is a 0.1 BPW 13B model 0.84 GB and not 0.16 GB?

Because quantization is applied to the transformer blocks only. The paper says so itself in the caption of Table 3: "components like the embedding layer and the lm_head remain in FP16."

Llama-2 has a 32,000-token vocabulary. The embedding table and the output head each hold 32000 x d_model parameters, and neither is shrunk. At FP16 that is a fixed cost no matter how hard the blocks are squeezed. We rebuilt the file size from the public architecture shapes (layer count, hidden size, FFN size, KV width):

Model	Block params	FP16 embed + head	Blocks at 0.1 BPW	Our estimate	Paper Table 3	Embedding share of file
Llama2-7B	6.48B	0.524 GB	0.081 GB	0.605 GB	0.63 GB	83%
Llama2-13B	12.69B	0.655 GB	0.159 GB	0.814 GB	0.84 GB	78%
Llama2-70B	68.45B	1.049 GB	0.856 GB	1.904 GB	1.98 GB	53%

Our estimates land 3% to 4% below the paper's sizes. We did not chase the remainder, which is plausibly rank rounding, norm weights and file overhead. The same reconstruction also tracks the paper at the other settings for 13B: 1.53 GB estimated against 1.51 GB at 0.55 BPW, and 1.13 GB against 1.15 GB at 0.3 BPW.

Two practical consequences for an on-device team:

Below roughly 0.3 BPW, the next saving is not in the blocks. At 0.1 BPW on 13B, going all the way to zero bits in the blocks would still leave 0.655 GB of embeddings. Quantizing the embedding table and lm_head is the remaining lever, and vocabulary size matters. A model with a 150,000-token vocabulary pays about 4.7x this embedding cost at the same hidden size.
Compression factors flatter large models. The headline 31x for 13B and 69.7x for 70B are file-size ratios against FP16. They grow with model size because the fixed embedding cost is a smaller fraction of a bigger file.

How do other sub-2-bit releases look at the file level?

"Bits per weight" is reported inconsistently across projects: sometimes nominal for the quantized tensors, sometimes for the whole file. The honest comparison metric for deployment is the same one you pay for in RAM: file bytes times 8, divided by total parameters. Here it is for LittleBit's published sizes next to three GGUF files from PrismML's Bonsai family on Hugging Face (Apache-2.0, based on Qwen3.6-27B and Qwen3.8-27B), measured the same way:

File	Size	File-level bits per parameter
LittleBit Llama2-70B, 0.1 nominal	1.98 GB	0.23
LittleBit Llama2-13B, 0.1 nominal	0.84 GB	0.52
LittleBit Llama2-7B, 0.1 nominal	0.63 GB	0.75
`Bonsai-27B-Q1_0.gguf`	3.80 GB	1.13
`Ternary-Bonsai-2-27B-PTQ1_0.gguf`	5.95 GB	1.77
`Ternary-Bonsai-27B-Q2_0.gguf`	7.17 GB	2.13

The Bonsai rows use the GGUF metadata total of 26,895,998,464 parameters and the file sizes from the Hugging Face API. The LittleBit rows use 16 x size / FP16 size from the paper's Table 3. Here is the script for the GGUF side, which works for any GGUF repository that reports a parameter total:

import json, urllib.request

def file_bits_per_param(repo, filename):
    url = f"https://huggingface.co/api/models/{repo}?blobs=true"
    m = json.load(urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "x"})))
    size = {s["rfilename"]: s["size"] for s in m["siblings"]}[filename]
    return size * 8 / m["gguf"]["total"]

file_bits_per_param("prism-ml/Bonsai-27B-gguf", "Bonsai-27B-Q1_0.gguf")  # 1.131

This table is a size comparison and nothing more. The rows are different models, different base architectures, different training procedures and different benchmarks, so it says nothing about which one is better. We have not inspected the Bonsai tensor layouts, and we do not know which of their tensors, if any, are kept at higher precision. We also have not tested whether any of these files runs well on a specific phone.

What does the quality cost?

The authors report WikiText-2 perplexity across model families, with STBLLM, a sub-1-bit baseline, as the comparison. For Llama2-7B (paper Table 1), LittleBit keeps perplexity within a factor of three of FP16 all the way down to 0.1 BPW, while STBLLM collapses at 0.3 BPW:

Llama2-7B WikiText-2 perplexity	FP16	0.55 BPW	0.3 BPW	0.1 BPW
LittleBit	5.47	10.47 (1.9x)	12.00 (2.2x)	15.92 (2.9x)
STBLLM	5.47	30.67	about 1,800	not reported

The multiplier grows with model difficulty. At 0.1 BPW the table gives 15.09 against 4.88 for Llama2-13B (3.1x), 26.11 against 6.10 for Llama3-8B (4.3x) and 35.26 against 6.34 for QwQ-32B (5.6x). The authors themselves write that "a quantization cliff appears between 0.3 and 0.1 BPW" and recommend 0.3 to 0.55 BPW as the sweet spot.

Perplexity is a gentle metric. Zero-shot accuracy is harsher (paper Table 2, Llama2-7B, average over seven tasks):

FP16 averages 62.97%. LittleBit at 0.55 BPW averages 47.26% and at 0.3 BPW 45.20%. Look at the individual tasks at 0.3 BPW and the picture is stark: WinoGrande, a two-choice task, scores 51.30% against a chance level of 50%, and ARC-Challenge, a four-choice task, scores 25.09% against a chance level of 25%. Tasks that need broader knowledge or reasoning are close to guessing. PIQA, at 65.83% against 78.12% for FP16, and BoolQ, at 61.80% against 71.37%, retain more.

Two caveats on how to read these numbers. First, the training data includes WikiText-2 (presumably its training split), and WikiText-2 perplexity is the headline metric, so the perplexity figures are partly in-distribution for the training run. The paper says it follows the training protocol of an earlier binarization paper, but the setup still flatters perplexity compared with a held-out domain. Second, the zero-shot tasks are common-sense multiple choice benchmarks. They tell you nothing about long-context behavior, tool calling or code, which are what most on-device assistants need.

What is the 11.6x speedup, and what is it not?

The paper reports a custom 1-bit GEMV CUDA kernel running a single Llama2-70B MLP layer (8,192 x 28,672) on an NVIDIA A100, peaking at 11.6x over an optimized FP16 baseline at 0.1 BPW. That is a kernel micro-benchmark on a data-center GPU.

The authors add that "the discrepancy between theoretical computational gains and practical latency is likely attributable to memory access dominance during small-batch inference" and that the kernel "has not yet reached the optimization level of industry-standard libraries." They list hardware co-design for NPUs as future work. So there is no published number for end-to-end tokens per second on a phone, and you should not infer one from 11.6x.

The authors also publish a code repository, SamsungLabs/LittleBit, whose description says it contains the NeurIPS 2025 implementation and the follow-up LittleBit-2. That follow-up, arXiv 2603.00042, says its geometric preconditioning "establishes a new state-of-the-art in the sub-1-bit regime" from 1 down to 0.1 bits per parameter and matches the fidelity of leading 1-bit baselines. We have not read its tables, so we make no claim about its numbers here.

When does this matter for an edge deployment?

Our reading, which is an opinion and not a measurement:

If your budget is a few hundred megabytes of weights, the 0.3 to 0.55 BPW range is where the authors' own data suggests the trade is defensible. It pairs with a use case that tolerates a large quality gap, such as a classifier, a router or a drafter, and not with open-ended chat.
If your target is a phone with 8 to 12 GB of RAM, a 4-bit or 2-bit file of a stronger model is likely the better starting point, because the quality gap at 0.1 BPW is large and the runtime stack for factorized binary weights is not mature. Measure on your own tasks before choosing.
If you are comparing quantization headlines, convert every claim to file-level bits per parameter first. It takes two lines of Python, and it turns marketing numbers into numbers that predict your RAM bill.

What are the limits of this analysis?

We did not run LittleBit or any Bonsai model. Quality numbers are the authors' own, from the arXiv HTML of the paper (v5, February 2026).
Our file-size reconstruction uses public Llama-2 shapes and assumes every block linear layer sits exactly at the nominal rate. It is within 4% of the paper, and the gap is unexplained.
Table 1 mixes Llama (first generation) and Llama2 columns. We used the Llama2-7B and Llama2-13B columns, which the paper's own text confirms for the 12.00 and 15.92 figures.
The Bonsai comparison is file size only. We did not verify press figures about those models beyond what Hugging Face metadata shows.
The paper's license is CC BY-NC-ND 4.0, so we summarize and cite and do not reproduce its tables beyond the figures quoted above.

FAQ

What does 0.1 bits per weight mean for LittleBit?
It is the average cost per weight of the quantized transformer blocks, counting sign bits and FP16 scales, after replacing each matrix with two rank-r binary factorizations. It does not include the embedding table and output head, which stay in FP16.

How big is Llama2-13B with LittleBit at 0.1 BPW?
0.84 GB according to the paper, against 26.06 GB in FP16. That is 31x smaller, and about 0.52 bits per parameter for the whole file.

Does LittleBit need training?
Yes. It is quantization-aware training with distillation, initialized from an SVD of the original weights, using WikiText-2 and C4 data for 5 epochs at sequence length 2048 in the authors' setup. It is not a post-training step you can run in minutes.

How much accuracy is lost at 0.1 bits per weight?
Perplexity on Llama2-7B rises from 5.47 to 15.92. At 0.3 BPW, zero-shot WinoGrande and ARC-Challenge for Llama2-7B are at chance level. The authors call 0.3 to 0.55 BPW the sweet spot.

Can I run it on a phone today?
We found no published phone benchmark. The reported 11.6x speedup is a single-layer kernel on an A100 GPU, and the authors state that the kernel is not yet at the level of industry-standard libraries.

Put the Decision Model Inside the Agent Loop, Not Beside It

GINIGEN AI — Tue, 20 Oct 2026 01:26:40 +0000

The cost nobody budgets for

Build an agent and count where the tokens go. A surprising share is not the work. It is the agent asking itself small questions.

Is this tool call safe to run. Does this document answer the question. Should this go to a human. Is this output in the right format. Did the last step actually succeed.

Each one is a yes, a pick, or a score. Each one currently costs a full generation: prompt in, reasoning out, parse the text, hope the format held. At a thousand of those a minute, the checking costs more than the task.

So teams stop checking. They drop the safety gate, skip the retrieval relevance test, let the agent run unguarded. Not because they decided the checks were unnecessary, but because the checks were too expensive to keep.

A different shape of model

A decision model answers without writing. Given the situation and a typed question, it returns a probability over the options in one forward pass.

Three shapes cover nearly everything an agent asks itself:

Shape	Returns	In the loop
yes or no	Probability the statement holds	Safety gate, retry decision, escalation
pick one	A probability per option	Routing, tool selection, triage
score	A probability per level	Relevance, risk, confidence

Nothing to parse. No format drift. No reasoning tokens to wait through.

Why calibration is the whole point

Accuracy alone does not let you build a gate. A gate needs a threshold, and a threshold needs probabilities that mean what they say.

If the model says 0.9 and is right nine times in ten, you can route everything above 0.9 automatically and send the rest to a human, and you can predict your error rate before shipping. If the model is accurate on average but its 0.9 means anything from 0.6 to 0.99, no threshold is safe and you are back to reading the output yourself.

This is why decision benchmarks score calibration, not just hit rate. Yes or no questions use Brier score. Choice questions score the probability mass on the correct option. Scales use expected score error. A model can top an accuracy table and still be useless as a gate.

What it changes in practice

Once checking is cheap you can check things you previously could not justify:

Gate every tool call instead of the risky-looking ones
Score every retrieved passage for relevance instead of trusting the top k
Ask after every step whether the step succeeded, instead of discovering failure at the end
Keep a confidence number on every decision so that the escalation rule is a threshold, not a guess

The architecture shift is small. The decision model sits inside the loop as a function call rather than beside it as another agent.

Reading the public boards

Decision models are now measured on public leaderboards such as S1MB, which currently covers 197 models across 137 benchmarks and ranks by aggregating ranks rather than averaging scores.

When comparing entries, read which column a claim quotes, the date it was read, and how many of the benchmarks the model was actually scored on. Most public boards mix self-reported and verified numbers.

Open-weight options exist in this category, so the approach is testable on your own traffic before it is a procurement decision. That is the right way to evaluate it: run your own decision log through a candidate and see whether the probabilities hold up at the threshold you intend to use.

A current reference point

If you want a concrete model to test this with, the S1MB english-v1 board currently lists Darwin-27B-ZTC-v9 first on Borda score at 90.38 across 137 benchmarks and 197 models, read on 20 October 2026. It is Apache-2.0, so you can run your own decision log through it rather than taking the board's word for it.

hf download FINAL-Bench/Darwin-27B-ZTC-v9 systemone_config.json --local-dir .
transformers serve FINAL-Bench/Darwin-27B-ZTC-v9 --decision-config systemone_config.json

The same board publishes a task-average column where that model is second at 72.20. Quote the column you mean.

LFM2.5-24B-A2B on Your Own Hardware: The RAM, KV Cache and Bandwidth Budget

GINIGEN AI — Mon, 19 Oct 2026 14:46:56 +0000

TL;DR

Liquid AI released LFM2.5-24B-A2B on Hugging Face this month: 23.97B total parameters, 2.45B active per token, 131,072-token context, GGUF on day one. For on-device use, three numbers decide whether it fits your hardware, and all three can be computed from config.json and the GGUF file list instead of taken from a launch post:

RAM floor: about 14.5 GB for the Q4_K_M weights (4.85 bits per weight). A 24B MoE costs 24B of memory even though it computes like a 2B model.
KV cache: 20 KiB per token in FP16, because only 10 of 40 layers use attention. A full 128K context is 2.5 GiB, not the 10 GiB a plain transformer of this width would need.
Decode ceiling: about 67 tokens per second per 100 GB/s of memory bandwidth at Q4_K_M, since each token streams roughly 1.5 GB of weights.

This is a budget calculation, not a benchmark. We did not run the model for this post. Everything below is derived from published files, and the one speed claim from the vendor is marked as theirs.

Why does a 24B model with 2.45B active parameters still need 14.5 GB of RAM?

Mixture-of-experts models split compute from storage. The router picks 4 of 64 experts per layer for each token, so compute looks like a 2B model. But the router can pick any expert on the next token, so every expert has to be resident, or paged in from fast storage, to keep decoding smooth.

The published files make this concrete. These are the GGUF sizes from the LiquidAI/LFM2.5-24B-A2B-GGUF repository, with bits per weight computed against the 23,971,587,712 parameters in the GGUF metadata:

Quant	File size	Bits per weight	Active weight bytes per token
QAD-Q4_0	13.58 GB	4.53	1.39 GB
Q4_K_M	14.53 GB	4.85	1.48 GB
Q5_K_M	17.03 GB	5.68	1.74 GB
Q6_K	19.69 GB	6.57	2.01 GB
Q8_0	25.49 GB	8.51	2.61 GB
BF16	47.96 GB	16.01	4.90 GB

The last column multiplies the 2.45B active parameters by the bits per weight. It is an approximation, because K-quants mix precisions per tensor, but it is close enough for sizing.

Two things stand out. First, QAD-Q4_0 is the smallest 4-bit file. QAD in the file name points to quantization-aware distillation, which, if it works as the name suggests, would make it a better 4-bit default than a post-hoc K-quant, but the repository README is short and we did not find an accuracy table for it. Treat that as a hypothesis to test on your own task, not a claim. Second, going from Q4_K_M to Q8_0 costs 11 GB more RAM for a 1.76x increase in bits per weight. On a 16 GB machine that difference decides whether the model loads at all.

Where do the 2.45B active parameters actually go?

You can reconstruct the active count from the config, which is a good habit before trusting any "A2B" label. From config.json: hidden size 2048, MoE intermediate size 1536, 64 experts, top-4 routing, 40 layers of which 2 are dense, so 38 MoE layers.

Assuming a SwiGLU expert has three projection matrices (gate, up, down):

h, moe_inter, experts, top_k, moe_layers = 2048, 1536, 64, 4, 38
per_expert = 3 * h * moe_inter                      # 9,437,184 params
all_experts = per_expert * experts * moe_layers     # 22.95 B
active_experts = per_expert * top_k * moe_layers    # 1.43 B
total = 23_971_587_712
always_on = total - all_experts                     # 1.02 B
print(always_on + active_experts)                   # 2.4548 B

The result, 2.4548B, matches the model card's 2.45B. That also tells us how the active budget splits: about 1.43B (58%) are the four selected experts, and about 1.02B (42%) are always-on weights: the attention and convolution blocks, the two dense feed-forward layers, and the tied 128,000-token embedding and output matrix.

Why this matters on-device: the always-on 1.02B is read for every single token no matter what the router does. If you ever offload experts to slower storage, the always-on part is what you pin in fast memory.

How big is the KV cache when 30 of 40 layers are convolutions?

This is the quiet edge-AI feature of the architecture. The config's layer_types list shows a repeating pattern of three conv layers followed by one full_attention layer, 30 convolution and 10 attention layers in total. Only attention layers keep a per-token KV cache. The convolution layers carry a small fixed-size state (conv_L_cache is 3), which does not grow with context.

With 8 KV heads and a head dimension of 64 (hidden size 2048 divided by 32 attention heads; the config has no explicit head_dim, so this is our assumption), the FP16 cost per token is:

2 (K and V) x 10 attention layers x 8 KV heads x 64 dims x 2 bytes = 20,480 bytes

Context	Hybrid, 10 attention layers	If all 40 layers were attention
8K	0.16 GiB	0.62 GiB
32K	0.62 GiB	2.50 GiB
64K	1.25 GiB	5.00 GiB
128K	2.50 GiB	10.00 GiB

The right-hand column is hypothetical, a same-width model with attention in every layer, shown only to isolate what the hybrid design saves. At 128K the saving is 7.5 GiB, which is larger than the entire RAM headroom of many laptops. With -ctk q8_0 -ctv q8_0 in llama.cpp the cache shrinks further (about 1.33 GiB at 128K using 1.0625 bytes per element), though a quantized V cache generally requires flash attention to be enabled, so check your build.

The convolution state is negligible: roughly 30 layers x 2048 channels x 3 positions x 2 bytes is about 0.35 MiB.

Will it fit on my machine?

Weights plus FP16 KV cache for Q4_K_M, before the operating system, the runtime's compute buffers, and anything else you have open:

Device RAM	Context	Weights + KV	Headroom
16 GiB	32K	14.15 GiB	1.85 GiB
16 GiB	128K	16.03 GiB	-0.03 GiB
24 GiB	32K	14.15 GiB	9.85 GiB
24 GiB	128K	16.03 GiB	7.97 GiB
32 GiB	128K	16.03 GiB	15.97 GiB

The honest reading: on a 16 GiB machine this model is borderline even at 32K context, because 1.85 GiB of headroom has to cover the OS, the compute graph, and your application. In practice that means 24 GiB is the comfortable minimum for Q4_K_M at long context, and 16 GiB is a "close everything else" configuration, or a case for mmap with some paging. Phones with 8 to 12 GiB cannot hold these weights at all, which is why the smaller LFM2.5 siblings exist for that tier.

What speed should I expect? A bandwidth ceiling, not a promise

Autoregressive decoding is bandwidth-bound: each new token must read every active weight from memory once. The ceiling is simply memory bandwidth divided by active bytes per token. For Q4_K_M that is 1.48 GB:

active_bytes = 2.45e9 * 4.85 / 8          # ~1.48e9 bytes per token
for bw_gbs in (50, 100, 200, 400):
    print(bw_gbs, round(bw_gbs * 1e9 / active_bytes, 1))
# 50 -> 33.7, 100 -> 67.4, 200 -> 134.7, 400 -> 269.4 tokens/s

The bandwidth tiers are illustrative round numbers, not specific devices. Real throughput lands below the ceiling because of compute, KV reads, expert routing overhead, and cache effects, so treat these as upper bounds you can sanity-check measurements against.

The vendor reports 184 tokens per second on an Apple M5 Max. We could not confirm which quantization was used for that figure from the model card. If it was a 4-bit-class format, 184 tokens per second times roughly 1.4 to 1.5 GB per token implies something like 255 to 270 GB/s of effective weight traffic. That is a consistency check on the claim, not a measurement of ours, and it suggests the number is plausible only on a high-bandwidth unified-memory machine, not on a typical thin laptop.

Two caveats on the ceiling. It applies to decode only; during prompt processing many tokens are in flight, the router touches most experts, and the workload becomes compute-bound. And the model is a pure reasoning model: the chat template appends <think> to every assistant turn, so a visible answer is preceded by reasoning tokens. Cheap tokens are good, but you will generate many of them.

How do I run it with a tight memory budget?

A starting point with llama.cpp, using flags that exist in the current server documentation:

llama-server \
  -m LFM2.5-24B-A2B-Q4_K_M.gguf \
  -c 32768 \
  -ctk q8_0 -ctv q8_0 -fa on \
  --jinja

If the weights do not fit in GPU memory, --n-cpu-moe N keeps the expert weights of the first N layers in system RAM, and --cpu-moe keeps all of them there. That maps well to the parameter split above: the 1.02B always-on weights stay on the accelerator, and the 1.43B-per-token expert traffic comes from RAM. Because only 4 of 64 experts are read per layer per token, the traffic per token is far smaller than the 14.5 GB resident size suggests. Measure it, because routing locality varies by workload.

Generation settings from the model card: temperature 0.1, top_k 50, repetition penalty 1.05. Tool calls are emitted as Pythonic lists between <|tool_call_start|> and <|tool_call_end|> tokens unless the system prompt asks for JSON.

What are the limits?

We did not benchmark it. The numbers here are arithmetic on published files. Quality on your task needs your own evaluation.
The model card says it is not recommended for knowledge-heavy tasks alone and suggests pairing it with retrieval or web search. That fits the on-device agent role it targets.
The license is a custom lfm1.0 license, listed as "other" on Hugging Face, not Apache 2.0 or MIT. Read the LICENSE file before shipping it in a product.
Vendor benchmark claims are the vendor's. The LiveClaw score of 48.09 against 38.19 for Qwen3.6-35B-A3B comes from the model card, and we have not reproduced it.
Head dimension 64 and the three-matrix expert structure are inferred from config values, not stated explicitly. The reconstruction landing on 2.4548B against the stated 2.45B supports the second one.

FAQ

How much RAM does LFM2.5-24B-A2B need?
About 14.5 GB for the Q4_K_M GGUF weights plus roughly 0.6 GiB of FP16 KV cache at 32K context, so plan for 16 GiB at the very least and 24 GiB for comfort.

Why is a 2B-active model not 2B in memory?
Because the router can select any of the 64 experts for the next token. All 23.97B parameters must be resident, or quickly pageable, even though only 2.45B are read per token.

How big is the KV cache at 128K context?
2.5 GiB in FP16, from 20,480 bytes per token across 10 attention layers with 8 KV heads of dimension 64. About 1.33 GiB with a q8_0 cache.

What decode speed is realistic on a laptop?
The ceiling is about 67 tokens per second per 100 GB/s of memory bandwidth at Q4_K_M. Measured speed will be lower. Check your machine's memory bandwidth first.

Which GGUF should I pick?
Start with Q4_K_M, or try QAD-Q4_0 if you need 1 GB less and can test quality on your own prompts. Move up to Q5_K_M only if you have the RAM and see quality problems.

사라진 명리 비기를 AI로 복원하다: 주명명리(宙明命理) 개발기

GINIGEN AI — Mon, 12 Oct 2026 10:29:04 +0000

TL;DR

지니젠AI가 만든 주명명리(宙明命理) 는 AI로 복원한 한국식 사주 풀이 서비스입니다. 생년월일과 태어난 시간을 넣으면 정밀 만세력을 뽑고, 그 위에 성격, 재물, 직업, 인연, 건강, 대운, 올해 운세, 궁합까지 이야기로 풀어 줍니다. 모든 풀이는 "아주 매운 정곡 찌르기 → 명식 근거 → 공감 → 용기와 개운법" 4단 흐름으로 나옵니다. 무료로 바로 체험할 수 있습니다.

링크: https://ginigen-ai-saju.hf.space/

왜 만들었나

명리학의 깊은 해석은 소수의 숙련자에게만 남아 있습니다. 좋은 풀이는 수십 년 경험에서 나오고, 흩어진 구결과 비기는 접근하기 어렵습니다. 우리는 이 간극을 AI로 메우려 했습니다. 고전의 틀(맹파 구결, 난강망 조후, 적천수 중화, 신살 비전)을 교차검증하고, 현대의 언어로 일관되게 풀어 주는 시스템을 만드는 것이 목표였습니다.

핵심 질문은 하나였습니다. "사람이 사주를 볼 때 거치는 계산과 해석을, 틀리지 않게 자동화할 수 있는가."

만세력 엔진은 어떻게 동작하나

풀이의 토대는 정확한 만세력입니다. 여기서 틀리면 그 위의 모든 해석이 무너집니다. 그래서 엔진은 다음을 직접 계산합니다.

사주 네 기둥(년월일시)의 간지와 절기 기반 월주 산정
납음 60갑자, 지장간 삼기(여기, 중기, 정기)
12신살, 귀인과 흉살, 합충형회파해
삼기귀인(천상 갑무경, 지하 을병정, 인중 임계신) 같은 드문 귀격
대운(양남음녀 순행과 역행, 대운수 산정)

정확도 검증은 실제 전문 만세력 앱 결과와 수작업으로 대조하며 맞췄습니다. 같은 사주에는 같은 결과가 나오도록 명식에 고유 식별자를 부여해, 한 번 계산한 사주는 다시 쓸 수 있게 했습니다.

풀이는 어떻게 생성되나

좋은 운세 서비스의 체감은 문체에서 갈립니다. 밋밋한 교과서체는 아무도 읽지 않습니다. 우리는 풀이를 네 박자로 설계했습니다.

매운 정곡: 읽는 순간 "이거 완전 내 얘기네" 하도록 정확히 찌른다.
근거: 명식의 글자로 왜 그런 기질인지 납득시킨다.
공감: "그러느라 힘들었겠다"고 안아 준다.
용기와 개운: 그 기질을 강점으로 뒤집고, 오늘부터 할 실천 개운법을 쥐여 준다.

단정과 과장, 공포 조장은 금지입니다. 매운맛으로 시작하되 뒷맛은 따뜻하게 닫습니다. 독자는 수준을 고를 수 있습니다. 용어 없이 쉬운 길잡이, 용어와 근거를 함께 보는 탐구, 원전에 가까운 비전 세 단계입니다.

서비스 아키텍처

가볍고 재현 가능한 구조를 택했습니다.

백엔드는 FastAPI, 프런트는 정적 페이지 한 장
만세력 계산은 파이썬 라이브러리 기반, 해석은 LLM 백엔드를 추상화해 호출
같은 사주, 같은 장은 한 번만 생성하고 재사용(캐시)
풀이는 탭으로 묶어, 사용자가 연 탭의 내용만 그때 생성(지연 생성)

지연 생성은 중요한 선택이었습니다. 모든 장을 미리 만들면 비용과 지연이 큽니다. 사용자가 보는 탭만 만들면 평균 생성량이 크게 줄고, 같은 사주는 전 사용자가 한 번만 만든 결과를 공유합니다.

X에서 댓글로 사주 보기

서비스를 더 널리 퍼뜨리기 위해, X(옛 트위터)에 자동응답 봇을 붙였습니다. 사용자가 게시물에 "생년월일 더하기 태어난 시간 더하기 성별"을 댓글로 남기면, 봇이 그 자리에서 만세력 요약과 맛보기 풀이를 답글로 달아 줍니다.

흐름은 이렇습니다.

멘션을 주기적으로 확인
자유 서식의 한국어에서 생년월일과 시간, 성별을 파싱(예: "1990년 1월 1일 오후 2시 남자")
만세력과 맛보기 풀이 생성
글자 수 한도 안에서 요약과 링크를 붙여 답글 게시

생년월일 파싱은 생각보다 까다롭습니다. "여명"처럼 여성을 뜻하는 표현, "02시"처럼 시만 적힌 경우, "생시 모름"까지 받아내야 실제 댓글을 놓치지 않습니다.

배운 점

토대가 먼저다. 해석 품질을 올리기 전에 만세력이 100퍼센트 맞아야 한다. 계산 오류 하나가 모든 문장을 거짓으로 만든다.
문체가 제품이다. 같은 데이터라도 네 박자 흐름으로 풀면 체감이 완전히 달라진다.
지연 생성이 비용을 가른다. 사용자가 실제로 보는 것만 만들면 평균 비용이 떨어진다.
입력 파싱이 전파를 가른다. SNS의 자유 서식을 폭넓게 받아내지 못하면 좋은 엔진도 사용자를 놓친다.

FAQ

정확한가요? 만세력 계산은 전문 앱 결과와 대조해 맞췄습니다. 해석은 고전의 경향을 현대어로 푼 것으로, 재미와 통찰을 위한 참고입니다.

생시를 모르면 쓸 수 있나요? 네. 생시 모름을 선택하면 시주 관련 해석은 단정 없이 유연하게 풀어 줍니다.

비용이 드나요? 무료로 체험할 수 있습니다.

궁합도 되나요? 상대의 생년월일을 넣으면 두 명식을 포개어 인연의 결을 봅니다.

어떤 언어를 지원하나요? 현재 한국어 중심입니다.

지금 체험하기

링크에서 생년월일을 넣고 바로 확인할 수 있습니다.

https://ginigen-ai-saju.hf.space/

재미와 통찰을 위한 서비스입니다. 운명을 단정하지 않습니다.

Zero-Token Decisions in the Browser: On-Device Multimodal Classification with EmbeddingGemma 2 and WebGPU

GINIGEN AI — Mon, 12 Oct 2026 03:38:49 +0000

TL;DR

Decide Live is a static web page that turns a camera frame, a microphone window, an image or a line of text into a typed decision, entirely in the browser. It runs a fine-tuned EmbeddingGemma 2 (Apache-2.0) through transformers.js on WebGPU, generates zero tokens, and never uploads anything. No server, no per-decision API call, weights cached once.

Live demo: https://huggingface.co/spaces/FINAL-Bench/Decision-AI-Web

Why a decision should not be a generation

If a model needs to answer "is this egg fertile", "is this scene safe to proceed", or "which of these four states are we in", asking a chat model to write a sentence is the wrong tool. Generation is slow, non-deterministic, and every token is a chance to drift. A decision has a type: a class, a yes or no, a score. You want that type directly.

Decide Live produces it with an embedding model and a calibrated similarity, so the number of generated tokens is zero. This is the on-device form of what we call ZTC (Zero-Token Confidence).

How a decision is computed

Everything below is in the page and the model card, so you can reproduce it.

Query embedding. The media placeholders come first, then task: classification | query: {question}, and \nContext: {context} when a context is given. The model embeds the media and the text together in one pass.
Option embeddings. Each option is written as title: none | text: {description}, embedded once and cached.
Pooling. Mean pooling over the attention mask, then L2 normalization, giving a 768-dimensional unit vector.
Scoring. P(option) = softmax(scale * cosine(query, option)). The scale is learned, 19.96 (a temperature near 0.05), shipped in decision.json.

P(option_i) = softmax_i( 19.96 * cos(e_query, e_option_i) )

No decoding loop appears anywhere in that path. One forward pass of the encoder per card, then a dot product.

The on-device stack

transformers.js 4.3.1 loads the model in a Web Worker and returns embeddings.
WebGPU runs the ONNX graphs, with a WASM (CPU) fallback when WebGPU is absent.
The ONNX files are shipped in q4f16 and in a portable q4 build (4-bit weights with plain 8-bit gathers, so the same files also run on the WASM runtime, which lacks GatherBlockQuantized), plus a 4-bit audio encoder. fp16 and fp32 are omitted to fit the 1 GB Space limit.
Weights download once and are cached by the browser. After that, decisions are local.

What you can actually do with it

Live: embed camera frames and score up to four decision cards, each with its own question, in the trained one-pass-per-card format. Each card shows probability bars, the current decision, and a confidence gate.
Microphone or sound file: the same cards over 2 second windows at 16 kHz mono; windows quieter than -55 dBFS are skipped.
Single input: one image, text, audio clip or short video, plus a question and 2 to 8 options written as key: description, or a yes/no question.
Sample tests: 18 image tests with gold answers, run in the browser with an accuracy report and a comparison of prompt formats.

Why this matters for edge AI

A decision that runs on the device is private by construction (camera frames and audio never leave the machine), free per call, and available offline after the first load. Pair that with the zero-token form and you can score every frame or every audio window without a cloud bill or a latency round trip. That is the shape of practical edge intelligence.

FAQ

Does anything get uploaded?
No. Camera frames, audio, images and text are processed on the device. Only the model weights are downloaded, once.

What model is behind it?
A decision fine-tune of google/embeddinggemma-2 (Apache-2.0), served as ONNX and run with transformers.js on WebGPU or WASM.

Why zero tokens?
The decision is a softmax over cosine similarity between a query embedding and option embeddings. There is no text generation, so it is deterministic and cheap.

Do I need a GPU?
No. WebGPU is used when available for speed; otherwise it falls back to CPU (WASM), slower but working.

Built by VIDRAFT / FINAL-Bench. Apache-2.0. Try it, then read the page source to see the whole decision path.

Open Quantum Challenge (OQC): a reproducible, hardware-free quantum benchmark

GINIGEN AI — Sun, 11 Oct 2026 10:08:55 +0000

Run it on your own GPU. No quantum hardware required.

The Open Quantum Challenge is a vendor neutral, fully reproducible benchmark for quantum computing methods. You download a frozen problem, solve it with any method on your GPU, and submit a prediction file. A private answer key scores every entry the same way, so the ranking is reproducible and independent of hardware.

Two seasons, $1,000 prize each

Season 1 is Quantum Simulation. Report the ground state energy of Fermi Hubbard systems (chains, ladders, plaquettes). Scored by mean absolute error against exact diagonalization. Deadline 28 February 2027.

Season 2 is QEC Decoder. Decode a frozen rotated surface code dataset (distance 5, p = 0.005, circuit level noise). Scored by logical error rate. Deadline 31 March 2027.

Prize per season: first $600, second $300, third $100.

How to enter

Get the public inputs: FINAL-Bench/oqc-data on Hugging Face.
Solve on your GPU. A reference harness is at github.com/final-bench/quantum-bench-harness.
Sign in with Hugging Face on the challenge Space and submit your prediction JSON.

Your entry is auto scored within minutes and appears on the live leaderboard under your Hugging Face profile. Answers stay private, the harness and frozen problems are public, and the ladder is the same for everyone.

One Self-Improving Model, Eleven Number-One Titles: What That Takes

GINIGEN AI — Sun, 11 Oct 2026 02:52:12 +0000

TL;DR

One self-improving model family now holds eleven public number-one benchmark records at the same time, across math, science, law, structured output, and decisions. The interesting part is not any single score, it is that a recursive self-improvement loop, bound to external verification, can push a whole spread of benchmarks to the top at once.

The eleven, at a glance

Math: AIME 2026 100%, HMMT 2026 100%
Science: GPQA Diamond 94.44%
Knowledge and multimodal: MMLU-Pro 88.12%, MMMU-Pro 79.48%
Law: LEXam 68.94%, LEXam-hard 45.72%
Extraction and structure: ExtractBench 90.29%, IFStruct 98.95%
Decisions: MDPBench 83.65%, S1MB Borda 89.58

Why a self-improving loop gets here

A model that only trains once is stuck with the data it saw. A self-improving (RSI) model runs a loop: it attempts problems, keeps the attempts that pass an external check, and trains on those. The reward is tied to verification outside the model, such as code execution, external evidence, or an answer key, not to the model grading itself. That is what keeps the loop from drifting into its own confident mistakes.

When the loop is honest, improvement on one reasoning skill tends to carry to others, which is why the records span unrelated fields instead of one narrow test.

The decision layer

For typed decisions, the family uses a zero-token method: it reads the problem in one forward pass and applies a calibrated probe to produce the decision, generating no text. That is deterministic and cheap, which also makes it a good fit for running decisions close to where the data is.

Open to use

S1MB number one model: https://github.com/final-bench/s1mb
ZTC method: https://github.com/final-bench/ztc
On-device models: https://github.com/final-bench/pocket

FAQ

How many number-one records, and across what?
Eleven at once, across math, science, law, structured output, and decisions.

What is a self-improving (RSI) model?
A model that runs a loop: it solves problems, keeps the solutions that pass an external check, and trains on them, so it improves without new human labels.

Why tie the loop to external verification?
If a model grades its own work, the loop can reinforce confident errors. External checks keep improvement real.

Is it open?
Yes, the decision model and method are public under Apache-2.0.

4-bit GGUF Quality for MoE Models: Why Only 3B of 180B Params Fire, and How to Prove Parity

GINIGEN AI — Tue, 06 Oct 2026 19:10:26 +0000

TL;DR

Mixture-of-Experts (MoE) models look enormous on disk, but only a small slice of the weights does work on any single token. A 180B-parameter MoE can activate roughly 3B parameters per forward pass. That sparsity is exactly why 4-bit GGUF quantization behaves so differently on MoE than on a dense model, and why a mixed-precision scheme such as the UD-Q4_K_XL style graft can hold accuracy that a flat 4-bit cast would lose.

The honest part of the story is not the compression ratio. It is verification. A quantized file that loads and produces fluent text is not proof of anything. You prove parity by scoring the quantized model on a held-out benchmark and comparing it to the full-precision baseline, number against number. In the example below, a 4-bit mixed graft reaches MMLU-Pro 87.65, equal to the full-precision baseline inside noise. This post shows the reasoning and the exact llama.cpp commands to reproduce that check yourself.

Why does only a fraction of an MoE model activate per token?

A dense transformer runs every weight for every token. A Mixture-of-Experts transformer replaces the big feed-forward block in each layer with many smaller expert blocks plus a router. For each token the router picks a few experts (often two) and ignores the rest. The attention layers and the router still run every time, but the overwhelming majority of the feed-forward parameters sit idle on any given token.

The practical consequence: the "180B" headline counts total parameters, while the compute and the active memory traffic track the much smaller active set, on the order of 3B. Two numbers describe an MoE and they answer different questions:

Total parameters set the disk and RAM footprint you must store.
Active parameters per token set the arithmetic you actually perform.

Quantization touches storage, so it operates on the total. Quality, however, is decided token by token on the active path. This gap is the whole reason MoE quantization is its own topic.

Why does 4-bit hurt MoE differently than a dense model?

Two forces pull in opposite directions.

First, MoE is more forgiving in aggregate. Averaged over a corpus, errors introduced into rarely used experts barely move the output, because those experts rarely fire. A dense model has no such luxury: every weight is on the hot path for every token, so quantization error accumulates everywhere at once.

Second, MoE is more fragile in specific places. The router is tiny but decisive. If quantization noise flips a routing decision, the token is suddenly served by a different expert, and the error is not a small perturbation but a discrete change of computation. Shared or "always-on" experts, the attention projections, and the embedding and output layers are also on the hot path for every token, so error there behaves like dense-model error.

So the right mental model is not "MoE tolerates 4-bit." It is "MoE has a small hot path that must stay precise and a large cold path that can be squeezed hard." A flat 4-bit cast ignores that structure and spends the same few bits on the router as on a seldom-touched expert. That is where quality quietly leaks.

What does a UD-Q4_K_XL style mixed-precision graft preserve?

Unsloth's "UD" (Unsloth Dynamic) GGUF variants, and similar mixed-precision schemes, treat the model as a graft rather than a uniform block. The idea is simple to state: keep the sensitive, always-active tensors at higher precision and push the rarely active expert tensors down to 4-bit. The _K_XL suffix signals a K-quant layout with an extra-large share of high-precision tensors retained.

In practice a graft like this tends to protect:

The token embedding and the final output projection, which every token reads and writes.
Attention query, key, value, and output projections, which run on every token in every layer.
The router or gate weights, so routing decisions stay stable and tokens keep landing on the intended experts.
Often the first and last transformer blocks, which empirically carry more than their share of the quality.

Everything else, principally the large pool of expert feed-forward weights, goes to 4-bit. Because that pool is both the biggest contributor to file size and the least active per token, you capture most of the compression while leaving the hot path near full precision. The result is a file a fraction of the original size whose per-token behavior barely changed, which is the only kind of compression worth shipping to an edge device.

The important caveat: which tensors you keep high is a model-specific choice, and vendors publish ready-made UD builds precisely so you do not have to guess. The reasoning above is the general principle, not a recipe to hand-tune.

How do you verify quality parity instead of trusting the quant?

Here is the rule that matters: a quant is guilty until it passes a benchmark. Perplexity on a tiny sample and a few good-looking chat replies are not evidence. You need a discriminating, held-out benchmark scored identically on both the full-precision model and the quantized model, then you compare the scores.

MMLU-Pro is a reasonable choice because it is broad, hard enough to separate models, and standard enough that the harness is well tested. The workflow is: build the quant, score the baseline, score the quant, and accept only if the gap is within benchmark noise.

Step 1: produce the quantized GGUF

If you are starting from a full-precision GGUF and want to roll your own 4-bit file (rather than download a vendor UD build), llama.cpp does the cast:

# Convert a Hugging Face checkpoint to a full-precision GGUF first (if needed)
python llama.cpp/convert_hf_to_gguf.py ./model-full \
    --outfile model-f16.gguf --outtype f16

# Quantize to a 4-bit K-quant. Q4_K_M is the common balanced target.
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# Inspect which tensors landed at which precision
./llama-gguf model-Q4_K_M.gguf | grep -iE "type|ffn|attn|token_embd"

For MoE specifically, prefer a published mixed-precision build (UD-Q4_K_XL or equivalent) when one exists, because the per-tensor precision map is the hard part and it has already been tuned for that architecture.

Step 2: score both models on the same benchmark

Run the identical harness against the baseline and the quant. Keep every knob fixed: same prompt template, same number of shots, same decoding settings, same subset of questions.

# Baseline: full-precision model
llama-perplexity --multiple-choice \
    -m model-f16.gguf \
    -bf mmlu-pro.bin \
    --ctx-size 4096 2>&1 | tee baseline.log

# Candidate: 4-bit mixed graft
llama-perplexity --multiple-choice \
    -m model-UD-Q4_K_XL.gguf \
    -bf mmlu-pro.bin \
    --ctx-size 4096 2>&1 | tee quant.log

# Compare the final accuracy lines
grep -i "final" baseline.log quant.log

Step 3: accept or reject on the number

baseline (f16)        MMLU-Pro: 87.6x
UD-Q4_K_XL (4-bit)    MMLU-Pro: 87.65

Equal inside noise is a pass. If instead the quant drops several points, do not ship it: either the graft protected the wrong tensors, the router drifted, or the target precision was too aggressive for this model. The benchmark told you something the demo never would.

A few guardrails so the comparison stays honest:

Fix the random seed and the question subset so both runs see the same items.
Report the margin, not just "it passed." A positive gap is not automatically a win if it is smaller than run-to-run variance.
Run at least two passes per model. If the two baseline runs disagree by more than the baseline-to-quant gap, your measurement is too noisy to conclude anything.
Beware caches and warm-up effects that help one run and not the other.

When is 4-bit MoE the right call, and when is it not?

It shines when the total parameter count is what blocks you (you cannot fit the file in RAM) while the active path is small enough that the device can actually run it. That is the classic edge case: a big MoE that no consumer machine can hold at full precision becomes loadable at 4-bit because the cold experts shrink dramatically.

It is the wrong call when the model is dense, when you need the last fraction of a point on a sensitive task, or when you have not budgeted time to verify. The compression is free; the confidence is not.

FAQ

Does a smaller GGUF always mean lower quality?
No. File size tracks total parameters, but MoE quality is decided on the small active path. A mixed-precision graft can shrink the file a lot while keeping the active path near full precision, so a much smaller file can score the same on a held-out benchmark.

What is the difference between Q4_K_M and a UD-Q4_K_XL style build?
Q4_K_M applies a mostly uniform 4-bit K-quant. A UD-Q4_K_XL style build keeps more of the sensitive, always-active tensors (embeddings, attention, router) at higher precision and pushes only the rarely active experts to 4-bit, which protects quality on MoE at a similar overall size.

Why not just read the perplexity number?
Perplexity on a small sample is a weak proxy and can look fine while task accuracy slips. A discriminating multiple-choice benchmark like MMLU-Pro, scored identically on both models, is far harder to fool and tells you whether real task behavior survived.

How close is close enough to call it parity?
Within the benchmark's own run-to-run noise. Score each model at least twice. If the baseline-to-quant gap is smaller than the spread between two baseline runs, treat it as equal. Report the actual margin rather than a pass or fail label.

Can I just download a vendor quant instead of making my own?
Usually yes, and for MoE it is often the better choice, because the per-tensor precision map is model specific and tuning it is the hard part. Still run the verification step yourself: trust the build, but confirm the number on your own harness.

Does this apply to dense models too?
The verification workflow applies to everything. The sparsity argument does not: dense models run every weight on every token, so 4-bit error accumulates across the whole network and there is no large cold path to squeeze cheaply.

Building an AI That Refuses to Say "Safe": Reliability Engineering for an Offline Disaster App

GINIGEN AI — Tue, 06 Oct 2026 13:12:57 +0000

Most AI reliability writing assumes a network. You call a bigger model to double-check, you log to a server, you push a hotfix when something goes wrong. A disaster app breaks every one of those assumptions. By the time someone opens it, the cell tower may be rubble and the phone may be in airplane mode for the next two days. There is no bigger model to call, no server to log to, and no hotfix that will arrive in time.

This post is about the engineering we did for HeliGO, an offline disaster and distress app built around an on-device 4B model. The runtime side (thread pinning and lazy loading on a phone) was covered earlier in this series. Here I want to focus on the harder problem: how do you make a small model's answers trustworthy when a wrong answer can get someone killed and nobody is online to catch it?

TL;DR

HeliGO runs entirely on-device in airplane mode: maps, contour lines, altitude, compass, rescue coordinates, and an AI chat, with no network call at any point.
The model is Edge-4B-TELL, built on Gemma-4 E4B, small enough to run on a phone chip.
Safety is not a prompt. It is a 3-stage gate around the model: block fatal outputs, substitute official guidance text, or refuse.
The gate is asymmetric on purpose. It never says a thing is safe to eat, but it always warns when something is poisonous.
Confidence comes from reading the model's hidden state (the TELL technique), not from asking the model how sure it is, because asking does not work.

Why "just prompt it to be careful" fails offline

The obvious approach is a system prompt: "You are a disaster assistant. Never give dangerous advice. Say you are unsure when you are unsure." On a 4B model with no fallback, this is wishful thinking for two reasons.

First, a prompt is a suggestion, not a constraint. A small model under distribution shift (a panicked, ungrammatical question about a mushroom in the dark) will still occasionally produce a confident, wrong, fatal answer. There is no second model downstream to catch it.

Second, and this is the part that surprised us most, the model's own stated confidence is not just useless, it is inverted. When we asked the model to attach a confidence number to its answers, wrong answers tended to come with higher stated confidence than right ones. Asking a model "are you sure?" and trusting the reply is worse than a coin flip in exactly the cases that matter.

So the reliability layer cannot live inside the model's text output. It has to wrap the model.

The 3-stage safety gate

Every answer the model produces passes through three stages before it reaches the screen. Think of it as a decision pipeline, not a filter.

model draft
   |
   v
[1] fatal-output block   -- is this a category that can kill if wrong?
   |  (if yes and risky) -> suppress, escalate to stage 2
   v
[2] official-text substitute -- is there a matching MOIS guideline?
   |  (if yes) -> answer with the official text, not the model's words
   v
[3] refuse               -- still not confident enough?
   |  (if yes) -> say "I am not sure", do not guess
   v
answer shown to user

Stage 1, fatal-output block. Some answer categories are life-or-death: "yes, you can eat this", "yes, this water is safe", "yes, go that way". For these, the model's confidence is irrelevant. If the content falls in a fatal category and carries any risk signal, it is suppressed regardless of how sure the model sounds. This is the direct consequence of the inverted-confidence finding: in the categories where being wrong is fatal, we do not let the model's certainty vote at all.

Stage 2, official-text substitution. We packaged 75 official response guidelines from Korea's Ministry of the Interior and Safety (MOIS) on the device. When a question maps to one of them (earthquake, flood, wildfire, first aid, and so on), the app answers with the official text verbatim instead of the model's paraphrase. The model is still useful here: it does the matching and retrieval. But the words the user reads are the authoritative source, not a 4B paraphrase that might drift.

Stage 3, refuse. If an answer is neither safely substitutable nor confidently verifiable, the app declines. "I am not sure" is a valid and often correct output for a disaster tool. A refusal costs the user a little time. A confident wrong answer can cost a life. The gate is built so refusal is the default failure mode, never a guess.

The asymmetry principle

The gate is deliberately not symmetric, and this is the design decision I am most sure about.

The app never says "this is safe." It always says "this is poisonous" when it has reason to.

The reasoning is a straightforward expected-cost argument. Consider the two ways the app can be wrong about something edible:

Situation	App says	If app is wrong	Cost
Edible plant	"I cannot confirm this is safe"	User stays hungry	Low, recoverable
Poisonous plant	"This may be poisonous"	User avoids safe food	Low, recoverable
Poisonous plant	"This is safe to eat"	User is poisoned	Catastrophic

Three of these outcomes are survivable. Only one is not. So the app is tuned to make the unsurvivable mistake nearly impossible, at the cost of being overly cautious in the harmless direction. The costs of silence are not equal on the two sides, so the gate is not equal either. We ship warnings for 12 venomous and poisonous species in the same spirit: a false alarm is cheap, a missed warning is not. Any edge safety system where errors have asymmetric cost should bias toward the cheap error.

TELL: reading the model instead of asking it

Stages 1 and 3 both need a trustworthy confidence signal, and we already established that the model's spoken confidence is inverted. The fix is to stop asking.

TELL reads the model's internal state while it produces an answer and derives a reliability estimate from that, rather than from any number the model writes in its text. The intuition: the computation a transformer performs on its way to an answer carries signal about whether that answer is well-grounded, and that signal is more honest than the self-report the model emits at the end. Internal state does not have an incentive to sound confident; the output text, apparently, does.

I am not going to publish the thresholds or the exact internal signals we read, for the obvious reason that they are what makes the gate hard to game. The point for this article is architectural: the confidence check lives outside the token stream. We treat the model as an instrument to be read, not a witness to be interviewed.

The TELL technique was released alongside the model. In HeliGO it currently backs the gate's suppress-or-refuse decisions, and a future update will surface low-confidence filtering more directly to the user.

What actually ships on the device

For an offline tool, "on-device" has to be literal. Everything the gate and the map need is packaged into the app so it works with the radio off:

4,998 terrain tiles with elevation data, and contour lines that redraw automatically by zoom range
16,568 mountain summit elevations
1,255 shelter, medical, water, and fire facilities, plus nearest-high-ground routing for floods
75 MOIS official response guidelines (the stage-2 substitution corpus)
12 venomous and poisonous species warnings
The Edge-4B-TELL model itself, for offline chat, photo identification, and voice input

None of this phones home. Location history in particular never leaves the device, which is both a privacy property and a reliability property: there is no network dependency to fail.

H2: How do you make an offline AI trustworthy without a bigger model to check it?

You move trust out of the model and into a deterministic wrapper. The model drafts and retrieves; a gate you control decides what reaches the user. Fatal categories are blocked on content, not on the model's confidence. Authoritative answers come from packaged official text. When neither applies, you refuse. The model never gets the final word on anything that can kill.

H2: Why not just use a larger model on-device?

Because the phone is the budget. A 4B model is what runs in airplane mode on a consumer chip with room left for maps and rendering. The interesting claim of this project is not that a big model is safe, it is that a small model plus the right wrapper can be safe enough to ship for life-or-death use. The reliability came from architecture, not from parameter count.

FAQ

Does HeliGO send anything to a server?
No. Maps, contours, altitude, compass, rescue coordinates, and the AI chat all run on-device in airplane mode. Location history does not leave the phone.

What is the model?
Edge-4B-TELL, built on Google's Gemma-4 E4B, published on Hugging Face. It handles offline chat, photo identification, and voice input on the phone's own chip.

Why does it never say something is safe to eat?
Because the cost of being wrong is asymmetric. A missed safe meal is hunger; a wrong "safe" is poisoning. The gate biases toward the recoverable error and only ever issues the poison warning side confidently.

Can I trust the AI's confidence when it answers?
The app does not trust the model's stated confidence, and neither should you in general. Measured on this model, wrong answers tended to be stated more confidently than right ones. HeliGO derives confidence from the model's internal state (TELL) instead, outside the text it writes.

Is this a replacement for calling emergency services?
No. HeliGO does not replace rescue services. If you have any signal, call emergency services first. It is a tool for when you have no connection.

Takeaways for edge builders

Put the safety logic outside the model. A prompt is not a constraint; a deterministic gate is.
Do not trust self-reported confidence. Measure whether it is calibrated before you rely on it. It may be inverted.
Make your error handling asymmetric when the costs are asymmetric. Bias toward the cheap mistake.
Package everything. For a true offline tool, every datum the app needs must live on the device.

Running a 180B Model on a Laptop With No GPU: How 4-bit GGUF Keeps Full Accuracy

GINIGEN AI — Tue, 06 Oct 2026 00:42:56 +0000

A frontier-class model used to mean a rack of data-center GPUs. That assumption is what this post takes apart.

POCKET-Darwin-180B-GGUF is a 4-bit build of Darwin-180B-RSI, a 180-billion-parameter model, packaged so it runs without a GPU. It ships as GGUF on Hugging Face and on ModelScope. The headline numbers, all measured in October 2026:

Size: 111 GB (4-bit GGUF, 4 files), down from 360 GB (BF16, 131 files)
CPU only: a single server CPU (16 threads) generates 18.4 to 21.0 tokens/s, peak memory 78.8 GB
Laptop: an RTX 5060 Laptop (8 GB VRAM) with 32 GB RAM runs it at 4.17 tokens/s
Mini PC: a 128 GB-RAM mini PC holds the whole model in memory, no GPU
Accuracy: MMLU-Pro, 2,000 questions, paired comparison, 87.65% original vs 87.65% quantized

TL;DR

A 180B mixture-of-experts model activates only about 3B parameters per token. Combine that sparsity with a 4-bit quantization that keeps the small fraction of the weights that actually changed during training at high precision, and you get a model that fits in laptop-class memory and still answers like the full model. You run it with llama.cpp in one command. This post explains why it works and how to reproduce it.

How does a 180B model fit on a laptop?

Two things make this possible, and neither is magic.

First, the architecture is sparse. Darwin-180B is a mixture-of-experts (MoE) model: 512 expert sub-networks, of which only 10 are selected per token. Of the 180B total parameters, roughly 3B are active for any given token. The dense parts (attention, routing, embeddings) run every step; the expert weights are read on demand. llama.cpp can memory-map the file and let the operating system page expert tensors in and out, so you do not need all 111 GB resident at once. On a 128 GB mini PC the whole thing sits in RAM; on a 32 GB laptop the OS pages against the GGUF file on disk, which is why the laptop is slower but still runs.

Second, the quantization is selective. The build starts from a community-verified 4-bit quantization of the base weights. On top of that, only the portion that the self-improvement training actually modified (about 3% of total volume) is kept at higher precision. You spend your bit budget where the signal is, not uniformly. That is the difference between "4-bit and lossy" and "4-bit and lossless on the benchmark."

Does 4-bit quantization hurt accuracy here?

On the measured benchmark, no. The honest way to check quantization damage is a paired comparison: same questions, same decoding, original versus quantized, item by item. On MMLU-Pro (2,000 questions) both the original and the 4-bit build scored 87.65%. Identical.

That result is specific and worth reading carefully. It does not mean 4-bit is free in general. It means that for this model, on this benchmark, with a selective scheme that protects the trained delta, the drop is below the measurement's resolution. Quantization error concentrates in the weights that carry the most task-relevant signal, so protecting that 3% is what preserves the score.

There is a second, subtler result. On SuperGPQA (1,000 graduate-level science questions, never used in training), the quantized POCKET build was compared against the same 4-bit quantization of its base model. POCKET scored 61.55% vs 59.10%, a 2.45-point gain that held as statistically significant, while using about 13% fewer tokens per question. The self-improvement training taught the model to commit to an answer instead of re-deriving it, and quantization preserved that behavior. A third-party developer reproduced this at the file level in a public thread.

How do you run POCKET-Darwin-180B-GGUF?

You need a recent llama.cpp build (b11048 or later), the four GGUF files (111 GB total), and enough memory to either hold or page the model. No cloud account, no GPU driver stack required for the CPU path.

# 1. Build llama.cpp (b11048 or later)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

# 2. Download the GGUF (4 files, ~111 GB) from Hugging Face
#    FINAL-Bench/POCKET-Darwin-180B-GGUF
#    (any standard huggingface download works; place all parts
#     in one directory so the loader finds the shards)

# 3. Run, CPU only, 16 threads
./build/bin/llama-cli \
  -m ./POCKET-Darwin-180B-Q4.gguf \
  -t 16 \
  -c 8192 \
  -p "Explain mixture-of-experts routing in two sentences."

A few practical notes from the measurements:

Threads matter. The server CPU result used 16 threads. Match -t to your physical cores. Oversubscribing threads past your core count usually slows generation, not speeds it.
Memory decides your path. With 128 GB of RAM the model stays resident and you get the top of the range. With 32 GB the OS pages expert tensors from disk, so throughput drops to a few tokens per second. A fast NVMe SSD helps the paging path noticeably.
Context length costs memory too. The 78.8 GB figure is peak at a working context. Raising -c raises the KV-cache footprint on top of the weights.
GPU offload is optional. On the 8 GB laptop GPU, only a fraction of layers fit in VRAM, so most of the work still lands on CPU and RAM. The 4.17 tok/s number reflects that mixed path.

When is a CPU-only 180B model actually the right call?

Throughput is the honest tradeoff. 21 tok/s on a server CPU is fine for batch jobs, drafting, extraction, and agent steps that tolerate latency. 4 tok/s on a laptop is a developer convenience, not a chat product. If you need interactive speed at scale, a GPU still wins.

Where the CPU path wins is where the data cannot leave the building. An air-gapped server, a mini PC on a factory floor, a defense or finance or public-sector box with no outbound connection: there a model that runs entirely on local CPU and RAM, with weights you downloaded once, is not a compromise. It is the only option that clears the policy. Edge AI is often less about milliseconds and more about where the bytes are allowed to sit.

FAQ

Can I run this in LM Studio or Ollama?
This post only verifies the llama.cpp path (b11048 or later), which is the engine GGUF targets. Treat other runners as untested here until you confirm they load this specific build.

Do I need the full 111 GB in RAM?
No. With 128 GB you can keep it resident for best speed. With 32 GB the OS memory-maps the GGUF and pages expert tensors on demand, trading throughput for a smaller footprint. You do need the full 111 GB on disk.

Why is the laptop so much slower than the server CPU?
Memory. The 32 GB laptop cannot hold the model, so it pages from disk, and only a few layers fit on the 8 GB GPU. The 16-thread server CPU with ample RAM keeps far more of the working set hot.

Is 4-bit always this lossless?
No, and do not generalize it. The 87.65% match is a measured paired result for this model and benchmark, enabled by keeping the trained delta (about 3% of the weights) at higher precision. Always verify your own quantization with a paired comparison rather than assuming parity.

What makes the active parameter count so low?
The MoE router picks 10 of 512 experts per token, so roughly 3B of the 180B parameters do work on any single token. That sparsity is what makes CPU inference and on-demand expert loading practical.

Where do I get it?
huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF and modelscope.cn/models/FINAL-Bench/POCKET-Darwin-180B-GGUF. The original is huggingface.co/FINAL-Bench/Darwin-180B-RSI.

DEV Community: GINIGEN AI

The Inference Hardware You Already Own Might Train Models Too: A 325M Pretraining Run on an NPU

TL;DR

Why should edge and on-prem teams care?

What was actually shown?

Is the NPU-trained model really as good?

How far has this line come?

What is the catch?

FAQ

Further reading

Chitos on the OWASP Benchmark: 77.8 Youden, on-prem

Reading the jump

Why the on-prem row matters most

0.1 Bits Per Weight Is Not 0.1 Bits Per Weight: Where the Bytes Go in Sub-2-Bit LLMs

TL;DR

What does "0.1 bits per weight" actually mean?

Why is a 0.1 BPW 13B model 0.84 GB and not 0.16 GB?

How do other sub-2-bit releases look at the file level?

What does the quality cost?

What is the 11.6x speedup, and what is it not?

When does this matter for an edge deployment?

What are the limits of this analysis?

FAQ

Further reading

Put the Decision Model Inside the Agent Loop, Not Beside It

The cost nobody budgets for

A different shape of model

Why calibration is the whole point

What it changes in practice

Reading the public boards

A current reference point

LFM2.5-24B-A2B on Your Own Hardware: The RAM, KV Cache and Bandwidth Budget

TL;DR

Why does a 24B model with 2.45B active parameters still need 14.5 GB of RAM?

Where do the 2.45B active parameters actually go?

How big is the KV cache when 30 of 40 layers are convolutions?

Will it fit on my machine?

What speed should I expect? A bandwidth ceiling, not a promise

How do I run it with a tight memory budget?

What are the limits?

FAQ

Further reading

사라진 명리 비기를 AI로 복원하다: 주명명리(宙明命理) 개발기

TL;DR

왜 만들었나

만세력 엔진은 어떻게 동작하나

풀이는 어떻게 생성되나

서비스 아키텍처

X에서 댓글로 사주 보기

배운 점

FAQ

지금 체험하기

Zero-Token Decisions in the Browser: On-Device Multimodal Classification with EmbeddingGemma 2 and WebGPU

TL;DR

Why a decision should not be a generation

How a decision is computed

The on-device stack

What you can actually do with it

Why this matters for edge AI

FAQ

Open Quantum Challenge (OQC): a reproducible, hardware-free quantum benchmark

Two seasons, $1,000 prize each

How to enter

Links

One Self-Improving Model, Eleven Number-One Titles: What That Takes

TL;DR

The eleven, at a glance

Why a self-improving loop gets here

The decision layer

Open to use

FAQ

4-bit GGUF Quality for MoE Models: Why Only 3B of 180B Params Fire, and How to Prove Parity

TL;DR

Why does only a fraction of an MoE model activate per token?

Why does 4-bit hurt MoE differently than a dense model?

What does a UD-Q4_K_XL style mixed-precision graft preserve?

How do you verify quality parity instead of trusting the quant?

Step 1: produce the quantized GGUF

Step 2: score both models on the same benchmark

Step 3: accept or reject on the number