DEV Community: GINIGEN AI

4-bit GGUF Quality for MoE Models: Why Only 3B of 180B Params Fire, and How to Prove Parity

GINIGEN AI — Tue, 06 Oct 2026 19:10:26 +0000

TL;DR

Mixture-of-Experts (MoE) models look enormous on disk, but only a small slice of the weights does work on any single token. A 180B-parameter MoE can activate roughly 3B parameters per forward pass. That sparsity is exactly why 4-bit GGUF quantization behaves so differently on MoE than on a dense model, and why a mixed-precision scheme such as the UD-Q4_K_XL style graft can hold accuracy that a flat 4-bit cast would lose.

The honest part of the story is not the compression ratio. It is verification. A quantized file that loads and produces fluent text is not proof of anything. You prove parity by scoring the quantized model on a held-out benchmark and comparing it to the full-precision baseline, number against number. In the example below, a 4-bit mixed graft reaches MMLU-Pro 87.65, equal to the full-precision baseline inside noise. This post shows the reasoning and the exact llama.cpp commands to reproduce that check yourself.

Why does only a fraction of an MoE model activate per token?

A dense transformer runs every weight for every token. A Mixture-of-Experts transformer replaces the big feed-forward block in each layer with many smaller expert blocks plus a router. For each token the router picks a few experts (often two) and ignores the rest. The attention layers and the router still run every time, but the overwhelming majority of the feed-forward parameters sit idle on any given token.

The practical consequence: the "180B" headline counts total parameters, while the compute and the active memory traffic track the much smaller active set, on the order of 3B. Two numbers describe an MoE and they answer different questions:

Total parameters set the disk and RAM footprint you must store.
Active parameters per token set the arithmetic you actually perform.

Quantization touches storage, so it operates on the total. Quality, however, is decided token by token on the active path. This gap is the whole reason MoE quantization is its own topic.

Why does 4-bit hurt MoE differently than a dense model?

Two forces pull in opposite directions.

First, MoE is more forgiving in aggregate. Averaged over a corpus, errors introduced into rarely used experts barely move the output, because those experts rarely fire. A dense model has no such luxury: every weight is on the hot path for every token, so quantization error accumulates everywhere at once.

Second, MoE is more fragile in specific places. The router is tiny but decisive. If quantization noise flips a routing decision, the token is suddenly served by a different expert, and the error is not a small perturbation but a discrete change of computation. Shared or "always-on" experts, the attention projections, and the embedding and output layers are also on the hot path for every token, so error there behaves like dense-model error.

So the right mental model is not "MoE tolerates 4-bit." It is "MoE has a small hot path that must stay precise and a large cold path that can be squeezed hard." A flat 4-bit cast ignores that structure and spends the same few bits on the router as on a seldom-touched expert. That is where quality quietly leaks.

What does a UD-Q4_K_XL style mixed-precision graft preserve?

Unsloth's "UD" (Unsloth Dynamic) GGUF variants, and similar mixed-precision schemes, treat the model as a graft rather than a uniform block. The idea is simple to state: keep the sensitive, always-active tensors at higher precision and push the rarely active expert tensors down to 4-bit. The _K_XL suffix signals a K-quant layout with an extra-large share of high-precision tensors retained.

In practice a graft like this tends to protect:

The token embedding and the final output projection, which every token reads and writes.
Attention query, key, value, and output projections, which run on every token in every layer.
The router or gate weights, so routing decisions stay stable and tokens keep landing on the intended experts.
Often the first and last transformer blocks, which empirically carry more than their share of the quality.

Everything else, principally the large pool of expert feed-forward weights, goes to 4-bit. Because that pool is both the biggest contributor to file size and the least active per token, you capture most of the compression while leaving the hot path near full precision. The result is a file a fraction of the original size whose per-token behavior barely changed, which is the only kind of compression worth shipping to an edge device.

The important caveat: which tensors you keep high is a model-specific choice, and vendors publish ready-made UD builds precisely so you do not have to guess. The reasoning above is the general principle, not a recipe to hand-tune.

How do you verify quality parity instead of trusting the quant?

Here is the rule that matters: a quant is guilty until it passes a benchmark. Perplexity on a tiny sample and a few good-looking chat replies are not evidence. You need a discriminating, held-out benchmark scored identically on both the full-precision model and the quantized model, then you compare the scores.

MMLU-Pro is a reasonable choice because it is broad, hard enough to separate models, and standard enough that the harness is well tested. The workflow is: build the quant, score the baseline, score the quant, and accept only if the gap is within benchmark noise.

Step 1: produce the quantized GGUF

If you are starting from a full-precision GGUF and want to roll your own 4-bit file (rather than download a vendor UD build), llama.cpp does the cast:

# Convert a Hugging Face checkpoint to a full-precision GGUF first (if needed)
python llama.cpp/convert_hf_to_gguf.py ./model-full \
    --outfile model-f16.gguf --outtype f16

# Quantize to a 4-bit K-quant. Q4_K_M is the common balanced target.
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# Inspect which tensors landed at which precision
./llama-gguf model-Q4_K_M.gguf | grep -iE "type|ffn|attn|token_embd"

For MoE specifically, prefer a published mixed-precision build (UD-Q4_K_XL or equivalent) when one exists, because the per-tensor precision map is the hard part and it has already been tuned for that architecture.

Step 2: score both models on the same benchmark

Run the identical harness against the baseline and the quant. Keep every knob fixed: same prompt template, same number of shots, same decoding settings, same subset of questions.

# Baseline: full-precision model
llama-perplexity --multiple-choice \
    -m model-f16.gguf \
    -bf mmlu-pro.bin \
    --ctx-size 4096 2>&1 | tee baseline.log

# Candidate: 4-bit mixed graft
llama-perplexity --multiple-choice \
    -m model-UD-Q4_K_XL.gguf \
    -bf mmlu-pro.bin \
    --ctx-size 4096 2>&1 | tee quant.log

# Compare the final accuracy lines
grep -i "final" baseline.log quant.log

Step 3: accept or reject on the number

baseline (f16)        MMLU-Pro: 87.6x
UD-Q4_K_XL (4-bit)    MMLU-Pro: 87.65

Equal inside noise is a pass. If instead the quant drops several points, do not ship it: either the graft protected the wrong tensors, the router drifted, or the target precision was too aggressive for this model. The benchmark told you something the demo never would.

A few guardrails so the comparison stays honest:

Fix the random seed and the question subset so both runs see the same items.
Report the margin, not just "it passed." A positive gap is not automatically a win if it is smaller than run-to-run variance.
Run at least two passes per model. If the two baseline runs disagree by more than the baseline-to-quant gap, your measurement is too noisy to conclude anything.
Beware caches and warm-up effects that help one run and not the other.

When is 4-bit MoE the right call, and when is it not?

It shines when the total parameter count is what blocks you (you cannot fit the file in RAM) while the active path is small enough that the device can actually run it. That is the classic edge case: a big MoE that no consumer machine can hold at full precision becomes loadable at 4-bit because the cold experts shrink dramatically.

It is the wrong call when the model is dense, when you need the last fraction of a point on a sensitive task, or when you have not budgeted time to verify. The compression is free; the confidence is not.

FAQ

Does a smaller GGUF always mean lower quality?
No. File size tracks total parameters, but MoE quality is decided on the small active path. A mixed-precision graft can shrink the file a lot while keeping the active path near full precision, so a much smaller file can score the same on a held-out benchmark.

What is the difference between Q4_K_M and a UD-Q4_K_XL style build?
Q4_K_M applies a mostly uniform 4-bit K-quant. A UD-Q4_K_XL style build keeps more of the sensitive, always-active tensors (embeddings, attention, router) at higher precision and pushes only the rarely active experts to 4-bit, which protects quality on MoE at a similar overall size.

Why not just read the perplexity number?
Perplexity on a small sample is a weak proxy and can look fine while task accuracy slips. A discriminating multiple-choice benchmark like MMLU-Pro, scored identically on both models, is far harder to fool and tells you whether real task behavior survived.

How close is close enough to call it parity?
Within the benchmark's own run-to-run noise. Score each model at least twice. If the baseline-to-quant gap is smaller than the spread between two baseline runs, treat it as equal. Report the actual margin rather than a pass or fail label.

Can I just download a vendor quant instead of making my own?
Usually yes, and for MoE it is often the better choice, because the per-tensor precision map is model specific and tuning it is the hard part. Still run the verification step yourself: trust the build, but confirm the number on your own harness.

Does this apply to dense models too?
The verification workflow applies to everything. The sparsity argument does not: dense models run every weight on every token, so 4-bit error accumulates across the whole network and there is no large cold path to squeeze cheaply.

Building an AI That Refuses to Say "Safe": Reliability Engineering for an Offline Disaster App

GINIGEN AI — Tue, 06 Oct 2026 13:12:57 +0000

Most AI reliability writing assumes a network. You call a bigger model to double-check, you log to a server, you push a hotfix when something goes wrong. A disaster app breaks every one of those assumptions. By the time someone opens it, the cell tower may be rubble and the phone may be in airplane mode for the next two days. There is no bigger model to call, no server to log to, and no hotfix that will arrive in time.

This post is about the engineering we did for HeliGO, an offline disaster and distress app built around an on-device 4B model. The runtime side (thread pinning and lazy loading on a phone) was covered earlier in this series. Here I want to focus on the harder problem: how do you make a small model's answers trustworthy when a wrong answer can get someone killed and nobody is online to catch it?

TL;DR

HeliGO runs entirely on-device in airplane mode: maps, contour lines, altitude, compass, rescue coordinates, and an AI chat, with no network call at any point.
The model is Edge-4B-TELL, built on Gemma-4 E4B, small enough to run on a phone chip.
Safety is not a prompt. It is a 3-stage gate around the model: block fatal outputs, substitute official guidance text, or refuse.
The gate is asymmetric on purpose. It never says a thing is safe to eat, but it always warns when something is poisonous.
Confidence comes from reading the model's hidden state (the TELL technique), not from asking the model how sure it is, because asking does not work.

Why "just prompt it to be careful" fails offline

The obvious approach is a system prompt: "You are a disaster assistant. Never give dangerous advice. Say you are unsure when you are unsure." On a 4B model with no fallback, this is wishful thinking for two reasons.

First, a prompt is a suggestion, not a constraint. A small model under distribution shift (a panicked, ungrammatical question about a mushroom in the dark) will still occasionally produce a confident, wrong, fatal answer. There is no second model downstream to catch it.

Second, and this is the part that surprised us most, the model's own stated confidence is not just useless, it is inverted. When we asked the model to attach a confidence number to its answers, wrong answers tended to come with higher stated confidence than right ones. Asking a model "are you sure?" and trusting the reply is worse than a coin flip in exactly the cases that matter.

So the reliability layer cannot live inside the model's text output. It has to wrap the model.

The 3-stage safety gate

Every answer the model produces passes through three stages before it reaches the screen. Think of it as a decision pipeline, not a filter.

model draft
   |
   v
[1] fatal-output block   -- is this a category that can kill if wrong?
   |  (if yes and risky) -> suppress, escalate to stage 2
   v
[2] official-text substitute -- is there a matching MOIS guideline?
   |  (if yes) -> answer with the official text, not the model's words
   v
[3] refuse               -- still not confident enough?
   |  (if yes) -> say "I am not sure", do not guess
   v
answer shown to user

Stage 1, fatal-output block. Some answer categories are life-or-death: "yes, you can eat this", "yes, this water is safe", "yes, go that way". For these, the model's confidence is irrelevant. If the content falls in a fatal category and carries any risk signal, it is suppressed regardless of how sure the model sounds. This is the direct consequence of the inverted-confidence finding: in the categories where being wrong is fatal, we do not let the model's certainty vote at all.

Stage 2, official-text substitution. We packaged 75 official response guidelines from Korea's Ministry of the Interior and Safety (MOIS) on the device. When a question maps to one of them (earthquake, flood, wildfire, first aid, and so on), the app answers with the official text verbatim instead of the model's paraphrase. The model is still useful here: it does the matching and retrieval. But the words the user reads are the authoritative source, not a 4B paraphrase that might drift.

Stage 3, refuse. If an answer is neither safely substitutable nor confidently verifiable, the app declines. "I am not sure" is a valid and often correct output for a disaster tool. A refusal costs the user a little time. A confident wrong answer can cost a life. The gate is built so refusal is the default failure mode, never a guess.

The asymmetry principle

The gate is deliberately not symmetric, and this is the design decision I am most sure about.

The app never says "this is safe." It always says "this is poisonous" when it has reason to.

The reasoning is a straightforward expected-cost argument. Consider the two ways the app can be wrong about something edible:

Situation	App says	If app is wrong	Cost
Edible plant	"I cannot confirm this is safe"	User stays hungry	Low, recoverable
Poisonous plant	"This may be poisonous"	User avoids safe food	Low, recoverable
Poisonous plant	"This is safe to eat"	User is poisoned	Catastrophic

Three of these outcomes are survivable. Only one is not. So the app is tuned to make the unsurvivable mistake nearly impossible, at the cost of being overly cautious in the harmless direction. The costs of silence are not equal on the two sides, so the gate is not equal either. We ship warnings for 12 venomous and poisonous species in the same spirit: a false alarm is cheap, a missed warning is not. Any edge safety system where errors have asymmetric cost should bias toward the cheap error.

TELL: reading the model instead of asking it

Stages 1 and 3 both need a trustworthy confidence signal, and we already established that the model's spoken confidence is inverted. The fix is to stop asking.

TELL reads the model's internal state while it produces an answer and derives a reliability estimate from that, rather than from any number the model writes in its text. The intuition: the computation a transformer performs on its way to an answer carries signal about whether that answer is well-grounded, and that signal is more honest than the self-report the model emits at the end. Internal state does not have an incentive to sound confident; the output text, apparently, does.

I am not going to publish the thresholds or the exact internal signals we read, for the obvious reason that they are what makes the gate hard to game. The point for this article is architectural: the confidence check lives outside the token stream. We treat the model as an instrument to be read, not a witness to be interviewed.

The TELL technique was released alongside the model. In HeliGO it currently backs the gate's suppress-or-refuse decisions, and a future update will surface low-confidence filtering more directly to the user.

What actually ships on the device

For an offline tool, "on-device" has to be literal. Everything the gate and the map need is packaged into the app so it works with the radio off:

4,998 terrain tiles with elevation data, and contour lines that redraw automatically by zoom range
16,568 mountain summit elevations
1,255 shelter, medical, water, and fire facilities, plus nearest-high-ground routing for floods
75 MOIS official response guidelines (the stage-2 substitution corpus)
12 venomous and poisonous species warnings
The Edge-4B-TELL model itself, for offline chat, photo identification, and voice input

None of this phones home. Location history in particular never leaves the device, which is both a privacy property and a reliability property: there is no network dependency to fail.

H2: How do you make an offline AI trustworthy without a bigger model to check it?

You move trust out of the model and into a deterministic wrapper. The model drafts and retrieves; a gate you control decides what reaches the user. Fatal categories are blocked on content, not on the model's confidence. Authoritative answers come from packaged official text. When neither applies, you refuse. The model never gets the final word on anything that can kill.

H2: Why not just use a larger model on-device?

Because the phone is the budget. A 4B model is what runs in airplane mode on a consumer chip with room left for maps and rendering. The interesting claim of this project is not that a big model is safe, it is that a small model plus the right wrapper can be safe enough to ship for life-or-death use. The reliability came from architecture, not from parameter count.

FAQ

Does HeliGO send anything to a server?
No. Maps, contours, altitude, compass, rescue coordinates, and the AI chat all run on-device in airplane mode. Location history does not leave the phone.

What is the model?
Edge-4B-TELL, built on Google's Gemma-4 E4B, published on Hugging Face. It handles offline chat, photo identification, and voice input on the phone's own chip.

Why does it never say something is safe to eat?
Because the cost of being wrong is asymmetric. A missed safe meal is hunger; a wrong "safe" is poisoning. The gate biases toward the recoverable error and only ever issues the poison warning side confidently.

Can I trust the AI's confidence when it answers?
The app does not trust the model's stated confidence, and neither should you in general. Measured on this model, wrong answers tended to be stated more confidently than right ones. HeliGO derives confidence from the model's internal state (TELL) instead, outside the text it writes.

Is this a replacement for calling emergency services?
No. HeliGO does not replace rescue services. If you have any signal, call emergency services first. It is a tool for when you have no connection.

Takeaways for edge builders

Put the safety logic outside the model. A prompt is not a constraint; a deterministic gate is.
Do not trust self-reported confidence. Measure whether it is calibrated before you rely on it. It may be inverted.
Make your error handling asymmetric when the costs are asymmetric. Bias toward the cheap mistake.
Package everything. For a true offline tool, every datum the app needs must live on the device.

Running a 180B Model on a Laptop With No GPU: How 4-bit GGUF Keeps Full Accuracy

GINIGEN AI — Tue, 06 Oct 2026 00:42:56 +0000

A frontier-class model used to mean a rack of data-center GPUs. That assumption is what this post takes apart.

POCKET-Darwin-180B-GGUF is a 4-bit build of Darwin-180B-RSI, a 180-billion-parameter model, packaged so it runs without a GPU. It ships as GGUF on Hugging Face and on ModelScope. The headline numbers, all measured in October 2026:

Size: 111 GB (4-bit GGUF, 4 files), down from 360 GB (BF16, 131 files)
CPU only: a single server CPU (16 threads) generates 18.4 to 21.0 tokens/s, peak memory 78.8 GB
Laptop: an RTX 5060 Laptop (8 GB VRAM) with 32 GB RAM runs it at 4.17 tokens/s
Mini PC: a 128 GB-RAM mini PC holds the whole model in memory, no GPU
Accuracy: MMLU-Pro, 2,000 questions, paired comparison, 87.65% original vs 87.65% quantized

TL;DR

A 180B mixture-of-experts model activates only about 3B parameters per token. Combine that sparsity with a 4-bit quantization that keeps the small fraction of the weights that actually changed during training at high precision, and you get a model that fits in laptop-class memory and still answers like the full model. You run it with llama.cpp in one command. This post explains why it works and how to reproduce it.

How does a 180B model fit on a laptop?

Two things make this possible, and neither is magic.

First, the architecture is sparse. Darwin-180B is a mixture-of-experts (MoE) model: 512 expert sub-networks, of which only 10 are selected per token. Of the 180B total parameters, roughly 3B are active for any given token. The dense parts (attention, routing, embeddings) run every step; the expert weights are read on demand. llama.cpp can memory-map the file and let the operating system page expert tensors in and out, so you do not need all 111 GB resident at once. On a 128 GB mini PC the whole thing sits in RAM; on a 32 GB laptop the OS pages against the GGUF file on disk, which is why the laptop is slower but still runs.

Second, the quantization is selective. The build starts from a community-verified 4-bit quantization of the base weights. On top of that, only the portion that the self-improvement training actually modified (about 3% of total volume) is kept at higher precision. You spend your bit budget where the signal is, not uniformly. That is the difference between "4-bit and lossy" and "4-bit and lossless on the benchmark."

Does 4-bit quantization hurt accuracy here?

On the measured benchmark, no. The honest way to check quantization damage is a paired comparison: same questions, same decoding, original versus quantized, item by item. On MMLU-Pro (2,000 questions) both the original and the 4-bit build scored 87.65%. Identical.

That result is specific and worth reading carefully. It does not mean 4-bit is free in general. It means that for this model, on this benchmark, with a selective scheme that protects the trained delta, the drop is below the measurement's resolution. Quantization error concentrates in the weights that carry the most task-relevant signal, so protecting that 3% is what preserves the score.

There is a second, subtler result. On SuperGPQA (1,000 graduate-level science questions, never used in training), the quantized POCKET build was compared against the same 4-bit quantization of its base model. POCKET scored 61.55% vs 59.10%, a 2.45-point gain that held as statistically significant, while using about 13% fewer tokens per question. The self-improvement training taught the model to commit to an answer instead of re-deriving it, and quantization preserved that behavior. A third-party developer reproduced this at the file level in a public thread.

How do you run POCKET-Darwin-180B-GGUF?

You need a recent llama.cpp build (b11048 or later), the four GGUF files (111 GB total), and enough memory to either hold or page the model. No cloud account, no GPU driver stack required for the CPU path.

# 1. Build llama.cpp (b11048 or later)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

# 2. Download the GGUF (4 files, ~111 GB) from Hugging Face
#    FINAL-Bench/POCKET-Darwin-180B-GGUF
#    (any standard huggingface download works; place all parts
#     in one directory so the loader finds the shards)

# 3. Run, CPU only, 16 threads
./build/bin/llama-cli \
  -m ./POCKET-Darwin-180B-Q4.gguf \
  -t 16 \
  -c 8192 \
  -p "Explain mixture-of-experts routing in two sentences."

A few practical notes from the measurements:

Threads matter. The server CPU result used 16 threads. Match -t to your physical cores. Oversubscribing threads past your core count usually slows generation, not speeds it.
Memory decides your path. With 128 GB of RAM the model stays resident and you get the top of the range. With 32 GB the OS pages expert tensors from disk, so throughput drops to a few tokens per second. A fast NVMe SSD helps the paging path noticeably.
Context length costs memory too. The 78.8 GB figure is peak at a working context. Raising -c raises the KV-cache footprint on top of the weights.
GPU offload is optional. On the 8 GB laptop GPU, only a fraction of layers fit in VRAM, so most of the work still lands on CPU and RAM. The 4.17 tok/s number reflects that mixed path.

When is a CPU-only 180B model actually the right call?

Throughput is the honest tradeoff. 21 tok/s on a server CPU is fine for batch jobs, drafting, extraction, and agent steps that tolerate latency. 4 tok/s on a laptop is a developer convenience, not a chat product. If you need interactive speed at scale, a GPU still wins.

Where the CPU path wins is where the data cannot leave the building. An air-gapped server, a mini PC on a factory floor, a defense or finance or public-sector box with no outbound connection: there a model that runs entirely on local CPU and RAM, with weights you downloaded once, is not a compromise. It is the only option that clears the policy. Edge AI is often less about milliseconds and more about where the bytes are allowed to sit.

FAQ

Can I run this in LM Studio or Ollama?
This post only verifies the llama.cpp path (b11048 or later), which is the engine GGUF targets. Treat other runners as untested here until you confirm they load this specific build.

Do I need the full 111 GB in RAM?
No. With 128 GB you can keep it resident for best speed. With 32 GB the OS memory-maps the GGUF and pages expert tensors on demand, trading throughput for a smaller footprint. You do need the full 111 GB on disk.

Why is the laptop so much slower than the server CPU?
Memory. The 32 GB laptop cannot hold the model, so it pages from disk, and only a few layers fit on the 8 GB GPU. The 16-thread server CPU with ample RAM keeps far more of the working set hot.

Is 4-bit always this lossless?
No, and do not generalize it. The 87.65% match is a measured paired result for this model and benchmark, enabled by keeping the trained delta (about 3% of the weights) at higher precision. Always verify your own quantization with a paired comparison rather than assuming parity.

What makes the active parameter count so low?
The MoE router picks 10 of 512 experts per token, so roughly 3B of the 180B parameters do work on any single token. That sparsity is what makes CPU inference and on-demand expert loading practical.

Where do I get it?
huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF and modelscope.cn/models/FINAL-Bench/POCKET-Darwin-180B-GGUF. The original is huggingface.co/FINAL-Bench/Darwin-180B-RSI.

Offline AI on a Phone: Thread Pinning, Lazy Loading, and a Safety Gate That Knows When to Stay Quiet

GINIGEN AI — Mon, 05 Oct 2026 08:56:19 +0000

TL;DR

HeliGO is Ginigen's offline disaster and survival app. In airplane mode it still gives you maps, contour lines, altitude, a compass, rescue coordinates, and an AI chat, all running from data and a model stored on the phone.
The model is Edge-4B-TELL, a 4-bit GGUF build based on Google's Gemma-4 E4B, published on Hugging Face (1,526 downloads and 35 likes at the time of writing).
Thread count is not a tuning detail on Android. With the screen off, our process was allowed only 4 cores. Whisper with 6 threads took 49.4 s; with 4 threads it took 2.1 s on the same phone (Galaxy S25). Read Cpus_allowed_list and size your thread pool from it.
Lazy-load rare features. Making the 0.92 GB vision projector always resident pushed model load time from about 10 s to about 1 minute, and slowed plain text chat too.
Do not trust a model's stated confidence. Asked to self-report confidence, the model was more confident on wrong answers (AUROC 0.441, worse than a coin flip). HeliGO uses a three-stage safety gate, and our TELL signal reads the model's internal state instead of its opinion.
Bonus from our sister company VIDRAFT: POCKET-Darwin-180B-GGUF shows the same "quantize without losing quality" idea at 180B scale: 111.3 GB, MMLU-Pro 87.65% vs 87.65% for the BF16 original.

Why build an AI app that never touches the network?

In a disaster, the network is usually the first thing to go. Cell towers fail in an earthquake. A hiker who loses the trail in the mountains loses signal at the same time. Yet the information you need right then (where to evacuate, how to treat an injury, which plant or mushroom not to eat) mostly lives on the internet.

HeliGO flips that assumption. Everything the app needs ships inside the phone. If you turn on airplane mode, nothing degrades. That is not a fallback mode; it is the only mode we design for.

Here is what is stored on the device:

Dataset	Records
Terrain tiles with elevation (auto-switching contour lines)	4,998
Summit elevations	16,568
Shelters, medical, water supply, and fire facilities	1,255
Korea Ministry of the Interior and Safety (MOIS) public action guidelines	75
Venomous species warnings	12

On top of that: nearest high ground for flood situations, sunrise and sunset, moon phase, and breadcrumb backtracking along the path you walked.

The AI is the newest layer, and it is also the one that needs the most care. Maps do not hallucinate. Language models do.

How do you run an LLM fully offline on a phone?

The short answer: a quantized GGUF model, llama.cpp compiled for ARM64, and a lot of discipline about memory and threads.

Our stack for the on-device model:

Model: Edge-4B-TELL. The inference weights are a verbatim mirror of Google's gemma-4-E4B-it-qat-q4_0-gguf. We host our own copy so that a disaster app does not break if an upstream path moves.
Files: a 4.8 GB text-generation GGUF and a 0.92 GB vision and audio projector (mmproj).
Runtime: llama.cpp, running as a native process under the app.
Speech: whisper.cpp for voice input.

Quantization-aware training (QAT) at 4 bits is what makes a 4B-class multimodal model fit a phone at all. A GGUF file packs the weights and the metadata llama.cpp needs, so the app can load it straight from local storage with no conversion step.

A generic way to try the same pattern on a development device (not our app's code, just the standard llama.cpp workflow):

# Build llama.cpp for Android with the NDK (on your workstation)
cmake -B build-android \
  -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-28 \
  -DGGML_OPENMP=OFF
cmake --build build-android -j --target llama-server llama-cli

# Push binary and model to the device
adb push build-android/bin/llama-server /data/local/tmp/
adb push gemma-4-E4B_q4_0-it.gguf /data/local/tmp/

# Run fully offline; bind to localhost only
adb shell "cd /data/local/tmp && ./llama-server \
  -m gemma-4-E4B_q4_0-it.gguf -c 4096 -t 4 \
  --host 127.0.0.1 --port 8080"

That flag -t 4 looks innocent. It is the single most important number in this article.

Why does thread count matter for on-device inference on Android?

Here is the incident. Our voice pipeline ran whisper with 4 threads. The Galaxy S25 has more cores than that, so we raised it to 6, expecting a speedup.

Translation latency jumped to about 56 seconds.

The cause: when the phone screen is off, Android moves a foreground-service app and its native child processes into a restricted cpuset. On the S25 that set was 4 cores. ggml's thread pool uses spin-waiting, and when you run more worker threads than you have cores, threads that are spinning steal time from threads that are doing real work. The pool does not degrade gracefully. It collapses.

Measured on the same phone, same 4 allowed cores, screen off:

whisper threads	Time
`-t 6`	49.4 s
`-t 4`	2.1 s

That is roughly a 23x difference from one integer.

Why our benchmarks missed it

We benchmarked through adb shell run-as <package>. A process started that way inherits the shell's cpuset, which allows all cores. So the benchmark said 6 threads was fine. The real app, with the screen off, saw only 4. adb benchmarks hide this class of bug.

The fix: read the allowed cores at runtime

Do not hardcode thread counts, and do not use the total core count. Ask the kernel what this process is actually allowed to use:

# Inside the app's own process context
grep Cpus_allowed_list /proc/self/status
# Cpus_allowed_list:  0-1,4-5   (example: 4 cores)

A small helper that turns that into a thread count:

fun allowedCoreCount(): Int {
    val line = java.io.File("/proc/self/status").readLines()
        .firstOrNull { it.startsWith("Cpus_allowed_list") } ?: return 4
    val spec = line.substringAfter(":").trim()
    return spec.split(",").sumOf { part ->
        val r = part.split("-").map { it.trim().toInt() }
        if (r.size == 2) r[1] - r[0] + 1 else 1
    }
}

val threads = minOf(allowedCoreCount(), 4)
// pass "-t $threads" to whisper / llama.cpp

Two practical rules we now follow:

Compute threads from Cpus_allowed_list, and cap them. Re-check when the app moves between foreground and background if your workload can run in both.
Benchmark under the real restriction. Pin the process to the same cores the app will get (for example taskset 33 for cores 0, 1, 4, 5) and test with the screen off, not only from an adb shell.

# Reproduce the restricted cpuset during a benchmark (mask 0x33 = cores 0,1,4,5)
adb shell "taskset 33 /data/local/tmp/whisper-cli -m ggml-base.bin -f test.wav -t 4"

Why should rare features be lazy-loaded on mobile?

The second lesson came from a fix that made things worse.

The vision projector (the mmproj file, 0.92 GB) is what lets the model look at a photo. We had a bug in the photo feature, and while fixing it we made the projector always load with the model. The photo feature worked again. But model load time went from about 10 seconds to about 1 minute, and ordinary text questions slowed down too.

What went wrong was not the code. It was the decision. We looked only at the broken feature and never measured the main path. Users chat with text all the time. They send a photo occasionally. "Keep it loaded so it is always ready" sounds safe, but it charges every user, every session, for a feature most sessions never touch.

The rule we adopted:

Before adding anything to global initialization, ask how often it is used.
Rare features load when first used (lazy loading), and can be released afterwards under memory pressure.
Measure the main path (cold start to first token for a text question) before and after every change. If you did not measure it, you did not fix it.

# Lazy pattern with llama.cpp: start text-only by default
./llama-server -m model.gguf -t $THREADS            # fast cold start

# Only when the user opens the camera feature, start (or switch to) a
# vision-capable instance with the projector:
./llama-server -m model.gguf --mmproj mmproj.gguf -t $THREADS

On a phone, memory and load time are the product. Every megabyte you keep resident is a megabyte the OS can use as a reason to kill you in the background.

How do you make an on-device AI refuse safely?

In a disaster app, a wrong answer can hurt someone. So HeliGO never passes the model's output straight to the user. Every answer goes through a three-stage safety gate:

Block life-threatening answers, regardless of confidence. An answer like "this is safe to eat" is stopped even if the model sounds certain.
Answer from official text when it applies. If the question matches one of the 75 MOIS public action guidelines, the app answers with the guideline text instead of the model's paraphrase.
Abstain when uncertain. If neither of the above settles it and the answer is not reliable, the app says it does not know.

The asymmetry principle

The gate is deliberately lopsided: it never says "safe", and it always says "poisonous".

The two possible errors do not cost the same. If the app wrongly withholds "safe", you skip a meal and stay hungry. If it wrongly says "safe" about something venomous or toxic, someone can die. When silence and speech carry such different costs, the design should not treat them symmetrically. This is less a machine learning decision than a product ethics decision, and we think every safety-critical on-device AI needs one written down.

Can you trust an LLM when it says how confident it is?

No, and our measurements were sharper than we expected.

We prompted the model to state its confidence with each answer on 665 Korean disaster-procedure questions. The average stated confidence was 0.863. When we used that number to separate correct from incorrect answers, the AUROC was 0.441. Anything below 0.5 means the signal points the wrong way: the model tended to sound more certain on the answers it got wrong.

So "Are you sure?" is not a weak safety check. It is a misleading one.

That is why we built TELL. Instead of asking the model for its opinion, TELL reads the model's internal state during generation and estimates whether the answer is likely to be correct. The readout ships alongside the weights in the Edge-4B-TELL repository, and the public numbers on the model card (held-out cross-validation on the same 665 questions) are:

Signal	AUROC
Self-reported confidence	0.441
Surface features (length, digits, formatting)	0.736
TELL (internal-state readout)	0.759

We publish the surface-feature baseline on purpose. TELL beats it by a real but modest margin, and we would rather show that honestly than claim a bigger win. Inside the app, the comparison that matters is TELL versus having no calibration signal at all.

Does 4-bit quantization destroy model quality?

Not if you are careful, and the evidence goes well beyond 4B.

Our sister company VIDRAFT recently released POCKET-Darwin-180B-GGUF, a 4-bit GGUF build of the 180B mixture-of-experts model Darwin-180B-RSI. Public facts from its model card:

Size: 111.3 GB in 4 files, down from 360 GB for the BF16 original.
Base layout: built on the widely used UD-Q4_K_XL GGUF layout.
Quality: MMLU-Pro, 2,000 paired questions: 87.65% vs 87.65% for the BF16 original.
Shorter answers: on the same 2,000 questions, answers averaged 3,694 tokens vs 4,322 for the same-format parent build.
Hardware: runs with llama.cpp (b11048 or newer), including a gaming laptop with an 8 GB GPU and 32 GB RAM (4.17 tokens/s) and a CPU-only server (18 to 21 tokens/s).
Sparsity: about 3B active parameters per token, with experts read on demand from SSD through memory mapping.

The phone and the 180B laptop build share one idea: edge AI is a systems problem, not just a model problem. Quantization makes the model fit. Threads, memory mapping, and load paths decide whether it is actually usable.

What is the practical checklist for shipping on-device LLMs?

Quantize with a format your runtime loads natively (GGUF for llama.cpp), and verify quality against the full-precision model on paired questions.
Size thread pools from /proc/self/status Cpus_allowed_list, not from the core count.
Benchmark with the screen off, under the real cpuset, not only through adb shell run-as.
Lazy-load rare modalities (vision projectors, extra adapters). Measure cold start to first token before and after every change.
Never ship raw model output in a safety-critical domain. Put a gate in front: block, defer to official text, or abstain.
Do not use self-reported confidence as a safety signal. Validate any confidence signal against a simple surface baseline.

FAQ

Does HeliGO really work in airplane mode?
Yes. Maps, contours, altitude, compass, rescue coordinates, facility lookups, and AI chat all run from data and a model stored on the phone. HeliGO does not replace emergency services; it is a tool for when you cannot reach them.

Which model runs inside HeliGO?
Edge-4B-TELL on Hugging Face. The inference weights are an unmodified mirror of Google's Gemma-4 E4B QAT Q4_0 GGUF; Ginigen's contribution is the TELL calibration readout shipped with it.

How many threads should I give llama.cpp or whisper.cpp on Android?
No more than the number of cores in your process's Cpus_allowed_list. On a Galaxy S25 with the screen off that was 4, and going to 6 made whisper about 23x slower.

Why not just ask the model if it is sure?
Because on our disaster-procedure questions the model's stated confidence ranked answers worse than chance (AUROC 0.441). Wrong answers came with higher stated confidence.

Can POCKET-Darwin-180B-GGUF run without a GPU?
According to its model card, yes: 18 to 21 tokens/s on one server CPU socket with llama.cpp b11048 or newer, and it also runs on a laptop with an 8 GB GPU and 32 GB RAM.

Why does the safety gate never say "safe"?
Because the costs are asymmetric. Wrongly withholding "safe" costs a meal; wrongly saying "safe" can cost a life.

DEV Community: GINIGEN AI

4-bit GGUF Quality for MoE Models: Why Only 3B of 180B Params Fire, and How to Prove Parity

TL;DR

Why does only a fraction of an MoE model activate per token?

Why does 4-bit hurt MoE differently than a dense model?

What does a UD-Q4_K_XL style mixed-precision graft preserve?

How do you verify quality parity instead of trusting the quant?

Step 1: produce the quantized GGUF

Step 2: score both models on the same benchmark

Step 3: accept or reject on the number

When is 4-bit MoE the right call, and when is it not?

FAQ

Further reading

Building an AI That Refuses to Say "Safe": Reliability Engineering for an Offline Disaster App

TL;DR

Why "just prompt it to be careful" fails offline

The 3-stage safety gate

The asymmetry principle

TELL: reading the model instead of asking it

What actually ships on the device

H2: How do you make an offline AI trustworthy without a bigger model to check it?

H2: Why not just use a larger model on-device?

FAQ

Takeaways for edge builders

Running a 180B Model on a Laptop With No GPU: How 4-bit GGUF Keeps Full Accuracy

TL;DR

How does a 180B model fit on a laptop?

Does 4-bit quantization hurt accuracy here?

How do you run POCKET-Darwin-180B-GGUF?

When is a CPU-only 180B model actually the right call?

FAQ

Further reading

Offline AI on a Phone: Thread Pinning, Lazy Loading, and a Safety Gate That Knows When to Stay Quiet

TL;DR

Why build an AI app that never touches the network?

How do you run an LLM fully offline on a phone?

Why does thread count matter for on-device inference on Android?

Why our benchmarks missed it

The fix: read the allowed cores at runtime

Why should rare features be lazy-loaded on mobile?

How do you make an on-device AI refuse safely?

The asymmetry principle

Can you trust an LLM when it says how confident it is?

Does 4-bit quantization destroy model quality?

What is the practical checklist for shipping on-device LLMs?

FAQ

Links