DEV Community: Jaydeep Shah (JD)

FP32, INT4, and Everything Between - What I Learned About Precision on Mobile

Jaydeep Shah (JD) — Mon, 18 May 2026 01:00:00 +0000

I was familiar with precision formats from my embedded systems work, but seeing labels like "INT4," "FP16," and "GPTQ" on HuggingFace model downloads hit differently. In that context, they are not just precision specs - they determine whether your model fits on a phone, how fast it runs, and how much accuracy you trade away. Here is the intuition that clicked once I started deploying to mobile.

I started by understanding what gets compressed

A neural network is a massive collection of numbers called weights. During inference, the model multiplies your input by these weights billions of times. Quantization answers one question: how precisely do you need to store each of those numbers?

The precision spectrum I had to learn

Each weight can be stored at different levels of precision:

Format	Bits per Weight	Relative Size	Typical Use
FP32 (full precision)	32 bits	1x (baseline)	Training, research
FP16 (half precision)	16 bits	0.5x	GPU inference, fine-tuning
FP8	8 bits	0.25x	Specialized silicon, datacenter inference
INT8	8 bits	0.25x	Server-side quantized inference
INT4	4 bits	0.125x	On-device / mobile deployment

Each step down roughly halves the memory footprint. A 2-billion parameter model at FP32 takes about 8 GB of memory (2 billion weights times 4 bytes each). At INT4, those same parameters fit in roughly 2.5 GB.

That is the difference between "will not fit on your phone" and "runs on your phone."

What these formats actually mean

FP32 stores each weight as a 32-bit floating-point number: a sign bit, 8 bits for the exponent, and 23 bits for the fractional part. This gives you roughly 7 decimal digits of precision. The weight 0.00314159 is stored almost exactly.

FP16 cuts that in half: 1 sign bit, 5 exponent bits, 10 fractional bits. About 3-4 decimal digits. That same weight might become 0.003143 - close, but not exact.

INT8 and INT4 are integer formats. Instead of floating-point representation, the weight range is mapped onto a fixed set of integer values. INT8 gives you 256 possible values. INT4 gives you just 16.

Think about that. A weight that could be any of billions of possible floating-point values is now forced into one of 16 buckets. That is aggressive compression. (In practice, INT4 quantization uses per-group scale factors, so the effective resolution is finer than 16 uniform buckets globally - but the core tradeoff holds.)

A note on FP8

FP8 sits in an interesting middle ground - 8 bits, but in floating-point format rather than integer. It preserves more of the dynamic range than INT8 while using the same amount of storage. I worked on FP8 optimization at the silicon level, and it is genuinely a compelling format for inference workloads where you need better accuracy than INT8 but cannot afford FP16's memory cost. Chip designers are increasingly building native FP8 support into AI accelerators precisely because of this tradeoff (NVIDIA Transformer Engine; Qualcomm AI Engine Direct).

The analogy that made it click for me

The easiest way I found to think about quantization: it is like reducing the number of decimal places in a measurement.

Imagine you are measuring the length of a room:

FP32: 3.14159 meters - laboratory precision
FP16: 3.142 meters - engineering precision
INT8: 3.1 meters - good enough for most construction
INT4: 3 meters - rough estimate

For most purposes, "3 meters" is fine. You can buy the right amount of carpet. But if you are doing precision cabinetry, rounding to the nearest meter will produce gaps.

This is exactly what happens inside a quantized model. For common, well-represented patterns, the model still produces good results. The weights do not need to be precise to the seventh decimal place because the overall pattern is strong enough. But for rare edge cases - unusual inputs, subtle distinctions, low-frequency knowledge - the model loses its ability to differentiate. The signal was in the decimals that got rounded away.

It is similar to reducing image resolution. A 4K photo and a 480p version both clearly show a person's face. But zoom in on the text on their T-shirt, and the 480p version is unreadable. Common features survive compression; fine details do not.

Real numbers: Gemma 4 E2B on a phone

Here is why this matters concretely. Gemma 4 E2B has 5.1 billion total parameters (2.3 billion effective, thanks to Per-Layer Embeddings). At full FP32 precision, it would need approximately 8 GB just for the weights - before accounting for the memory the operating system, other apps, and the inference runtime itself consume.

Most phones today have 6-12 GB of total RAM, shared across everything. An 8 GB model would leave virtually nothing for Android to run your UI, manage the camera, or keep background apps alive. The system would kill your app or crash.

At INT4, the standard GPU model compresses to 2.59 GB. That fits. That runs. That is why quantization is not optional for mobile - it is a prerequisite.

When we built Redacto, our on-device PII redaction app for the Qualcomm x Google LiteRT Developer Hackathon 2026, we shipped the standard Gemma 4 E2B model at 2.59 GB (INT4 quantization, specifically dynamic_wi4_afp32 - INT4 weights with FP32 activations). The fine-tuned version of the same model, exported with different quantization granularity, came in at 4.7 GB.

The performance difference was dramatic:

Metric	Standard (2.59 GB)*	Fine-tuned (4.7 GB)*
Avg latency per inference	5,693 ms	10,626 ms
Throughput	12.8 tok/s	9.0 tok/s
Avg current draw	101 mA	301 mA

*Different quantization granularity from different export pipelines - not a pure apples-to-apples comparison, but a real illustration of how model size drives hardware cost.

The larger model was 1.9x slower and drew 3x more power. On a phone, power draw translates directly to battery life and thermal throttling.

The tradeoff I did not expect

Here is the part nobody told me upfront. Quantization, fine-tuning, and model accuracy are locked in a three-way tradeoff:

You must quantize to run on mobile. There is no way around this. FP32 models do not fit.
Quantization degrades accuracy. You are rounding billions of numbers. Some information is lost.
Fine-tuning recovers accuracy on your specific task. By sharpening the model on exactly the patterns your app needs, you compensate for what quantization blurred.

This is why fine-tuning is not a luxury for on-device AI - it is part of the deployment strategy. You are already accepting precision loss from quantization. Fine-tuning lets you direct the remaining precision toward the things that matter most for your use case.

In our benchmarks, on one specific domain (tactical law enforcement data), the standard model scored 63.7% entity recall while the fine-tuned model scored 76.8%. That 13-point improvement came from training the model on domain-specific examples - teaching it to use its limited precision budget on the patterns that actually matter for that task. (The overall picture is more nuanced - I cover the full comparison in a later post in this series.)

Quantization schemes I encountered along the way

When you find quantized models in the wild, here are the formats you will see most often:

dynamic_wi4_afp32 - INT4 weights, FP32 activations. This is what LiteRT-LM uses for on-device export. The weights are aggressively compressed to INT4, but activations stay at full FP32 precision during inference. This preserves more accuracy than quantizing everything (LiteRT documentation).
GPTQ - A post-training quantization method that uses calibration data to minimize the error introduced by quantization. It processes the model layer by layer and adjusts remaining weights to compensate for rounding errors in already-quantized layers. Widely supported by the open-source ecosystem (Frantar et al., 2022).
AWQ (Activation-Aware Weight Quantization) - Observes which weights matter most by looking at activation magnitudes, then protects those important weights from aggressive quantization. Often produces better quality than GPTQ at the same bit width (Lin et al., 2023).
bitsandbytes - A library that provides easy INT8 and INT4 quantization integrated with the HuggingFace ecosystem. Commonly used for QLoRA fine-tuning, where the base model stays in INT4 and only the small LoRA adapter trains in higher precision (Dettmers et al., 2023).

Each has its own tradeoffs in quality, speed, and tooling compatibility. For on-device deployment via LiteRT-LM, dynamic_wi4_afp32 is currently the standard path.

What I took away from all this

Quantization is not a magic trick and it is not free. It is a deliberate engineering decision: trade precision for the ability to run on constrained hardware. The skill is in understanding exactly what you are trading away and whether your application can tolerate it.

For most common use cases, INT4 quantization works remarkably well. The model still understands language, still follows instructions, still generates coherent output. But the edges get soft. Rare patterns, subtle distinctions, unusual inputs - these are where you feel the loss.

If your app lives on those edges, invest in fine-tuning to sharpen them back up. If your app handles common cases, INT4 out of the box might be all you need.

Either way, now you know what the acronym actually means.

Sources

Gemma 4 Model Card - PLE architecture, parameter counts
LiteRT Documentation - on-device quantization formats
GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers - Frantar et al., 2022
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration - Lin et al., 2023
QLoRA: Efficient Finetuning of Quantized LLMs - Dettmers et al., 2023
NVIDIA Transformer Engine FP8 Documentation
Qualcomm AI Engine Direct Overview
Benchmark data: Redacto project, Samsung Galaxy S25 Ultra (Snapdragon 8 Elite, SM8750)

Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.

Last updated: May 2026
4th of 22 posts in the "Edge AI from the Trenches" series

What I Learned Untangling LiteRT, LiteRT-LM, and TFLite

Jaydeep Shah (JD) — Sun, 17 May 2026 23:15:00 +0000

When we started building Redacto - an on-device PII redaction app running Gemma 4 E2B on a Snapdragon 8 Elite - we kept tripping over three names: TFLite, LiteRT, and LiteRT-LM. Google's own docs sometimes use them interchangeably. Forum posts and community discussions mix them freely. We invested a good amount of time trying to figure out if they were the same thing, different versions, or completely separate tools.

Here is the distinction I wish someone had spelled out for me on day one.

The name that kept following me around: TFLite

TensorFlow Lite (TFLite) was Google's original on-device inference runtime, announced at Google I/O 2017 as the mobile companion to TensorFlow. It ran classical ML models - image classification, object detection, pose estimation - on phones and embedded devices. It consumed .tflite model files: small, optimized graphs dispatched to CPU, GPU, or specialized accelerators.

The rebrand that confused everyone: LiteRT

In September 2024, Google renamed TensorFlow Lite to LiteRT (short for "Lite Runtime"). The core runtime, APIs, and .tflite file format stayed the same. What changed was branding: LiteRT is no longer tied to TensorFlow. You can convert models from TensorFlow, PyTorch, JAX, or other frameworks. The old name implied a dependency that no longer existed.

The migration is still ongoing - you will find both names in code, packages, and docs. If you see org.tensorflow.lite in a Gradle dependency and LiteRT in Google's marketing, they are the same thing.

The one that is actually different: LiteRT-LM

This is where I got tripped up the longest. LiteRT-LM is not a rebrand. It is a separate runtime for running large language models on device, built on top of LiteRT. It adds capabilities the base runtime does not have:

Conversation management - system prompts, user turns, multi-turn history
Tokenization - BPE/SentencePiece tokenizer bundled with the model
Chat template handling - model-specific formatting (like Gemma's <start_of_turn> tags)
Streaming token output - callback-based delivery so your app shows text as it generates
Sampling configuration - temperature, top-k, top-p controls

LiteRT-LM consumes .litertlm files, not .tflite files. A .litertlm file is a compiled bundle: quantized weights, tokenizer, chat template, and an execution graph optimized for a specific device.

What I learned the hard way: they are not interchangeable

You cannot run a .litertlm file with plain LiteRT or TFLite. The base runtime has no concept of conversations, tokenizers, or streaming callbacks.
You cannot run a .tflite model with LiteRT-LM. The LLM runtime expects the bundled tokenizer and conversation-aware execution graph that only .litertlm provides.

They solve different problems. LiteRT runs classical ML inference. LiteRT-LM runs LLM inference with conversational scaffolding. In Redacto, we use LiteRT-LM exclusively - our pipeline sends system prompts through a multi-step conversation chain, with streaming callbacks and sampling configuration that do not exist in base LiteRT.

The stack that made it click

LiteRT-LM and MediaPipe both sit on top of LiteRT, but they serve different purposes and do not overlap. MediaPipe provides high-level task APIs (face detection, image segmentation) that use LiteRT as the engine underneath. LiteRT-LM provides conversational LLM inference. For a deeper runtime comparison including llama.cpp, ONNX Runtime, and ExecuTorch, see my earlier post on the HuggingFace-to-phone pipeline.

The short version

TFLite = the old name for Google's on-device ML runtime. Being replaced by LiteRT branding.
LiteRT = TFLite renamed, with broader framework support. Same runtime, same .tflite format.
LiteRT-LM = a separate runtime for LLMs, built on LiteRT. Different file format (.litertlm), different capabilities.

If you need an LLM on device, you want LiteRT-LM. If you need a classifier or detector, you want LiteRT. If you see "TFLite" in code, it is the old name for LiteRT.

The naming is confusing. But once you see the stack, it clicks.

Sources:

Last updated: May 2026
3rd of 22 posts in the "Edge AI from the Trenches" series

What I Learned Turning a HuggingFace Model Into Something My Phone Can Run

Jaydeep Shah (JD) — Sun, 17 May 2026 08:10:00 +0000

You found a model on HuggingFace. It looks promising - maybe Gemma, maybe Llama, maybe something smaller. You want to run it on a phone. You click "Download," and then... what? The file is 5 GB of .safetensors splits. There is no APK, no .tflite, no obvious next step. The HuggingFace README says "Usage: model = AutoModelForCausalLM.from_pretrained(...)" - a Python API that does not exist on Android.

This was exactly where I got stuck when I started building Redacto - an on-device PII redaction app running Gemma 4 E2B entirely on a Galaxy S25 Ultra. The distance between "I found a good model" and "it runs on my phone" turned out to be much bigger than I expected. Here is what I learned along the way.

What is actually inside a HuggingFace repo

The first thing I had to understand was what I was actually downloading. HuggingFace model repos for LLMs are not just a single weights file - they contain an entire ecosystem of artifacts.

Here is what I found inside a typical LLM repo:

Model weights (.safetensors or .bin files) - the learned parameters. For Gemma 4 E2B, this is a single 10.2 GB file. Larger models split weights across multiple files (model-00001-of-00004.safetensors, etc.).
config.json - the model architecture definition: number of layers, hidden dimensions, attention heads, vocabulary size. This is the blueprint the runtime needs to reconstruct the model's computational graph.
Tokenizer files (tokenizer.json, tokenizer_config.json, tokenizer.model) - the mapping between text and token IDs. The model does not see words; it sees integers. The tokenizer defines how "Mrs. Chen" becomes [4521, 18, 9832] (or whatever the model's vocabulary dictates).
chat_template (embedded in tokenizer_config.json or as a standalone .jinja file) - a Jinja2 template that wraps your messages into the format the model was trained on. Gemma expects <start_of_turn>user\n...<end_of_turn>. Llama expects [INST]...[/INST]. Get this wrong and the model produces garbage without any error message.
Model card (README.md) - documentation covering training data, intended use, limitations, license, and benchmark scores.

Not all repositories are the same. Some are raw research checkpoints with no tokenizer. Some include GGUF quantized versions alongside the originals. I learned quickly that for on-device deployment, you want repositories that have already been prepared for your target runtime.

Why I could not just download and run

This was my first real lesson. Those .safetensors files are PyTorch-format tensors at full or half precision. They are designed to be loaded into GPU memory on a workstation running Python. A phone cannot use them directly, and it took me a while to understand why:

Size. Gemma 4 E2B at FP32 is roughly 10 GB. A phone with 12 GB of RAM cannot load that while also running the OS, the app, and everything else. You need quantization - compressing weights from 32-bit floats to 4-bit integers - to get the model down to a size that fits.

Format. Android does not have a PyTorch runtime. The phone's CPU, GPU, and NPU each have their own instruction sets and memory layouts. The model's computational graph needs to be compiled into operations that these hardware targets can execute.

Runtime. An LLM is not just a forward pass. You need conversation management (tracking turns), tokenization (text to integers and back), KV-cache management (storing previous computations so generation does not recompute from scratch every token), and streaming (delivering tokens one at a time for responsive UX). The raw weights have none of this.

Once I understood these three gaps, the compilation pipeline started to make sense.

The pipeline I had to learn

Getting from HuggingFace to a phone-ready model turned out to be a multi-stage process:

Quantization - making it fit

Quantization converts the model's weights from high-precision floating point (FP32 or FP16) to lower-precision integers (INT8 or INT4). It is a lossy compression step - the model loses some accuracy, but the file shrinks dramatically.

For Gemma 4 E2B, INT4 quantization (specifically dynamic_wi4_afp32 - INT4 weights with FP32 activations) brings the model from ~10 GB down to ~2.58 GB. That is the difference between "impossible on a phone" and "fits in memory with room for the app."

Quantization is not optional for mobile. It is a hard requirement. This was not obvious to me at first - I kept looking for ways around it before accepting that every on-device model goes through this step.

Export and compilation - the tool I did not know existed

This is where I discovered litert-torch - a pip-installable package from Google that takes a HuggingFace model and produces a .litertlm file. During export, the tool:

Reads the model architecture from config.json
Loads and quantizes the weights
Embeds the tokenizer
Embeds the chat template (from tokenizer_config.json or an override)
Compiles the computational graph into LiteRT operations
Packages everything into a single .litertlm bundle

The official export command from Google's Gemma 4 documentation:

litert-torch export_hf \
  --model=google/gemma-4-E2B-it \
  --output_dir=/tmp/gemma4_2b \
  --externalize_embedder \
  --jinja_chat_template_override=litert-community/gemma-4-E2B-it-litert-lm

The --model flag takes a HuggingFace model ID or local path. --externalize_embedder separates the embedding table for memory efficiency. The --jinja_chat_template_override points to a known-compatible chat template - this flag exists because some model templates use Jinja features that the on-device parser does not support. I learned this the hard way, and I cover that story in a later post in this series.

For fine-tuned models, you can point --model at a local directory and add quantization:

litert-torch export_hf \
  --model=./my_finetuned_model \
  --output_dir=./output \
  --externalize_embedder \
  --quantization_recipe=dynamic_wi4_afp32 \
  --jinja_chat_template_override=litert-community/gemma-4-E2B-it-litert-lm

Device-specific variants - one model, two files

This was a surprise. The exported .litertlm runs on CPU and GPU out of the box. But if you want to target the NPU (Neural Processing Unit) - which on a Snapdragon 8 Elite delivers 41.7 tok/s versus 24.5 tok/s on GPU - you need a second compilation step.

The NPU variant goes through the Qualcomm QNN toolchain, which compiles certain operations into DISPATCH_OP custom ops that run directly on the Hexagon V79 DSP. This produces a separate, larger .litertlm file that is tied to a specific chip.

The standard GPU/CPU file works across all ARM64 Android devices. The NPU file works only on the exact SoC it was compiled for. I did not expect to need two different model files for what is technically the same model.

Push to device - the easy part

The final step turned out to be the simplest:

adb push gemma-4-E2B-it.litertlm \
  /sdcard/Android/data/com.example.redacto/files/gemma4.litertlm

The model lives in the app's private external storage. On first load, LiteRT-LM parses the bundle, sets up the KV-cache, initializes the appropriate hardware delegate, and for NPU generates an AOT (ahead-of-time) compilation cache. Cold start takes about 10 seconds for GPU and 14 seconds for NPU. With the AOT cache in place, subsequent launches drop to around 2 seconds.

How it actually went with Redacto

For Redacto, we did not run the export pipeline ourselves for the production model. This is something I wish I had known earlier: Google's litert-community organization on HuggingFace publishes pre-compiled .litertlm files for popular models, including Gemma 4 E2B.

We downloaded from litert-community/gemma-4-E2B-it-litert-lm:

File	Size	Target Hardware
`gemma-4-E2B-it.litertlm`	2.59 GB	CPU/GPU (all ARM64 Android devices)
`gemma-4-E2B-it_qualcomm_sm8750.litertlm`	3.02 GB	Snapdragon 8 Elite NPU only

Two files. Same model. Same weights at the same precision. The size difference (~430 MB) comes from the QNN-compiled custom ops embedded in the NPU variant. Those ops will crash if you try to run them on a GPU - and the GPU variant cannot dispatch to the NPU. They are not interchangeable.

We did, however, run litert-torch export_hf ourselves when we fine-tuned the model. That is when we hit the chat template trap: tokenizer.save_pretrained() bundled the HuggingFace-native Jinja template (which uses map.get()), and LiteRT-LM's on-device template parser does not support that function. The model loaded, the tokenizer initialized, and then inference produced garbage. No error, no crash - just wrong output. We had to manually swap the template with an older compatible version before re-exporting. (I cover this trap in detail in a later post in this series.)

What I learned about the runtime landscape

As I went through this process, I also had to figure out where LiteRT-LM fits among the other on-device inference options. Here is the comparison I wish I had found earlier:

Runtime	Focus	Model Format	Hardware Targets	LLM Features
LiteRT-LM	LLMs on Android/iOS	`.litertlm`	CPU, GPU, NPU (Qualcomm, MediaTek)	Tokenizer, chat template, KV-cache, streaming, conversation management
TFLite / LiteRT	Classical ML (image, audio, NLP)	`.tflite`	CPU, GPU, NPU, Edge TPU	None (no tokenizer, no chat, no streaming)
ONNX Runtime	Cross-platform inference	`.onnx`	CPU, GPU (DirectML, CUDA)	Limited (community extensions)
llama.cpp	LLM inference, CPU-focused	`.gguf`	CPU (NEON/AVX), some GPU (Metal, CUDA, Vulkan)	Tokenizer, chat template, KV-cache, streaming
MediaPipe	ML pipelines for media tasks	`.tflite` + config	CPU, GPU	LLM Inference API (wraps LiteRT under the hood)

The distinctions that mattered most for my use case:

LiteRT-LM vs TFLite/LiteRT. I initially confused these. TFLite (now rebranded as LiteRT) handles classical ML models - image classifiers, object detectors. LiteRT-LM is built on top of LiteRT specifically for LLMs. You cannot run a .litertlm file with the TFLite interpreter, and you cannot run a .tflite model with LiteRT-LM. Same infrastructure, different model types.

LiteRT-LM vs llama.cpp. llama.cpp is excellent for CPU-based inference on laptops and desktops. On phones, LiteRT-LM's advantage is its deep integration with vendor-specific NPU delegates - on Snapdragon 8 Elite, NPU inference through LiteRT-LM runs at 41.7 tok/s versus the ~10-15 tok/s range typical of CPU-only execution for a model this size.

LiteRT-LM vs MediaPipe. Google's MediaPipe provides a higher-level LLM Inference API that wraps LiteRT under the hood. Simpler API, less control. LiteRT-LM gives you more control over engine initialization, backend selection, and sampling configuration - which I needed for our multi-step redaction pipeline.

The mental model that made it click

If I had to summarize everything I learned in one sentence: a HuggingFace model repository is the source code, and a .litertlm file is the compiled binary.

You would not try to run a .c file on a microcontroller without compiling it first. You would not try to load .safetensors on a phone. Both need a compilation step that transforms human-friendly source into machine-friendly executable. The difference is that model compilation also includes quantization (compression), hardware specialization (targeting specific silicon), and runtime packaging (embedding the tokenizer, chat template, and execution graph).

Once this clicked for me, the rest of the on-device AI stack made sense. The .litertlm file is not just a model - it is a self-contained inference package that knows how to tokenize input, format conversations, run the forward pass on your target hardware, and stream output back to your app.

Finding the right model on HuggingFace is step one. Getting it to your phone is the actual engineering.

Related in this series

What I Learned Untangling LiteRT, LiteRT-LM, and TFLite - clarifies the naming confusion in the stack this post describes
FP32, INT4, and Everything Between - what quantization does and why it is mandatory for mobile
What's Inside a .litertlm File? - deep dive into the compiled bundle at the end of the pipeline
The Chat Template Trap - what happens when the export pipeline embeds an incompatible template

Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware.
Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.

Sources:

HuggingFace Model Hub
litert-community/gemma-4-E2B-it-litert-lm - pre-compiled .litertlm files
Gemma 4 export documentation - official export commands
litert-torch on PyPI - the export tool
LiteRT-LM overview

Last updated: May 2026
2nd of 22 posts in the "Edge AI from the Trenches" series

What I Learned by Dissecting gemma-4-E2B-it_qualcomm_sm8750.litertlm

Jaydeep Shah (JD) — Sun, 17 May 2026 05:00:00 +0000

I recently started exploring on-device AI inference, and honestly, the initial experience was overwhelming. Hundreds of models on HuggingFace, unfamiliar architecture names, quantization formats, chip-specific variants - it felt like drinking from a firehose. When I first saw a filename like gemma-4-E2B-it_qualcomm_sm8750.litertlm, it looked like alphabet soup.

But as I dug deeper - reading model cards, building an actual app, benchmarking on real hardware - each piece of that filename started to make sense. Every segment encodes a specific decision about the model's lineage, its architecture, how it was trained, what chip it was compiled for, and what runtime will execute it.

This post breaks that filename apart, piece by piece, so you do not have to go through the same confusion I did.

The anatomy of the name

`gemma` - the model family

Gemma is Google's family of open-weight large language models. "Open-weight" means Google publishes the trained model weights so you can download, deploy, fine-tune, and build on them without per-token API fees or cloud dependencies.

But Gemma is not one model - it is four generations of architectural evolution, each shifting what "small enough for a phone" actually means. Understanding this lineage matters because the generation determines the architecture, the context window, the chat template format, and ultimately what your on-device app can do.

The Gemma family tree

A few things worth noticing in this evolution:

Attention mechanisms evolved with each generation. Gemma 1 used MQA on the small model (fewer key-value heads = less memory) and standard MHA on the large model. Gemma 2 unified everything to GQA - a middle ground where key-value heads are shared across groups of query heads, reducing KV-cache size without the quality loss of MQA. This sliding-window + global attention alternation in Gemma 2 was a direct response to the memory bandwidth problem during autoregressive decoding - the exact bottleneck that matters on a phone.

The licensing change in Gemma 4 is significant. Generations 1-3 used Google's custom "source-available" license with usage restrictions. Gemma 4 moved to Apache 2.0 - fully permissive, no usage restrictions. This matters for commercial on-device apps: you can ship Gemma 4 in a production app without worrying about license compliance beyond standard Apache 2.0 attribution.

The context window grew dramatically. 8K (Gemma 1-2) to 128K (Gemma 3) to 128K on edge / 256K on cloud models (Gemma 4). The E2B model we run on the Galaxy S25 Ultra has a 128K context window - the same as Gemma 3. The 256K window is reserved for the heavyweight 26B MoE and 31B dense models. On-device, you rarely use the full window anyway - a 4K KV-cache is common for mobile deployment - but the larger training window means the model understands longer documents even when you truncate at inference time.

`4` - the generation

The number tells you which generation of the Gemma family this model belongs to. Higher generation means better quality at the same parameter count - improved architecture, better training data, and lessons from prior generations baked in.

For on-device developers, the generation also determines which chat template the model expects. Gemma 3 and 4 use <start_of_turn>user\n...<end_of_turn> formatting. Older generations use different markup. Getting this wrong does not produce an error - it produces garbage output. (More on this silent failure mode in a future post on chat templates.)

`E2B` - effective 2 billion parameters

This is the most misunderstood part of the name, and it is worth getting right.

The E does not stand for a generic prefix. According to Google's model card on HuggingFace, it stands for effective parameters. The smaller Gemma 4 models (E2B and E4B) use a technique called Per-Layer Embeddings (PLE) that fundamentally changes how parameters are counted.

What Per-Layer Embeddings do

In a standard transformer, there is one shared embedding table that converts tokens to vectors at the input and converts vectors back to tokens at the output. Every decoder layer in between works with the same token representations.

PLE gives each decoder layer its own small embedding table for every token. Instead of sharing one embedding, each layer has a private lookup table that adapts the token representation to that layer's specific role in the network.

Why this changes the parameter count

These per-layer embedding tables are large in raw parameter count - they add up significantly. Gemma 4 E2B has 5.1 billion total parameters, but only 2.3 billion active/effective parameters. The difference is entirely PLE overhead.

But those PLE tables are lookup tables, not compute-heavy matrix multiplications. A lookup is an O(1) memory read per token, not an O(n) matrix multiply. So while the chip has to hold 5.1B parameters worth of memory-mapped weights, the execution engine only computes 2.3B parameters worth of matrix multiplications per token step.

This is the key insight: PLE maximizes parameter efficiency specifically for on-device deployment. You get the quality benefits of having more specialized parameters (5.1B of them), but the inference cost (latency, memory bandwidth, power) stays in the ~2B compute class.

What this means for your phone

Variant	Total Params (with PLE)	Active/Effective Params	Base .litertlm Size	Context	Target Hardware
E2B	5.1B	2.3B	~2.58 GB	128K	Phones, Raspberry Pi, edge devices
E4B	7.9B	4.5B	~3.65 GB	128K	High-end phones (12+ GB RAM)
26B MoE	26B	Subset active per token	Large	256K	Workstations, servers
31B Dense	31B	All active	Large	256K	Servers, high-end GPUs

For Redacto - our on-device PII redaction app - we chose E2B because even with 5.1B total parameters, the INT4-quantized .litertlm is only ~2.58 GB, leaving headroom for the OS, ML Kit OCR, and four independent LLM conversations running in our redaction pipeline, all on a Galaxy S25 Ultra with 12 GB RAM. The PLE architecture means we get quality from 5.1B parameters worth of specialization, at the compute cost of only 2.3B active parameters - a budget that fits within a mobile power envelope.

`-it` - instruction-tuned

This suffix changes everything about how the model behaves.

A base model (no -it) is trained to predict the next token. Give it text, it continues it. It does not follow instructions or understand "system prompt" vs "user message."

An instruction-tuned model (-it) is the base model further trained on instruction-response pairs. It understands conversational structure: system prompts, user turns, assistant responses. It follows directions.

For on-device apps, -it is almost always what you want. In Redacto, each pipeline step sends a system prompt like "You are a medical PII detector. Find all names, dates of birth, medical record numbers..." A base model would ignore this and generate plausible-looking but unstructured text. The -it variant follows the prompt, stays in role, and produces structured output that the next pipeline step can parse.

If you see two versions on HuggingFace - with and without -it - and your app needs the model to follow instructions, pick -it.

`_qualcomm_sm8750` - the compilation target

This suffix tells you the model has been compiled for a specific chip: qualcomm is the vendor, sm8750 is the Snapdragon 8 Elite system-on-chip.

Modern phones have specialized AI silicon - a Neural Processing Unit (NPU) - that runs neural network operations far faster than the CPU or GPU. But using it requires translating the model's computation graph into the chip's native instruction format. Same concept as compiling C for x86 vs arm64 - the math is identical, the binary is not.

File	Target	Decode Speed	TTFT
`gemma-4-E2B-it.litertlm`	CPU / GPU (generic)	24.5 tok/s	366ms
`gemma-4-E2B-it_qualcomm_sm8750.litertlm`	Snapdragon 8 Elite NPU	41.7 tok/s	92ms

The NPU variant is larger (2.8 GB vs 2.4 GB) because it bundles QNN-compiled execution graphs with DISPATCH_OP custom operations targeting the Hexagon V79 DSP. But the speed gain is substantial: 1.7x faster decode throughput, 4x faster time-to-first-token. No chip suffix means the generic variant that runs on any ARM CPU or mobile GPU. (I cover NPUs and hardware delegates in detail in upcoming posts in this series.)

`.litertlm` - the file format

The file extension for LiteRT-LM, Google's on-device LLM inference runtime. A .litertlm file is not raw weights - it is a compiled bundle containing quantized weights, tokenizer, chat template, and execution graph, all packaged for immediate on-device inference.

You cannot fine-tune a .litertlm file. It is the end of the pipeline: train on cloud, fine-tune (optionally), quantize, compile, package, deploy. It is distinct from .tflite (classical ML), .gguf (llama.cpp), .onnx (cross-platform), or .safetensors (raw weights for storage/transfer). (I dig into the internals of this format in a separate post.)

Navigating HuggingFace

Two organizations matter when searching for Gemma models ready for on-device deployment:

google - official weights in .safetensors format, for fine-tuning or conversion. This is where you start if you need to customize the model.
litert-community - pre-compiled .litertlm bundles, including chip-specific NPU variants, ready for on-device use. This is where you go if you want to run inference immediately.

Check the model card for quantization level, supported hardware, and license. Look at the file listing for chip-specific variants (_qualcomm_sm8750, etc.). For production on-device apps, litert-community gives you deployment-ready files; google gives you the starting point for fine-tuning.

Putting it all together

Every segment in Redacto's model filename maps to a decision we made:

Segment	What it means	Why we chose it
`gemma`	Google's open-weight LLM family	Apache 2.0, strong on-device ecosystem, Google-backed
`4`	Fourth generation	Best quality per parameter, Apache 2.0 license, PLE architecture
`E2B`	5.1B total / 2.3B effective via PLE	~2.58 GB at INT4, fits in phone RAM with headroom for OCR + 4 LLM calls
`it`	Instruction-tuned	Follows system prompts - critical for our 4-step redaction pipeline
`qualcomm_sm8750`	Compiled for Snapdragon 8 Elite NPU	41.7 tok/s, 92ms TTFT on Hexagon V79
`.litertlm`	LiteRT-LM compiled bundle	Tokenizer + chat template + weights + graph, ready to infer

The next time you see a model filename that looks like alphabet soup, read it left to right: family, generation, size architecture, training variant, target hardware, runtime format. Each piece narrows what the model is, what it can do, and where it can run.

Related in this series of "Edge AI from the Trenches"

From HuggingFace to Your Phone - the next logical read: what happens after you decode the filename
What Does "On-Device" Actually Mean? - the privacy and deployment context behind choosing an on-device model

Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware.
Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.

Sources:

Google Gemma 4 E2B Model Card - PLE architecture, effective parameter explanation
Google Gemma announcements (Gemma 1, 2, 3)
litert-community on HuggingFace
LiteRT documentation
Benchmark data: Redacto project, Samsung Galaxy S25 Ultra (Snapdragon 8 Elite, SM8750)

Last updated: May 2026
1st out of 22 posts in the "Edge AI from the Trenches" series

DEV Community: Jaydeep Shah (JD)

FP32, INT4, and Everything Between - What I Learned About Precision on Mobile

I started by understanding what gets compressed

The precision spectrum I had to learn

What these formats actually mean

A note on FP8

The analogy that made it click for me

Real numbers: Gemma 4 E2B on a phone

The tradeoff I did not expect

Quantization schemes I encountered along the way

What I took away from all this

Sources

What I Learned Untangling LiteRT, LiteRT-LM, and TFLite

The name that kept following me around: TFLite

The rebrand that confused everyone: LiteRT

The one that is actually different: LiteRT-LM

What I learned the hard way: they are not interchangeable

The stack that made it click

The short version

What I Learned Turning a HuggingFace Model Into Something My Phone Can Run

What is actually inside a HuggingFace repo

Why I could not just download and run

The pipeline I had to learn

Quantization - making it fit

Export and compilation - the tool I did not know existed

Device-specific variants - one model, two files

Push to device - the easy part

How it actually went with Redacto

What I learned about the runtime landscape

The mental model that made it click

Related in this series

What I Learned by Dissecting gemma-4-E2B-it_qualcomm_sm8750.litertlm

The anatomy of the name

gemma - the model family

The Gemma family tree

4 - the generation

E2B - effective 2 billion parameters

What Per-Layer Embeddings do

Why this changes the parameter count

What this means for your phone

-it - instruction-tuned

_qualcomm_sm8750 - the compilation target

.litertlm - the file format

Navigating HuggingFace

Putting it all together

Related in this series of "Edge AI from the Trenches"

`gemma` - the model family

`4` - the generation

`E2B` - effective 2 billion parameters

`-it` - instruction-tuned

`_qualcomm_sm8750` - the compilation target

`.litertlm` - the file format