In Part 1, we started with a simple question:
Does every AI task need the biggest model we can afford?
Often, the answer is no.
A model that is considerably smaller than a frontier model can be the better choice for a well-defined workload—especially when latency, cost, privacy, or on-device execution matters.
But that immediately creates a harder question:
How do we build a smaller model that is still good enough?
This is where the conversation moves from strategy to engineering.
You cannot take a 400-billion-parameter model, delete 99% of its parameters, and expect the remaining billion parameters to magically retain everything the original model knew.
Model compression is not a file-size problem.
It is a capability-preservation problem.
And there are several different tools for attacking it.
The most important ones are:
Knowledge distillation
Quantization
Pruning
Fine-tuning
Parameter-efficient fine-tuning, especially LoRA and QLoRA
They sound similar because they all help make AI systems more efficient.
Under the hood, however, they do very different things.
First, let's separate two problems
Before getting into the techniques, there is an important distinction.
Suppose you have a 7B model and your application needs a 3B model.
There are actually two questions:
How do I make the model cheaper to run?
and:
How do I make the model better at my particular task?
Quantization primarily attacks the first problem.
Fine-tuning primarily attacks the second.
Distillation can address both, because it can transfer useful behavior from a larger teacher into a smaller student.
This distinction is useful because you can combine these techniques.
For example:
Large Teacher
|
| Distillation
v
Small Base Model
|
| Fine-tuning / LoRA
v
Domain Specialist
|
| Quantization
v
Efficient Deployment
That is not one optimization.
It's a pipeline.
1. Knowledge Distillation: Teach the Small Model
Let's start with the technique most closely associated with the idea of transferring capability from a larger model.
Knowledge distillation uses a larger model—the teacher —to help train a smaller student model.
The concept predates today's LLMs. Hinton, Vinyals, and Dean described a method for transferring knowledge from an ensemble of models into a smaller model that is easier to deploy.
The basic idea is beautifully simple.
Instead of asking the student to learn only from the original training labels, we also let it learn from the behavior of the teacher.
Conceptually:
Input
|
+-----+------+
| |
v v
Teacher Student
| |
v v
Teacher output Student output
| |
+-----+------+
|
v
Loss
|
v
Update Student
The teacher already contains useful information about the task.
The student learns to approximate that behavior.
Hard targets vs. soft targets
This is one of the most important ideas in traditional knowledge distillation.
Imagine a classification problem with three classes:
Cat 0.92
Dog 0.07
Rabbit 0.01
The hard label might simply be:
Cat
The hard label tells the student which answer is correct.
The soft distribution tells it something more subtle:
"This is overwhelmingly a cat, but it has some resemblance to a dog and very little resemblance to a rabbit."
That additional structure can contain useful information.
The original distillation work showed how these soft targets can transfer information from a larger model into a smaller one.
For modern language models, the picture becomes more complicated because the output is a sequence of tokens rather than a single class.
But the principle remains:
Don't just teach the student the answer. Teach it something about how the teacher behaves.
Distillation for LLMs is more complicated
With a language model, the teacher might generate:
"The crash is most likely caused by an invalid pointer dereference in the native library."
A straightforward approach is to train the student to reproduce that response.
But there are many ways to distill an LLM.
You can transfer:
generated answers,
token-level probabilities,
reasoning-oriented examples,
task-specific demonstrations,
intermediate representations,
preferences,
or specialized behavior.
Modern LLM distillation research has become a large field in its own right, with different approaches targeting algorithms, skills, and domain specialization.
This gives us an important correction to a common oversimplification:
Distillation isn't simply "copy the big model into the small model."
It is a family of training techniques for transferring useful behavior.
Why distillation is so attractive for SLMs
Imagine a large model is excellent at a particular task but far too expensive to deploy at scale.
You can use that model offline as a teacher.
Generate high-quality examples.
Train a smaller model on those examples.
Then deploy the smaller model for the high-volume workload.
The expensive teacher doesn't necessarily have to serve every production request.
That gives us an architecture like this:
EXPENSIVE / OFFLINE
+---------------+
| Large Teacher |
+-------+-------+
|
Generate examples
|
v
+---------------+
| Training Data |
+-------+-------+
|
v
+---------------+
| Small Student |
+-------+-------+
|
v
CHEAP / ONLINE
Millions of requests
|
v
Small Model
This separation between expensive capability acquisition and cheap production inference is one of the reasons distillation is so interesting for SLMs.
But distillation has a catch
A student cannot learn what the teacher never demonstrates.
If your distillation dataset contains only easy questions, the student may become excellent at easy questions and terrible at edge cases.
If the teacher itself makes systematic mistakes, those mistakes can be transferred too.
And if the student is substantially smaller than the teacher, there is a limit to how much information it can absorb.
So a good distillation pipeline isn't:
Teacher → dump outputs → train student → ship.
It's closer to:
Teacher
|
v
Generate candidate data
|
v
Filter / validate
|
v
Balance easy + hard examples
|
v
Train student
|
v
Evaluate against real workloads
|
+----> Fail? ----> Improve dataset
|
v
Deploy
The dataset becomes part of the engineering
2. Quantization: Fewer Bits, Less Baggage
Distillation changes the model itself.
Quantization attacks representation.
Neural-network parameters are stored as numerical values.
A model might commonly use formats such as:
FP32 — 32-bit floating point
FP16 — 16-bit floating point
BF16 — 16-bit brain floating point
INT8 — 8-bit integer
INT4 — 4-bit integer
The basic intuition is straightforward:
If we can represent the model's numbers using fewer bits, we can reduce its memory footprint.
For example, ignoring overhead and implementation details, storing a billion parameters at 16 bits requires roughly 2 GB just for the weights.
At 8 bits, it is roughly 1 GB.
At 4 bits, roughly 0.5 GB.
Those are simplified calculations, because real systems contain additional metadata, scaling factors, buffers, KV cache, runtime overhead, and other components.
But the intuition is important.
The numerical representation matters
Quantization isn't just "round the numbers"
A naive approach would be:
FP16 value
|
v
Round it
|
v
INT4 value
But neural networks contain distributions that aren't always friendly to naive quantization.
Some values matter disproportionately.
That means good quantization methods use scaling, calibration, mixed precision, or other techniques to preserve important information.
LLM.int8(), for example, demonstrated an approach that handled outlier features separately while performing the majority of multiplication in 8-bit precision. The authors reported that this allowed large models to be run with substantially lower memory requirements without the performance degradation they observed from simpler approaches.
The broader lesson is:
Quantization is an optimization problem, not simply a bit-counting exercise.
What quantization buys you
Depending on the model, hardware, and runtime, quantization can provide:
lower model memory,
lower bandwidth requirements,
the ability to run models on smaller GPUs,
improved feasibility for CPU or edge inference,
and potentially higher throughput.
But there are trade-offs.
Quantization can affect:
accuracy,
perplexity,
reasoning performance,
generation quality,
and sometimes latency.
And the trade-off isn't identical for every model.
A 4-bit model isn't automatically "better" than a 8-bit model.
You need to measure.
3. Pruning: Remove What You Don't Need
Quantization changes how parameters are represented.
Pruning tries to remove parameters or structures altogether.
The intuition is similar to trimming a tree.
Some branches contribute more than others.
If certain weights contribute very little to the final behavior, perhaps they can be removed.
There are several forms of pruning, including:
unstructured pruning,
structured pruning,
neuron pruning,
head pruning,
layer pruning,
and other architecture-specific approaches.
The trade-off is that sparsity is only useful if the hardware and runtime can exploit it.
Imagine removing 50% of the weights but still performing essentially the same dense matrix multiplication.
You may have a smaller file.
You may not have a faster model.
This is an important engineering distinction:
Compression does not automatically translate into acceleration.
A technique can reduce storage while providing little real-world latency benefit.
The three techniques so far
At this point, we can summarize the difference:
| Technique | What changes? | Primary goal |
|---|---|---|
| Distillation | Training behavior | Transfer capability |
| Quantization | Numerical representation | Reduce memory/compute cost |
| Pruning | Model structure | Remove unnecessary computation |
| Fine-tuning | Model behavior | Specialize for a task |
| LoRA / QLoRA | Trainable parameters | Make adaptation cheaper |
And these techniques can be combined.
That's where things get powerful.
4. Fine-tuning: Make the Model Care About Your Problem
Pre-trained language models are generalists.
They have learned from enormous and diverse datasets.
But your application probably doesn't need a generalist.
It needs something specific.
Imagine a device diagnostics application.
The model doesn't need to be an expert in Shakespeare.
It needs to understand:
crash signatures,
error messages,
device metadata,
component names,
severity levels,
and your organization's troubleshooting vocabulary.
Fine-tuning allows us to adapt a pre-trained model to a particular task or domain.
Instead of starting from zero:
Random weights
|
v
Train enormous model
|
v
General-purpose model
we start from an existing model:
Pre-trained model
|
v
Task-specific data
|
v
Fine-tuned specialist
This is dramatically more practical.
But traditional fine-tuning still has a problem.
You have to update the model's parameters.
For a large model, that can be expensive.
That's where parameter-efficient fine-tuning enters.
5. LoRA: Don't Rewrite the Whole Model
LoRA—Low-Rank Adaptation of Large Language Models—takes a clever approach.
Instead of updating all of the original model weights, LoRA freezes the pre-trained model and introduces small trainable matrices into the model's layers.
Conceptually:
Original Model
+--------------+
| Frozen |
Input ------>| Weights |----+ +--------------+ | +----> Output +--------------+ | Input ------>| LoRA Adapter |----+ | Trainable | +--------------+
The base model stays frozen.
The adapter learns the task-specific modification.
The original LoRA paper demonstrated that this can dramatically reduce the number of trainable parameters and memory requirements compared with full fine-tuning while achieving comparable or better quality on the tasks they evaluated.
This changes the economics of specialization.
Instead of storing a complete copy of a model for every task, you can conceptually maintain:
Base Model
|
+------------+------------+
| | |
v v v
Adapter A Adapter B Adapter C
Medical Support Coding
The same base model can therefore support multiple specialized behaviors.
6. QLoRA: Quantization Meets LoRA
Now combine two ideas.
LoRA says:
Don't update the entire model.
Quantization says:
Don't store the model using unnecessarily high numerical precision.
QLoRA combines these ideas.
The base model is loaded in a quantized representation, while LoRA adapters are trained on top of it.
The QLoRA paper demonstrated that this approach could reduce memory requirements enough to fine-tune a 65B-parameter model on a single 48 GB GPU while maintaining the authors' reported 16-bit fine-tuning performance on their evaluated setup.
QLoRA introduced several components, including:
4-bit NormalFloat (NF4),
double quantization,
paged optimizers,
and LoRA adapters.
The important architectural idea is simpler than the terminology:
Keep the expensive base model compressed and frozen; learn a small amount of task-specific information on top.
This is where SLM engineering gets interesting
Now we can combine the techniques.
Suppose you want a specialized model for a production application.
A possible pipeline is:
Large Teacher
|
| Distillation
v
Small Base Model
|
| QLoRA / Fine-tuning
v
Domain Specialist
|
| Quantization
v
Deployment Model
|
+-------+-------+
| |
v v
Cloud Edge
Notice what happened.
We didn't simply "make an LLM smaller."
We built a specialized model optimized for a workload.
That is a fundamentally different mindset.
The hidden trade-off: every optimization changes something
There is a tendency in AI discussions to present optimization techniques as if they are free.
They're not.
Every technique introduces a trade-off.
Distillation
Can reduce model size while transferring useful behavior.
But the student can lose capabilities the teacher had, particularly outside the distilled distribution.
Quantization
Can dramatically reduce memory requirements.
But aggressive quantization can affect model quality, and the impact varies by model and task.
Pruning
Can reduce the number of parameters or operations.
But irregular sparsity may not translate into real speedups on the target hardware.
Fine-tuning
Can dramatically improve performance on a domain.
But a poorly designed dataset can cause overfitting, unwanted behavior, or loss of general capabilities.
LoRA
Can make specialization much cheaper.
But the adapter still depends on the underlying base model, and the chosen rank, target modules, training data, and task determine how effective it is.
QLoRA
Can make fine-tuning much more memory-efficient.
But quantized training introduces its own numerical and implementation considerations.
There is no magic compression button.
The benchmark trap
This is where experienced engineers should be particularly skeptical.
Suppose someone tells you:
"Our 3B model is almost as good as a 70B model."
The next question should be:
At what?
A model can perform extremely well on one benchmark while failing badly on another.
Even worse, a benchmark may not resemble your production workload.
Consider a model used for structured extraction.
The benchmark might measure semantic accuracy.
Your application might require:
{
"severity": "critical",
"component": "camera",
"confidence": 0.93
}
If the model instead produces:
The severity appears to be critical and the affected componentis probably the camera. I would estimate confidence at around 93%.
a human might consider that a good answer.
Your parser might consider it a complete failure.
This is why application-level evaluation matters.
For an SLM, you should measure things like:
task accuracy,
structured-output validity,
latency,
memory usage,
throughput,
energy consumption where relevant,
failure rate,
escalation rate,
and recovery behavior.
The best model is the one that performs well across the metrics your application actually cares about.
A production SLM is more than a model file
This is perhaps the most important engineering lesson from Part 2.
When you deploy an SLM, you aren't deploying:
"model.bin"
You're deploying a system.
That system may include:
User Request
|
v
+---------------+
| Preprocessor |
+-------+-------+
|
v
+---------------+
| SLM |
+-------+-------+
|
v
+---------------+
| Schema |
| Validation |
+-------+-------+
|
+----------+----------+
| |
Valid Invalid
| |
v v
Accept Retry /
Escalate
|
v
LLM
This is especially important for on-device applications.
A model might produce a semantically correct answer but violate the application's output contract.
Your runtime needs to handle that.
A model might run beautifully for a 2K-token context and then exhaust memory at 16K.
Your application needs to handle that.
A quantized model might be fast on one device and slower on another because the runtime doesn't have optimized kernels.
Your deployment system needs to handle that.
The model is only one component.
So, how do you actually choose?
There isn't one universally optimal compression strategy.
Instead, start with the deployment constraint.
If memory is the primary problem
Start by investigating quantization.
If task performance is the primary problem
Investigate fine-tuning or distillation.
If training cost is the primary problem
Investigate LoRA/QLoRA and parameter-efficient methods.
If the model contains unnecessary capacity
Investigate distillation or pruning.
If the device is extremely constrained
Combine techniques.
For example:
Distill → specialize → quantize → benchmark on the actual device.
And don't forget the final step.
Benchmark on the actual device.
A model that looks fantastic on an A100 benchmark may behave very differently on a phone.
The real optimization target
At the beginning of this article, we asked:
How do we make a model smaller without losing everything useful?
The answer isn't a single algorithm.
It's a sequence of trade-offs.
Distillation asks:
What knowledge can we transfer?
Quantization asks:
How precisely do we need to represent it?
Pruning asks:
What computation can we remove?
Fine-tuning asks:
What behavior does this application actually need?
LoRA asks:
How little of the model do we need to change?
QLoRA asks:
Can we do that while keeping the base model heavily compressed?
Together, these techniques let us move from a general-purpose model toward a model that is smaller, more specialized, and easier to deploy.
But that still leaves one major problem.
We've optimized the model.
We haven't yet optimized the system.
From model optimization to system optimization
Imagine we have three models:
Small model → Fast, cheap, limited reasoning
Medium model → Balanced
Large model → Expensive, powerful, broad reasoning
Which one should receive the next request?
If we always choose the large model, we've thrown away much of the benefit of SLMs.
If we always choose the small model, we'll eventually encounter tasks it can't handle.
The interesting solution is to make the system decide.
Request
|
v
+-----------+
| Router |
+-----+-----+
|
+------------+------------+
| | |
v v v
Small Medium Large
Model Model Model
Now the optimization problem becomes much more interesting.
We're no longer asking:
"How do I make one model do everything?"
We're asking:
"How do I use the right amount of intelligence for every request?"
That is the bridge between Part 2 and Part 3.
And it leads us to the final—and perhaps most important—idea in this series:
The future of efficient AI may not be a smaller model. It may be a system that knows when to use a small model.
What's next?
In Part 3, we move from model engineering to system orchestration.
We'll look at:
intelligent model routing;
SLM + LLM hybrid architectures;
confidence-based escalation;
agents and tool use;
local versus cloud execution;
measuring energy and water consumption;
the difference between model efficiency and system efficiency;
and whether an SLM-first architecture actually makes AI more sustainable.
Because "small model = green AI" is an appealing story.
But the real story is much more complicated.
And much more interesting.
Part 3: The Orchestrated Future
When the smartest AI system isn't the one with the smartest model—but the one that knows which model to use.
Top comments (0)