Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
You can often understand a lot of what an LLM is doing by asking one very old question:
How much information can I throw away while still preserving what I care about?
That is the central question of rate-distortion theory, developed by Claude Shannon in the foundations of information theory.
It sounds like a theory about telephone lines, image compression, and noisy communication.
It is also a remarkably useful mental model for modern language models.
A tokenizer throws away some structure in raw text.
A context window throws away most of the universe and preserves only a selected slice of it.
Model weights compress enormous training corpora into a finite parameter vector.
Quantization throws away numerical precision.
Prompt compression throws away words.
Knowledge distillation throws away model capacity.
And, perhaps most interestingly, pretraining itself can be viewed as a gigantic lossy compression process: the model cannot remember everything it saw, so it learns a representation that preserves information useful for predicting language.
The engineering question is therefore rarely:
"Can we compress this?"
It is:
"What information are we willing to lose, and what is the economic value of keeping it?"
That is rate-distortion intuition.
1. Shannon's Original Problem Was Close to This Scenario
Claude Shannon was working at Bell Labs when he published A Mathematical Theory of Communication in 1948.
The practical world around him consisted of telephone networks, radio, cryptography, switching systems and increasingly sophisticated communications hardware. Shannon's contribution was to strip away the implementation details and ask what the fundamental limits were.
One of the insights was that a message need not necessarily be transmitted exactly.
Suppose I send you a photograph.
I could demand perfect reconstruction: every pixel must be identical.
Or I could say:
"I don't care about tiny changes in color. Keep the face recognizable."
Those are radically different engineering requirements.
The second requirement permits much more compression.
This led to a basic tradeoff:
more bits -> better fidelity
fewer bits -> more distortion
The key insight is that "distortion" is not necessarily the same thing as "difference."
That distinction becomes extremely important for LLMs.
Suppose the original sentence is:
The quarterly revenue increased by 7.3% because enterprise demand accelerated in Europe.
A compressed representation might lose "quarterly," the exact percentage, or the geographic qualifier.
Those losses are not equally bad.
If your application is:
"Was revenue increasing?"
then the representation may be perfectly adequate.
If your application is:
"Produce the exact financial statement"
then losing 7.3% is disastrous.
So distortion must be defined relative to the task.
This is the part of Shannon's idea that becomes especially powerful for AI.
Shannon himself was an unusually good prototype for this way of thinking. At Bell Labs he worked on communication theory while also building whimsical machines, including Theseus, an electromechanical mouse that could learn paths through a maze. The same person who was developing mathematical limits for information transmission was also playing with machines that stored and exploited information.
The fundamental theory came from communications.
The same abstraction now shows up whenever an LLM turns a huge source into a smaller representation.
2. The Rate-Distortion Curve
Let:
- R = rate, roughly "how many bits we spend"
- D = distortion, roughly "how much useful information we lose"
Then rate-distortion theory asks for the smallest possible rate for a given acceptable distortion:
R(D) = minimum information rate needed to achieve distortion <= D
You can visualize the curve like this:
rate
^
|\
| \
| \
| \
| \__
| \____
+-----------------> distortion
The precise shape depends on the source distribution and the definition of distortion, but the engineering intuition is general.
Going from very bad compression to decent compression can produce enormous quality improvements.
Going from very good compression to almost-perfect reconstruction often becomes extremely expensive.
This is why compression engineering tends to have diminishing returns.
The LLM analogy is immediate.
Imagine compressing a model from 16-bit weights to 8-bit weights.
You have roughly halved the storage required.
Perhaps quality barely changes.
Then you go from 8-bit to 4-bit.
You halve storage again, while potentially still retaining most capabilities.
But go from 4-bit to 2-bit and you may encounter a much steeper quality penalty.
So the interesting region is not simply:
maximum compression
It is:
maximum useful compression
That distinction matters enormously in production.
3. The Most Important Trick: Distortion Depends on What You Care About
Imagine that you have a 1 GB document collection and only 10 MB of representation budget.
What should the 10 MB contain?
The naïve answer is:
"The most important information."
But important according to whom?
Consider a medical assistant.
Removing a repeated sentence such as:
"The patient should return for follow-up."
might have virtually zero distortion.
Removing:
"The patient is allergic to penicillin."
could be catastrophic.
So a good compression system does not minimize some abstract notion of textual difference.
It minimizes task-relevant distortion.
Formally, you can imagine a distortion function:
D(x, x_hat)
where:
- x = original information
- x_hat = reconstructed or compressed information
In classical signal processing, D might be mean squared error.
For an LLM application, D might instead be:
D = probability the answer to the downstream task changes
or:
D = loss in benchmark accuracy
or:
D = expected monetary loss from incorrect output
That is a much more interesting engineering objective.
Suppose you are building a customer-support model.
You have 100,000 tokens of raw conversation history but can afford only 10,000 tokens in the prompt.
You could compress it by removing:
"Hi there!"
"Thanks!"
"Okay."
"Sounds good."
while preserving:
Order ID: 847291
Refund requested: $1,249
Customer claims duplicate charge
Previous agent promised refund within 5 business days
Both representations have lost information.
Only one has incurred large task-relevant distortion.
This gives a useful LLM engineering rule:
Compression quality cannot be measured independently of the task.
4. An LLM Is Already a Giant Lossy Compressor
Now we get to the more interesting part.
Suppose a model trains on a trillion tokens.
After training, the model does not contain a trillion-token database.
Instead, it contains a finite parameter vector:
theta = {w1, w2, ..., wN}
That parameter vector is much smaller than the information content of its training corpus.
So where did the information go?
Most of it was discarded.
The model retains statistical regularities that help predict future tokens.
That means training can be viewed as a kind of lossy compression:
training corpus
|
v
training
|
v
finite parameter vector
The remarkable thing is that the compressed representation can reconstruct useful behavior without reconstructing the original dataset verbatim.
This is very different from ZIP.
ZIP tries to preserve the original bytes.
An LLM preserves something closer to:
"What regularities in this data are useful for predicting language?"
A recent line of research explicitly develops this perspective, arguing that LLM pretraining can be understood as lossy compression in which information irrelevant to the training objective is discarded. Their experiments connect the degree of compression to downstream behavior across multiple models.
This gives us an important conceptual distinction:
ZIP:
source -> exact reconstruction
LLM:
source -> useful statistical reconstruction
And that explains one of the stranger properties of LLMs.
A model can "know" something without being able to reproduce the original wording.
It may have compressed:
"The Eiffel Tower is 330 meters tall."
into something more like:
Paris
+ landmark
+ Eiffel Tower
+ approximate height
+ French context
That representation can be sufficient for many questions.
But not necessarily for:
"Quote the source verbatim."
The second task requires much lower distortion.
5. Why Bigger Models Help: You're Buying More Rate
Now consider the parameter count.
A model with:
1 billion parameters
has considerably less representational capacity than one with:
100 billion parameters
At a crude information-theoretic level, the larger model has more "budget" with which to represent useful structure.
This does not mean:
100B parameters = 100B x more knowledge
because neural network parameters are not independent bits.
But the analogy is still useful.
Imagine trying to summarize the internet using a notebook.
With 100 pages, you can retain:
major topics
major entities
common relationships
With 100,000 pages, you can retain:
rare facts
subtle distinctions
long-tail linguistic patterns
specialized domains
The additional capacity lets the model operate farther along the rate-distortion curve.
That is one reason scaling works.
But there is another constraint: training data.
Suppose your model has enormous capacity but only sees 10 million tokens.
There simply isn't much statistical structure available to learn.
This is exactly where the Chinchilla result became economically important.
DeepMind trained hundreds of models and found that, for a given training-compute budget, model size and training tokens should scale together rather than putting almost everything into model parameters. Their 70B Chinchilla model, trained on roughly four times as much data as the 280B Gopher model under a similar compute budget, substantially outperformed it on many evaluations.
In rate-distortion language:
more representation capacity
+
more source information
=
better useful compression
A giant empty dictionary is not very useful.
Neither is an enormous compression budget applied to a tiny source.
6. Quantization Is Literally a Rate-Distortion Problem
The rate-distortion picture becomes especially concrete when we quantize model weights.
Suppose a model contains:
70 billion parameters
At 16 bits per parameter:
70B x 16 bits
= 1.12 trillion bits
= 140 GB
That is before accounting for implementation details and runtime memory overhead.
Now use 8-bit weights:
70B x 8 bits
= 560 Gb
= 70 GB
You've roughly halved the weight-storage requirement.
At 4 bits:
70B x 4 bits
= 280 Gb
= 35 GB
Again, another factor of two.
The seductive mistake is to think:
"Then 2 bits must be better economically."
Only if the resulting distortion is acceptable.
Some weights are much more sensitive to quantization than others. Some layers or channels can tolerate substantial numerical error; others contribute disproportionately to model quality.
This is almost exactly what rate-distortion theory tells us to expect.
Don't allocate bits uniformly.
Allocate more bits where distortion is expensive.
That is the basic logic behind rate-distortion optimization approaches to LLM quantization. Recent work has explicitly formulated LLM weight quantization in these terms and demonstrated compression-quality tradeoffs at very large model scales.
You can think of the optimization as:
minimize:
model cost
subject to:
quality >= required quality
or equivalently:
minimize:
distortion + lambda * rate
where lambda represents the economic value of saving bits.
That parameter is conceptually important.
A mobile device might have:
very high value of memory savings
while a data center serving a high-value financial workload might have:
very high value of model accuracy
Same model.
Different optimal point on the curve.
7. The Operational Meaning: You're Always Buying Information With a Budget
This is where rate-distortion becomes less like an information-theory curiosity and more like an engineering framework.
Every LLM system has multiple information budgets.
Context budget
You might have:
1M token context window
but only actually need:
20K useful tokens
The rest imposes latency and compute cost without necessarily adding value.
Prompt compression is therefore a rate-distortion optimization problem:
keep fewer tokens
vs.
lose relevant context
The right question isn't:
"How do I make the prompt shorter?"
It is:
"How many tokens do I need before additional context stops paying for itself?"
Model-memory budget
Suppose you can deploy either:
70B @ 4-bit = ~35 GB
or:
70B @ 8-bit = ~70 GB
Perhaps the 8-bit model improves your evaluation score by 0.7 percentage points.
Is that worth another 35 GB of accelerator memory?
This is no longer an ML question alone.
It is an operations question.
Maybe 4-bit lets you fit onto one GPU instead of two.
Then the economics may change dramatically:
2 GPUs -> 1 GPU
can mean lower capital cost, lower power consumption, lower networking overhead, simpler scheduling, and higher replica density.
A tiny increase in distortion can therefore produce a huge reduction in serving cost.
Training budget
Chinchilla is another example.
Suppose you have a fixed compute budget:
C = constant
You can spend it on:
larger model
or:
more training tokens
The optimum is not at either extreme.
You want the combination that gives the best reduction in downstream error for the available compute.
That is rate-distortion thinking translated into a compute-allocation problem.
Human attention
There is an even stranger version.
Your model may technically support a huge context window.
But humans generating the context are themselves lossy compressors.
An engineer takes a 500-page incident report and writes:
Root cause:
cache invalidation bug
Impact:
12% traffic affected
Fix:
deployed version 4.18
Remaining concern:
stale replicas after failover
That five-line summary is a rate-distortion system.
The engineer has performed compression.
The LLM then compresses the summary again into internal representations.
The final answer is another reconstruction.
In a modern AI system, there can therefore be an entire chain:
world
|
documents
|
retrieval
|
prompt
|
tokens
|
activations
|
weights
|
generated tokens
|
human interpretation
At every stage, information is discarded.
The engineering problem is to make sure the discarded information is the cheap information to lose.
8. The Back-of-the-Envelope Rule
A useful mental model for developers is to assign a monetary value to distortion.
Suppose you are serving a customer-support model.
Assume:
1 million requests / month
A compression technique saves:
$0.002 per request
So:
monthly savings = 1,000,000 x $0.002
= $2,000
But suppose it increases the probability of an expensive support failure by just:
0.01 percentage points
That's:
100 additional failures / month
If each failure costs:
$50
then:
100 x $50 = $5,000
The "better compressed" system is economically worse.
This is the core operational lesson of rate-distortion theory:
The acceptable distortion is determined by the value of the information, not by the compression algorithm.
For a toy chatbot, a minor factual degradation may be irrelevant.
For a code-generation system, changing one character in an API call can be catastrophic.
For an autonomous agent, losing one piece of state about an external side effect may be far more costly than losing 1,000 tokens of conversational history.
The distortion function is fundamentally application-dependent.
9. A Useful Mental Model for LLM Engineers
When you encounter an LLM optimization problem, ask four questions.
What is the source?
training data?
prompt?
context?
weights?
activations?
KV cache?
What is the rate?
tokens?
bits?
bytes?
GPU memory?
latency?
FLOPs?
What is the distortion?
Not merely "how different is the compressed representation?"
Ask:
What user-visible or system-level behavior gets worse?
What is one unit of distortion worth?
This is the missing piece in many optimization discussions.
If losing one token costs almost nothing, compress aggressively.
If losing one token containing an authentication constraint can cause a production incident, preserve it.
The general optimization is something like:
maximize utility
= useful information retained
- cost of representation
Or, in a slightly more formal form:
minimize D + lambda * R
where:
D = task-relevant distortion
R = resource usage
lambda = economic price of resource usage
Change lambda, and the optimal LLM system changes.
Cheap inference encourages larger contexts and higher precision.
Expensive inference encourages aggressive compression.
Memory-constrained edge deployment encourages quantization.
Latency-sensitive applications encourage smaller models.
High-stakes applications tolerate much less distortion.
Same information theory.
Different economics.
10. The Bigger Idea
Rate-distortion theory gives a clean way to think about what an LLM fundamentally does.
A language model is not a database of everything it has seen.
It is a finite representation of an enormous distribution.
Training compresses data into weights.
Inference compresses the relevant world into a context.
Attention decides which parts of that context deserve representational bandwidth.
Quantization compresses numerical precision.
Distillation compresses capability into a smaller model.
Prompt compression compresses language.
And deployment decisions determine how much distortion the business is willing to tolerate.
That suggests a useful reframing of LLM engineering.
Instead of asking:
"How do I make the model bigger?"
or:
"How do I make inference cheaper?"
ask:
"What information is actually valuable, and what is the cheapest representation that preserves it?"
That is a much more general question.
It also explains why some apparently magical LLM behaviors make sense. A model does not need to retain a complete copy of its training data to answer useful questions about it. It needs a representation that preserves the statistical structure relevant to the queries we care about.
In other words, intelligence under finite compute is partly a problem of deciding what information not to keep.
And that is almost exactly the problem Shannon was studying in 1948.
So when you next see a paper claiming that an LLM can be made 4x smaller, a prompt can be reduced by 80%, or a KV cache can be compressed by 10x, there is a useful question hiding underneath all the benchmark numbers:
What distortion did we buy to get that rate reduction, and was it actually worth the price?
That's the rate-distortion lens.
What LLM optimization problem do you think is most naturally understood as a rate-distortion tradeoff: model quantization, context compression, training-data selection, or something else?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)