While running one morning, I found myself staring at the shadows cast by a stone bridge railing. The railing had a series of identical, carved镂空 (hollowed-out) structures. As I ran past, the angle of the sun shifted, and the shadows on the ground transformed.
At one moment, the shadow looked like a swimming fish; a few steps later, it looked like a rolling wave; later still, it was just an abstract, elongated blob.
It struck me that I was witnessing a perfect physical metaphor for how human language—and Large Language Models (LLMs)—generate meaning.
The stone bridge hole is the concept. The sunlight is the context. The shadow on the ground is the utterance.
This realization perfectly mirrors the "Central Problem" of language modeling, beautifully stripped of its usual jargon in Joseph L. Breeden’s paper, The Simple Mathematics of Large Language Models.
The Problem of the Single Shadow
In his paper, Breeden highlights a fundamental flaw in early AI approaches: treating words as static, unchanging symbols. He uses the classic example of the word "bank":
- "I deposited money at the **bank." (a financial institution)
- "We walked along the river **bank." (the land beside a river)
- "The plane **banked* sharply left."* (tilted)
If we assign a single, fixed vector to the word "bank" in a high-dimensional space, we are essentially trying to capture all three meanings in one static shape. But as Breeden points out, a fixed vector for "bank" cannot simultaneously be close to "money," "river," and "tilt."
In my running metaphor, a single, fixed vector is like looking at the bridge hole only at exactly high noon. You get one specific shadow. But language is dynamic. The meaning of "bank" depends entirely on the "angle of the light" provided by the surrounding words.
The Mathematics of Light: Learned Weighted Averaging
How does an LLM solve this? Breeden explains that the core mechanism of a Transformer is not magic; it is Learned Weighted Averaging.
Let’s map Breeden’s mathematics to the bridge railing:
1. The Bridge Hole (The Token Embedding $e$
Before any context is applied, the word "bank" is just an initial embedding vector ($e$). It is the physical stone structure itself. It contains the potential for all its meanings, but it is currently unshaped by the environment.
2. The Angle of the Sunlight (Influence Scores,$a_i \cdot b_j$)
To determine what the shadow will look like, we need light. In an LLM, the "light" comes from the surrounding words (the context).
Breeden explains that the model computes an influence score between the target word and its context. It does this by projecting the vectors through learned matrices ($W^A$ and $W^B$).
- When the context is "deposited money," the "light" hits the bridge hole from a financial angle.
- When the context is "river," the light hits it from a geographical angle. The dot product ($a_i \cdot b_j$) measures how perfectly the angle of the light aligns with the shape of the hole.
3. The Shadow on the Ground (Context-Dependent Representation, $h_i$)
Finally, the light passes through the hole and casts a shadow. Mathematically, the model takes a weighted average of the context vectors (using the content projection $W^C$), resulting in a brand new vector ($h_i$).
This new vector is the shadow. It is a context-dependent representation. The "bank" in the first sentence is mathematically pulled toward the geometry of finance; the "bank" in the second sentence is pulled toward the geometry of nature. They are the same material, but entirely different products.
Why This Metaphor Matters
What makes Breeden’s explanation so profound is that it removes the anthropomorphic jargon of "Queries," "Keys," and "Values." There is no little homunculus inside the computer "looking up" information in a database.
There is only geometry and light.
When an LLM reads the sentence "The plane banked sharply left," it doesn't "know" what a plane is. Instead, the high-dimensional vectors for "plane" and "sharply left" act as a specific angle of light. This light passes through the "bank" bridge hole and casts a shadow that aligns perfectly with the geometric region of "physical motion" in the model's latent space.
Furthermore, this explains the difference between languages. In English, "bank" is a single, multi-purpose bridge hole that requires complex, dynamic lighting (Attention mechanisms) to cast the right shadow. In Chinese, the language has already carved three separate, distinct bridge holes: 银行 (bank), 岸 (riverbank), and 倾斜 (to tilt). The Chinese model requires less "light manipulation" because the physical structures themselves are already disentangled.
Conclusion: Meaning is a Projection
As I finished my run, I realized that the bridge railing wasn't just casting shadows; it was performing a physical projection.
Large Language Models do not store a dictionary of definitions. They store a high-dimensional geometry of concepts. Meaning is not a static object you can hold in your hand; it is a projection. It is the dynamic, mathematical shadow cast when the light of context passes through the structure of language.
And just like the shadows on the pavement, the beauty of language lies not in the stone itself, but in how it changes with the light.
References:
Breeden, J. L. (2026). The Simple Mathematics of Large Language Models. CRC Working Papers.
post according to talk with Qwen 3.7 plus model
Top comments (0)