Hello, DEV Community! 🙌 It’s my first time writing here.
As I was studying how models handle text today, I ran into a funny realization: computers don't understand human words at all. They only understand numbers.
So how do we explain to a model that a cat is a pet, an apple can be a fruit or a tech company?
Here is what I learned today about Word Embedding, explained from a pure logic and geometry perspective!
1. The First step: Making a Massive List(One-Hot Encoding)
If you ask a beginner programmer how to turn words into numbers, the most obvious idea is to give every single word a simple code or an index in a giant list.
Imagine taking an entire dictionary of 100,000 words and marking 1 for the word you want and 0 for everything else.
-
cat=[1, 0, 0, 0, ...] -
dog=[0, 1, 0, 0, ...] -
apple=[0, 0, 1, 0, ...]
Why this fails:
- It Wastes Endless Space: You end up creating massive arrays full of useless zeros just to represent one tiny word. RIP RAM. 🪦
-
Computers Got Zero "Street Smarts": In pure math, these simple list positions have nothing in common. To the computer,
cathas as much in common withdogas it does withbanana. It has no idea that two animals belong together.
2. The Solution: Giving Words Map Coordinates
Instead of a giant empty list, researchers like Mikolov et al. at Google introduced Word Embedding (such as Word2Vec and GloVe).
Think of it like plotting points on a map. In physics or math class, you plot points using X, Y, and Z coordinates. Word embedding does the exact same thing, except instead of 3 directions, they use 50 to 300 invisible dimensions!
These dimensions represent hidden concepts like is_animal, is_food, is_tech, or grammatical_type.
- "cat" gets coordinates near animal concepts.
- "dog" gets coordinates right next to "cat".
- "apple" gets pulled into a coordinate spot sitting right in between the fruit neighborhood and the iPhone neighborhood!
How does the computer learn these coordinates? By sliding a "context window" across millions of sentences on the internet. It observes which words consistently live as neighbors in the same sentences so if words appear in similar contexts, they get pulled toward the same coordinates in vector space.
Because the computer reads millions of sentences, it notices which words "live" in the same neighborhoods.
3. Math Geometry Magic
Because every word becomes a point in space, we can measure how close two words are by calculating the angle or distance between their points.
If two words point in nearly the exact same direction, they share similar meanings!
Doing Algebra with Words
My discovery today was that you can actually do math equations with word coordinates:
King - Man + Woman = Queen
This famous vector arithmetic concept was demonstrated in the original Word2Vec paper by Mikolov et al. (2013)."
If you start at the coordinate for King, subtract the "male" direction, and add the "female" direction, your new coordinate in space lands right on Queen. 🤯
Final Thoughts
Learning that word embedding are just coordinate maps helped me bridge the gap between machine language (a.k.a math) and human language.
In any case, word embedding are nothing but high dimensional maps that make words street-smart. The idea of understanding text via coordinates really clicked with me!
References & Further Reading
- Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781
- Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. Stanford NLP
- Alammar, J. (2019). The Illustrated Word2Vec. jalammar.github.io
Top comments (1)
Good work on the blog, and welcome to the Tech blogging community.
However, in your blog, it is not easy to connect the dots and understand where you are coming from.
You also need to share what you learned, what you found surprising, and what you struggled with.
You should include a code snippet so that we can learn with you what you have explored
Overall, you have done fantastic by exploring the math behind word embeddings