How Token IDs Learn Meaning: Embeddings Explained

Token IDs, input to the NLP model, are arbitrary integers with no inherent meaning. Embeddings are how a meaningless integer becomes a vector that actually represents what a word means.

One-Hot Encoding

The most naive way to turn a token ID into a vector is one-hot encoding - a type of sparse vector.
It, essentially, creates a vector as long as your entire vocabulary, sets every position to 0, and a single 1 at the position matching that token's ID.

Example
Vocabulary (5 tokens): {"the": 0, "cat": 1, "sat": 2, "on": 3, "mat": 4}
One-hot encoding produces:

cat  → [0, 1, 0, 0, 0]
sat  → [0, 0, 1, 0, 0]
mat  → [0, 0, 0, 0, 1]

Problem with One-Hot Encoding

1. It doesn't scale
   A real vocabulary has tens of thousands of entries (e.g. GPT-2 has ~50,000-token vocabulary). Every single token would need a 50,000-number vector, almost entirely 0s, just to represent one word. This is enormously wasteful — both in memory and in the amount of computation every matrix multiplication downstream would have to do on mostly-zero data.

2. Every word is equally "different" from every other word
   With one-hot vectors, every single word's vector has exactly one 1 and all the rest 0s. 1 is in a different place for every word and only 0 position is in common. Mathematically, every pair of one-hot vectors is exactly the same distance apart as every other pair (distance is measured as Euclidean distance[1]). One-Hot Encoding has no way to express that "cat" and "dog" are more similar to each other than "cat" and "airplane". This directly conflicts with the standard principle of linguistics: meaning depends on relationships between words.

Dense Vectors or Embeddings

Embeddings fix both problems listed above by representing each token as a dense vector. It is a much shorter list of numbers (typically a few hundred, not tens of thousands), where most or all values are non-zero and meaningful, rather than mostly 0s with a single 1.

Example
Instead of a 50,000-token one-hot vector, "cat" is represented as a 4 number dense vector (simplified for illustration):

cat → [0.92, 0.15, -0.30, 0.71]

Unlike one-hot encoding, these numbers aren't arbitrary placeholders — they're learned during training. Each number captures some dimension of the word's usage and meaning, inferred from patterns in massive amounts of text.

The key payoff is that words with similar meanings end up with similar vectors. "Cat" and "dog" get vectors that are numerically close to each other, while "cat" and "airplane" end up far apart overcoming the limitations associated with one-hot encoding.

Semantic Space

Because each embedding is a vector, you can treat it as a coordinate — a specific point in a multi-dimensional space (a "semantic space"). Every token in the vocabulary occupies its own point in this space, and the geometric distance between two points reflects how semantically related those two words are.

Here is a sample code that visualizes embeddings using an open-source embedding model all-MiniLM-L6-v2 in the sentence-transformers Python library:

from sentence_transformers import SentenceTransformer
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

model = SentenceTransformer("all-MiniLM-L6-v2")  

words = ["cat", "dog", "hamster", "car", "truck", "bicycle"]
embeddings = model.encode(words)  # 384-dimensional embeddings

# Reduce 384 dimensions down to 2 using PCA (for plotting)
pca = PCA(n_components=2)
coords_2d = pca.fit_transform(embeddings)

plt.scatter(coords_2d[:3, 0], coords_2d[:3, 1], c='#1baf7a', label='Pets')
plt.scatter(coords_2d[3:, 0], coords_2d[3:, 1], c='#2a78d6', label='Vehicles')
for i, word in enumerate(words):
    plt.annotate(word, coords_2d[i])
plt.legend()
plt.savefig('real-semantic-space.png')

Output:

Cosine Similarity

Since similar meaning equals close together in semantic space, the actual mathematical way to measure "how close" is using cosine similarity.

Rather than measuring the straight-line (Euclidean[1]) distance between two vectors, cosine similarity measures the angle between them — how much they point in the same direction, regardless of their length. The output is a number between -1 and 1.

• 1 = pointing in exactly the same direction (maximally similar)
• 0 = pointing in completely unrelated directions (no similarity)
• -1 = pointing in exactly opposite directions (maximally dissimilar)

Example

cat = [0.9, 0.1]
dog = [0.85, 0.15]
car = [0.1, 0.9]

• "Cat" and "dog" point in almost the same direction (both mostly weighted on the first number) — their cosine similarity would be close to 1.
• "Cat" and "car" point in very different directions (one weighted on the first number, the other on the second) — their cosine similarity would be close to 0.
• This is exactly the mathematical operation that lets a model answer "how related are these two words," using nothing but the vectors themselves.

Here's what cosine similarity looks like using an open-source embedding model 'all-MiniLM-L6-v2' in the sentence-transformers Python library:

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer("all-MiniLM-L6-v2")

words = ["cat", "dog", "airplane"]
embeddings = model.encode(words)

print("Embedding shape:", embeddings[0].shape)
print("cat vs dog:", cosine_similarity([embeddings[0]], [embeddings[1]])[0][0])
print("cat vs airplane:", cosine_similarity([embeddings[0]], [embeddings[2]])[0][0])

Output:

Embedding shape: (384,)
cat vs dog: 0.62
cat vs airplane: 0.18

A few things worth connecting back to the concepts above:

• The embedding shape is (384,) — a dense, 384-number vector for a single word, nowhere near the ~50,000-number one-hot vector the same vocabulary would have required.
• "cat" and "dog" score notably higher 0.62 than "cat" and "airplane" 0.18 — this is the semantic clustering from the previous section, meaning related concepts land closer together in the embedding space, exactly as cosine similarity is designed to detect.
• These specific numbers will vary slightly depending on the exact model used, but the pattern — related words scoring higher than unrelated ones — is the reliable part.

Where This Leaves Us?

A token ID on its own is meaningless — one-hot encoding turns it into a vector but keeps it meaningless and wastefully large. Dense embeddings solve both problems at once: a compact vector, learned from data, positioned in a semantic space where distance and direction actually correspond to meaning, measurable directly via cosine similarity.

But embeddings alone have a limitation worth flagging: each word gets exactly 1 fixed vector, regardless of context. "Bank" gets the same embedding whether it means a riverbank or a financial institution — the ambiguity problem from Article 1[2] hasn't actually been solved yet, just moved one step deeper. Letting a word's representation shift based on the words around it is exactly what self-attention does, and that's next.

---

[1] Euclidean distance is the straight line distance between two points given by the formula:

distance = √[(a₁-b₁)² + (a₂-b₂)² + ... ]

[2] For details, refer to https://dev.to/rishikadhar/rules-to-learning-why-nlp-needed-transformers--ojp