How Emoji Semantic Search Works: Embeddings, Cosine Similarity, and 310k Vectors in Milliseconds

Type "cozy morning coffee" into our emoji search and it returns breakfast, warm-light and calm-mood emoji combinations — none of which share a single keyword with the query. No synonym table, no hand-tagging. That swap from keyword lookup to meaning-based matching is powered by the most boring-sounding stack in modern ML: text embeddings plus cosine similarity, scanned over ~310,000 precomputed vectors. This is the engineering tour of how it works and what we learned building it.

Keyword tables don't generalize

Classic emoji search is a lookup table: "happy" maps to 😊, and if you type "joyful" you get nothing. Every synonym must be hand-tagged and every phrasing anticipated.

Embedding-based search flips the problem. Instead of matching strings, you match points in a high-dimensional space:

• Every query is converted into a vector of numbers.
• Every item in the catalog (for us, 310k+ emoji combinations) is also a vector, precomputed ahead of time.
• "Similar" simply means "geometrically close" — and geometry generalizes to words the system has never seen.

The embedding pipeline, step by step

We use an open-source multilingual model (BGE-M3, 1024 dimensions). Every text — queries at runtime, the whole catalog offline — goes through the same four stages:

1. Tokenization — the model sees subword tokens, not words: "cozy morning coffee" → ["co", "##zy", "morning", "coffee"]. Rare words split into known pieces, so the model can embed text it has never seen.
2. Transformer encoding — each token flows through self-attention layers, exchanging information with every other token. "coffee" absorbs the fact that "morning" is next to it.
3. Mean pooling — one per-token vector each is averaged into a single 1024-dimension sentence vector.
4. L2 normalization — the vector is scaled to unit length, landing on the surface of a unit sphere.

That last step is load-bearing. Once every vector has length 1, comparing directions is all that's left.

The math that ranks everything

Cosine similarity measures the angle between two vectors, ignoring length — from −1 (opposite) through 0 (unrelated) to +1 (same direction):

cos(θ) = (q · c) / (|q| × |c|)

Because we normalized everything to unit length, the denominator is always 1 and the whole formula collapses to a plain dot product — one multiply-and-add per dimension. There's also a neat identity that falls out:

|q − c|² = 2 − 2 × cos(θ)

Smaller Euclidean distance and larger cosine similarity are the same statement for unit vectors. Whether your nearest-neighbor code sorts by dot product, cosine, or Euclidean distance, the ranking is identical.

Here's the entire mechanism in three dimensions, small enough to check by hand:

import numpy as np

query = np.array([0.80, 0.60, 0.00])   # "coffee"
combos = np.array([
    [0.70, 0.70, 0.14],                # "hot drink"
    [0.20, 0.30, 0.93],                # "birthday party"
])

cosine = combos @ query / (
    np.linalg.norm(combos, axis=1) * np.linalg.norm(query)
)
print(cosine)   # [0.98 0.34]  -> hot drink wins by a mile

Scale that from 3 dimensions to 1024, and from 2 candidates to 310,000, and you have semantic emoji search.

The lifecycle of one search

Offline, once: embed every catalog item and store its vector alongside its text. Vectors never change until the catalog does, so this cost is paid once — not per search.

Online, every search: embed the query with the same model, compute the dot product against the catalog, sort, return the top 100. Mixing models between index time and query time is the classic bug that quietly destroys result quality — two models produce two incompatible geometries.

Exact scan vs. ANN index

1024 dimensions × 310k vectors is ~318 million multiply-adds per query. That sounds scary and isn't: a tight vectorized loop finishes in well under a second, and an exact scan returns the provably best matches. Most systems switch to approximate nearest-neighbor (ANN) indexes — graph or cluster structures that trade a sliver of recall for an order-of-magnitude speedup — only past a few million vectors. Know your threshold before you reach for the fancy index.

Lessons we learned the hard way

1. Normalize your vectors at write time — cosine becomes a dot product and rankings stay consistent.
2. Use one model everywhere. Index and query vectors must share a geometry.
3. Precompute aggressively. Only the live query should pay the embedding cost.
4. Watch p99, not the average. Inference has cold starts; warm instances matter more than average speed.
5. Evaluate with real queries. Cosine scores don't feel like relevance — the only test that counts is whether humans like what comes back.

If you want to see it ranking live, every query on emojimagic.io/text-to-emoji runs exactly this pipeline — this post on our blog has the full worked example, and the site itself is free with no signup if you'd rather throw queries at it than read about them.