Vector Embeddings Explained with Semantic Search Example

Introduction

Vector embeddings are powerful tools for representing data in a high-dimensional space. In this article, we'll explore vector embeddings using a simple semantic search example. This will help you understand how to use vectors for finding similar items based on their semantic meaning.

Understanding Vector Embeddings

Vector embeddings are used to represent data in a form that can be easily processed by machine learning algorithms. For example, text can be converted into numerical vectors, enabling computers to 'understand' the text.

Encoding Text into Vectors

Let's start by encoding some text data into vectors. We'll use the popular gensim library for this. The gensim library provides utilities for working with word embeddings and semantic text similarity.

from gensim import corpora, models, similarities
from gensim.models import TfidfModel
from gensim.corpora import Dictionary

# Example text data
texts = [
    'The quick brown fox jumped over the lazy dog',
    'The quick brown dog and the lazy fox',
    'The lazy dog and the quick brown fox',
    'The lazy dog and the quick dog'
]

# Create dictionary
dictionary = Dictionary(texts)

# Create bag-of-words model
corpus = [dictionary.doc2bow(text) for text in texts]

# Create tf-idf model
tfidf = TfidfModel(corpus)

corpus_tfidf = tfidf[corpus]

In this code, we first create a dictionary of unique words from the text data. We then convert the text data into a bag-of-words model, which represents each document as a set of word frequencies. We also create a tf-idf (term frequency-inverse document frequency) model, which emphasizes words that appear frequently in a document but less in the overall corpus.

Creating Vector Embeddings

Next, we'll convert the corpus into vector embeddings.

# Create vector embeddings
lens = [sum([w[1] for w in doc]) for doc in corpus_tfidf]

vectors = [w[0] for w in corpus_tfidf]

# Normalize vector lengths
norms = [len(l) ** 0.5 for l in lens]

vectors = [[v[0], v[1] / norms[w[1]]] for w, v in zip(corpus_tfidf, vectors)]

# Convert to array
vectors = [v[1] for v in vectors]

Here, we calculate the vector representation by normalizing the word frequencies. This normalization ensures that the vectors are of comparable lengths and can be used for comparison.

Semantic Search with Vector Embeddings

Now that we have our vector embeddings, let's perform a semantic search. We'll use the cosine_similarity function to find the most similar vectors.

# Create index
index = similarities.MatrixSimilarity(vectors)

# Create query
query = 'The lazy dog and the quick brown fox'

text_query = dictionary.doc2bow(query.lower().split())

text_query_tfidf = tfidf[corpus_tfidf[corpus.index(text_query)]]

text_query_vector = [w[0] for w in text_query_tfidf]

text_query_vector = [v[1] / norms[corpus.index(text_query)] for v in text_query_vector]

text_query_vector = [text_query_vector[0]] + [0] * (len(vectors[0]) - 1)

text_query_vector = np.array(text_query_vector)

# Find most similar vectors
index = similarities.MatrixSimilarity(vectors)

sim_scores = index[corpus_tfidf]

docs = [texts[i] for i in sim_scores[0].argsort()[-3:]]

docs

In this example, we create an index of our vector embeddings and then find the most similar documents to the query 'The lazy dog and the quick brown fox'. The sim_scores array contains the cosine similarity scores between the query vector and the document vectors. We then sort these scores in descending order to find the most similar documents.

Common Pitfalls and Considerations

• Vector Space Issues: Vectors can span a very large space, leading to noise or irrelevant results. Techniques like dimensionality reduction (e.g., PCA) can help reduce this space.
• Vector Embedding Quality: The quality of vector embeddings depends on the training data and model. Careful selection and preprocessing of the training data can improve the quality of embeddings.
• Similarity Metrics: Other similarity metrics like cosine similarity are also available. Choose the one that best suits your needs.

Conclusion

In this article, we explored vector embeddings and their application in semantic search. By converting text data into vector embeddings and using similarity metrics, we can find related items based on their meaning. Understanding vector embeddings is crucial for building more advanced NLP and machine learning models.

Description

This article explains vector embeddings with a semantic search example, covering encoding text, creating vector embeddings, and finding similar items based on semantic similarity.