Anup Karanjkar

Posted on Jul 19 • Originally published at wowhow.cloud

The Complete Guide to AI Embeddings (With Python Code)

#aiembeddings #vectordatabase #embeddingsexplained #similaritysearch

If you build AI applications, you need to understand embeddings. They power semantic search, RAG systems, recommendation engines, clustering, anomaly detection, and dozens of other applications. Yet most developers treat them as a black box.

Let's open that box.

What Are Embeddings?

An embedding is a list of numbers (vector) that represents the meaning of text. Similar meanings produce similar vectors. Different meanings produce different vectors.

# "The cat sat on the mat" → [0.23, -0.45, 0.67, ...]  (1536 numbers)
# "A feline rested on a rug" → [0.21, -0.43, 0.65, ...]  (similar!)
# "Stock prices rose sharply" → [-0.89, 0.12, -0.34, ...] (very different!)

The mathematical distance between vectors corresponds to the semantic distance between concepts. This is what makes them useful — you can search by meaning, not just keywords.

Generating Embeddings in Python

Option 1: OpenAI Embeddings

from openai import OpenAI

client = OpenAI()

def get_embedding(text: str, model: str = "text-embedding-3-large") -> list[float]:
    response = client.embeddings.create(
        input=text,
        model=model
    )
    return response.data[0].embedding

# Generate embeddings
embedding = get_embedding("How do neural networks learn?")
print(f"Dimensions: {len(embedding)}")  # 3072 for text-embedding-3-large

Option 2: Open-Source with sentence-transformers

from sentence_transformers import SentenceTransformer

# Free, runs locally, no API key needed
model = SentenceTransformer('all-MiniLM-L6-v2')

texts = [
    "How do neural networks learn?",
    "What is backpropagation in deep learning?",
    "Best restaurants in New York City"
]

embeddings = model.encode(texts)
print(f"Shape: {embeddings.shape}")  # (3, 384)

Similarity Search

The core operation: given a query, find the most similar items.

import numpy as np
from numpy.linalg import norm

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (norm(a) * norm(b))

# Compare similarity
query = model.encode("machine learning algorithms")
documents = [
    model.encode("supervised learning classification methods"),
    model.encode("best pizza delivery services"),
    model.encode("neural network training techniques"),
]

for i, doc in enumerate(documents):
    sim = cosine_similarity(query, doc)
    print(f"Document {i}: {sim:.4f}")

# Output:
# Document 0: 0.7823  (related!)
# Document 1: 0.1234  (not related)
# Document 2: 0.8156  (very related!)

Vector Databases

For production use, you need a vector database that handles billions of vectors efficiently.

Pinecone Example

from pinecone import Pinecone

pc = Pinecone(api_key="your-key")
index = pc.Index("my-index")

# Upsert vectors
vectors = [
    {"id": "doc1", "values": embedding1, "metadata": {"title": "ML Basics"}},
    {"id": "doc2", "values": embedding2, "metadata": {"title": "Pizza Guide"}},
]
index.upsert(vectors=vectors)

# Query
results = index.query(
    vector=query_embedding,
    top_k=5,
    include_metadata=True
)

Supabase pgvector (Free Option)

-- Enable the extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create a table with a vector column
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding vector(1536)
);

-- Insert with embedding
INSERT INTO documents (content, embedding)
VALUES ('Machine learning basics', '[0.1, 0.2, ...]');

-- Similarity search
SELECT content, 1 - (embedding  '[0.15, 0.25, ...]') AS similarity
FROM documents
ORDER BY embedding  '[0.15, 0.25, ...]'
LIMIT 5;

Practical Applications

1. Semantic Search Engine

Replace keyword search with meaning-based search. Users can search "how to fix slow website" and find documents about "performance optimization" even if those words don't appear.

2. Duplicate Detection

Find near-duplicate content by comparing embedding similarity. Useful for content moderation, plagiarism detection, and deduplication.

3. Recommendation Systems

Embed user preferences and item descriptions. Recommend items whose embeddings are closest to the user's preference vector.

4. Clustering and Classification

Group similar items automatically using k-means or HDBSCAN on embeddings. No labels needed — the structure emerges from the data.

Choosing an Embedding Model

OpenAI text-embedding-3-large: Best quality, $0.13/million tokens
OpenAI text-embedding-3-small: Good quality, $0.02/million tokens
Cohere embed-v4: Competitive quality, good for multilingual
all-MiniLM-L6-v2: Free, runs locally, 384 dimensions, great for prototyping
BGE-large-en-v1.5: Free, runs locally, 1024 dimensions, production-ready quality

DEV Community