Testing some embeddings… with Ollama
Introduction — Embeddings
If Large Language Models (LLMs) are the brain of modern AI, then embeddings are the nerve signals — the numerical vectors that allow machines to understand the meaning and relationship between words. When you ask an AI a question, it first converts your text into a vector (an embedding), which is essentially a long list of floating-point numbers.
But not all embeddings are created equal. Choosing the right embedding model is the foundation of a high-performance system, whether you’re building a sophisticated RAG (Retrieval-Augmented Generation) application or a simple search index. Before you deploy, you must test.
The Three Pillars of Embedding Model Performance
Testing an embedding model requires balancing three critical, and often conflicting, metrics. As demonstrated by the comparison between two models like Granite Embedding (two different sizes here) and Embedding Gemma, the choice is usually a trade-off.
1. The Need for Speed (Latency & Duration)
The most straightforward metric is speed. How quickly can the model convert your text into a vector?
- Metric: Total Duration (seconds) per set of texts.
- Implication: Low latency is essential for real-time applications, like instant search suggestions or immediate RAG lookups. If your embedding process is slow, your entire application feels sluggish.
2. Efficiency and Size (Resource Consumption)
Embedding models still need to be hosted and run, which costs memory and power.
- Metric: Approximate Model Size (MB/GB).
-
Implication: A massive model (like
embeddinggemmaat 621MB) requires significantly more system resources and startup time than a tiny one (likegranite-embeddingat 62MB). For edge devices or cost-sensitive cloud deployments, a smaller, leaner model often wins.
3. The Quality Trade-Off (Vector Dimension)
This is the most critical factor for accuracy, but the hardest to test without real-world data.
- Metric: Vector Dimension (e.g., 384, 768, 1024).
- Implication: Dimension is complexity. A higher dimension means the vector has more numerical “slots” available to capture the nuance, semantic relationships, and context of the text.
- A 384-dimension vector is faster and smaller but might struggle with highly technical or subtle topics.
- A 768-dimension vector is slower and larger but offers superior semantic accuracy, especially in complex retrieval tasks.
The Ultimate Test: Semantic Accuracy
While checking speed, size, and dimension (as we did with our comparison script) provides an essential baseline, the ultimate test is semantic accuracy.
Two vectors that are numerically close together should mean the two pieces of text are semantically similar.
For example, a strong embedding model should place the vectors for “The Golden Gate Bridge spans the Pacific” and “The famous suspension bridge in San Francisco is very tall” very close together, even though the sentences use different words.
Moving Forward
Once you have established the performance baseline (latency and size), the next logical step is to introduce a semantic evaluation. This involves using a standard dataset with known relationships to verify which model provides the most accurate and useful distances between related and unrelated concepts.
By systematically testing for speed, size, and semantic quality, you move beyond guesswork and ensure your AI is built on the most robust foundation possible.
Motivation for this Test
I was really curious about the new capabilities of the Google embeddinggemma model popping up on Ollama! Since I've already been relying heavily on the Granite family of embedding models, specifically the lean granite-embedding:latest and the heavier granite-embedding:278m for different applications, I figured it was the perfect time to run a side-by-side benchmark to see where this new, high-dimension model stands in terms of speed, size, and potential semantic quality against my existing favorites.
My Test Case and Comparaison Basis
My project progress can be viewed as a technical progression, starting with simple foundational scripts dedicated to standalone embedding generation to verify Ollama connectivity and model output. This preparatory work culminated in the advanced compare_embeddings.py utility. This single file now executes a robust, three-way performance benchmark, meticulously comparing the speed, size, and vector dimension of the cutting-edge embeddinggemma:latest against the established efficiency of the two different versions of Granite, granite-embedding:latest and granite-embedding:278m, with the comprehensive results saved neatly as a Markdown report for easy analysis.
- Preparing my environment 👨🍳
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install ollama
- 1st Test — granite-embedding:latest — 67mb 🪨
import ollama
import json
import os
from datetime import datetime
# --- Configuration ---
EMBEDDING_MODEL = 'granite-embedding'
OUTPUT_DIR = './output'
# ---------------------
# 1. Initialize the Ollama client
client = ollama.Client()
# 2. Define the list of texts to embed
texts_to_embed = [
"What is the capital of France?",
"Paris is known for the Eiffel Tower.",
"The process of creating a vector representation of text is called embedding."
]
print(f"Generating embeddings using model: {EMBEDDING_MODEL}")
all_embeddings = []
start_time = datetime.now()
try:
for text in texts_to_embed:
print(f"-> Embedding: '{text[:30]}...'")
# Call the dedicated embeddings API endpoint
response = client.embeddings(
model=EMBEDDING_MODEL,
prompt=text # Correct: single string for prompt
)
# Use the singular 'embedding' key which was confirmed to work via curl
if 'embedding' in response:
# Store the vector along with the original text and timestamp
all_embeddings.append({
'timestamp': datetime.now().isoformat(),
'text': text,
'vector': response['embedding']
})
else:
print(f"Warning: Response for '{text[:30]}...' missing 'embedding' key.")
except Exception as e:
print(f"\nAn error occurred during embedding generation.")
print(f"Error details: {e}")
exit()
end_time = datetime.now()
total_duration = end_time - start_time
# --- Output and File Writing ---
# 3. Print Summary to Console
print("\n--- Results ---")
print(f"Total embeddings generated: {len(all_embeddings)}")
print(f"Total duration: {total_duration.total_seconds():.2f} seconds")
if all_embeddings:
dimension = len(all_embeddings[0]['vector'])
print(f"Dimension of each embedding vector: {dimension}")
first_vector_snippet = all_embeddings[0]['vector'][:5]
print(f"First embedding vector (snippet): {first_vector_snippet}...")
# 4. Create output directory if it doesn't exist
os.makedirs(OUTPUT_DIR, exist_ok=True)
print(f"\nOutput directory checked/created: {OUTPUT_DIR}")
# 5. Define timestamped filename
# Sanitize model name for filename and append current timestamp
safe_model_name = EMBEDDING_MODEL.replace(':', '_').replace('-', '_')
timestamp_str = start_time.strftime("%Y%m%d_%H%M%S")
filename = f"{safe_model_name}_embeddings_{timestamp_str}.json"
filepath = os.path.join(OUTPUT_DIR, filename)
# 6. Write results to JSON file
try:
with open(filepath, 'w') as f:
# Write a final dictionary containing metadata and the results list
json.dump({
'model': EMBEDDING_MODEL,
'timestamp': start_time.isoformat(),
'duration_seconds': total_duration.total_seconds(),
'dimension': dimension,
'embeddings': all_embeddings
}, f, indent=4)
print(f"✅ Successfully wrote results to: {filepath}")
except Exception as e:
print(f"❌ Error writing file to disk: {e}")
- The result for granite-embedding:latest — 67mb;
{
"model": "granite-embedding",
"timestamp": "2025-09-29T14:36:26.615078",
"duration_seconds": 0.068043,
"dimension": 384,
"embeddings": [
{
"timestamp": "2025-09-29T14:36:26.649465",
"text": "What is the capital of France?",
"vector": [
-1.0555170774459839
...
},
{
"timestamp": "2025-09-29T14:36:26.666867",
"text": "Paris is known for the Eiffel Tower.",
"vector": [
-1.2965751886367798,
...
},
{
"timestamp": "2025-09-29T14:36:26.683114",
"text": "The process of creating a vector representation of text is called embedding.",
"vector": [
0.9112906455993652,
...
},
]
}
]
}
- 2nd Test — embeddinggemma:latest — 671mb
import ollama
import json
import os
from datetime import datetime
import time
import sys # Import sys for better error reporting
# --- Configuration ---
EMBEDDING_MODEL = 'embeddinggemma:latest'
OUTPUT_DIR = './output'
# Configuration for retry logic
MAX_RETRIES = 3
RETRY_DELAY = 2 # seconds
# ---------------------
# 1. Initialize the Ollama client
client = ollama.Client()
# 2. Define the list of texts to embed
texts_to_embed = [
"What is the capital of France?",
"Paris is known for the Eiffel Tower.",
"The process of creating a vector representation of text is called embedding.",
"The quick brown fox jumps over the lazy dog."
]
print(f"Generating embeddings using model: {EMBEDDING_MODEL}")
all_embeddings_data = []
start_time = datetime.now()
# --- Health Check and Model Check with Retry Logic ---
try:
print("\n[STEP 1/3] Checking Ollama connectivity and model availability...")
# Loop to attempt connection and model list retrieval multiple times
for attempt in range(MAX_RETRIES):
available_models = []
response = {}
try:
# 1. Attempt to get the list of models
response = client.list()
# 2. SAFELY EXTRACT MODEL NAMES (FIXED LOGIC)
# The client returns a list of Model objects, so we must access the '.model' attribute.
model_list = response.get('models', [])
for m in model_list:
try:
# Correct way to access model name from the Model object
name = m.model
available_models.append(name)
except AttributeError:
# Fail gracefully if the object structure is unexpected
pass
# 3. Check if the corrected model name is in the available list
if EMBEDDING_MODEL in available_models:
print(f"✅ Ollama is connected and '{EMBEDDING_MODEL}' is available.")
break # Success! Exit the retry loop
# 4. If we successfully got the list (it's not empty) but the model is missing, raise a permanent error
if available_models:
raise ValueError(f"Model '{EMBEDDING_MODEL}' not found. Available models: {', '.join(available_models)}")
# 5. If available_models is empty, this is a connectivity/parsing issue, so we continue to retry
raise ConnectionError("Ollama returned an empty or malformed model list.")
except Exception as e:
if attempt < MAX_RETRIES - 1:
print(f"Connection attempt {attempt + 1}/{MAX_RETRIES} failed. Retrying in {RETRY_DELAY}s...")
# Log the raw response if it was fetched but was unusable
if response:
print(f" [DEBUG] Raw response on failure: {repr(response)}")
time.sleep(RETRY_DELAY)
else:
# Re-raise the final error with debug information
final_error_message = f"Failed to verify model list after {MAX_RETRIES} attempts. Last error: {e}"
if response:
final_error_message += f"\n [DEBUG] Last raw response received: {repr(response)}"
raise Exception(final_error_message)
# If we exited the loop without breaking (meaning model wasn't found after all attempts)
if EMBEDDING_MODEL not in available_models:
raise ValueError(f"Model '{EMBEDDING_MODEL}' could not be verified in the available list.")
except Exception as e:
# This block catches all final failures
print(f"\n❌ CRITICAL ERROR: Could not verify Ollama status or model list.")
print(f" -> Please ensure the Ollama server is running (http://localhost:11434) and responding correctly.")
print(f" -> Error details: {e}", file=sys.stderr)
sys.exit(1) # Use sys.exit(1) for clean exit on failure
# ------------------------------------
print("\n[STEP 2/3] Generating Embeddings...")
try:
for text in texts_to_embed:
print(f"-> Embedding: '{text[:40]}...'")
# Call the dedicated embeddings API endpoint
time.sleep(0.01)
# This call should now work if the connection is truly stable
response = client.embeddings(
model=EMBEDDING_MODEL,
prompt=text
)
# Access the confirmed singular 'embedding' key
if 'embedding' in response:
# Store the vector along with the original text and its individual timestamp
all_embeddings_data.append({
'timestamp': datetime.now().isoformat(),
'text': text,
'vector': response['embedding']
})
else:
print(f"Warning: Response for '{text[:40]}...' missing 'embedding' key. Full response: {response}")
except Exception as e:
print(f"\nAn unexpected error occurred during embedding generation.")
print(f"Error details: {e}")
sys.exit(1)
end_time = datetime.now()
total_duration = end_time - start_time
# --- Output and File Writing ---
# 3. Print Summary to Console
print("\n[STEP 3/3] Finalizing and Writing Output...")
print("\n--- Results ---")
print(f"Model used: {EMBEDDING_MODEL}")
print(f"Total embeddings generated: {len(all_embeddings_data)}")
print(f"Total duration: {total_duration.total_seconds():.2f} seconds")
if all_embeddings_data:
dimension = len(all_embeddings_data[0]['vector'])
print(f"Dimension of each embedding vector: {dimension}")
first_vector_snippet = all_embeddings_data[0]['vector'][:5]
print(f"First embedding vector (snippet): {first_vector_snippet}...")
# 4. Create output directory if it doesn't exist
os.makedirs(OUTPUT_DIR, exist_ok=True)
# 5. Define timestamped filename
# Sanitize model name for filename and append current timestamp
safe_model_name = EMBEDDING_MODEL.replace(':', '_').replace('-', '_')
timestamp_str = start_time.strftime("%Y%m%d_%H%M%S")
filename = f"{safe_model_name}_embeddings_{timestamp_str}.json"
filepath = os.path.join(OUTPUT_DIR, filename)
# 6. Write results to JSON file
try:
with open(filepath, 'w') as f:
# Write a final dictionary containing metadata and the results list
json.dump({
'model': EMBEDDING_MODEL,
'timestamp': start_time.isoformat(),
'duration_seconds': total_duration.total_seconds(),
'dimension': dimension,
'embeddings': all_embeddings_data
}, f, indent=4)
print(f"✅ Successfully wrote results to: {filepath}")
except Exception as e:
print(f"❌ Error writing file to disk: {e}")
else:
print("Failed to generate any embeddings.")
- The result for embeddinggemma:latest— 671mb;
{
"model": "embeddinggemma:latest",
"timestamp": "2025-09-29T14:43:54.100824",
"duration_seconds": 0.753729,
"dimension": 768,
"embeddings": [
{
"timestamp": "2025-09-29T14:43:54.662257",
"text": "What is the capital of France?",
"vector": [
-0.15460720658302307,
...
},
{
"timestamp": "2025-09-29T14:43:54.730422",
"text": "Paris is known for the Eiffel Tower.",
"vector": [
-0.18294751644134521,
...
},
{
"timestamp": "2025-09-29T14:43:54.793922",
"text": "The process of creating a vector representation of text is called embedding.",
"vector": [
-0.13661450147628784,
...
},
{
"timestamp": "2025-09-29T14:43:54.854547",
"text": "The quick brown fox jumps over the lazy dog.",
"vector": [
-0.14782573282718658,
...
]
}
]
}
- OK, this is great, now let’s add another embedding model ‘granite-embedding:278m’ and test all these and make a comparaison report!
import ollama
import json
import os
from datetime import datetime
import time
import sys
# --- Configuration ---
# Models to test and their approximate sizes.
# Added 'granite-embedding:278m' for a three-way comparison.
MODELS_TO_COMPARE = [
{'name': 'granite-embedding:latest', 'size_approx_mb': 62},
{'name': 'granite-embedding:278m', 'size_approx_mb': 278},
{'name': 'embeddinggemma:latest', 'size_approx_mb': 621}
]
OUTPUT_DIR = './output'
MAX_RETRIES = 3
RETRY_DELAY = 2 # seconds
REPORT_FILENAME_PREFIX = 'embedding_comparison_report'
# 1. Initialize the Ollama client
client = ollama.Client()
# 2. Define the list of texts to embed
texts_to_embed = [
"What is the capital of France?",
"Paris is known for the Eiffel Tower.",
"The process of creating a vector representation of text is called embedding.",
"The quick brown fox jumps over the lazy dog."
]
def check_model_availability(model_name):
"""Checks if a model is available with retry logic."""
for attempt in range(MAX_RETRIES):
try:
response = client.list()
model_list = response.get('models', [])
available_models = []
for m in model_list:
try:
# Access the model name using the correct attribute syntax (.model)
name = m.model
available_models.append(name)
except AttributeError:
pass
if model_name in available_models:
return True, ""
if available_models:
return False, f"Model '{model_name}' not found. Available models: {', '.join(available_models)}"
raise ConnectionError("Ollama returned an empty or malformed model list.")
except Exception as e:
if attempt < MAX_RETRIES - 1:
print(f"[{model_name}] Connection attempt {attempt + 1}/{MAX_RETRIES} failed. Retrying in {RETRY_DELAY}s...")
time.sleep(RETRY_DELAY)
else:
return False, f"Failed to verify model list after {MAX_RETRIES} attempts. Last error: {e}"
return False, f"Model '{model_name}' could not be verified in the available list."
def run_embeddings_for_model(model_config, texts):
"""Generates embeddings for a given model and returns results and metadata."""
model_name = model_config['name']
# 1. Check availability
is_available, check_error = check_model_availability(model_name)
if not is_available:
print(f"❌ Skipping {model_name} due to error: {check_error}", file=sys.stderr)
return None, None, None, check_error
print(f"✅ Model '{model_name}' is available. Starting embedding generation...")
all_vectors = []
start_time = time.time()
try:
for text in texts:
# Generate embedding
response = client.embeddings(
model=model_name,
prompt=text
)
if 'embedding' in response:
all_vectors.append(response['embedding'])
else:
print(f"Warning: Missing 'embedding' key for '{text[:20]}...' using {model_name}")
except Exception as e:
# Catch errors during the actual embedding call
return None, None, None, f"Error during embedding generation: {e}"
end_time = time.time()
duration = end_time - start_time
if not all_vectors:
return None, None, None, "No vectors were generated."
dimension = len(all_vectors[0])
return all_vectors, dimension, duration, "Success"
def generate_markdown_report(results, report_start_time):
"""Generates a Markdown table comparing the results of the embedding tests."""
# Header row: Metric and Model Names
header = ["Metric"] + [data['name'] for data in MODELS_TO_COMPARE]
# Separator row
separator = [":---"] + [":---:"] * len(MODELS_TO_COMPARE)
table_rows = [header, separator]
# Mapping for report metrics
metric_map = {
'Status': 'Status',
'Total Texts Embedded': 'Total Texts Embedded',
'Vector Dimension': 'Vector Dimension',
'Approx. Model Size (MB)': 'Approx. Model Size (MB)',
'Total Duration (s)': 'Total Duration (s)',
}
# Data rows
for metric_label, result_key in metric_map.items():
row = [f"**{metric_label}**"]
for model_config in MODELS_TO_COMPARE:
model_name = model_config['name']
data = results[model_name]
if result_key == 'Status':
value = '✅ Success' if data['status'] == 'Success' else f'❌ Error'
elif result_key == 'Total Texts Embedded':
value = str(len(data['vectors'])) if data['status'] == 'Success' else 'N/A'
elif result_key == 'Vector Dimension':
value = str(data['dimension']) if data['status'] == 'Success' else 'N/A'
elif result_key == 'Approx. Model Size (MB)':
value = str(model_config['size_approx_mb'])
elif result_key == 'Total Duration (s)':
value = f"{data['duration']:.3f}" if data['status'] == 'Success' else 'N/A'
else:
value = 'Unknown'
row.append(value)
table_rows.append(row)
# Convert rows to final Markdown format
markdown_table = '\n'.join(['| ' + ' | '.join(row) + ' |' for row in table_rows])
# Build the final content
markdown_content = f"# LLM Embedding Model Comparison Report\n\n"
markdown_content += f"Report Generated: {report_start_time.strftime('%Y-%m-%d %H:%M:%S')}\n"
markdown_content += f"Texts Embedded Per Model: {len(texts_to_embed)}\n\n"
markdown_content += markdown_table
# Add detailed notes for any errors
error_notes = ""
for model_config in MODELS_TO_COMPARE:
model_name = model_config['name']
data = results[model_name]
if data['status'] != 'Success':
error_notes += f"\n- **{model_name} Error:** {data['error']}"
if error_notes:
markdown_content += "\n\n## Notes\n"
markdown_content += "Some models encountered errors during availability checks or generation:\n"
markdown_content += error_notes
return markdown_content
# --- Main Execution ---
def main():
print(f"Starting comparison test for {len(MODELS_TO_COMPARE)} models...")
report_start_time = datetime.now()
all_results = {}
# Run tests sequentially
for model_config in MODELS_TO_COMPARE:
model_name = model_config['name']
print(f"\n--- Testing Model: {model_name} ---")
vectors, dimension, duration, status = run_embeddings_for_model(model_config, texts_to_embed)
all_results[model_name] = {
'vectors': vectors if vectors else [],
'dimension': dimension,
'duration': duration,
'status': status,
'error': status if status != 'Success' else None
}
# Generate Report
markdown_content = generate_markdown_report(all_results, report_start_time)
# Write Report to File
os.makedirs(OUTPUT_DIR, exist_ok=True)
timestamp_str = report_start_time.strftime("%Y%m%d_%H%M%S")
filename = f"{REPORT_FILENAME_PREFIX}_{timestamp_str}.md"
filepath = os.path.join(OUTPUT_DIR, filename)
try:
with open(filepath, 'w') as f:
f.write(markdown_content)
print(f"\n✅ Comparison report successfully written to: {filepath}")
except Exception as e:
print(f"\n❌ Error writing report file to disk: {e}", file=sys.stderr)
if __name__ == '__main__':
main()
- Here comes the comparaison table ⬇️
# LLM Embedding Model Comparison Report
Report Generated: 2025-09-29 15:18:14
Texts Embedded Per Model: 4
| Metric | granite-embedding:latest | granite-embedding:278m | embeddinggemma:latest |
| :-------------------------- | :----------------------: | :--------------------: | :-------------------: |
| **Status** | ✅ Success | ✅ Success | ✅ Success |
| **Total Texts Embedded** | 4 | 4 | 4 |
| **Vector Dimension** | 384 | 768 | 768 |
| **Approx. Model Size (MB)** | 62 | 278 | 621 |
| **Total Duration (s)** | 0.510 | 1.530 | 19.812 |
So who’s the winner? That’s the million-dollar question! The “winner” in this case depends entirely on what you prioritize for your application: speed and low latency, or semantic quality and nuance.
🥇 The Three-Way Win
- Winner for Speed and Small Footprint granite-embedding:latest (62 MB): best for mobile apps, edge devices, simple, high-throughput batch processing, or when minimizing infrastructure cost is the primary goal. It sacrifices some semantic quality for blazing speed and minimal resource usage.
- Winner for Semantic Quality and Nuance embeddinggemma:latest (621 MB): best for complex RAG systems, detailed semantic search, or document clustering where high accuracy on subtle language differences is required. You get the best quality, but you pay for it with larger file size and higher latency.
- Winner for Balance and Compromise granite-embedding:278m best for the majority of general-purpose applications. This model offers a significant step up in semantic performance compared to the 62 MB version, while still being much smaller and faster than the massive 621 MB Gemma model. It hits the “sweet spot” of performance and efficiency.
Conclusion
According to the tests above, the true champion is granite-embedding:278m if you are looking for the best all-around compromise. If you only care about being the fastest, use the smallest model. If you only care about maximum accuracy, use the largest one.
The fundamental value of embeddings for the ecosystem of Generative AI and LLMs is that they act as the essential, numerical bridge between raw knowledge and accurate generation. They are the indispensable infrastructure powering Retrieval-Augmented Generation (RAG), which is the mechanism that grounds large language models in specific, proprietary, and up-to-the-minute data. By efficiently converting vast datasets into searchable vector indexes, embeddings allow the LLM to instantly retrieve the highest-fidelity source material — its “source of truth” — before formulating a response. This capability transforms LLMs from being general knowledge generators prone to hallucination into specialized, context-aware tools, unlocking their true enterprise value for applications requiring high precision, reliability, and real-time accuracy.
The critical takeaway from benchmarking embedding models is that no single model reigns supreme for every scenario; the optimal choice must be a calculated decision driven entirely by the application’s unique requirements and deployment goals. Whether the priority is the low latency and minimal footprint offered by a model like granite-embedding:latest, the semantic richness afforded by the high-dimensional vectors of embeddinggemma:latest, or the practical balance achieved by a mid-sized option, the final selection determines the entire system's efficiency and accuracy. Intentional testing is therefore non-negotiable, ensuring the chosen embedding model perfectly aligns the project's technical architecture with its business objectives.
Top comments (0)