DEV Community

Rhea Kapoor
Rhea Kapoor

Posted on

What I Learned from Benchmarking Elastic Cloud vs. Zilliz Cloud for Vector Search

Introduction
When I began testing vector search systems at scale, I was curious whether mature platforms like Elastic Cloud, with vector search plugins, could match purpose-built solutions like Zilliz Cloud. I’ve worked on multiple LLM-based apps—primarily Retrieval-Augmented Generation (RAG) pipelines—and I wanted to understand not just what each platform could do, but how they behave under real workloads. What I discovered reshaped how I think about retrofitting traditional databases for high-dimensional similarity search.

1. Vector Plugins vs. Vector Databases: What's the Real Difference?

Early on, I assumed adding a vector search plugin to a general-purpose database (like Elasticsearch) would be enough for semantic similarity. But I soon realized it’s like stuffing a jet engine into a hatchback. Sure, you’ll get motion, but not without heat and noise—especially at scale.
Analogy:

  • Elastic Cloud (with ANN plugin): Think of it as an after-market solution. You get the capability, but it’s constrained by legacy internals (e.g., static sharding, shared compute, fixed insert/query coupling).
  • Zilliz Cloud: Built ground-up for vector operations. Query, ingestion, storage, and indexing are separate, allowing for optimization along each axis. This distinction surfaced clearly during benchmarking.

2. Benchmark Setup: Apples to Apples

To get actionable results, I ran two real-world dataset configurations using VectorDBBench:

Image description

Tested Systems:

  • Zilliz Cloud (1cu-perf): Optimized for performance.
  • Zilliz Cloud (1cu-cap): Optimized for cost efficiency.
  • Elastic Cloud: Up to 2.5vCPU and 8GB RAM instance. All systems were tested on near-identical hardware footprints.

3. Results: QPS, Latency, and Cost

3.1 Queries per Second (QPS)

On Dataset A, Zilliz Cloud outperformed Elastic Cloud by:

  • 34x (perf mode)
  • 22x (capacity mode) On Dataset B:
  • 26x and 13x, respectively. This shows Elastic Cloud’s architecture struggles under large vector loads, especially with high-dimensional embeddings.

3.2 Queries per Dollar (QP$)

Zilliz Cloud was:

  • 102x more cost-efficient on Dataset A (perf mode)
  • 65x better on Dataset A (cap mode)
  • 79x and 38x, respectively, on Dataset B If you’re running production pipelines, this cost difference accumulates fast, especially for vector-heavy RAG workloads.

3.3 P99 Latency

For me, latency is non-negotiable—especially when inference APIs are in the loop. The results were stunning:

Image description

Elastic Cloud’s latency curve degrades significantly with vector count growth. Zilliz Cloud retained sub-50ms P99 latencies even under pressure.

4. Feature Comparison: When the Details Start to Matter

Here's what I compiled when aligning platform capabilities:
Image description

Elastic does win on binary vector support, so if your application uses fingerprint-like data (e.g., malware hashes), that’s something to consider.

5. Real-World Tradeoffs and Design Choices

What I liked about Zilliz Cloud is its dynamic segment placement—as your data grows, it doesn’t require pre-sharding. Elastic Cloud still depends on static sharding, which caused performance cliffs in my high-ingest tests.
Also, Zilliz supports stream + batch ingestion, whereas Elastic’s ingestion model is more batch-oriented—something that matters if you’re doing real-time updates in a chatbot or recommendation engine.

6. Migration Notes: Elastic to Zilliz

While Elastic Cloud offers ease of setup and RESTful APIs, migrating vector workloads to a purpose-built system pays dividends. I used the official migration guide and was able to:

  • Export vector fields via elasticsearch-dump
  • Normalize and transform embedding data
  • Import via pymilvus.bulk_insert() The API differences were minor, but the operational differences were night and day.

7. Code Snippet: Using Zilliz Cloud with Python 

from pymilvus import Collection, utility
collection_name = "rag_vectors"
dim = 768
schema = CollectionSchema([
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=dim)
])
collection = Collection(name=collection_name, schema=schema)
collection.load()
results = collection.search(
    data=[[...]],  # query vector
    anns_field="embedding",
    param={"metric_type": "L2", "params": {"nprobe": 10}},
    limit=5
)
Enter fullscreen mode Exit fullscreen mode

This took under 5 minutes to deploy using Zilliz’s UI and Python SDK.

8. Reflection: What This Taught Me

I used to think feature parity was enough—but architecture matters more than features. Elastic Cloud is a capable platform, but it’s fundamentally retrofit for vectors. Zilliz Cloud is built with vectors as the first-class citizen, and it shows in every performance and usability metric.
What I’ll Explore Next:

  • GPU-accelerated vector search (e.g., CAGRA or GPU_FLAT indexes)
  • Query planning under hybrid search with 100M+ entries
  • Comparing open-source Milvus on Kubernetes vs. Zilliz Cloud in air-gapped environments

Top comments (0)