HNSW indexes in pgvector are tuned with critical parameters like m and ef_construction to balance speed and accuracy in high-dimensional vector searches. These settings directly impact the efficient use of system resources and the quality of search results, especially in applications with large datasets and real-time search requirements. Optimizing vector database performance begins with finding the right m and ef_construction values and requires an iterative process based on the application's specific needs.
What is HNSW and Why is it Important?
HNSW (Hierarchical Navigable Small World) is an Approximate Nearest Neighbor (ANN) algorithm used to quickly find nearest neighbors in high-dimensional vector spaces. This algorithm organizes vectors within a multi-layered graph structure; each layer represents the density of connections between vectors. Upper layers have sparser and broader connections, while lower layers have denser and more localized connections.
The search process starts from the upper layers, approaching the target vector, and then performs a more precise search in the lower layers to find the nearest neighbors. This hierarchical structure logarithmically reduces search time, mitigating the "curse of dimensionality" problem. Especially in scenarios involving millions of vectors, HNSW indexes offer much more efficient results compared to a full scan.
pgvector HNSW Parameters: m and ef_construction
When optimizing HNSW indexes in pgvector, two primary parameters are used: m and ef_construction. These parameters directly affect how the index is built, and consequently, both search accuracy (recall) and memory usage and query performance. Correct settings require a careful balance based on your application's requirements.
m(Maximum number of connections per layer): Determines the maximum number of neighbors a node can have in each layer. Highermvalues lead to a denser graph, which offers more path options during search, thereby increasing recall. However, it also increases the index's memory footprint and build time because each node needs to store more neighbors. Themvalue is typically set between 5 and 48, with a default of16in pgvector.ef_construction(Size of the dynamic list for constructing the graph): Specifies the number of candidate nearest neighbors to search for each new node during index construction. Higheref_constructionvalues result in a higher quality and more accurate index structure, potentially leading to better recall. However, this significantly prolongs the index build time. This parameter determines the fundamental quality of the index but does not directly impact memory usage as much asm. Anef_constructionvalue of at least twicemis generally recommended, with a default of64in pgvector.
These two parameters interact with each other, and the ideal combination depends on your dataset size, vector dimensionality, and your application's recall/performance tolerance.
How to Balance Recall and Performance?
The balance between recall and performance is a critical aspect of vector search systems. High recall refers to the system's ability to find all truly relevant results, while high performance indicates fast query response times and low resource usage. Increasing m and ef_construction parameters generally increases recall because the index becomes more comprehensive and accurate.
However, this comes with a performance cost, as it requires a larger index size, longer index build times, and more computation per query. For example, in an e-commerce application, a very high recall might be targeted for product recommendations, while in a content moderation system, fast response times might be a higher priority.
| Parameter | Effect when Increased (+) | Effect when Decreased (-) |
|---|---|---|
m |
Recall increases, memory increases, build time increases | Recall decreases, memory decreases, build time decreases |
ef_construction |
Recall increases, build time increases | Recall decreases, build time decreases |
ef_search |
Recall increases, query time increases | Recall decreases, query time decreases |
Practical Application and Index Creation
When creating an HNSW index in pgvector, you can specify parameters using the CREATE INDEX command. While default values are often a good starting point for most scenarios, more specific adjustments may be needed in production environments. Typically, values like 16 or 32 for m and 64 to 200 for ef_construction are commonly used. However, these values vary depending on vector dimensionality, data distribution, and application tolerance.
The example below demonstrates how to create a basic HNSW index and set its parameters. The vector dimension (here 1536) will vary depending on the embedding model you use. For OpenAI's text-embedding-3-small model, 1536 dimensions is a common value.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE items (
id serial PRIMARY KEY,
embedding vector(1536)
);
-- Example data insertion (in a real scenario, there could be millions of rows)
INSERT INTO items (embedding) VALUES
(ARRAY[0.1, 0.2, ..., 0.9]::vector(1536)),
(ARRAY[0.3, 0.4, ..., 0.1]::vector(1536)),
(ARRAY[0.9, 0.8, ..., 0.7]::vector(1536));
-- Create HNSW index
-- m: Number of neighbors per layer (default: 16)
-- ef_construction: Number of candidates to search during index construction (default: 64)
CREATE INDEX ON items USING hnsw (embedding vector_l2_ops) WITH (m = 16, ef_construction = 100);
The vector_l2_ops operator class is used for L2 (Euclidean) distance.
Once the index is created, the ef_search parameter also affects search quality during queries. This parameter determines the number of candidate neighbors to search during a query and can be set on a session basis or within application code using a command like SET hnsw.ef_search = 100;. The higher the ef_search value, the higher the recall, but also the longer the query time. The default value for ef_search is 40.
Memory Usage and Disk Space Optimization
The memory usage of HNSW indexes is directly related to the m parameter. Since m neighbors' information is stored for each vector, increasing the m value significantly increases the index size on disk and thus its memory requirement. When working with large datasets, this can put serious pressure on server resources.
ℹ️ Calculating Index Size
Index size increases significantly as the
mparameter and the number of vectors increase. The actual size will be higher due to additional index metadata and PostgreSQL's own overhead. You can query the actual size of the index using thepg_indexes_size('your_index_name')function.
To optimize memory usage, it's important to keep the m value at the lowest level acceptable for the application. If recall requirements are not very strict, lower m values (e.g., 8 or 12) can be tried. Additionally, ensuring that PostgreSQL's shared_buffers and work_mem settings are sufficient for indexing and query processes is critical for performance. Insufficient values for these can lead to frequent disk write/read operations, reducing performance and negatively impacting the system's overall response time.
Monitoring and Iterative Improvement
Continuous monitoring and iterative improvement are essential to ensure HNSW index parameters are correctly set. Regularly tracking recall and performance metrics is crucial for understanding the impact of changes. This process requires continuous observation of system dynamics rather than a "set and forget" approach.
- Recall Measurement: To measure recall, a small test dataset, often referred to as "ground truth," where the true nearest neighbors are known beforehand, is typically used. The results of queries performed on this dataset are compared with the expected correct results to calculate the recall rate. This is usually done in a QA or test environment.
- Performance Measurement: Monitoring query times with
EXPLAIN ANALYZEis very useful for understanding index effectiveness and the query planner's behavior. Additionally, tools likepg_stat_statementscan be used to identify overall query performance and slow queries. In real-time systems, API response times and database latency metrics should also be closely monitored. - Resource Monitoring: CPU, memory, and disk I/O usage indicate the impact of indexing and querying operations on the system. High memory usage or disk I/O may suggest that
mandef_constructionvalues need to be re-evaluated. Especially intense resource usage during index creation indicates that timing needs to be carefully managed to avoid affecting other server workloads.
Conclusion
Tuning HNSW indexes in pgvector is a critical step that directly impacts your application's vector search performance and accuracy. The m and ef_construction parameters determine the index structure and, consequently, resource consumption and search quality. The choice of these parameters depends on whether pure recall or speed and memory efficiency are prioritized.
It's important to remember that optimal parameters will vary based on your dataset, vector dimensionality, and the specific requirements of your application. Therefore, conducting comprehensive experiments in test environments, carefully monitoring recall and performance metrics, and making iterative adjustments based on your findings is the most appropriate approach. This way, you can achieve a vector search solution that is both high-performing and accurate.
Top comments (0)