Vector Databases in AI Projects: Are They Really Necessary?

Vector Databases in AI Projects: Are They Really Necessary?

AI projects, especially those based on Large Language Models (LLMs), are rapidly gaining popularity. A cornerstone of these projects often turns out to be "vector databases." Popular patterns like Retrieval-Augmented Generation (RAG) necessitate storing and querying vector representations of texts to generate meaningful and contextual responses. However, like any technology, vector databases come with their own challenges and costs. So, do you truly need these complex databases at every stage of your project? Let's delve deeper into this topic.

In this post, I will explain why vector databases have become popular, what advantages and disadvantages they offer, what alternative approaches exist, and provide concrete examples from my own experiences. My goal is to question this technology and help you make the right decision for your projects. Remember, the best architecture is always the simplest one.

Why Have Vector Databases Become Popular?

Even though LLMs are trained with billions of parameters, their access to current or specific information on a particular topic is limited. This is precisely where the RAG architecture comes in. We convert texts into numerical vectors (embeddings) that carry the semantic meaning of words or sentences and store these vectors in specialized databases. When a query arrives, we create its vector and search the database for the closest vectors (i.e., the most relevant information). By adding this relevant information to the LLM, we enable it to provide more accurate and up-to-date responses.

For example, imagine you are developing a customer service bot. When a customer asks, "What is your return policy?", we create a vector for this question and find the vectors in your database that are closest to texts related to "return policy." By sending these texts to the LLM, we enable the bot to explain the current and correct return policy. Vector databases excel at performing this "similarity search" operation within milliseconds, even with large datasets.

Advantages Offered by Vector Databases

The biggest advantage of vector databases is their ability to perform very fast and efficient similarity searches on high-dimensional vectors. This provides a significant advantage, especially when working with large datasets. Traditional databases search for exact word matches or specific keywords within text, while vector databases operate on "semantic similarity." This means that even if the exact words of your query are not in the database, it can still find results that are semantically close.

This capability is not limited to LLM projects. It can also be used in many areas such as image recognition, recommendation systems, and anomaly detection. For example, you can use vector representations of product images to recommend similar products on an e-commerce site. If a customer is looking for a specific style of shoe, the system can recommend other shoes whose vectors are close to that shoe. Vector databases offer a powerful tool for solving such complex matching problems.

Disadvantages and Costs of Vector Databases

However, vector databases are not a panacea. First, their setup and management are generally more complex. Some vector databases may require specialized hardware or pose challenges in terms of scaling. For instance, solutions like Milvus or Pinecone are often preferred as SaaS (Software as a Service) rather than being set up and managed on your own infrastructure. This translates to monthly subscription costs.

Another significant disadvantage is cost. Especially when working with large datasets, storing vectors can require substantial disk space. Furthermore, the query performance of these databases can vary depending on the data size, the index used, and the query complexity. In my own projects, I've seen that when working with hundreds of millions of vectors, storage costs alone can constitute a significant budget item. When you add query time and scaling costs on top of that, the total cost can become quite high.

⚠️ Cost Analysis

In one project, 100 million 768-dimensional embeddings required approximately 150 GB of disk space. We faced significant infrastructure investment or monthly SaaS costs for managing, backing up, and querying this data.

Alternative Approaches: RAG Without a Vector Database

So, are we forced to use a vector database? The answer is no. Especially in the early stages of your project or if your dataset is not very large, simpler and more cost-effective solutions are available.

Traditional databases have gained vector search capabilities in recent years. For example, the pgvector extension for PostgreSQL allows you to store vectors in your database and perform similarity searches. This enables you to manage your vector data using your existing PostgreSQL infrastructure, freeing you from the burden of setting up and managing a separate vector database.

Similarly, search engines like Elasticsearch also offer vector search capabilities. If you are already using Elasticsearch, it might make sense to use it as a vector store for your RAG projects. These approaches can offer very practical solutions, especially if your dataset is in the tens of millions of vectors and you don't have complex scaling needs.

My Experiences with PostgreSQL pgvector

While improving a demand forecasting model for a production ERP system, I wanted to perform semantic analysis of product descriptions. I had a dataset containing approximately 5 million product descriptions. Initially, I considered a separate vector database, but I decided to try installing the pgvector extension on my existing PostgreSQL 14 server.

The installation was quite simple. I pulled the PostgreSQL image with Docker and added pgvector to the shared_preload_libraries setting. Then, I generated embeddings and stored them in a VECTOR type column in my PostgreSQL table. For querying, I used functions like vector_cosine_distance.

For example, when I wanted to find similar product descriptions for a given product description, I ran a query like this:

SELECT
  id,
  product_description,
  embedding <-> 'my_query_vector' AS distance
FROM
  products
ORDER BY
  distance
LIMIT 10;

This query returned similar product descriptions from approximately 5 million rows within an average of 200-300 milliseconds. This performance was perfectly sufficient for me, and I avoided the cost of managing a separate database. This approach is quite sensible, especially during initial prototyping phases or for medium-sized datasets.

💡 pgvector Advantages

• Uses existing PostgreSQL infrastructure.
• Does not require separate database management.
• A cost-effective solution.
• Provides relatively easy integration.

When Is a Vector Database Truly Necessary?

So, when should we opt for a separate vector database? Generally, a few situations can trigger this decision:

1. Very Large Datasets: If you are working with billions or trillions of vectors, the performance and scalability capabilities offered by traditional databases might be insufficient. At this point, databases specifically optimized for vector searches (like Pinecone, Weaviate, Milvus, Qdrant) come into play. These systems offer better performance at scale thanks to their distributed architectures and specialized indexing algorithms.

2. High Query Speed and Low Latency Requirements: In real-time applications, even milliseconds can be critical. Separate vector databases generally have more advanced query processing mechanisms and distributed query capabilities. This allows them to handle a large number of concurrent queries with lower latency.

3. Advanced Vector Features and Indexing Options: Some vector databases offer more advanced indexing algorithms like HNSW (Hierarchical Navigable Small Worlds). These algorithms can help you strike a better balance between search accuracy and speed. Additionally, features like metadata filtering and hybrid search (vector + keyword) are necessary for more complex search scenarios.

4. Specific Management and Scaling Needs: If you don't want to or can't manage your own infrastructure, SaaS vector databases (like Pinecone, Weaviate Cloud) might be more suitable for you. These services handle infrastructure management and scaling on your behalf.

While working on a financial analysis platform, we wanted to analyze the texts of millions of financial reports and extract summaries of specific trends. Our dataset was around 500 million documents. At this scale, anticipating that PostgreSQL's performance might degrade over time, we decided to try Weaviate. Thanks to Weaviate's metadata filtering and hybrid search capabilities, we were able to find the reports we were looking for much faster and more accurately.

Vector Indexing Techniques and Trade-offs

One of the most important factors determining the performance of vector databases is the indexing technique used. Vector searches are typically performed using Approximate Nearest Neighbor (ANN) algorithms. This is because Exact Nearest Neighbor (ENN) search is computationally very expensive in high-dimensional spaces.

Some popular ANN indexing techniques include:

• HNSW (Hierarchical Navigable Small Worlds): Generally offers a good balance of high speed and accuracy. However, memory usage can be high, and index creation time can be long.
• IVF (Inverted File Index): Uses less memory and has faster index creation time. However, search accuracy may not be as high as HNSW.
• LSH (Locality-Sensitive Hashing): Effective especially for high-dimensional data, but search accuracy is generally lower.

The indexing technique you choose directly impacts search speed, search accuracy, memory usage, and index creation time. For example, if speed is critical for you, HNSW might be a good option, but if you have memory constraints, IVF might be more suitable. Understanding these trade-offs is vital for selecting the right vector database and configuration.

A Pragmatic Approach: Choosing Based on Need

In conclusion, whether or not to use a vector database in AI projects depends on your project's specific requirements. Instead of imposing the same solution on every project, you can make a more informed decision by following these steps:

1. Determine Your Dataset Size: How many vectors will you store? Millions or billions?
2. Understand Your Performance Requirements: How critical is query latency for you? Are you building a real-time application or performing batch analyses?
3. Evaluate Costs: Consider factors such as infrastructure costs, SaaS subscriptions, and management effort.
4. Review Your Existing Infrastructure: Are you already using databases like PostgreSQL or Elasticsearch? Can the vector capabilities of these tools meet your needs?
5. Assess Technical Depth: Does your team have the expertise to manage complex vector databases?

If your dataset is medium-sized and your team is proficient in existing database management, starting with solutions like pgvector or Elasticsearch is generally the most sensible approach. This reduces both costs and operational complexity. As your project grows or your performance/scalability needs increase, you can consider migrating to specially designed vector databases.

Remember, the most complex technology is not always the best solution. The key is to accurately analyze your needs and build the most suitable and sustainable architecture for your project.