[Free E-Book] The World of Vector Databases & AI Applications!

Understanding Vector Databases

Everything about artificial intelligence (AI) continues to explode — from the amount of AI applications introduced every day to the ever-expanding capabilities of OpenAI’s ChatGPT. But even within this overall explosion, hot spots still exist. They’re marked by vector databases.

Vector databases are a critical component in building AI and ChatGPT-based applications and are also where businesses run into some of their biggest challenges in creating these applications (Chapters 5-8 cover apps in more detail).

Technical professionals, database architects, developers, and organizations of all kinds succeed or fail based on how they use vector databases and their associated functions. Understanding what vector databases are, and recognizing the key traits and features of a vector database, can help you choose the tools and approaches you need to meet these challenges where others fall short.

Defining Vector Databases

Vector databases provide the ability to store data in a numerical format. Imagine you have words like Boy, King, Prince, Princess, Queen, Girl, and Woman. A vector representation of each word may be something like 2.12, 3.25, 2.34, 1.23. Each numerical representation is similar to latitude and longitude coordinates of that word to represent their location in a multi-dimensional space (longitude and latitude are two-dimensional).

When a word or a piece of data (an image or a sentence or even
an entire paragraph) is converted into a vector representation, it inherently has meaning associated with it by the virtue of the distance between each vector. For example, when the words Boy and King are converted to vectors, they are inherently closer to each other than the words Girl, Woman, or Princess.

The market for vector databases is exploding. The world’s largest technology companies depend on these databases to achieve their AI use cases. And most companies are increasingly dependent on real-time applications, too. The reason why data is often converted and stored as vectors is because it then enables software and programs to do semantic search.

Looking at the Three Key Steps for Vector Search

When vectorizing and searching data, three steps are involved:
1. Vectorize data
Converting a word, sentence, paragraphs, audio, or even images requires using transformer models. These are now generally available through application programming interface (API) calls. Some of the transformers are open source and free, while others, like OpenAI, require paying a small fee to convert data to a vector. The output of this step is vectors that now need to be stored in a vector store or database.

2. Store data
After the data has been converted into vectors, it needs to be
stored in a database. Given that the vectors are in essence numbers separated by commas, these can be stored in a number of data types, including in a binary/BLOB storage. During this step, vectors are also stored as indexes so they’re faster to retrieve and search in the last step. There are a number of different algorithms to create and store vectors as indexes.

3. Perform a semantic search
A semantic search over the vectors is carried out by using one
of the following algorithms:
• Cosine similarity: This involves measuring the angle
between two objects. The smaller the angle, the closer the
objects.
• Euclidean distance: This involves measuring the distance
between the two objects.

There is lot more in this e-book about GenAI applications and vector databases.

The new book “Vector Databases & AI Applications for Dummies” is now available. Register to get your free copy.

Visit GitHub for online supporting material for this book.

Also take a look at my vector database article.

WTF Is a Vector Database: A Beginner's Guide!

Pavan Belagatti ・ Aug 25 '23

#database #ai #devops #developers