AI Explainer: What’s Our Vector, Victor?

The reference in the title is from the 1980 film “Airplane.” If you recognized it, I’m sorry to inform you that you’re getting old (like me).

Let’s talk about vector databases. A vector database is a type of database that stores and organizes data in the form of vectors, which are mathematical representations of objects or entities in a multidimensional space. These databases are designed to efficiently handle vector data, enabling various applications in data retrieval, similarity search and machine learning. In the context of large language models (LLMs), vector databases can be used to store and retrieve vector representations of words, sentences or documents.

I try to keep these “AI Explainer” blog posts relatively short so they’re easily consumable, but this one is longer because there are several aspects worth covering, even in an introduction to vector databases. You can skip to the part(s) you’re interested in reading — the first section is a history of vector databases, the second is comparing vector databases to traditional relational databases, and the third explains how vector databases are used by LLMs.

History of Vector Databases

Vector databases have been around for several decades, and their development has evolved alongside advancements in database technology, machine learning and data retrieval. The concept of organizing and querying data based on vectors is rooted in mathematical principles and has found applications in various fields.

Here's a brief timeline highlighting key milestones in the development of vector databases:

• 1960s-1970s - Early Database Systems: The foundations of relational database systems were laid in the 1960s and 1970s. While these early systems focused on tabular data and SQL queries, the idea of indexing and querying based on mathematical representations was present in some contexts.
• 1980s-1990s - Spatial Databases: In the 1980s and 1990s, spatial databases emerged to handle spatial data, such as maps and geographical information. These databases used geometric and vector-based representations for efficient spatial queries.
• 2000s - Content-Based Image Retrieval: In the 2000s, vector databases gained prominence in content-based image retrieval systems. These systems used vector representations of images to enable similarity searches and retrieval based on visual content.
• 2010s - Advances in Machine Learning: With the rise of machine learning, especially deep learning, vector representations became prevalent in natural language processing and other domains. Embeddings, which are dense vector representations, became a standard way to represent words, sentences and documents.
• 2010s-Present - General-Purpose Vector Databases: In recent years, there has been an increased focus on developing general-purpose vector databases that can handle a wide range of vector data, not limited to spatial or image data. These databases are designed to support efficient vector storage, indexing and similarity search operations.
• Distributed Vector Databases: More recently, there has been a trend toward developing distributed vector databases that can scale horizontally to handle large datasets and support high-throughput queries. This is particularly relevant in the context of big data and large-scale machine learning applications.

While the term "vector database" may not have been widely used in the early years, the underlying principles of organizing and querying data based on vector representations have been present in various forms throughout the history of database and information retrieval research. Today, vector databases play a crucial role in supporting applications in machine learning, information retrieval and similarity search across diverse domains.

Vector Databases vs. Traditional Relational Databases

Let's break down the difference between a vector database and a traditional relational database in simpler terms:

• Nature of Data:
  • Relational Database: Traditional relational databases store data in tables with rows and columns. Each row represents a record, and each column represents a specific attribute or property of that record. The data is typically structured and organized in a tabular format.
  • Vector Database: Vector databases store data as vectors, which are mathematical representations in a multidimensional space. Instead of tables, data is organized based on these vectors, capturing relationships and patterns between different entities.
• Data Retrieval:
  • Relational Database: Retrieving data from a relational database often involves SQL queries that specify conditions, joins and aggregations. The focus is on extracting rows and columns that meet certain criteria.
  • Vector Database: Retrieval in a vector database is based on similarity search. Given a vector (e.g., representing an image or a piece of text), the database finds other vectors that are similar, allowing for tasks like finding similar images or documents.
• Use Cases:
  • Relational Database: Traditional relational databases are well suited for structured data with clear relationships between entities. They are commonly used in business applications, financial systems, and scenarios where data is highly organized.
  • Vector Database: Vector databases excel in scenarios where the emphasis is on similarity and pattern recognition. They are used in applications like content-based image retrieval, recommendation systems, and natural language processing, where understanding similarity between entities is crucial.
• Representation:
  • Relational Database: Data in a relational database is represented as tables with rows and columns. Relationships between tables are established using keys.
  • Vector Database: Data in a vector database is represented as vectors, which are numerical representations in a multidimensional space. Similarity between vectors is a key aspect of data representation.
• Flexibility:
  • Relational Database: Traditional databases provide a structured and rigid way to represent data with predefined schemas. Changes to the schema can be challenging.
  • Vector Database: Vector databases offer more flexibility, especially in handling unstructured or semistructured data. They can adapt to varying data representations and are well suited for scenarios where the data may not fit neatly into tables.

In essence, the main difference lies in how data is organized and retrieved. Traditional relational databases focus on structured data and SQL queries, while vector databases are designed for similarity searches and pattern recognition using vector representations. Vector databases are particularly powerful in scenarios where understanding the similarity between different entities is essential, such as in machine learning applications and content-based retrieval systems.

Vector Databases and LLMs

Here's how vector databases are used by large language models:

1. Vector Representations: Large language models, like those based on transformer architectures, represent words, sentences or documents as vectors in a high-dimensional space. These vectors capture semantic relationships and contextual information.
2. Embeddings: Words or phrases are mapped to numerical embeddings, which are dense vector representations. These embeddings are often learned during the pre-training phase of the model, where the model is exposed to vast amounts of text data to understand language patterns.
3. Storage in Vector Database: The learned vector representations can be stored in a vector database. Each vector corresponds to a specific word or phrase, and the database is optimized for efficient storage and retrieval of these vectors.
4. Similarity Search: Vector databases excel at similarity search. Given a query vector, the database can quickly retrieve vectors that are most similar to the query vector. In the context of language models, this means finding words or phrases that are semantically similar to a given input.
5. Semantic Retrieval: Vector databases facilitate semantic retrieval. They can be used to retrieve documents or passages with similar semantic meaning to a given query, based on the vector representations of the text.
6. Efficient Search Operations: Vector databases are designed for efficient search operations in high-dimensional spaces. They often employ techniques such as indexing and nearest-neighbor search algorithms to optimize the retrieval of vectors.
7. Applications in Language Models: Large language models leverage vector databases for tasks like information retrieval, semantic search and contextual similarity analysis. For example, a language model might use a vector database to find the most contextually similar phrases when generating text.
8. Operational Efficiency: Using a vector database allows language models to perform complex semantic operations more efficiently than exhaustive searches through large datasets. This is particularly important for real-time applications.

In summary, vector databases play a crucial role in the operational efficiency of large language models by enabling rapid and accurate retrieval of vector representations. They are particularly valuable in tasks that involve semantic similarity and context-based operations, enhancing the overall capabilities of language models.

If you’d like to see a demo of some cool things we’re doing with AI, click here.