A vector database isn’t just a fancy index; it’s a fundamental shift in how we store and query information, treating meaning and similarity as first-class citizens.
Let’s see this in action. Imagine we have a collection of product descriptions and we want to find similar items.
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient, models
# Load a pre-trained model
model = SentenceTransformer('all-MiniLM-L6-v2')
# Initialize Qdrant client (connecting to a local instance for this example)
client = QdrantClient("localhost", port=6333)
# Define a collection
collection_name = "product_descriptions"
client.recreate_collection(
collection_name=collection_name,
vectors_config=models.VectorParams(size=model.get_sentence_embedding_dimension(), distance=models.Distance.COSINE)
)
# Sample data
products = [
{"id": "1", "description": "A comfortable, ergonomic office chair with lumbar support."},
{"id": "2", "description": "Stylish gaming chair with adjustable armrests and head pillow."},
{"id": "3", "description": "Executive leather chair, perfect for a home office."},
{"id": "4", "description": "Lightweight and portable laptop stand for remote work."},
{"id": "5", "description": "Mechanical keyboard with RGB backlighting and tactile switches."}
]
# Encode and upload data
points_to_upsert = []
for product in products:
embedding = model.encode(product["description"]).tolist()
points_to_upsert.append(
models.PointStruct(
id=product["id"],
vector=embedding,
payload={"description": product["description"]}
)
)
client.upsert(
collection_name=collection_name,
wait=True,
points=points_to_upsert
)
# Query for similar items to "A chair for working from home"
query_text = "A chair for working from home"
query_vector = model.encode(query_text).tolist()
search_result = client.search(
collection_name=collection_name,
query_vector=query_vector,
limit=3 # Get top 3 results
)
for hit in search_result:
print(f"ID: {hit.id}, Score: {hit.score:.4f}, Description: {hit.payload['description']}")
This code demonstrates the core loop: embed text into vectors, store these vectors, and then query by embedding a search term and finding the nearest neighbors. The "magic" is in the vector index (like HNSW or IVF) that Qdrant uses, allowing for approximate nearest neighbor (ANN) search that’s orders of magnitude faster than brute-force comparisons for high-dimensional data.
The problem vector databases solve is information retrieval beyond keyword matching. Traditional databases are great for structured data and exact matches. But what if you want to find documents conceptually similar to a query, or detect duplicate images? You need to represent the meaning or content of the data as numerical vectors (embeddings). Vector databases are optimized for storing and querying these high-dimensional vectors at scale. Internally, they use specialized indexing algorithms to make similarity searches (like cosine similarity, dot product, or Euclidean distance) feasible on millions or billions of vectors. You control the dimensionality of your vectors (tied to your embedding model), the distance metric used for comparison, and the trade-offs in accuracy vs. speed for the ANN index.
Most people understand that vector databases find "similar" things. What they often miss is that the definition of similarity is entirely up to them, encoded in the choice of embedding model and the distance metric. An embedding model trained to understand sentiment will produce vectors where points close together represent similar sentiments, even if the words are different. A model trained on image features will find visually similar images. The vector database just provides the mechanism to find points close in that chosen vector space. You’re not just querying data; you’re querying a high-dimensional semantic space.
The next step in understanding is exploring different ANN index parameters and their impact on recall and latency.