Vector databases are surprisingly good at not storing vectors.
Let’s spin up Qdrant and Weaviate on Kubernetes and see how they handle similarity search. Imagine we have a bunch of product descriptions, and we want to find similar products.
First, we need to get our text into vector form. This is usually done with an embedding model. For this example, we’ll assume we’ve already done that and have pairs of (product_id, vector_embedding).
Now, let’s deploy Qdrant. We’ll use Helm for this:
helm repo add qdrant https://helm.qdrant.tech/
helm repo update
helm install qdrant qdrant/qdrant --namespace qdrant --create-namespace --values values.yaml
Here’s a minimal values.yaml for a single node deployment:
cluster:
enabled: false
persistence:
enabled: true
storageClassName: "standard" # Or your preferred storage class
size: "10Gi"
Once Qdrant is up, we can interact with it via its API. Let’s say we have a collection named products. We can add some data:
import qdrant_client
client = qdrant_client.QdrantClient("localhost", port=6333) # Assuming port-forwarding
client.recreate_collection(
collection_name="products",
vectors_config=qdrant_client.models.VectorParams(size=1536, distance=qdrant_client.models.Distance.COSINE),
)
# Add some dummy data
points = [
qdrant_client.models.PointStruct(
id=1,
vector=[0.1] * 1536, # Replace with actual embeddings
payload={"name": "Laptop", "price": 1200},
),
qdrant_client.models.PointStruct(
id=2,
vector=[0.15] * 1536,
payload={"name": "Gaming Laptop", "price": 1800},
),
qdrant_client.models.PointStruct(
id=3,
vector=[0.8] * 1536,
payload={"name": "Desk Chair", "price": 300},
),
]
client.upsert(collection_name="products", points=points, wait=True)
To find products similar to a query vector (e.g., a new laptop description), we’d run:
query_vector = [0.12] * 1536 # Embedding for a new laptop
search_result = client.search(
collection_name="products",
query_vector=query_vector,
limit=2,
)
print(search_result)
This would return the Laptop and Gaming Laptop points, ordered by their similarity to the query vector.
Now, let’s look at Weaviate. We’ll also use Helm:
helm repo add weaviate https://weaviate.io/helm
helm repo update
helm install weaviate weaviate/weaviate --namespace weaviate --create-namespace --values weaviate-values.yaml
A simple weaviate-values.yaml:
cluster:
enabled: false
persistence:
enabled: true
storageClassName: "standard" # Or your preferred storage class
size: "10Gi"
Weaviate uses a GraphQL-like API and a schema. First, we define the schema:
import weaviate
import os
client = weaviate.Client(
url="http://localhost:8080", # Assuming port-forwarding
)
class_name = "Products"
schema = {
"classes": [
{
"class": class_name,
"description": "Product information",
"vectorizer": "none", # We provide vectors ourselves
"properties": [
{
"name": "name",
"dataType": ["text"],
},
{
"name": "price",
"dataType": ["number"],
},
]
}
]
}
client.schema.create(schema)
When using vectorizer: "none", Weaviate expects you to provide the vectors during data import.
# Add some dummy data
with client.batch as batch:
batch.add_data_object(
data_object={
"name": "Laptop",
"price": 1200,
},
class_name=class_name,
uuid="a1b2c3d4-e5f6-7890-1234-567890abcdef", # Example UUID
vector=[0.1] * 1536 # Replace with actual embeddings
)
batch.add_data_object(
data_object={
"name": "Gaming Laptop",
"price": 1800,
},
class_name=class_name,
uuid="b2c3d4e5-f6a7-8901-2345-67890abcdef1",
vector=[0.15] * 1536
)
batch.add_data_object(
data_object={
"name": "Desk Chair",
"price": 300,
},
class_name=class_name,
uuid="c3d4e5f6-a7b8-9012-3456-7890abcdef12",
vector=[0.8] * 1536
)
To perform a similarity search in Weaviate:
query_vector = [0.12] * 1536
response = (
client.query
.get(class_name, ["name", "price"])
.with_near_vector({
"vector": query_vector
})
.with_limit(2)
.do()
)
print(response)
This query will return the Laptop and Gaming Laptop items.
The core mechanism in both systems for efficient similarity search relies on Approximate Nearest Neighbor (ANN) algorithms. Instead of comparing your query vector to every single vector in the database (which would be O(N) and prohibitively slow for large datasets), ANN algorithms build an index that allows them to find close neighbors much faster, usually with logarithmic or sub-linear complexity. Qdrant and Weaviate abstract away the specifics of these algorithms (like HNSW or IVF), but understanding that they are building and querying specialized index structures is key.
One critical aspect of vector database performance, especially under load, is the trade-off between index build time, memory usage, and search accuracy. When you configure ANN parameters (like ef_construct and M for HNSW in Qdrant, or ef and ef_construction in Weaviate), you’re directly tuning this trade-off. Higher values generally lead to better recall (more accurate results) and slower indexing/searches, while lower values are faster but might miss some true nearest neighbors. The default values are often a good starting point, but for performance-critical applications, experimentation is essential.
With data indexed and queried, the next logical step is to explore hybrid search, combining vector similarity with traditional keyword-based filtering.