Vector Databases: The Missing Piece of the AI Stack

01 Why Vector Databases Matter

RAG (retrieval-augmented generation) requires storing document embeddings — high-dimensional vectors that represent the semantic meaning of text. A vector database stores these vectors and can find the most similar vectors to a query in milliseconds. This is the foundation of every RAG system. Without a vector database, the LLM has no memory beyond its context window. With one, it can search millions of documents and ground its responses in verified information.

02 The Benchmark

We tested 6 vector databases with 10 million 1536-dimensional vectors (the dimensionality of OpenAI embeddings). Milvus led at 15,000 queries/second. Qdrant was close at 12,000. Pinecone (managed service) hit 8,500. Chroma (designed for small datasets) managed 1,200. pgvector (PostgreSQL extension) managed 380 — 40x slower than Milvus. The choice depends on scale: Chroma for prototypes (<100K vectors), Pinecone for managed production (1M-100M), Milvus for large-scale self-hosted (100M+).

03 The pgvector Surprise

pgvector is the slowest in raw performance but the most popular in practice. Why? Because most applications already have PostgreSQL. Adding vector search to an existing database means no new infrastructure, no new ops, and ACID transactions across relational and vector data. For applications with under 1 million vectors and moderate query volume (<100 QPS), pgvector is sufficient and dramatically simpler to operate. The 40x speed difference only matters at scale, and most applications don't need that scale.