Embeddings & Vector Search

Why keyword search keeps failing you

You build a search box over your docs. A user types "how do I reset my password", and gets nothing, because your help article is titled "Account recovery steps." Same meaning, zero matching words. Keyword search only finds documents that share *literal tokens*; it has no idea that "reset password" and "account recovery" are the same idea.

Embeddings fix this. They turn text into numbers that encode meaning, so "reset my password" lands right next to "account recovery" even though they share no words. Vector search is how you find those nearby meanings fast, and it's the retrieval engine underneath every RAG system.

What an embedding actually is

An embedding is a list of numbers (a vector) that represents a piece of text as a point in a high-dimensional space, positioned so that texts with similar meaning sit close together.

That's it. You hand text to an embedding model and it returns a fixed-length array of floats, often 384, 768, or 1,536 of them. Each number is a coordinate. "The cat sat on the mat" might become [0.021, -0.118, 0.337, ...]. The individual numbers mean nothing to you; what matters is the position they describe and how close it is to other positions.

The model learned these coordinates by reading enormous amounts of text. It discovered that words and sentences used in similar contexts should land in similar places. The result is a space where geometric distance is a proxy for semantic difference.

On a real map, Paris and Lyon are close, and both are far from Tokyo. In embedding space, "reset password" and "account recovery" are close, and both are far from "export invoice as PDF." Search becomes geometry: to find relevant text, you find the nearest points.

The full pipeline, end to end

1. 1

   Embed your documents

   Run every chunk of your corpus through the embedding model once, offline. You get one vector per chunk.

2. 2

   Store the vectors in an index

   Load them into a vector database (Pinecone, Qdrant, pgvector, Weaviate). The DB builds an index, usually HNSW, to make search fast.

3. 3

   Embed the user's query

   At query time, send the user's question through the *same* model to get a query vector in the same space.

4. 4

   Find nearest neighbors

   The index returns the top-k vectors closest to the query vector, ranked by a similarity metric like cosine.

5. 5

   Use the results

   Return them as search hits, or stuff them into an LLM prompt as context, that second path is RAG.

Similarity metrics: how "close" is measured

"Nearest" needs a definition of distance. Three metrics dominate, and they answer subtly different questions. The wrong choice quietly degrades your results.

For text, cosine is almost always right. Two sentences about the same topic point in the same direction even if one is a paragraph and the other a phrase, cosine cares about *direction*, not *length*. A handy fact: if you normalize your vectors to unit length first, cosine similarity and dot product become identical, and dot product is cheaper. Many vector DBs do this for you, which is why their docs say "use dot product for cosine results."

Cosine similarity in code

Here's the whole idea in one runnable script: embed two texts, then measure how aligned their vectors are. Swap the model for any embedding API or local model, the math is identical.

import numpy as np
from openai import OpenAI

client = OpenAI()

def embed(text: str) -> np.ndarray:
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return np.array(resp.data[0].embedding)

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    # dot product divided by the product of magnitudes
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

v1 = embed("how do I reset my password")
v2 = embed("steps to recover my account")
v3 = embed("export an invoice as a PDF")

print(round(cosine_similarity(v1, v2), 3))  # ~0.82  -> very similar
print(round(cosine_similarity(v1, v3), 3))  # ~0.21  -> unrelated

Two phrasings of the same intent score high (~0.82); an unrelated task scores low (~0.21), with zero shared keywords. That gap is the entire value of embeddings. Vector search is just doing this comparison against millions of stored vectors and returning the highest scorers.

Exact vs approximate search: the recall/speed trade-off

The naive way to find nearest neighbors is exact search: compute the similarity between the query and *every* stored vector, then sort. This is perfectly accurate, and perfectly unscalable. Ten thousand vectors? Fine. Ten million? Every single query now does ten million comparisons. Latency dies.

Approximate Nearest Neighbor (ANN) search trades a sliver of accuracy for orders-of-magnitude speed. Instead of scanning everything, it builds a smart index that lets a query jump to the right neighborhood and only compare against a small candidate set. Two families dominate:

• HNSW (Hierarchical Navigable Small World), builds a layered graph where each vector links to its neighbors. A query enters at the top layer, greedily hops toward closer nodes, and descends. Excellent recall and very fast queries; the trade-off is higher memory use. The default in most vector DBs.
• IVF (Inverted File Index), clusters vectors into buckets (via k-means), then at query time only searches the few buckets nearest the query. Lower memory, great for huge datasets; recall depends on how many buckets you probe (the nprobe knob).

The key word is *approximate*: ANN might miss the true #3 result and return the true #4 instead. We measure this with recall, what fraction of the true top-k the index actually returned. Tuning ANN is choosing a point on a curve: push parameters (HNSW's ef_search, IVF's nprobe) up for higher recall and slower queries, or down for blazing speed and a few missed neighbors. For most apps, 95-99% recall at a fraction of the latency is a trade you'll happily take.

Common mistakes that cost hours

1. Mismatched embedding models. Embedding documents with one model and queries with another puts them in different coordinate systems. Results look random. Pin one model for both sides and re-embed everything if you ever switch.
2. Wrong distance metric. Using Euclidean on vectors built for cosine (or vice versa) silently ranks the wrong things first. Check your model card, it tells you which metric it was trained for, and configure your index to match.
3. No normalization. If you use dot product expecting cosine behavior but never normalize to unit length, longer documents get unfairly boosted by their larger magnitude. Normalize, or use a metric that ignores magnitude.
4. Embedding raw, oversized chunks. Cramming a 4,000-word page into one vector blurs many ideas into one average point. Chunk to coherent passages (see RAG architecture) so each vector represents a single idea.
5. Chasing 100% recall. Demanding perfect ANN recall throws away the entire speed advantage. Set a recall target (say 98%), tune to it, and move on.

Takeaways

Where to go next

Embeddings and vector search are the *retrieval* half of modern AI apps. The natural next step is wiring them into a generation loop, that's exactly what RAG does. And if the geometry here felt magical, understanding how the models that produce these vectors are trained will demystify the rest.

• RAG Architecture Explained, how retrieved chunks become grounded LLM answers.
• How LLMs Actually Work, the model machinery behind both embeddings and generation.
• AI Engineer career path, the full track from foundations to production AI systems.