LLM Embeddings and Vector Search

Real-World Scenario

A bike shop’s help centre has hundreds of articles. A customer types “my tyre keeps losing air”. The article that helps, about tubeless tyres whose sealant plugs punctures, never uses the words “losing” or “air”. Keyword search can’t connect the two. Embeddings can, because they compare what texts mean instead of which words they share. They are the retrieval layer under most RAG systems, semantic search boxes and “similar items” features.

This lesson explains how embeddings work and how to use them well. The examples use only NumPy and scikit-learn, so they run anywhere without downloading a model. The retrieval and reranking lesson runs a real neural embedding model (all-MiniLM-L6-v2 via sentence-transformers) end to end.

What an Embedding Is

An embedding model turns a piece of text into a fixed-length vector of floating-point numbers: 384 numbers for a small model like all-MiniLM-L6-v2, and commonly 768 to 3,072 for larger models, including hosted ones. Every input gets the same length, whether it is a three-word query or a paragraph, which is what makes any two texts comparable.

The individual numbers mean nothing on their own. There is no “tyre” dimension. What matters is direction: the model is trained so that texts with related meaning produce vectors pointing in similar directions. “Restart the router” and “reboot the modem” end up close together, and “refund my order” ends up somewhere else, even though none of the three share a content word.

Three properties follow from that, and the rest of this lesson builds on them:

• Similarity is geometry. Comparing meaning becomes comparing vectors, which is fast arithmetic.
• Vectors belong to one model. Each model defines its own space. A vector from one model can’t be compared with a vector from another, even at the same length.
• Embed once, compare many times. Documents are embedded once at indexing time and stored. Only the query is embedded per search.

Sparse Keyword Vectors, and Where They Fail

Before neural embeddings, text was vectorized with one dimension per vocabulary word, weighted by TF-IDF (how often the word appears in this text, discounted by how common it is overall). These vectors are sparse: almost every entry is zero.

from sklearn.feature_extraction.text import TfidfVectorizer

articles = {
    "Fixing a flat tyre":     "Lever the tyre off the rim, find the puncture in the inner tube and patch it.",
    "Tubeless setup":         "Tubeless tyres use sealant, so small punctures seal themselves while you ride.",
    "Hydraulic brake bleed":  "A spongy brake lever means air in the hydraulic line; bleed it with fresh fluid.",
    "Choosing chain lube":    "Wet lube suits rain; dry lube keeps the chain clean in dusty summer air.",
    "Refund timing":          "Refunds reach your card five to ten days after we receive the item.",
}
tfidf = TfidfVectorizer(stop_words="english")
vectors = tfidf.fit_transform(articles.values())

if __name__ == "__main__":
    print(f"{vectors.shape[1]} dimensions, {vectors.nnz / vectors.shape[0]:.1f} non-zero per article")
    query = tfidf.transform(["my tyre keeps losing air"])
    scores = (vectors @ query.T).toarray().ravel()
    for title, score in sorted(zip(articles, scores), key=lambda p: -p[1]):
        print(f"{score:.2f}  {title}")

$ python sparse_miss.py
40 dimensions, 8.4 non-zero per article
0.27  Choosing chain lube
0.24  Fixing a flat tyre
0.13  Hydraulic brake bleed
0.00  Tubeless setup
0.00  Refund timing

Five short articles already need 40 dimensions, one per distinct word, and each article uses only about 8 of them. A real help centre has tens of thousands of dimensions, almost all zero.

The ranking is the real problem. The top result is about chain lube, because it mentions dust in “summer air”. The brake-bleed article comes third for “air in the hydraulic line”. The tubeless article, the one that answers the question, scores exactly 0.00, because it shares no word with the query. A sparse vector can only match words that are literally shared: it has no way to know that “losing air” and “punctures seal themselves” describe the same problem.

LSA (latent semantic analysis) compresses TF-IDF vectors into a few hundred dense dimensions using word co-occurrence, and it is sometimes presented as a fix. It helps only when your own corpus shows the related words appearing together. In a help centre where no article ever puts “air” and “puncture” in the same text, LSA has no evidence that they are related, so it can’t bridge the gap either. That knowledge has to come from far more text than one site has, which is what neural embedding models are trained on.

How Neural Embedding Models Learn Meaning

A text embedding model is a transformer encoder, usually much smaller than a chat model, trained for one job: placing related texts near each other.

1. Tokenize and encode. The text is split into tokens (see the tokenization lesson), and the encoder produces one vector per token, each informed by the tokens around it.
2. Pool into one vector. The per-token vectors are combined into a single vector for the whole text, commonly by averaging them (mean pooling) or by taking one special token’s vector. Many models then normalize it to length 1.
3. Contrastive training. The model is trained on large numbers of pairs that belong together: a question and its answer, a title and its article, two paraphrases. The loss pulls each pair’s vectors together and pushes them away from the other texts in the batch. After millions of pairs, “losing air” lands near “puncture” because such texts kept appearing as matching pairs.

Some practical consequences:

• Inputs have a maximum length. Text beyond the model’s token limit is truncated, usually silently. all-MiniLM-L6-v2, for example, cuts input off at 256 tokens. This is one reason documents are chunked before they are embedded.
• Queries and documents may need different prefixes. Some models are trained asymmetrically and expect markers such as query: and passage: in front of the text. Leaving them out quietly lowers retrieval quality, so check the model card.
• Domain matters. A model trained mostly on web text may place specialized terms (legal, medical, internal product names) poorly. Test it on your own queries before choosing it.

Measuring Similarity: Cosine, Dot Product, Euclidean

Three measures are in common use. Small hand-made vectors show how they differ.

import numpy as np

a = np.array([0.8, 0.6, 0.0])        # "reset the router"
b = np.array([0.6, 0.8, 0.0])        # "restart the router": similar direction
c = np.array([0.0, 0.3, 0.95])       # "refund my order": different direction
long_a = a * 3                       # same direction as a, three times the length

def cosine(x, y):
    return x @ y / (np.linalg.norm(x) * np.linalg.norm(y))

if __name__ == "__main__":
    for name, y in [("b", b), ("c", c), ("long_a", long_a)]:
        print(f"a vs {name:6}  cosine {cosine(a, y):5.2f}  dot {a @ y:5.2f}"
              f"  euclidean {np.linalg.norm(a - y):5.2f}")

    unit = lambda v: v / np.linalg.norm(v)
    print("after normalizing, dot == cosine:", np.isclose(unit(a) @ unit(b), cosine(a, b)))

$ python similarity.py
a vs b       cosine  0.96  dot  0.96  euclidean  0.28
a vs c       cosine  0.18  dot  0.18  euclidean  1.28
a vs long_a  cosine  1.00  dot  3.00  euclidean  2.00
after normalizing, dot == cosine: True

• Cosine similarity measures only the angle. a and b point almost the same way (0.96), and c points elsewhere (0.18). long_a scores a perfect 1.00, because it has the same direction as a, just three times longer.
• Dot product mixes angle and length. It agrees with cosine for b and c, because a and b have length 1 and c nearly so, but gives long_a 3.00. If vector length tracks something like text length or popularity, an unnormalized dot product rewards that instead of relevance.
• Euclidean distance measures the straight-line gap. It calls long_a further away (2.00) than the unrelated c (1.28), even though long_a has exactly the same direction as a.

The last line is the practical rule. Normalize every vector to length 1 once, when you store it. Then the dot product equals cosine similarity and is the cheapest thing to compute, and Euclidean distance gives the same ranking (practice task 1 shows why). Use the measure the model was trained with; for most text embedding models that is cosine.

Dimensions, Storage and Search Cost

Every stored vector costs memory, and a brute-force search compares the query with every one of them. Here that is measured for 100,000 vectors at three common sizes. Random vectors stand in for real embeddings, since the cost doesn’t depend on the values.

import time
import numpy as np

rng = np.random.default_rng(0)
for dims in (384, 768, 1536):
    docs = rng.standard_normal((100_000, dims), dtype=np.float32)
    docs /= np.linalg.norm(docs, axis=1, keepdims=True)       # normalize once, at index time
    query = docs[123] + 0.1 * rng.standard_normal(dims, dtype=np.float32)
    runs = []
    for _ in range(5):                                          # best of 5, to skip warm-up noise
        start = time.perf_counter()
        scores = docs @ query                                   # cosine, since docs are unit length
        top = np.argpartition(-scores, 5)[:5]
        top = top[np.argsort(-scores[top])]
        runs.append((time.perf_counter() - start) * 1000)
    ms = min(runs)
    print(f"{dims:5} dims  {docs.nbytes / 1e6:4.0f} MB  search {ms:5.1f} ms  best match {top[0]}")

$ python cost.py
  384 dims   154 MB  search   5.3 ms  best match 123
  768 dims   307 MB  search  11.4 ms  best match 123
 1536 dims   614 MB  search  21.5 ms  best match 123

Memory is exact arithmetic: 100,000 × dimensions × 4 bytes. 1,536 dimensions cost four times what 384 do, and at 10 million documents that is 61 GB instead of 15 GB. Search time grows roughly in step with the dimension count; timings are from one CPU and will vary on yours. At this size a single matrix multiplication searches everything in milliseconds, and the planted neighbour (document 123) is found at every size.

This leads to the usual scaling path:

• Up to about a million vectors, brute force with NumPy, or a database extension such as pgvector, is usually fast enough and exact.
• Beyond that, vector databases use approximate nearest neighbour (ANN) indexes such as HNSW (a navigable graph of neighbours) or IVF (clusters searched selectively). They check a small fraction of vectors, trading a little recall for large speedups. Measure recall on your own data when you tune them.
• To shrink storage, store vectors at lower precision (float16, int8 or even binary), or use models that support shortened vectors, where the first N dimensions still work as a smaller embedding. Always re-check retrieval quality after compressing.

Choosing an Embedding Model

How to decide:

1. Start from your constraints. Language coverage, maximum input length, whether data may leave your servers, and budget rule options out before quality does.
2. Shortlist from a benchmark. The MTEB leaderboard ranks models across retrieval, clustering and classification tasks, and is a reasonable place to find candidates.
3. Decide on your own data. Take 30 to 100 real queries with known correct documents, and compare candidates with recall@k and MRR, as in the evaluation lesson. Benchmark rankings often reorder on domain-specific text.
4. Plan for change. Record which model and version produced each stored vector. Switching models means re-embedding the whole collection, because old and new vectors can’t be mixed.

Where Embeddings Are Used

• Semantic search. Embed the query and return the nearest documents. It is usually combined with keyword search (BM25) in hybrid search, because embeddings can miss exact identifiers such as error codes and part numbers, which keyword search matches perfectly.
• RAG retrieval. The same search, feeding the top chunks into an LLM prompt. Retrieval quality caps answer quality, so this is where most RAG failures start; see RAG failure diagnosis.
• Reranking. A fast embedding search finds candidates, and a slower cross-encoder rescores the top ones by reading query and document together. See the reranking lesson.
• Clustering and topic discovery. Run k-means or HDBSCAN over ticket or feedback embeddings to find recurring themes without labelling anything first.
• Near-duplicate detection. Pairs of texts above a similarity threshold are candidate duplicates, which is useful for cleaning fine-tuning data and search indexes. Tune the threshold on labelled pairs, because a good value differs between models.
• Classification with few labels. Train a simple classifier, such as logistic regression, on embeddings of a few hundred labelled examples. It is often competitive with fine-tuning and far cheaper.
• Recommendations. “More like this” is a nearest-neighbour search starting from an item’s own embedding, excluding the item itself.

Common Pitfalls

• Mixing models or versions in one index. The scores look normal and mean nothing.
• Forgetting to normalize before using the dot product, so longer texts win.
• Embedding whole documents past the model’s input limit. The tail is silently truncated and never becomes searchable.
• Trusting raw similarity scores as confidence. A 0.62 can be an excellent match with one model and noise with another. Set thresholds per model, from labelled examples.
• Skipping keyword search. Exact strings such as E-204 or VX2-7A91 need lexical matching, so use hybrid search.

Practice

1. For vectors normalized to length 1, squared Euclidean distance equals 2 - 2 × cosine. Check that with the vectors from similarity.py, and explain why it means Euclidean distance and cosine rank results the same way once vectors are normalized.

   Solution

   import numpy as np
   from similarity import a, b, c, long_a, cosine
   
   unit = lambda v: v / np.linalg.norm(v)
   for name, y in [("b", b), ("c", c), ("long_a", long_a)]:
       d = np.linalg.norm(unit(a) - unit(y))
       print(f"a vs {name:6}  euclidean {d:.2f}  2 - 2*cosine {2 - 2 * cosine(a, y):.2f}  d^2 {d * d:.2f}")

   $ python sol_distance.py
   a vs b       euclidean 0.28  2 - 2*cosine 0.08  d^2 0.08
   a vs c       euclidean 1.28  2 - 2*cosine 1.64  d^2 1.64
   a vs long_a  euclidean 0.00  2 - 2*cosine 0.00  d^2 0.00

   The squared distance matches 2 - 2 × cosine in every row, so distance is a decreasing function of cosine: the closest vector is always the one with the highest cosine. long_a now sits at distance 0.00 from a, because normalizing removed the length difference that made plain Euclidean distance call it far away.

2. In sparse_miss.py, add the word “puncture” to the query, as a query-expansion step might. What happens to the ranking, and why does the tubeless article still score 0?

   Solution

   from sparse_miss import articles, tfidf, vectors
   
   for q in ["my tyre keeps losing air", "my tyre keeps losing air puncture"]:
       scores = (vectors @ tfidf.transform([q]).T).toarray().ravel()
       ranked = sorted(zip(articles, scores), key=lambda p: -p[1])
       print(q, "->", [f"{t} {s:.2f}" for t, s in ranked[:3]])
   print("'puncture' and 'punctures' are separate dimensions:",
         "puncture" in tfidf.vocabulary_, "punctures" in tfidf.vocabulary_)

   $ python sol_expand.py
   my tyre keeps losing air -> ['Choosing chain lube 0.27', 'Fixing a flat tyre 0.24', 'Hydraulic brake bleed 0.13']
   my tyre keeps losing air puncture -> ['Fixing a flat tyre 0.41', 'Choosing chain lube 0.23', 'Hydraulic brake bleed 0.11']
   'puncture' and 'punctures' are separate dimensions: True True

   “Fixing a flat tyre” now ranks first, because it contains “puncture” exactly. The tubeless article still scores 0, because it says “punctures”, which is a different dimension to TF-IDF. Keyword matching needs exact tokens, so every synonym and word form has to be handled by hand, with stemming or synonym lists. That is the work neural embeddings do for you.

3. Storing vectors as float16 halves memory. Using the 384-dimension setup from cost.py, check whether it changes the top 10 results.

   Solution

   import numpy as np
   
   rng = np.random.default_rng(0)
   docs = rng.standard_normal((100_000, 384), dtype=np.float32)
   docs /= np.linalg.norm(docs, axis=1, keepdims=True)
   query = docs[123] + 0.1 * rng.standard_normal(384, dtype=np.float32)
   
   half = docs.astype(np.float16)
   top32 = np.argsort(-(docs @ query))[:10]
   top16 = np.argsort(-(half @ query.astype(np.float16)).astype(np.float32))[:10]
   print(f"float32 {docs.nbytes / 1e6:.0f} MB, float16 {half.nbytes / 1e6:.0f} MB")
   print("same top 10:", list(top32) == list(top16), " overlap:", len(set(top32) & set(top16)))

   $ python sol_float16.py
   float32 154 MB, float16 77 MB
   same top 10: True  overlap: 10

   Memory drops from 154 MB to 77 MB, and both the set and the order of the top 10 survive the rounding. That’s not guaranteed — float16 has roughly 3 decimal digits of precision, so two documents scored within about 0.001 of each other could swap places after quantization — it just didn’t happen for this query’s margins. For retrieval that feeds a reranker or an LLM, a swap between two near-tied results rarely matters. For anything that depends on exact order or on a similarity threshold, measure the effect on your own data before compressing.