Dense Retrieval in vector search

Module 08 · Embeddings & Search · 17 concepts

In depth / 15 lines

Dense Retrieval, in more detail

Dense retrieval searches by meaning rather than by words.

Query and documents are both embedded, and the closest vectors are returned.

Every number in the vector carries some signal, which is what dense refers to.

It finds documents that share no vocabulary with the query at all.

A search for car insurance surfaces a page about vehicle coverage.

Synonyms, paraphrase and loose description are handled naturally.

That makes it far better than keyword search at conversational questions.

It is the retrieval step in essentially every RAG system.

Its blind spot is exactness, since a vector is a summary of meaning.

Part number XR-4471B and XR-4471C mean nearly the same thing to an embedding model.

Rare proper nouns, identifiers, codes and dates all suffer the same way.

It also cannot represent operators such as must contain or not.

Documents must be chunked to fit the model's input limit, and chunking is its own problem.

Retrieval quality is capped by the embedding model, which cannot be prompted out of a weakness.

This is why hybrid search exists, and why keyword search refuses to die.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Dense Retrieval

Two encoders, one vector space

s(q, d) = E_Q(q)^{\top} E_D(d)

$E_Q$ : the query encoder.
$E_D$ : the document encoder.

Queries and documents are embedded separately and compared by a dot product, which is why this shape is called a bi-encoder.

The separation is the entire engineering argument. Documents can be embedded once, offline, and the query is the only thing encoded at search time.

A cross-encoder reading the pair together scores better and cannot be indexed, because it has no vector for a document until it has seen the query.

The training objective

\mathcal{L} = -\log \frac{\exp\big(s(q, d^{+}) / \tau\big)}{\exp\big(s(q, d^{+}) / \tau\big) + \sum_{j=1}^{N} \exp\big(s(q, d_j^{-}) / \tau\big)}

$d^{+}$ : the passage that answers the query.
$d_j^{-}$ : negatives, passages that do not.
$\tau$ : a temperature, typically 0.05.

This is contrastive learning: pull the right pair together and push the wrong ones apart, in one softmax over the candidates.

This is where in-batch negatives come in.

The cheap trick that made it practical is to use the other documents in the same batch as negatives, so a batch of 1,024 gives 1,023 negatives per query at no extra cost.

Larger batches therefore train better retrievers, which is why this task is unusually sensitive to how much memory the training hardware has.

Hard negatives are where the quality is

A random negative is easy. A passage about the wrong football club is separated from the right one long before training is useful.

d^{-} \sim \text{top-}k \text{ from the current model}

Hard negatives are passages the current retriever already ranks highly and which do not answer the query, mined by running the model over the corpus.

Training on them is what teaches the fine distinctions, and it usually takes two or three rounds of mine, retrain, mine again.

The risk is false negatives: a mined passage that does answer the query, labelled as wrong, actively teaches the model the opposite of the truth.

One system, in order

Fine-tune the two encoders on query and passage pairs with in-batch negatives.
Mine hard negatives with the resulting model and train again.
Embed the whole corpus once and build an approximate index.
At query time, embed the query and search the index.
Rerank the top candidates with a cross-encoder if the latency allows.

Where it beats keywords and where it does not

\text{"car"} \approx \text{"automobile"}

Dense retrieval matches meaning, so it finds passages sharing no words with the query, which is exactly what BM25 cannot do.

Try a query like "error XT-4491".

It is weak on exact identifiers, rare product codes and names that were not in its training distribution, where a keyword index is perfect and effortless.

It also does not transfer. A retriever trained on web questions performs poorly on legal or medical corpora, while BM25 has no domain to be wrong about.

That complementarity is why hybrid search exists, and why very few serious systems run dense retrieval alone.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does dense retrieval tend to miss?

Embeddings & Search

17 concepts in this module, then a graded test with every answer explained.

Study this module →