Reranking is a second, more expensive scoring pass over the initial results.
First-stage retrieval fetches perhaps 50 candidates cheaply.
A cross-encoder then reads the query and each document together.
That joint reading is far more accurate than comparing two separate vectors.
You keep the top 5 after reranking, which sharply improves final quality.
Reranking is a second scoring pass over the results the first stage returned.
Retrieval is built for speed across millions of documents.
Reranking is built for accuracy across fifty.
Fetch broadly and cheaply, then judge carefully and expensively.
The first stage compares two vectors that were computed separately.
The query never influenced the document vector, since it did not exist yet.
A cross-encoder instead reads the query and the document together in one pass.
Every word of the query can attend to every word of the document.
That joint reading is far more accurate at judging genuine relevance.
It is also far too slow to run against a whole corpus.
So it runs on the fifty candidates and returns the best five.
The gain is usually large, often the single biggest quality win available to a RAG system.
Cohere Rerank, BGE and Jina are common hosted or open options.
The cost is latency, typically a few hundred milliseconds.
Retrieve more candidates than you think you need, since the reranker can only sort what it is given.
Retrieval has to score millions of documents, so it uses a model cheap enough to precompute. Reranking scores a hundred, so it can use one that is far better.
The query and the document are concatenated and read together by a transformer, so every query token can attend to every document token.
A bi-encoder compresses the document into a vector before it knows the query, which is what makes it indexable and what limits how well it can match.
The cross-encoder never compresses. That is the whole source of its accuracy and the whole reason it cannot be an index.
Reranking 100 candidates costs 100 passes over a query-document pair, which is tens of milliseconds on a GPU and hundreds on a CPU.
Applied to the whole corpus that is impossible, which is why the two stages exist rather than one good model.
Step five matters more than it looks. A reranker produces a calibrated relevance score, so it can say that nothing retrieved is relevant, which a rank alone cannot.
The reranker can only reorder what it is given, so the retrieval stage must have already found the answer somewhere in its .
Below 50 the recall ceiling usually binds. Above 200 the extra latency buys almost nothing, since the retriever's ranking below that depth is mostly noise.
The right way to set it is to measure retrieval recall at several depths and pick the point where the curve flattens.
Late interaction models such as ColBERT keep one vector per token and compare them at query time, which lands between the two families in both cost and quality.
Using a general language model as the reranker works and costs an order of magnitude more per candidate, which limits it to small or to offline evaluation.
A dedicated cross-encoder of a few hundred million parameters beats a much larger general model at this specific task, and it is the standard choice for that reason.
One question on this card. Every option explains itself once you pick, right or wrong.
17 concepts in this module, then a graded test with every answer explained.