Reranking in RAG Pipelines: When Cross-Encoders Pay Off

Reranking in a RAG pipeline is a second scoring stage: a fast retriever pulls a wide pool of candidate chunks from the index, then a stronger relevance model rescores each candidate against the query and keeps the best handful. It pays off when the correct document reliably appears somewhere in your top 50 results but lands outside your top 5, which describes most production retrieval over corpora past a few thousand documents. The price is added latency on every query, typically between 30 and 500 milliseconds depending on the model and where it runs.

That trade, precision at the top of the list in exchange for milliseconds on the hot path, is the whole subject. Rerankers are among the cheapest quality upgrades available to a RAG system when the conditions are right, and pure overhead when they are wrong. The sections below cover the architecture, the main options as of mid-2026, how to size the candidate pool, how to prove the second stage is earning its latency, and the cases where leaving it out is correct.

How does the retrieve-then-rerank pattern work?

A query hits the index once and gets back k candidates, where k is deliberately generous: 50, 100, sometimes 200. The first stage is optimized for recall. Its only job is getting the correct chunk anywhere into the pool at a cost low enough to run against millions of vectors. The second stage reorders that pool with a model far too expensive to run against the whole corpus but easily affordable against 100 candidates. The top n survivors, usually 3 to 10, go into the generator's context.

Web search has run this architecture for two decades: a cheap first pass over an inverted index, followed by learning-to-rank models over a bounded candidate set. RAG teams keep rediscovering it because the underlying economics have not changed. Scoring functions cheap enough for corpus scale are too crude to order the top of the list well, and scoring functions accurate enough to order the list well are too slow for corpus scale. Splitting the work lets each stage do the thing it is good at.

One honest constraint before the details. A reranker reorders the candidates it receives. If the correct document is missing from the top-k pool, the reranker cannot promote it. If the chunking pass split an answer across two fragments, the reranker scores each fragment as the partial evidence it is. The upstream failure modes that kill RAG systems in production, drifted schemas, duplicate-polluted indexes, badly cut chunks, mismatched embeddings, all pass straight through a reranker untouched. Fix recall first. Rerank second.

Why do cross-encoders score query-document pairs jointly?

The retriever in most RAG stacks is a bi-encoder. It embeds the query into one vector and every document into its own vector, computed once at index time, and scores relevance as the similarity between the two. That precomputation is what makes vector search fast, and it carries a structural blind spot: the document embedding was produced without any knowledge of the query. Everything the model knew about the document had to be compressed into a single fixed-length vector before anyone asked a question.

A cross-encoder removes the compression. Query and document are concatenated into one input and pass through the model together, so attention layers can connect specific query tokens to specific document tokens. That joint pass lets the model resolve cases similarity search fumbles: whether "revenue before 2019" matches a chunk about 2022 revenue, whether a negation flips relevance, which of two entities a dollar figure belongs to. The sentence-transformers library ships this as the CrossEncoder class, and its ms-marco MiniLM checkpoints, trained on the MS MARCO passage-ranking dataset, remain the standard starting point.

The cost is symmetric with the benefit. Nothing can be precomputed, so every query pays a full forward pass per candidate. That is why cross-encoders live in stage two.

What does reranking actually cost?

The tax has three components, and it lands on the hot path of every request, unlike embedding cost, which is paid once at index time.

Latency comes first. A small cross-encoder such as ms-marco-MiniLM-L-6-v2 scores 100 candidates in tens of milliseconds on a modest GPU and in several hundred milliseconds on CPU. Larger open models like bge-reranker-v2-m3 (roughly 568 million parameters) run slower per pair but stay in the low hundreds of milliseconds for a 100-candidate pool on reasonable hardware. Hosted APIs add a network round trip on top of inference; a Cohere or Voyage call over 100 documents commonly lands between 100 and 400 milliseconds end to end. Document length matters as much as candidate count, since each pair is a full sequence through the model.

Money comes second. Self-hosting means a GPU provisioned for peak query load rather than average. Hosted rerankers price differently from embedding APIs: Cohere charges per search (one query against a batch of documents), Voyage prices by token, and both scale linearly with how many candidates you send. A pipeline reranking 150 candidates per query at production traffic can find the reranker line item rivaling the generation line item.

Operational surface comes third and gets discussed least. A reranker is one more model to version, one more dependency that can drift, one more thing to ablate when quality regresses. It sits between retrieval and generation, so when it degrades, the symptom reads as "worse answers" with no obvious culprit.

Which reranker options are on the menu in 2026?

Four families cover the practical choices.

Open-weight cross-encoders. The sentence-transformers ms-marco checkpoints are small, fast, English-centric, and thoroughly road-tested. BAAI's bge-reranker family, with bge-reranker-v2-m3 as the current workhorse, adds multilingual coverage and stronger accuracy at a larger footprint. mixedbread's mxbai-rerank-large-v2 ships under Apache 2.0 and benchmarks competitively with hosted offerings on public retrieval suites like BEIR. All of these run wherever you can stand up inference, keep document text inside your network, and cost whatever your infrastructure costs.

Hosted rerank APIs. Cohere Rerank 4.0 (rerank-v4.0-pro, with a faster rerank-v4.0-fast variant and a 32k context window) and Voyage's rerank-2.5 (with a lighter rerank-2.5-lite variant) are single API calls: send a query and a list of documents, get back relevance scores. They handle multilingual input and long documents, and they scale without any serving work on your side. The trade is per-query fees, a network hop, and shipping corpus text to a third party, which some data-governance postures rule out on the spot.

LLM-as-reranker. Prompting a general-purpose model to order a candidate list, the pattern the RankGPT paper formalized as listwise reranking, produces strong relevance judgments, particularly on queries where relevance requires actual reasoning. It is also the slowest and most expensive option by an order of magnitude, and the output ordering can wobble between runs. It earns its keep in offline evaluation, in generating training labels for smaller rerankers, and in low-volume pipelines where seconds of added latency are acceptable.

Late interaction. ColBERT gets its own section below, since it occupies a different point on the architecture curve.

Treat the latency column as an order-of-magnitude guide. Measured numbers move with document length, hardware, and batching, so benchmark on your own corpus before committing.

Where does ColBERT fit?

ColBERT keeps a vector per token rather than per document. At index time, every document is encoded into token-level embeddings and stored. At query time, the query is encoded the same way and relevance is computed by MaxSim: each query token finds its best-matching document token, and those maxima sum into the score. The result is token-level matching granularity, most of what a cross-encoder buys you, at query latency close to a bi-encoder, because the document side was still encoded offline.

ColBERTv2 and its PLAID indexing pipeline compressed the storage overhead substantially, but the overhead remains real: per-token vectors multiply index size several-fold versus single-vector search, and the engines with native multivector support (Vespa, Qdrant, and LanceDB among them) form a shorter list than the engines supporting plain vector search. If you are in the middle of choosing a vector store, multivector support belongs on the evaluation sheet. Late interaction fits best at high query volume, where per-query cross-encoder inference is unaffordable but bi-encoder precision is measurably too low.

How large should the candidate pool be?

Two numbers to set: retrieve-k, the candidates pulled from stage one, and top-n, the survivors passed into context.

Set k from your recall curve. Compute recall@k on a held-out evaluation set of (query, relevant-document) pairs at k of 10, 25, 50, 100, and 200. The curve typically climbs steeply and then flattens; k belongs at the knee. If recall@50 is 0.92 and recall@200 is 0.93, retrieving 200 buys one point of ceiling for four times the reranking cost. If the curve is still climbing at 200, the retriever has a recall problem no reranker will fix, and the work moves upstream.

Set n from the context budget and the generator's behavior. Chunks past the first handful add tokens, cost, and position-related degradation; the Lost in the Middle results (Liu et al., 2023) showed language models use long context unevenly, favoring the start and end. In practice, an n of 3 to 10 covers most pipelines. Push it higher when answers require synthesizing several documents. Pull it lower when precision@3 is already strong.

How do you prove the reranker helps?

Three layers of evidence, cheapest first, all running against the same held-out evaluation set.

Recall@n at the final cutoff. Compare the fraction of queries where the relevant document survives into the top n with the reranker on, against the raw retriever top n with it off. This is the reranker's core claim, and it is deterministic to score.

nDCG@10 for ordering quality. Recall treats position 1 and position 9 identically; nDCG rewards putting the best document first, which matters when the generator weights early context more heavily.

Answer correctness end to end. Retrieval metrics are proxies. Run a graded answer-quality evaluation, human or model-graded, on the same queries with the reranker toggled on and off. If nDCG improves and answer quality does not move, the generator was already finding the document at position 7, and the added latency has no downstream payoff.

Record p95 latency next to every quality number. A reranker that lifts answer correctness by four points while adding 350 milliseconds at p95 is a product decision, and different products will decide it differently.

When should you skip reranking?

Small corpora. Under a few thousand chunks, a decent retriever's top 5 and top 50 contain nearly the same documents. There is nothing to promote.

Already-precise retrieval. If recall@5 sits within a point or two of recall@50 on your evaluation set, the ordering is already right. This happens on corpora with distinctive vocabulary, where lexical and semantic signals agree.

Hard latency budgets. Interactive surfaces with sub-second total budgets (autocomplete, voice, live agent assist) often cannot spend 200 milliseconds on rescoring. Late interaction or a stronger first-stage embedding model is the better spend.

Upstream problems. Degraded retrieval after an embedding change points at the embedding model migration, and poor chunk quality points at ingestion. A reranker layered over either failure adds latency to a broken pipeline and muddies every evaluation you run afterward.

The pragmatic sequence follows from all of this. Build the held-out evaluation set first, since every decision above depends on it. Measure the recall curve and repair upstream failures before adding stages. Then trial the cheapest reranker that fits your deployment constraints, an open cross-encoder if you have inference capacity, a hosted API if you do not, and keep it only if recall@n and answer correctness both move. A reranker earns a permanent place in the pipeline the same way any component does: by winning its ablation.

Diagest is ixprt's data-for-AI pipeline, the ingestion and data-quality layer that sits upstream of retrieval. For the framing behind that layer, see what data-for-AI means in practice.