Rerank does not have a context window in the sense a chat model does. It has a per-document token budget applied to every document independently, and text past it is truncated rather than rejected — which is why a rerank pipeline can quietly stop working well without ever erroring.
The documented limits
Cohere’s rerank documentation and the rerank API reference document the ceilings. On the v2 endpoint the relevant parameter is max_tokens_per_doc, documented with a default of 4,096: a document longer than that is truncated to that length before scoring. There is also a documented maximum number of documents per request — in the low thousands — and exceeding it is a validation error rather than a silent drop.
The per-document default and the per-request document cap have both been revised across rerank model generations, and the v1 endpoint expressed the same idea differently, with a max_chunks_per_doc parameter that split long documents into chunks rather than cutting them. Read the reference for the endpoint version and rerank model you are calling; the numbers here are the documented values at the time of writing.
The query shares the budget
The important structural point about rerank is that it is a cross-encoder rather than an embedding model. It does not compute a vector for the query and a vector for each document and compare them. It reads the query and one document together and outputs a relevance score, which is why it is more accurate than vector similarity and why it costs a forward pass per document.
That is also why the token budget is not the document’s alone: the query occupies part of the same input. A long query — and in a RAG system the “query” is often a rewritten, expanded, multi-sentence thing rather than what the user typed — eats directly into how much of each document gets read. If your queries grew from 20 tokens to 400 during a prompt-engineering pass, every document just lost 380 tokens of tail, silently.
What truncation actually costs you
Truncation keeps the head of the document. That interacts badly with two very common document shapes.
- Documents whose answer is at the end. A support article with a boilerplate header, a contract with the operative clause in section 9, a transcript where the decision is in the last minute. If the relevant passage falls past the cut, the reranker scores a document it never saw the relevant part of, and it will rank it below a shorter document that mentioned the topic in its first line.
- Documents that all share a prefix. Every page carrying the same 300-token legal header means 300 tokens of every budget spent identically across the whole candidate set, contributing nothing to the ranking while displacing text that would.
The failure has no error and no warning. Scores come back, results are ordered, the pipeline runs. The only symptom is worse retrieval, which is indistinguishable from a hundred other causes unless you go looking for this one specifically. The defence is to chunk before reranking rather than relying on truncation — you control which part survives, and each chunk is scored on its own merits instead of inheriting a score from its document’s opening paragraph. Count the chunks with Cohere’s own tokenizer rather than by character count, since the ratio differs by language.
A request that respects the limits
curl https://api.cohere.com/v2/rerank \
-H "Authorization: Bearer $CO_API_KEY" \
-H "content-type: application/json" \
-d '{
"model": "rerank-v3.5",
"query": "restock date for the Gazelle Ultimate C8 in Utrecht",
"documents": [
"Utrecht stock: Gazelle Ultimate C8, 0 units. Restock expected 19 August 2026.",
"Amsterdam stock: Gazelle Ultimate C8, 3 units available.",
"Returns policy: unused bikes may be returned within 30 days."
],
"top_n": 2,
"max_tokens_per_doc": 2048
}'
The response is indices and scores, not documents:
{
"results": [
{"index": 0, "relevance_score": 0.981},
{"index": 1, "relevance_score": 0.412}
],
"meta": {"billed_units": {"search_units": 1}}
}
index refers to the position in the array you sent, so you keep your own documents and reorder them yourself. relevance_score is between 0 and 1 and is comparable within one response but not across queries — it is not a probability and there is no universally correct threshold. Setting top_n below the document count reduces the size of the response, not the amount of work: every document is scored regardless, which is why the cost model below is what it is.
A chunking policy for rerank
If truncation is the failure and chunking is the defence, the policy needs to be explicit rather than inherited. A workable one:
- Chunk on structure before size. Section headings, paragraph boundaries, clause numbers. A chunk that ends mid-sentence scores worse than one that ends at a boundary, because the reranker is reading it as text and half a sentence reads as irrelevant.
- Target well under the per-document budget. Aiming at 4,096 leaves nothing for the query and no margin for a tokenizer that counts your language less efficiently than you assumed. Something in the region of half the budget is a comfortable default, measured with Cohere’s tokenizer rather than by characters.
- Overlap adjacent chunks by a sentence or two. A fact that straddles a boundary otherwise appears in neither chunk in a scoreable form. Overlap costs duplicate tokens and buys you against the single most annoying retrieval failure.
- Prefix each chunk with its document title or path. A chunk from the middle of a document has no context about what it belongs to, and the reranker sees only what you send. One short line of provenance often changes the ranking more than any parameter.
- Deduplicate after scoring, not before. Several chunks of one document can all score highly; keeping the best one per document before you pass results to the model avoids filling the context with near-identical passages, which is where a long context window gets spent on nothing.
The one thing not to do is to strip the document down to a summary before reranking. It is tempting — it fits the budget and it is cheap — but the reranker’s whole advantage over vector search is that it reads the actual text against the actual query, and a summary has already thrown away the specific sentence the query was about.
How rerank is billed
Rerank is not billed in tokens. It is billed in search units, and the definition is what makes the token limits matter for cost as well as quality: one search unit covers a single query against a documented number of documents, so a request with more documents than that consumes proportionally more units. The count comes back in meta.billed_units.search_units on every response, which is the number to log.
Latency scales with the candidate count in the same way, and for the same reason: every document is a forward pass. Reranking 500 candidates is not five times the work of 100 in the abstract — it is exactly five times the passes — and it sits on the critical path before the model has generated a single token. A retrieval stage that adds 400ms to time-to-first-token is a real cost to weigh against the quality it buys, and it is worth measuring at your actual candidate count rather than at the example one.
Two consequences for pipeline design. Reranking your entire corpus is expensive in a way that scales with corpus size, so the standard shape — retrieve a broad candidate set cheaply with vectors or BM25, then rerank the top 50 to 200 — is a cost decision as much as a latency one. And chunking to avoid truncation increases the document count, which increases the units billed. The right chunk size is the one that keeps relevant text intact without multiplying the candidate set fivefold, and it is worth working out deliberately rather than inheriting from whatever your vector store was configured with.
Top comments (0)