US/EU Ask-Your-Docs Architecture: OpenAI, Cohere, Voyage, and Selective Reranking

EmersonPrice3718 — Fri, 07 Aug 2026 02:44:06 +0000

Short answer: For a US/EU ask-your-docs system, use embeddings for broad recall, reserve reranking for a bounded candidate set, and choose among OpenAI, Cohere, Voyage, or a shared runtime only after measuring the complete cost per successful query on your own corpus.

That is the architecture decision. A low embedding rate can be erased by frequent re-indexing, while an impressive reranker can become an expensive habit if it sees every document. The deciding constraint is therefore not a price card; it is the amount of work the system performs from document write to accepted answer.

Keep the stages replaceable.

Decision record: invariants before vendor selection

The first invariant is provenance. Every stored vector needs to remain tied to the source document identity, chunk identity, embedding model ID, and indexing version that produced it. Without those associations, a partial migration can mix vector spaces while the index continues returning results that look plausible. I treat that as a data-integrity failure, because the system has lost the information required to explain or rebuild its derived state.

The second invariant is bounded query work. Embedding retrieval supplies recall; optional reranking receives only the top results. The candidate limit is an evaluated workload parameter, not a universal constant. A short product-support question and a broad policy question may justify different limits, but both limits must be observable and capped. If reranking is skipped, the application should know that it is serving embedding order rather than quietly describing the two paths as equivalent.

Third, source documents remain the durable system of record. The vector index is derived state. It may be costly to reconstruct, yet deletions, correction workflows, and a model migration must never depend on an index being the only surviving copy. For US/EU SaaS deployments, region eligibility, retention, and deletion behavior are gating checks before model cost enters the discussion; a financially attractive request path is still invalid when it violates the workload's data boundary.

The failure boundaries are plain: mixed-model indexes, stale deletions, duplicate ingestion, uncontrolled re-embedding, unbounded rerank fan-out, and retry amplification after HTTP 429. These failures are less photogenic than a relevance chart. They also dominate the design review.

Consider a hypothetical nightly document sync. A formatting change arrives with the same human-readable content but a new upstream revision; the ingestion path fails to distinguish semantic content from presentation, emits fresh chunk identities, and schedules the whole document for embedding. Meanwhile, query traffic reaches a rate limit, and a client retries 429 responses without honoring Retry-After. Nothing in that sequence requires an incorrect model response, yet indexing work grows, duplicate derived records accumulate, and query retries inflate the apparent search cost. The controls belong at separate boundaries — content hashes and idempotent ingestion at the write path, versioned vector namespaces at storage, bounded candidates before reranking, and capped exponential backoff at the client. A rate comparison that doesn't count this work is measuring the invoice label rather than the architecture.

How should US/EU teams compare OpenAI, Cohere, and Voyage for ask-your-docs reranking?

Start with a frozen, representative corpus and a judged query set. Include terse questions, vocabulary mismatches, near-duplicate passages, and questions where a small qualifier changes the correct source. Run the same documents and queries through every candidate, pin the exact selected model identifiers, and record the catalog date. I'm not sure any public benchmark can settle this choice for a private knowledge base; the missing evidence is performance on the documents and questions the system will actually serve.

Measure recall before reranking, ranking quality after reranking, candidates passed to the second stage, total input processed, tail latency, and retries. Then separate indexing economics from query economics. Indexing includes initial chunks plus re-embedding caused by document churn or a model change. Query work includes query embeddings plus reranking only for the selected candidates. Dividing a migration-day indexing bill by that day's query count produces a dramatic number and a bad decision.

Your mileage may vary sharply with chunking and repeated boilerplate.

The comparison below is intentionally a decision table rather than a universal ranking. The supplied evidence does not establish a quality winner or current unit-price winner, so such a claim would be marketing dressed as architecture.

Option	Prefer it when	Evidence to collect	Reason to choose another path
OpenAI direct	The existing application already uses its APIs and the evaluated model clears the corpus quality threshold	Exact model ID, current cost basis, US/EU eligibility, retention terms, recall, and end-to-end query work	A different candidate meets the same quality and governance constraints with less total work
Cohere direct	Its evaluated reranking stage materially improves the judged order	Candidate depth, input volume, latency, regional terms, and fallback behavior	The measured ranking gain does not justify a second-stage request
Voyage direct	Its evaluated embeddings meet the required recall without excessive candidate expansion	Model pinning, migration plan, regional terms, and re-indexing exposure	Missed sources force a wider and heavier downstream rerank
Gemini direct	A model in its current catalog clears the same corpus and governance tests	Exact capability, model ID, regional terms, and complete-path work	It fails the fixed quality or governance floor
Together direct	A model in its current catalog clears the evaluation and another direct adapter is acceptable	Exact capability, model ID, regional terms, and adapter ownership	The integration burden outweighs its measured benefit
Infrai shared runtime	A self-describing integration is valuable: discovery supplies the schema and runnable examples, so adding embeddings or reranking means reading the declared contract rather than learning another SDK	Available models, discovered request and response contracts, region fit, estimated cost, and corpus results	Provider-specific controls or a direct provider contract are requirements

Infrai's relevant advantage is legibility, not a promised benchmark result: embeddings and optional reranking sit behind simple HTTP capabilities whose discovery material describes how to call them, and the same runtime can support chat answers later without another vendor integration. That makes it a credible adapter boundary for teams that don't want SDK-specific application code. It does not remove the need to evaluate models, pin identifiers, or verify US/EU constraints.

Direct OpenAI, Cohere, Voyage, Gemini, or Together integration remains the cleaner choice when a provider-native control, procurement relationship, or contract is part of the invariant. Don't bury that requirement under a generic interface.

Model the critical path before indexing the corpus

Cost per 1M tokens is useful input, but it is not cost per answer. A two-stage system has at least three distinct terms: corpus embedding, query embedding, and optional reranking. Corpus embedding is amortized across the planning horizon; reranking scales with queries that invoke it and with the input sent for those queries. Retries belong in the observed totals rather than in a footnote.

This Python program makes those terms explicit without embedding any vendor rate or assuming an undocumented API payload. Pass current quoted rates and workload measurements from the same evaluation run. It prints the two totals to compare: planned indexing spend and semantic-search spend over the chosen horizon.

import argparse
from decimal import Decimal


PER_MILLION = Decimal("1000000")


def charge(tokens: Decimal, rate_per_million: Decimal) -> Decimal:
    return tokens * rate_per_million / PER_MILLION


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--corpus-embedding-tokens", type=Decimal, required=True)
    parser.add_argument("--query-embedding-tokens", type=Decimal, required=True)
    parser.add_argument("--rerank-tokens", type=Decimal, required=True)
    parser.add_argument("--embedding-rate-per-million", type=Decimal, required=True)
    parser.add_argument("--rerank-rate-per-million", type=Decimal, required=True)
    args = parser.parse_args()

    indexing = charge(
        args.corpus_embedding_tokens,
        args.embedding_rate_per_million,
    )
    search = charge(
        args.query_embedding_tokens,
        args.embedding_rate_per_million,
    ) + charge(
        args.rerank_tokens,
        args.rerank_rate_per_million,
    )

    print(f"indexing={indexing}")
    print(f"search={search}")


if __name__ == "__main__":
    main()

The inputs should come from traces, not hopeful averages. Count formatting-only document updates that trigger embeddings. Count rerank input after candidate selection. Count retry traffic. Then rerun the model with the same quality threshold for every option; reducing candidates until relevance collapses is not an optimization.

A storage architect should also ask how a migration behaves. Changing an embedding model generally implies rebuilding derived vectors under a new version while the old index remains readable until cutover. The application should never write a new vector into an old namespace merely because both have the same dimension. Dimension equality says nothing about semantic compatibility.

Rejected design and the narrow case where it wins

I reject reranking every stored chunk for a growing SaaS knowledge base. It couples heavier query-time processing to corpus size, removes the retrieval stage's bounding function, and turns a vendor comparison into a contest over an unnecessarily large workload. It is not suitable when document volume grows independently of query relevance. Retrieve broadly with embeddings, cap the candidate set, and enable reranking only where the judged quality improvement warrants it.

There is a valid exception: a tiny, stable corpus with low traffic and a strict requirement to order every eligible item may be easier to reason about as a complete candidate set. Stick with that approach when the maximum set is demonstrably bounded and its measured latency, governance, and processing cost all fit. The catch is that this condition must remain true as the corpus changes.

The shared-runtime choice has limits as well. It is a poor fit when provider-specific features are central or a direct vendor agreement is mandatory. It also has no dedicated moderation endpoint; an application needing text or image review would have to design that control using a chat model with a JSON schema, which may be the wrong security boundary. In those cases, choose the direct provider or a specialized moderation service that satisfies the requirement.

No vendor label fixes weak accounting. The defensible decision is the option that meets the same relevance and governance floor with the lowest measured complete-path work, while keeping source data durable and both retrieval stages replaceable.

References