US/EU Ask-Your-Docs Cost: Comparing Cheap Embeddings and Rerank Alternatives

Short answer: for an ask-your-docs support system, compare cheap embeddings and rerank in the same semantic search eval, build the tenant usage ledger first, and choose the smallest pipeline that meets answer quality and US/EU policy requirements.

The concrete workload matters. A support team is enriching a product catalog from messy descriptions: “small black charger, works with old tablet” may need to become a structured accessory record, even when the source text is inconsistent. Semantic search finds candidate descriptions; reranking decides which few passages deserve the answer model's context. The winning design is the one that keeps that decision visible in US and EU tenant-level accounting.

Cheap isn't a retrieval strategy.

What must a US/EU tenant ledger capture before semantic search is tuned?

Start with an event contract, not a model shortlist. Every catalog-enrichment request needs a tenant identifier, region, pipeline version, corpus snapshot, operation name, input token count, retry count, and final status. Keep raw descriptions out of the cost ledger. A stable request identifier is enough to join an approved debugging trace when an engineer needs to inspect one answer.

There is a useful failure mode here: a batch worker can merge records from several tenants, emit one provider invoice, and still preserve separate internal usage events. If it doesn't, the first cost comparison is already untrustworthy. I keep the region and pricing snapshot date with the event, and I count a retry after a 429 as usage rather than deleting it from the report. Back off; don't tight-loop. A per-1M-token rate is only meaningful after this allocation exists.

The ledger also needs a clear ownership rule for shared work. Suppose a nightly catalog refresh embeds a normalized description once, then several tenants reference the same product record. The system should define whether that indexing event belongs to the catalog owner, is apportioned by tenant, or is treated as a platform cost outside tenant usage. Any of those policies can be defensible; silently switching between them cannot. The same question applies to cached query embeddings, rerank requests triggered by a fallback rule, and answer-generation retries. Write the rule beside the event schema, test it with a multi-tenant fixture, and expose the resulting buckets in the evaluation report. Otherwise an apparently precise comparison of OpenAI, Cohere, Voyage, or another backend is only precise about an accounting convention that nobody agreed on.

Use the same accounting shape for changed index text, query text, rerank candidates, and generation context. A shared total can look healthy while one tenant quietly pays for long descriptions or an unusually high rerank rate.

The regional gate comes before the weighted score. Verify processing location, retention, deletion, subprocessors, and contract terms for both US and EU deployments. I'm not sure a particular provider will meet a company's policy without reading its current terms and approvals; your mileage may vary by tenant and data class.

I keep the comparison sheet intentionally plain. For each candidate, the row records the same evidence: retrieval quality, rerank lift, token volume, latency, region eligibility, retention terms, and adapter effort. A provider that wins one column does not automatically win the workload.

Decision dimension	What to record	Why it changes the choice
Retrieval	Recall at the candidate cutoff and answer attribute accuracy	A low unit rate is irrelevant if the SKU evidence is absent
Reranking	Lift at the final context cutoff and added input tokens	A second stage should earn its latency and token volume
Tenant cost	Changed index tokens, query tokens, retries, and generation input by tenant	Shared averages hide expensive catalog shapes
Region and policy	Processing location, retention, deletion, and approved contract terms	A quality pass cannot override a policy failure
Operations	Adapter work, rate limits, telemetry, and rollback path	Portability has a maintenance cost of its own

How should an ask-your-docs pipeline compare embeddings, rerank, and cost?

Now freeze the experiment before comparing OpenAI, Cohere, Voyage, or a self-hosted alternative. Use the same catalog snapshot, chunking rule, query set, relevance labels, embedding dimension policy, candidate count, context limit, and answer prompt. Otherwise the comparison is a change of several variables disguised as a model comparison.

I use three gates in the eval harness. First, does embedding retrieval put relevant catalog evidence inside the candidate set? Second, does reranking improve the ordering at the context cutoff? Third, does the final answer extract the right product attributes and abstain when the description is too vague? A better ranking score with no answer-quality gain is not a reason to add another paid stage.

The first failure is counting only query tokens. Catalog enrichment also pays for initial indexing and every changed description. If a cleaning job rewrites every record's whitespace, a supposedly incremental index can turn into a full re-embedding run. Store a normalized-content hash and count only records whose embedding input actually changed.

The second failure is treating rerank input as free. A top-50 candidate set with long passages can cost more and add more latency than a top-10 set, even when the reranker has a compelling per-1M-token rate. The correct comparison is cost per evaluated answer, separated into indexing, retrieval, rerank, generation, retries, and observability. Add p50 and p95 latency beside the cost; a tenant's support workflow has a user waiting at the other end.

Here is a small, provider-neutral accounting model. It does not invent a rate. It turns the experiment into volumes that can be joined with an approved rate sheet later.

from dataclasses import dataclass


@dataclass(frozen=True)
class TenantRun:
    tenant_id: str
    region: str
    changed_index_tokens: int
    queries: int
    query_tokens: int
    candidates_per_query: int
    average_candidate_tokens: int
    rerank_fraction: float
    generation_tokens_per_query: int


def usage(run: TenantRun) -> dict[str, int | str]:
    if not run.tenant_id or run.region not in {"US", "EU"}:
        raise ValueError("tenant_id and an approved region are required")
    if not 0.0 <= run.rerank_fraction <= 1.0:
        raise ValueError("rerank_fraction must be between 0 and 1")

    reranked = round(run.queries * run.rerank_fraction)
    return {
        "tenant_id": run.tenant_id,
        "region": run.region,
        "embedding_index_tokens": run.changed_index_tokens,
        "embedding_query_tokens": run.queries * run.query_tokens,
        "rerank_input_tokens": (
            reranked
            * run.candidates_per_query
            * run.average_candidate_tokens
        ),
        "generation_input_tokens": (
            run.queries * run.generation_tokens_per_query
        ),
    }

The numbers in a run should come from production-like histograms, not from this example. I log the rejected input too: an invalid region or negative token count should fail before a request reaches a model. That little boundary prevents a later cost report from confusing an application bug with model usage.

When does selective reranking earn its place?

Reranking is useful when the evidence is already in the candidate set but vector similarity places it below the context cutoff. It cannot recover a missing catalog record, repair a bad chunk, or add a product attribute that was never present in the source. That diagnosis is the whole point of the eval.

Start with three replayable conditions: embeddings only, rerank every query, and rerank only queries that match a recorded ambiguity rule. The rule might cover near-duplicate product families, conflicting versions, or a low margin between the first two retrieval scores. Keep the rule deterministic enough to replay in a notebook and production.

Measure recall at the candidate cutoff, nDCG at the context cutoff, attribute accuracy, groundedness, abstention, p50/p95 latency, and each token bucket from the accounting model. Break results down by tenant, region, language, and description quality. An average can hide a single EU tenant whose policy fails, or a catalog class whose relevant passage never survives the first retrieval stage.

Short pipeline. Long audit trail.

How do stable adapters make retrieval comparisons fair?

Keep provider-specific response shapes behind a small internal contract. The application should pass normalized text and receive stable chunk IDs, scores, usage metadata, and a status category. It should not know whether a backend calls the operation embedding, rerank, or something else.

from dataclasses import dataclass
from typing import Protocol, Sequence


@dataclass(frozen=True)
class RankedChunk:
    chunk_id: str
    score: float


class RetrievalBackend(Protocol):
    def embed(self, texts: Sequence[str]) -> list[list[float]]: ...

    def rerank(
        self, query: str, chunk_ids: Sequence[str], limit: int
    ) -> list[RankedChunk]: ...

The adapter owns authentication, request serialization, retry policy, region selection, and response validation. It must reject duplicate chunk IDs, missing usage metadata, and scores that cannot be parsed. A clear ValueError at the boundary is easier to investigate than a silent drop that changes recall three stages later.

OpenAI, Cohere, Voyage, and self-hosted alternatives can all be treated as candidates in this test, but their names should stay in the experiment record rather than leak into application logic. The useful comparison is the normalized contract and the measured result. Keep the provider-specific evidence attached to the row.

This is also where a plain HTTP interface can be helpful: any language can implement the same contract without making the rest of the application depend on an SDK. It does not remove the need for an adapter or an eval harness. Portability is an internal design property, not a marketing adjective.

For the comparison, run each adapter against the same frozen inputs and store model identifiers, request settings, corpus version, and evidence date beside the result. Do not rank providers from a single query or from a published unit price. The rate sheet answers arithmetic; the replay answers whether the system still finds the right catalog evidence.

What is the right alternative when embeddings or rerank do not fit?

The catch is that one pipeline cannot satisfy every tenant. Keep embeddings-only retrieval when the candidate set already clears the quality threshold and the extra stage adds latency without improving answers. Choose a direct provider when its approved processing terms or specialized controls are required. Choose self-hosting when policy prohibits external processing and the team can own hardware, patching, capacity planning, and model reevaluation.

An exact-match or metadata filter can be the better alternative for SKU, region, or compatibility fields. A lexical search stage can rescue rare model numbers that semantic similarity smooths away. For vague descriptions, the honest answer may be “insufficient product evidence,” followed by a clarification request; spending more tokens on a confident guess is a product defect.

I would ship the smallest pipeline that passes the tenant-specific eval, then add reranking as an evaluated exception. Keep the result ledger, not just the final provider choice. Catalogs change, tenant mixes change, and a cost per 1M tokens is only one input to the decision.

References

• OpenAI Function Calling guide: https://platform.openai.com/docs/guides/function-calling
• ElevenLabs documentation: https://elevenlabs.io/docs