Text Summarization Quality and Cost Explained — A Media Catalog Decision Guide

Choose a summarization model by measured catalog outcomes and tenant-level accounting, not by a universal quality ranking. A media catalog has short titles, duplicated metadata, rights notes, and occasionally a description that is really an HTML fragment; the best API is the one whose errors you can detect and whose usage you can charge to the right tenant.

Short answer: keep the model behind a provider-neutral interface, evaluate OpenAI, Claude, and Gemini on the same redacted catalog set, and record input tokens, output tokens, latency, retries, and validation results for every tenant. There is no honest quality-cost winner without that workload-specific record.

What exactly are we comparing?

The useful unit is not a single prompt. It is a catalog enrichment job with a contract. For each messy description, the service should return a bounded summary, preserved identifiers, and a reason when it refuses to infer a fact. The summary can be shorter than the source, but it must not silently turn a rating, release date, cast member, or licensing restriction into an invented field.

That contract changes the evaluation. A fluent paragraph that drops a rights restriction is worse than an awkward paragraph that preserves it. In media systems, I treat omission of a business-critical constraint as a correctness failure, not a style preference.

At first, I treated summary quality as a writing question. In email and OTP flows, I learned to look for delivery gaps and rate-limit boundaries instead; catalog enrichment needs the same habit of looking for the failure that a happy-path sample cannot show.

The decision record has four invariants:

• Every request carries a tenant ID and a catalog item ID through the whole path.
• Raw text is bounded before it reaches a model, and the output is checked against a schema after it returns.
• Usage is recorded from the provider response when available, with an explicit estimated state when it is not.
• A retry cannot create a second billable enrichment accidentally.

The failure boundaries are equally important. A malformed description belongs at input validation. A provider timeout belongs in retry policy. A summary that violates the contract belongs in output validation. Mixing those states into one generic 500 makes tenant cost analysis and operational triage needlessly difficult.

That is the boundary.

Option	Useful boundary	Cost-accounting implication	Quality question to test
OpenAI API	A broad text-generation API with a separately maintained tokenizer library, tiktoken, for local estimates	Compare local estimates with response usage and keep the estimate flag	Does the summary preserve catalog constraints under the chosen context limit?
Claude API	A text-generation API with its own request and token-counting conventions	Do not reuse another provider's token count; store provider and counting method	How does it handle long, repetitive descriptions and explicit refusal rules?
Gemini API	A text-generation API with provider-specific counting and request semantics	Keep counting and retry metadata provider-specific	Does the output stay schema-valid when descriptions contain markup or mixed languages?

These are engineering boundaries, not a ranking. Tokenization, context behavior, safety settings, response metadata, and retry semantics are not interchangeable. A tokenizer can estimate work; it cannot predict whether a model will preserve the one sentence an editor actually needed.

How should a text summarization API handle quality and cost?

Start with a small, labeled set. I would use at least 100 descriptions split across ordinary blurbs, duplicated synopses, HTML-heavy text, missing fields, and rights-sensitive notes. The number is a test-design choice, not a claim that 100 examples proves general quality. It is large enough to expose recurring categories before the team argues about prose polish.

Score each result on separate axes:

1. Factual support: every claim in the summary must be traceable to the input.
2. Field retention: required facts such as language, format, and restriction notes survive.
3. Length compliance: the result fits the product's display and storage limits.
4. Parseability: the response matches the expected JSON shape without repair by guesswork.
5. Editorial usefulness: a human can understand the item without rereading the source.

A single average hides the dangerous cases. Keep a failure matrix by category, provider, model configuration, and prompt revision. Ten excellent summaries do not compensate for one class of descriptions that consistently loses rights information. That is the kind of edge case that passes a demo and reaches production.

Do not compare raw output tokens as if they were identical units. The OpenAI tiktoken library is useful for local BPE-based estimates, but accounting should still prefer usage returned by the actual request. For the other APIs, use their documented counting mechanism or response metadata. Store the source of the number. An estimate and an observed count are different facts.

The practical limitation is that a cross-provider score is only comparable after the input contract, output limit, and counting method are held constant. A provider may be a poor fit for a tenant that needs a particular retention rule, latency ceiling, or data-handling policy even when its average editor score is higher.

Quality also has a cost. A longer answer can score better with an editor while consuming more output tokens and occupying more storage. Set a maximum output length and evaluate the business result at that limit. Otherwise the experiment rewards verbosity that the catalog never asked for.

How do we make tenant cost visible?

The accounting record should be append-only from the application point of view. It needs a request ID, tenant ID, provider, model label, input and output token counts, count source, latency, retry number, validation result, and a status such as completed, rejected, or failed. Keep the raw description out of this record unless retention policy explicitly permits it.

Here is the critical path in Python. The client is deliberately generic; the adapter is where provider-specific authentication, request format, and usage parsing belong.

from dataclasses import dataclass
from typing import Any, Protocol
import hashlib
import json


class TextModel(Protocol):
    def summarize(self, prompt: str, request_id: str) -> dict[str, Any]:
        ...


@dataclass(frozen=True)
class Usage:
    input_tokens: int | None
    output_tokens: int | None
    source: str


def stable_key(tenant_id: str, item_id: str, source_hash: str) -> str:
    raw = f"{tenant_id}:{item_id}:{source_hash}".encode()
    return hashlib.sha256(raw).hexdigest()


def summarize_item(
    model: TextModel,
    tenant_id: str,
    item_id: str,
    description: str,
) -> tuple[dict[str, Any], Usage, str]:
    bounded = description[:10_000]
    source_hash = hashlib.sha256(bounded.encode()).hexdigest()
    request_id = stable_key(tenant_id, item_id, source_hash)
    prompt = (
        "Summarize the catalog description. Preserve supported facts, "
        "especially rights and format notes. Return JSON only with keys "
        "summary and warnings.\n\n" + bounded
    )
    result = model.summarize(prompt, request_id)
    usage_data = result.get("usage", {})
    usage = Usage(
        input_tokens=usage_data.get("input_tokens"),
        output_tokens=usage_data.get("output_tokens"),
        source="provider_response" if usage_data else "estimated_or_unknown",
    )
    payload = json.loads(result["text"])
    if not isinstance(payload.get("summary"), str):
        raise ValueError("summary must be a string")
    return payload, usage, request_id

The idempotency key is derived from the tenant, item, and bounded source, so a retry can look up the same enrichment. It is not a substitute for an idempotency guarantee in the adapter or job store; that boundary must be explicit. The example also makes a deliberate choice to reject invalid JSON instead of quietly repairing it. A repair step can be valid, but it needs its own event and cost record.

For dashboards, aggregate by tenant first, then by provider and model label. Show estimated and observed usage separately. A monthly total that combines them looks precise while hiding the uncertainty that matters during a provider migration.

This is a trade-off, not a universal optimization. Per-tenant attribution adds ledger fields and storage work, but omitting it makes a shared catalog service impossible to audit fairly.

Why did I reject one routing shortcut?

I would not route every description to the cheapest-looking model based on a static quality score. The shortcut is attractive: classify the input, pick a provider, and watch aggregate spend fall. It fails when the classifier is wrong, when a prompt revision changes token usage, or when the supposedly easy category contains a rights note with unusual wording.

A valid use case remains. Static routing is reasonable for a narrow, stable category with a labeled regression set, a hard output contract, and a measured fallback path. Keep the routing decision in configuration, version it, and log it with the request. Then a tenant can answer, “Why did this item use that route?” without reconstructing application state from scattered logs.

A safer rollout is less clever. Run the same 100-plus-item evaluation set through each candidate, shadow a small production slice with tenant identifiers preserved, and compare validation failures, retained fields, latency, and observed token counts. Add a human review sample for the categories where the automated score cannot see a business loss.

The final decision rule is plain: choose the configuration that meets the catalog's retention and format thresholds, then compare its observed cost per accepted enrichment. “Accepted” matters. A cheap response that needs manual correction is not a cheap response; it is an unpaid second workflow.

The architecture keeps that judgment reversible. Provider adapters can change. The tenant ledger, evaluation set, output contract, and failure taxonomy should not.

References

• OpenAI, tiktoken tokenizer library: https://github.com/openai/tiktoken
• OpenAI API text generation guide: https://platform.openai.com/docs/guides/text-generation
• Anthropic Messages API documentation: https://docs.anthropic.com/en/api/messages
• Anthropic token counting documentation: https://docs.anthropic.com/en/docs/build-with-claude/token-counting
• Google Gemini API counting tokens documentation: https://ai.google.dev/gemini-api/docs/tokens
• Cohere Rerank overview: https://docs.cohere.com/docs/rerank-overview