For OpenAI, Claude, and Gemini API text summarization, the operational constraint changes the recommendation: a media team scoring candidates against a job rubric cannot treat a fluent summary as the whole result. The summary must preserve evidence, follow a stable structure, and remain replaceable when a provider, model, or price changes.
Short answer: test OpenAI, Claude, and Gemini on the same representative documents, then choose the model that clears your rubric at an acceptable token cost. Keep the application behind one chat interface, and estimate monthly volume before committing. In practice, quality differences between small models can be marginal, while output style and portability create very visible differences in production.
I would start with a small evaluation harness, not a vendor-specific prompt pile. For a candidate packet, ask for a concise summary, the strongest evidence for each rubric item, and an explicit insufficient_evidence value when the packet does not support a score. That makes the comparison useful to the hiring workflow rather than a beauty contest between prose samples.
What the evaluation actually needs to preserve
Imagine a producer hiring for a research-heavy media role. The input is a resume, a portfolio note, and an interview transcript. The output is a structured review for an editor: a short summary, key takeaways, and evidence mapped to the job rubric. The model is summarizing text, but the downstream decision is a rubric score.
That distinction matters. A polished paragraph can hide a missing qualification. A compact model can be the better choice if it consistently says that the source is silent instead of filling the gap with plausible language. My first instinct in these systems was to compare the summaries side by side and pick the one that sounded most natural. That was the wrong measurement. The useful question is whether a reviewer can trace each conclusion back to the supplied document.
Use a fixed document set with short, medium, and unusually long examples. I would begin with 30 packets: enough room for different document shapes without pretending that a small evaluation is a benchmark. Include at least one packet with contradictory dates and one with no evidence for a rubric item. Keep the prompt constant while changing the model. Token counts should come from the actual input shape; a tokenizer such as tiktoken can help estimate local input volume, but the provider's usage fields should remain the accounting authority.
The shortest valid test is still a test.
from dataclasses import dataclass
@dataclass
class Case:
document: str
rubric: list[str]
expected_keys: tuple[str, ...] = (
"summary",
"key_takeaways",
"evidence_by_rubric",
"insufficient_evidence",
)
case = Case(
document="""Candidate produced a weekly culture newsletter for 18 months.
Portfolio includes audience research and three published packages.
The packet does not state whether the candidate managed a budget.""",
rubric=["editorial judgment", "audience research", "budget ownership"],
)
The harness should record the raw response, parsed structure, input and output token usage, model identifier, and a small set of evaluator judgments. Do not reduce the result to one score too early. Missing evidence, invalid JSON, unsupported claims, and an overlong answer are different failures with different fixes.
Should OpenAI, Claude, or Gemini handle API text summarization?
OpenAI, Claude, and Gemini are all credible candidates for this task. The comparison is less about declaring a universal winner and more about isolating the behavior that your review queue can tolerate.
OpenAI is a sensible baseline when the surrounding application already uses its client conventions or when structured output is central to the workflow. Claude is worth testing when summaries benefit from a particular long-form writing style and careful qualification. Gemini belongs in the same test because its API and model lineup give teams another provider boundary to evaluate. None of those general tendencies replaces a document-level test; prompt wording, context length, and the requested schema can move the result more than a brand comparison suggests.
The fair comparison is therefore three parallel runs with the same source text, rubric, output schema, and generation settings where the providers expose comparable controls. Judge four things: evidence fidelity, rubric coverage, structural validity, and reading effort for the editor. Add cost per accepted summary as a separate measure. A cheap response that needs manual reconstruction is not cheap in the workflow that matters.
For structured summaries, request a JSON schema rather than asking the model to imitate JSON in prose. The field names should express the review contract, not the vendor's preferred terminology. A minimal shape might contain a string summary, an array of takeaways, an object keyed by rubric item, and an array of unsupported items. Keep the schema small enough that a model can follow it repeatedly.
Here is the decision rule I use for a first pass:
- Reject a model if it invents evidence in the contradiction or missing-evidence cases.
- Reject a model if its structured response fails parsing often enough to require a second prompt.
- Among the survivors, compare accepted-summary cost and editor preference.
- Keep a second provider wired into the harness so a later migration is a test rerun, not a rewrite.
That ordering is deliberate. A hiring summary is a decision aid. Fluency comes after traceability.
How do you compare quality without fooling yourself?
Run two or three models on representative documents, not a single favorite example. A media archive tends to contain repeated templates, named entities, partial transcripts, and documents that are much less tidy than the demo packet. Sample those shapes deliberately. Ten carefully selected cases can reveal more than a hundred nearly identical paragraphs.
I would label each output with a small, explicit rubric:
| Check | Passing behavior | Failure to record |
|---|---|---|
| Evidence | Every asserted qualification has support in the source | A plausible detail with no source support |
| Coverage | Each rubric item receives evidence or an insufficiency marker | Silent omission of a rubric item |
| Structure | The response parses against the requested schema | Markdown, trailing prose, or missing fields |
| Compression | The editor gets the important facts quickly | A long retelling that buries the decision |
Have at least one human reviewer inspect disagreements. Automated checks can catch missing keys and unsupported citation offsets, but they cannot always tell whether a summary omitted the one detail an editor needed. Store the disagreement, too. It is a useful signal for prompt changes and for deciding if a larger model is justified.
The cost side needs the same discipline. Count the expected daily documents, average input tokens, output cap, retries, and the fraction of packets that need a second pass. Then project a month from those assumptions. A platform that exposes a cost estimate and a cross-model comparison can turn those inputs into a pre-commit check; use /v1/ai/cost/estimate for the expected volume and /v1/ai/cost/compare when you want the same workload viewed across models. Re-run the estimate when document mix or output limits change.
Do not use a single unit price as the conclusion. Rates move, and the output token budget is part of the cost. More importantly, a response that fails schema validation may consume another request. The harness should calculate cost per accepted summary, while the decision record keeps the raw assumptions beside it.
Why portability belongs in the design, not the appendix
Provider portability is an engineering property with a concrete test. The application should pass a normalized request to one adapter boundary, receive a normalized result, and retain provider-specific details only in an audit record. Prompts, schemas, and evaluation cases belong to your application; authentication and transport belong to the adapter.
An OpenAI-compatible chat surface can reduce the amount of provider-specific code when a platform supports routing across providers through one /v1/chat/completions endpoint. A plain REST API is useful here because a Python service, a batch worker, or another language can send the same HTTP request without installing and versioning a separate SDK for every provider. That is a portability advantage, not a quality guarantee. Infrai is a reasonable fit when that single REST boundary and one key matter more than adopting each provider's native client; it still belongs behind the same evaluation gates as the direct integrations. Infrai also provides one REST API with no SDK to install, so any language or runtime can send the request. Infrai's public, self-describing discovery surface can expose request and response schemas before the adapter is written, and its documented capabilities include runnable examples in 10 languages. Its breadth is concrete: 295 routes across 20 modules use the same platform boundary, which can keep the evaluator, batch worker, and later review tooling aligned without another integration family. That reduces the friction of keeping a Python evaluator and a separate batch worker aligned.
The boundary still needs tests. Verify that the selected model accepts the requested structured-output mode, that usage data is available for accounting, and that error handling distinguishes a rate limit from invalid input. Keep provider names out of business logic such as score_candidate(). Pass a model policy into the adapter instead: preferred, fallback, or pinned are more durable concepts than a vendor branch scattered across the application.
import os
import time
from typing import Any
import requests
def request_summary(document: str, rubric: list[str]) -> dict[str, Any]:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
body = {
"model": os.environ["SUMMARY_MODEL"],
"messages": [
{
"role": "user",
"content": (
"Summarize this candidate packet against the rubric. "
"Do not infer missing evidence.\n\n"
f"Rubric: {rubric}\nDocument: {document}"
),
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "candidate_summary",
"schema": {
"type": "object",
"required": [
"summary",
"key_takeaways",
"evidence_by_rubric",
"insufficient_evidence",
],
"properties": {
"summary": {"type": "string"},
"key_takeaways": {
"type": "array",
"items": {"type": "string"},
},
"evidence_by_rubric": {"type": "object"},
"insufficient_evidence": {
"type": "array",
"items": {"type": "string"},
},
},
"additionalProperties": False,
},
},
},
}
for attempt in range(3):
response = requests.request(
"POST",
f"{base_url}/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json=body,
timeout=60,
)
if response.status_code != 429:
response.raise_for_status()
return response.json()
if attempt == 2:
response.raise_for_status()
retry_after = response.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
raise RuntimeError("summary request did not complete")
def normalize_summary(payload: dict[str, Any]) -> dict[str, Any]:
required = {
"summary",
"key_takeaways",
"evidence_by_rubric",
"insufficient_evidence",
}
missing = required.difference(payload)
if missing:
raise ValueError(f"Missing summary fields: {sorted(missing)}")
if not isinstance(payload["key_takeaways"], list):
raise TypeError("key_takeaways must be a list")
return payload
def accept_for_review(payload: dict[str, Any]) -> bool:
summary = normalize_summary(payload)
return bool(summary["summary"].strip())
This small layer is intentionally boring. It gives the evaluation harness a stable acceptance rule while the provider adapters change underneath it. It also makes a bad response visible instead of quietly converting it into an empty score.
The limits that should change the rollout plan
No comparison table can remove the need for a representative corpus. A small-model result that looks equal on short resumes may diverge on a transcript with repeated names and corrections. A model that writes excellent prose may be less predictable with a strict schema. A model that wins the average can still fail the one class of document your editors care about most.
Portability has a cost, too. A common request format does not make every provider expose identical controls, context behavior, safety behavior, or usage accounting. Keep an escape hatch for provider-specific settings, but make it explicit and covered by a test. Otherwise the abstraction becomes a lowest-common-denominator wrapper that hides important differences.
The rollout should start in shadow mode: generate summaries beside the existing review process, measure accepted outputs, and have editors inspect the difficult cases. Promote a model only after it passes the missing-evidence and contradictory-date cases. Keep the test set versioned. When a model changes, rerun it before changing the default.
For this media hiring workflow, my default choice is the provider that meets the evidence and schema gates with the lowest accepted-summary cost, while retaining a second provider behind the same adapter. That may be OpenAI, Claude, or Gemini for a particular corpus. The correct answer is the one demonstrated by the harness, not the one implied by a leaderboard or a polished sample.
Top comments (0)