Short answer: the OpenAI vs Claude vs Gemini decision for text classification and JSON tagging in a Europe-US app backend requires tenant-level accuracy and cost evidence; no universal best simple API exists.
For customer-support catalog enrichment, the deciding constraint is per-tenant cost visibility. A model can return clean JSON and still be the wrong default if retries, oversized descriptions, or an expanding label set make one tenant's workload impossible to explain. The useful unit isn't “one API call.” It's one accepted classification attached to a tenant, schema version, model configuration, and usage record.
This changes the experiment. The simple approach sends every messy product description to one provider and checks whether the response parses. The better approach treats parsing as the first gate, label correctness as the second, and attributable usage as the condition for shipping. No leaderboard can make that decision for a particular catalog.
Why valid JSON is the wrong finish line
Parsing proves syntax, not usefulness. A support agent needs stable tags that make products searchable and route tickets correctly. Input may contain copied HTML, abbreviated dimensions, old category names, or contradictory prose. Output should contain only allowed labels plus a small amount of evidence that an application can validate.
The test set must represent those rough edges. Keep ordinary items, ambiguous items, descriptions with missing attributes, and descriptions that should produce an explicit needs_review result. Freeze the set before comparing configurations. Otherwise, a prompt edit and a dataset edit happen at once, and nobody can tell why the score moved.
Use the same prompt contract and JSON Schema for every candidate. OpenAI, Anthropic's Claude, and Google's Gemini are comparison candidates here, not recommendations. Their public APIs, model catalogs, and billing terms change independently, so a copied winner is stale faster than the application contract. The fair comparison is the behavior observed through the same adapter on the same frozen examples.
Four outcomes are enough for a first pass:
| Gate | Question | Ship signal |
|---|---|---|
| Transport | Did the request complete within the app's latency budget? | Recorded duration and terminal status |
| Structure | Did the response validate against the exact schema? | Zero parser or schema errors |
| Semantics | Are tags supported by the description and allowed taxonomy? | Human-reviewed score on the frozen set |
| Economics | Can usage be assigned to one tenant and one accepted result? | Complete usage ledger with retries included |
Structure is binary. Semantics isn't. Two valid JSON objects can disagree, and the more confident-looking one can still invent a material or product type. I'm not sure a public benchmark can resolve that local ambiguity; a reviewed slice of the actual catalog can.
How should a Europe-US app backend compare JSON text classification accuracy and cost?
Run the same evaluation worker near each production region, but don't confuse network geography with regulatory compliance. Record region, request duration, provider, model identifier, schema version, prompt version, input size, reported usage, attempt count, and validation result. Those fields let an operator separate a slow network path from a verbose tenant catalog or a configuration that needs extra attempts.
Accuracy needs two scores. Exact-tag agreement catches taxonomy drift and supports clean regression tests. A second, human-reviewed score handles defensible alternatives, such as whether “weather resistant” supports both outdoor and durable. Define that rubric before inspecting provider output. If reviewers change the rule after seeing a model name, the comparison has already tilted.
Cost should be computed from the provider's current billing record and stored usage, outside the prompt and outside business logic. Don't bake prices into the classifier. Keep a dated rate table in the reporting layer, join it to immutable usage events, and retain the original units so a later pricing change doesn't rewrite historical reports. Your mileage may vary with cached input, batch programs, or account terms; the invoice and each provider's current documentation resolve those details.
One long example exposes why tenant attribution matters. Suppose tenant northwind-eu uploads 8,000 terse descriptions while contoso-us uploads 800 descriptions copied from supplier pages. Request count makes Northwind look ten times larger. Yet Contoso's inputs may be longer, may trigger more needs_review results, and may require more attempts after schema rejection. A shared monthly total hides all three causes. An event for every attempt, followed by a separate accepted-result event, shows what happened without assigning all overhead to the busiest tenant. It also lets a team cap or queue one tenant without changing model selection for everyone else.
That's the pivot.
Why does the app need its own typed boundary?
The application should own the schema, taxonomy, and accounting envelope. A provider adapter should do only three jobs: translate the generic request, return text plus reported usage, and preserve a neutral terminal status. Keep provider-specific response objects out of catalog tables.
Here is a focused TypeScript boundary. It deliberately contains no vendor endpoint or SDK call; each adapter can follow its provider's current official documentation without leaking that surface into the enrichment workflow.
type Region = "eu" | "us";
type Candidate = "openai" | "claude" | "gemini";
type CatalogTag = "apparel" | "electronics" | "home" | "outdoor";
type Classification = {
tags: CatalogTag[];
needs_review: boolean;
};
type ModelUsage = {
input_units: number;
output_units: number;
};
type ModelReply = {
text: string;
usage: ModelUsage;
model: string;
};
type ClassifyRequest = {
tenantId: string;
region: Region;
description: string;
schemaVersion: "catalog-tags-v1";
};
type CallModel = (
candidate: Candidate,
request: ClassifyRequest,
) => Promise<ModelReply>;
const allowedTags = new Set<CatalogTag>([
"apparel",
"electronics",
"home",
"outdoor",
]);
function parseClassification(text: string): Classification {
const value: unknown = JSON.parse(text);
if (typeof value !== "object" || value === null) throw new Error("invalid_shape");
const record = value as Record<string, unknown>;
if (!Array.isArray(record.tags) || typeof record.needs_review !== "boolean") {
throw new Error("invalid_shape");
}
if (!record.tags.every((tag) => typeof tag === "string" && allowedTags.has(tag as CatalogTag))) {
throw new Error("invalid_tag");
}
return {
tags: record.tags as CatalogTag[],
needs_review: record.needs_review,
};
}
async function classifyCatalogItem(
candidate: Candidate,
request: ClassifyRequest,
callModel: CallModel,
) {
const startedAt = Date.now();
const reply = await callModel(candidate, request);
try {
const classification = parseClassification(reply.text);
return {
classification,
usageEvent: {
tenantId: request.tenantId,
region: request.region,
candidate,
model: reply.model,
schemaVersion: request.schemaVersion,
durationMs: Date.now() - startedAt,
validation: "accepted" as const,
...reply.usage,
},
};
} catch (error) {
return {
classification: null,
usageEvent: {
tenantId: request.tenantId,
region: request.region,
candidate,
model: reply.model,
schemaVersion: request.schemaVersion,
durationMs: Date.now() - startedAt,
validation: "rejected" as const,
reason: error instanceof Error ? error.message : "unknown_validation_error",
...reply.usage,
},
};
}
}
The parser is intentionally strict. Extra prose, an unknown tag, or a wrong shape is a rejected result, even if a developer can guess the intended meaning. That rule keeps downstream support automation boring. It also makes failures comparable: an adapter cannot quietly “repair” one candidate's response while another is judged raw.
In production, validate with a maintained JSON Schema library rather than expanding this small example into a homegrown validator. Pin the schema version in both the request and the event. A taxonomy migration then becomes a controlled evaluation, not a silent prompt tweak.
Where the tenant usage ledger belongs
A simple API at the application layer means one stable function, not one permanent upstream. Begin with a default chosen from the frozen evaluation. Route by tenant only when a documented requirement justifies it: a regional deployment constraint, a latency objective, an approved-model policy, or a budget threshold based on complete usage data. Persist the reason with the routing rule.
Do not retry blindly across all three candidates. Each attempt consumes time and may incur billable usage, even when its output is rejected locally. Set a small attempt budget, record every attempt against the originating tenant, and send exhausted or semantically uncertain items to review. Fast failure is useful.
The catch is that a multi-provider adapter costs engineering time. It is not suitable when the catalog is tiny, classification is non-critical, and one provider already meets the measured quality and regional requirements. Stick with the single adapter in that case, but keep the app-owned schema and usage ledger. At the other extreme, a conventional supervised classifier or embeddings-based pipeline may fit a stable taxonomy with a large labeled dataset better than repeated generative calls. Embeddings represent inputs as vectors and are commonly used for classification; they still require an evaluation tied to this catalog.
There is another boundary: deployment region alone doesn't prove where every request, log, or support operation is handled. Legal and security owners must check the current contract, data-processing terms, retention controls, and subprocessor information for the selected account. An architecture diagram cannot substitute for that review.
When should a team rerun the classifier comparison?
Ship only after the frozen set passes an agreed semantic threshold and every accepted or rejected attempt appears in the tenant ledger. Then watch schema rejection rate, reviewed-tag agreement, p50 and p95 duration by region, attempts per accepted item, usage units per accepted item, and review-queue rate. Break each measure down by tenant and schema version; global averages conceal the workload this design is meant to expose.
Re-run the set when the taxonomy, prompt, model identifier, provider configuration, or adapter changes. Compare candidates blind where practical, because product labels create expectations before reviewers inspect evidence. Keep the old result beside the new one rather than overwriting it.
The selection rule remains deliberately plain: choose the candidate that clears the catalog's quality and regional constraints with complete per-tenant attribution. If two clear them, prefer the configuration with less operational complexity. If none clear them, improve the taxonomy, examples, or review path before shopping for another logo.
Top comments (0)