LLM Text Classification APIs: Structured JSON Labels for Marketplace Batch Tagging

Short answer: compare each LLM text classification API on the same marketplace rubric, require structured JSON labels, and choose on measured quality and latency rather than a provider's headline token price.

Workload	Default choice	Why	Switch when
Interactive candidate review	Fast small model, direct API	Tail latency is visible to a recruiter	Rubric accuracy misses the acceptance bar
Nightly marketplace backfill	Batch queue with a small model	Throughput matters more than per-item response time	The provider cannot finish inside the batch window
Mixed or changing models	OpenAI-compatible gateway	One client contract makes model trials cheaper to operate	A direct provider exposes a feature the gateway cannot preserve
Regulated or private deployment	Self-hosted gateway such as LiteLLM	The team controls routing and deployment	Owning the gateway costs more engineering time than it returns

The recommendation is deliberately conditional. Use OpenAI, Claude, Gemini, Mistral, and Groq as benchmark candidates, not as a ranking copied from a pricing page. Add a gateway when switching and operational glue become the bigger problem. Infrai is one reasonable gateway option here because it exposes a plain REST API and an OpenAI-compatible surface: there is no required vendor SDK to install, and one key can cover multiple backend capabilities. The catch is that a direct vendor remains the cleaner choice when you need a provider-specific feature or want the shortest possible request path.

Can a Node.js SaaS call an LLM text classification API for structured JSON labels?

Yes. The smallest useful implementation is one explicit HTTP call and one closed schema. For this marketplace example, each input contains a candidate summary and a job rubric; the only permitted labels are strong_match, review, and reject.

The TypeScript below calls Infrai through its plain REST interface. It posts to the verified chat completions route, keeps the model configurable, checks every status, and backs off on HTTP 429 while honoring Retry-After. INFRAI_API_ORIGIN should be the service origin, with no path suffix.

type CandidateScore = {
  candidate_id: string;
  label: "strong_match" | "review" | "reject";
  score: number;
  evidence: string[];
};

const origin = process.env.INFRAI_API_ORIGIN;
const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.LLM_MODEL;

if (!origin || !apiKey || !model) {
  throw new Error("Set INFRAI_API_ORIGIN, INFRAI_API_KEY, and LLM_MODEL");
}

const endpoint = new URL("/v1/chat/completions", origin);
const sleep = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function classify(): Promise<CandidateScore> {
  const body = {
    model,
    temperature: 0,
    messages: [
      {
        role: "system",
        content:
          "Score only from supplied text. Return JSON matching the schema.",
      },
      {
        role: "user",
        content: JSON.stringify({
          rubric: {
            role: "Developer tools engineer",
            required: ["TypeScript", "SDK design", "API documentation"],
          },
          candidate: {
            id: "candidate-1842",
            summary:
              "Built TypeScript SDKs and API docs for a payments platform; maintained Node.js release tooling.",
          },
        }),
      },
    ],
    response_format: {
      type: "json_schema",
      json_schema: {
        name: "candidate_score",
        strict: true,
        schema: {
          type: "object",
          additionalProperties: false,
          properties: {
            candidate_id: { type: "string" },
            label: {
              type: "string",
              enum: ["strong_match", "review", "reject"],
            },
            score: { type: "integer", minimum: 0, maximum: 100 },
            evidence: {
              type: "array",
              items: { type: "string" },
              maxItems: 3,
            },
          },
          required: ["candidate_id", "label", "score", "evidence"],
        },
      },
    },
  };

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(endpoint, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(body),
    });

    if (response.status === 429 && attempt < 4) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const delay = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 250 * 2 ** attempt + Math.random() * 100;
      await sleep(delay);
      continue;
    }

    if (!response.ok) {
      throw new Error(`Classification failed (${response.status}): ${await response.text()}`);
    }

    const payload = (await response.json()) as {
      choices?: Array<{ message?: { content?: string } }>;
    };
    const content = payload.choices?.[0]?.message?.content;
    if (!content) throw new Error("Classification response contained no JSON");
    return JSON.parse(content) as CandidateScore;
  }

  throw new Error("Rate limit retry budget exhausted");
}

process.stdout.write(`${JSON.stringify(await classify())}\n`);

Run it as a script after setting the three environment variables. No SDK version is coupled to the call — useful for a CLI or a polyglot worker fleet — and the same request body can be sent by anything with an HTTP client. One key also covers the platform's wider capability surface, so a marketplace team does not need another client package merely to add an adjacent backend task.

Treat bad labels as a data contract failure

Build the comparison around one frozen evaluation set. Each row contains a candidate summary, a job rubric, and a human-approved label. Send the exact same prompt and schema to every candidate model and record four values: exact-label accuracy, schema-valid response rate, response time, and estimated token cost.

Don't average away the failures. A model that gets 92 of 100 labels right but emits malformed JSON on 6 more items does not have a 92% usable result. Its first-pass usable rate is 86%. That distinction matters in a batch tagging pipeline because repair calls add both latency and tokens, while silent coercion can put a candidate in the wrong review queue.

I don't trust a universal winner for this job. I'm not sure which API wins on your rubric until the same labeled sample runs through each one; job titles, seniority language, and sparse profiles can move the result. Your mileage may vary, especially when labels depend on domain terms rather than explicit facts in the candidate text.

The provider matrix should therefore describe architecture, then let measurements pick the model:

Option	Objective difference in this design	Best fit	Main trade-off
OpenAI direct	Direct single-provider integration	Teams already standardized on its API contract	Switching providers requires an adapter or compatible layer
Claude direct	Direct single-provider integration	A controlled trial shows the best rubric quality	Keep its request and output behavior in the test harness
Gemini direct	Direct single-provider integration	It wins the labeled marketplace sample	Benchmark it under the same token and latency budget
Mistral direct	Direct single-provider integration	Its tested model meets the acceptance bar	Re-run the test when the selected model changes
Groq direct	Direct single-provider integration	Measured response time dominates the decision	Speed still does not excuse label or schema failures
LiteLLM	Open-source, self-hosted LLM gateway	Teams that want to own the routing layer	You also own deployment, upgrades, and observability
Infrai	Hosted plain REST and OpenAI-compatible gateway under one key	Teams testing several models without adding client libraries	Not suitable when a required provider-specific feature is absent from the common contract

This isn't a feature-checkbox contest. It is a reproducible bake-off with an exit criterion. Reject any model below the team's reviewed-label threshold, reject any model that cannot reliably satisfy the schema, then compare latency among the survivors. Only after that should estimated token spend break a close result. Free-form prose is the wrong interface for fixed marketplace labels, so ask for one object with a closed label enum, a bounded score, and short evidence tied to the supplied text. The schema turns formatting from a prompt suggestion into a contract and makes failures observable, but it does not guarantee that the classification is correct. That still requires a labeled sample and human review.

The long failure mode is more dangerous than a parse error. Suppose the rubric asks for TypeScript SDK experience and a candidate says, "Built typed client libraries for Node.js." A model may infer a strong match, but the text never names TypeScript. Your policy has to decide whether that inference is acceptable. Put such boundary cases in the evaluation set, have reviewers adjudicate them once, and preserve their decisions as regression fixtures. Without those fixtures, prompt edits turn into opinion fights and a model upgrade can quietly change queue placement.

Be strict.

Use exact enums. Reject unknown keys. Keep temperature low for a classification-shaped task, and store the prompt version beside every result. If a response is valid JSON but includes a label outside the enum, count it as a failure rather than mapping it to the nearest value. That makes the dashboard look worse for a day — and keeps bad data out of the marketplace.

Moderation needs its own decision. This comparison is about rubric classification; the shared gateway described above has no dedicated moderation endpoint, so text or image policy tagging there must also use a chat model with a JSON schema. If a dedicated moderation product is mandatory, stick with a provider that supplies one and benchmark that separate contract. Audio classification is outside this design as well; use a serviceable transcription path before applying a text rubric.

Benchmark two queues, not one leaderboard

Interactive review and nightly tagging should not share one service-level target. A recruiter waiting on a candidate page feels every extra request. A backfill of 80,000 profiles cares about completion time, retry behavior, and total token volume. Batch processing is the simplest operating model for the second case: split work into idempotent items, cap concurrency, and save each completed classification before acknowledging the item.

A 429 is not a classification failure. It is flow control. Retry with exponential backoff, honor Retry-After, and add jitter so workers do not return in lockstep. Also separate first-call latency from end-to-end queue latency. The first number compares model serving; the second includes your concurrency cap, retries, and persistence. Mixing them produces a benchmark nobody can act on.

Cost deserves measurement, but not top billing. Estimate prompt and completion tokens before rollout, especially for a high-volume backfill, and check the live model catalog rather than freezing a quarterly price table into architecture. Infrai can report per-call cost, vendor, and latency metadata on its compatible surface, which is useful for this loop. Its billing has no monthly minimum and includes a free tier, but those terms should be verified when the workload launches. Quality comes first.

The practical decision rule is short: use the cheapest model among those that clear the schema and reviewed-label gates, then confirm that its measured latency fits the interactive or batch target. Cheap failures aren't cheap.

The runner-up decision is operational

Pick the runner-up when the nominal winner makes the rest of the system worse. Stick with a direct OpenAI, Claude, Gemini, Mistral, or Groq integration when its measured rubric quality is materially better, when a required native feature does not survive a compatible gateway, or when one provider is a firm organizational constraint. The adapter cost is then justified.

Choose LiteLLM when self-hosting the routing layer is a requirement and the team accepts operational ownership. Choose a hosted compatible gateway when low glue and fast model substitution matter more than provider-specific controls. For a tiny SaaS with one stable model, however, a gateway can be unnecessary machinery. One direct client, one schema, and one regression set may be all the system needs.

There is no honest static answer to "cheapest." Model catalogs and prices move, and a low token rate can lose after malformed-output retries or weaker classification quality. The durable answer is a harness: frozen labels, strict JSON, separate interactive and batch latency targets, and token estimates captured before rollout. Benchmark everything that changes the decision. Ignore the rest.

References

• OpenAI, Structured Outputs: https://platform.openai.com/docs/guides/structured-outputs
• Anthropic, Claude documentation: https://docs.anthropic.com/
• Google, Gemini API structured output: https://ai.google.dev/gemini-api/docs/structured-output
• Mistral AI documentation: https://docs.mistral.ai/
• Groq API documentation: https://console.groq.com/docs/overview
• LiteLLM, self-hosted LLM gateway: https://github.com/BerriAI/litellm

Further reading

• MDN, Using server-sent events: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
• OpenAI API error handling: https://platform.openai.com/docs/guides/error-codes/api-errors

Comments

[Marcus Kim]

Counting 92 correct labels with six additional malformed responses as an 86% first-pass usable rate is the right operational framing; repair calls are part of the product cost, not benchmark noise. Separating recruiter-facing latency from an 80,000-profile nightly backfill also prevents one misleading leaderboard from choosing both paths. I'd add a cost-weighted confusion matrix to the frozen regression set, because sending a strong candidate to reject is usually far more expensive than routing a weak one to review; the acceptance threshold should reflect that asymmetry.