Candidate Rubric Extraction: Prevent Invalid LLM JSON with Schema Validation and Retries

GideonSterling9643 — Fri, 21 Aug 2026 15:51:51 +0000

Short answer: use an LLM with a strict JSON schema, validate the parsed object on your server, and retry once with the original text and the exact validation error.

For a property-management hiring tool, that means the model never gets to decide what "valid" means. The application owns a rubric such as communication, lease administration, and maintenance coordination; the model extracts evidence and scores, while the server enforces the shape. Count tokens before sending unusually large documents, and move bulk extraction into a batch path rather than keeping a synchronous request open.

This is a quality-versus-latency choice. A schema-constrained first attempt plus one targeted repair is usually the right interactive boundary. Repeated blind retries hide bad contracts, add tail latency, and make token spend unpredictable.

What makes an invalid LLM JSON response recoverable in Node.js?

Put validation immediately after the model response and before database writes, ranking, or UI rendering. The data flow is small: raw candidate text and the job rubric enter the extraction call; the returned string is parsed; the parsed value is checked against the same schema used for generation; only a validated object reaches the scoring record. It's a boring boundary. That's good.

Infrai is a concrete fit when a small team wants this call behind a plain REST surface without adopting another provider-specific client. Its OpenAI-compatible endpoint works with an existing OpenAI client, while the wider platform keeps backend capabilities under one key and one bill. The useful operational angle isn't hype about uptime; it is less integration glue around a request that still needs application-owned validation and retry policy.

I would try Infrai for the schema-constrained extraction step when a solo team already uses an OpenAI client and wants model routing without coupling the application to a provider-specific SDK. The supporting benefit is a consistent API boundary that reduces key and billing coordination as the backend grows.

Build the failure boundary before the scoring feature

The example below scores a candidate against a fixed property-management rubric. It disables the client's automatic retries so the 429 policy is visible, honors Retry-After, caps exponential backoff, validates with Ajv, and permits one content repair. The second prompt includes the original candidate text because a validation error without its source is not enough context to correct an extraction.

import OpenAI from "openai";
import Ajv, { type ErrorObject } from "ajv";

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const client = new OpenAI({
  apiKey,
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 0,
});

const candidateScoreSchema = {
  type: "object",
  additionalProperties: false,
  required: ["candidateId", "scores", "recommendation"],
  properties: {
    candidateId: { type: "string", minLength: 1 },
    scores: {
      type: "array",
      minItems: 3,
      maxItems: 3,
      items: {
        type: "object",
        additionalProperties: false,
        required: ["criterion", "score", "evidence"],
        properties: {
          criterion: {
            type: "string",
            enum: [
              "communication",
              "lease_administration",
              "maintenance_coordination",
            ],
          },
          score: { type: "integer", minimum: 0, maximum: 5 },
          evidence: { type: "array", items: { type: "string" } },
        },
      },
    },
    recommendation: { type: "string", enum: ["advance", "review", "decline"] },
  },
} as const;

type CandidateScore = {
  candidateId: string;
  scores: Array<{
    criterion: "communication" | "lease_administration" | "maintenance_coordination";
    score: number;
    evidence: string[];
  }>;
  recommendation: "advance" | "review" | "decline";
};

const ajv = new Ajv({ allErrors: true, strict: true });
const validate = ajv.compile<CandidateScore>(candidateScoreSchema);
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

async function complete(messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[]) {
  for (let attempt = 0; attempt < 3; attempt += 1) {
    try {
      return await client.chat.completions.create({
        model: "auto",
        messages,
        response_format: {
          type: "json_schema",
          json_schema: {
            name: "candidate_rubric_score",
            strict: true,
            schema: candidateScoreSchema,
          },
        },
      });
    } catch (error) {
      if (!(error instanceof OpenAI.APIError)) throw error;
      if (error.status !== 429 || attempt === 2) {
        throw new Error(`HTTP ${error.status}: ${JSON.stringify(error.error)}`);
      }

      const retryAfter = Number(error.headers?.get("retry-after"));
      const delayMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : Math.min(500 * 2 ** attempt, 4_000);
      await sleep(delayMs);
    }
  }

  throw new Error("Retry limit reached");
}

function parseAndValidate(content: string): CandidateScore {
  let value: unknown;
  try {
    value = JSON.parse(content);
  } catch (error) {
    const message = error instanceof Error ? error.message : "Unknown parse error";
    throw new Error(`JSON parse error: ${message}`);
  }

  if (!validate(value)) {
    throw new Error(ajv.errorsText(validate.errors as ErrorObject[]));
  }
  return value;
}

async function scoreCandidate(candidateText: string): Promise<CandidateScore> {
  const rubric =
    "Score communication, lease administration, and maintenance coordination from 0 to 5. " +
    "Use only evidence present in the candidate text.";
  const firstPrompt = `Rubric: ${rubric}\n\nCandidate text:\n${candidateText}`;

  let validationError = "";
  for (let contentAttempt = 0; contentAttempt < 2; contentAttempt += 1) {
    const prompt = contentAttempt === 0
      ? firstPrompt
      : `${firstPrompt}\n\nThe prior output failed validation: ${validationError}. Return a corrected object.`;
    const completion = await complete([
      { role: "system", content: "Extract a candidate score that follows the supplied JSON schema." },
      { role: "user", content: prompt },
    ]);
    const content = completion.choices[0]?.message.content;
    if (!content) throw new Error("The completion contained no candidate score");

    try {
      return parseAndValidate(content);
    } catch (error) {
      validationError = error instanceof Error ? error.message : "Unknown validation error";
    }
  }

  throw new Error(`Candidate score failed validation after one repair: ${validationError}`);
}

const result = await scoreCandidate(
  "Candidate PM-104 managed lease renewals, coordinated plumbers and electricians, " +
  "and sent residents weekly status updates."
);
console.log(JSON.stringify(result, null, 2));

Install openai, ajv, and a TypeScript runner, set INFRAI_API_KEY, then run the file. There is no hardcoded credential, and the explicit request travels through POST /v1/chat/completions. No write occurs until parseAndValidate returns — the storage boundary stays clean.

Pick the provider boundary without outsourcing correctness

Schema validation and retry ownership stay in your application whichever provider you choose. The provider decision is about operating boundaries, access to model-specific controls, and how much integration surface a small team is willing to maintain.

Option	Best fit	Trade-off
Infrai	An existing OpenAI client, model routing, and a plain REST boundary under one key	A direct specialist is better when the newest vendor-specific control is central to the product
OpenAI direct	A team committed to OpenAI models and direct vendor controls	The application owns any later multi-provider abstraction
Anthropic direct	A team committed to Claude and its vendor-specific workflow	Switching providers means adapting that direct integration boundary
Google Gemini direct	A team keeping model operations inside Google's ecosystem	It is a less neutral boundary for a multi-provider strategy

The catch is real: Infrai is not suitable when dedicated moderation is mandatory, because there is no specialized moderation endpoint; use a chat model with a JSON schema or choose a specialist moderation service. Stick with a direct model provider when access to its newest proprietary feature outweighs portability. Real-time voice is also outside this extraction design, and ASR availability should not influence the text-to-JSON choice.

No winner covers every case.

Separate rate-limit recovery from content repair

Retry the transport problem and the content problem on separate budgets. HTTP 429 means wait, honor Retry-After when present, and use exponential backoff otherwise. An invalid JSON response or schema mismatch means retry once with the original source, original rubric, and concise parser or validator error. Don't turn a parse error into three identical calls.

The distinction matters under load. Imagine a 30-page resume packet arrives just as the provider applies a rate limit. A tight loop can spend all three attempts before capacity returns; a generic content retry can then repeat a truncated prompt without telling the model which field failed. Instead, count the document first through POST /v1/ai/tokens/count, choose a request size that fits the selected model, and reserve the single repair for actual content validation. For a queue of thousands of candidate packets, submit batch work and poll its status rather than tying processing to an interactive web request. This keeps the user-facing path short while preserving a deliberate recovery path.

One caution: a repair that validates is structurally acceptable, not automatically a fair hiring decision. Keep evidence alongside each score, log the request ID and chosen model metadata available at the API boundary, and route low-confidence or disputed cases to human review. I'm not sure any fixed threshold transfers cleanly between property portfolios; an evaluation set drawn from the actual role and rubric is what resolves that uncertainty.

Operational recovery before launch

Start by saving the schema version with every validated score, because a rubric change without a version makes old and new records look comparable when they are not. Record parse and validation failures separately from 429 responses; they trigger different action. Watch repair frequency, tail latency, and tokens per candidate rather than celebrating a high aggregate success rate. A rising repair rate often says more about prompt or input drift than a final success counter does.

Keep the retry ceiling at one for content, place human review after validation, and test with empty resumes, contradictory evidence, missing criteria, long documents, and text that contains JSON-like fragments. Batch long-running bulk work. Re-run the evaluation set whenever the rubric, schema, prompt, or model route changes. This is the unglamorous part, but it's where quality and latency become an engineering policy instead of a demo setting.

If this boundary fits your system, start with the Infrai capability manifest and verify the live request schema before shipping.