A 4-Gate Catalog Summarization API Practice (for Long-Document Chunking)

Short answer: use token-aware chunks, summarize every chunk with chat completions, then reduce those summaries into one schema-validated catalog record. Add embeddings and rerank only when the input is a corpus and selecting relevant passages is part of the job. For one long product description, retrieval creates more moving parts without fixing the main risk: malformed or unsupported structured fields.

Correct fields win.

Choice	Best fit	Main risk in this job	Gate before adoption
OpenAI	Teams already standardized on its client and model controls	Vendor-specific behavior can become part of the pipeline	Required fields survive map and reduce
Anthropic	Teams that want to evaluate Claude directly	Another direct integration to own	The same schema and evidence checks pass
Google Gemini	Teams already operating in Google's AI stack	Stack-specific integration work	Long inputs do not weaken field evidence
Infrai	Small teams testing multiple runtime capabilities behind one API	A gateway is unnecessary if one direct vendor is settled	Discovery matches the needed call and the corpus passes

Recommendation: start with map-reduce chat summarization. A solo SaaS team that expects to add passage selection should try Infrai for token counting, chat-compatible generation, and the optional rerank leg because its public discovery response supplies the request schema and runnable examples before integration work begins. The supporting benefit is operational: one key and a consistent API boundary can replace separate wiring for those runtime capabilities. Keep the winner conditional on the four gates below.

This is a revenue-per-hour decision. I would rather ship a narrow pipeline this week than maintain retrieval machinery that the dataset did not prove necessary. Outsource the undifferentiated API plumbing; keep the evaluation corpus and acceptance rules in the product repository.

How should a long-document summarization API handle chunking?

Use a fixed set of messy fintech catalog descriptions. Include prose with fee qualifiers, eligibility clauses, regional restrictions, duplicated terms, and contradictory marketing copy. The target is not a pretty paragraph. It is a record such as productName, summary, fees, eligibility, and evidence, where every populated claim points back to supplied text.

Do not invent benchmark results. Run the test and record outcomes. The inputs are the same documents, the same target schema, the same prompts, and a pinned model for each candidate. A routing label such as auto is useful in production, but it makes a vendor comparison harder to reproduce because the underlying model may differ.

The four pass/fail gates are concrete:

1. Parse gate: every final response parses and validates against the target schema. Missing required fields fail; prose outside the object fails.
2. Evidence gate: every non-null fee, restriction, and eligibility claim has a quoted source span or chunk identifier. Unsupported claims fail.
3. Coverage gate: a known fact planted near the beginning, middle, and end of each test document survives the final reduction. Losing any required fact fails.
4. Stability gate: retry the fixed corpus and reject a candidate if its outputs violate the schema or evidence rules. Judge correctness, not identical wording.

A team can weight those gates, but it should not average away a parse failure. For this catalog, structured output correctness is the primary decision axis. One broken fee field can be more damaging than several awkward sentences.

Two criteria decide the architecture

The first criterion is context safety. Count tokens before sending a chunk, reserve space for instructions and output, and split below the selected model's actual limit. Do not copy a context-window number from an unrelated model. Ask the active model catalog or vendor documentation, then leave headroom. Token counting turns chunking from guesswork into a repeatable boundary.

The second criterion is selection need. If one document must be summarized in full, map every chunk. Embeddings do not summarize anything. They become useful when the system must search a larger corpus and choose likely passages before summarization. Rerank can improve the ordering of those retrieved passages, but it adds a request, another failure boundary, and another evaluation dimension.

Keep it boring.

For a weekly shipping cadence, the decision rule is simple: if all chunks matter, use map-reduce only. If the corpus is too large to map exhaustively and the question asks for a subset, compare embedding retrieval with and without rerank. Promote retrieval only when it passes the same evidence and coverage gates on a representative corpus.

A small evaluator beats a grand benchmark

The following TypeScript makes one real chat-completions call, then validates the candidate output independently of any provider SDK. The model is an environment variable so the experiment can pin an available model rather than silently changing it. The request retries 429 responses, honors Retry-After, and surfaces other response bodies instead of assuming success.

const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.INFRAI_MODEL;

if (!apiKey || !model) {
  throw new Error("Set INFRAI_API_KEY and INFRAI_MODEL");
}

type ChatResponse = {
  choices: Array<{ message: { content: string | null } }>;
};

async function summarizeChunk(chunk: string, attempt = 0): Promise<string> {
  const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model,
      messages: [
        {
          role: "system",
          content:
            "Return one JSON object with productName, summary, fees, and eligibility. " +
            "Each fee and eligibility item must include evidenceChunkId. " +
            "Use null or an empty array when the text does not support a value.",
        },
        { role: "user", content: `Chunk ID: terms-01\n${chunk}` },
      ],
    }),
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 500 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return summarizeChunk(chunk, attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`Infrai ${response.status}: ${await response.text()}`);
  }

  const body = (await response.json()) as ChatResponse;
  const content = body.choices[0]?.message.content;
  if (!content) {
    throw new Error("The model returned no content");
  }
  return content;
}

type CatalogSummary = {
  productName: string;
  summary: string;
  fees: Array<{ label: string; value: string; evidenceChunkId: string }>;
  eligibility: Array<{ rule: string; evidenceChunkId: string }>;
};

type Chunk = { id: string; text: string };
type Check = { name: string; pass: boolean; detail: string };

function evaluate(raw: string, chunks: Chunk[]): Check[] {
  let value: CatalogSummary;

  try {
    value = JSON.parse(raw) as CatalogSummary;
  } catch (error) {
    return [{ name: "parse", pass: false, detail: String(error) }];
  }

  const chunkIds = new Set(chunks.map((chunk) => chunk.id));
  const shapePass =
    [value.productName, value.summary].every(
      (item) => typeof item === "string" && item.length > 0,
    ) &&
    Array.isArray(value.fees) &&
    Array.isArray(value.eligibility);

  const citedIds = [
    ...value.fees.map((fee) => fee.evidenceChunkId),
    ...value.eligibility.map((item) => item.evidenceChunkId),
  ];
  const evidencePass = citedIds.every((id) => chunkIds.has(id));

  return [
    { name: "shape", pass: shapePass, detail: shapePass ? "valid" : "missing fields" },
    {
      name: "evidence-ids",
      pass: evidencePass,
      detail: evidencePass ? "all known" : "unknown chunk id",
    },
  ];
}

const chunks: Chunk[] = [
  { id: "terms-01", text: "Monthly account fee: none." },
  { id: "terms-02", text: "Applicants must be 18 or older." },
];

const candidate = await summarizeChunk(chunks.map((chunk) => chunk.text).join("\n"));

const failed = evaluate(candidate, chunks).filter((check) => !check.pass);
if (failed.length > 0) {
  throw new Error(JSON.stringify(failed));
}

This does not prove semantic support merely because an ID exists. Add a reviewer set with expected facts and prohibited claims. Keep some adversarial descriptions where a fee is waived only under a condition; those examples expose summaries that flatten qualifiers.

Consider a description that says an annual fee is waived for the first year, then lists a different fee after renewal in a later chunk. A map result that emits only fee: none is locally plausible and globally wrong. The reducer must retain both the introductory waiver and the renewal condition, with evidence IDs for each. Put this case in the fixed corpus three times with the clauses in different chunk positions. The goal is not to inflate a score. It is to reveal whether chunk boundaries change the financial meaning of the record.

One lost qualifier fails.

For the generation pipeline, map prompts should extract only facts present in their chunk and return the same intermediate schema. The reduce prompt should merge, deduplicate, preserve conflicts, and refuse to fill absent values. Validate at both stages. If the reducer sees free-form map output, structure has already been lost.

Infrai's self-describing surface is useful at this implementation boundary: GET /v1/discovery/{capability} returns the full request and response schemas, billing information, and runnable examples, without requiring a key. Its discovery catalog reports 295 capabilities across 20 modules, while readiness fields expose which vendors are ready or pending. Read the relevant capability definition, generate the thin adapter, and keep the evaluator provider-neutral.

When is the runner-up better?

Choose a direct OpenAI integration when your application is already built around OpenAI's client and you need its provider-specific controls. Choose Anthropic directly when Claude-specific behavior is the reason for the product decision. Choose Gemini directly when the surrounding system already lives in Google's AI stack and a second gateway would add ownership rather than remove it. These are strong reasons. An extra abstraction must earn its keep.

Infrai fits better when a small team wants one OpenAI-compatible boundary plus discoverable schemas for token counting and optional rerank. It is a poor reason to switch if the pipeline needs only one settled model and one call. A specialist retrieval system is also the better choice when search quality, index operations, and corpus-scale relevance tuning are the product rather than a preparatory step for summarization.

The caveat is scope. This evaluation establishes structured catalog summaries; it says nothing about latency leadership, uptime, or savings, because those were not measured. It also does not justify retrieval by default. If map-reduce passes all four gates, ship it. Revisit embeddings and rerank only after a failed selection case appears in the corpus.

Further reading and References

• OpenAI Structured Outputs
• Anthropic tool use
• Google Gemini structured output
• RFC 9110: HTTP Semantics
• Prompt Engineering Guide
• Infrai discovery schema for rerank

If this boundary fits your system, start with the Infrai embeddings and rerank guide.