For a marketplace knowledge base, the cheapest model for a text summarization API is the one that answers from the right passages on the first attempt. Compare complete workflow cost, not a provider's isolated unit.
The least complex option is a small provider-neutral adapter around one summarization contract. Keep the marketplace data model, prompt policy, token accounting, and retry rules in your application. Keep provider-specific request and response shapes at the edge.
That design answers the cost question more honestly than a static list of model prices. The cheapest request is the one that fits the input budget, produces a useful answer on the first attempt, and can move providers without a rewrite when quality, latency, or policy changes.
For a one-person SaaS, I would use this decision matrix before choosing a text API:
| Option | Best fit | Main cost risk | Portability |
|---|---|---|---|
| One direct provider client | A prototype with one narrow workflow | Retry and payload logic spreads through the codebase | Low |
| A local adapter with a fixed contract | A production marketplace knowledge base | Adapter maintenance | High |
| A routing layer with measured fallbacks | Multiple providers or strict availability needs | Duplicate requests and hard-to-explain quality changes | Medium to high |
My recommendation is the middle row first. Add routing only after you have measurements from real marketplace questions. It keeps the revenue-per-hour calculation sane: ship the useful workflow weekly, and outsource the undifferentiated transport details to a replaceable boundary.
Measure twice.
A fixed adapter is a poor fit for a prototype that will be deleted next week, and a routing layer is a poor fit when there is no replay set or availability requirement. The extra boundary buys portability, but it also adds tests, telemetry, and another place to debug. That is a real trade-off for a solo founder.
What should the summary contract answer?
A marketplace knowledge base is not a pile of paragraphs. It may contain seller policies, listing rules, delivery notes, and support answers with different freshness dates. A summary that hides scope or invents a policy creates a support problem, even if the request was inexpensive.
Start with a contract that makes the output inspectable. The application should know which tenant and document set produced a summary, how much source text was admitted, and whether the result is ready to show. The model response itself is not your data model.
Here is a deliberately small TypeScript boundary. It has no provider SDK types in the rest of the application.
type SummaryRequest = {
tenantId: string;
question: string;
passages: Array<{
id: string;
text: string;
updatedAt: string;
}>;
maxOutputTokens: number;
};
type SummaryResult = {
answer: string;
sourceIds: string[];
usage: {
inputTokens?: number;
outputTokens?: number;
};
providerLabel: string;
};
interface Summarizer {
summarize(request: SummaryRequest): Promise<SummaryResult>;
}
The optional usage fields matter. Some providers expose token counts and some expose a different accounting vocabulary. Your contract can preserve the information that exists without making every provider pretend to report identical telemetry. The application can then compare observed usage after normalization.
Do not put a long, unbounded knowledge base into one call. Retrieve a small set of passages, preserve their identifiers, and require the answer to cite those identifiers internally before rendering it to a customer. The retrieval stage and the generation stage have different failure modes. A reranker, for example, changes which passages are selected; it does not replace the summarizer. The distinction is documented in the reranking overview from Cohere, which is useful here as a vocabulary reference rather than a product recommendation.
How should a model text summarization API cost comparison work?
Compare complete workflow cost, not a model's advertised unit. For one marketplace question, record these values:
- input tokens admitted after retrieval and truncation;
- output tokens requested and actually returned;
- retrieval and reranking calls;
- retries, timeouts, and fallback calls;
- the share of answers that need a human review or a second request.
A simple estimate is enough for the first dashboard:
type RunCost = {
inputTokens: number;
outputTokens: number;
retrievalCalls: number;
retries: number;
reviewRequired: boolean;
};
function workflowUnits(run: RunCost): number {
const tokenUnits = run.inputTokens + run.outputTokens;
const retryUnits = run.retries * (run.inputTokens + run.outputTokens);
const reviewUnits = run.reviewRequired ? 1 : 0;
return tokenUnits + retryUnits +
run.retrievalCalls * 100 + reviewUnits * 500;
}
The constants above are accounting units, not prices. Replace them with the internal values that reflect your business. The point is to make hidden work visible. A cheaper model that causes twice as many retries can lose on total workflow cost. A more expensive call that returns a grounded answer on the first attempt may win on revenue per hour because it removes support work.
For a fair comparison, replay the same redacted question set through each adapter. Keep the retrieval result fixed. Otherwise you are comparing two changes at once: different evidence and different generation. Label the set by question type, such as “return eligibility” or “seller onboarding,” because a single blended score hides the expensive edge cases.
I initially treated output length as the main lever. That was too narrow. The larger lever is the amount of irrelevant marketplace context admitted before generation. A concise, current passage set makes every compatible provider easier to evaluate.
Which portability boundary survives a provider change?
Portability is not achieved by hiding every difference. It is achieved by naming the differences and testing the behavior that matters. Keep these three layers separate:
- The domain layer decides what a marketplace answer must contain: a direct answer, source IDs, freshness checks, and an escalation state.
- The runtime layer converts that request into a provider call, parses the result, and reports usage and errors.
- The policy layer decides when to retry, shorten the context, switch a provider, or ask for human review.
Provider-specific controls belong in the runtime layer. A provider may expose different sampling settings, structured-output features, limits, or error classes. Do not leak those names into a controller that also handles tenant permissions. The controller should ask for a summary and receive either a valid result or a typed failure.
type SummaryFailure =
| { kind: "invalid-input"; message: string }
| { kind: "temporary"; message: string; retryAfterMs?: number }
| { kind: "limit"; message: string }
| { kind: "policy"; message: string };
type SummaryOutcome =
| { ok: true; value: SummaryResult }
| { ok: false; error: SummaryFailure };
function canRetry(error: SummaryFailure): boolean {
return error.kind === "temporary";
}
This is where standards help. Use JSON Schema or an equivalent validation approach at the adapter boundary, and emit traces with a consistent semantic vocabulary. OpenTelemetry's specification gives you a vendor-neutral place to record spans and attributes; it does not decide your business policy. Keep sensitive passage text out of normal logs. Record IDs, sizes, status, duration, and a redacted correlation key instead.
The practical test is a contract suite. Feed every adapter the same fixtures: an answer with two sources, a passage set with conflicting update dates, an empty retrieval result, a context-limit response, and a transient failure. Assert on the domain result, not the exact prose. Exact text assertions make provider substitution look impossible even when the answer behavior is acceptable.
Where should the Node.js implementation make a choice?
The application-facing function should own timeouts, cancellation, and bounded retries. It should not know how an individual provider names its endpoint or usage fields. A small dependency-free runner is enough to make the policy explicit.
async function runSummary(
summarizer: Summarizer,
request: SummaryRequest,
signal: AbortSignal,
): Promise<SummaryOutcome> {
if (!request.tenantId || !request.question.trim()) {
return { ok: false, error: { kind: "invalid-input", message: "Missing tenant or question" } };
}
const boundedRequest = {
...request,
passages: request.passages.slice(0, 8),
};
try {
const value = await summarizer.summarize(boundedRequest);
if (signal.aborted || !value.answer.trim()) {
return { ok: false, error: { kind: "temporary", message: "No usable answer" } };
}
return { ok: true, value };
} catch {
return { ok: false, error: { kind: "temporary", message: "Summary request failed" } };
}
}
The exact timeout mechanism belongs around the adapter call in the production implementation. The important boundary is visible here: input validation and passage bounds happen before a provider request, while provider parsing stays behind Summarizer. The example intentionally does not claim a universal token limit. You should measure the limits and behavior of the providers you actually support.
A routing layer becomes justified when you can state its rule in one sentence. For example: use the primary adapter for normal requests, use a second adapter after a classified temporary failure, and send low-confidence answers to review. If the rule needs a spreadsheet to explain, it is probably adding operational surface before it has earned its keep.
When is the runner-up the better choice?
A direct client is the better choice for a throwaway prototype or a single internal workflow with no expected provider change. An adapter can be needless ceremony when there is no second implementation and the domain contract is still moving daily. Keep the boundary small so removing it later is possible.
A routing layer is the better choice when availability requirements are explicit, the question set is large enough to support replay tests, and the team can explain how fallback affects answer quality and auditability. It is a poor choice when it silently sends the same request twice, loses source IDs, or makes tenant data cross an unintended boundary.
The marketplace case has one extra rule: freshness outranks fluency. If a policy passage is old or contradictory, return an escalation state instead of polishing the conflict into confident prose. That rule will matter more than a small difference in request cost.
Start with one adapter, one fixed fixture set, and one weekly report containing input size, output size, retries, latency, groundedness checks, and review rate. After four reports, decide if a second adapter solves a measured problem. This keeps the implementation portable without turning portability into a project of its own.
Top comments (0)