Short answer: for an edtech SaaS app, compare OpenRouter, direct OpenAI, and Claude API by the accepted code-review finding, not by a single token quote. The cheapest path is the one that produces that accepted finding with the fewest counted tokens, retries, and human corrections. Measure equivalent review packets across a direct endpoint, a second direct endpoint, and a routing intermediary, then charge every attempt to the same review ID. Keep the finding schema and acceptance test above the provider adapter.
The important unit is not a request. It is an accepted result. A failed attempt that is retried at a second provider has consumed tokens twice, and a response that parses but changes the meaning of severity still costs a reviewer’s time. I treat every log line as stored bytes and every label as cardinality, so I want a small ledger with four numbers per review: input tokens, output tokens, attempts, and accepted result.
The architecture decision record
The invariants are narrow: one review ID follows the request; the submitted diff is bounded and reproducible; each provider response is normalized into one schema; validation happens before persistence; and a fallback is allowed only before an accepted result exists. The failure boundaries are just as important. A timeout is different from a schema violation, which is different from a valid finding that says “no issue.” Treating all three as retryable is how a cost report becomes fiction.
For an edtech product, the review job can contain student-facing code, private tests, and a rubric. Provider portability matters because that packet must remain meaningful after a route changes. The adapter may translate fields, but it must not silently change the rubric or turn a missing finding into an empty array.
| Path | What it gives the ledger | Portability boundary | Main test |
|---|---|---|---|
| Direct OpenAI API | A direct provider attempt with its own request and response records | Provider-specific request semantics stay inside one adapter | Does the normalized result pass the same schema and rubric checks? |
| Direct Claude API | A second direct provider attempt that can be evaluated against the same packet | Prompt and output translation must preserve finding meaning | Do severity, file, line, and rationale retain their definitions? |
| OpenRouter | An intermediary route whose selected upstream must be recorded | Routing policy and upstream identity become part of attribution | Can the ledger explain which attempt produced the accepted result? |
This table is not a ranking. It is a boundary map. Direct calls make the upstream boundary explicit; an intermediary can add route choice; both require an application-owned contract if the surrounding SaaS is meant to remain portable.
The cost equation should stay boring:
review_cost = sum(input_tokens * input_rate + output_tokens * output_rate) + retry_cost + correction_cost
The last two terms are where a token-only comparison fails. retry_cost includes attempts that never reach a useful response. correction_cost represents review work caused by a result that passed transport but failed semantic validation. I would rather omit a noisy label than create a high-cardinality dashboard that cannot answer which route produced accepted findings.
What should a SaaS API compare before choosing a direct or unified fallback?
The portable contract is a document, not a shared URL. Define it before comparing routes. For example, a finding needs a stable identifier, file path, line range, severity, message, and evidence. Define whether a line is one-based, whether a missing line is invalid, and whether “no findings” is an empty list or a separate status. These decisions belong to the SaaS domain.
A provider adapter should emit a candidate result plus metadata, not write directly to the review table. The validator checks shape, allowed values, references to the submitted diff, and a small set of deterministic rules. A malformed candidate is a failed attempt. A valid candidate with no findings is a successful result. That distinction is operationally useful and easy to lose in a generic fallback helper.
Here is the critical path as a deliberately plain request. The endpoint is illustrative; the important part is that the application sends one review packet and receives one normalized contract.
curl -X POST https://runtime.example.test/reviews \\
-H 'Content-Type: application/json' \\
-H 'Idempotency-Key: review-7f2c-attempt-1' \\
--data '{"review_id":"review-7f2c","tenant_id":"school-42","rubric_version":"2026-01","diff_sha":"abc123","files":[{"path":"grader.ts","patch":"@@ -18,3 +18,4 @@"}]}'
The idempotency key is scoped to an attempt, while review_id scopes the business result. If the same attempt is replayed after a network failure, the server needs a way to distinguish a new operation from a duplicate submission. HTTP semantics describe this problem in terms of method safety, idempotence, and retry behavior; the application still has to choose its key scope and persistence rules.
How do you compare the cheapest path without hiding retries?
Build a fixed corpus of review packets. Ten packets are enough to expose basic shape drift; a larger corpus is useful when rubrics contain several severity levels and file layouts. Keep the packet, rubric version, and acceptance test constant. Random prompts are poor accounting instruments because a change in output length can look like a provider difference.
For every attempt, record:
-
review_id, tenant class, corpus case, and rubric version - route label, upstream identity when available, model identifier, and adapter version
- input and output token counts, latency bucket, outcome, and retry reason
- schema validation result and whether a human or deterministic check accepted it
Do not put raw diff text or full model output in every metric label. That is both a cardinality problem and a data-retention problem. Store detailed evidence in a bounded event record; expose low-cardinality dimensions such as route, outcome, and rubric version in metrics. Retain enough detail to audit one review, not every possible string.
Then compare accepted-result cost, not mean request cost. Suppose a direct route uses one attempt for 82 of 100 corpus cases, while a fallback route produces 18 second attempts and four rejected candidates. The useful question is not which first attempt was shorter. It is which route has the lower sum after those 22 extra attempts and the four validation failures. I would report the distribution, not a single winner: p50 and p95 tokens per accepted result, acceptance rate, retry rate, and the number of corrections.
Batch work belongs in a separate lane. A nightly rubric audit can tolerate delayed completion and can be evaluated with a batch-specific ledger, while an editor waiting for a pull-request review cannot. The batch interface documented by OpenAI is evidence that asynchronous processing has different operational semantics; it is not a reason to blend interactive and offline figures into one average.
The rejected option: one universal response passthrough
I first assumed a universal provider response would make fallback trivial. It did the opposite. A passthrough preserves surface differences, so downstream code learns to inspect provider-shaped fields. The second adapter then becomes a migration project, and observability starts counting provider vocabulary instead of business outcomes.
The rejected design is still valid for a narrow use case: an internal experiment where a single engineer owns the caller and raw responses are the deliverable. It is a poor boundary for a multi-tenant code-review product because a tenant should receive the same finding semantics after a route change.
A unified key also solves less than its name suggests. It can reduce secret distribution, but it does not decide which attempts are retryable, who owns token attribution, or what constitutes an accepted finding. Those are application decisions. Keep credentials behind the adapter, and keep the acceptance contract above it.
Decision rule for a portable review runtime
Choose a path only after it survives the same corpus, schema, retry policy, and retention policy. Record provider-specific facts at the edge, then aggregate by application outcomes. If a route is cheaper per request but produces more invalid candidates, it has not answered the SaaS cost question. If it is more expensive per accepted result but sharply reduces correction work, that trade-off belongs in the decision record rather than being hidden in a token chart.
The practical rule is compact: one business review, one ledger, one acceptance contract. Keep route selection replaceable, keep retries bounded, and keep labels small enough that the observability bill remains explainable. That preserves provider portability without pretending that three different APIs have identical behavior.
References
- OpenAI, “Batch API guide”: https://platform.openai.com/docs/guides/batch
- RFC 9110, “HTTP Semantics”: https://www.rfc-editor.org/rfc/rfc9110
- JSON Schema, “The JSON Schema specification”: https://json-schema.org/specification
Top comments (0)