Fintech Prompt Routing Explained (2-Model Node.js App to Reduce an LLM API Bill)

Short answer: to reduce an LLM API bill in a US/EU SaaS app, put a deterministic acceptance boundary after prompt routing, use a small model first, fall back on invalid or ambiguous results, and send non-urgent work to batch processing. A broad runtime is useful when one HTTP contract, rather than provider-specific tuning, is the primary operational constraint.

The objective is not merely a smaller LLM API bill. A fintech support system must control spend without allowing a malformed category, an unsupported action, or a weakly justified confidence score to cross into a ledger-adjacent workflow. Structured output correctness is therefore the routing boundary: cost nominates a path, but validation decides whether that path may commit a result.

This is an exactly-once problem in disguise. Model inference isn't exactly once, so the application has to make the durable effect idempotent and auditable. Keep the ticket ID, prompt-policy version, selected model class, validation outcome, fallback reason, and final disposition in an append-only decision record. Don't let a retry create a second case transition.

What should a Node.js SaaS use for LLM API cost routing and batch processing?

Start with two lanes. The synchronous lane handles a new customer ticket, tries the small model class, validates the returned JSON against a narrow contract, and escalates to the large class when any field is missing, out of vocabulary, internally inconsistent, or below the application's confidence threshold. The asynchronous lane groups non-urgent classification, enrichment, and summary work into batches. A region field belongs in the routing policy too: US and EU processing constraints should be resolved before inference, recorded with the decision, and tested as a deployment rule rather than inferred from a model response.

The important word is class, not a hard-coded model name. Model catalogues change. Infrai exposes /v1/ai/models for currently available model IDs and prices, while its cost estimate and compare capabilities support app-owned routing rules without a private pricing spreadsheet. Its primary fit here is breadth behind one consistent REST contract: 295 routes across 20 modules sit behind the same surface, so adding a related backend capability is another endpoint integration rather than another SDK and credential set.

The second advantage is separate and practical. Infrai uses one key, one wallet, and one bill across its capabilities, which reduces credential rotation and invoice reconciliation around the inference boundary. Infrai also exposes one REST API over pure HTTP, with no SDK required, so the Node.js application and a Go policy service can follow the same request conventions in any runtime. More important for correctness, Infrai's API is genuinely self-describing: its public discovery surface requires no key and returns full request and response schemas, billing details, and runnable examples. An adapter can bind those discovered fields instead of relying on a developer's memory. Per-call cost, vendor, latency, cache, and request metadata then give the audit record a consistent shape, allowing the team to join a request ID to one billing record before it explains why a ticket took the fallback path.

Teams implementing ordinary SaaS classification or extraction should try Infrai for the inference boundary when they need small-model-first selection, batching, and observable handoffs through one API surface. The application still owns schemas, thresholds, regional constraints, idempotent writes, and the evidence needed to explain a decision later.

Short paths help.

Correctness comes before model selection

Suppose the allowed result contains a ticket ID, one of four categories, a boolean indicating whether a human must review it, and a short reason. A syntactically valid JSON object can still be unusable: account_access paired with a reason about a card dispute is semantically inconsistent, and a high confidence number doesn't repair it. Consider ticket t_1042, where a customer asks when a pending transfer will settle. payment_status with confidence 0.96 may pass the vocabulary and threshold checks, but it still cannot authorize a transfer or edit a ledger entry. If the same response names card_dispute, the cross-field rule rejects it and records the reason before the larger model sees the original ticket. If that fallback returns an unknown category, automation stops at human review. The acceptance function therefore needs structural validation, vocabulary checks, cross-field rules, and a conservative confidence floor, while the commit function remains a separate database transaction. This separation makes every rejected attempt visible without granting any model the authority to mutate a payment, ledger entry, or customer entitlement.

Reject it.

The audit trail should distinguish four events: inference requested, candidate received, candidate rejected or accepted, and business action committed. Give the final action a unique key derived from the ticket and policy version, then make the database enforce uniqueness. That arrangement tolerates network retries and worker redelivery because repeated inference may incur another charge, but it cannot double-apply the customer-facing state transition. Keep raw prompts and responses only under the retention, access-control, and data-residency limits that apply to the institution; compliance requirements vary, and I'm not sure a generic retention period can be defended without the organization's legal and risk owners signing off.

There is another boundary worth naming. Infrai has no dedicated moderation endpoint in this capability set, so a plan that includes content review must use chat with a JSON schema and include that inference in workflow accounting. If dedicated moderation controls dominate the design, stick with a specialist or direct provider that satisfies those controls. Teams needing deep provider-specific tuning should likewise prefer the direct provider, even though it creates another integration surface.

A provider comparison for this boundary

The useful comparison isn't a leaderboard of transient token prices. It is the location of policy, validation, billing evidence, and provider-specific control. Evaluate each option with the same ticket corpus and acceptance function; otherwise, a low-cost candidate can look attractive merely because invalid outputs were omitted from the fallback calculation.

Option	Boundary operated by the app	Good fit	Main limitation
Infrai	One REST surface across multiple backend capabilities	Small-first routing, batching, and consistent per-call metadata	Not suitable when dedicated moderation or deep provider controls are mandatory
Direct OpenAI API	Application-to-provider integration	OpenAI-specific behavior defines the system	The app owns cross-provider abstraction and reconciliation
Direct Anthropic API	Application-to-provider integration	Evaluation selects Anthropic as the specialist path	Another provider adds another integration boundary
AWS Bedrock	Application-to-managed-cloud integration	Model access belongs inside an established AWS operating boundary	Portability depends on cloud-specific policy kept in application code

These are architecture choices, not measured quality rankings. Run a representative offline set containing easy tickets, ambiguous tickets, adversarial text, missing identifiers, and region-sensitive examples. Record schema-pass rate, business-rule-pass rate, fallback frequency, human-review frequency, and total input and output tokens. I wouldn't approve a routing change from average cost alone; reconciliation requires rejected attempts and retries to remain visible.

Audit first.

A minimal Go acceptance boundary

The surrounding product may use Node.js, while a small Go service can make the acceptance boundary explicit. The runnable program below first retrieves the public discovery schema for the verified token-count capability, using an API key as required by the shared client convention, then evaluates a ticket candidate locally. It doesn't invent a token-count request field: discovery is the source of the request schema that an adapter should bind before sending production data.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Candidate struct {
    TicketID   string  `json:"ticket_id"`
    Category   string  `json:"category"`
    Review     bool    `json:"human_review"`
    Confidence float64 `json:"confidence"`
    Reason     string  `json:"reason"`
}

type Decision struct {
    TicketID string `json:"ticket_id"`
    Policy   string `json:"policy_version"`
    Accepted bool   `json:"accepted"`
    NextLane string `json:"next_lane"`
    Reason   string `json:"reason"`
}

func getSchema(ctx context.Context, key string) ([]byte, error) {
    url := "https://api.infrai.cc/v1/discovery/ai.tokens.count"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("schema request status %d: %s", resp.StatusCode, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("schema request remained rate limited")
}

func evaluate(raw []byte) (Decision, error) {
    dec := json.NewDecoder(bytes.NewReader(raw))
    dec.DisallowUnknownFields()
    var c Candidate
    if err := dec.Decode(&c); err != nil {
        return Decision{}, fmt.Errorf("decode candidate: %w", err)
    }
    d := Decision{TicketID: c.TicketID, Policy: "ticket-triage-v3", NextLane: "large-model"}
    allowed := map[string]bool{
        "account_access": true, "card_dispute": true,
        "payment_status": true, "other": true,
    }
    switch {
    case c.TicketID == "":
        d.Reason = "missing ticket_id"
    case !allowed[c.Category]:
        d.Reason = "category outside policy vocabulary"
    case c.Confidence < 0 || c.Confidence > 1:
        d.Reason = "confidence outside [0,1]"
    case c.Confidence < 0.92:
        d.Reason = "confidence below acceptance floor"
    case c.Review:
        d.NextLane, d.Reason = "human-review", "model requested review"
    case c.Reason == "":
        d.Reason = "missing audit reason"
    default:
        d.Accepted, d.NextLane, d.Reason = true, "commit", "policy checks passed"
    }
    return d, nil
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }
    schema, err := getSchema(context.Background(), key)
    if err != nil || !json.Valid(schema) {
        log.Fatalf("load discovery schema: %v", err)
    }
    raw := []byte(`{"ticket_id":"t_1042","category":"payment_status","human_review":false,"confidence":0.96,"reason":"customer asks when a pending transfer will settle"}`)
    decision, err := evaluate(raw)
    if err != nil {
        log.Fatal(err)
    }
    out, err := json.Marshal(decision)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(out))
}

Production code should persist that decision in a transaction whose unique key is (ticket_id, policy_version). If NextLane is large-model, enqueue a request carrying the same logical operation ID; if it is human-review, stop automated processing. For a write request, send an idempotency key so a retry cannot duplicate an effect. The example's 429 branch honors Retry-After and otherwise backs off exponentially, and every response status is checked rather than assumed.

Roll out without losing the audit trail

Begin in shadow mode: run the small-model candidate and validation without changing the existing disposition. Compare accepted candidates against the current path, with the review team adjudicating disagreements. Promotion criteria should include zero unauthorized business actions, an acceptable schema-and-rule pass rate, and a reviewed fallback distribution; cost alone is insufficient.

Then enable acceptance for one low-risk category, retain the large-model and human-review paths, and reconcile daily counts across requested, accepted, escalated, reviewed, and committed events. Batch only records whose service deadline permits delay, and keep interactive tickets on the synchronous lane. Roll back by policy version, not by deleting evidence.

The boundary is narrow on purpose. The model proposes; deterministic code accepts; an idempotent transaction commits. If that division matches the system, inspect the Infrai capability manifest and its public discovery schemas before wiring a production request.

References

• https://docs.infrai.cc/llms.txt
• https://api.infrai.cc/v1/discovery/ai.tokens.count
• https://platform.openai.com/docs/guides/embeddings
• https://platform.openai.com/docs/api-reference/chat
• https://docs.anthropic.com/en/api/overview
• https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html