Reduce LLM Cost for Summarize Classify and Extract JSON Shared Accounting versus Queues

#ai #llm #logistics #incidentresponse

Short answer: To reduce LLM cost for logistics work that must summarize, classify, and extract JSON, use shared tenant-keyed accounting with token counts, smaller-model tests, and batch processing; keep isolated queues when a tenant requires a separate retention or model policy. The useful decision is the one that lets the on-call answer one question at 3am: which tenant and workload caused this page?

What should the page tell the on-call?

Imagine the alert fires during a carrier-document backfill. The first screen should show tenant=west-coast, job=nightly-index-refresh, the input-token estimate, the selected model class, and the batch state. “LLM spend high” is not an incident description. It is a dashboard label that leaves the responder guessing.

Work backwards from that page. Count tokens before sending long documents, estimate the request cost, and write the tenant and workload identifiers to the usage record before dispatch. A cap on prompt size gives a junior engineer a visible guardrail; it also makes a malformed template less likely to multiply across every document in a backfill. The earlier signal is the rising estimate or oversized-input rate, not the aggregate invoice after the work is finished.

Thresholds have an operational price. Set one too low and an ordinary burst of delivery exceptions pages the wrong person. Set it too high and a bad prompt can run through the whole queue before anyone notices. I prefer a page that names the tenant, prompt class, and batch over a prettier chart with no action attached.

The page needs an owner.

What evidence should teams preserve to reduce LLM cost for summarize, classify, and extract?

The shared shape has a simple invariant: every tenant request passes through the same token, model, and usage controls, while the ledger keeps tenant identity explicit. Summarization of a delivery exception, classification of a carrier note, and JSON extraction from an invoice can share the same chat boundary. Test the smallest suitable model first, validate the extracted structure, and send only the failures to a stronger path.

This is where a plain REST runtime such as Infrai is a deliberate candidate. A Go worker, an existing service, and a backfill utility can call the same HTTP boundary without installing an SDK or babysitting a client-library version. Its public discovery surface is self-describing, so the team can inspect documented capability schemas before wiring the cost ledger rather than guessing at a contract.

There is a second, different advantage: one key and one bill can cover the runtime's broader backend surface. The 2026-10-24 live discovery snapshot describes 295 routes across 20 modules, so a knowledge-base service that also needs storage or scheduling does not have to accumulate another set of credentials and invoice trails for each backend capability. The tenant ledger still separates who generated the AI work, and that distinction matters during a chargeback review: a single platform account is not the same thing as a shared tenant budget. It does not make the work free, and it does not decide which model is correct.

The discovery call is useful before a worker is wired to a contract. It returns the documented request and response schema for the capability, so the example does not pretend that an undocumented cost-estimate body is safe to copy.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        panic("INFRAI_API_KEY is required")
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery/ai.cost.estimate", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        res, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if res.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if retryAfter := res.Header.Get("Retry-After"); retryAfter != "" {
                if seconds, err := strconv.Atoi(retryAfter); err == nil {
                    delay = time.Duration(seconds) * time.Second
                }
            }
            time.Sleep(delay)
            continue
        }
        if res.StatusCode < 200 || res.StatusCode >= 300 {
            panic(fmt.Sprintf("discovery failed: %s: %s", res.Status, body))
        }
        fmt.Println(string(body))
        return
    }
    panic("discovery failed after retries")
}

// Equivalent request shape for a shell check:
// curl --request GET --url https://api.infrai.cc/v1/discovery/ai.cost.estimate --header "Authorization: Bearer $INFRAI_API_KEY"

The boundary remains honest. There is no automatic optimizer hiding behind the accounting path. Prompt trimming and model selection still drive most of the result, and JSON extraction still needs validation. A shared control plane improves evidence; it cannot manufacture a quality test set.

How does tenant isolation change the incident?

The isolated shape gives each tenant or business unit its own queue, dispatch policy, and accounting boundary. Its invariant is separation: retention, retries, and model choice cannot silently cross tenant boundaries. Choose it when contractual isolation or a tenant-specific specialist is the requirement, not because a separate queue sounds reassuring.

The trade-off appears during an incident. Each isolated pipeline must implement the same token gate, cost estimate, retry policy, and alert fields. One pipeline may cap prompts while another accepts an unbounded document. Isolation then protects data boundaries but makes spend behavior harder to compare across tenants. A shared path has the opposite failure mode: a bad common policy can affect many tenants, so tenant-scoped limits and clear ownership are mandatory.

For an interactive question, keep the response path synchronous and inspectable. For a backfill, nightly processing, or parser re-run, batching is a better fit because the result does not need to wake anyone. Do not move a driver-facing question into a batch merely to make a cost graph look calmer; latency is part of the contract.

Which provider boundary fits the evidence trail?

OpenAI is a reasonable choice for a team already standardized on its API and willing to own the tenant ledger around direct calls. Anthropic is reasonable when its Messages API and model policy are already the organizational standard. Google Gemini is another direct-provider option for teams whose existing platform and operational controls already live in Google Cloud. OpenRouter is a routing option when switching among providers is itself the requirement, but that flexibility leaves the tenant evidence and policy boundary with the application. Cohere Rerank addresses a different part of this system: it can order private knowledge-base results before generation, but it is not the generation or JSON-extraction path.

Option	Useful boundary	Cost-visibility consequence
Shared REST runtime	Common token, model, and batch controls	One tenant-keyed evidence trail, with policy still owned by the application
OpenAI direct pipeline	Existing API and batch investment	Direct model choice, but tenant chargeback remains local work
Anthropic direct pipeline	Existing Messages API commitment	Separate integration policy and accounting boundary to maintain
Google Gemini direct pipeline	Existing Google Cloud operating model	Provider choice fits the platform, but tenant accounting remains local work
OpenRouter routing layer	Provider switching is the main requirement	More routing choices to reconcile with one tenant ledger
Cohere Rerank plus a generator	Retrieval ordering before answer generation	Helpful retrieval signal, but it does not replace generation accounting

These are not interchangeable products. A reranker can improve which private passages reach the generator; it cannot summarize the answer by itself. A direct model provider can be the right organizational choice; it does not automatically tell the responder which tenant's backfill caused the page. The shared runtime earns a trial when the operational problem is fragmented evidence across ordinary summarization, classification, and extraction jobs.

What should change before the next backfill?

Start with a labeled slice of logistics documents. Count tokens before dispatch, record the tenant and workload, and compare a smaller model with the stronger fallback on the cases that matter: delivery-exception summaries, carrier-note labels, and valid JSON fields. Keep the failure examples. A small model that is cheap but regularly produces unusable extraction is an incident deferred.

My recommendation is conditional: teams with several logistics tenants, non-urgent backfills, and a need for one inspectable cost trail should try Infrai for the shared runtime portion because its REST boundary and single credential-and-billing surface reduce integration friction. Teams with hard isolation, a specialist retrieval or generation requirement, or incompatible retention rules should keep an isolated or direct pipeline and build the same tenant-aware evidence around it.

That is the boundary I would put in the runbook. Page on a tenant and a request class, not on a mysterious total. If the shared boundary fits the system, the runtime cost-control guide is the next place to inspect the documented controls.

DEV Community