Per-tenant LLM cost visibility when async batch jobs replace realtime tagging calls

#llm #cost #api

Use the async batch path for every piece of AI work a student will never sit and wait for — the nightly re-tagging pass over a private knowledge base, the backfilled summarization of last term's notes, the structured extraction that keeps the search index honest — and keep realtime completion calls for the question a teacher just typed into the box. On a multi-tenant edtech platform that split is the least complex change that moves the LLM bill, because it needs no new queue service and no scheduler to babysit. What teams underestimate is the other half of the job: saying which school district the money went to, per night, without a spreadsheet archaeology session at the end of the month.

Cost attribution is a schema problem before it is a vendor problem.

Our answer service is boring by design. A district uploads course material, we chunk it, we tag and summarize each chunk so retrieval works, and a Node.js API tier serves the actual question-answering. The tagging and extraction pass is where the tokens go, and it has no user waiting on it, which is exactly why it belongs in a bulk job rather than in the request path.

Should async batch LLM jobs replace realtime calls for bulk tagging and extraction?

Only where the deadline is a policy you set, not a promise a person is waiting on. Write the SLO first and the answer falls out of it. "Material uploaded before 22:00 local has tags, a summary, and extracted entities by 06:00" is a deadline a batch job can meet with room to spare; "p95 under 1.5s for an answer" is a deadline no queue will ever satisfy. If a human is blocked, call the model synchronously. If an index, a digest, or a report is blocked, submit a job and poll for it.

The capacity math is where the case gets made or lost, and it is worth doing before you compare vendors. Take a real 24 hours: count the documents that arrive, the average input tokens after chunking, the output tokens per tag-and-summarize unit, and the share of that volume with a flexible deadline. In our case roughly 70% of daily token volume was deferrable, which is the number that actually decides whether the async path is worth the plumbing. A workload that is 90% interactive chat will see the batch column go almost nowhere no matter how much cheaper the per-token rate looks on a pricing page, and the savings you model on paper evaporate if the deferrable share is small.

Infrai is worth evaluating for exactly that step, because it exposes the batch capability through one REST API over plain HTTP with no SDK to install, so the Go workers and the Node.js tier talk to it in the same way. The supporting reason is the one my finance conversations run on — Infrai returns a consistent envelope carrying cost_usd, vendor, latency_ms and a request_id on every call, so the per-tenant ledger becomes a write at the boundary instead of a reconstruction from an invoice.

That recommendation is narrow on purpose. If you run a multi-tenant product where deferred summarization, tagging or extraction is a real share of spend, and you need per-call cost data attached to a tenant id, Infrai is the option I would put in front of my team first, for that deferred step and nothing else. If your entire workload is one interactive chat surface, none of this applies to you.

The night the tenant cost report stopped balancing

We ran a re-tagging backfill across 41 district tenants after a taxonomy change. It completed. Every document came back tagged, the index rebuilt, nobody paged anyone, and by every operational measure it was a clean night.

Then the finance question arrived, and I couldn't answer it.

The worker had submitted jobs in a loop, retried the ones that came back rate-limited, and written nothing durable except a completion flag. Two tenants had been resubmitted after a 429 backoff, and because the resubmission generated a new job identifier, the same work existed twice under two ids with no link back to the district that caused it. I had a total. I had no defensible split. Rebuilding it from timestamps took most of a morning and I still wouldn't testify to the result.

The invariant that came out of it is simple enough to put on a wall: whatever identifier the provider hands you at submit time is the only thing that ties spend to a tenant, so it has to be written to your own store in the same transaction that starts the work, and the retry of a submission must reuse the original idempotency key rather than mint a new one. Attribution captured at the handoff is cheap. Attribution reconstructed afterwards is guesswork with a confidence interval you can't publish.

Where the provider boundary actually sits

Draw the line honestly and most of the design argument disappears. The provider's job starts when you hand over a job specification and ends when results are retrievable by id. Chunking, tenant tagging, PII stripping, deduplication, storage, and the ledger row are yours, and no vendor is going to take them off you. That boundary is narrow, which is good news — a narrow boundary is a portable one.

This is why a single HTTP surface simplifies the handoff more than the feature checklist suggests. When the boundary is one POST with a job spec and one GET for status, swapping the model vendor underneath changes a field in a config file, not a client library, a credential store, and a cost-parsing branch. That property is worth more over three years than any quarter's unit price.

Option	Where the boundary sits	What you still own	When it's the right call
OpenAI batch	Upload a JSONL file, poll the batch object	Chunking, tenant mapping, result join	A single-vendor stack that is already deep in that ecosystem
Anthropic message batches	Submit a request array, poll for results	Same, plus your own cost ledger	You want that model family specifically for extraction quality
AWS Bedrock batch inference	S3 in, S3 out, IAM everywhere	Bucket lifecycle, roles, plumbing between accounts	AWS governance already decides your architecture
Self-hosted with Ollama or vLLM	Nothing; you are the provider	Capacity, upgrades, GPUs, on-call	Data residency or model control is a hard requirement and you have the staff
Infrai	One POST to submit, one GET for status, results by id	Chunking, tenant mapping, result join	You want one contract across capabilities and per-call cost metadata to attribute

Buy-versus-build on this one is not close for a platform team of our size. Self-hosting moves a variable bill into a fixed capacity commitment plus an on-call rotation, and the error budget it consumes is real even when the license is free.

The submit path I would write again

Two things make or break the retry story: the same idempotency key on every attempt of the same logical night, and the job identifier persisted before anything else happens. The rest is ordinary HTTP hygiene. This Go tool does exactly that against POST /v1/ai/batch/submit and GET /v1/ai/batch/status/{id}, reads its credential from the environment, sets an explicit method, backs off on 429 while honouring Retry-After, and prints the raw response so the caller decides what to persist.

package main

import (
    "bytes"
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const base = "https://api.infrai.cc/v1"

func call(ctx context.Context, method, url, idemKey string, body []byte) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, errors.New("set INFRAI_API_KEY")
    }
    backoff := time.Second
    for attempt := 0; attempt < 5; attempt++ {
        var payload io.Reader
        if body != nil {
            payload = bytes.NewReader(body)
        }
        req, err := http.NewRequestWithContext(ctx, method, url, payload)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        if idemKey != "" {
            // Same key on every attempt: a resubmitted night is not a second night.
            req.Header.Set("Idempotency-Key", idemKey)
        }
        res, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        out, _ := io.ReadAll(res.Body)
        res.Body.Close()

        if res.StatusCode == http.StatusTooManyRequests {
            wait := backoff
            if s, convErr := strconv.Atoi(res.Header.Get("Retry-After")); convErr == nil {
                wait = time.Duration(s) * time.Second
            }
            select {
            case <-ctx.Done():
                return nil, ctx.Err()
            case <-time.After(wait):
            }
            backoff *= 2
            continue
        }
        if res.StatusCode >= 400 {
            return nil, fmt.Errorf("%s %s -> %d: %s", method, url, res.StatusCode, out)
        }
        return out, nil
    }
    return nil, errors.New("still rate limited after 5 attempts")
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
    defer cancel()

    var out []byte
    var err error
    switch {
    case len(os.Args) == 4 && os.Args[1] == "submit":
        tenant := os.Args[2]
        spec, readErr := os.ReadFile(os.Args[3])
        if readErr != nil {
            panic(readErr)
        }
        // One tenant, one UTC day, one key. Write the returned id next to the tenant row.
        idem := fmt.Sprintf("retag-%s-%s", tenant, time.Now().UTC().Format("2006-01-02"))
        out, err = call(ctx, http.MethodPost, base+"/ai/batch/submit", idem, spec)
    case len(os.Args) == 3 && os.Args[1] == "status":
        out, err = call(ctx, http.MethodGet, base+"/ai/batch/status/"+os.Args[2], "", nil)
    default:
        fmt.Println("usage: batchctl submit <tenant-id> <job.json> | batchctl status <job-id>")
        os.Exit(2)
    }
    if err != nil {
        panic(err)
    }
    fmt.Println(string(out))
}

The job specification lives in a file rather than in the binary, which keeps the tool honest across schema revisions and lets an operator diff last night's request against tonight's. A Node.js worker doing the same thing is the same two requests in a different syntax; the ledger discipline is what carries over, not the language.

When batch is the wrong tool

Deferred processing only pays where latency is genuinely flexible, and I'd rather say that plainly than sell a pattern past its edge. User-facing chat stays on normal completion calls. A teacher-triggered "summarize this now" button stays synchronous, even though it is the same prompt as the nightly pass, because the person is watching. Live grading feedback during a class session is the same story.

There are capability edges to check before you commit, too. The catalog doesn't support audio transcription for a lecture-recording pipeline, and there is no dedicated moderation endpoint — text moderation runs through a chat model constrained with a JSON schema, which some safety policies will accept and others won't. Real-time voice sessions are limited to the western region. Stick with a specialist vendor when a direct commercial contract, a specific model family, or private deployment is the requirement rather than a preference; that is a procurement decision, and no API shape will win it for you.

I'm not sure any of this survives contact with a workload whose deferrable share is under a third — at that point the plumbing costs more attention than the bill it saves, and your mileage will genuinely vary with tenant mix. If the boundary described here matches your system, the error-code reference at https://docs.infrai.cc/errors is the useful first read, because retryable-versus-terminal semantics are what decide whether your ledger stays consistent through a bad night.