Edtech Extraction Operations: 4 Direct Billing, Retry, and Rate-Limit Trade-Offs

For an in-app invoice chatbot, the OpenRouter-versus-direct-OpenAI, Anthropic, or Gemini question is an account-boundary decision; it won't rescue a weak job boundary. A supplier uploads a document, the extraction spinner stalls, and the page says the queue is old. On-call can see a growing backlog but cannot tell whether the provider is slow, requests are being throttled, or workers are duplicating attempts. The invoice may still be processed. It may also be processed three times. Those are different incidents hiding behind one alert.

TL;DR: choose between a routing intermediary and direct model accounts by choosing the failure boundary your team can observe and operate. Keep one internal request contract, assign an idempotency key before any provider call, cap retry time, and record four signals: queue age, attempt count, provider outcome, and estimated usage. This makes OpenRouter, direct OpenAI, direct Anthropic, and direct Gemini replaceable execution paths rather than four application architectures. The least complex option is the one whose billing and rate-limit boundary matches the people carrying the pager.

What should an in-app chatbot compare for direct billing, retries, and rate limits?

The page arrives at 09:12: oldest_invoice_job_age has crossed the service objective. A useful alert includes the tenant, queue, oldest job identifier, and worker deployment. It does not need a model name in its title. Model labels change; the user-visible failure does not.

I've been paged by both missed jobs and duplicate deliveries. That history makes one rule non-negotiable: count durable work separately from network attempts.

Start with the job record. For an edtech finance workflow, that record should distinguish the uploaded document from each extraction attempt. The stable job owns the supplier ID, document hash, schema version, and final state. Each attempt owns a provider-neutral outcome, start and end times, usage estimate, and a correlation ID. Raw invoice text and model output need separate access controls and retention rules because observability should not turn sensitive documents into log payloads.

Then ask four questions in order:

1. Is work arriving faster than it completes? Compare enqueue and terminal-state rates, then inspect oldest age.
2. Are attempts multiplying? Count attempts per stable job and separate scheduled retries from duplicate deliveries.
3. Where does time go? Split queue wait, provider latency, validation, and persistence.
4. Is usage rising without completed invoices? Compare estimated usage with unique terminal jobs, grouped by outcome.

This sequence matters. A latency percentile can look healthy while a throttled queue stops making calls. A request error rate can look bad while bounded retries recover within the job deadline. The page must describe impact; the dashboard can explain mechanism.

One graph can't carry that distinction.

Work backward to the signal that should have fired

A queue-age page is late by design. It reports accumulated user impact. The earlier warning is retry amplification: attempts rise faster than unique completed jobs. One transient failure is ordinary. Repeated calls after the job deadline are waste.

The ratio needs context, not a universal threshold. Invoice batches are lumpy around finance cutoffs, documents vary in complexity, and provider limits may apply at account or project boundaries. Track a rolling numerator of started attempts and a denominator of unique jobs reaching a terminal state. Segment both by extraction schema version and execution path. Never use retries as the denominator; that hides the amplification you are trying to see. The trade-off is a little more ledger storage and metric cardinality in exchange for an incident trail that distinguishes one invoice from its many attempts. I accept that cost because a single aggregate request counter can't prove which supplier jobs reached a terminal state, and it can't tell finance why observed usage rose during a retry storm.

A second early signal is headroom. Measure admitted calls against the configured limit at the same boundary that enforces it. If the boundary is unknown, label it unknown and page on exhausted capacity rather than pretending a guessed quota is authoritative. Configuration belongs beside the metric so an operator can tell whether a deployment changed concurrency or the external boundary changed behavior.

Billing follows the same rule. A provider invoice is a reconciliation source, not the only incident signal. Before dispatch, estimate the input units using the adapter's accounting method; after a successful response, store the returned usage when available. Keep estimated and reported values in separate fields. A discrepancy is diagnosable. One overwritten number is not.

Instrument the attempt boundary once

The application should not decide retry policy inside a chat handler. Put it around a narrow extraction interface, after durable job creation and before provider dispatch. This Go sketch leaves provider-specific parsing inside adapters while keeping deadlines, attempt records, and idempotency in one place.

package extraction

import (
    "context"
    "errors"
    "time"
)

type Result struct {
    SupplierName string
    InvoiceID    string
    TotalMinor   int64
    Currency     string
}

type Usage struct {
    InputUnits  int64
    OutputUnits int64
}

type Extractor interface {
    Extract(ctx context.Context, document []byte, idempotencyKey string) (Result, Usage, error)
}

type AttemptStore interface {
    Begin(ctx context.Context, jobID, attemptID, path string, started time.Time) error
    Finish(ctx context.Context, attemptID, outcome string, usage Usage, ended time.Time) error
}

type RetryClassifier interface {
    Retryable(error) bool
}

var ErrDeadlineBudget = errors.New("retry deadline exhausted")

The idempotency key should come from stable application data, such as the invoice job ID plus extraction schema version. Do not derive it from the attempt number. A redelivery must converge on the same logical operation even when the queue assigns a new delivery identifier. Persist the attempt before the network call; otherwise, a worker can disappear after dispatch and leave no evidence that a chargeable call happened.

Retry only errors the adapter has classified as transient, and stop when the job's time budget cannot accommodate another attempt. Backoff needs jitter so a fleet does not wake at once. The worker must also re-check whether another delivery already completed the stable job before it calls the provider.

func Run(
    ctx context.Context,
    jobID string,
    schemaVersion string,
    path string,
    deadline time.Time,
    extractor Extractor,
    attempts AttemptStore,
    classifier RetryClassifier,
    document []byte,
) (Result, error) {
    key := jobID + ":" + schemaVersion
    waits := []time.Duration{250 * time.Millisecond, time.Second, 3 * time.Second}

    for n := 0; ; n++ {
        if time.Now().After(deadline) {
            return Result{}, ErrDeadlineBudget
        }

        attemptID := key + ":" + time.Now().UTC().Format(time.RFC3339Nano)
        started := time.Now()
        if err := attempts.Begin(ctx, jobID, attemptID, path, started); err != nil {
            return Result{}, err
        }

        result, usage, err := extractor.Extract(ctx, document, key)
        outcome := "success"
        if err != nil {
            outcome = "failed"
        }
        if storeErr := attempts.Finish(ctx, attemptID, outcome, usage, time.Now()); storeErr != nil {
            return Result{}, storeErr
        }
        if err == nil {
            return result, nil
        }
        if !classifier.Retryable(err) || n == len(waits) {
            return Result{}, err
        }

        timer := time.NewTimer(waits[n])
        select {
        case <-ctx.Done():
            timer.Stop()
            return Result{}, ctx.Err()
        case <-timer.C:
        }
    }
}

This is a control-flow example, not a claim that every remote API honors the supplied key. The application still needs its own uniqueness constraint around finalization. Exactly-once delivery is not the premise. Idempotent effects are.

Compare account boundaries, not logo counts

The four named choices answer a narrower question than many comparison tables imply. OpenRouter puts an intermediary account and routing boundary between the application and model providers. Direct OpenAI, direct Anthropic, and direct Gemini integrations place separate provider relationships at the adapter boundary. That structural difference changes where operators reconcile usage, discover throttling, and isolate credentials. It does not remove the need for an internal ledger or retry budget.

Execution path	Account boundary to operate	Portability consequence	Incident question
OpenRouter	One intermediary relationship	Routing can sit outside application code, but the intermediary remains a dependency	Is impact at the intermediary boundary or beyond it?
Direct OpenAI	Separate direct relationship	The adapter must preserve the application's neutral contract	Is this path exhausting its own configured headroom?
Direct Anthropic	Separate direct relationship	The same contract needs an independently tested adapter	Does this path classify outcomes consistently?
Direct Gemini	Separate direct relationship	Schema and usage mapping remain adapter responsibilities	Can jobs move paths without changing finalization semantics?

Do not select from this table by counting setup steps. Select the ownership model. A small team with one pager may prefer one operational boundary. A team that requires independent account isolation may accept three direct integrations and three reconciliation paths. Neither choice is inherently cheaper or easier because those words omit support load, audit requirements, traffic shape, and failure isolation.

Each path has a real limitation. An intermediary is not suitable when policy requires a separate direct provider relationship, and it adds another dependency to investigate during an incident. Direct accounts are a poor fit when the team cannot staff separate credential, limit, adapter-test, and reconciliation work. Supporting all four paths improves optionality but expands the test matrix; portability has an ongoing operations bill.

Provider portability is proven by replay, not by an interface name. Keep a versioned corpus of representative invoices with expected fields and validation rules. Run it against every enabled adapter before deployment. Compare field validity and terminal outcomes, while treating model output as untrusted input. Embeddings and vector search can support supplier lookup or retrieval, but they are separate subsystems; the OpenAI embeddings guide and pgvector project documentation are useful implementation references, not evidence that retrieval belongs in every extraction flow.

Set thresholds with the false-positive bill attached

After instrumentation ships, replace the single queue-age alert with a warning on retry amplification and a page on sustained user impact. The exact values must come from the service objective and observed workload; inventing a universal percentage would create false precision. Record the evaluation window, minimum traffic floor, and required duration in the runbook. Low-volume tenants need an absolute-count guard because one retry can produce a dramatic ratio.

There is a cost to sensitivity. If every short burst pages, on-call learns to distrust the warning and may disable the route that would have caught a real backlog. If the threshold is too loose, duplicated attempts consume capacity until queue age finally trips. Review both false positives and missed early warnings after each incident, then change one threshold at a time.

Quiet isn't the same as healthy.

The closing decision rule is operational: use the execution path whose account boundary the team can meter, limit, test, and reconcile without changing invoice-job semantics. Preserve a stable job record and adapter contract, and switching paths becomes a controlled deployment rather than a rescue operation. Four signals are enough to start. Their labels and ownership matter more than a large dashboard.

Further reading

• OpenAI, "Embeddings guide": https://platform.openai.com/docs/guides/embeddings
• pgvector, "Open-source vector similarity search for Postgres": https://github.com/pgvector/pgvector