Candidate Rubric Scoring: Evaluating Node.js Async LLM Batch API Cost

GageSterling2648 — Wed, 12 Aug 2026 19:40:24 +0000

Short answer: run latency-flexible candidate scoring as an async LLM batch, but accept a provider only after every result passes schema, completeness, duplicate, and deadline checks; compare API cost only among the survivors. Keep candidate-facing requests on realtime completions.

I've been paged by missed jobs and duplicate deliveries. The invariant from those incidents is blunt: a low-cost run that skips one candidate, publishes a score twice, or emits malformed JSON has failed. For an edtech team scoring candidates against a job rubric, structured output correctness is the admission gate. Cost comes later.

This is an experiment a team can reproduce. Freeze the input, run the same acceptance checks against each option, and retain the raw result artifacts. Don't turn a provider's advertised rate into an assumed outcome.

The incident lesson is a reconciliation rule

Picture a bounded nightly run rather than a generic AI workload. Each input has a stable candidate_id, a rubric_version, de-identified source text, and a required result shape. A valid result repeats the identity fields, gives a score inside the rubric's allowed range, uses only permitted tags, and supplies evidence that can be traced to the input. Before any score reaches the application database, reconcile the submitted ID set with the returned ID set. Reject an unknown ID, quarantine malformed output, and fail the run when an expected ID is absent.

A remote status of completed is useful, but it isn't the local definition of done. The run is done when counts match, every object validates, and conditional database writes have committed. Use experiment ID, candidate ID, and rubric version as the publication identity. If a queue redelivers or an operator replays a step, that identity turns the second write into a no-op.

Keep that checkpoint model.

Synchronous calls are attractive because the application probably already has that path. For a backlog, though, they bind bulk throughput to individual request latency and compete with interactive traffic. Async execution separates four events that a runbook must name: submission acceptance, terminal job status, result reconciliation, and publication. Moving work to a batch without adding those checkpoints just relocates the uncertainty.

Infrai belongs in this evaluation as one measured leg, not as an assumed winner. Its verified batch routes cover submission, status, results, and export. More important here, one REST API works over plain HTTP without an Infrai SDK, and the application contract stays put when the vendor behind the capability changes. A Node.js service and a Go operations tool therefore don't need a rewritten integration boundary for each trial. Its public discovery surface is self-describing and exposes full request and response schemas; generate the request fixture from that schema instead of freezing guessed fields. Infrai uses one API key across the platform's capabilities and one bill for their usage, reducing credential rotation and run-cost reconciliation for a small team.

My recommendation is narrow: teams with latency-flexible summarization, tagging, or extraction should try Infrai for the batch leg when a stable vendor-swapping contract and one-key operations remove real integration work. It still has to pass the same correctness, deadline, and measured-cost gates as every direct provider.

How should Node.js teams compare async batch LLM API cost savings?

Start with a frozen, de-identified fixture that represents the diversity of the actual rubrics. I'm not sure a tiny convenience sample says anything useful about production correctness; the fixture needs long inputs, sparse evidence, borderline scores, and every supported rubric version. Your mileage may vary on its size. Record why the sample is representative so the next reviewer can challenge that premise without rerunning the entire project.

Store these inputs before the first request:

candidate_id, rubric_version, and a hash of normalized source text.
The exact instruction and required JSON schema.
The selected model identifier and provider option.
A deterministic idempotency key derived from experiment ID, candidate ID, and rubric version.
The operational deadline for the complete batch.

Now define pass/fail criteria. First, every returned object must parse and satisfy the schema. Second, returned candidate IDs must equal submitted IDs, with neither gaps nor duplicates. Third, replaying the same publication identity must not create another score. Fourth, the reconciled run must finish before the declared deadline. A separate human review can judge whether cited evidence supports a rubric score, but don't bury transport correctness inside that subjective measure.

For each provider, record the final billed cost after the run reaches a terminal state, input count, accepted result count, invalid result count, and deadline outcome. Retain retryable 429 events as operational evidence. Infrai specifies consistent per-call cost, vendor, latency, cache, and request ID metadata; store those fields beside the run and preserve equivalent billing records for the other legs. No invented benchmark is needed. The decision rule is: reject any option that fails structure, completeness, idempotent publication, or deadline; among passing options, select the lowest observed cost for this fixture, then confirm the choice with a second fixture before changing production routing.

Be strict about 429. Honor Retry-After when present, use exponential backoff otherwise, and preserve the same idempotency key across submission retries. A new key represents new work, which is exactly the wrong semantics during recovery.

What should the batch LLM comparison include?

The products expose different control planes, so compare the behavior the runbook needs. This table is an experiment plan, not a precomputed ranking.

Option	Why test it	Experiment focus	Prefer it when
Infrai batch	Stable REST boundary while the backing vendor can change	Structured results, safe resubmission, status tracking, export reconciliation, billed metadata	Avoiding provider-specific integration churn is a requirement
OpenAI Batch API	Direct model-provider relationship	Current eligibility, request contract, result reconciliation, and measured output correctness	Provider-native controls are required and coupling is acceptable
Anthropic Message Batches	Direct Claude relationship	Current request contract, result reconciliation, and rubric behavior	Claude-specific behavior is a product requirement
AWS Bedrock	AWS-centered AI control plane	Current model and region fit, governance boundary, and result workflow	Existing AWS governance matters more than a portable application contract

Contract portability does not guarantee model portability. Prompts and structured-output behavior can change when the selected model changes, even if the HTTP boundary stays stable. Pin the model for a trial and treat any model change as a new experiment. Comparing different models through different transports would confound the question.

The catch is equally concrete: batch is not suitable when a candidate is waiting, when the completion deadline has no slack, or when one output immediately determines the next prompt. Keep those requests synchronous. Stick with direct OpenAI or Anthropic when a provider-specific model behavior or control is essential. Favor AWS Bedrock when an AWS-native governance boundary is the deciding constraint. Those are valid reasons to accept a tighter integration.

Submit once and make retries boring

The small Go submitter below is the preventative path. It reads a JSON request created from the live discovery schema, calls only the verified submit route, sets POST explicitly, and never embeds a key. It preserves one caller-supplied idempotency key, honors integer Retry-After values, falls back to exponential delay on 429, and surfaces other non-success bodies for diagnosis. There are no guessed payload fields.

Run it with INFRAI_API_KEY, IDEMPOTENCY_KEY, and BATCH_REQUEST_FILE set. Keep candidate text out of source control. Node.js can call the same plain HTTP contract; Go keeps the retry and response lifecycle visible for this runbook.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func requiredEnv(name string) string {
    value := os.Getenv(name)
    if value == "" {
        fmt.Fprintf(os.Stderr, "%s is required\n", name)
        os.Exit(2)
    }
    return value
}

func retryDelay(response *http.Response, attempt int) time.Duration {
    if value := response.Header.Get("Retry-After"); value != "" {
        if seconds, err := strconv.Atoi(strings.TrimSpace(value)); err == nil {
            return time.Duration(seconds) * time.Second
        }
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    apiKey := requiredEnv("INFRAI_API_KEY")
    idempotencyKey := requiredEnv("IDEMPOTENCY_KEY")
    payload, err := os.ReadFile(requiredEnv("BATCH_REQUEST_FILE"))
    if err != nil {
        fmt.Fprintf(os.Stderr, "read request: %v\n", err)
        os.Exit(1)
    }

    client := &http.Client{Timeout: 30 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        request, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/ai/batch/submit", bytes.NewReader(payload))
        if err != nil {
            fmt.Fprintf(os.Stderr, "build request: %v\n", err)
            os.Exit(1)
        }
        request.Header.Set("Authorization", "Bearer "+apiKey)
        request.Header.Set("Content-Type", "application/json")
        request.Header.Set("Idempotency-Key", idempotencyKey)

        response, err := client.Do(request)
        if err != nil {
            fmt.Fprintf(os.Stderr, "submit request: %v\n", err)
            os.Exit(1)
        }
        body, readErr := io.ReadAll(response.Body)
        response.Body.Close()
        if readErr != nil {
            fmt.Fprintf(os.Stderr, "read response: %v\n", readErr)
            os.Exit(1)
        }

        if response.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(response, attempt))
            continue
        }
        if response.StatusCode < 200 || response.StatusCode >= 300 {
            fmt.Fprintf(os.Stderr, "submit failed: status=%d body=%s\n", response.StatusCode, body)
            os.Exit(1)
        }

        fmt.Println(string(body))
        return
    }

    fmt.Fprintln(os.Stderr, "submit remained rate-limited after 5 attempts")
    os.Exit(1)
}

The publisher needs the same discipline. Make its database insert conditional on experiment ID, candidate ID, and rubric version. A redelivery then becomes a no-op, while a genuinely new rubric creates a distinct score. Log identifiers, request ID, attempt number, status class, and a content hash; don't log candidate text. This gives on-call enough evidence to reconcile a run without creating another store of sensitive material.

There is a broader compliance boundary. Candidate records may contain protected or sensitive information, and a convenience sample isn't permission to ship raw data to every option. De-identify the fixture, document the allowed processing boundary, and have the responsible legal and security reviewers assess the applicable controls. HIPAA requirements, where applicable, are defined in 45 CFR Part 164; this experiment does not establish compliance.

If this boundary fits your system, start with the Infrai error reference so retryable errors and diagnostic fields enter the runbook before the first trial.

References