Short answer: batch LLM jobs can be cheaper than realtime candidate scoring, but move to async processing only when the same job rubric survives review and every result meets the hiring deadline.
| Choice | Best fit | Main risk | Decision signal |
|---|---|---|---|
| Realtime request | A recruiter is waiting on one candidate | Paying for immediacy the workflow does not need | A human needs the result during the current interaction |
| Async batch | Many applications can finish behind a deadline | Late results can block the next hiring step | The whole set can wait and accepted-score cost is lower |
| Local queue plus realtime calls | Arrival traffic is bursty, but each result must return soon | Queue and provider retry behavior can duplicate work | Short waits are acceptable, long batch windows are not |
The least complex choice is realtime for an interactive review and async batch for a nightly scoring backlog. Don't migrate because a batch price looks attractive in isolation. Migrate when a representative replay proves that the slower path keeps rubric quality inside your tolerance and lowers the cost of results that your hiring team can actually use.
No magic here.
1. How should quality and latency govern async batch LLM jobs?
Treat the comparison as a deadline-and-quality test, not a feature checklist. In this property-management example, the input is a set of applications for maintenance coordinators, leasing staff, and property managers. Each record is scored against a fixed job rubric. A score that arrives after the recruiter has started interviews has little operational value, while a fast score that cites the wrong evidence can be worse than no score.
Start by freezing the contract. Give both paths the same application text, rubric version, output schema, model class, and validation rules. Compare cost per accepted result, where an accepted result parses, contains every required rubric field, and passes the same evidence checks. Raw request cost leaves out retry spend and unusable output. Total batch cost should include input and output usage, failed-item retries, storage, queue workers, and the engineering time needed to operate the path. The realtime side gets the same accounting treatment.
Then define a deadline before running the trial. “Async” isn't a deadline. For one team it might mean results must exist before the next morning's review; another may need them before a two-hour scheduling block. I'm not sure where your cutoff belongs until the hiring process is timed, and provider completion promises alone can't answer that workflow question. Measure arrival-to-result latency at the median and at a tail percentile your team actually cares about. Also record the share completed before the business deadline. An average can hide the exact late jobs that stall a recruiter.
Quality needs a paired evaluation. Sample outputs from both paths, hide which path produced them, and have a qualified reviewer judge rubric correctness and evidence support. This is the hard part — and the part a pricing spreadsheet skips. If the batch path changes model behavior, truncates source material, or produces more schema failures, its nominal saving may disappear when humans repair the scores.
2. Audit accepted-result cost
The useful denominator is accepted scores, not submitted candidates. Write the calculation down before the test:
accepted unit cost = (inference + retries + storage + workers + review) / accepted scores
That formula prevents a familiar accounting trick: comparing the advertised inference line on one side with the fully loaded production path on the other. It also gives finance and engineering the same object to inspect. Keep currency, measurement window, and usage units beside every input. If a provider changes a rate or a model changes token behavior, rerun the sheet rather than preserving an old percentage in a slide.
Use three cohorts: short applications, long applications, and documents near your input limit. Property-management roles can produce very different evidence density; a concise leasing application and a long maintenance history should not be averaged into one synthetic “typical” record. Report accepted unit cost for each cohort and for the weighted production mix. Your mileage may vary as that mix changes.
One warning deserves its own paragraph.
Don't count a rejected score as cheap.
3. Observe deadlines, retries, and queue age
Batching trades immediate response for scheduling freedom. That is useful only if the product can expose the wait honestly. Store a job identifier, rubric version, candidate identifier, submission time, deadline, and status. Make submission idempotent so a worker restart cannot score the same candidate twice without detection. The result writer should also be idempotent; duplicate delivery is a normal condition to design for in asynchronous systems, not a reason to silently overwrite a newer rubric result.
Watch the queue as a business process. Useful signals include oldest-job age, completion-before-deadline ratio, retry count, invalid-output ratio, and review overrides. A queue with low average latency can still be unhealthy if a small old tail never clears. Alert on age relative to the hiring deadline, not just worker CPU or queue depth.
The local-queue option is the runner-up when jobs may wait for minutes but not for a provider's bulk completion window. It smooths bursts and gives you application-level backpressure while retaining per-request completion. The catch is config bloat: another queue, retry policy, dead-letter path, and dashboard can cost more attention than the workload deserves. I benchmark the end-to-end path because a clean API call surrounded by five moving parts is not clean DX.
4. Implement a paired replay in TypeScript
The harness below stays vendor-neutral. It compares two adapters under one rubric and records the fields needed for a decision. The sample thresholds are policy inputs, not industry benchmarks; set them from your own hiring process and reviewed dataset.
type Candidate = {
id: string;
application: string;
};
type Rubric = {
version: string;
criteria: readonly string[];
};
type Score = {
candidateId: string;
rubricVersion: string;
ratings: Record<string, number>;
evidence: Record<string, string>;
};
type RunResult = {
score?: Score;
inputUnits: number;
outputUnits: number;
inferenceCost: number;
retryCost: number;
submittedAt: number;
completedAt: number;
};
interface ScoringPath {
run(candidates: readonly Candidate[], rubric: Rubric): Promise<RunResult[]>;
}
function isAccepted(result: RunResult, rubric: Rubric): boolean {
const score = result.score;
if (!score || score.rubricVersion !== rubric.version) return false;
return rubric.criteria.every((criterion) =>
Number.isFinite(score.ratings[criterion]) &&
score.evidence[criterion]?.trim().length > 0
);
}
function summarize(
results: readonly RunResult[],
rubric: Rubric,
deadlineMs: number,
operatingCost: number
) {
const accepted = results.filter((result) => isAccepted(result, rubric));
const inferenceAndRetries = results.reduce(
(sum, result) => sum + result.inferenceCost + result.retryCost,
0
);
const onTime = accepted.filter(
(result) => result.completedAt - result.submittedAt <= deadlineMs
);
return {
submitted: results.length,
accepted: accepted.length,
completedBeforeDeadline: onTime.length,
costPerAccepted:
accepted.length === 0
? null
: (inferenceAndRetries + operatingCost) / accepted.length
};
}
Run the adapters on the same frozen sample. Keep human review outside isAccepted: structural validity can be automated, but evidence quality needs a labeled decision from someone authorized to assess the hiring rubric. Record disagreements by criterion. A single aggregate accuracy number won't show that one path consistently weakens, say, safety-procedure evidence for maintenance roles.
This also exposes a subtle deployment trap. If the realtime and batch adapters don't share types and validation, the experiment is testing two integrations rather than two execution modes. Keep the interface boring. One schema. One rubric version. One result sink. The fewer mode-specific knobs, the easier it is to explain a regression and the less glue an SDK or CLI has to hide.
5. Can batch LLM jobs be cheaper than realtime candidate scoring?
Stick with realtime when a recruiter is actively reviewing one application, when a downstream interview scheduler needs the score immediately, or when the rubric changes too often to leave queued work on an older version. It is also the safer default at low volume: the absolute saving may not justify queue operations, reconciliation, delayed deletion, and another access-control surface. Async bulk processing is not suitable when the completion deadline is shorter than the observed tail latency.
Batch is a stronger fit for a large, stable backlog with a clear next-day deadline, versioned rubrics, and a review sample that shows no material quality loss. Even then, deploy by cohort. Start with one role and keep a realtime escape path for urgent candidates. Don't route the same candidate through both paths unless the experiment requires it; duplicate automated assessments complicate audit history and can confuse reviewers.
Candidate data also deserves a narrow collection and retention policy. Keep only fields needed by the rubric, separate raw application access from aggregate metrics, log who can retrieve results, and set deletion behavior before launch. If a workflow handles regulated health information, the HIPAA Security and Privacy Rules are primary material for the compliance review; they are not a generic claim that every candidate record is covered. AWS lists managed foundation-model capabilities through Bedrock, but platform availability does not settle your data classification, hiring obligations, or deadline. Those remain system decisions.
The final decision is intentionally plain: choose the path with the lowest measured accepted-score cost that meets both the quality tolerance and the business deadline. If neither path clears all three gates, change the workflow or rubric before changing the transport.
Top comments (0)