Structured data extraction from e-commerce code reviews fails dangerously when an LLM rate limit returns 429, but a schema-invalid finding that reaches a pull request is worse. It can point at the wrong file, omit the evidence, or turn a suggestion into a blocking issue.
Short answer: put every LLM extraction request through a region-local queue, retry 429 responses with bounded backoff, and acknowledge work only after the returned JSON passes a strict schema gate. Use a batch API for deferrable repository sweeps, not for feedback that a developer is waiting to read.
That order matters. Backoff protects capacity. Validation protects the product.
This build log assumes an existing synchronous extractor. The migration sequence is schema gate first, queued execution second, and regional or batch partitioning last. Each phase can be rolled back without weakening the output contract.
Migration starts with one publishable state
Start from the state transition that matters: a finding is either validated and publishable, or it is not a finding yet. HTTP success, parseable JSON, and a non-empty array are intermediate observations. None earns the right to annotate a pull request.
For this system, acceptance requires the expected schema version, the same change ID that entered the worker, and findings whose file, line, severity, summary, and evidence fields pass runtime checks. That is stricter than TypeScript alone. A compile-time type says nothing about unknown bytes received while the process is running.
Write the failure ledger before the queue. Four terminal labels are enough for the migration: transport_rejected, retry_exhausted, schema_rejected, and accepted. Only the last label can trigger publication. Keeping them distinct stops a dashboard full of “completed” requests from masking unusable output, and it gives schema changes a measurable blast radius without pretending every failure came from rate limiting.
Deploy this gate around the current synchronous call before changing its scheduler. Observe rejection labels without publishing new behavior, compare accepted output with the existing path, and set a rollback condition for unexpected schema rejection. This is the constraint that changes the architecture. Queue throughput becomes secondary to the integrity of that transition.
Phase 1 replaces hopeful parsing
The useful abstraction is an adapter, not a vendor SDK woven through the queue. The worker needs one operation: accept text and return unknown data. Everything after that is local code, which makes the correctness path easy to test and keeps config bloat out of the call site.
type Region = "us" | "eu";
type Severity = "info" | "warning" | "blocking";
type ReviewFinding = {
file: string;
line: number;
severity: Severity;
summary: string;
evidence: string;
};
type ReviewResult = {
schemaVersion: 1;
changeId: string;
findings: ReviewFinding[];
};
type ExtractionRequest = {
changeId: string;
region: Region;
diff: string;
};
type ModelAdapter = {
extract(input: string, signal: AbortSignal): Promise<unknown>;
};
const isString = (value: unknown): value is string =>
typeof value === "string" && value.length > 0;
function isFinding(value: unknown): value is ReviewFinding {
if (typeof value !== "object" || value === null) return false;
const row = value as Record<string, unknown>;
return (
isString(row.file) &&
Number.isInteger(row.line) &&
(row.line as number) >= 1 &&
["info", "warning", "blocking"].includes(String(row.severity)) &&
isString(row.summary) &&
isString(row.evidence)
);
}
function parseReviewResult(value: unknown, changeId: string): ReviewResult {
if (typeof value !== "object" || value === null) {
throw new Error("invalid_result: expected an object");
}
const row = value as Record<string, unknown>;
if (
row.schemaVersion !== 1 ||
row.changeId !== changeId ||
!Array.isArray(row.findings) ||
!row.findings.every(isFinding)
) {
throw new Error("invalid_result: schema or correlation mismatch");
}
return row as ReviewResult;
}
The correlation check is easy to miss. Without it, a syntactically perfect response can still be attached to the wrong change after an asynchronous retry or batch reconciliation. changeId is therefore part of the accepted output, not metadata that disappears at the model boundary.
Consider a hypothetical checkout diff that changes both tax display and inventory reservation. The extractor returns two well-formed findings, but its changeId belongs to the previous commit because asynchronous results were reconciled by arrival order. Every field validator passes. Publishing that object would place accurate-looking evidence on the wrong code, which is precisely why structural correctness includes identity rather than stopping at JSON shape. The worker rejects the whole object, records schema_rejected, and leaves publication untouched; an operator can then compare the submitted manifest, returned ID, and schema version without guessing which queue position moved. This example needs no special queue feature. It needs a contract that survives every scheduler used during the migration.
Keep the ID.
This validator is intentionally boring. For a larger schema, generate runtime validation from one checked-in schema definition so the TypeScript type, test fixtures, and production gate cannot wander apart. The acceptance rule stays the same: unknown enters; a validated domain object leaves.
At this phase, do not add retries or regional routing. Swap one boundary, run the contract fixtures, and retain the old publication switch as the rollback point. A migration that changes parsing, scheduling, and placement in one release leaves no clean explanation for a rejected finding.
How should Node.js migrate LLM extraction through a 429 rate limit queue?
A retry must retain the original item ID, region, attempt count, and deadline. It must not leap into another lane. It also must not hold a concurrency permit while waiting, because sleeping workers turn a transient limit into self-inflicted starvation.
Phase 2 moves only validated requests behind the queue and keeps the publication contract unchanged. Run the synchronous and queued schedulers against the same fixed fixtures before shifting live work. The comparison target is accepted findings by changeId, not merely an equal count of successful requests.
Here is the scheduling core. The adapter can surface a typed rate-limit error from whatever transport it uses; the queue does not need to know a provider route or response shape.
class RateLimitError extends Error {
constructor(readonly retryAfterMs?: number) {
super("rate_limited");
}
}
const delay = (milliseconds: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
function retryDelayMs(attempt: number, retryAfterMs?: number): number {
if (retryAfterMs !== undefined) return Math.min(retryAfterMs, 30_000);
const ceiling = Math.min(500 * 2 ** attempt, 30_000);
return Math.floor(Math.random() * ceiling);
}
async function runExtraction(
request: ExtractionRequest,
adapter: ModelAdapter,
signal: AbortSignal,
): Promise<ReviewResult> {
const maxAttempts = 5;
for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
try {
const raw = await adapter.extract(request.diff, signal);
return parseReviewResult(raw, request.changeId);
} catch (error) {
const canRetry = error instanceof RateLimitError && attempt < maxAttempts - 1;
if (!canRetry) throw error;
await delay(retryDelayMs(attempt, error.retryAfterMs));
}
}
throw new Error("retry_budget_exhausted");
}
Five attempts and a 30-second cap are example policy values, not universal tuning advice. Benchmark the actual workload. Record queue wait, active time, attempt count, validation rejection count, and end-to-end latency by lane; averages alone conceal the burst that users feel. I care more about the p95 queue wait and the invalid-output rate than a pretty requests-per-second number, because those two measurements expose both overload and correctness loss.
Notice what the code does not retry: schema rejection. Blindly replaying the same invalid extraction spends capacity without changing the contract or input. Route that outcome to a bounded repair policy only if the repair prompt and acceptance criteria are separately tested. Otherwise, fail explicitly and preserve the raw response under the applicable regional retention policy for diagnosis.
No magic.
Where does identity live during US, EU, and batch handoffs?
Treat admission, execution, and acceptance as separate states. Admission decides whether a code diff belongs in the US lane, the EU lane, or a deferred lane. Execution owns concurrency and 429 recovery. Acceptance applies the same JSON contract everywhere.
Add these partitions in Phase 3. The safe migration key is (region, changeId, schemaVersion); it remains stable while the scheduling mechanism changes, so results from a drain period cannot be confused with newer work.
| Lane | Work | Scheduling rule | Completion rule |
|---|---|---|---|
| US interactive | A developer's current change | Low concurrency, short bounded retry | Validated finding or explicit failure |
| EU interactive | A developer's current change | Independent capacity and retry state | Validated finding or explicit failure |
| Deferred batch | Repository or backlog sweep | Bulk submission outside the interactive path | Every item reconciled by stable ID |
Region selection must happen before enqueueing. Don't send a rejected EU item through a US worker just because that queue happens to be shorter; operational convenience is not a data-location policy. Keep payload storage, logs, retry metadata, and result handling inside the same regional boundary. The exact legal and deployment requirements vary, and I'm not sure a generic architecture diagram can settle them. A documented data-flow review can.
Batch is appropriate when the caller does not need a result during the code-review interaction: a nightly repository sweep, migration audit, or backfill can trade latency for smoother admission. Give each input a stable changeId, persist the submitted manifest, and reconcile output by ID rather than array position. Partial completion must be visible. One missing item cannot silently turn a 1,000-change sweep into 999 apparent successes.
Stick with the interactive lanes when a developer is waiting, when a finding blocks a merge, or when the diff may become stale before a delayed result returns. The catch is operational duplication: live queues and batch jobs need the same schema gate, fixtures, regional rules, and observability. If the batch path has a second parser, the two paths will eventually disagree about what “valid” means.
Batch is also unsuitable when the upstream system cannot reconcile asynchronous IDs safely. Fix that ownership boundary first. Faster bulk submission won't repair ambiguous identity.
Rollout gates come before adaptive admission
The first change would be adaptive admission per region, driven by observed throttling and queue delay rather than a single hard-coded concurrency value. I would keep a hard ceiling as a safety rail, reduce permits when 429 responses rise, and restore them gradually. The controller needs dampening; an eager increase/decrease loop can oscillate and manufacture bursts.
Next, I would version the output contract and build a fixed evaluation set from representative e-commerce changes: price calculation, inventory reservation, tax display, and checkout state transitions. A deployment should pass exact structural checks before traffic moves. Semantic review quality needs a separate rubric, because valid JSON can still contain a weak finding. Transport success, structural correctness, and review usefulness are three different signals.
There are limits. A queue cannot create provider capacity, schema validation cannot prove that a finding is true, and regional lanes do not by themselves establish compliance. For tiny, low-volume repositories, three durable queues may be needless machinery; an in-process limiter plus strict validation can be the better choice. For high-volume or regulated work, durable state and explicit regional ownership justify their operational weight. That's the decision line I would use.
The final architecture is deliberately plain: classify, enqueue, execute, validate, publish. Migrate one verb at a time. Every extra layer has to improve one of them and show it in a benchmark. If it doesn't, delete the config.
Top comments (0)