Node.js Review Findings: Structured LLM Extraction Under Regional 429 Rate Limits

#node #llm #architecture

Short answer: put each US or EU extraction worker behind a region-local, rate-aware queue; validate every structured finding before completion; and use provider batch processing only for review work whose deadline can absorb asynchronous turnaround. A 429 is an admission-control signal, not permission to multiply requests with eager retries.

For a code-review service, the governing trade-off is quality versus latency. The fast path should return a small, schema-valid set of findings while the diff is still useful to the developer. Deeper analysis can wait. Regional isolation, bounded attempts, and explicit completion states keep overload from turning one delayed review into duplicate comments or a cross-border data leak.

How should a Node.js structured data extraction LLM queue react to a 429?

Stop admitting fresh work to the constrained lane, honor an explicit server retry delay when one is supplied, and otherwise apply capped exponential backoff with jitter. The queue must preserve the original job identity across attempts. It also needs separate concurrency control for each provider, model class, tenant, and processing region because a single global cap hides which quota is actually exhausted.

Pause admission first.

Do not sleep inside a request handler. A handler that accepts a review should persist the job, choose its region from policy, and return a job identifier. A worker leases that job when its lane has capacity. If the upstream response is 429, the worker records the next eligible time and releases the lease; another worker must not race the same job. This distinction matters during a burst: ten blocked workers retrying independently can produce ten new bursts, while a queue-level pause reduces admission before the next attempt.

Backoff alone is incomplete. Read any standard rate-limit metadata that the upstream contract documents, but don't assume every service uses the same headers or quota unit. Some limits count requests, some account for input or output volume, and some are scoped more narrowly than an account. I'm not sure which constraint is active unless the provider documents it or exposes it in response metadata, so the scheduler should treat the observed scope as configuration rather than infer it from one 429.

Keep the retry budget small and explicit. A malformed input, an authentication failure, and a schema validation failure are not congestion, so they should never enter the 429 retry path. Exhausted congestion retries move to a delayed or terminal state with a reason that operations can inspect. No spin loop.

Residency, validation, and the retry feedback loop

The ADR chooses durable, region-partitioned queue lanes with adaptive admission control for interactive reviews, plus an optional asynchronous batch lane for non-urgent repository sweeps. The unit of work is one immutable review job, even when the implementation splits a large diff into model-sized parts. Aggregation belongs to the job state machine, not to a web request.

Four invariants shape the design:

A job's source text and structured findings remain in its assigned US or EU data plane. Routing metadata may be global only when it contains no review content.
Every successful result passes syntactic JSON parsing and application-level schema checks before it is marked complete. A valid JSON object with an unknown file path is still an invalid review result.
One logical job can publish findings once. Retries reuse an idempotency key and publication uses a compare-and-set transition from validated to published.
Interactive and batch work never share an unbounded backlog. The interactive lane gets a latency budget; bulk work yields when that budget is threatened.

The failure boundary is deliberately narrow. The API owns authentication, policy lookup, region selection, and durable enqueue. A regional worker owns upstream admission, extraction, validation, and job state. The publisher owns deduplication and delivery to the code-review surface. This gives each stage a useful failure mode: accepted, waiting, running, retry-scheduled, validated, published, or terminal. running without a lease expiry is forbidden because abandoned work would have no recovery path. I've seen rate-limit handling turn a 429 into repeated pressure when workers treated their own clocks as the source of truth; the state machine avoids that pattern by making the next eligible time shared queue state. It also makes compliance review less vague. Region is a required field chosen before content persistence, logs carry job IDs rather than source snippets, and dead-letter inspection follows the same access policy as the original diff. Redaction is not a substitute for residency: a patch can reveal customer identifiers, internal hostnames, or secrets in context that a generic scrubber misses. Observability should answer operational questions, not merely count errors. Track queue age by region and lane, admitted concurrency, 429 responses by configured quota scope, retry scheduling delay, validation failures by schema version, and publish deduplication outcomes. Do not place prompts, diffs, or raw model output in ordinary metrics labels or logs — they are high-cardinality and may contain sensitive code.

Residency is a hard boundary.

Spend the latency budget by lane

Lane	Best fit	Latency behavior	429 handling	Quality control	Main limitation
Inline request	Tiny internal tools with low traffic	Caller waits for extraction	Handler must wait or fail	Validation blocks the response	Ties web capacity to upstream latency
Durable interactive queue	Pull-request review and user-triggered checks	Bounded by queue age and worker time	Lane pauses and reschedules jobs	Validate before publishing	Requires queue operations and job-status UX
Asynchronous batch	Repository sweeps and offline re-analysis	Completion is intentionally deferred	Submission and collection use their own admission budgets	Validate each item and reconcile the manifest	Not suitable for a developer waiting on a current review

The durable interactive queue is the default because it gives latency policy somewhere to live. The API can reject or degrade work when predicted queue age breaches the product's review window, rather than accepting an unlimited backlog and calling it reliability. For example, a degraded pass may inspect only changed hunks and request a compact finding schema; a later batch pass can examine broader repository context. That is a product choice, not an invisible model tweak, so label the review depth in the result.

Quality controls must be identical across lanes. Define a versioned schema for fields such as file path, line, severity, category, evidence, and suggested action. Constrain enums, reject extra properties where appropriate, and verify semantic references against the submitted diff. If one chunk claims a line that isn't present, quarantine that finding rather than publishing a confident-looking comment. Schema retries also need a separate budget from 429 retries; mixing them makes capacity incidents look like quality incidents.

The catch is operational weight. A durable queue adds leases, reconciliation, regional deployment, and on-call work. Stick with the inline option when requests are rare, callers tolerate upstream latency, source data is non-sensitive, and losing an in-flight request is acceptable. Choose asynchronous batch when freshness has little value and throughput matters more than immediate feedback.

Make every retry a state transition

A Node.js service can implement these same interfaces with its queue library and HTTP client. The control loop below is Python because the important artifact is the state transition: acquire admission, call the abstract extraction adapter, validate, publish once, or reschedule only the congestion case. Provider-specific URLs and headers stay inside the adapter so the worker doesn't invent API contracts.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from random import random
from typing import Protocol


@dataclass(frozen=True)
class ReviewJob:
    job_id: str
    region: str
    attempt: int
    schema_version: str
    diff_ref: str


class RateLimited(Exception):
    def __init__(self, retry_after_seconds: float | None = None):
        self.retry_after_seconds = retry_after_seconds


class ExtractionAdapter(Protocol):
    def extract(self, job: ReviewJob) -> dict: ...


class JobStore(Protocol):
    def load_diff(self, region: str, diff_ref: str) -> str: ...
    def reschedule(self, job_id: str, eligible_at: datetime, reason: str) -> None: ...
    def publish_once(self, job_id: str, findings: list[dict]) -> None: ...
    def finish_terminal(self, job_id: str, reason: str) -> None: ...


def backoff_seconds(attempt: int) -> float:
    cap = 60.0
    base = min(cap, 2.0 ** attempt)
    return base * (0.5 + random())


def run_job(
    job: ReviewJob,
    adapter: ExtractionAdapter,
    store: JobStore,
    max_attempts: int,
) -> None:
    store.load_diff(job.region, job.diff_ref)

    try:
        payload = adapter.extract(job)
    except RateLimited as exc:
        if job.attempt >= max_attempts:
            store.finish_terminal(job.job_id, reason="rate_limit_budget_exhausted")
            return

        delay = (
            exc.retry_after_seconds
            if exc.retry_after_seconds is not None
            else backoff_seconds(job.attempt)
        )
        eligible_at = datetime.now(timezone.utc) + timedelta(seconds=delay)
        store.reschedule(job.job_id, eligible_at, reason="rate_limited")
        return

    findings = validate_findings(payload, schema_version=job.schema_version)
    store.publish_once(job.job_id, findings)

The sample numbers are policy defaults, not claims about a provider: the backoff is capped at 60 seconds, jitter ranges from half to one-and-a-half times the exponential value, and max_attempts comes from the lane configuration. In production, datetime.now() should be paired with a queue that performs atomic scheduled transitions. The adapter should pass through a documented retry delay only after parsing it according to that provider's contract.

There is one subtle edge. load_diff proves that content lookup occurs inside the assigned region, but the adapter must receive the content through a region-bound dependency even though the simplified protocol only accepts the job. Don't let a convenient global client silently choose an endpoint. Dependency construction should fail deployment when a US worker is wired to an EU content store, or the reverse.

Why a single FIFO was rejected

The rejected design is a single FIFO queue with fixed worker concurrency and worker-local exponential backoff. It looks adequate in a load test because all work eventually drains. Under mixed traffic, however, a bulk repository scan can sit ahead of an interactive pull request, and independent sleeping workers retain leases while doing no useful work. A shared queue also makes regional routing and tenant fairness implicit, which is precisely where compliance and noisy-neighbor mistakes hide.

It still has a valid use case. A small, single-region internal service with one workload class, one documented quota scope, and no interactive deadline may reasonably choose a FIFO queue. Keep job identities stable, validate outputs, cap retries, and measure queue age. Upgrade to lanes when the workload gains a second latency class or residency boundary; don't prepay the operational complexity before the boundary exists.

Batch processing was also rejected as the universal path. It is not suitable when a developer expects findings during an active review, when cancellation must take effect quickly, or when each result unlocks a user-visible next step. Use it for scheduled re-analysis, evaluation sets, and backlog cleanup where delayed completion is honest. The boundary should appear in the API contract: interactive jobs expose queue status, while batch jobs expose manifest progress and item-level reconciliation.

This architecture doesn't guarantee a particular review quality or response time. Those outcomes depend on model behavior, prompt and schema design, diff size, upstream limits, and the latency objective chosen by the team. It does make the trade-off visible: admit less work, reduce review depth, or accept more delay. Hiding that choice behind automatic retries just moves it into an outage-shaped backlog.

DEV Community