DEV Community

Aryan Rajput
Aryan Rajput

Posted on

Preserving Requested Model Telemetry Across Quota Rejections in GitLab AI Gateway

The Hook

When an enterprise AI gateway dynamically resolves model aliases or rejects requests at the usage-quota middleware layer, reading the model identifier after route execution means your access logs either record the post-resolution fallback model or log None on every 429/403 quota denial—blinding capacity planners to what models users actually requested.

The Deep Dive

GitLab's AI Gateway (ai-assist) powers Code Suggestions and GitLab Duo across cloud and self-hosted model deployments using FastAPI and Starlette. In the /v2/code/completions pipeline, incoming SuggestionsRequest payloads specify a client-requested model (model_provider, model_name), which the route handler may subsequently overwrite during model resolution or reject upfront if has_sufficient_usage_quota fails. While authoring MR !7935 (fix: preserve requested model in access logs, released in v1.944.1), I investigated two critical failure modes in how request-model metadata was captured for structured access logging.

First, if telemetry extraction runs after the quota dependency or after in-place payload mutation, rejected requests never record which model exhausted the quota, and successful requests log the resolved backend model where the requested model should be. Second, in FastAPI route decorators, naively inspecting positional args[0] or untyped kwargs to extract the request body is brittle: FastAPI injects dependencies like Configuration as keyword arguments, so a naive decorator can mistake an injected Configuration object for the SuggestionsRequest payload. Furthermore, self-hosted model payloads can carry sensitive fields (model_api_key, model_endpoint, model_identifier, or AWS IAM role ARNs) that must never leak into starlette_context access logs.

In MR !7935, I extracted a dedicated record_requested_model() route decorator middleware (ai_gateway/api/middleware/route/requested_model.py) that explicitly validates isinstance(payload, SuggestionsRequest) on kwargs.get("payload") and writes only meta.requested_model_provider and meta.requested_model_name into starlette_context before quota enforcement and model resolution execute. Even when has_sufficient_usage_quota short-circuits the request or the handler rewrites the target model, the structured access log retains the exact requested provider and model name with zero credential leakage.

The GitLab Hero Angle

GitLab's AI Gateway (MR !7935) provides enterprise platform teams with tamper-proof observability across multi-model routing and quota boundaries—capturing exact requested-vs-resolved model telemetry in starlette_context while strictly scrubbing self-hosted API keys and endpoints.

The Takeaway

In FastAPI/Starlette AI gateways, capture requested model identifiers into context-local storage via a type-checked decorator before quota guards and model-alias resolvers run, and explicitly whitelist logged attributes to prevent credential leakage.

Top comments (0)