This system has two main lines: one is model management and model assets, answering "who can use what and how much"; the other is the AI gateway, enforcing those decisions on every request. This article follows "the life of a model" and "the life of a request" in turn, and finally sees where the two lines mesh.
1. Assets and traffic
The core object of traditional API management is the interface; the core object of AI platform management is the model. Both appear to handle HTTP requests, yet what they care about differs:
| What interface management cares about | What model management additionally cares about |
|---|---|
| Address, protocol, authentication, routing | Model listing, selection, permissions, quotas, unit price, retirement |
| Whether the request succeeded | How many input and output tokens were consumed |
| Whether the upstream is healthy | Whether instances are healthy, whether quota remains, whether the provider needs to be switched |
| Who the caller is | Which application, user and cost center the call belongs to |
The left column is what the traffic side cares about; the right column is what model management adds. Instance health and remaining quota are ultimately executed on the data plane — the asset side states the requirements and consumes the results. If the people who manage assets and the people who manage traffic were two separate systems, each holding half the picture, they would drift apart: approvals, permissions and quotas would stay inside the asset system and fail to constrain real requests; the gateway would forward traffic without knowing why a model was listed, who is allowed to use it, or whose cost the call books to.
Their real relationship is a loop:
Model listing → selection and application → authorization and quota generation
→ gateway authentication, rate limiting, routing, rewriting
→ usage and status flow back → cost allocation, scale-out, migration and retirement
The loop runs in two directions: decisions flow down — models, permissions, quotas and routing rules enter the gateway through the configuration plane; results flow up — token usage, callers and call outcomes return from the gateway to the asset side.
Three planes keep the loop turning: the business plane makes decisions and owns model assets and usage rules; the configuration plane translates and delivers those decisions, turning business rules into router, instance and plugin configuration the gateway can execute; the data plane executes decisions and reports results back — built on an AI gateway, it is responsible only for authentication, rate limiting, routing, rewriting, forwarding and recording along the request path. Each plane keeps to its boundary: the business plane does not touch inference traffic, the data plane does not hold asset rules, and the configuration plane only translates and delivers.
From here on, this article uses two words: the side that manages the rules is the asset side, corresponding to the business plane of the three planes; the side that sends requests is the caller — both applications such as low-code agent platforms and microservices, and direct personal office use.
The hard part is here: the same model, token and caller must point to one and the same thing in both the asset system and the traffic system. Next, two lines: first the life of a model, then the life of a request.
2. Model management and model assets: the life of a model
One running example throughout: the customer support team wants to connect a new inference model to their low-code agent platform. From arrival to exit, the model passes through six gates:
2.1 Selection: verify first, then add to the asset catalog
They first tried several candidates side by side in the model marketplace: tuning Temperature, Top_p and Max_tokens one by one to see how the parameters shape the output; once the parameters were settled they were frozen, and the models were swapped while re-running the same set of prompts — keeping the model as the only variable — to compare the differences.
Selection is not just picking a model name; it also means judging whether reasoning capability is needed, how high the quality bar is, and whether there are security risks. Security risk is especially decisive — it directly determines whether to go with a private deployment or a cloud service, and the later application and routing approach follow from it.
2.2 Application: establish ownership first, then open production access
The application form must state the usage mode (low-code agent platform, microservice calls, or personal office use) and the environment (test or production).
Production access must be bound to an explicit scenario. Without a scenario, calls stop at the token and never reach the application or cost center, so auditing, cost attribution and problem diagnosis are limited to that coarse layer. So what this step really establishes is the three-way relationship between model, caller, and cost ownership.
2.3 Tokens: the carrier of permission, not an isolated key
One token can carry permissions for multiple models — you do not need a separate key per model. The deletion boundary for tokens is deliberately tight: delete the last model permission and the token is invalidated with it; if an agent still references the token, deletion is refused. A token that is "still valid but whose user is unknown" must never exist in the system.
Once the application is approved, the rules continue down the chain. The enterprise/application path issues a token, creates a Consumer (the gateway-side object representing the caller, holding its credentials and plugin configuration), writes model permissions and quota, and adds the route to a whitelist; the personal path only issues a token, with no Consumer — authentication and quota checks happen in real time on the asset side. Until this step is complete, "authorized" in the asset system cannot become "may call" in the request path.
2.4 Invocation: asset relationships carried into the request
Once real calls begin, the SDK documentation is only the surface; underneath it, model management rules are expressed on the request side:
- Exceeding either RPM or TPM causes a rejection, constraining request frequency, model throughput and the cost budget;
- Keeping
X-Session-IDconstant within a conversation lets consistent hashing land requests on the same instance as much as possible, reusing the KV cache; - Passing user information via
baggage: sentry-user(baggageis the standard request header OpenTelemetry uses to propagate key-value context along a trace) lets the call be correlated with the log fieldcaller_userwhen the caller carries it as agreed, enabling end-to-end attribution.
These request headers are not business decoration; they are the carriers through which session, user and cost ownership enter the data plane.
2.5 Billing: usage must flow back to the asset relationship
A single user interaction can trigger several model calls; counting by interactions or requests alone cannot explain real consumption, and capacity and cost have to be measured in tokens.
Token-based measurement supports three things: capacity sizing — production statistics yield the average token consumption per interaction (P95 as well; the mean hides the long tail of multi-turn loops), which is fed into load tests to verify the RPM / TPM, concurrency and latency ceilings so that business volume can be converted into resource requirements; cost allocation aggregates by token to the application or sub-product, drillable down to the user; cost estimation uses DAU × interactions per user per day × average tokens per interaction × unit price to project daily cost — for the latter two, DAU and interaction counts come from product-side instrumentation, while the gateway supplies only the token and unit-price bases, with input and output tokens priced separately.
The gateway is the feedback entry point here: it parses upstream usage, records input and output tokens, model instance, caller and trace information, and hands them to quota settlement, detail queries and cost allocation.
2.6 Retirement: remove references first, then delete the asset
A model cannot be retired by deleting it from a list. Published routes, issued tokens, agent references and in-flight migrations must all be handled in sync. The system uses a migration protection list to block new routes being created against the old model, and route creation also validates service template compatibility.
The correct order is: switch the calling relationships first, then reclaim tokens and routes, and only then remove the model asset. Doing it in reverse does not work: delete the asset before cutting traffic, and the asset relationships drift apart from real traffic.
The six stages do not really end here. The three dashed lines in Figure 2 are the feedback flow: usage, quality and cost feedback supports scale-out plans and provider trade-offs; permission and quota adjustments flow back to the application step; and the slot freed by retirement is filled by a replacement model, re-entering selection evaluation.
3. The AI gateway: the life of a request
The asset relationships established in Chapter 2 finally take effect in real requests — some as short as milliseconds, others as long as several minutes.
Once the low-code agent platform is officially connected, every request on the enterprise path enters the gateway, and the gateway must answer five questions in order:
- Who are you: verify the enterprise credentials.
- May you call this model: check model permissions and the routing whitelist.
- May this request be admitted: run security validation plus the admission checks on RPM, concurrency and input tokens.
- Where does this request go: filter out unhealthy or quota-exhausted candidate instances to complete routing, then re-check instance-level quota.
- How much was actually used: parse the response usage, write logs, and report back to the asset side.
Figure 3 expands these five questions into eight stages; the extra request rewriting and forwarding are execution details. Identity and permission must be confirmed first, the admission check comes before routing, and actual usage can only be settled after the response — implementations may vary; this order cannot.
Behind all five questions are the decisions from Section 2: who you are comes from the ownership established at application time; whether you may call comes from token permissions; whether it is admitted comes from the quota verdict; where it goes comes from the instance list determined by selection and retirement; how much was used returns to the billing basis. The two lines mesh right here.
3.1 Two kinds of callers, two paths
The low-code agent platform takes the enterprise path; personal office use takes the personal path. The two kinds of callers have different data sources and do not fit into one authentication model:
- Enterprise/application path: the token is bound to a fixed caller, and the gateway completes identity, model permission and quota checks locally.
- Personal path: no fixed consumer object is created; the asset side validates token and quota in real time, and actual usage is reclaimed after the request ends.
The two kinds of calls use different scenario domains. By the time a request reaches the gateway, the domain has already determined whether it hits the enterprise route or the personal route — it is not a choice made by route labels after entry.
3.2 Usage is parsed exactly once
The gateway parses upstream response usage exactly once, and hands the same result to token quota settlement, personal usage reporting, logs and metrics, avoiding the inconsistent accounting that arises when each of them computes it separately.
This step turns "the request succeeded" into operational information: not only whether the call completed, but which instance served it, how many tokens it burned, and whose account it books to.
Request and response bodies may contain prompts, business data, and even personal information; "can be recorded" does not mean "retained in full by default". In production, sampling decisions must also be made per scenario, together with the corresponding masking, access control and retention periods.
3.3 Instance pool
One model may map to self-hosted instances, a cloud service and a backup provider at the same time. Before forwarding, health status, remaining quota and alert-based demotion jointly decide whether an instance may enter the candidate set; after forwarding, an upstream 429/5xx triggers candidate switching according to the retry conditions. These conditions must converge into a single routing-and-retry chain, otherwise an instance just removed by health checks may be re-selected by another logic path, and a high-priority instance that has exhausted its quota may keep receiving traffic.
3.4 A real request
In the figure, the red-flagged "unavailable" instance was removed from the candidates after failing an active health check. This request first reached the preferred instance, which returned 429 — excluded only for this request, marked "tried this round" in the figure, and not written into health-check state. Fallback is on, the time since the last failure is still within the retry window, and retries remain, so the router then switched to backup one. The low-code agent platform notices nothing: no retry of its own, no need to change address or credentials.
Request-level exclusion, health-check removal, quota filtering and alert-based demotion — four mechanisms that do not interfere with one another, yet all act on the same candidate pool. Only after the preferred instance keeps failing past a threshold is it demoted by alert integration, and it re-enters the candidate set according to policy once probing recovers.
4. Where the two lines mesh
The asset side makes decisions and the traffic side executes them, connected by quota and configuration; the capability map fixes the boundary between the two sides.
4.1 Quota: computed on the asset side, executed on the data plane
The asset side holds model permissions, plans and cost rules, so it is the right place to answer "how much should be granted"; the data plane sees every real request, so it is the right place to answer "how much has been used". The two are not interchangeable.
The enterprise/application path connects them with a five-step chain (the "Business side" in the figure is this article's asset side):
The core boundary in one sentence: on the enterprise path, the gateway does not hold quota rules — only the computed verdict. The personal path goes further: authentication and quota checks are all done in real time on the asset side.
Token rate limiting has two stages: check remaining headroom before forwarding, then book against real usage after the response. That way a call can be rejected before it happens and settled accurately afterward. The cost is that concurrent in-flight requests have no reserved quota, so actual usage can briefly exceed the threshold.
To users, this implementation ultimately surfaces only as the rate-limit dimensions RPM / TPM, with exact values on the model marketplace detail page; the performance tiers (standard mode / high-performance mode, etc.) are separate information annotated on the model marketplace page.
Time-window rate limiting follows the same logic: the time window is a variable read when the request arrives, so busy/off-peak quotas switch at runtime under the same rules, with no daily batch edit of live configuration.
4.2 Configuration delivery
Model permissions, quotas and instance state all change, but changes cannot be made by overwriting the gateway configuration wholesale — a single business-side delivery could wipe out manual operations adjustments.
So the configuration stage enforces exactly two boundaries. Source-of-truth boundary: managed plugins treat the database as the single source of truth; unmanaged plugins keep their current data-plane state; the gateway is called only after the database transaction commits, and external IO stays out of the transaction. Failure boundary: a failed delivery records status, alerts and retries — a database rollback is never used to paper over "not delivered"; if a rollback is genuinely needed, restore the gateway first, then roll back the database.
The configuration plane only changes target fields and reads back to confirm after delivery. The goal is not that a given call never fails, but that after a failure you still know where the source of truth is and the data plane can re-converge.
4.3 Capability map
The asset side and the traffic side each hold a segment — where does the boundary lie? The starting point was a list of 13 capability categories:
Management · Traffic governance · Connections and timeouts · Routing and load balancing · Request rewriting · Observability
Health checks · Model management · Security · Multi-region · Protocol conversion · Extensibility · High availability
Laid item by item against a traditional API gateway, the differences cluster in governance and operations, not in forwarding — and among the 13 categories, observability and request rewriting were the densest. Grouped by where the capability finally executes, they fall into 8 domains:
The most important thing about this map is not the number of features but the boundaries: A–G decide how a request executes; H decides why it executes that way. Traditional API gateways mainly cover the former; to make models first-class citizens, the latter must be supplied by model management and the business control plane.
5. How feedback becomes decisions
Operational results flow back from three places, carrying the traffic side's execution results back to the asset side:
| Feedback trigger | Who catches it | What decision it changes |
|---|---|---|
| Persistent upstream 429 / 5xx triggering an alert | Data-plane alert integration demotes priority | The instance's routing priority, directly affecting every subsequent routing decision |
| Token usage and call details | Quota and analytics layer | Cost allocation aggregates to the application or sub-product (drillable to the user), and supports the capacity and cost basis (2.5) |
| Instance health status and usage growth trends | Operations and model operations | Scale out instances, degrade, or add providers — eventually entering the retirement and migration of 2.6 |
The first row needs no human: once the state changes, routing priority changes with it. The latter two need a human decision: data first becomes bills and dashboards, and only then does someone decide to scale out, switch providers, or retire.
Conclusion
Model management decides who can use what and how much; the AI gateway is responsible for enforcing those decisions on every request; and after each request, usage, callers and outcomes flow back into model management to support cost allocation, scale-out, migration and retirement.
This is also where the two main lines intersect: the life of a model establishes and changes asset relationships, and the life of a request executes and verifies them. Without the first, the gateway does not know why it routes the way it does; without the second, permissions and quotas in model management are just rules on paper.
Connect these things, and model assets, traffic governance and cost management finally form a genuine closed loop.
A note from the author
I drove this system end to end — from the capability inventory and three-plane architecture design, through custom plugins, the full model lifecycle, rate-limiting migration, and the operations console.
If you are also working on model platforms, AI gateways or LLMOps, I would be glad to compare notes.
Further reading
- Selection criteria: What Kind of Gateway Makes a Good AI Gateway?
- Another rate-limiting shape: Rethinking LLM Gateway Rate Limiting: From an In-Process Leaky Bucket to a Distributed Leaky-Bucket Matrix
- Protocol conversion: Efficient Design and Development in the AI Era: Building an Anthropic → OpenAI Protocol Conversion Plugin
- Production troubleshooting: One Upgrade, Two Independent Bugs: Tracing Streaming 500s and CPU Pegged at 100% Under Low Traffic in an API7 AI Gateway
This article is the English edition of a post originally published in Chinese.
Top comments (0)