The customer service team launches intelligent Q&A, the R&D team builds a knowledge assistant, and the operations team uses large language models to generate content. Each team integrates models separately and maintains its own service endpoints, API Keys, and invocation records. As the number of applications increases, the platform team needs to consolidate scattered invocation records, reconcile model usage, and check each application's connection to upstream services when service issues occur. When switching providers, multiple applications may also need to adjust their integration configurations at the same time.
With CSGHub AI Gateway, the platform team can connect enterprise-managed inference and third-party model services through a single entry point, centrally maintaining caller identities, upstream connections, and routing configurations. Combined with CSGHub's model management and inference deployment capabilities, enterprises can manage model calls from business applications, reconcile consumption, and adjust service configurations based on actual results.
In CSGHub Enterprise Edition (EE), the platform team can use AI Gateway to organize model integration, usage metering, quotas, and reporting.
01 As Models Multiply Enterprises Need a Unified Entry Point
AI Gateway sits between applications and model services. An application submits a request, and the gateway forwards it to the appropriate upstream service based on the caller's identity, service configuration, and routing rules. The upstream model service performs inference and returns the result through the gateway.
The platform team centrally maintains upstream connections, while applications use enabled services through the gateway, reducing repeated development of integration logic across applications. When upstream authentication methods or service endpoints change, administrators first update the service configuration, and application teams then verify compatibility.
Administrators maintain model sources and upstream credentials, while developers use gateway credentials to call services they are authorized to access. Once multiple teams use the same entry point, administrators can monitor usage by caller identity, check upstream health, and locate service issues.
When multiple applications share models, the platform team needs to maintain access scopes and service capacity. When enterprise-managed models coexist with third-party APIs, it also needs to handle authentication methods for different sources. Enterprises can organize this work through the gateway, clarify usage ownership, and monitor resource consumption and service quality.
For example, a knowledge assistant and a content generation application can share model services while using their own caller identities. The platform team maintains the shared upstream services, application teams are responsible for prompts and business logic, and administrators reconcile usage against the corresponding identities.
02 Evaluate AI Gateways with Real Business Workloads First
When evaluating AI Gateways, enterprises should first list the models, interfaces, and business requests to be integrated, then examine integration, stability, usage management, and operational capabilities. Testing protocol compatibility with multi-turn conversations, streaming output, tool calls, and structured output can reveal parameter differences that simple Q&A tests do not cover.
The platform team can conduct an evaluation of CSGHub and confirm the availability of capabilities in the target version, license, and deployment configuration.
Focus Area What to Validate in CSGHub
Integration Scope Connect enterprise-managed inference and third-party APIs and test business requests
Interface Compatibility Check streaming output, tool calls, and required parameters
Traffic Distribution Configure round-robin routing, weights, or session hashing and observe request distribution
Failure Handling Simulate upstream unavailability and recovery and check health status and request results
Usage Ownership Use Keys from different applications and reconcile identities, usage, and quotas
Operational Observability Compare traffic, latency, and node status over the same period
CSGHub organizes unified access, multi-upstream scheduling, authentication, usage management, and operational observability into interconnected capabilities. Deployment conditions should also be included in acceptance criteria during evaluation: whether the required features are available in the target version, whether upstream services provide the corresponding interfaces, and whether existing models and resources support the planned inference approach.
03 Connect Model Management with Business Application Calls in CSGHub
CSGHub is an open-source large language model asset management platform that supports managing models, datasets, code, and application Spaces, and offers private deployment. Enterprises can organize model assets in their own environments, perform inference, fine-tuning, and evaluation, and then integrate model services into business applications.
AI Gateway can connect both third-party model APIs and model services hosted on CSGHub. The platform team manages both types of sources through a single entry point, and business applications select enabled models as needed. Model repositories manage files and versions, inference services run models, and the gateway manages application access and invocations.
Connect Third-Party Model APIs
In "AI Gateway → Commercial APIs," administrators configure the model name exposed to applications, the upstream Endpoint, and provider credentials, and check whether the service is enabled and reachable. When multiple upstream services are needed, they then configure routing according to the target model's compatibility requirements.
Administrators maintain provider credentials on the upstream side, while applications use gateway API Keys issued by the platform. When an enterprise adds a model provider or changes an upstream address, administrators update the service configuration, and application teams verify compatibility with the interfaces and parameters they use.
Connect Models Hosted on CSGHub
For models deployed by the enterprise itself, the platform team can create Public Inference services or Dedicated Instances in CSGHub. Administrators create Public Inference services from a model's details page in Asset Management and, after startup, manage them in the gateway's Public Inference list. Users create Dedicated Instances from the model page and view the instance's API endpoint and invocation instructions.
Unified enterprise access to vLLM is also part of this workflow. Deployment staff select the vLLM runtime when supported by the model and resource conditions, start the inference service, and then bring it under the gateway's access and invocation management. vLLM performs model inference, the platform team manages deployment and resources through CSGHub, and AI Gateway configures application access.
For a model that supports Dedicated Instances, the deployment and integration workflow is as follows:
- On the model page, select "Model Deployment → Dedicated Instance."
- Configure the instance name, region, and resources, and select an available vLLM runtime and the required parameters.
- Set the access method, environment variables, and replica configuration as needed, then start the instance.
- Confirm that the instance is available, obtain its service endpoint and model identifier, and configure gateway access through the service management workflow supported by the target deployment.
- Verify invocation results using business requests, then make the service available to applications. Deployment staff configure resources, runtime, and access methods on the instance creation page. The available models, runtime versions, and parameters are those shown on the current page.
Applications Invoke Models through a Unified Entry Point
After obtaining a gateway API Key authorized to use the target service, developers can query available models and use the corresponding identifier in requests. For OpenAI-compatible text interfaces, developers configure the gateway service endpoint and API Key, then select the third-party model or CSGHub-hosted model to call. For Dedicated Instances, confirm endpoint and identity requirements according to the instance instructions.
Application teams need to check context length, tool calls, structured output, and multimodal parameters to confirm that the interface and selected model jointly support business requirements. After switching upstream services, updating models, or changing inference frameworks, teams rerun business examples and retain model versions, deployment configurations, and validation results.
Teams create clearly named Keys for different applications and maintain each Key's validity period and usage ownership separately. Git Access Tokens are used for Git or platform API access; applications use the corresponding gateway credentials when calling models.
04 LLM Routing and Gateways Direct Requests to the Appropriate Upstream Service
Distinguish Model Selection from Upstream Scheduling
The application team selects a model for the knowledge assistant based on business requirements, then uses gateway routing configurations to assign available upstream services to the target model.
For example, when the same target model has two compatible nodes, administrators can configure request distribution between them. Assigning tasks across different models requires separate business rules and an evaluation of model performance.
Configure Routing Based on Request Characteristics
The CSGHub administration interface provides configurations for round-robin routing, weights, and session hashing. Independent calls are suitable for distribution using round-robin routing and weights. Multi-turn conversations can use session hashing so that requests with the same identifier are directed to the same node whenever possible. Administrators can specify the request header used to identify sessions.
Session hashing provides routing affinity. Conversation history must still be sent by the caller according to interface requirements. Requests may also be redirected to other nodes after an upstream issue or a configuration change. Configuration should balance cache reuse and load distribution to prevent large volumes of requests from concentrating on a small number of nodes.
Figure 4 Routing policy and upstream node configuration interface
Verify Failover and Capacity Control
The gateway provides health checks, circuit breaking, fault isolation, and request-level failover. Backup upstream services must be configured correctly and meet access requirements. Tests should check normal distribution, request results during failures, and distribution behavior after upstream recovery.
If a streaming request is interrupted after some content has already been output, the application needs to retain the results received before deciding whether to retry. When tool write operations are involved, duplicate execution must also be avoided.
Administrators can set concurrency limits to control the number of requests processed simultaneously, and use RPM and TPM to control requests per minute and tokens per minute, respectively. For requests exceeding the concurrency limit, administrators can choose rejection or queuing. In queue mode, the tokens-per-minute and requests-per-minute limits in that upstream service's capacity policy do not take effect. Application teams need to handle waiting timeouts and service errors.
05 Enterprise Token Cost Control Starts with Understanding Consumption
Account for Third-Party APIs and Enterprise-Managed Inference Separately
When calculating the cost of third-party model APIs, the platform team needs to reconcile input, output, and cached usage with the provider's billing rules. Charges arising from failed requests, interrupted streams, and repeated calls are based on actual records.
Enterprise-managed inference also includes computing resources, runtime, utilization, and operational effort. The same token volume may have different costs under different resource configurations and load conditions. Enterprises can use token usage to analyze business volume and combine it with resource investment to estimate the cost of each effective task. During accounting, the platform team compares gateway records with resource data.
For example, repeated retries by a content generation application because of formatting failures may increase consumption per final usable article. A knowledge assistant with a lower per-call price may frequently provide unusable answers, increasing manual review and additional calls. Enterprises should record the number of calls needed to complete tasks and compare output quality and usage using the same set of examples.
First Distinguish Usage Quotas Rates and Monetary Limits
Concept What It Manages
Token Usage Actual consumption, reflecting the scale of business usage
Usage Quota Limits cumulative usage in the configured unit
TPM Tokens used per minute, controlling short-term load
Spending Limit Limits the monetary spending of a Key when billing is enabled
CSGHub EE provides usage metering, quotas, and organizational ownership management. Platform pricing, account balances, deductions, top-ups, invoices, and financial settlement belong to separately enabled billing and settlement capabilities. During deployment, enterprises should confirm the units of measurement used on management pages and check quantity constraints and monetary constraints separately.
Configuring separate, clearly named Keys for different applications helps identify usage sources. Credentials for organizational applications should be configured through the corresponding organization's entry point, and ownership should then be checked against the usage records and reports available in the current version. When further accounting for department or project costs is needed, these records can be combined with the enterprise's internal cost rules.
Measure Optimization Benefits through Business Outcomes
Before optimization, select a set of real business examples and record model versions, input and output volumes, success rates, time to first token, full response times, and actual costs or resource investment.
Then make adjustments one at a time: streamline repeated context, control ineffective retries, evaluate models against task quality requirements, and check output length and inference configurations. For repeated context or multi-turn conversations, evaluate whether session hashing can increase opportunities to reuse upstream Prompt/KV Cache. Caching is provided by the upstream inference service, and actual benefits depend on upstream support, load, and billing rules.
Scaling down idle Public Inference services can reduce resource consumption, but the next request requires a cold start. For workloads sensitive to time to first token, compare the resource cost of keeping replicas running with the responsiveness benefits. Retest every adjustment using the same examples and traffic conditions to confirm that answer quality has not been compromised by cost reductions.
06 Start with Two Applications to Establish an Enterprise Model Service Entry Point
An enterprise runs an internal knowledge assistant and an operations content generation application at the same time. The knowledge assistant uses an enterprise-managed vLLM service, while the content application uses a third-party model API. The platform team wants to manage integration centrally, understand each application's usage separately, and monitor the operational quality of model services.
Implementation Workflow
The platform team first organizes the two applications' model and interface requirements, confirming service deployment locations, available models, and required parameters. Administrators configure Public Inference services, Dedicated Instances, or third-party upstream services and check networks, credentials, and service status.
The two applications use separately identifiable Keys. Models are determined by their respective business configurations. Multi-upstream routing is configured when compatible backup nodes exist for the same target model. The knowledge assistant can evaluate session hashing, while independent content generation requests can evaluate round-robin and weighted distribution.
Administrators configure capacity based on what the services can handle and set available usage quotas according to enterprise management requirements. In environments with billing enabled, administrators also set monetary allowances. Application teams verify streaming responses, error handling, and retries, while the platform team retains configuration and test records.
Acceptance Testing and Ongoing Adjustments
During acceptance testing, application teams test normal calls, invalid credentials, quota limits, traffic bursts, upstream failures, and streaming interruptions. The platform team checks whether requests reach the expected services, whether caller identities are correct, and whether responses during failures and recovery behavior match the configuration.
For example, testers pause an upstream node that is handling requests in the test environment, record request results and node status, and then restore the node to observe subsequent traffic. In another test, testers have the client disconnect after receiving part of a streaming response, check whether the application retains the content and makes duplicate calls, and reconcile the usage already incurred. The team can use these results to develop incident-handling procedures for production.
After launch, compare usage, latency, and errors over the same time range. If the knowledge assistant becomes slower, first check context size, caching, concurrency, and instance status. If the content application's usage is abnormal, check output length, repeated requests, and retries. Adjust the configuration after identifying the cause, then retest using the same business examples.
After Integration Continue Monitoring and Adjusting
CSGHub connects model management, inference deployment, and AI Gateway invocation management. Enterprises can start with one application, validate integration, identities, usage, and failure handling, and then gradually expand to more teams and model services. As usage grows, the platform team can follow the same management workflow to check service quality, reconcile consumption, and evaluate the next model or deployment adjustment.
Top comments (0)