Together AI is revolutionizing how AI models are deployed with its innovative capacity-aware routing architecture for dedicated model inference. This system is designed to provide predictable, scalable, and efficient deployment of AI models, a crucial step in bringing advanced AI capabilities to a wider audience. By focusing on actual resource capacity rather than fixed percentages, Together AI’s approach allows for dynamic deployments, sophisticated A/B testing, and seamless zero-downtime updates.
The Foundation: Endpoints, Deployments, and Configurations
At its core, Together AI's dedicated model inference system is built upon three interconnected components: endpoints, deployments, and configurations.
- Configurations (Configs): These act as immutable recipes, defining the essential parameters for inference. A config specifies the inference engine, the required GPU specifications (type and count), and the desired optimization profile (latency, throughput, or balanced). Because configs are immutable, they ensure consistent behavior for a deployment. Any changes necessitate a new revision, providing a stable rollback point.
- Deployments: Each deployment is associated with a specific model revision and a particular config. Crucially, deployments include autoscaling policies that are tied to the specific model version being served. This allows the system to automatically adjust the number of running instances based on demand.
- Endpoints: These are stable, user-facing names that serve as the entry point for traffic. Endpoints direct incoming requests to one or more deployments based on their assigned weights and the number of ready replicas.
This layered structure offers granular control over AI model serving, enabling robust management of even the most complex AI workloads.
Capacity-Aware Routing: The Engine of Efficiency
The true innovation of Together AI's system lies in its capacity-aware routing mechanism. Instead of relying on static percentage-based traffic splits, this system dynamically allocates traffic based on the actual capacity of each deployment.
The traffic split is defined as a list of {deployment_id, weight} pairs. The system then calculates each deployment's effective capacity by multiplying its assigned weight by its current number of ready replicas. Traffic is subsequently routed proportionally to this calculated capacity. For example, if two deployments have equal weights but one has twice as many active replicas, it will naturally receive approximately two-thirds of the traffic, while the other receives one-third. This intelligent design ensures that as a deployment scales up, its traffic load automatically adjusts, preventing over-saturation and minimizing the risk of idle resources.
This capacity-driven approach is particularly beneficial when dealing with fluctuating resource availability. Deployments with zero ready replicas, perhaps due to cold starts or temporary failures, contribute zero capacity and therefore receive no traffic. This ensures that traffic seamlessly flows to available and healthy deployments, guaranteeing continuous service without manual intervention. This stands in stark contrast to traditional percentage-based splits, which can lead to inefficient resource utilization or service degradation when replica counts change.
Advanced Deployment Strategies for Seamless Updates
Together AI's architecture not only ensures efficient scaling but also facilitates advanced deployment strategies, including zero-downtime updates.
- Rollouts: New model versions can be gradually rolled out to a subset of users, allowing for careful monitoring before a full deployment.
- A/B Testing: The system supports A/B testing cohorts, enabling developers to allocate fixed traffic shares using integer percentages that sum to 100%. These adjustments can be made dynamically via simple API updates, allowing for real-time experimentation and optimization of model performance.
- Shadow Experiments: This feature allows new model versions to receive a copy of the live traffic without impacting user experience. This is invaluable for testing the performance and impact of a new model in a production-like environment before committing to a full rollout.
These advanced strategies collectively contribute to achieving zero-downtime updates, ensuring that users experience uninterrupted service even during model updates and changes.
Simplifying Deployment with Pre-Benchmarked Profiles
Recognizing that crafting optimal configurations can be complex, Together AI offers pre-benchmarked 'deployment profiles' for many supported models. These profiles represent certified model-and-config pairs that have been rigorously tested and refined by the platform.
For instance, users can explore profiles for models like the Gemma 4 31B deployment, selecting hardware and optimization settings that best align with their specific needs. When choosing a config, developers can select the accelerator type, GPU count, and critically, the optimization profile.
- Latency Profiles: These prioritize rapid response times, making them ideal for interactive applications and real-time services.
- Throughput Profiles: These are designed to maximize token output, which is essential for batch processing and large-scale data tasks.
- Balanced Profiles: These offer a versatile default, suitable for mixed workloads that require a blend of low latency and high throughput.
The immutability of configurations is a key feature, ensuring that a deployment's behavior remains consistent. This stability is crucial for reliable AI model serving.
Real-World Validation and Performance Insights
Together AI has demonstrated the effectiveness of its capacity-aware routing through practical experiments. In one test, two single-H100 deployments were configured with equal weights and one replica each, resulting in a 50/50 traffic split. When one deployment scaled to two replicas, the traffic split dynamically adjusted to approximately 66.7/33.3, precisely mirroring the increased capacity. This real-world scenario validates the system's ability to dynamically manage load based on available resources.
Furthermore, the company has presented performance benchmarks for models like the MiniMax M3 inference, comparing different optimization profiles under varying concurrency levels. These benchmarks highlight that a profile excelling at low concurrency might significantly degrade under higher loads. This underscores the critical importance of tailoring inference configurations to specific traffic patterns and workloads, a capability that Together AI's platform is designed to facilitate.
The mission of Together AI Dedicated Model Inference is to democratize access to advanced AI inference capabilities. With a few simple commands, developers can deploy models, configure endpoints, and begin serving traffic, leveraging sophisticated infrastructure without being burdened by extensive operational overhead. This combination of ease of use, intelligent routing, and powerful scaling mechanisms positions Together AI as a leading solution for deploying AI models efficiently and effectively at scale. This refined approach to how we together refines model deployment is set to significantly impact the AI landscape.
For those interested in scalable solutions for demanding AI tasks, understanding how platforms like Databricks databricks scales real-time fraud detection offers valuable context on robust AI infrastructure.
tags: ai, machine learning, model deployment, inference, capacity-aware routing, cloud computing, technology, startuphub.ai
Top comments (0)