Together AI has announced Together Compiler, a system in which AI agents continuously optimize how open models are served. The company presents it as the first step toward what it calls Polymorphic Inference: serving that adapts to new model architectures, shifting traffic and changing chips instead of relying on one fixed configuration.
What Together Compiler does
Together describes the tool as a loop that treats inference tuning as a search problem for agents instead of a classic compiler translation task. A swarm of agents works against a persistent engine that supports fast iteration. According to the company, it runs in production today on some of its most popular serverless endpoints and has found further gains on code that was already heavily optimized. The company calls the practice Continuous Compilation. Previously a new workload meant new hand-written kernels; the claim now is that the system adapts to new traffic shapes automatically.
The source gives no speed or cost figures for these gains, so the size of the improvement is unconfirmed.
Why Together says static setups fall short
The post argues that typical open-source serving recipes ship with one tuned configuration, which may not fit another setting. It cites mixture-of-experts kernels that perform well on random tokens but may not carry over to real traffic. It also notes that 80% of OpenRouter traffic goes to models released in the last three months, and that the ten most popular open models each use a different architecture. Together says its own earlier work on megakernels taught it that fusing everything into one kernel is the wrong goal for this environment.
What is available and what comes later
- Together Compiler is available to end customers in a limited beta; access is by request.
- Soon, customers are meant to optimize dedicated endpoints for their own workloads at the click of a button.
- Other items on the research list: speculative decoding that adapts per request, a router for sending requests across different deployments, post-training of open models against a customer's own tools, and workload-aware routing across several models.
No dates, prices or plan names are given for any of these.
Practical reading
The announcement is mainly a statement about how Together intends to keep open-model serving fast as models turn over quickly. Teams that run production workloads on open models there should treat the beta as an option to evaluate, and should test against their own traffic, since the claims are the vendor's own and come without published benchmarks.
Top comments (0)