When building applications with LLM APIs, choosing a model is only part of the challenge.
Teams also have to think about:
- Cost: How do you prevent unexpected inference expenses?
- Latency: How do you handle unpredictable response times?
- Reliability: What happens when a provider fails or rate-limits requests?
- Routing: How do you decide which model or provider should handle a request?
- Visibility: Can you explain why a particular routing decision was made?
At CarbonLayer, we're exploring these questions as part of our research into inference trade-offs and decision transparency.
Rather than assume which problems matter most, we'd like to hear directly from developers.
Share your experience
We've put together a short 2–3 minute developer survey covering model selection, cost controls, latency, reliability, and existing tooling.
Take the LLM Routing & Reliability Survey
You don't need to use CarbonLayer to participate.
And if you'd rather discuss it here, I'd love to hear:
What's the one LLM infrastructure problem you wish were easier to control?
CarbonLayer's current public API prototype uses simulated receipts, and its Playground explores model-based cost and latency scenarios. Real per-inference environmental impact measurements are not yet available.
Top comments (0)