The latest articles on DEV Community by Asankhaya Sharma (@codelion86). https://dev.to/codelion86 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2745210%2F625d5b8c-8651-49f2-8d10-f469c1dd9615.jpeg https://dev.to/codelion86 en Asankhaya Sharma Wed, 22 Jan 2025 03:30:08 +0000 https://dev.to/codelion86/adaptive-classifier-cut-your-llm-costs-with-smart-query-routing-324-cost-savings-demonstrated-1a51 https://dev.to/codelion86/adaptive-classifier-cut-your-llm-costs-with-smart-query-routing-324-cost-savings-demonstrated-1a51 <p>Hey Folks! I'm excited to share a new open-source library that can help optimize your LLM deployment costs. The adaptive-classifier library learns to route queries between your models based on complexity, continuously improving through real-world usage.</p> <p>We tested it on the arena-hard-auto dataset, routing between a high-cost and low-cost model (2x cost difference). The results were impressive:</p> <ul> <li>32.4% cost savings with adaptation enabled</li> <li>Same overall success rate (22%) as baseline</li> <li>System automatically learned from 110 new examples during evaluation</li> <li>Successfully routed 80.4% of queries to the cheaper model</li> </ul> <p>Perfect for setups where you're running multiple LLama models (like Llama-3.1-70B alongside Llama-3.1-8B) and want to optimize costs without sacrificing capability. The library integrates easily with any transformer-based models and includes built-in state persistence.</p> <p>Check out the repo for implementation details and benchmarks. Would love to hear your experiences if you try it out!</p> <p>Repo - <a href="https://github.com/codelion/adaptive-classifier" rel="noopener noreferrer">https://github.com/codelion/adaptive-classifier</a></p> ai opensource machinelearning python