The latest articles on DEV Community by Chris Hefley (@chrishefley). https://dev.to/chrishefley https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1787069%2F39a3e82b-20a3-4a4f-95f8-18856151bcc4.png https://dev.to/chrishefley en Chris Hefley Mon, 15 Jul 2024 20:44:42 +0000 https://dev.to/chrishefley/how-are-you-dealing-with-openai-outages-in-your-applications-4glk https://dev.to/chrishefley/how-are-you-dealing-with-openai-outages-in-your-applications-4glk <p>We wanted to add some cool AI chat features to our product – and getting it to work with code examples was easy – but reliably scaling it and doing it securely was a lot harder.</p> <p>Even at moderate scale, keeping it reliable and available was a major headache. Provider outages caused our product to fail at the worst times (like product demos).</p> <p>What does it take to get our AI features Production-Ready?</p> <p>We realized we needed multiple LLM providers so that we could gracefully failover to another when not if they had an outage.</p> <p>Different providers had different rate limits, so we added the ability to retry a request to different providers whenever we hit a rate limit. </p> <p>An let’s not forget EU customers. Without data tenancy settings to route AI chat requests to LLM providers in the EU, they wouldn’t be able to use our software. </p> <p>We added response caching, a developer control panel, customer token allowances, secure API key storage, load shedding, and PII data redaction, too. </p> <p>And now we’ve packaged up everything we’ve learned for you to use in your applications. </p> <p>Head on over to <a href="https://llmasaservice.io/" rel="noopener noreferrer">https://llmasaservice.io/</a> and check it out. We're looking for application developers to pilot it. Get in touch, and let's get your AI features into production!</p> ai chatgpt sass