The latest articles on DEV Community by Aishwarya Goel Inferless ( A.G.I) (@aginfer). https://dev.to/aginfer https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1188057%2F07f7efe6-eef8-4c57-91d6-1db184e4d723.jpeg https://dev.to/aginfer en Aishwarya Goel Inferless ( A.G.I) Wed, 15 Nov 2023 17:30:00 +0000 https://dev.to/aginfer/the-state-of-serverless-gpu-part-2-4l6d https://dev.to/aginfer/the-state-of-serverless-gpu-part-2-4l6d <p>In the evolving landscape of AI Infrastructure, Serverless GPUs have been a game changer. Six months on <a href="https://news.ycombinator.com/item?id=35738072">from our last guide,</a> which sparked multiple discussions &amp; created more awareness about the space, we've returned with fresh insights on the state of "True Serverless" offerings and I am here sharing performance benchmark &amp; cost effectiveness analysis for <a href="https://huggingface.co/meta-llama/Llama-2-7b-hf">Llama 2-7Bn</a> &amp; <a href="https://huggingface.co/meta-llama/Llama-2-7b-hf">Stable Diffusion 2-1</a> model. 
</p> <p>📊 <strong>Performance Testing Methodology:</strong> We put the spotlight on popular serverless GPU contenders: Runpod, Replicate, Inferless, and Hugging Face Inference Endpoints, specifically testing for:</p> <p><strong>1. Cold Starts:</strong> Varied across platforms. Latency minus inference time, represents the delay due to initializing a dormant Serverless function.</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--HBLseXZq--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jlnrmym69bycm5w6sw83.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--HBLseXZq--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jlnrmym69bycm5w6sw83.png" alt="Image description" width="800" height="453"></a></p> <p><strong>2. Variability:</strong> We don't just trust one-off results; we test over 5 days to ensure stability. We observed differences in consistency.


</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--hO10M_ZH--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3b3xib8w02bwt3y1ng53.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--hO10M_ZH--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3b3xib8w02bwt3y1ng53.png" alt="Image description" width="800" height="567"></a></p> <p><strong>3. Autoscaling:</strong> Simulated traffic peaks to assess how well platforms scale under pressure ,we tried the simulation on what happens when we receive 200 requests with a concurrency of 5. Not all platforms could manage linear scaling efficiently, leading to varied latencies under load.


</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--o_KaiaEt--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3w1kbgbvdrbh4iyqra7z.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--o_KaiaEt--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3w1kbgbvdrbh4iyqra7z.png" alt="Image description" width="800" height="474"></a></p> <p><strong>4. Decoding Serverless Pricing:</strong></p> <p>4.1 We modeled a scenario where you process 1,000 documents daily with the Llama 2 7Bn model. Here's the TL;DR on costs:</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--Ohs0SYEE--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a1rpwv5nmvdj4k4g1ua9.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--Ohs0SYEE--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a1rpwv5nmvdj4k4g1ua9.png" alt="Image description" width="800" height="289"></a></p> <p>4.2 For the image processing (stable diffusion) use case, only the number of processed items and cold start times differ. Instead of 1,000 documents, we're considering 1,000 images daily.</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s---bmCCHb3--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tzejelcrav5vhm77401w.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s---bmCCHb3--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tzejelcrav5vhm77401w.png" alt="Image description" width="800" height="293"></a></p> <p><strong>🔮 Overall Insights:</strong> The serverless GPU sector is advancing, notably in reducing cold-start times and improving cost efficiency. However, the best choice depends on specific use cases. While AWS Lambda is a leader in general serverless solutions, specialized tasks, particularly those GPU-intensive, may find better options elsewhere.</p> <p><strong><em>Detailed Blog link:</em></strong> <a href="https://www.inferless.com/learn/the-state-of-serverless-gpus-part-2">https://www.inferless.com/learn/the-state-of-serverless-gpus-part-2</a></p> <p>This analysis aims at shedding light on the serverless GPU arena. We welcome feedback and aim for precision in our findings.</p> serverless gpu machinelearning llm