The latest articles on DEV Community by Aleksandr Zakharov (@xezed). https://dev.to/xezed https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F614818%2F32310f9c-bb1d-4d34-8404-3b56c773b13b.jpeg https://dev.to/xezed en Aleksandr Zakharov Wed, 20 Oct 2021 02:10:55 +0000 https://dev.to/xezed/lessons-weve-learned-after-burning-many-thousands-thanks-to-aws-lambda-expect-no-mercy-from-aws-39ph https://dev.to/xezed/lessons-weve-learned-after-burning-many-thousands-thanks-to-aws-lambda-expect-no-mercy-from-aws-39ph <h2> Preface. </h2> <p>A year ago, we decided to make a transition towards serverless architecture. Our management was very excited about it, and its excitement resulted in many tries and failures for developers(including me). So one Monday, we started our working day and realized that one of our lambdas had been going right into the rabbit hole the whole weekend. We were astonished, management was dissatisfied, and I was happy with the new material for the current article.</p> <h2> Our setup. </h2> <p>The staple part of several microservices at our disposal heavily relies upon S3 event notifications. So what happened. <br> A developer screwed up and invoked Lambda from within the same Lambda for the same file in S3, which initially triggered Lambda. These invocations created other S3 files, which started different lambdas... You got the idea. </p> <p>Dev wasn't fired or sanctioned in any way. Because it's an architectural problem, anyone can make a silly mistake.</p> <h2> How much we've lost? Tenths of thousands. </h2> <p>We filled the ticket afterward and got compensated 5k only because we spent this much before the alarm came through.</p> <h2> Precautions we implemented to prevent future incidents. </h2> <ol> <li><p>We set budget notifications and created alarms to email, slack channel, and mobile phone of key tech company figures. </p></li> <li><p>Most of the Lambdas must have reserved concurrency parameters set.</p></li> <li><p>Most of the Lambdas must be invoked via SQS only.</p></li> <li><p>We also implemented AWS Config rule to check all our Lambdas for reserved concurrency.</p></li> </ol> <p>With reserved concurrency, we avoid calling functions more than we should. This way, essentially throttle it. </p> <p>And SQS helps us to prevent data loss. In case of facing concurrency limit, Lambda will wait before obtaining the following message from the queue. </p> <h2> Questions to think about. </h2> <ol> <li>Why is there no option to kill all AWS activities after reaching some usage threshold? </li> <li>Is it this complicated to create an intelligent tool to help AWS customers catch this situation and avoid money loss?</li> </ol> aws serverless cloud terraform