AWS Fault Injection Service for AWS Lambda - Part 4 FIS Lambda Action invocation-error

Introduction

In the part 2 article of the series, we learned how to create a sample application for conducting experiments with AWS FIS Lambda Actions, and how to create and start AWS FIS Experiment Template for the specific aws:lambda:invocation-add-delay Lambda Action. In the part 3 we covered and how to create and start AWS FIS Experiment Template for the specific aws:lambda:invocation-http-integration-response Lambda Action.

In this part of the series, we'll create and start the experiment with the aws:lambda:invocation-error Lambda Action.

For this we'll re-use the sample application from the part 2. Also please read there how to create AWS FIS Experiment Template for Lambda Action, as we'll only focus on the difference requried for aws:lambda:invocation-http-integration-response Action.

Create and start AWS FIS Experiment Template for aws:lambda:invocation-http-error

We'll focus mainly on the Action itself :

give it a name and description, for the "Action type" select Lambda and "aws:lambda:invocation-error". In the "Action parameters" specify 10 minutes "duration" of the experiment, "invocation percentage" of 100% (means all invocations) and enable "prevent execution". Which means return Lambda (invocation) error for all invocations of the target Lambda function without executing its business logic. Of course you can specify other values. Then hit "Save".

For Targets, we'll select another Lambda function, please search for GetProductByIdJava21FISLambda :

When we start newly created experiment, AWS FIS service will create experiment object in the subfolder configs of the vadym-fis S3 bucket for the as experiment target specified Lambda function GetProductByIdJava21FISLambda :

This object file contains the experiment details like:

{"faults":[{"actionId":"aws:lambda:invocation-error",
"parameters"{"invocationPercentage":"100","preventExecution":"true","replaceOutputWithString":"{\"errorMessage\":\"FIS Injected Fault\",
\"errorType\":\"FISInjectedFault\",\"stackTrace\":[]}","injectAsError":"true","qualifier":"$ANY","minimumRequiredExtensionVersion":"1.0.0"},
"expiration":1751303485580,"startTime":1751303322914}]}

The FIS Lambda extension that we deployed for our Lambda functions as a part our application will recognize that FIS experiment is going on when GetProductByIdJava21FISLambda Lambda function is invoked and apply experiment settings (return Lambda invocation error for the all invocations), like explained in the article Introducing AWS Fault Injection Service Actions to Inject Chaos in Lambda functions :

Sometimes it takes up to several minutes or even a new Lambda execution environment to recognize that the experiment has been started and apply experiment settings to the Lambda invocation. For more details, please read the Use the AWS FIS aws:lambda:function actions article.

Let's invoke the GetProductByIdJava21FISLambda Lambda function using the load test tool hey with:

hey -q 5 -z 1m -c 10 -H "X-API-Key: a6ZbcDefQW12BN56WEAW7" {$API_GATEWAY_URL}/prod/products/1

But you can use whatever tool you want, like Serverless-artillery or Postman.

We see many Lambda invocation errors in the CloudWatch :

Let's explore GetProductByIdJava21FISLambda Lambda CloudWatch log :

Experiment details have been logged as well:

And we see many metrics sent to CloudWatch. Let's explore them:

There are FIS Fault Active/Injected metrics:

And there are FIS Lambda extension configuration metrics:

For each experiment you can also find its logs in the logs subfolder of the S3 bucket.

You can also use Amazon DevOps Guru - a machine learning service designed to detect abnormal operating patterns. I wrote 2 articles Amazon DevOps Guru for the Serverless applications - Anomaly detection on Lambda invocations and Amazon DevOps Guru for the Serverless applications -Continuing with anomaly detection on Lambda invocations where I described how DevOps Guru can detect many types of errors that can occur during Lambda function invocation/execution (throttling, errors during initialization phase or during executions)

Conclusion

In this article of the series, we learned how to create and start AWS FIS Experiment Template for the specific aws:lambda:invocation-error Lambda Action.