AWS FIS Lambda: chaos experiments step by step

With AWS FIS Lambda actions you can inject errors and delays into your Lambda functions without touching a single line of application code. The topic keeps me busy because almost every serverless application looks resilient on paper: retries here, a dead-letter queue there, alarms somewhere. Whether all of that actually works together in an emergency, you only find out once something has broken for real. Usually at night.

Chaos engineering turns this around. You break things on purpose, during the day, announced in advance and with an emergency stop. In this article you build a small, fully runnable example with the Serverless Framework: an API for a bike rental shop, a queue and a worker. After that you run two experiments and look at what happens.

In short

AWS Fault Injection Service (FIS) can inject errors, startup delays and fake HTTP responses into Lambda invocations through a Lambda extension that AWS maintains. The setup consists of a layer, two environment variables and an S3 bucket that FIS uses to distribute the configuration. The biggest pitfall is timing: while idle, the extension only checks once a minute, and the experiment duration, the queue’s visibility timeout and the retry count have to fit together, otherwise you are testing something other than you think.

What you will learn in this article

• How the FIS extension for Lambda works under the hood
• The three Lambda actions and what each one is good for
• A complete setup with Serverless Framework, alarms and experiment templates
• How to write a hypothesis and evaluate the result
• Typical mistakes, costs and safety rules for experiments

What is chaos engineering for Lambda?

Chaos engineering means: you state a testable assumption about your system, inject a controlled fault and compare the outcome with your assumption. It has little to do with breaking things at random. A good experiment is closer to a scientific trial, with a metric, a limited blast radius and a clear stop condition.

For Lambda this is particularly useful, because the interesting failures rarely sit in your own code. Things get exciting between the services: what does the SQS event source mapping do when the worker suddenly loses every third invocation? Does your client timeout hold when the API needs three extra seconds? Does the alarm fire at all? Those are exactly the questions an experiment answers.

A hypothesis instead of gut feeling

Every experiment starts with a sentence of the form “If X happens, Y stays within these limits”. For example: “If 30 percent of worker invocations fail for five minutes, no message ends up in the dead-letter queue, and all rental contracts exist no later than ten minutes after the experiment ends.” This sentence is measurable. That is why you can say afterwards, without any doubt, whether it was true.

How AWS FIS Lambda works under the hood

Since October 2024, AWS FIS has shipped its own actions for Lambda. Technically there is a Lambda extension behind them, which AWS provides as a public layer. An extension is an additional process inside your function’s execution environment. This one sits as a proxy in front of the Lambda Runtime API, the interface your runtime uses to fetch the next event and return the response. That lets it delay an invocation, mark it as failed or replace the response.

Communication runs through S3. When you start an experiment, FIS writes a small configuration file to a bucket you defined beforehand. The extension polls that location regularly. Without a running experiment this happens every 60 seconds by default, during an experiment every 20 seconds. That also explains why the fault does not start and stop to the second.

How AWS FIS injects faults through the Lambda extension: configuration via S3, polling every 60 or 20 seconds, alarm as emergency stop.

Per function, the setup needs four things: the layer, read access to the configuration folder, the variable AWS_FIS_CONFIGURATION_LOCATION with the S3 ARN of the folder, and AWS_LAMBDA_EXEC_WRAPPER set to /opt/aws-fis/bootstrap. The wrapper script starts the runtime in a way that makes it talk through the extension’s proxy.

Which Lambda actions does AWS FIS offer?

There are currently three actions. All of them take a duration in ISO 8601 format (for example PT5M for five minutes) and an optional share of affected invocations, invocationPercentage.

Action	What it does	Typical question
aws:lambda:invocation-add-delay	delays the start of the handler by startupDelayMilliseconds (default 1,000, maximum 900,000)	Do client timeouts and latency alarms hold?
aws:lambda:invocation-error	marks the invocation as failed, with preventExecution even without running the code	Do retries, the DLQ and error alarms kick in?
aws:lambda:invocation-http-integration-response	returns an HTTP status code and content type of your choice	How do callers behind API Gateway or an ALB react to a 500 or 429?

The delay feels like a cold start, but it hits every execution environment and counts towards billed duration. If you set it higher than the function timeout, you also get a real timeout. That comes in handy when you want to check how your application behaves when a downstream service hangs.

The example: a bike rental shop with a queue

Our example is a small bike rental station. A client sends POST /rentals to an HTTP API. The createRental function validates the input, puts a message on an SQS queue and immediately answers with 202. The issueContract worker picks up the messages and writes the rental contract idempotently to DynamoDB. Whatever is left after five failed attempts ends up in a dead-letter queue (DLQ). You may already know the pattern from my article on the asynchronous API with Lambda.

🎬 Bike rental flow with injected faults – this animated diagram only runs on my site: watch it on sascha-ms.de

We want to run two experiments. The first makes 30 percent of worker invocations fail and checks whether the retries catch everything. The second delays half of the API invocations by three seconds and checks whether a client with a five-second timeout gets through without aborts.

Step by step: setting up AWS FIS Lambda with the Serverless Framework

Prerequisites

• an AWS account for testing or staging (not production) and credentials with permissions for CloudFormation, Lambda, IAM, S3, SQS, DynamoDB, CloudWatch and FIS
• Node.js 24 and npm
• Serverless Framework v4 (npm i -g serverless, run serverless login once)
• AWS CLI v2 to start the experiments

Step 1: create the project

Create an empty directory and install the dependencies. Serverless Framework v4 compiles TypeScript itself with esbuild during deployment, so you only need tsc for the tests and the load script.

mkdir radstation-chaos && cd radstation-chaos
npm init -y
npm install @aws-sdk/client-sqs @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb
npm install -D typescript @types/node @types/aws-lambda

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "commonjs",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "dist",
    "rootDir": "."
  },
  "include": ["src", "scripts", "test"]
}

Step 2: write the API function

The validation lives in its own file. That way you can test it without AWS.

// src/validate.ts
export interface RentalRequest {
  bikeId: string;
  customerId: string;
}

export type ValidationResult =
  | { ok: true; value: RentalRequest }
  | { ok: false; error: string };

const ID_PATTERN = /^[A-Za-z0-9-]{3,40}$/;

export function parseRentalRequest(body: string | undefined): ValidationResult {
  if (!body) return { ok: false, error: "Body is missing" };
  let data: unknown;
  try {
    data = JSON.parse(body);
  } catch {
    return { ok: false, error: "Body is not valid JSON" };
  }
  const { bikeId, customerId } = (data ?? {}) as Record<string, unknown>;
  if (typeof bikeId !== "string" || !ID_PATTERN.test(bikeId)) {
    return { ok: false, error: "bikeId is missing or invalid" };
  }
  if (typeof customerId !== "string" || !ID_PATTERN.test(customerId)) {
    return { ok: false, error: "customerId is missing or invalid" };
  }
  return { ok: true, value: { bikeId, customerId } };
}

// src/create-rental.ts
import { randomUUID } from "node:crypto";
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";
import type { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from "aws-lambda";
import { parseRentalRequest } from "./validate";

const sqs = new SQSClient({});
const queueUrl = process.env.RENTAL_QUEUE_URL!;

export const handler = async (
  event: APIGatewayProxyEventV2,
): Promise<APIGatewayProxyResultV2> => {
  const parsed = parseRentalRequest(event.body);
  if (!parsed.ok) {
    return { statusCode: 400, body: JSON.stringify({ message: parsed.error }) };
  }

  const rentalId = randomUUID();
  await sqs.send(
    new SendMessageCommand({
      QueueUrl: queueUrl,
      MessageBody: JSON.stringify({ rentalId, ...parsed.value }),
    }),
  );

  // 202: request accepted, the rental contract is created asynchronously
  return { statusCode: 202, body: JSON.stringify({ rentalId }) };
};

Step 3: write the worker

The worker reports failed messages individually (partial batch response) and writes with a condition, so a duplicate delivery does not create a second contract. I covered both in more detail in the articles on SQS partial batch response and idempotency in Lambda. For the experiment this matters, because repeated deliveries are exactly what we are about to provoke.

// src/issue-contract.ts
import { DynamoDBClient, ConditionalCheckFailedException } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb";
import type { SQSBatchResponse, SQSEvent } from "aws-lambda";

const db = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const tableName = process.env.CONTRACT_TABLE!;

export const handler = async (event: SQSEvent): Promise<SQSBatchResponse> => {
  const batchItemFailures: SQSBatchResponse["batchItemFailures"] = [];

  for (const record of event.Records) {
    try {
      const { rentalId, bikeId, customerId } = JSON.parse(record.body);
      await db.send(
        new PutCommand({
          TableName: tableName,
          Item: { rentalId, bikeId, customerId, issuedAt: new Date().toISOString() },
          // Idempotent: a second delivery of the same message creates no duplicate
          ConditionExpression: "attribute_not_exists(rentalId)",
        }),
      );
    } catch (err) {
      if (err instanceof ConditionalCheckFailedException) continue; // already done
      console.error("Could not create contract", { messageId: record.messageId, err });
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }

  return { batchItemFailures };
};

Step 4: serverless.yml with layer and experiments

Now for the main part. You do not copy the extension’s layer ARN by hand; you take it from the public SSM parameter that AWS maintains per region. The Serverless Framework resolves ${ssm:…} during deployment. The environment variables apply to both functions here, because the whole stack is a staging stack. The file also contains the queues, the table, two alarms as stop conditions, the FIS role and both experiment templates.

# serverless.yml
service: radstation-chaos
frameworkVersion: "4"

provider:
  name: aws
  runtime: nodejs24.x
  architecture: arm64
  region: eu-central-1
  stage: ${opt:stage, 'staging'}
  # FIS extension as a layer: the ARN comes from a public SSM parameter maintained by AWS
  layers:
    - ${ssm:/aws/service/fis/lambda-extension/AWS-FIS-extension-arm64/1.x.x}
  environment:
    AWS_FIS_CONFIGURATION_LOCATION: arn:aws:s3:::${self:custom.fisBucketName}/FisConfigs/
    # Only set together with the layer – without it every invocation fails
    AWS_LAMBDA_EXEC_WRAPPER: /opt/aws-fis/bootstrap
    AWS_FIS_EXTENSION_METRICS: all
    # Recommended with preventExecution=true so the extension has a fresh configuration
    AWS_FIS_POLL_MAX_WAIT_MILLISECONDS: "2000"
  iam:
    role:
      statements:
        - Effect: Allow
          Action: s3:ListBucket
          Resource: arn:aws:s3:::${self:custom.fisBucketName}
          Condition:
            StringLike:
              s3:prefix: FisConfigs/*
        - Effect: Allow
          Action: s3:GetObject
          Resource: arn:aws:s3:::${self:custom.fisBucketName}/FisConfigs/*
        - Effect: Allow
          Action: sqs:SendMessage
          Resource:
            Fn::GetAtt: [RentalQueue, Arn]
        - Effect: Allow
          Action: dynamodb:PutItem
          Resource:
            Fn::GetAtt: [ContractTable, Arn]

custom:
  fisBucketName: radstation-fis-${aws:accountId}-${sls:stage}

functions:
  createRental:
    handler: src/create-rental.handler
    timeout: 10
    environment:
      RENTAL_QUEUE_URL:
        Ref: RentalQueue
    events:
      - httpApi:
          method: POST
          path: /rentals

  issueContract:
    handler: src/issue-contract.handler
    timeout: 20
    environment:
      CONTRACT_TABLE:
        Ref: ContractTable
    events:
      - sqs:
          arn:
            Fn::GetAtt: [RentalQueue, Arn]
          batchSize: 10
          functionResponseType: ReportBatchItemFailures

resources:
  Resources:
    FisConfigBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.fisBucketName}
        PublicAccessBlockConfiguration:
          BlockPublicAcls: true
          BlockPublicPolicy: true
          IgnorePublicAcls: true
          RestrictPublicBuckets: true

    RentalDlq:
      Type: AWS::SQS::Queue
      Properties:
        MessageRetentionPeriod: 1209600

    RentalQueue:
      Type: AWS::SQS::Queue
      Properties:
        VisibilityTimeout: 120
        RedrivePolicy:
          deadLetterTargetArn:
            Fn::GetAtt: [RentalDlq, Arn]
          maxReceiveCount: 5

    ContractTable:
      Type: AWS::DynamoDB::Table
      Properties:
        BillingMode: PAY_PER_REQUEST
        AttributeDefinitions:
          - AttributeName: rentalId
            AttributeType: S
        KeySchema:
          - AttributeName: rentalId
            KeyType: HASH

    # Stop condition 1: as soon as a message lands in the DLQ, the experiment stops
    DlqNotEmptyAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmDescription: Messages in the rental DLQ
        Namespace: AWS/SQS
        MetricName: ApproximateNumberOfMessagesVisible
        Dimensions:
          - Name: QueueName
            Value:
              Fn::GetAtt: [RentalDlq, QueueName]
        Statistic: Maximum
        Period: 60
        EvaluationPeriods: 1
        Threshold: 0
        ComparisonOperator: GreaterThanThreshold
        TreatMissingData: notBreaching

    # Stop condition 2: more than 5 server errors per minute on the HTTP API
    Api5xxAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmDescription: 5xx errors on the bike rental API
        Namespace: AWS/ApiGateway
        MetricName: 5xx
        Dimensions:
          - Name: ApiId
            Value:
              Ref: HttpApi
        Statistic: Sum
        Period: 60
        EvaluationPeriods: 1
        Threshold: 5
        ComparisonOperator: GreaterThanThreshold
        TreatMissingData: notBreaching

    FisLogGroup:
      Type: AWS::Logs::LogGroup
      Properties:
        LogGroupName: /aws/fis/radstation-${sls:stage}
        RetentionInDays: 14

    FisExperimentRole:
      Type: AWS::IAM::Role
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Principal:
                Service: fis.amazonaws.com
              Action: sts:AssumeRole
        Policies:
          - PolicyName: fis-lambda-faults
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Effect: Allow
                  Action:
                    - s3:PutObject
                    - s3:DeleteObject
                  Resource: arn:aws:s3:::${self:custom.fisBucketName}/FisConfigs/*
                - Effect: Allow
                  Action:
                    - lambda:GetFunction
                    - tag:GetResources
                  Resource: "*"
                - Effect: Allow
                  Action:
                    - logs:CreateLogDelivery
                    - logs:PutResourcePolicy
                    - logs:DescribeResourcePolicies
                    - logs:DescribeLogGroups
                  Resource: "*"

    # Experiment 1: 30 % of worker invocations fail for five minutes
    WorkerErrorExperiment:
      Type: AWS::FIS::ExperimentTemplate
      Properties:
        Description: Bike rental – worker errors (30 %)
        RoleArn:
          Fn::GetAtt: [FisExperimentRole, Arn]
        Targets:
          WorkerFunction:
            ResourceType: aws:lambda:function
            ResourceArns:
              - Fn::GetAtt: [IssueContractLambdaFunction, Arn]
            SelectionMode: ALL
        Actions:
          workerErrors:
            ActionId: aws:lambda:invocation-error
            Parameters:
              duration: PT5M
              invocationPercentage: "30"
              preventExecution: "true"
            Targets:
              Functions: WorkerFunction
        StopConditions:
          - Source: aws:cloudwatch:alarm
            Value:
              Fn::GetAtt: [DlqNotEmptyAlarm, Arn]
        LogConfiguration:
          LogSchemaVersion: 2
          CloudWatchLogsConfiguration:
            LogGroupArn:
              Fn::GetAtt: [FisLogGroup, Arn]
        ExperimentOptions:
          EmptyTargetResolutionMode: fail
        Tags:
          Name: radstation-worker-errors

    # Experiment 2: half of the API invocations start 3 seconds late
    ApiLatencyExperiment:
      Type: AWS::FIS::ExperimentTemplate
      Properties:
        Description: Bike rental – API latency (+3 s for 50 %)
        RoleArn:
          Fn::GetAtt: [FisExperimentRole, Arn]
        Targets:
          ApiFunction:
            ResourceType: aws:lambda:function
            ResourceArns:
              - Fn::GetAtt: [CreateRentalLambdaFunction, Arn]
            SelectionMode: ALL
        Actions:
          apiDelay:
            ActionId: aws:lambda:invocation-add-delay
            Parameters:
              duration: PT5M
              invocationPercentage: "50"
              startupDelayMilliseconds: "3000"
            Targets:
              Functions: ApiFunction
        StopConditions:
          - Source: aws:cloudwatch:alarm
            Value:
              Fn::GetAtt: [Api5xxAlarm, Arn]
        LogConfiguration:
          LogSchemaVersion: 2
          CloudWatchLogsConfiguration:
            LogGroupArn:
              Fn::GetAtt: [FisLogGroup, Arn]
        ExperimentOptions:
          EmptyTargetResolutionMode: fail
        Tags:
          Name: radstation-api-latency

  Outputs:
    WorkerErrorExperimentId:
      Value:
        Ref: WorkerErrorExperiment
    ApiLatencyExperimentId:
      Value:
        Ref: ApiLatencyExperiment

A few details deserve a second look. The Lambda role may only read from FisConfigs/, and the FIS role may only write and delete there. FIS needs lambda:GetFunction to look up where in the function’s configuration it should write. And the logs permissions are required so FIS can write its experiment log to the log group.

Step 5: a small load script

No load, no experiment. The script sends a fixed number of requests with limited parallelism and reports latency percentiles and status codes. Status 0 stands for a client timeout.

// scripts/stats.ts
export function percentile(sortedMs: number[], p: number): number {
  if (sortedMs.length === 0) return 0;
  const index = Math.ceil((p / 100) * sortedMs.length) - 1;
  return sortedMs[Math.min(Math.max(index, 0), sortedMs.length - 1)];
}

export function summarize(durationsMs: number[], statusCodes: number[]) {
  const sorted = [...durationsMs].sort((a, b) => a - b);
  const byStatus: Record<string, number> = {};
  for (const code of statusCodes) byStatus[code] = (byStatus[code] ?? 0) + 1;
  return {
    requests: durationsMs.length,
    p50: percentile(sorted, 50),
    p95: percentile(sorted, 95),
    p99: percentile(sorted, 99),
    byStatus,
  };
}

// scripts/load-test.ts
// Usage: API_URL=https://... node dist/scripts/load-test.js
import { summarize } from "./stats";

const apiUrl = process.env.API_URL;
const totalRequests = Number(process.env.REQUESTS ?? 300);
const concurrency = Number(process.env.CONCURRENCY ?? 10);
const timeoutMs = Number(process.env.CLIENT_TIMEOUT_MS ?? 5000);
const pauseMs = Number(process.env.PAUSE_MS ?? 0);

async function oneRequest(i: number): Promise<{ ms: number; status: number }> {
  const started = performance.now();
  try {
    const res = await fetch(`${apiUrl}/rentals`, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ bikeId: `bike-${i % 25}`, customerId: `customer-${i}` }),
      signal: AbortSignal.timeout(timeoutMs),
    });
    await res.text();
    return { ms: performance.now() - started, status: res.status };
  } catch {
    // 0 = client timeout or network error
    return { ms: performance.now() - started, status: 0 };
  }
}

async function main() {
  if (!apiUrl) throw new Error("API_URL is missing");
  const durations: number[] = [];
  const statuses: number[] = [];
  let next = 0;

  const workers = Array.from({ length: concurrency }, async () => {
    while (next < totalRequests) {
      const { ms, status } = await oneRequest(next++);
      durations.push(Math.round(ms));
      statuses.push(status);
      if (pauseMs > 0) await new Promise((r) => setTimeout(r, pauseMs));
    }
  });
  await Promise.all(workers);
  console.log(JSON.stringify(summarize(durations, statuses), null, 2));
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

The tests for validation and statistics run with the built-in Node.js test runner:

// test/stats.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { percentile, summarize } from "../scripts/stats";

test("calculates percentiles on sorted values", () => {
  const values = Array.from({ length: 100 }, (_, i) => i + 1);
  assert.equal(percentile(values, 50), 50);
  assert.equal(percentile(values, 95), 95);
  assert.equal(percentile(values, 99), 99);
});

test("counts status codes", () => {
  const result = summarize([120, 80, 3100], [202, 202, 0]);
  assert.deepEqual(result.byStatus, { "0": 1, "202": 2 });
  assert.equal(result.p50, 120);
});

npx tsc && node --test dist/test/*.test.js

Expected result: # pass 5 and # fail 0.

Step 6: deploy and start the first experiment

serverless deploy --stage staging
serverless info --stage staging --verbose

The output contains the API URL and both experiment template IDs (WorkerErrorExperimentId, ApiLatencyExperimentId). Wait a minute or two after the deployment and send a few requests, so the execution environments including the extension are up. Then start the load and the experiment in parallel in two terminals:

# Terminal 1: about 10 requests per second for almost seven minutes
export API_URL=https://abc123.execute-api.eu-central-1.amazonaws.com
REQUESTS=4000 CONCURRENCY=10 PAUSE_MS=1000 CLIENT_TIMEOUT_MS=5000 npm run load

# Terminal 2: start experiment 1 and query its state
aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7 --tags Name=worker-errors-1
aws fis get-experiment --id EXP1a2b3c4d5e6f7 --query "experiment.state"

The experiment ID in the response of start-experiment differs from the template ID. The state moves from initiating via running to completed. If it says stopped, an alarm pulled the emergency brake.

Step 7: evaluate the result

Check the DLQ first, then compare the number of contracts in DynamoDB with the number of 202 responses from the load script.

aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names ApproximateNumberOfMessages
aws dynamodb scan --table-name "$CONTRACT_TABLE" --select COUNT

CloudWatch Logs Insights shows you when the extension stepped in. The extension writes its messages to the function’s regular log group:

fields @timestamp, @message
| filter @message like /AWS FIS EXTENSION/
| sort @timestamp asc
| limit 50

With AWS_FIS_EXTENSION_METRICS=all, metrics also land in the aws-fis-extension namespace. For experiment 2 you only swap the template ID. There you expect a p95 just above three seconds in the load script, but no status codes 0 or 5xx.

Why timing decides the outcome

Before you celebrate the first experiment, do some quick math. The queue has a visibility timeout of 120 seconds. When an invocation fails, the message reappears two minutes later at the earliest. In a five-minute experiment a message can therefore hit a faulty invocation at most three times. With maxReceiveCount: 5, the DLQ simply cannot fill up. The hypothesis holds, but it tells you less than it seems.

If you really want to know whether the retry count is enough, extend the experiment to 15 minutes. That is where it gets interesting: with a 30 percent error rate, the probability that a message fails five times in a row is 0.35 ≈ 0.24 percent. With 4,000 messages that is, on paper, almost ten contracts in the DLQ, and the alarm stops the experiment. Whether you then raise maxReceiveCount or document a redrive process is a business decision. What matters is that you make it consciously.

A second timing effect concerns the start: while idle, the extension only checks every 60 seconds. So expect up to a minute of ramp-up and about 20 seconds of ramp-down. For very short, rare invocations it can take even longer. That is what AWS_FIS_POLL_MAX_WAIT_MILLISECONDS is for: it makes the function wait briefly until an in-flight poll has finished.

Typical mistakes with AWS FIS Lambda

• Wrapper without layer: if AWS_LAMBDA_EXEC_WRAPPER is in the configuration but the layer is missing, every invocation fails. When removing it, delete the variable first and the layer second.
• Wrong architecture: for arm64 you need the parameter with arm64 in its name, for x86_64 the other one.
• Response streaming: functions with response streaming are not supported. The extension even suppresses streaming while no fault is being injected.
• Versions and aliases: without a qualifier, FIS only looks at the $LATEST configuration. If production traffic goes through an alias, include it in the target ARN.
• SnapStart: with SnapStart the extension more often waits for the full polling interval before it notices the experiment.
• No alarm as emergency brake: source: none is technically allowed. Without a stop condition, however, the experiment keeps running even when something unexpected happens.

What does an experiment cost and how do you keep it safe?

FIS charges per action-minute, currently 0.10 US dollars in the commercial regions. One action running for five minutes therefore costs 50 cents. On top come the usual costs for Lambda, SQS, DynamoDB and a few S3 requests from the extension. The startup delay counts towards billed duration. You can estimate how extra milliseconds add up across many invocations with the Lambda cost calculator.

For safety I stick to three rules. First: run experiments in a dedicated staging stack, never in production first. Second: every template gets at least one alarm as a stop condition, and the blast radius stays small (few functions, limited percentage, short duration). Third: control who may start experiments via IAM, with fis:StartExperiment scoped to specific templates. The layer itself is not free while idle either, since it polls S3 every minute and takes a little runtime. In production I would therefore only add it if the team actually runs experiments there regularly.

My take

I consider the Lambda actions in FIS a real step forward. Before, you needed your own extensions or libraries in your code, and that turned chaos engineering on serverless into a tinkering project for many teams. Now it is configuration, and it fits neatly into an infrastructure-as-code project.

The technology is still the smaller part. The bigger benefit comes when the team prepares the experiment together: write down the hypothesis, agree on roles, and evaluate the results without blame afterwards. That is essentially a game day, a planned practice incident. Anyone who has facilitated a blameless postmortem knows the drill. And if you want to practise the incident commander role once without pressure, this is the perfect opportunity.

Where I disagree: chaos engineering does not replace tests. If the worker lacks idempotency, a unit test finds that faster and cheaper. FIS answers questions that only arise when real services interact. For exactly those questions, though, it is the best tool I know in the AWS world.

FAQ

Do I need code changes for AWS FIS Lambda?

No. You add a layer, set two environment variables and give the function read access to the S3 folder. The handler stays unchanged.

Which runtimes does the FIS extension support?

The extension works at the level of the Runtime API, so it does not depend on the language. For container images you can copy the extension from the layer into the image under /opt. Functions with response streaming are not supported.

Can I run FIS experiments in production?

Technically yes, and in the end that is the goal. Start in staging, though, use alarms as stop conditions and limit percentage, duration and target functions. Only once the team and the monitoring are solid is production the next step.

Why does the fault not start immediately?

Without a running experiment the extension only checks for a new configuration every 60 seconds. You can lower the interval with AWS_FIS_SLOW_POLL_INTERVAL_SECONDS, but you pay for it with more S3 requests and a little runtime.

Your next step

Pick a single function from your staging system for which you are sure that retries and alarms work. Write down the hypothesis before you deploy anything, and run a five-minute experiment with a 10 percent error rate. It costs less than a coffee and takes less than an hour.

I am curious: have you ever run a chaos experiment with Lambda, and what surprised you most? Let me know in the comments.

Sources

• AWS FIS User Guide: Use the AWS FIS aws:lambda:function actions
• AWS FIS Actions reference (section AWS Lambda actions)
• AWS FIS: Available versions of the AWS FIS extension for Lambda
• AWS Cloud Operations Blog: Introducing AWS Fault Injection Service Actions to Inject Chaos in Lambda functions
• CloudFormation: AWS::FIS::ExperimentTemplate
• AWS FIS Pricing

This post was created with AI assistance.

---

👉 Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator – it runs right in your browser.

📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en