DEV Community: Corentin Doue

Top 10 common errors I wish I hadn’t made using SQS

Corentin Doue — Thu, 13 Jun 2024 11:56:11 +0000

Amazon SQS is a powerful service for messaging in traditional or Serverless applications, but it comes with its own set of challenges. I've compiled a list of common mistakes and best practices to help you navigate SQS more effectively. I also released the Swarmion SQS contract that helps you avoid these pitfalls and focus on your business logic.

TL;DR;

Configuration

❌ Don’t expect SQS messages to be consumed in the order they are sent
❌ Don’t set a too small Retention Period
❌ Don’t set a too small Visibility Timeout
❌ Don’t use Lambda reserved concurrency to control throughput

Producer

❌ Don’t send messages that the consumer can’t process
❌ Don’t send messages individually
❌ Don’t send too many messages to SendMessageBatchCommand or batch too big messages
❌ Don’t forget to handle SendMessageBatchCommand results
❌ Don’t send too many messages to SQS FIFO queues
❌ Don’t forget to use MessageGroupeId for SQS FIFO Queues
❌ Don’t use uuid() as MessageDeduplicationId

Consumer

❌ Don’t throw errors while processing a batch of SQS messages

🤔 Yeah, there are 12. But a top 10 title sounded better

SQS Configuration

🙅 Error: Expecting messages to be consumed in the order they are sent

💥 Impact: 🐛 Stability. Messages are processed in an unpredictable order.

✅ Solution: Use SQS FIFO if you need to process messages in a precise order

🙅 Error: Setting a too small Retention Period

💥 Impact: 🐛 Stability. Messages can be deleted before they are processed, especially with delays or multiple retries. This can be a debugging nightmare.

✅ Solution: Set a generous retention period if you plan to use delays or retries. Retention is not billed.

🙅 Error: Setting a too small Visibility Timeout

💥 Impact: 🐛 Stability. Messages can be processed several times if their visibility timeout expires before their first processor delete them.

✅ Solution: AWS recommends setting a visibility timeout three time longer than the expected message processing duration.

🙅 Error: Using Lambda reserved concurrency to control throughput

💥 Impact: 🐛 Stability. Messages can be lost due to throttle errors returned by the Lambda service, which can result in them being sent to the DLQ without being processed.

✅ Solution: Use the MaxConcurency parameter of the event source mapping instead of Lambda reserved concurrency.

Producer

🙅 Error: Sending messages that the consumer can’t process

💥 Impact: 🐛 Stability. Consumers will throw errors, causing messages to be lost or behave unpredictably.

✅ Solution: Enforce a strong interface between your producers and consumers.

💡You can use Swarmion contracts to create and enforce interfaces between your lambdas and the services that use them.

🙅 Error: Sending messages individually

💥 Impact: ⚡💰 Performance and cost. Each message is sent as an HTTP request, increasing both time and cost.

✅ Solution: Use SendMessageBatchCommand to batch messages up to 10. One batch request is billed as one request.

💡You can use the sendMessages utility of Swarmion SQS contract to send multiple messages without bothering with technical aspects

🙅 Error: Sending too many messages to `SendMessageBatchCommand` or batch too large messages

💥 Impact: 🐛 Stability. SendMessageBatchCommand can batch up to 10 messages with a total size below 256Kb. Exceeding these limits will cause the batch to be rejected, potentially losing messages.

✅ Solution: Batch messages up to 10, ensuring the total size is within limits.

💡 The sendMessages utility of Swarmion SQS contract provides automatic batching that follow these rules. Just pass an array of messages and it handles the rest.

🙅 Error: Forgetting to handle `SendMessageBatchCommand` results

💥 Impact: 🐛 Stability. SendMessageBatchCommand doesn’t throw if messages are throttled, they are returned in the response.

✅ Solution: Handle failed batch items returned by SendMessageBatchCommand and retry them.

💡The sendMessages utility of Swarmion SQS contract automatically retries throttled messages and throws errors by default to avoid silent bugs.

🙅 Error: Sending too many messages to SQS FIFO queues

💥 Impact: 🐛 Stability. FIFO queues are throttled at 300 requests per second, causing some messages to be lost if not handled.

✅ Solution: Use high throughput FIFO queues or/and control the throughput rate of your sender.

💡The sendMessages utility of Swarmion SQS contract provides a throughputCallsPerSecond parameter to precisely control throughput.

🙅 Error: Forgetting to use `MessageGroupeId` for SQS FIFO Queues

💥 Impact: ⚡Performance. All messages will be processed one at the time.

✅ Solution: use MessageGroupeId to enable parallel processing of message groups. Group messages by related usage to allow unrelated messages to be processed in parallel.

🙅 Error: Using `uuid()` as `MessageDeduplicationId`

💥 Impact: 🐛 Stability. Messages can be processed multiple times.

✅ Solution: MessageDeduplicationId must be a hash of your message content

Consumer

🙅 Error: Throwing errors while processing a batch of SQS messages

💥 Impact: ⚡🐛 Performance and stability. The entire batch will be retried after the visibility timeout is reached. Some messages will be processed or partially processed multiple times. As no message is deleted, this jam the queue.

✅ Solution: Catch errors individually and delete successfully processed messages. With lambda event source mapping, use ReportBatchItemFailures function response type and send back the unprocessed messageIds.

💡You can use the getHandler utility of Swarmion SQS contract to generate a wrapper around your handler to process messages individually, catch errors and report failed messages to SQS.

Conclusion

I hope these insights help you avoid the common mistakes I’ve encountered while working with SQS. Please share your experiences and any other tips you have for using SQS effectively.

Enforcing ESLint rules: A guide to taming codebase chaos

Corentin Doue — Tue, 02 Jan 2024 15:14:53 +0000

In the dynamic world of javascript development, ESLint plays a vital role in maintaining code quality. When developers install ESLint or introduce new rules, it often reveals tons of errors. How to manage them?

📝 TL;DR

Use errors to make the rules blocking and enforce it in your CI.
Comment out all existing ESLint errors using a script like eslint-disable-inserter.
Merge this as soon as possible to prevent new errors from being added.
Then fix the errors at your own pace.

⚠️ The temptation of the Warnings️

When developers first face ESLint errors, the volume of violations across files can be daunting. Some may be tempted to convert these errors into warnings to quickly pass the linter without addressing the underlying issues.

This is a bad practice, nothing will prevent new errors from being added. The developers will not know what is an acceptable warning or an error disguised as a warning.

⚔️ Another Pitfall: Try to fix them all at once

Some developers can be tempted to fix all existing errors before applying new ESLint rules to the codebase. While this approach seems logical, it often results in massive pull requests that are challenging to review and rebase. As developers work to resolve errors, others may simultaneously introduce new ones, making it an endless battle.

🤖 A Better Approach: `eslint-disable-inserter`

To address this issue and ease new rule usage, I created eslint-disable-inserter, a npm package that simplifies commenting existing ESLint errors.

CorentinDoue / eslint-disable-inserter

Inserts eslint-ignore comments automatically

eslint-disable-inserter

Easily insert eslint-disable-next-line comments into your code.

When moving to a new ESLint config, or when adopting ESLint for the first time it's common to have tons of violations that you want to silence for now.

This library exposes a helpful utility, eslint-disable-inserter, that will do all the heavy lifting, and insert // eslint-disable-next-line ... or {/* eslint-disable-next-line ... */} comments into your code.

It handles JSX detection, and will insert the correct comment in the correct places.

This utility is idempotent, so it can be used each time you add a new ESlint rule.

Example (Before/After)

With the following file, which has some violations and a existing comment:

export const MyComponent = () => {
  let count = 0
  count += 1
  const messages: any = undefined
  return (
    <div>
      <h1>MyComponent</h1>
      <p>Count: {

…

View on GitHub

How `eslint-disable-inserter` Works

eslint-disable-inserter automatically inserts // eslint-disable-next-line ... or {/* eslint-disable-next-line ... */} comments into your code, silencing existing ESLint errors. It handles JSX detection and is idempotent, allowing repeated use without duplicates.

Example: Transforming Code

Suppose you have the following TypeScript file with ESLint violations:

export const MyComponent = () => {
  let count = 0;
  count += 1;
  const messages: any = undefined;
  return (
    <div>
      <h1>MyComponent</h1>
      <p>Count: {count + messages.myMessage}</p>
      {/* eslint-disable-next-line eqeqeq -- my comment */}
      <p>Is Zero: {count == 0 ? messages.yes : messages.no}</p>
    </div>
  );
};

Running the following command:

eslint --format json . | eslint-disable-inserter

Will transform the file to:

export const MyComponent = () => {
  let count = 0;
  count += 1;
  // eslint-disable-next-line @typescript-eslint/no-explicit-any -- FIXME
  const messages: any = undefined;
  return (
    <div>
      <h1>MyComponent</h1>
      {/* eslint-disable-next-line @typescript-eslint/no-unsafe-member-access -- FIXME */}
      <p>Count: {count + messages.myMessage}</p>
      {/* eslint-disable-next-line @typescript-eslint/no-unsafe-member-access, eqeqeq -- FIXME my comment */}
      <p>Is Zero: {count == 0 ? messages.yes : messages.no}</p>
    </div>
  );
};

♻️ Fix them all at your own pace

You have explicit comments for all existing ESLint errors. And no more errors will be added.

You can now decide with your team how you will handle this legacy and how to fix them.

You can visualize the size of the task with another package I created: eslint-disabled-stats

CorentinDoue / eslint-disabled-stats

Compute statistics about the eslint rules disabled

eslint-disabled-stats

Compute statistics about the eslint rules disabled

It could be useful to track the correction of legacy eslint errors commented with a tool such as https://github.com/CorentinDoue/eslint-disable-inserter on your codebase.

The number of analyzed files and number of analyzed lines could be useful to track the evolution of the eslint errors compared to the evolution of the codebase.

Usage

$ npx eslint-disabled-stats -g -p "example/**/*.(js|ts)"

ℹ Analysing 2 files...
✔ Statistics computed

Rules disabled by rule:
• prefer-const: 1
• eqeqeq: 1
• curly: 1
• ALL_RULES: 1

Rules disabled by file:
• example/index.ts: 3
• example/legacy/legacy-file.js: 1

Total rules disabled:  4

Analysed files:        2
Analysed lines:        19

✔ Done

Options

The --pattern / -p flag allows specifying the glob pattern of files on which the statistics are computed The default pattern is **/*.(js|ts|jsx|tsx)
The --quiet / -q flag makes the console output lighter The details of the errors by…

View on GitHub

$ npx eslint-disabled-stats -g -p "example/**/*.(js|ts)"

ℹ Analysing 2 files...
✔ Statistics computed

Rules disabled by rule:
• prefer-const: 1
• eqeqeq: 1
• curly: 1
• ALL_RULES: 1

Rules disabled by file:
• example/index.ts: 3
• example/legacy/legacy-file.js: 1

Total rules disabled:  4

Analysed files:        2
Analysed lines:        19

✔ Done

Depending on the estimated time to fix the errors, you can fix them by batch or opt for the Boy Scout approach.

With the Boy Scout approach, you leave the codebase cleaner after your intervention.

You fix the errors as you encounter them in your daily work. You will fix them when you have context on the code, and this will reduce the risk of regression.

Make sure to not accept any Pull Request with commented ESLint errors to ensure a continuous improvement of your codebase quality. Some tools can help you to automate this part of the review, such as Danger JS.

You can also use IDE extensions such as TODO highlight or TODO Tree to make visible and track the remaining errors directly in your IDE.

🚀 Conclusion

Enforcing ESLint rules from day one is a pivotal step towards codebase excellence. By leveraging the strategies discussed in this article, you can pave the way for a cleaner, more maintainable codebase.

But the journey doesn't end here. If you have questions or want to share your experiences, don't hesitate to reach out on X/Twitter or GitHub

Power up your serverless application with AWS Lambda extensions

Corentin Doue — Tue, 06 Jun 2023 11:14:12 +0000

AWS Lambda service comes packed with impressive features that might still be new to you. One of them is the ability to run additional code side-by-side with the handler code. This feature is called extensions.

This article teaches you how to use extensions in order to create your own monitoring tool for lambdas without compromising their performances. The implementation uses NodeJs Lambda deployed with AWS CDK.

⚡ The Lambda extension feature

Let's start by a quick recap on Lambda extension. If you are already familiar with it feel free to jump to the creation part.

🤔 What is an Lambda extension?

An Lambda extension is a piece of code that runs in the same execution environment as the lambda handler.

It can be:

internal: it runs in the same process as the handler code.
external: it runs in a separate process.

In both case it shares the memory, timeout and ephemeral storage of the execution environment with the handler.

It can interact with the Lambda service to be notified on each event received by the handler and can receive telemetry events from the Lambda service.

An internal extension can be used to add capabilities directly to the handler code because it runs in the same process.

For example, an internal monitoring extension can override the http library to intercept and log all http calls.

Because it runs in a separate process, an external extension can add capabilities to the handler without impacting its performance.

For example, an external monitoring extension can send metrics to an external monitoring service after the handler has returned its response to avoid degrading an API performance.

🧩 How to use a Lambda extension

A Lambda extension comes as a Lambda layer.

To add an external extension, you only need to add the layer to your lambda. The extension process will be spawn automatically.

To add an internal extension, you need to add the layer to your lambda but you also have to configure the runtime to execute the entrypoint of the extension alongside with the handler. This configuration is done using runtime specific environment variable or wrapper script.

For example, for NodeJs, you can set the NODE_OPTIONS environment variable to --require /opt/my-extension.js or set the AWS_LAMBDA_EXEC_WRAPPER environment variable to the path of a wrapper script.

⚙️ How to create an extension

Let's create a monitoring extension that logs all http calls made by the lambda and send them to an external service for storage and visualisation.

We will use an internal extension to intercept all http calls and forward them to an external extension which sends a monitoring recap after each lambda invocation.

All the code following code is available on github:

CorentinDoue / lambda-extensions-example

Minimal repository to create and use both internal and external Lambda extensions with a NodeJs lambda

Lambda extensions example

Code of the article Power up your serverless application with AWS Lambda extensions

This example shows how to create a simple Lambda with a monitoring tool composed of:

an internal extension that logs all http calls made by the Lambda
an external extension that aggregate those logs and send them to an hypothetical monitoring tool

Install

Change the https://webhook.site/* urls in src/urls.ts to your own webhook urls.

 pnpm install
 pnpm cdk bootstrap
 pnpm run deploy

Test

pnpm integration-test

View on GitHub

🔍 The interceptor internal extension

The interceptor must be executed in the same process as the handler to be able to intercept all http calls.

📝 The interceptor code

For simplicity, we use msw as an out-of-the-box node interceptor.

// src/layers/monitorExtension/partial-interceptor.ts

import { rest } from 'msw';
import { setupServer } from 'msw/node';

const server = setupServer(
  rest.all('*', async req => {
    const url = req.url.toString();
    const method = req.method;
    console.log(`request intercepted in interceptor: ${method} ${url}`);

    return req.passthrough();
  }),
);

server.listen({ onUnhandledRequest: 'bypass' });

Then we forward the logs to the external extension with a local http call.

💡We must use the sandbox url because it's the only one authorized in the lambda execution environment. The port is free, so I chose a random one.

// src/layers/monitorExtension/interceptor.ts

import { rest } from 'msw';
import { setupServer } from 'msw/node';
import fetch from 'node-fetch';

console.log('Executing interceptor extension code...');

const LOG_EXTENSION_SERVER_URL = 'http://sandbox:4243';
const server = setupServer(
  rest.all('*', async (req, res, ctx) => {
    const url = req.url.toString();

    // Bypass the calls made by this code to the local server to avoid infinite loop
    if (url.includes(LOG_EXTENSION_SERVER_URL)) {
      return req.passthrough();
    }

    const method = req.method;
    const headers = req.headers;
    const body = await req.text();

    console.log(`request intercepted in interceptor: ${method} ${url}`);
    fetch(LOG_EXTENSION_SERVER_URL, {
      method: 'POST',
      body: JSON.stringify({
        url,
        method,
        headers,
        body,
        date: new Date().toISOString(),
      }),
    }).catch(error => console.error('error sending logs in interceptor extension', error));

    return req.passthrough();
  }),
);

server.listen({ onUnhandledRequest: 'bypass' });

🚀 Deploy the internal extension

First we need to make the extension code executable by the lambda. The simplest way is to use esbuild to bundle it in a single cjs file.

💡 That's basically what is done with the handler code under the wood by CDK when using the NodejsFunction construct. But here, we need to do it manually.

{
  "scripts": {
    "build:interceptor": "./node_modules/.bin/esbuild  ./src/layers/monitorExtension/interceptor.ts --bundle --outfile='./dist/layers/monitorExtension/interceptor.js' --platform=node --main-fields=module,main"
  }
}

Then we need to ship the code to the lambda thanks to a lambda layer.

With CDK, we can use the LambdaLayerVersion construct to create a layer which will ship all the content of a directory into the /opt folder of the lambda.

// lib/partial-stack.ts

export class AppStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const layer = new LayerVersion(scope, 'MonitorLayer', {
      code: Code.fromAsset('dist/layers/monitorExtension'),
    });

    const helloFunction = new NodejsFunction(this, 'Hello', {
      runtime: Runtime.NODEJS_18_X,
      handler: 'handler',
      entry: path.join(__dirname, `/../src/functions/hello/handler.ts`),
      layers: [layer],
    });
  }
}

Finally, we need to configure the lambda to execute the interceptor code alongside the handler code.

For that we will use a runtime specific environment variables. For NodeJs we need to set the NODE_OPTIONS environment variable to --require /opt/interceptor.js.

// lib/stack.ts

export class AppStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const layer = new LayerVersion(scope, 'MonitorLayer', {
      code: Code.fromAsset('dist/layers/monitorExtension'),
    });

    const helloFunction = new NodejsFunction(this, 'Hello', {
      runtime: Runtime.NODEJS_18_X,
      handler: 'handler',
      entry: path.join(__dirname, `/../src/functions/hello/handler.ts`),
      layers: [layer],
      environment: {
        NODE_OPTIONS: '--require /opt/interceptor.js',
      },
    });
  }
}

That's it! We have created our internal extension.

👀 Results

Lets deploy it on a simple lambda, that make http calls:

// src/functions/hello/handler.ts

import fetch from 'node-fetch';

export const hello = async () => {
  await fetch('https://webhook.site/87c3df17-c965-40d9-a616-790c4002a162');

  await fetch('https://webhook.site/87c3df17-c965-40d9-a616-790c4002a162', {
    method: 'POST',
    body: JSON.stringify({
      message: 'hello world',
    }),
  });

  return {
    statusCode: 200,
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      message: 'OK',
    }),
  };
};

export const handler = hello;

We can see those logs:

The interceptor code ran alongside the handler code and intercepted the http calls 🚀.

But we can see that it fails to forward them to the local http server because it doesn't exist. Let's create the external extension to fix that.

📊 The monitoring external extension

We want to aggregate the logs and send them to an external service without impacting the lambda performance. So we will use an external process, aka an external extension, to handle those logs.

📝 The external extension code

First, we want our external process to be aware of whats happening in the lambda. To do so, we can use the lambda extension API to:

register the process as an extension
subscribe to various lambda events or telemetry events

If you're not familiar with the lambda extension API, I recommend you to read the wonderful article Simplifying internal AWS Lambda APIs of Wakeem's World.

I published a node package inspired by his article to abstract the communication with the lambda extension API.

CorentinDoue / lambda-extension-service

SDK to easily implement nodejs lambda extension

Lambda extension service

SDK to easily build Lambda extensions in NodeJs and typescript.

Inspired by Simplifying internal AWS Lambda APIs

Installation

pnpm add lambda-extension-service

or if using yarn

yarn add lambda-extension-service

or if using npm

npm install lambda-extension-service

Usage

import { EventTypes, ExtensionAPIService, TelemetryEventTypes } from "lambda-extension-service";
(async () => {
  const extensionApiService = new ExtensionAPIService({ extensionName: "my-extension" });
  await extensionApiService.register([EventTypes.Invoke, EventTypes.Shutdown]);
  extensionApiService.onTelemetryEvent((event) => 
      console.log("Telemetry event received: ", JSON.stringify(event))
  );
  await extensionApiService.registerTelemetry([
      TelemetryEventTypes.Function,
      TelemetryEventTypes.Platform,
      TelemetryEventTypes.Extension,
  ]);

  while (true) {
      const event = await extensionApiService.next();
      console.log

…

View on GitHub

With it, we can easily register our external extension and subscribe to all the invocation events and be notified when the lambda is about to shutdown.

// src/layers/monitorExtension/partial-monitor.ts

import { EventTypes, ExtensionAPIService } from 'lambda-extension-service';

console.log('Executing monitor extension code...');

const main = async () => {
  const extensionApiService = new ExtensionAPIService({
    extensionName: 'monitor',
  });
  await extensionApiService.register([EventTypes.Invoke, EventTypes.Shutdown]);

  while (true) {
    const event = await extensionApiService.next();
    console.log('Received event', event);
  }
};

main().catch(error => console.error(error));

Now we can start a http server to receive the logs from the internal extension.

// src/layers/monitorExtension/logServer.ts

import { createServer } from 'http';
import { Log } from './types';

type LogServerOptions = {
  port: number;
};
export const listenForLog = (onLogReceived: (log: Log) => void, { port }: LogServerOptions = { port: 4243 }) => {
  const server = createServer(function (request, response) {
    if (request.method == 'POST') {
      let body = '';
      request.on('data', function (data) {
        body += data;
      });
      request.on('end', function () {
        try {
          onLogReceived(JSON.parse(body));
        } catch (e) {
          console.error('failed to parse logs', e);
        }
        response.writeHead(200, {});
        response.end('OK');
      });
    } else {
      console.error('unexpected request', request.method, request.url);
      response.writeHead(404, {});
      response.end();
    }
  });

  server.listen(port, 'sandbox');
  console.info(`Listening for logs at http://sandbox:${port}`);
};

And then aggregate and send them to an external service on each new invocation or on shutdown.

// src/layers/monitorExtension//monitor.ts

import { EventTypes, ExtensionAPIService } from 'lambda-extension-service';
import { Log } from './types';
import { LogAggregator } from './logAggregator';
import { listenForLog } from './logServer';
import { LambdaContext } from './lambdaContext';
import { forwardLogs } from './forwardLogs';

console.log('Executing monitor extension code...');

const main = async () => {
  const logAggregator = new LogAggregator();
  const lambdaContext = new LambdaContext();
  const onLogReceived = (log: Log) => {
    logAggregator.addLog(log, lambdaContext.getRequestId());
  };
  listenForLog(onLogReceived);

  const extensionApiService = new ExtensionAPIService({
    extensionName: 'monitor',
  });
  await extensionApiService.register([EventTypes.Invoke, EventTypes.Shutdown]);

  while (true) {
    const event = await extensionApiService.next();
    const lastContext = lambdaContext.getContext();
    lambdaContext.updateContext(event);

    if (lastContext !== undefined) {
      await forwardLogs({
        context: lastContext,
        logs: logAggregator.getLogs(lastContext.requestId),
      });
    }
  }
};

main().catch(error => console.error(error));

🚀 Deploy the external extension

First, we also need to make the extension code executable by the lambda. But this time it should be a standalone executable file.

💡 We reuse the same esbuild command as with the interceptor extension, but we need to add a shebang to make the file executable with the --banner:js='#!/usr/bin/env node' option.

{
  "scripts": {
    "build:monitor": "./node_modules/.bin/esbuild  src/layers/monitorExtension/monitor/index.ts --bundle --outfile='./dist/layers/monitorExtension/monitor.js' --platform=node --main-fields=module,main --banner:js='#!/usr/bin/env node'"
  }
}

Then we need to ship the code to the lambda thanks to a lambda layer.

💡 By outputting the build of the monitor code in the same folder as the interceptor code (./dist/layers/interceptorExtension), the layer will ship both files without changing our CDK configuration.

Finally, we need to make the lambda execute the monitor process.

By default, the lambda service will try to execute all files in the /opt/extensions folder when starting the lambda environment.

So let's add a bash script into ./dist/layers/monitorExtension/extensions to start the monitor code.

# dist/layers/monitorExtension/extensions/monitor

#!/bin/bash
set -euo pipefail

OWN_FILENAME="$(basename $0)"
LAMBDA_EXTENSION_NAME="$OWN_FILENAME" # (external) extension name has to match the filename
NODE_OPTIONS="" # Needed to reset NODE_OPTIONS set by Lambda runtime. Otherwise, the internal interceptor extension will be loaded in the external process too.

exec "/opt/${LAMBDA_EXTENSION_NAME}.js"

This script will be shipped to /opt/extensions/monitor and will be executed automatically by the lambda service.

💡Note that the environment variables of the lambda are shared between processes. So the external process will have the NODE_OPTIONS set to require the interceptor code. We need to reset NODE_OPTIONS to avoid this and cause the interceptor code to be load twice and intercept the monitor calls.

That's it! We finished our monitoring extension.

👀 Results

Let's deploy it again on our previous lambda. Here are the final logs:

🟠 The lambda service inits the lambda execution environment
🟢 The monitoring external extension initializes
🟣 The interceptor internal extension initializes
🟠 The lambda service acknowledges the initialization of the monitoring extension
🟠 The lambda service starts the first event handling
🟣 The interceptor internal extension intercepts the two http calls
🟠 The lambda service ends the first event handling
🟠 The lambda service starts the second event handling
🟢 The monitoring external extension sends the logs of the first event to the external monitoring service
🟣 The interceptor internal extension intercepts the two http calls
🟠 The lambda service ends the first event handling
🟢 At lambda shutdown, the monitoring external extension sends the logs of the second event to the external monitoring service

On the fake external monitoring service, we received aggregated logs by event 🥳:

⏭️ Test it on your own AWS account using the dedicated repository:

CorentinDoue / lambda-extensions-example

Minimal repository to create and use both internal and external Lambda extensions with a NodeJs lambda

Lambda extensions example

Code of the article Power up your serverless application with AWS Lambda extensions

This example shows how to create a simple Lambda with a monitoring tool composed of:

an internal extension that logs all http calls made by the Lambda
an external extension that aggregate those logs and send them to an hypothetical monitoring tool

Install

Change the https://webhook.site/* urls in src/urls.ts to your own webhook urls.

 pnpm install
 pnpm cdk bootstrap
 pnpm run deploy

Test

pnpm integration-test

View on GitHub

Conclusion

Now know how to create internal and external extensions for lambda. It's a powerful way to implement and share support tools for your lambdas without impacting their performances and without directly modifying the handler code written by other developers.

Feel free to share your thoughts and questions in the comments below. I will be happy to answer them.

5 control points to implement serverless integration tests like a boss 👔

Corentin Doue — Wed, 08 Feb 2023 15:09:52 +0000

Serverless applications are composed of many services interacting with one another. A lot more than traditional web applications.

Service configurations replace some application code. Parts of the application are black box managed by a cloud provider. Those configuration and their effects can only be tested with integration tests.

That’s why it is crucial for all serverless applications to have relevant integration test coverage.

Without integration tests, you expose your application to hard-to-fix bugs because they don’t come from your application code but from a misconfiguration.

🤔 Intent

There is no unique perfect way to implement integration tests. It depends on the context such as the application or the team.

But after 2 years of experimentation on serverless integration tests on several projects, I have seen some patterns emerge. I summed them up in 5 control points.

I will illustrate each one of them with OK ✅ and KO ❌ real-life examples. Those examples are meant to help you forge your own opinion on the subject and help you implement your own integration tests according to your context.

🎯 The 5 controls points

Write one test case per integration ⛓️
Test deployed resources 🚀
Keep the feedback loop below 1 min ♻️
Don’t create side effects 💥
Run integration tests in CI before merge 🚦

Let's detail them.

1. Write one test case per integration ⛓️

To ensure good coverage without writing too many tests. Determine what services are communicating and test every interaction once.

✍️ Real-life example

Let’s take an async lambda triggered by event-bridge which gets an item in dynamoDb and then put a generated file in a S3 bucket.

There are 5 configurations to tests:

invoke: SQS ➡️ Lambda
getItem: Lambda ➡️ DynamoDB
putObject: Lambda ➡️ S3
retry: SQS ➡️ Lambda
onMaxRetry: SQS ➡️ DLQ

❌ Bad

Forget to test that a wrong message ends up in the DLQ

⇒ ❌ The retry and onMaxRetry configurations are not tested.

Test that different files are generated if you put different fixtures in the table

⇒ The first test tested invoke & getItem & putObject

⇒ ❌ Subsequent tests add no value because they will test again the same configurations and an integration test shouldn't test all business logic.

⇒ ❌ Moreover it slows down your test phase because integration tests are slower than unit ones. If you write 10 integration test cases per lambda each time 2 are required, your whole testing suite will be 5 times slower than it could be.

💡 You should use unit tests to test the business logic

✅ Good

Two test cases:

A first test with a fixture item in the table, a real event that triggers the lambda, and an assertion on the object creation in S3

⇒ ✅ Test invoke & getItem & putObject

A malformed event that triggers the lambda and ends up in the DLQ after 3 retries

⇒ ✅ Test retry & onMaxRetry

2. Test deployed resources 🚀

In order to test the effect of a configuration on a service, it must be deployed in real conditions.

✍️ Real-life example

❌ Bad

Mock / emulate a service locally

⇒ ❌ Open source mock can’t perfectly implement the behavior of a cloud provider service and keep up with new features.

Import and execute a lambda handler locally in the test

⇒ ❌ the upstream event is not tested.

⇒ ❌ the lambda configuration is not tested.

✅ Good

Deploy your resources in a real stack and let your test code interact with the interfaces

⇒ ✅ The configuration of the upstream service is tested.

⇒ ✅ The configuration of the lambda is tested.

⇒ ✅ The code of the lambda is tested.

⇒ ✅ The configurations of the downstream services are tested.

💡 For lambda interacting with external services, use real services if possible. If they are not stable or too expensive, consider replacing the url of the external service with the url of a mock

3. Keep the feedback loop below 1 min ♻️

Tests must be easy to write and debug if you want your team to write and maintain them.

A new change must be deployed as quickly as possible to be tested without delay.

A test must be quick to run without pain.

✍️ Real-life example

❌ Bad

Use separate stacks to e2e and integration testing

⇒ ❌ Keeping two stacks synced is a pain for developers.

💡 This doesn’t mean you should use the same resources for your integration tests and your e2e usage. But they should be in the same stack to synchronize the deployment.

Trigger a full deployment between two development iterations

For example, use sls deploy each time you change your code.

⇒ ❌ Those deployments are too long to iterate quickly.

✅ Good

Use real-time deployment tools

Serverless frameworks provide tools to keep your dev stack synced with your code in real-time:

✅ sst start for SST
✅ cdk watch for CDK
⚠️ sls deploy -f myFunctionName for serverless framework

ℹ️ sls deploy -f requires you to manually trigger the deployment between each change

4. Don’t create side effects 💥

To run your tests in parallel and without breaking your development environment, you need to isolate the tests from the rest of the app.

✍️ Real-life example

❌ Bad - Code Level

Require an empty state to run

// ❌ Requires running the tests synchronously
beforeEach(async () => truncateAllData());

it('creates a user', async () => {
  await axios.post(`${URL}/users`, user);

  const users = await User.findAll();
  // ❌ Breaks if another user is in the database
  expect(users).toEqual([user]);
});

✅ Good - Code Level

Create isolated data

const userId = uuidV4(); // ✅ No collision const user = { userId, /_ ... _/ };

afterEach(async () => deleteUser(userId)); // ✅ Delete only test data

it('creates a user', async () => {
  await axios.post(`${URL}/users`, user);

  const users = await User.findAll();
  // ✅ Another user from another test or e2e data can be in the list
  // without impacting the test result
  expect(users).toContainsEqual(user);
});

❌ Bad - Infrastructure level

Let async test events propagate and break downstream components in event driven architectures

ℹ️ This only concerns event driven architectures

⇒ Test 1 will create events that trigger the lambda 2 & 3 in an uncontrolled context.

⇒ ❌ If want to control the context of lambda 2 & 3 while testing lambda 1, the complexity explodes with the size of the async flow 🤯. It becomes an e2e test, that's not what we want here.

⇒ ❌ If you let the events propagate, you will end with lots of “normal” errors that will prevent you to detect real errors

⇒ ❌ If the async flow can affect upstream data, race conditions will appear in the tests

✅ Good - Infrastructure level

Breaks the async flows in event driven architectures to isolate tests

ℹ️ This only concerns event driven architectures

⇒ Duplicate instances of event services and plug the lambda into the rights services thanks to the environment variables.

⇒ ✅ The side effects created by a lambda are not propagated to the other lambdas.

⚠️ This can be a little scary but the other alternatives are: letting the event propagate (❌ cf above) or modifying production code to filter out tests events

💡 I’m working on the automation of the duplication and the wiring of environment variables to abstract this away from the developer. Stay tuned 🔊

5. Run integration tests in CI before merge 🚦

Integration tests break often. And it’s good, they catch for you all sorts of mistakes. You need to run them as soon as possible in your automated tests suite.

Even if it requires more configuration at first, take the time to make them run in your CI for every pull request/merge request.

You also need to think about keeping your integration test job fast enough to be awaited without pain.

✍️ Real-life example

❌ Bad

Run automated integration tests in preprod/staging

Write a new service integration & the corresponding test on your developer environment
Push your code & open a pull request/merge request
1. run units tests in CI
2. review code
After the merge on the staging branch, deploy the code on the staging environment
Run all integration tests on the staging environment

⇒ Developers will not run every test on every change in their developer environment, sometimes unexpected side effects will break unexpected tests.

⇒ ❌ In that case the tests will fail in the staging branch which will impact all other developers who need this environment as stable as possible.

⇒ ❌ The fix must be done quickly & is expensive because it requires starting the development flow from scratch: create a new branch, make the fix, test it properly, open a new pull request, pass the CI, pass the code review and then merge.

✅ Good

Deploy and run integration test on every pull request/merge request on a dedicated environment

ℹ️ Serverless heavily relies on infrastructure as code and pay-as-you-use resources. Therefore it’s easy and cheap to deploy multiple environments

Write a new service integration & the corresponding test on your developer environment
Push your code & open a pull request/merge request
1. run units tests in CI
2. deploy a dedicated test environment & run all integration tests on it
3. review code
After the merge on the staging branch, deploy the code on the staging environment

⇒ ✅ If a test fails, you only need to push a fix in the existing branch.

⇒ ✅ It checks that the infrastructure as code is working properly and you are sure that staging deployment will succeed.

💡 To speed up test environment deployment, you can work with a pool of test stacks and only deploy changes on them.

⇒ More details about how to implement it:

Blazing fast CI for serverless integration tests

Corentin Doue for Serverless By Theodo ・ Mar 22 '22

#aws #serverless #testing #node

theodo / test-stack-orchestrator

Orchestrate your serverless test stacks

stack-orchestrator

An easy way to manage the availability of multiple serverless stacks.

Use case

This API helps to implement integration or e2e testing per feature branch It enables to request a stack for a specific branch, deploy the app on this stack, test on it, then release the stack for the next feature branch.

A stack is a group of ressources that could be identified by a string, its stack name.

If you use the serverless framework you can deploy your app for a specific stack using serverless deploy --stage $stackName. Most of the ressources created will be marked with the stackName.

Routes

Postman documentation

Request stack

Gets an available stack, locks it and return its stack name and last deployed commit.

The returned stack is

one of the same branch if it exists and is available
the older stack available (based on the last requested date)
a…

View on GitHub

Conclusion

I hope those 5 control points will help you to write better tests and to have a better development experience.

Keep them in mind and try to apply them as soon as possible in a project life cycle, tests are a pain to implement/refactor afterward. The bad examples mostly come from projects I've worked on. I know how painful it is to have to refactor tests because your team is stuck. Don't wait until it's too late 😉

Feel free to share your thoughts and questions in the comments below. I will be happy to answer them.

Stop using a local environment to develop Serverless applications

Corentin Doue — Thu, 07 Apr 2022 14:00:26 +0000

We, developers like to know the soonest when something is wrong with our code to fix it before it becomes officially a bug. That's why we use lots of tools to run our code as soon as possible in the real world. A replica of the production environment that runs locally on our machine is often the best solution to achieve this goal. But Serverless introduces new paradigms that question the necessity of a complete local environment.

TL;DR

Only deploy to your development environment in the cloud once your code is completely ready. To catch bad code earlier, the Test Driven Development (TDD) is your best ally. Theses tricks make it more powerful with lambdas:

Use copies of real lambda events as input of your unit and integration tests
Mock Cloud Provider SDK in your unit tests
Don't mock and interact with real services in your integration tests

Local development environment - The solution to develop with legacy architectures

Note: all schema use AWS services but the article is also valid for other Cloud providers. I personally applied it to CGP and AWS serverless projects

🤔 What problems does the local environment solve?

When you develop your app, you want to test it and be sure it will behave the same on the production server. You can blindly duplicate your production server and work on it. But if your team is composed of several developers you will need to share the development server or duplicate it. The first scenario can create conflicts on the server and the second is expensive. Neither of them is scalable.

Moreover, you need a good internet connection to stay synchronized with your server. So you will try to replicate the production server on each developer's computer and probably add some developers tools to ease the developer experience: you created some local environments. Each local environment is a completely autonomous environment that should behave as close as possible like the production server.

🔄 What is the development workflow with a local environment?

Working with a local environment means setting it up the first time. This job can be long but is only done once. Then you can completely develop and validate a feature on your machine, from unit to e2e tests (manual or automated). Those checks can be done offline, with no waiting time. Once the feature is validated you deploy it on a production-like server for the final checks. The feature is ready to be in production.

✅️ What makes a local environment a good development environment?

A good development environment must be easy to install for all developers. Lots of tools such as docker have been created to ease the installation of a whole environment locally.

A good development environment must also be production-like. If something is wrong on the production server and can't be reproduced locally, you have two choices:

Fix the behavior gap between your local machine and the server.
Blindly fix the code assuming the server behavior.

Both are painful and time-consuming but hopefully rare with classical architectures. The local environments tend to be close to production.

Developers have no time to lose, a good development environment needs to be fast. Local environments generally add some tools such as hot-reloader to speed up the feedback loop of the developers.

Each developer needs a development environment, they need to be cheap to scale with the team size. With local environments, the environment is on the developer's computer so there is no extra cost after equipping the developer.

Old Habits die hard - A local environment for serverless apps is inefficient

If like most developers, you always have worked in a local environment. Old habits die hard. So when it comes to a Serverless project, you will try to reproduce the server on your machines. It will allow you to keep the same development workflow as always. Once the environment is set up you can develop and test a feature end to end.

💥 To completely emulate a whole cloud provider is impossible

The server you try to duplicate locally is not an empty Linux virtual machine with ten programs, it's a whole cloud provider. The cloud provider services are not made to be installed on personal computers so you need to use services mock (which are often maintained by the community).

Those mocks are by definition not real services so there are some behavior differences between the local environment and the cloud provider. For example, AWS API Gateway emulated by serverless-offline doesn't handle VTL locally the same as AWS does.

Each time a difference creates a bug, you will have to fix the mock or blindly fix the code assuming the real service behavior. As multiple services are used, the risk is much higher than with a monolith architecture.

♻️🤯 An environment based on multiple services emulations is a nightmare to maintain

The risk is also increased because the community-maintained mocks are often outdated compared to the latest version of the service developed by an army of developers. Some new services could also not be available.

Moreover, each time a new service is used in the project, the local environment must be updated and every developer must set it up again. This not uncommon in Serverless projects that tend to use a lot of cloud provider services.

Finally, to maintain your codebase over time, if you update to the latest version of your favorite service, you need to ensure your code still works. But you also need to ensure that the service mock has also been updated and works as expected.

Remote development environment - A slower development feedback loop but an exact replication of production

💡 Why should you use a remote environment?

Because most serverless services have on-demand pricing and cloud providers support several IaC (Infrastructure as Code) tools, creating a full cloud environment for each developer is now cheap and easy. It allows the developers to validate their features on an exact replicate of the production infrastructure and to never encounter any behavior differences between their development environment and the production.

The setup of an environment is now a part of the deployment process. You can set up your developer's environment by fetching the code, entering your name in an environment variable, and deploying it.

🔄 What is the naive development workflow with a remote environment?

To develop a feature you will first locally write your function code then test it with your favorite test framework. When you want to validate it in integration with another service or end to end, you need to deploy it. If something goes wrong you need to correct it and deploy it again.

⚠️ What are the drawbacks?

It's rare to develop right the first time. So you will need to deploy multiple times before your feature will be production-ready. Even if a serverless deployment is relatively short (between 30secs and 5mins) compared to a monolithic deployment, waiting for it ten times a day will make it feel a lot longer. As developers rather fix bugs than wait, they could fall back onto the local environment option.

Moreover, all offline development is merely impossible.

Get the best of both worlds - Hybrid workflow to the rescue

🧫 What could be optimized?

Comparing the drawbacks of a serverless local environment and a cloud environment, it appears easier to try to reduce the waiting time than to perfectly replicate a whole cloud provider on a local machine.

w ai t in g T im e = n u mb er O f De pl oy m e n t s * d e pl oy m e n tT im e

🚀 How to implement a hybrid development workflow?

1️⃣ Deploy an empty function that logs all the events

Serverless functions are functions triggered by an event (an HTTP call, a message from another service, …). The first step is to configure its trigger regardless of what the function is supposed to do. This could be achieved with an empty function. Making it log its input really helps to understand the events.

You deployed once.

2️⃣ Copy all possible events locally

The function does nothing but you can trigger it with all the events it's supposed to handle once the feature is done. You can use e2e means to trigger it because your function is deployed in a production-like environment. Get the logs and copy all different events locally to be able to replay them later.

3️⃣ Test-driven develop your function in integration with real services

You got real events to replay. You can put them as input of your function in unit or integration tests. You will be able to develop your function knowing its behavior for each event you got without any effort.

Your function probably makes side effects interacting with other services. Handle them differently according to the test:

In unit tests: Mock all cloud providers' SDKs or client interface to check your code behave as expected, assuming all external services it uses.
In integration tests: Let the application interact with the real services. Those tests are really powerful because it test the production behavior with a short feedback loop

4️⃣ Deploy your feature

Your function is tested unitary and in integration with the rest of the world. You can safely deploy it and expect your e2e validation not to fail.

You deployed only twice.

If the e2e validation fails you can copy the event of the failure case and add it to your list of events. You will be able to replay it locally and the debug will be easier.

Conclusion - Use hybrid workflow as your default development workflow for Serverless application

You now know that you should stop using a local stack to develop serverless applications. You can create a real development stack for each developer. You should use TDD to work around the drawbacks of remote stacks. You should use both units and integration tests to ensure your code work before deploying it.

What is your current development workflow? Have you some tricks to improve mine? All feedbacks are welcome.

Implementing integration tests is challenging. I specifically wrote a series of articles on the subject:

How to test your serverless app in integration with real AWS services

Corentin Doue for Serverless By Theodo ・ Jan 22 '22

#aws #serverless #testing #node

Blazing fast CI for serverless integration tests

Corentin Doue for Serverless By Theodo ・ Mar 22 '22

#aws #serverless #testing #node

Check them out.

Blazing fast CI for serverless integration tests

Corentin Doue — Tue, 22 Mar 2022 15:56:53 +0000

Running integration tests in a CI can be quite challenging. It's even more challenging with serverless applications because they interact with lots of services.

In previous articles I explained how to test your serverless app in integration with real AWS services.

How to test your serverless app in integration with real AWS services

Corentin Doue for Serverless By Theodo ・ Jan 22 '22

#aws #serverless #testing #node

5 control points to implement serverless integration tests like a boss 👔

Corentin Doue for Serverless By Theodo ・ Feb 8 '23

#aws #serverless #testing #node

This article describes one way to implement the logical next step: running those integration tests in your CI for each feature branch of your team as fast as possible.

TL;DR

Set up a test account with multiple production-like copies of your application. This pool of stacks will be available for the CI to speed up the setup of the integration tests. Dimension it according to your team size and CI frequency.

For each CI job:

Get an existing production-like stack and dedicate it to the job.
Update the stack. It's far quicker than creating it.
Get the environment variables of the resources of the stack.
Run the tests in integration with the ressources of the stack.
Release the stack for the next job on success or failure.

# 1. Request an available stack and lock it
requestStackResult=$(curl --location --request POST 'https://stack-orchestrator.theodo.org/requestStack' \
   --header "X-API-Key: $ORCHESTRATOR_KEY" \
   --header "Content-Type: application/json" \
   --data-raw "{
     \"branch\": \"$GIT_BRANCH\"
   }")
stackName=$(echo "$requestStackResult" | jq -r .stackName)

echo "$stackName will be used"

# 2. Deploy the stack
yarn sls deploy --stage "$stackName"

# 3. Get the environment variables
cfnOutputs=$(aws cloudformation list-exports)
get_cfn_output_value() {
  echo "$cfnOutputs" |
    jq -r --arg cfnOutputName "$1" \
      '.Exports[] | select(.Name==$cfnOutputName) | .Value'
}

echo "TABLE_NAME=$(get_cfn_output_value "table-name-$stackName")" >> .env
echo "BUS_NAME=$(get_cfn_output_value "bus-name-$stackName")" >> .env

# 4. Execute the tests
yarn test:integration

# 5. Release the stack
curl --location --request POST 'https://stack-orchestrator.theodo.org/releaseStack' \
  --header "X-API-Key: $ORCHESTRATOR_KEY" \
  --header "Content-Type: application/json" \
  --data-raw "{ \"stackName\": \"$stackName\" }"

The plan

You can run locally the tests you created following the previous article. The test interact with real AWS services of your dev account.

You now want to execute the tests in your CI to protect your main branch. Your team works on multiple branches at the same times. You will have to orchestrate multiple test stacks and then execute the tests against the right stack.

1. Orchestrate multiple stacks

A CI job can be triggered on different branches that have different services and tests. Moreover, the CI could have multiple jobs concurrently. Therefore, each job must have a dedicated stack to use for its tests. The obtention and update of these stacks must be as efficient as possible.

Having multiple stacks is not a problem. Most serverless frameworks can identify which stack to deploy with a prefix that is added on most resources.

For example with Serverless framework you can use

$ yarn serverless deploy --stage test-1
$ yarn serverless deploy --stage test-2

to deploy your application twice.

If you only use serverless services with on-demand pricing, having one, two, or ten stacks will not increase your bill.

But deploying a whole new stack is slow. It shouldn't be done for each CI job. Instead, you could reuse a stack from one job to another. The deployment will be a lot faster because it will only deploy the difference between the last time the stack has been used and the state of the feature branch.

A job must be able to know what stack it should use. A job mustn't be able to choose the same stack which is used by another job to avoid conflicts.

I developed a small API to handle the orchestration of those stacks.

theodo / test-stack-orchestrator

Orchestrate your serverless test stacks

stack-orchestrator

An easy way to manage the availability of multiple serverless stacks.

Use case

A stack is a group of ressources that could be identified by a string, its stack name.

If you use the serverless framework you can deploy your app for a specific stack using serverless deploy --stage $stackName. Most of the ressources created will be marked with the stackName.

Routes

Postman documentation

Request stack

Gets an available stack, locks it and return its stack name and last deployed commit.

The returned stack is

one of the same branch if it exists and is available
the older stack available (based on the last requested date)
a…

View on GitHub

It enables to:

Request an available stack and lock it.
Release the stack when the job is done.

# 1. Request an available stack and lock it
requestStackResult=$(curl --location --request POST 'https://stack-orchestrator.theodo.org/requestStack' \
  --header "X-API-Key: $ORCHESTRATOR_KEY" \
  --header "Content-Type: application/json" \
  --data-raw "{
    \"branch\": \"$GIT_BRANCH\"
  }")
stackName=$(echo "$requestStackResult" | jq -r .stackName)

echo "$stackName will be used"

# 2. Deploy the stack
yarn sls deploy --stage "$stackName"

# 3. Execute the tests
# ...

# 4. Release the stack
curl --location --request POST 'https://stack-orchestrator.theodo.org/releaseStack' \
  --header "X-API-Key: $ORCHESTRATOR_KEY" \
  --header "Content-Type: application/json" \
  --data-raw "{ \"stackName\": \"$stackName\" }"

Note: The stack orchestrator API also enables you to store the last commit deployed of each stack. Then you can deploy only the code affected since the last deployment.

2. Run your tests in interaction with the right ressources

Multiple stacks mean multiple services. Each CI job must configure its tests to run in interaction with its corresponding stack.

The tests use environment variables to identify the resources to use. Those variables are loaded from a .env file.

Let's assume being in CI job which has requested and deployed the stack test-1. You need to build a .env with the TABLE_NAME and BUS_NAME of the DynamoDB table and EventBridge bus of the stack test-1.

Lets use CloudFormation Outputs to easily gets those values.

"Outputs" : {
  "DynamoDbTableName" : {
    "Value" : { "Ref": "DynamoDbTable" },
    "Export" : { "Name": "table-name-test-1" }
  },
  "EventBusName" : {
    "Value" : { "Ref": "EventBus" },
    "Export" : { "Name": "bus-name-test-1" }
  }
}

The name of the exports must contain the stack name. If you use the Serverless framework, use the stage variable: "table-name-${sls:stage}".

After the deployment of a stack, you can now get the names of the DynamoDB table and the EventBridge bus of this stack using the list exports command of the AWS CLI.

cfnOutputs=$(aws cloudformation list-exports)
get_cfn_output_value() {
  echo "$cfnOutputs" |
    jq -r --arg cfnOutputName "$1" \
      '.Exports[] | select(.Name==$cfnOutputName) | .Value'
}

echo "TABLE_NAME=$(get_cfn_output_value "table-name-$stackName")" >> .env
echo "BUS_NAME=$(get_cfn_output_value "bus-name-$stackName")" >> .env

The execution of the test will be similar to locally. No further argument is required.

yarn test:integration

A complete bash script

# 1. Setup common environment variables
cp .env.ci.test .env.test

# 2. Get an available test stack
requestStackResult=$(curl --location --request POST 'https://stack-orchestrator.theodo.org/requestStack' \
  --header "X-API-Key: $ORCHESTRATOR_KEY" \
  --header "Content-Type: application/json" \
  --data-raw "{
    \"branch\": \"$GIT_BRANCH\"
  }")
stackName=$(echo "$requestStackResult" | jq -r .stackName)
lastDeployedCommit=$(echo "$requestStackResult" | jq -r .lastDeployedCommit)

echo "$stackName will be used"
echo "STAGE=$stackName" >>.env.test

release_stack() {
  curl --location --request POST 'https://stack-orchestrator.theodo.org/releaseStack' \
    --header "X-API-Key: $ORCHESTRATOR_KEY" \
    --header "Content-Type: application/json" \
    --data-raw "{
        \"stackName\": \"$stackName\"
    }"
}

# 3. Deploy stack
yarn sls deploy --stage "$stackName"
# Release stack and exit script if deploy failed
if [ $? -ne 0 ]; then
  echo "Deploy failed"
  release_stack
  exit 1
fi
# Set last deployed commit
curl --location --request POST 'https://stack-orchestrator.theodo.org/setLastDeployedCommit' \
  --header "X-API-Key: $ORCHESTRATOR_KEY" \
  --header "Content-Type: application/json" \
  --data-raw "{
      \"stackName\": \"$stackName\",
      \"lastDeployedCommit\": \"$(git rev-parse HEAD)\"
  }"


# 4. get environment variables of the stack
cfnOutputs=$(aws cloudformation list-exports --profile test-profile)
get_cfn_output_value() {
  echo "$cfnOutputs" |
    jq -r --arg cfnOutputName "$1" \
      '.Exports[] | select(.Name==$cfnOutputName) | .Value'
}

echo "TABLE_NAME=$(get_cfn_output_value "table-name-$stackName")" >>.env.test
echo "BUS_NAME=$(get_cfn_output_value "bus-name-$stackName")" >>.env.test

# 5. Run migrations only if there is new ones
if git diff --name-only "$lastDeployedCommit" HEAD | grep migrations; then
   yarn migrate --stage "$stackName"

   # Release stack and exit script if migration failed
   if [ $? -ne 0 ]; then
     echo "Migrate failed"
     release_stack
     exit 1
   fi
fi

# 6. Run integration tests
yarn test:integration

# Release stack and exit script if tests failed
if [ $? -ne 0 ]; then
  echo "Test failed"
  release_stack
  exit 1
fi

# 7. Release the stack
release_stack

Conclusion

You now know how to test your serverless application in integration with real AWS services before each merge on your main branch. Those tests are quite powerful. I use integration tests running in my CI for 6 months and it prevented at least a dozen of regressions and bugs.

As your code grows the integration test job can become slower. Depending on your architecture, lots of micro enhancements can be added to improve the speed of the job such as parallelism or deploy only affected code.

Feedbacks are welcome 😃

How to test your serverless app in integration with real AWS services

Corentin Doue — Sat, 22 Jan 2022 12:22:26 +0000

Integration tests are part of the well known test pyramid. They test the interaction between services. On average Serverless applications are composed of more services than more traditional architectures. It's, therefore, more important to implement integration tests in a Serverless application.

This article is a step-by-step guide to adding integration tests to your serverless project.

All examples are written in Typescript, using Jest testing engine, on an AWS serverless stack. They can easily be used in javascript projects. Core concepts also apply to other languages, test frameworks, and cloud providers.

Update february 2023: I also wrote a technology-agnostic checklist to help implement your own integration tests according to your context. Check it out.

5 control points to implement serverless integration tests like a boss 👔

Corentin Doue for Serverless By Theodo ・ Feb 8 '23

#aws #serverless #testing #node

TL;DR

This article uses a simple serverless application as example.

The application is composed of one lambda which:

Gets an item from DynamoDB.
Then publishes an event in EventBridge.

The corresponding integration test asserting the lambda behaves as expected will:

Push fixture data in the DynamoDB
Execute the application code in integration with the real DynamoDB and EventBridge services
Assert the event have been published

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { InvokeCommand, LambdaClient } from '@aws-sdk/client-lambda';
import { DeleteCommand, DynamoDBDocumentClient, PutCommand } from '@aws-sdk/lib-dynamodb';
import { EventBridge } from 'sls-test-tools';

const ddbDocClient = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const lambdaClient = new LambdaClient({});

it('gets the item and publishes it as an event', async () => {
  // Arrange
  await ddbDocClient.send(
    new PutCommand({
      TableName: process.env.TABLE_NAME,
      Item: {
        PK: 'MyPartitionKey',
        SK: 'MySortKey',
        payload: 'MyPayload',
      },
    }),
  );
  const eventBridge = await EventBridge.build(process.env.BUS_NAME);
  // Act
  const { FunctionError } = await lambdaClient.send(new InvokeCommand({ FunctionName: 'my-lambda' }));
  expect(FunctionError).toBeUndefined();
  // Assert
  const events = eventBridge.getEvents();
  expect(events).toHaveEventWithSource('my-lambda');
  // Clean
  await ddbDocClient.send(
    new DeleteCommand({
      TableName: process.env.TABLE_NAME,
      Key: {
        PK: 'MyPartitionKey',
        SK: 'MySortKey',
      },
    }),
  );
  await eventBridge.destroy();
});

Use a specific test config

The first thing to do is to configure your test framework.

It's easier to use the same test framework for your unit and integrations tests. But unit and integration tests must run separately and with different configurations.

Add a jest.integration.config.ts at the root of your project next to your jest.config.ts. It can extend the jest.config.ts but but it overrides some parameters.

Don't use mocks

Integration tests interact with real services. If you have mocked some of them for unit test purposes, they should be ignored.

// jest.integration.config.ts
export default {
  ...commonJestConfig,
  modulePathIgnorePatterns: ['_mock_'],
};

Use specific files

Unit and integration tests must be executed separately, the easiest way to do it is to use different files for the integrations test: myLambda.integration-test.ts or myLambda.integration-spec.ts or __integrationTests__/myLambda.ts.

The integration configuration must only execute those files.

// jest.integration.config.ts
export default {
  ...commonJestConfig,
  testRegex: '(/__integrationTests__/.*|(\\.|/)(integration-test|integration-spec))\\.(jsx?|js?|tsx?|ts?)$',
};

Increase the timeout

Integration tests require on average more time to run than unit tests because they often wait for the responses of external services.

// jest.integration.config.ts
export default {
  ...commonJestConfig,
  testTimeout: 10000, // 10s
};

Add a setup file

A setup file contains some code that runs before test initialization. It will be useful in the next part to load environment variables.

// jest.integration.config.ts
export default {
  ...commonJestConfig,
  setupFiles: ['<rootDir>/test/setup_integration.ts'],
};

Run sequentially

Integration tests create side effects. It's recommended to run them sequentially to avoid mixing the side effects.

yarn jest -c jest.integration.config.ts --runInBand

Note: runInBand can only be specified as a CLI param, it cannot be configured using jest.integration.config.ts

Interact with a real AWS Stack

As explained at the beginning of this article, integration tests aim at testing your application’s code in integration with your ecosystem of services. To achieve this, your tests should run against an actual AWS account and interact with AWS services within. Running your test will therefore require the use of your existing dev stack or a specific test stack.

Nice to have: Set up a test stack

If you want to fully control the behavior of the stack used for the integration tests I recommend you set up a dedicated stack per developer. The data on this stack will only be test fixtures and can be wiped out between tests.

Serverless applications generally have on-demand pricing, therefore if you use infrastructure as code you can easily create an exact copy of your stack. The resulting cost is the same running all tests on one stack or splitting the tests in separate stacks per developer.

The main drawback of this implementation is that you need to deploy two stacks after each change.

Use AWS SDK

The AWS SDK makes it easy to interact with AWS services from your local computer.

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, PutCommand } from '@aws-sdk/lib-dynamodb';

const ddbDocClient = DynamoDBDocumentClient.from(new DynamoDBClient({}));

it('myTest', async () => {
  await ddbDocClient.send(
    new PutCommand({
      TableName: 'MyTable',
      Item: {
        PK: 'MyPartitionKey',
        SK: 'MySortKey',
      },
    }),
  );
});

If you execute this test locally, it would put an item in the real DynamoDB table named MyTable of the default region of your default AWS account. It would act nearly the same as if the PutCommand were executed in a lambda in the default region of your default AWS account.

Thus, you can use the same code that you use in your lambda to push test fixtures or to expect a side effect happened. You can also use the same utils you generally use in your lambda to interact with services, such as an ORM, DynamDBToolBox or TypeBridge.

Load environment variables

For most AWS services you need at least one identifier to interact with a service instance. In the latest example, it's the tableName of the DynamoDB table. The best way to pass them to your application code is to use environment variables.

At runtime, the environment variables are stored by AWS Lambda and injected into the executed code.

To interact with the services in integrations tests the same way it's done in lambda, the environment variables must be loaded in the test process.

Let's use the test setup file to inject the environment variables in each integration test suite.

// test/setup_integration.ts

process.env.TABLE_NAME = 'MyTable';
process.env.BUS_NAME = 'MyBus';

To easily manage your environment variables and avoid committing them, I recommend you to use dotenv to load your variables.

# .env.test
TABLE_NAME=MyTable
BUS_NAME=MyBus

// test/setup_integration.ts
import { config as loadEnv } from 'dotenv';

loadEnv('.env.test');

Allow access to AWS services

To interact with an AWS service you need to be authenticated and authorized.

At runtime, the AWS SDK resolves its identity using multiple methods.

When it's executed in lambda, it resolves the role assumed by the lambda and uses it to interact with the services.

When it's executed locally you can choose how it resolves the credentials to use. I recommend using AWS profiles.

Create an AWS programmatic user in the account of your test stack. Note: if you already have a dev profile with enough rights, you can use it and skip steps 1. to 3.
Attach him at least the union of the policies of you lambdas (to be able to run every lambda locally). If you use your own dev account you can use admin rights to be able to interact with all services freely.
Configure an AWS profile

   aws configure --profile my-test-profile

Add an AWS_PROFILE environment variable which must be loaded in the test process.

   # .env.test
   AWS_PROFILE=my-test-profile
   # ... the other environment variables

Write your first test

You are now ready to write the first integration test of the sample application.

The application code is the following:

// handler.ts
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { EventBridgeClient, PutEventsCommand } from '@aws-sdk/client-eventbridge';
import { DynamoDBDocumentClient, GetCommand } from '@aws-sdk/lib-dynamodb';

const ddbDocClient = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const eventBridgeClient = new EventBridgeClient({});

export const handler = async (): Promise<void> => {
  const { Item } = await ddbDocClient.send(
    new GetCommand({
      TableName: process.env.TABLE_NAME,
      Key: {
        PK: 'MyPartitionKey',
        SK: 'MySortKey',
      },
    }),
  );

  if (Item === undefined) {
    throw new Error('The item must be defined');
  }

  await eventBridgeClient.send(
    new PutEventsCommand({
      Entries: [
        {
          EventBusName: process.env.BUS_NAME,
          Source: 'my-lambda',
          DetailType: 'ItemFound',
          Detail: JSON.stringify(Item),
        },
      ],
    }),
  );
};

1. Arrange: Setup and load fixtures in a real stack

To be executed without error, this application code needs a specific item in the DynamoDB table. The fixture must be loaded before executing it.

// handler.integration-test.ts
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, PutCommand } from '@aws-sdk/lib-dynamodb';

const ddbDocClient = DynamoDBDocumentClient.from(new DynamoDBClient({}));

it('gets the item and publishes it as an event', async () => {
  // Arrange
  await ddbDocClient.send(
    new PutCommand({
      TableName: process.env.TABLE_NAME,
      Item: {
        PK: 'MyPartitionKey',
        SK: 'MySortKey',
        payload: 'MyPayload',
      },
    }),
  );
});

2. Act: Import the local code or execute a lambda

You can now execute your application code. You can either run the application code locally or in a lambda.

2.a Execute application code in Lambda

// handler.integration-test.ts
import { InvokeCommand, LambdaClient } from '@aws-sdk/client-lambda';

const lambdaClient = new LambdaClient({});

it('gets the item and publishes it as an event', async () => {
  // Arrange [...]
  // Act
  const { FunctionError } = await lambdaClient.send(new InvokeCommand({ FunctionName: 'my-lambda' }));
  expect(FunctionError).toBeUndefined();
});

The application code is executed as it would be in production. It's, therefore, the most reliable. However, you need to deploy the function each time you modify the code.

2.b Execute application code locally

// handler.integration-test.ts
import { handler } from './handler';

it('gets the item and publishes it as an event', async () => {
  // Arrange [...]
  // Act
  await handler();
});

The application code is executed locally. The feedback loop is shorter. You can use --watch mode to execute the test each time you change the code.

Use the same node version locally to be as close as possible to the lambda environment.

But there are some errors you can't catch locally:

The unset environment variable errors because you have all environment variables in your .env.test.
Your local profile is not the same as the lambda role. You could have the right to publish in event bridge locally and forget to pass it to the lambda role.

2.c Use both

I recommend you execute your application code locally while you are developing the test or the application code to benefit from the short feedback loop.

Once you're sure your code act correctly, deploy it and change the test to execute the deployed lambda to catch environment and rights issues.

3. Assert: Expect real change on the stack

You can now check that the event has been published.

With most services, you can also use the AWS SDK to interact with it and notice its state has changed.

With Eventbridge it's a little more complicated because you need to intercept the events. Hopefully, you can use sls-test-tools to easily intercept EventBridge events and assert the event has been published.

// handler.integration-test.ts
import { EventBridge } from 'sls-test-tools';

it('gets the item and publishes it as an event', async () => {
  // Arrange [...]
  const eventBridge = await EventBridge.build(process.env.BUS_NAME);
  // Act [...]
  // Assert
  const events = eventBridge.getEvents();
  expect(events).toHaveEventWithSource('my-lambda');
});

4. Clean: Remove the trace of the side effects before the next test

The last thing to do is to clean the test stack to avoid tests interfering.

// handler.integration-test.ts
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, PutCommand } from '@aws-sdk/lib-dynamodb';
import { EventBridge } from 'sls-test-tools';

const ddbDocClient = DynamoDBDocumentClient.from(new DynamoDBClient({}));

it('gets the item and publishes it as an event', async () => {
  // Arrange [...]
  const eventBridge = await EventBridge.build(process.env.BUS_NAME);
  // Act [...]
  // Assert [...]
  // Clean
  await ddbDocClient.send(
    new DeleteCommand({
      TableName: process.env.TABLE_NAME,
      Key: {
        PK: 'MyPartitionKey',
        SK: 'MySortKey',
      },
    }),
  );
  await eventBridge.destroy();
});

The complete test file

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { InvokeCommand, LambdaClient } from '@aws-sdk/client-lambda';
import { DeleteCommand, DynamoDBDocumentClient, PutCommand } from '@aws-sdk/lib-dynamodb';
import { EventBridge } from 'sls-test-tools';

const ddbDocClient = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const lambdaClient = new LambdaClient({});

it('gets the item and publishes it as an event', async () => {
  // Arrange
  await ddbDocClient.send(
    new PutCommand({
      TableName: process.env.TABLE_NAME,
      Item: {
        PK: 'MyPartitionKey',
        SK: 'MySortKey',
        payload: 'MyPayload',
      },
    }),
  );
  const eventBridge = await EventBridge.build(process.env.BUS_NAME);
  // Act
  const { FunctionError } = await lambdaClient.send(new InvokeCommand({ FunctionName: 'my-lambda' }));
  expect(FunctionError).toBeUndefined();
  // Assert
  const events = eventBridge.getEvents();
  expect(events).toHaveEventWithSource('my-lambda');
  // Clean
  await ddbDocClient.send(
    new DeleteCommand({
      TableName: process.env.TABLE_NAME,
      Key: {
        PK: 'MyPartitionKey',
        SK: 'MySortKey',
      },
    }),
  );
  await eventBridge.destroy();
});

Conclusion and next step

You can now configure and write integration tests. You can run them locally in interaction with a test stack.

The next step is to run them in CI. It's a tricky subject that I detailed in a second article.

Blazing fast CI for serverless integration tests

Corentin Doue for Serverless By Theodo ・ Mar 22 '22

#aws #serverless #testing #node

DEV Community: Corentin Doue

Top 10 common errors I wish I hadn’t made using SQS

TL;DR;

SQS Configuration

🙅 Error: Expecting messages to be consumed in the order they are sent

🙅 Error: Setting a too small Retention Period

🙅 Error: Setting a too small Visibility Timeout

🙅 Error: Using Lambda reserved concurrency to control throughput

Producer

🙅 Error: Sending messages that the consumer can’t process

🙅 Error: Sending messages individually

🙅 Error: Sending too many messages to SendMessageBatchCommand or batch too large messages

🙅 Error: Forgetting to handle SendMessageBatchCommand results

🙅 Error: Sending too many messages to SQS FIFO queues

🙅 Error: Forgetting to use MessageGroupeId for SQS FIFO Queues

🙅 Error: Using uuid() as MessageDeduplicationId

Consumer

🙅 Error: Throwing errors while processing a batch of SQS messages

Conclusion

Enforcing ESLint rules: A guide to taming codebase chaos

📝 TL;DR

⚠️ The temptation of the Warnings️

⚔️ Another Pitfall: Try to fix them all at once

🤖 A Better Approach: eslint-disable-inserter

CorentinDoue / eslint-disable-inserter

Inserts eslint-ignore comments automatically

eslint-disable-inserter

Example (Before/After)

How eslint-disable-inserter Works

Example: Transforming Code

♻️ Fix them all at your own pace

CorentinDoue / eslint-disabled-stats

Compute statistics about the eslint rules disabled

eslint-disabled-stats

Usage

Options

🚀 Conclusion

Power up your serverless application with AWS Lambda extensions

⚡ The Lambda extension feature

🤔 What is an Lambda extension?

🧩 How to use a Lambda extension

⚙️ How to create an extension

CorentinDoue / lambda-extensions-example

Minimal repository to create and use both internal and external Lambda extensions with a NodeJs lambda

Lambda extensions example

Install

Test

🔍 The interceptor internal extension

📝 The interceptor code

🚀 Deploy the internal extension

👀 Results

📊 The monitoring external extension

📝 The external extension code

CorentinDoue / lambda-extension-service

SDK to easily implement nodejs lambda extension

Lambda extension service

Installation

Usage

🚀 Deploy the external extension

👀 Results

CorentinDoue / lambda-extensions-example

Minimal repository to create and use both internal and external Lambda extensions with a NodeJs lambda

Lambda extensions example

Install

Test

Conclusion

5 control points to implement serverless integration tests like a boss 👔

🤔 Intent

🎯 The 5 controls points

1. Write one test case per integration ⛓️

✍️ Real-life example

Forget to test that a wrong message ends up in the DLQ

Test that different files are generated if you put different fixtures in the table

Two test cases:

2. Test deployed resources 🚀

✍️ Real-life example

Mock / emulate a service locally

Import and execute a lambda handler locally in the test

Deploy your resources in a real stack and let your test code interact with the interfaces

3. Keep the feedback loop below 1 min ♻️

🙅 Error: Sending too many messages to `SendMessageBatchCommand` or batch too large messages

🙅 Error: Forgetting to handle `SendMessageBatchCommand` results

🙅 Error: Forgetting to use `MessageGroupeId` for SQS FIFO Queues

🙅 Error: Using `uuid()` as `MessageDeduplicationId`

🤖 A Better Approach: `eslint-disable-inserter`

How `eslint-disable-inserter` Works