DEV Community: Khaan Sampaokit

Customize the deployment of AWS Lambda Layer anyhow you like

Khaan Sampaokit — Thu, 29 Dec 2022 23:56:23 +0000

Separate NPM Dependencies into layers

One way to improve deployment time when developing AWS Lambda functions is to separate your function's dependencies into layers. You can save time by deploying the layers once so that the subsequent deployments of your main functions code are as lightweight as possible.

Serverless Framework has built-in support for deploying layers by creating your desired dependencies in a separated package.json file and putting the file in the nodejs directory under the layer path.

# directory structure
layers/
  stablelibs/
    nodejs/
      package.json
  otherLibs/
    nodejs/
      package.json  
serverless.yml
package.json

# serverless.yml

layers:
  commonlibs:
    path: layers/commonlibs

functions:
  ...

The problems

However, this method has some problems.

You have to maintain multiple package.json files. It will be very difficult to keep the dependencies version in sync in every file, especially if you are using Github's dependabot to keep updating the dependencies version.
It will involve multiple manual steps for each deployment. For example, you have to go into each layer and run yarn install to populate node_modules before running the actual deploy command.
The deployment time can be even worst because it now has to take time to install dependencies and package each layer for every deployment.

For the second and third points, the existing plugin such as serverless-layer is a good solution, but in exchange, you will lose some built-in features such as the ability to exclude unnecessary files inside node_modules for some packages.

Different projects have different requirements for the workflow. It would be unfortunate to let the framework or plugins limit what you want to do.

I have been researching this problem, and It turns out that Serverless Framework is more flexible and powerful than I thought. It is very easy to customise the deployment workflow anyhow I like. And what follows is an example of what I have done with my current project.

1. Separate layers deployment from functions

In my project, I already have multiple serverless.yml for each of my API gateways. For example, I have serverless.internal-api.yml and serverless.external-api.yml for my internal and external API gateways.

The functions in both files share the same dependencies. So I create another file dedicated to the layers only.

serverless.internal-api.yml
serverless.external-api.yml
serverless.layers.yml <-- this
package.json

Let's start with the simple version of serverless.layers.yml

service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
  platformlibs:
    path: layers/platformlibs
    name: ${self:service}-${sls:stage}-platformlibs
  stablelibs:
    path: layers/stablelibs
    name: ${self:service}-${sls:stage}-stablelibs
  otherlibs:
    path: layers/otherlibs
    name: ${self:service}-${sls:stage}-otherlibs

And then use these layers with the functions in serverless.internal-api.yml and serverless.external-api.yml


package:
  individually: true

  # include only the functions code in the dist folder
  patterns:
    - "!**"
    - "dist/**"

providers:
  layers:
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-awslibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-platformlibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-stablelibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-otherlibs:latest

plugins:
  # This plugin will replace the 'latest' after the ARNs to the actual version number
  - serverless-latest-layer-version

Now I can deploy the layers independently from the functions. I have this npm script in package.json

scripts: {
  "deploy:layer": "serverless deploy -c serverless.layers.yml",
  "deploy:internal-api": "serverless deploy -c serverless.internal-api.yml",
  "deploy:external-api": "serverless deploy -c serverless.external-api.yml"
}

2. Use hooks to populate layer's files at built-time

Notice that we will need the following folder structure for the layer files.

# directory structure
layers/
  awslibs/
    nodejs/
      node_modules
      package.json
  platformlibs/
    nodejs/
      node_modules
      package.json  
  stablelibs/
    nodejs/
      node_modules
      package.json  
  otherlibs/
    nodejs/
      node_modules
      package.json  
serverless.internal-api.yml
serverless.external-api.yml
serverless.layers.yml
package.json

We will populate these files dynamically when running the deploy:layers script. To do that, I insert dependencyIncludeRegexPatterns for each layer configuration to tell which dependencies to be copied from the main package.json to the layer's package.json.

service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
    installCommand: yarn install
    dependencyIncludeRegexPatterns:
      - "^@aws-sdk"
      - "^aws"
  platformlibs:
    path: layers/platformlibs
    name: ${self:service}-${sls:stage}-platformlibs
    installCommand: yarn install
    dependencyIncludeRegexPatterns:
      - "^@opentelemetry"
      - "^@nestjs"
  stablelibs:
    path: layers/stablelibs
    name: ${self:service}-${sls:stage}-stablelibs

    dependencyIncludeRegexPatterns:
      - "^date-fns$"
      - "^rxjs$"
  otherlibs:
    path: layers/otherlibs
    name: ${self:service}-${sls:stage}-otherlibs
    dependencyIncludeRegexPatterns:
      - "."

Then I use serverless-plugin-scripts to define a hook script that will run right after we kick off the script deploy:layer.

plugins:
  - serverless-plugin-scripts

custom:
  scripts:
    commands:
      hooks:
        "before:package:initialize": ts-node tools/initializeLayerDependencies.ts serverless.layers.yml

Basically, we can do anything in tools/initializeLayerDependencies.ts, but here I populate the package.json for each layer and run the installCommand

import { mkdirSync, readFileSync, rmSync, writeFileSync } from "fs";
import * as yaml from "js-yaml";
import packageJson from "../package.json";
import { run } from "./libs/process-utils";

const configFile = process.argv[2];

initializeLayerDependencies(configFile);

// ################################################################

interface LayerConfig {
    path: string;
    installCommand: string;
    yarnCacheFolder: string;
    dependencyIncludeRegexPatterns: string[];
    dependencyExcludeRegexPatterns?: string[];
}

async function initializeLayerDependencies(configFile: string) {
    const config = yaml.load(readFileSync(configFile, "utf8")) as {
        layers: { [layerName: string]: LayerConfig };
    };

    const remainingDependencies = packageJson.dependencies;
    for (const [_layerName, layerConfig] of Object.entries(config.layers)) {
        console.log({ _layerName, layerConfig });

        if (!validConfig(layerConfig)) {
            console.warn("Invalid layerConfig!");
            continue;
        }
        const dependencyPath = `${layerConfig.path}/nodejs`;
        rmSync(dependencyPath, { recursive: true, force: true });
        console.log(`Initializing dependencies for ${layerConfig.path} ...`);
        const layerDependencies = Object.keys(remainingDependencies).reduce(
            (filteredDeps, key) => {
                if (
                    shouldInclude(
                        key,
                        layerConfig.dependencyIncludeRegexPatterns,
                        layerConfig.dependencyExcludeRegexPatterns,
                    )
                ) {
                    filteredDeps[key] = remainingDependencies[key];
                    delete remainingDependencies[key];
                }

                return filteredDeps;
            },
            {},
        );

        mkdirSync(dependencyPath, { recursive: true });
        writeFileSync(
            `${dependencyPath}/package.json`,
            JSON.stringify({ dependencies: layerDependencies }, null, 4),
        );

        writeFileSync(`${dependencyPath}/yarn.lock`, "");
        await run(layerConfig.installCommand, dependencyPath, {
            YARN_CACHE_FOLDER: layerConfig.yarnCacheFolder,
        });
    }
}

function shouldInclude(
    packageName: string,
    includePatterns?: string[],
    excludePatterns?: string[],
) {
    if (
        excludePatterns &&
        excludePatterns.some((regex) => packageName.match(regex))
    ) {
        return false;
    }

    if (
        includePatterns &&
        includePatterns.some((regex) => packageName.match(regex))
    ) {
        return true;
    }

    return false;
}

function validConfig({
    path,
    yarnCacheFolder,
    installCommand,
    dependencyIncludeRegexPatterns,
}: LayerConfig) {
    return (
        path &&
        yarnCacheFolder &&
        installCommand &&
        dependencyIncludeRegexPatterns &&
        true
    );
}

export async function run(
    cmd: string,
    cwd: string,
    env: Record<string, string> = {},
) {
    try {
        execSync(cmd, {
            cwd,
            env: {
                ...process.env,
                ...env,
            },
            maxBuffer: 1024 * 1024 * 500,
            stdio: "inherit",
        });
    } catch (error) {
        console.log(error.stdout?.toString());
        throw error;
    }
}

3. Improve installment time with yarn cache

We can further speed up the yarn install by utilizing yarnCacheFolder

service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
    installCommand: yarn install
    yarnCacheFolder: ../../../.yarn/cache <-- here
    dependencyIncludeRegexPatterns:
      - "^@aws-sdk"
      - "^aws"

4. Save the dependencies and skip deployment

Imagine the sequence to run in our CI/CD would be like

  - yarn install
  - yarn deploy:layer # serverless deploy -c serverless.layers.yml
  - yarn deploy:internal-api # serverless deploy -c serverless.internal-api.yml
  - yarn deploy:external-api # serverless deploy -c serverless.external-api.yml

So far so good, but it still takes non-trivial time to build the layers. Serverless Framework can check the deployment hash and skip the deployment, but it does not know when to skip the build (packaging) step.

This is easy. First, we will cache the dependencies from the main package.json in the SSM parameter store together with the deployment package.

# serverless.layers.yml

resources:
    Resources:
        LayerDependencies:
            Type: AWS::SSM::Parameter
            Properties:
                Type: String
                Name: /${self:service}/${sls:stage}/dependencies
                Value: ${file(./tools/getDependenciesAsString.js):value}

# tools/getDependenciesAsString.js

module.exports = async () => {
    const fs = require("fs");
    const packageJson = JSON.parse(fs.readFileSync("package.json", "utf-8"));
    return {
        value: JSON.stringify(packageJson.dependencies),
    };
};

Next, we can create a command to compare the cached dependencies with the current dependencies in the main package.json and run yarn deploy:layer when we detect the diff.

# serverless.layers.yml

custom:
    scripts:
        commands:
            update: ts-node tools/updateDependencyLayers.ts "/${self:service}/${sls:stage}/dependencies" ap-southeast-1 "yarn deploy:lambda:layer"
        hooks:
            "before:package:initialize": ts-node tools/initializeLayerDependencies.ts serverless.layers.yml

# tools/updateDependencyLayers.ts

import packageJson from "../package.json";
import _ from "lodash";
import { GetParameterCommand, SSMClient } from "@aws-sdk/client-ssm";
import { run } from "./libs/process-utils";

main(process.argv[2], process.argv[3], process.argv[4]);

async function main(ssmPath: string, region: string, deployCommand: string) {
    const ssm = new SSMClient({ region });
    const response = await ssm.send(
        new GetParameterCommand({
            Name: ssmPath,
        }),
    );
    const cachedDependenciesString = response.Parameter.Value;

    if (!cachedDependenciesString) {
        console.log("Not found dependencies in cache, should deploy layers");
        await run(deployCommand, ".");
        return;
    }
    const cachedDependencies = JSON.parse(cachedDependenciesString);
    if (!_.isEqual(cachedDependencies, packageJson.dependencies)) {
        console.log(
            "Found dependencies in cache and is not equal to dependencies in package.json, should deploy layers",
        );
        await run(deployCommand, ".");
        return;
    }

    console.log("The dependencies are up-to-date, should skip deploy layers");
}

Now we can run npx serverless update -c serverless.layers.yml to invoke this script.

Conclusion

It will be much faster for developers to edit the function code and deploy it because the function code is lightweight.

Only when adding new libraries or updating their versions should we then deploy the layers.

The CI/CD pipeline can always run the update, which will skip the deployment if the dependencies are up-to-date.

If you have a hard time to say something, let maw say it.

Khaan Sampaokit — Sat, 09 Jul 2022 19:57:38 +0000

Maw (แมว) in Thai means cats, and they are special organism. When you did anything stupid, just blame the cat and you will become innocent. When you are filming a boring video for your Youtube channel, just add a cat, and it will become viral.

So when you want to say something difficult, let's use mawsay. Like cowsay, but maw.

Credit: the cat ascii art is from https://ascii.co.uk/art/cats

Edit your text.
Click here capture the image and share it to your Facebook.

Poor Man's Bionic Reading

Khaan Sampaokit — Sat, 25 Jun 2022 04:53:55 +0000

I find that Bionic Reading is super cool. But their Chrome extension is not quite pleasant to use.

So I create a quick extension. But the algorithm is not as complex as the original one. This extension simply "bionifies" 60% of word length.


function bionifyText(text) {
  const words = text.split(/\s+/);
  return words.map(bionifyWord).join(" ");
}

function bionifyWord(word) {
  const wordLength = word.length;
  const numBionifiedCharacters = Math.floor((wordLength * 60) / 100);
  const bionifiedToken = word.slice(0, numBionifiedCharacters);
  const theRest = word.slice(numBionifiedCharacters);
  return `<b>${bionifiedToken}</b>${theRest}`;
}

How to use

Clone the repository
Open Chrome and go to chrome://extensions/
Enable Developer Mode (probably at the top right corner)
Click "Load unpacked" and choose the repository root directory
Open any website, highlight the text you want to bionify
Right click, choose "bionify", and boom!

How to customize

Change the implementation of the function bionifyWord in bionify.js however you like, go back to chrome://extensions/ and click the reload icon

Example Usage

Why your API should explicitly take resource owner id as a parameter instead of inferring from authentication.

Khaan Sampaokit — Fri, 03 Jun 2022 09:37:22 +0000

Imagine your are designing a REST API to process a resource (e.g. an order). The API should be restricted to be available only when the caller user is authenticated and he is the owner of the resource.

Suppose that we are using JWT token for authentication. The token has the userId in the claim payload.

{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022,
  "userId": "..."
}

The classic approach

You then have an option to infer the userId from the token and use it to check for the authorization.

@GetMapping("/orders/{orderId}")
public Order getOrderDetail(String orderId) {
   String userId = getUserIdFromAuthMiddlewareSomehow();
   Order order = getOrderFromDatabase(orderId);
   if (!order.getUserId().equals(userId)) {
      throw new RuntimeException("Not authorized");
   }
   return order;
}

If you choose this approach, your API call might look like this.

jwt_token=...
curl -H "Authorization: Bearer ${jwt_token}" https://your-api.example.com/orders/12345

This approach has 2 drawbacks.

Your API will need to run some, if not full, business logic and access the database even though the caller user is not authorized. This will be more serious if this API consumes great server's resource and performance.
You mix the authentication concern in your business logic where it can be separated. Imagine your API has a complex implementation, and you want to reuse it for not only when the caller is the resource owner itself, but also for an administrator user as well, and so on. The complex logic to check for the "who" will be scattered all over the place, eventually it will be harder to understand "what" your API intends to do.

The better approach

Another option is to put the userId explicitly in the API parameter, and move the authorization logic to AOP middleware. For example, you can write a custom annotation @CheckTokenPayload to tell the middleware which field to check.

@CheckTokenPayload(
   userId = "#userId"
)
@GetMapping("/get-order")
public Order getOrderDetail(@Param String orderId, @Param  String userId) {
   Order order = getOrderFromDatabase(orderId, userId);
   return order;
}

With this approach, your business logic is clean, and your API does not need to execute any resource consumption logic if the user is not authorized.

But you must be careful to use the userId together with the orderId when query from the database, otherwise the order will be accessible to authenticated user if he know the orderId.

Multi-dimensional ownership

In real world, it will not be as simple as checking the resource owner's userId because a resource can has many kinds of "owner".

For example, a merchant can also be an owner of an order, which makes some of the merchant's staffs eligible to see the order.

And, there will administrator staffs, support (call center) staffs, etc.

Fortunately, the user types of the system is the aspect that is not dynamic. You can always design your access token and the middleware to handle every type of the user. For example, your JWT token might look like.

// token payload for customer user
{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022,
  "userType": "CUSTOMER",
  "customerId": "..."
}

// token payload for merchant staff
{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022,
  "userType": "MERCHANT_STAFF",
  "merchantStaffId": "...",
  "merchantId": "...",
  "permissions": ["order:view", "order:edit", ...]
}

And this is an example write APIs that checks and allows different types of the user without having to touch the database.

@CheckTokenPayload(
   authTypeAllows = {"CUSTOMER"},
   customerId = "#customerId"
)
@GetMapping("/customer/get-order")
public Order getOrderDetailForCustomer(@Param String orderId, @Param  String customerId) {
   Order order = getOrderFromDatabase(orderId, userId);
   return order;
}

@CheckTokenPayload(
   authTypeAllows = {"MERCHANT_STAFF", "ADMIN_STAFF"},
   // you may write the middleware to omit merchantId check if the authType is ADMIN_STAFF
   merchantId = "#merchantId",
   permissionsContain = {"order:view"}
)
@GetMapping("/merchant/get-order")
public Order getOrderDetailForMerchantOrAdmin(@Param String orderId, @Param String merchantId) {
   Order order = getOrderFromDatabase(orderId, merchantId);
   return order;
}

Reverse-engineering the pin code authentication flow for mobile apps

Khaan Sampaokit — Sun, 26 Dec 2021 11:23:22 +0000

Investigating The Existing Applications

I am developing a mobile application for financial usage. I want to make it as secure as the existing apps on the market. Many apps ask the user to enter a PIN code to unlock the app after period of time or when the user wants to perform some important actions such as transferring the money.

But I cannot find a standard (similar to OAuth) or reliable document about its implementation or the reason behind the flow design. Every app seems to have some variations. For example

V1) Some apps bind the pin code with a device, if the user uses another device, they have to set up a new pin code. While the other apps allow the user to use the same pin code for many devices at the same time.
V2) Some apps store and validate the pin code at the server-side, while the others use the pin code to encrypt a kind of token, which can be used to exchange for an access token, and store the encrypted value at the device.
V3) Some apps force the user to set up a new pin code every time after the successful username/password authentication, while the others allow reusing the pin code of the previous session (the user is not asked to set up the pin code after the login if they have done it before for the same device)

Since I cannot find the standard, what I can do at best is to try reverse-engineer using my imagination to figure out the reason behind the flow. But I am no security expert, please feel free to share your experience or point out which parts are wrong.

The Flow

At the most basic level, we have the username/password method, in which the server returns an accessToken to the app on a successful user's authentication. The app saves this token in the device's memory (RAM) and attaches it to every subsequent request to access authorized resources.
```
sequenceDiagram

User-->>Device: username/password
Device->>Server: username/password
Server->>Device: accessToken
User-->>Device: perform actions
Device->>Server: Authorization: bearer ${accessToken}
```

But when the app is terminated and re-opened, the above flow alone must ask the user to re-enter username/password, which is not a good user experience, therefore we typically use a sort of rememberMe token (a.k.a refreshToken in OAuth). The server returns this token along with the accessToken on successful authentication. The app saves this token in the device's secured storage and uses it to obtain a new accessToken when the old one is not available or expired.

sequenceDiagram

User-->>Device: username/password
Device->>Server: username/password
Server->>Device: accessToken,refreshToken
Device->>Device: persist refreshToken
User-->>Device: perform actions
Device->>Server: Authorization: bearer ${accessToken}
User-->>User: Idle for a while
User-->>Device: perform actions
Device->>Server: renewToken (refreshToken)
Server->>Device: accesToken,refreshToken
Device->>Device: persist refreshToken
Device->>Server: Authorization: bearer ${accessToken}

But then, most financial apps have an additional security requirement to re-authenticate the user after a while (session timeout), or every time the user re-open the app. In the web application era, users don't have too much problem about having to re-enter the username/password because they don't use the app (website) that often. But that is not the case for mobile apps nowadays. Fortunately, we more or less can assume that a mobile device is private to a user and can be used for authentication.

Here comes the pin code authentication. The app generate, upon installation, an app's specific universal unique id, call it a deviceId, save it in the device secured storage. Then after a successful authentication, the server returns a short-lived, one-time token just for setting up the pin code. The app asks the user the set up a 5-6 digits pinCode, then send the pinCode and deviceId to the server along with the one-time token, The server hash the combination of pinCode and deviceId and save the hash in a database table as one of the user's allowed devices, and then returns an accessToken to the app. Optionally, we can send an email to the user to allow the device explicitly.
```
sequenceDiagram

User-->>Device: Open app first time
Device->>Device: generate and persist deviceId
User-->>Device: username/password
Device->>Server: username/password + deviceId
Server->>Server: Validate username/password
alt Optional
Server->>Server: Send email asking the user to allow the device
end
Server->>Device: temporaryToken

Device-->>User: Ask for pinCode
User-->>Device: Enter a pinCode
Device->>Server: pinCode + deviceId + temporaryToken
Server->>Server: save hash of pinCode + deviceId
alt Optional
Server->>Server: save hash of deviceId to check if it is the same device in the next login
end
Server->>Device: accessToken
User-->>User: Idle for a while
User-->>Device: perform actions
Device-->>User: ask for pinCode
User-->>Device: Enter pinCode
Device->>Server: renewToken (pinCode + deviceId)
Server->>Device: accesToken
```
If my imagination so far is the correct reason for the pin code flow, I then can conclude that, in the variations V1 above, the apps that allow the user to use the same pinCode across multiple devices are not using the pin code flow properly. It implies that they save the pinCode as plain text, or as a hash separated from the deviceId and it is no better than an alternative weaker password.
As an alternative to saving the hash of pinCode + deviceId on the server-side, we can use the pinCode to encrypt the rememberToken or the refreshToken of the typical OAuth flow, and save the encrypted result in the device's secured storage. When we want to re-authenticate the user, we ask the user the enter the pinCode and use it to decrypt the token and use the token to renew an accessToken. This is more secured because the refreshToken is not only specific to a device, but also specific to a user's authentication.
```
sequenceDiagram

User-->>Device: username/password
Device->>Server: username/password
Server->>Device: accessToken,refreshToken
Device-->>User: ask for pinCode
User-->>Device: enter a pinCode
Device->>Device: persist encrypt(refreshToken,pinCode)
User-->>User: Idle for a while
User-->>Device: perform actions
Device-->>User: Ask for pinCode
User-->>Device: Enter a pinCode
Device->>Device: refreshToken = decrypt(ciphertext, pinCode)
Device->>Server: renewToken (refreshToken)
Server->>Device: accesToken,refreshToken
Device->>Device: persist encrypt(refreshToken,pinCode)
Device->>Server: Authorization: bearer ${accessToken}
```
Again if this is correct, I can conclude that the variations in V2 are both valid. We can choose to store and validate the secret at either the server or the device. If we do it at the device, we avoid the risk that it may be compromised at the server, but with the limitation that leads to the variations V3. We must ask the user to set up a new pinCode after login because we need it to encrypt the refreshToken. If we do it on the server, we have an option to let the user use the old pinCode. In this case, the server has to also keep the hash of the deviceId (in addition to the has of deviceId + pinCode) to be able to determine (without using pinCode) if the user is logged in on the same device. It is also possible to manage the devices such as blocking or removing from the allowed list

Serverless Push Notification With AWS IoT

Khaan Sampaokit — Tue, 29 Jun 2021 14:35:41 +0000

AWS IoT is an option to create IoT devices without back-end servers as it provides MQTT as a service for us. Having a mobile application to interact with the devices is also possible without servers thanks to AWS Cognito that enables us to access many AWS services securely from the mobile application.

With this setup, serverless push notification from IoT devices to the mobile application can be achieved with only a few more configuration. And since I rarely find any example on this topic so I put it here for anyone exploring the option.

TL;DR

Use IoT Rule SQL like this to transform the data published into a topic to a valid GCM/FCM payload and forward it to Platform Application Endpoints.

Prerequisites

This post focuses only on how to set up the push notification with AWS IoT. Readers are assumed to be familiar with following topics.

How to connect microcontrollers, such as ESP32, to AWS IoT.
How to set up Firebase Cloud Messaging (FCM) to work with an Android application in your framework of choice, though in this example, I use Ionic and its Capacitor plugin.
How to set up AWS Cognito, and possiblity Amplify Library, to work with a mobiile application.

Runtime Architecture

IoT devices publish to an MQTT topic with a JSON data.
An IoT Rule is configured to transform the published data to a valid FCM notification payload and forward the payload to an SNS Topic.
The SNS Topic forwards the payload to all subscribed Platform Application Endpoints.
AWS sends requests to FCM with the payload, and the push notifications are delivered to the mobile application

Setup

1. Aquire FCM ServerKey from Firebase Console and use it to create SNS Platform Application.

aws sns create-platform-application \
    --name dev.foo-project.foo-mobile-devices.sns-app \
    --platform GCM \
    --attributes PlatformCredential=<Your FCM ServerKey>

2. Create the SNS Topic.

aws sns create-topic --name dev__foo-project__push-notification__sns-topic

3. The IoT rule we are about to create will need an IAM role that grants a permission to trigger the above SNS Topic.

echo '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "iot.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole"
            ]
        }
    ]
}' > trust-policy.json

echo '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sns:Publish",
            "Resource": "arn:aws:sns:<your_region>:<your_account_id>:dev__foo-project__push-notification__sns-topic"
        }
    ]
}' > policy.json

aws iam create-role --role-name dev.foo-project.iot-rule-action.role --assume-role-policy-document file://trust-policy.json
aws iam put-role-policy --role-name dev.foo-project.iot-rule-action.role --policy-name dev.foo-project.iot-rule-action.policy --policy-document file://policy.json

4. And finally, the main piece of today recipe, the IoT Rule.

echo '{
    "Sql": "SELECT concat('{\"priority\":\"high\",\"notification\":{\"title\":\"', title ,'\",\"body\":\"', body ,'\",\"mutable_content\":true,\"sound\":\"Tri-tone\",\"badge\":1,\"click_action\":\"foo\"},\"data\":{}}') as GCM, '' as default FROM 'dev.foo-project/push-notification'",
    "Actions": [
        {
            "Sns": {
                "MessageFormat": "JSON",
                "TargetArn": "arn:aws:sns:<your_region>:<your_account_id>:dev__foo-project__push-notification__sns-topic",
                "RoleArn": "arn:aws:iam::<your_account_id>:role/dev.foo-project.iot-rule-action.role"
            }
        }
    ]
}' > rule-payload.json

aws iot create-topic-rule \
    --rule-name dev__foo_project__push_notification__iot_rule \
    --topic-rule-payload file://rule-payload.json

It can transform a payload of the form { "title": "...", "body": "..." } to a valid FCM/GCM payload.

User Notification Registration Flow

1. The mobile application is set up to use AWS Cognito + Amplify Authentication. Here is an example as a React Component.

import Amplify, { Auth } from "aws-amplify";
import { Authenticator, SignIn } from "aws-amplify-react";

Amplify.configure({
    aws_project_region: 'your_region',
    aws_cognito_identity_pool_id: 'your_cognito_identity_pool_id',
    aws_cognito_region: 'your_region',
    aws_user_pools_id: 'your_user_pool_id',
    aws_user_pools_web_client_id: 'your_user_pool_client_id',
    aws_appsync_authenticationType: "AMAZON_COGNITO_USER_POOLS",
    oauth: {},
});
Auth.configure({
    authenticationFlowType: "USER_PASSWORD_AUTH",
});
Amplify.Logger.LOG_LEVEL = "VERBOSE";

const AppWithAuth: React.FC = () => {
    const [isSignedIn, setSignedIn] = useState(false);
    return (
        <Authenticator           
            onStateChange={(state) => {
                if (state === "signedIn") {
                    setSignedIn(true);
                }
            }}>
            <SignIn />
            <App ready={isSignedIn} />
        </Authenticator>
    )
}

2. After the user signin, we can use the credentials from Cognito to create any AWS SDK Clients

import { SNSClient } from "@aws-sdk/client-sns";
import { IoTClient } from "@aws-sdk/client-iot";

const amplifyCredentialProvider = async () => {
    const awsCredentials = await Auth.currentCredentials();
    return Auth.essentialCredentials(awsCredentials);
};

export const iotClient = new IoTClient({
    region: config.region,
    credentials: amplifyCredentialProvider,
});

export const snsClient = new SNSClient({
    region: config.region,
    credentials: amplifyCredentialProvider,
});

3. The application retrieves FCM token from Firebase and use SNSClient to create the platform application endpoint and subscribe to the SNS topic

import { ActionPerformed, PushNotifications, PushNotificationSchema, Token } from "@capacitor/push-notifications";
import { CreatePlatformEndpointCommand, SubscribeCommand } from "@aws-sdk/client-sns";

const App: Rect.FC<{ready: boolean}> = ({ ready }) => {
    useEffect(() => {
        if (ready) {
            // Retrieve FCM Token
            PushNotifications.addListener("registration", async (token: Token) => {
                const endpoint = await snsClient.send(
                    new CreatePlatformEndpointCommand({
                        PlatformApplicationArn: config.snsPlatformApplicationArn,
                        Token: token.value,
                    })
                );
                const subscriptionResult = await snsClient.send(
                    new SubscribeCommand({
                        Protocol: "application",
                        TopicArn: config.snsTopicArn,
                        Endpoint: endpoint.EndpointArn,
                    })
                );
            });
            // FCM Registration Error
            PushNotifications.addListener("registrationError", (error: any) => {
            });
            // Notification received when the app is in foreground
            PushNotifications.addListener("pushNotificationReceived", (notification: PushNotificationSchema) => {
            });
            // Notification data when the user tab the notification in system tray
            PushNotifications.addListener("pushNotificationActionPerformed", (notification: ActionPerformed) => {
            });

            PushNotifications.register();
        }
    }, [ready])
    return (
        <BlaBlaBla />
    )
}

Testing

1. We will use AWS Console' MQTT Test Client tool to simulate a device publishing to the MQTT topic.

2. Yey!, the notification arrives at my phone.

Axios Interceptor Model And Pitfalls

Khaan Sampaokit — Mon, 26 Apr 2021 22:36:49 +0000

It is common for people using Axios in their Javascript/Typescript project to use Interceptors to handle middleware stuff such as authentication headers or request/response logging.

There are plenty of examples out there that show how simple it is to use the Interceptors. You use axios.interceptors.request.use(onFulfilled, onRejected) to modify the request before it is fired, and use axios.interceptors.response.use(onFulfilled, onRejected) to handle the response before it is returned to the caller location.

But for people like me, who have a background in Java and are already familiar with the model of simple interceptors such as in Spring's RestTemplate, it is easy to misuse Axios Interceptors due to misunderstanding as we mix up the concept. Spring's RestTemplate is simple, we handle the request and response in an interceptor chain as if we call a normal method and the framework ensures that any error (Exception) in an interceptor will interrupt the chain and can be handled by the preceding interceptor with a simple try-catch.

Axios Interceptors, on the other hand, do not have this kind of chain interruption and error handling. What will happen if you write interceptors like this.

const myAxios = axios.create();

myAxios.interceptors.request.use(
  (config) => { console.log('interceptor2 handle config'); return config },
  (error) => { console.log('interceptor2 handle error') },
);

myAxios.interceptors.request.use((config) => {
  throw new Error('something is wrong in interceptor1');
});

myAxios
  .get('https://dev.to')
  .then(response => {
    console.log('caller handle response');
    console.log(response);
  })
  .catch(error => {
    console.log('caller handle error');
    console.log(error);
  });

It turns out to be some mysterious error.

$ ts-node --files main.ts
interceptor2 handle error
caller handle error
TypeError: Cannot read property 'cancelToken' of undefined

Why was the message something is wrong in interceptor1 missing? How come interceptor2 was invoked? And what was that 'cancelToken' all about? It was not clear to me at first. So I dig into it and draw this diagram to explain how it works.

This diagram assumes that the interceptors are registered in the order as in the following example code. You can modify this code to test and see the result yourself.


// # Use these commands to init project
// yarn init -y
// yarn add axios typescript @types/node ts-node log4js 
// npx tsc --init
// echo "console.log('hello world')" > main.ts
// npx ts-node --files main.ts

import axios from "axios";
import log4js from "log4js";

log4js.configure({
  appenders: {
    out: {
      type: "stdout",
      layout: {
        type: "pattern",
        pattern: "%d %p %f{1}(%l) %m%n",
      },
    },
  },
  categories: {
    default: { appenders: ["out"], level: "debug", enableCallStack: true },
  },
});

const logger = log4js.getLogger();

const myAxios1 = axios.create();

myAxios1.interceptors.request.use(
  config => { logger.debug('Request Interceptor3 OnFulfilled'); return config; },
  error => { logger.debug('Request Interceptor3 OnRejected: %s', error); return error; }
);

myAxios1.interceptors.request.use(
  config => { logger.debug('Request Interceptor2 OnFulfilled: %s', config); return config; },
  error => { logger.debug('Request Interceptor2 OnRejected: %s', error); return error; }
);

myAxios1.interceptors.request.use(
  config => { logger.debug('Request Interceptor1 OnFulfilled'); return config; },
  error => { logger.debug('Request Interceptor1 OnRejected: %s', error); return error; }
);

myAxios1.interceptors.response.use(
  response => { logger.debug('Response Interceptor1 OnFulfilled: %s', response.statusText); return response; },
  error => { logger.debug('Response Interceptor1 OnRejected: %s', error); return error; }
);

myAxios1.interceptors.response.use(
  response => { logger.debug('Response Interceptor2 OnFulfilled: %s', response.statusText); return response; },
  error => { logger.debug('Response Interceptor2 OnRejected: %s', error); return error; }
);

myAxios1.interceptors.response.use(
  response => { logger.debug('Response Interceptor3 OnFulfilled: %s', response.statusText); return response; },
  error => { logger.debug('Response Interceptor3 OnRejected: %s', error); return error; }
);

myAxios1
  .get("https://dev.to") // to test case api success
  // .get("https://dev.to/oh-no") // to test case api error
  .then((response) => {
    logger.debug('Caller response: %s', response.status);
  })
  .catch((err) => {
    logger.debug('Caller error: %s', err);
  });

And here is the explanation for each label in the diagram.

The first Request Interceptor will only have onFulfilled() invoked. You usually don't want to register any onRejected() for this interceptor.
The second Request Interceptor can have either onFulfilled() or onRejected() invoked depending on the return value of the first interceptor. If the return value is equivalent to Promise.resolve(), then onFulfilled() will be invoked, and if it is equivalent to Promise.reject(), then onRejected() will be called.

Please note that the following code is equivalent to Promise.resolve():
```
myAxios1.interceptors.request.use(
  config => { return config; },
);

myAxios1.interceptors.request.use(
  config => { return Promise.resolve(config); },
);
```
And the following is equivalent to Promise.reject():
```
myAxios1.interceptors.request.use(
  config => { throw 'error'; },
);

myAxios1.interceptors.request.use(
  config => { return Promise.reject('error'); },
);
```
The third interceptor doesn't care about which method is invoked previously in the second interceptor. It only cares about whether the return value is equivalent to Promise.resolve() or Promise.reject(). For example, throwing an error inside onFulfilled() of the second interceptor can invoke onRejected() on the third interceptor. Likewise, return a resolved promise in onRejected() of the second interceptor can invoke onFulfilled() on the third interceptor.
If the return value of the third interceptor is equivalent to Promise.reject(), no matter it is from which method, it will invoke onRejected() on the Response Interceptor3 without sending the request to the server.
If the last request interceptor's onReject() returns a resolved promise, or anything equivalent, that thing will be treated as a config object to prepare a request to send to the server. Therefore, if it is actually not a config object, a nasty exception may be thrown. This is what happened with the example in the introduction section.
Some errors can be populated at Axios middleware such as the error due to request timeout or internet connection problem.
If the response status code is in range 2XX, onFulfilled() on the first response interceptor will be called, and onRejected() otherwise. This logic can be customized by rewriting the function validateStatus on the config object.
Like the request interceptors, which method will be called for the subsequent response interceptors is depending on the resolved/rejected promise of the previous interceptor.
Finally at the caller location, the then() will be invoked if the last response interceptor yields a resolved promise, otherwise catch() will be invoked.

remark1: the onRejected() is optional, you can only provide the onFulfilled() and the default implementation will be used for onRjected() which basically is like (error) => Promise.reject(error).
remark2: When every interceptor has only the onFulfilled() defined, it is the special case in which any error that occurs inside an interceptor equivalent to a rejected promise will be chained automatically and invisibly to reach the catch() at the caller location. This effect is similar to the chain interruption but it is not the case because actually all interceptors will always be invoked behind the scene.

Example

To complete the research, the following code and diagram shows an example of a minimal interceptor chain.


// # Use these commands to init project
// yarn init -y
// yarn add axios typescript @types/node ts-node log4js @types/uuid
// npx tsc --init
// echo "console.log('hello world')" > main.ts
// npx ts-node --files main.ts

import axios, { AxiosInstance, AxiosRequestConfig, AxiosResponse } from "axios";
import log4js from "log4js";
import { v4 as uuidv4 } from 'uuid';

/**
 * Configure Logger
 */

log4js.configure({
  appenders: {
    out: {
      type: "stdout",
      layout: {
        type: "pattern",
        pattern: "%d %p %f{1}(%l) %m%n",
      },
    },
  },
  categories: {
    default: { appenders: ["out"], level: "debug", enableCallStack: true },
  },
});

const logger = log4js.getLogger();

/**
 * Declare helper types and functions
 */

export interface AxiosRequestInterceptor {
    onFulfilled: (
        config: AxiosRequestConfig
    ) => AxiosRequestConfig | Promise<AxiosRequestConfig>;
    onRejected: (
        error: any
    ) => any;
}

export interface AxiosResponseInterceptor {
    onFulfilled: (
        response: AxiosResponse<any>
    ) => AxiosResponse<any> | Promise<AxiosResponse<any>>;
    onRejected: (
        error: any
    ) => any;
}

const registerRequestInterceptor = (
    instance: AxiosInstance,
    interceptor: AxiosRequestInterceptor
) => {
    instance.interceptors.request.use(
        interceptor.onFulfilled,
        interceptor.onRejected
    );
};

const registerResponseInterceptor = (
    instance: AxiosInstance,
    interceptor: AxiosResponseInterceptor
) => {
    instance.interceptors.response.use(
        interceptor.onFulfilled,
        interceptor.onRejected
    );
};

/**
 * Begin building the interceptor chain
 */

const resourceAxios = axios.create();
const authAxios = axios.create();

const state = {
    accessToken: ''
}

const RequestBasicHeadersInterceptor: AxiosRequestInterceptor = {
    onFulfilled: (config) => {
        logger.debug('RequestBasicHeadersInterceptor.onFulfilled');
        config.headers['Authorization'] = `Bearer ${state.accessToken}`;
        config.headers['X-Request-Id'] = uuidv4();
        return config;
    },
    onRejected: (error) => {
        logger.debug('RequestBasicHeadersInterceptor.onRejected');
        return Promise.reject(error);
    },
};
const RequestLoggingInterceptor: AxiosRequestInterceptor = {
    onFulfilled: (config) => {
        logger.debug('RequestLoggingInterceptor.onFulfilled');
        logger.info('%s|%s|%s|%s|%s', config.method, config.url, JSON.stringify(config.params), JSON.stringify(config.data), JSON.stringify(config.headers));
        return config;
    },
    onRejected: (error) => {
        logger.debug('RequestLoggingInterceptor.onRejected');
        return Promise.reject(error);
    },
};
const ResponseLoggingInterceptor: AxiosResponseInterceptor = {
    onFulfilled: (response) => {
        logger.debug('ResponseLoggingInterceptor.onFulfilled');
        logger.info('%s|%s|%s|%s|%s', response.config.headers['X-Request-Id'], response.status, response.statusText, JSON.stringify(response.data), JSON.stringify(response.headers));
        return response;
    },
    onRejected: (error) => {
        logger.debug('ResponseLoggingInterceptor.onRejected');
        if (error.response) {
            const response: AxiosResponse = error.response;
            logger.info('%s|%s|%s|%s|%s', response.config.headers['X-Request-Id'], response.status, response.statusText, JSON.stringify(response.data), JSON.stringify(response.headers));
        } else {
            logger.info(error);
        }
        return Promise.reject(error);
    },
};

const appConfig = {
    authEndpoint: '...',
    clientCredentialsBase64: '...'
}
const AuthRetryInterceptor: AxiosResponseInterceptor = {
    onFulfilled: (response) => {
        logger.debug('AuthRetryInterceptor.onFulfilled');
        return response;
    },
    onRejected: async (error) => {
        logger.debug('AuthRetryInterceptor.onRejected');
        if (error.response) {
            const response: AxiosResponse = error.response;
            if (response.status === 403 && !response.config.headers['X-Auth-Retry']) {
                logger.debug('AuthRetryInterceptor.onRejected: start auth retry ...');
                try {
                    const authResponse = await authAxios.post(appConfig.authEndpoint!, {}, {
                        headers: {
                            'Authorization': `Basic ${appConfig.clientCredentialsBase64}`,
                            'Content-Type': 'application/x-www-form-urlencoded',
                            'X-Request-Id': response.config.headers['X-Request-Id']
                        },
                        params: {
                            'grant_type': 'client_credentials'
                        }
                    });
                    if (authResponse.data && authResponse.data['access_token']) {
                        const accessToken = response.data['access_token'] as string;
                        state.accessToken = accessToken;
                        response.config.headers['X-Auth-Retry'] = true;
                        return resourceAxios.request(response.config);
                    } else {
                        return Promise.reject('Not found access token');
                    }
                } catch (authError) {
                    return Promise.reject(authError);
                }
            }
        }
        return Promise.reject(error);
    },
};

class ApiErrorDomain extends Error {
    constructor(msg: string) {
        super(msg);
    }
}
class ApiClientErrorDomain extends ApiErrorDomain {
    constructor(msg: string) {
        super(msg);
    }
}
class ApiServerErrorDomain extends ApiErrorDomain {
    constructor(msg: string) {
        super(msg);
    }
}
class ApiUnknownErrorDomain extends ApiErrorDomain{
    constructor(msg: string) {
        super(msg);
    }
}
const DomainMapperInterceptor: AxiosResponseInterceptor = {
    onFulfilled: (response) => {
        logger.debug('DomainMapperInterceptor.onFulfilled');
        return response;
    },
    onRejected: (error) => {
        logger.debug('DomainMapperInterceptor.onRejected');
        if (error.response) {
            const response: AxiosResponse = error.response;
            if (response.status >= 400 && response.status < 500) {
                throw new ApiClientErrorDomain(response.data);
            } else if (response.status >= 500) {
                throw new ApiServerErrorDomain(response.data);
            } else {
                throw new ApiUnknownErrorDomain(response.data);
            }
        }
        throw new ApiUnknownErrorDomain(error);
    },
};

registerRequestInterceptor(resourceAxios, RequestLoggingInterceptor);
registerRequestInterceptor(resourceAxios, RequestBasicHeadersInterceptor);

registerResponseInterceptor(resourceAxios, ResponseLoggingInterceptor);
registerResponseInterceptor(resourceAxios, AuthRetryInterceptor);
registerResponseInterceptor(resourceAxios, DomainMapperInterceptor);

registerRequestInterceptor(authAxios, RequestLoggingInterceptor);
registerResponseInterceptor(authAxios, ResponseLoggingInterceptor);

/**
 * Test and see the result
 */

resourceAxios
  .get("https://dev.to") // to test case api success
  // .get("https://dev.to/oh-no") // to test case api error
  .then((response) => {
    logger.debug('Caller response: %s', response.status);
  })
  .catch((err) => {
    logger.debug('Caller error: %s', err);
  });

Flasher and Monitor For API Integration Testing

Khaan Sampaokit — Sat, 27 Mar 2021 18:07:31 +0000

Problems In Integration Testing

Writing API integration test scripts is easy to be messed up with. There are 2 important problems that I think are the root of chaos. The first problem is about how to prepare test data, the second is how to see and validate the side effects that are not immediately observable through the output of the API.

It is common for a non-trivial system to have integration with many external dependencies. For example, a monolith API service might have a minimal architecture like this.

Test engineers often ask me how they can write a script to connect to the database to modify some data before running the test and to validate the states in the database that are not observable through the output of the API. This is an unavoidable situation, I know this because I have once had a failed attempt trying to force that the integration test scripts must use only the available APIs as being used by the client applications. For example, to test a product checkout API of an e-commerce system, the test script must use existing APIs to create users, products (and their categories), and shopping cart, and so on, before the actual test logic can be executed. The script turns out to be too complex, time-consuming, slow, unmaintainable, and fragile.

The script can be a lot simpler if we just allow it to modify and query the database to validate the result, thanks to the universal and simple nature of SQL. But when it is not just a database, the script becomes messed up again as, for example, it has to connect to a file storage service to upload a prerequisite file and then test if the file has been processed as expected. The script sometimes has to call the third-party API services to check the result. I believe you can imagine the more complex cases as the system goes bigger. We have to "hack" the infrastructure and security of the system to allow testing to be possible. And the test script becomes dependent on this hacking as well as all the dependencies behind the API service under testing. At the worst, you end up duplicating all the middleware and integration logic of the API service, and how do you know that such logic inside the test script is less buggy than that of the service?

Manual Operations

That said, what I have seen people achieve so far is to rely on manual operations to prepare all the prerequisite data in every dependency before running the test. Ones can have some good strategies to create a pile of database dumps and label each of them to be for a specific test case. Anyways, it is not easy to automate the integration test script in CI/CD pipeline.

Insight From IoT

This problem has been stuck in my head for some time until recently I started learning IoT development. At the time I had zero knowledge of electronics but somehow managed to randomly buy an ESP8266 microcontroller and a Raspberry Pi.

My first experience with Pi is interesting. The seller told me I should buy a monitor together with the Pi but I rejected it thinking I am a programmer so I don't need to waste my money with a monitor as I can SSH to the Pi from my Mac. I followed all the instructions to boot the Pi with a WIFI connection. But I stuck as I could not SHH to it for some unknown reasons. I had to go back to the store to buy a monitor, then I could see what was going on inside the Pi and fix all problems in no time. That is when I understand the power of a monitor. Without it, I have to "hack" the Pi somehow (if I can) to be able to understand the system. Then I realize this is similar to the problem in integration testing. If we can have a "monitor" that shows all the states related to the API under testing, we can have enough information to validate the system without having to sneak into it. So I have an idea, what if we architect our system such that when developers build an API, they have to build a monitor together with it.

But that only solves half of the problems because a monitor cannot be used to prepare the test data. So I extend my idea and introduce a "flasher" counterpart of the monitor. A monitor shows the "state" of the system related to API under testing, on the other hand, a flasher takes the expected "state" and then "flash" the system to be in such state before running the test. Here I use the terminology "flash" as a metaphor because it is similar to when I have to "flash" my ESP8266 microcontroller with a specific version of my program before running my test script in a specific case.

The following diagram depicts my idea.

Please notice that, with this strategy, the test script does not have to care about the external dependencies. The system under testing is responsible for query states from all dependencies and combines them with its own state to produce the final information for the monitor. And to prepare the test data, the test script defines the expected state of the whole integral system and puts it to the flasher. The system under testing is, again, responsible for updating its own state and all dependencies according to the expected state. I think this makes very much sense because the system under testing has all the infrastructure ready to interact with its dependencies.

An API with built-in monitors and flashers

I have experimented with this idea in my IoT project which uses Serverless Framework to create Lambda functions for the OTA feature. But I will show only the general code to exemplify the idea of monitor and flasher. The readers don't have to care about what is Serverless Framework and OTA. I hope you can apply the idea in any case as you wish.

The functionality to be tested here is an API to publish a firmware version. Imagine you have built a program for your microcontroller in my-iot project as a .bin file, e.g. firmware-v99.bin. You have uploaded the file to a storage service, I will use AWS S3 here. Now you are going to publish the just uploaded firmware file so that every device knows which is the latest firmware version to use. Though the firmware files are kept in S3, the data about the project and what is the latest firmware version is kept in a database table. The following picture depicts the relation between them.

Notice that the latest_version for project my-iot in the table is pointing to firmware-v98.bin and you are going to update it to the just uploaded firmware my-iot/firmware-v99.bin. To publish the firmware version, you have to call a rest API like this.

curl -X POST https://my-api-url/firmwares/publish?projectKey=my-iot&firmwareKey=firmware-v99.bin -H "x-api-key: xyz"

If the request has been processed successfully, the response should have status 200 with an empty body, and the latest_version in the database table should become firmware-v99.bin for project my-iot. But it can also fail with response status 400 for the cases when projectKey or firmwareKey does not exist in the system.

Let's start with the test script for the case when projectKey does not exist. Here I use Jest as the test engine, and Axios as the HTTP client, the language is Typescript.

import axios from 'axios';

const apiKey = 'xyz';
const apiUrl = 'https://my-api-url'

interface State {
    [key: string]: {
        latest: string | null;
        firmwares: string[];
    };
}

async function fetchMonitor(): Promise<State> {
    const response = await axios.get(`${apiUrl}/firmwares/publish`, {
        headers: {
            'x-api-key': apiKey,
            'x-mode': 'monitor',
        },
    });
    expect(response.status).toBe(200);
    expect(response.headers['x-schema']).toBe('MONITOR');
    return response.data;
}

async function flash(state: State): Promise<void> {
    const response = await axios.put(
        `${apiUrl}/firmwares/publish`,
        {
            ...state,
        },
        {
            headers: {
                'x-api-key': apiKey,
                'x-mode': 'flash',
            },
        }
    );
    expect(response.status).toBe(200);
    expect(response.headers['x-schema']).toBe('FLASH');
}

async function flashRemoveAllProjects(): Promise<void> {
    await flash({})
}

it(`should reply 400 with body { "erroCode": "UNKNOWN_PROJECT_KEY" } when input projectKey does not exist in the system`,
    async (done) => {
        const mockProjectKey = 'mockProjectKey';
        await flashRemoveAllProjects();
        const state = await fetchMonitor();
        expect(Object.keys(state)).not.toContain(mockProjectKey);

        axios.post(`${apiUrl}/firmwares/publish`, null, {
            headers: {
                'x-api-key': apiKey,
            },
            params: {
                projectKey: mockProjectKey,
                firmwareKey: 'anything',
            },
        }).catch((err) => {
            expect(err.response.status).toBe(400);
            expect(response.headers['x-schema']).toBe('ERROR');
            expect(err.response.data).toHaveProperty('errorCode', 'UNKNOWN_PROJECT_KEY');
            done();
        });
    }
);

Please first notice the declaration of the State interface. It defines the structure of the output of our monitor, as well as the input of our flasher. As per the theory, the state should show all information behind the system related to the API under testing, which in this case are the name of all projects, the latest firmware version of each project, and the list of available firmware files in the S3 bucket for each project. An example of State data is as follows.

{
    "my-iot": {
        "latest": "firmware-v98.bin",
        "firmwares": [
            "firmware-v1.bin",
            "firmware-v2.bin",
            "firmware-v98.bin"
        ]
    },
    "my-smartfarm": {
        "latest": "firmware-v2.bin",
        "firmwares": [
            "firmware-v1.bin",
            "firmware-v2.bin"
        ]
    }
}

Following the State declaration is the function to fetchMonitor and to flash the system. The former sends a GET request with the header x-mode: monitor to the same endpoint as the API under testing. The latter sends a PUT request with the header x-mode: flash. I have set up the middleware that enforces every API to include handlers to handle these request accordingly, which I will talk about shortly.

The function flashRemoveAllProjects does just as its name implies. It invoke the function flash() with the empty state {} which indicates that we expect no project to exist in the system.

In the test function, we use it to make our assumption before executing the actual test.

const mockProjectKey = 'mockProjectKey';
await flashRemoveAllProjects();
const state = await fetchMonitor();
expect(Object.keys(state)).not.toContain(mockProjectKey);

Next is the example code to test the case of unknown firmwareKey

async function flashSetProject(projectKey: string): Promise<void> {
    const state: State = {};
    state[projectKey] = {
        firmwares: [],
        latest: null,
    };
    await flash(state);
}

it(`should reply 400 with { errorCode: "UNKNOWN_FIRMWARE_KEY" } when input firmwareKey does not exist in the system`,
    async (done) => {
        const mockProjectKey = 'mockProjectKey';
        const mockFirmwareKey = 'mockFirmwareKey';
        await flashSetProjectKey(mockProjectKey);
        const state = await fetchMonitor();
        expect(Object.keys(state)).toContain(mockProjectKey);
        expect(state[mockProjectKey].firmwares).not.toContain(mockFirmwareKey);

        axios.post(`${apiUrl}/firmwares/publish`, null, {
            headers: {
                'x-api-key': apiKey,
            },
            params: {
                projectKey: mockProjectKey,
                firmwareKey: mockFirmwareKey,
            },
        }).catch((err) => {
            expect(err.response.status).toBe(400);
            expect(response.headers['x-schema']).toBe('ERROR');
            expect(err.response.data).toHaveProperty('errorCode', 'UNKNOWN_FIRMWARE_KEY');
            done();
        });
    }
);

In this case, we define the function flashSetProject to set up an empty project in the system. The empty project has no latest firmware version and no firmware files in the list. Then we use it in the test the ensure our assumption that the input projectKey exists but the firmwareKey doesn't.

const mockProjectKey = 'mockProjectKey';
const mockFirmwareKey = 'mockFirmwareKey';
await flashSetProject(mockProjectKey);
const state = await fetchMonitor();
expect(Object.keys(state)).toContain(mockProjectKey);
expect(state[mockProjectKey].firmwares).not.toContain(mockFirmwareKey);

The final test case is for the happy case. We have to ensure that both of our inputs projectKey and firmwareKey exist before running the test logic. Here we define another function flashSetProjectAndUploadFirmwareKey for this.

async function flashSetProjectAndUploadFirmwareKey(projectKey: string, previousFirmwareKey: string): Promise<void> {
    const state: StateType = {};
    state[projectKey] = {
        latest: null,
        firmwares: [previousFirmwareKey],
    };
    await flash(state);
}

Please notice that we don't actually upload any files to S3. It is up to the system under testing to implement this flashing logic. It may upload some dummy file behind the scene to the real S3 service or entirely mock the S3 system with its own convenience. And here is how we use it in the main test script.

it(`should be able to publish the firmware successfully.`,
    async () => {
        const mockProjectKey = 'my-iot';
        const mockFirmwareKey = 'firmware-v99.bin';
        await flashSetProjectAndUploadFirmwareKey(mockProjectKey, mockFirmwareKey);
        const state = await fetchMonitor();
        expect(Object.keys(state)).toContain(mockProjectKey);
        expect(state[mockProjectKey].firmwares).toContain(mockFirmwareKey);
        expect(state[mockProjectKey].latest).toBe(null);

        await axios.post(`${apiUrl}/firmwares/publish`, null, {
            headers: {
                'x-api-key': apiKey,
            },
            params: {
                projectKey: mockProjectKey,
                firmwareKey: mockFirmwareKey,
            },
        })
        const stateAfter = await fetchMonitor();

        expect(Object.keys(stateAfter)).toContain(mockProjectKey);
        expect(stateAfter[mockProjectKey].firmwares).toContain(mockFirmwareKey);
        expect(stateAfter[mockProjectKey].latest).toBe(mockFirmwareKey);
    }
);

Here, after publishing the firmware, I fetch the stateAfter from the monitor and check that stateAfter[mockProjectKey].latest actually becomes mockFirmwareKey as expected.

With all these examples, we can see the power of our "flasher" and "monitor". Without them, we have to write code in our test script to clean up and upload files to S3 and write some SQL to update the database table. We also have the bothersome task of setting up infrastructure to allow the test script to access the S3 bucket and the database, which should be private to the API service in production. We also might have to set up some middleware that knows how to structure the firmware files in each project in S3 (because S3 doesn't have a concept of a directory). If the API service has decided to change the database and file storage service to other vendors, the test script should also be updated too. All of which makes our test script unmaintainable.

Finally, let me show you the middleware of the API that can enforce developers to always include the handler for flasher and monitor in every API.

The following code is the entry point of the Lambda function, it is like a controller in other frameworks if you are not familiar with Lambda. Anyways, please focus on the Middleware.integrate(...), which is my custom middleware.

import Middleware from 'my-middleware'
import handler from 'publish-firmware/handler'
import { monitor, flasher } from 'common/handlers'

export async function publishFirmware(event): APIGatewayProxyHandler {
    return Middleware.integrate(
        'POST',
        event,
        handler,
        monitor,
        flasher
    );
};

I won't show the code inside Middleware.integrate for brevity. Here I would like to exemplify how you can architect your code structure to enforce that every API controller should return a Middleware.integrate(...) which, in turns, is responsible to route the request to either handler, monitor, or flasher depending on your strategy. For mine, I make it that every request with method PUT and header x-mode: flash goes to the flasher, the request with method GET and header x-mode: monitor goes to the monitor, and the request with method matching the first parameter, i.e. POST in this case, goes to the main handler. I also have a configuration to disable this behavior in production. And please notice that the monitor and flasher do not have to be specific to only one API handler. You can have a shared monitor and flasher for a group of APIs. Here please notice that I put them in common/handlers instead of publish-firmware/handler

Conclusion

In this post, we have seen the idea of making every API to have a built-in monitor and flahser. Some may argue that this would take time and may not worth the effort. Anyways, I found it is very useful for me as a developer because this strategy leads me to think more thoroughly about all the possible states affected by my API. I can uncover some bugs which I could not have seen if I were not writing the code to handle the monitor and flasher.

I think the greatest benefit of this method is that it leverages automated integration testing in CI/CD pipeline. Integration testing is the real proof that our system works as per specifications or requirements. But most of the time we have to rely only on the coverage of unit testing because it is too complex to test some cases in integration. But unit testing is more or less of an Art that you cannot enforce a deterministic strategy for how and where should be tested. On the contrary, it is easy to set up a rule that every API should have the corresponding integration test script that covers at least what is in the specification. And now with the monitor and flasher strategy, we eliminate the indeterministic factors from the integration test script. Every developer knows how and has the same pattern to write the test script. All the complex responsibilities of dealing with dependencies are delegated to the API under testing, which already has the full infrastructure to handle the dependencies.

Shell Script Template

Khaan Sampaokit — Sat, 27 Feb 2021 12:56:25 +0000

Record everything in a shell script

Usually when I work with source code in a software project, no matter it is Java, Python, or any other languages, there are some infrastructure set up tasks that need the use of shell scripting.

It is very useful to record EVERY shell commands that involve the source code. For example, when you initialize a NextJs project, you would write yarn create next-app your-app. It is not a waste of time to record this in a utility script so that you and your friend know EVERYTHING commands that involve the project.

#!/bin/bash

init_project() {
    yarn create next-app your-app
}

Suppose you have a command to be used in CI/CD configuration tools such as Gitlab CI, you write all the core command in this shell script and have Gitlab CI invoke your script.

# In your utility.sh
build_and_deploy_to_k8s() {
    yarn build
    build_docker_image
    publish_docker_image
    deploy_to_k8s
}

# In your .gitlab-ci.yml
deploy:
    stage: deploy
    script:
        - ./utility.sh build_and_deploy_to_k8s

With this practice, you verbosely declare almost everything in the source code. The future you and your friends will share equal knowledge of what happens in the whole life cycle of the project and can investigate and reproduce any issue.

To do this, your utility.sh will need some common features:

You can add a new group of commands in a bash function and invoke the function easily, like ./utility.sh your_new_function.
You can add documentation to your commands so that they can be shown when the user executes the script with ./utility.sh help
The ./utility.sh help command can list all the available commands and their documentation.
You can add private commands that do not show to the user and cannot be invoked externally.
The script can handle when the user enters an invalid or no command.
The script has an easy error handling mechanism.
The script is in compliance with shellcheck standard.

My template

I share my template that work for me in most case in my Github, please check it out.

Example Usage

$ ./example.sh

alert
    To print a normal message with a cute cow.

all_commands
    Show all available commands

error
    To print an error message with a cute cow, and then exit the script with non-zero exit code

ex_happy
    Test happy case

ex_oops
    What will happens when there is an error that your forgot to handle with && and ||

ex_public_command
    All function that starts with _ will now be listed and cannot be invoked externally.
 But you can still invoke the function inside the script itself.

ex_success_or_failure
    This example show you the error handling behavior.
 You can use || and && to handle in case your command fails.
 If a command in a pipe fails, all the pipe also fails

ex_unhappy
    Test the "error" command

greeting_n_times
    your documentation

help
    Show all available commands and their documentation

$ ./example.sh greeting_n_times

 ______________________
  < Missing 1st parameter for name of the person >
 ----------------------
        \   ^__^
         \  (xx)\_______
            (__)\       )\/
             U  ||----w |
                ||     ||

$ ./example.sh greeting_n_times John

 ______________________
  < Missing 2nd parameter for the number of time to say hello >
 ----------------------
        \   ^__^
         \  (xx)\_______
            (__)\       )\/
             U  ||----w |
                ||     ||


$ ./example.sh greeting_n_times John 3
Hello John
Hello John
Hello John