DEV Community: Sc0ra

Serverless Contracts to the Rescue of your Fullstack Application

Sc0ra — Tue, 22 Nov 2022 13:00:55 +0000

Clear and steady interfaces for stable applications 🧘

Introducing a breaking change in the input or output of your API and not updating the frontend client might be the most common error a fullstack developper makes at the start of its career (and even later 👴).

Interfaces between different services are one of the weakest, if not the weakest, point in an application (right after copy pasted stackoverflow one liners 👨‍🔬).

Thats why you should put a lot of effort in defining those interfaces. However, it can become quite repetitive, and that's what we wanted to tackle with Serverless contracts.

Serverless Contracts 📑

At Kumo, we build fullstack applications using Serverless and TypeScript. During our multiple projects, we encountered the problem of varying interfaces a lot. That is why we decided to build Swarmion, a tool to build stable serverless application at scale.

One of its most important feature is Serverless Contracts.

What we wanted 🙏

When you want to build a tech tool, you should keep a product approach as much as possible. It can help you defining where the value sits and where you should be putting work.

So when we first thought about contracts, we focused on our main use case: interfaces between frontend and backend microservices, with the backend deployed behind API Gateway.

With that in mind, we wanted to address the folowing pain points with Serverless Contracts:

Usable in both the backend and the frontend
Strong type safety during static analysis
IntelliSense in the IDE
Runtime input and output validation at the backend for client-side errors handling
Reduced boilerplate in provider (API) and consumer (frontend) for every handler/request
Human readable definition

What we built 🔨

Now, lets dive into some code, with your first Serverless contract

// contracts/users-contracts/src/contracts/getUser/contract.ts
import { ApiGatewayContract } from '@swarmion/serverless-contracts';

import { pathParametersSchema, userEntitySchema } from './schemas';

const getUserContract = new ApiGatewayContract({
  id: 'users-getUser',
    integrationType: 'httpApi',
  authorizerType: undefined,
  path: '/users/{userId}',
  method: 'GET',
  pathParametersSchema,
  queryStringParametersSchema: undefined,
  headersSchema: undefined,
  bodySchema: undefined,
  outputSchema: userEntitySchema,

});

export default getUserContract;

// contracts/users-contracts/src/contracts/getUser/schemas.ts
export const pathParametersSchema = {
  type: 'object',
  properties: {
    userId: { type: 'string' },
  },
  required: ['userId'],
  additionalProperties: false,
} as const;

export const userEntitySchema = {
  type: 'object',
  properties: {
    userId: { type: 'string' },
    userName: { type: 'string' },
    email: { type: 'string' },
  },
  required: ['userId', 'userName', 'email'],
  additionalProperties: false,
} as const;

There is a lot of things to see here:

The getUserContract 👥

This class instance contains all the informations defining your endpoint interface:

Implementation details of your endpoint:
- A unique identifier
- The api gateway type (V1 or V2)
- The authorizer type (cognito, jwt, lambda or none)
The path and method of your endpoint
The different parts of the incoming request:
- path parameters
- query string parameters
- headers
- body
The format of the output of your endpoint.

The last 2 categories are defined using JSON Schemas.

JSON Schemas 💽

We use JSON Schemas for multiple reasons:

They are a well known interface definition format in the javascript ecosystem.
With the json-schema-to-ts lib, we can define our TypeScript types directly from those schemas without any duplication and provide fully typed helpers as we will see later.
They allow us to perform run time validation with ajv.

In this contract, we define an outputSchema and a pathParameterSchema. The other options are left undefined because those inputs are not used by our endpoint.

Now that we are settled with our contract, we can start using it in our application

Provider-side features

First, we want to define our endpoint using our contract.
With the Serverless framework, you do it by defining 2 elements: the function trigger and its handler. Serverless contracts provide helpers for both.

getTrigger ⚡

// config.ts
import { getTrigger } from '@swarmion/serverless-contracts';
import { getUserContract } from '@my-app/contracts/getUserContract';

export default {
  handler: getHandlerPath(__dirname),
  events: [getTrigger(getUserContract)],
};

This method defines the path and method that will trigger the lambda based on the contract information.

If you define an authorizer in your contract, you can provide it there directly, and it will prevent you to overwrite the trigger values.

// config.ts
import { getTrigger } from '@swarmion/serverless-contracts';
import { getUserContract } from '@my-app/contracts/getUserContract';

export default {
  handler: getHandlerPath(__dirname),
  events: [
    getTrigger(getUserContract, {
      method: 'delete', // ts will throw an error because 'delete' != 'get'
            authorizer: 'arn::aws...' // will also throw here because we did not 
                                                                // define an authorizer in this contract
    }),
  ],

getHandler ⚙️

// handler.ts
import { getHandler } from '@swarmion/serverless-contracts';
import { getUserContract } from '@my-app/contracts/getUserContract';

const handler = getHandler(getUserContract)(async event => {
  event.pathParameters.userId; // will have type 'string'

  event.toto; // will fail type checking
  event.pathParameters.toto; // will also fail

  return { id: '123456', name: 'Doe' }; // also type-safe!
});

export default handler;

The getHandler helpers does multiple things:

Full typing for input and output in IDE
Body parsing and output serialization
HTTP errors handling
input and output runtime validation using ajv

Consumer features

Once the endpoint is defined, we want to request it from another service (a frontend application for example).
To do so, you can use consumer helpers to generate type safe requests.

getFetchRequest 🌐

// useGetUser.ts
import { getFetchRequest } from '@swarmion/serverless-contracts';
import { getUserContract } from '@my-app/contracts/getUserContract';

await getFetchRequest(getUserContract, fetch, {
  pathParameters: { userId: '15' }, // correctly typed
  headers: {
    'my-header': 'hello', // will fail type checking
  },
  baseUrl: 'https://my-app.com',
});

Conclusion ⭐

Swarmion Serverless contracts helps you build scalable applications with multiple services and strong interfaces between them. We presented you API Gateway contracts but we also recently implemented EventBridge contracts for message interfaces between services.

Also, Swarmion comes with more features, including a project and service generator which implements all the best practices we gathered at Kumo.

You can check our developer guide here, everything is open source, feel free to contact us and contribute!

Prevent AWS from Reading Your Step Functions Data

Sc0ra — Mon, 31 Aug 2020 14:24:24 +0000

AWS Step Functions is the perfect tool to handle long-running, complex or business-critical workflows such as payment or subscription flows. However, a naive implementation could put sensitive data at risk.

What is AWS Step Functions?

AWS Step Functions is an Amazon cloud service designed to create state machines to orchestrate multiple Lambda functions, branching logic and external services in one place. It presents most of the advantages of the serverless services: the scaling is immediate and the pricing is based only on the computing time and the number of step transitions.

The state machines can be defined directly in the AWS Console with a JSON configuration and you directly have a graph representation of your workflow.

AWS Step Functions state diagram

At Theodo, we use Serverless as our framework with the Step Functions plugin to write our AWS Lambda functions, define our resources (DynamoDB and Queues) and setup AWS Step Functions and the state machines directly in our codebase.

Example: an identity verification workflow with Step Functions

The previous picture shows an identification verification workflow. It is a piece of a bigger validation flow for banking applications.

What it does is request an identification URL to a third-party system, send this link by text message to the applicant and wait for them to complete the identity verification.
When it is done, the third-party system calls a webhook, which restarts the state machine and either triggers a retry, sends a rejection email, or continues the global validation process.

The identification flow

To perform all of those actions, sensitive information will flow through the inputs and outputs of the state machine steps:

👤 Personal information from the applicant which are sent to the third-party service to check his/her identity
📱 The phone number of the applicant to send the SMS
✉️ The email address of the applicant to send the mail
✔️ The result of the identity verification

Problem: we are exposing sensitive data

With this implementation, we see that sensitive data flow through the inputs and outputs of your state machine directly.

state-machine.yaml
----------

SendIdentificationRequest:
  Type: Task
  Resource: arn:aws:states:::lambda:invoke
  Parameter:
    FunctionName:
      Fn::ImportValue: ${self:provider.stage}-sendSMS-Function-arn
    Payload:
      creditApplicationId.$: $.creditApplicationId
      creditApplicant.$: $.creditApplicant
      phoneNumber.$: $.phoneNumber
      message.$: $.smsBody
  ResultPath: $.identificationSMS
  Next: WaitIdentification

An exemple of sensitive data flowing through our inputs and outputs

The problem is: nowadays, most DevOps engineer, developers and even some product managers with access to AWS and can see that information. Not very GDPR friendly, right? Furthermore, AWS Lambda dumps are stored in an S3 bucket which is not encrypted with the default configuration. Any malicious access to your AWS stack could leak sensitive information far more easily than by corrupting your database for example. The more you spread sensitive information between multiple services, the more you put yourself at risk. Finally, as it is stated in AWS documentation:

Any data that you enter into Step Functions or other services might get picked up for inclusion in diagnostic logs.

With that in mind, you may completely dump AWS Step Functions, or you may try to find a solution because the tool is still awesome and perfectly suits your needs!

Our solution

To prevent leaking sensitive information, we need to clean every step function input and output. Here are two ways of doing it:

Keep the data in the input and output but encrypt and decrypt it between each step
Never transmit sensitive data between steps and retrieve it directly during their execution

Both solutions are viable. However, we prefered the second one because we wanted to keep our sensitive data in a single safe spot, where we put a lot of attention to the security.

We were already using AWS DynamoDB for the waiting tokens of our lambda functions, so we decided to create a new table to store sensitive data between steps.

We configured it like below, using an external KMS key.

resources.yaml
----------

StepSentiveInputTable:
  Type: AWS::DynamoDB::Table
  DeletetionPolicy: Retain
  Properties:
    TableName: step-sensitive-input-table
    AttributeDefinitions:
      - AttributeName: sensitiveInputId
        AttributeType: S
    KeySchema:
      - AttributeName: sensitiveInputId
        KeyType: HASH
    BillingMode: PAY_PER_REQUEST
    PointInTimeRecoverySpecification:
      PointINTimeRecoveryEnabled: true
    SSESpecification:
      SSEEnabled: true
      SSEType: KMS
      KeyId: hsm-key-id

DynamoDB table definition in Serverless

We were already using DynamoDB from our lambdas to save our waiting task tokens and we chose to add another table to save sensitive data between steps. Furthermore, on AWS, you can choose to save the encryption keys of your DynamoDB tables outside of AWS, which was a big plus for us.

To avoid repeating ourselves, we coded a wrapper that we use on all our lambdas. It does the following things:

After the return of the lambda, the wrapper:

Looks for sensitive data keys in the output, remove them and build an object with that information
Save the sensitive data objects in a dynamo entry
Add the id of the new dynamo entry under a “sensitiveData” key in the output

Before the execution of the main lambda function, the wrapper:

Looks for “sensitiveData” keys in the input
Retrieve the content at the given IDs
Replaces the "sensitiveData" keys with the resulting object in the input

// encryption-wrapper.ts

export default (lambdaHandler: LambdaHandler): LambdaHandler => {
  const decryptHandler = async (event: any) => {
    const decryptedInformation = await getSensitiveData(event.sensitiveInputId);
    return lambdaHandler({ ...event, ...decryptedInformation });
  };

  return async (event: any) => {
    const clearOutput = await decryptHandler(event);
    const sensitiveInputId = await storeSensitiveData(clearOutput);

    for (const sensitiveKey of sensitiveInputKeys) {
      delete clearOutput[sensitiveKey];
    }

    return { sensitiveInputId, ...clearOutput };
  };
};

Our custom encryption wrapper

Last of all, we needed to encrypt the first input at the start of our state machine from the API to finally secure our users information all the way!

Takeways

Take care about what information you have in your lambdas and state machine events on AWS
Always think twice about the data you put in your logs
Spread your secrets and security implementation in as few services as possible
Use a wrapper if you want to repeat behavior in multiple lambda functions.

Thank you very much for reading my article. Feel free to comment, ask for implementation details, and feel free to share the problems that you encountered with AWS Step Functions.