DEV Community: scl

JSON schema parser

scl — Fri, 24 Jan 2025 00:04:40 +0000

Been thinking about a model validation/parsing issue lately.

Background

I use JSON schema in AWS API gateway to validate the incoming event payload. If the payload gets to reach my Lambda codes downstream, those Lambda codes parse the payload into a strongly typed object using Zod.

Problem statement

How then can we prevent manual duplication and have a single source of JSON schema for API gateway (for validation) and our Lambda codes (for parsing into strongly typed objects)?

Zod doesn't really interoperate with JSON schema without some third party adapters. Hence that sent me on a search for a parser that plays well with JSON schema.

Probable solution

That search led me to a promising candidate that's used by some 30 million people:
https://ajv.js.org/guide/typescript.html#utility-type-for-jtd-data-type

A reference CDK stack

scl — Thu, 20 Jun 2024 09:36:58 +0000


import { Construct, IConstruct } from 'constructs';
import * as cdk from 'aws-cdk-lib/core';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';

export interface MyStackProps extends cdk.StackProps {
  stageName: string;
  prefix: string;
  retainData: boolean;
}

export class CdkApiSqsStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: MyStackProps) {
    super(scope, id, props);

    // Create an SQS queue
    const queue = new sqs.Queue(this, 'BooksQueue', {
      visibilityTimeout: cdk.Duration.seconds(300),
      queueName: `${props.prefix}-books-queue-${props.stageName}`,
    });

    // Create a DynamoDB table
    const carsTable = new dynamodb.Table(this, 'CarsTable', {
      tableName: `${props.prefix}-cars-${props.stageName}`,
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      partitionKey: { name: 'carId', type: dynamodb.AttributeType.STRING },
      removalPolicy: props.retainData ? cdk.RemovalPolicy.RETAIN : cdk.RemovalPolicy.DESTROY, // NOT recommended for production code
    });

    // Create a REST API
    const api = new apigateway.RestApi(this, 'Api', {
      restApiName: `${props.prefix}-api-${props.stageName}`,
      deployOptions: {
        stageName: props.stageName,
      },
      retainDeployments: false,
    });

    // Define the /books resource
    const books = api.root.addResource('books');

    // Add a POST method to the /books resource
    books.addMethod('POST', new apigateway.AwsIntegration({
      service: 'sqs',
      path: `${cdk.Aws.ACCOUNT_ID}/${queue.queueName}`,
      integrationHttpMethod: 'POST',
      options: {
        credentialsRole: new iam.Role(this, 'ApiGatewaySqsRole', {
          assumedBy: new iam.ServicePrincipal('apigateway.amazonaws.com'),
          managedPolicies: [
            iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSQSFullAccess')
          ]
        }),
        requestParameters: {
          'integration.request.header.Content-Type': "'application/x-www-form-urlencoded'"
        },
        requestTemplates: {
          'application/json': `Action=SendMessage&MessageBody=$util.urlEncode($input.body)`
        },
        integrationResponses: [
          {
            statusCode: '200',
            responseTemplates: {
              'application/json': `{"status": "message queued"}`
            }
          },
          {
            statusCode: '500',
            responseTemplates: {
              'application/json': `{"status": "error"}`
            }
          }
        ]
      }
    }), {
      methodResponses: [
        {
          statusCode: '200'
        },
        {
          statusCode: '500'
        }
      ]
    });

    // Grant API Gateway permissions to send messages to the SQS queue
    queue.grantSendMessages(new iam.ServicePrincipal('apigateway.amazonaws.com'));

    // Define the /cars resource
    const cars = api.root.addResource('cars');

    const carSchema = {
      type: 'object',
      required: ['name', 'age'],
      properties: {
        name: { type: 'string' },
        age: { type: 'number' }
      }
    };

    // Create a model for request validation
    const requestModel = api.addModel('RequestModel', {
      contentType: 'application/json',
      modelName: 'RequestModel',
      schema: {
        schema: apigateway.JsonSchemaVersion.DRAFT4,
        type: apigateway.JsonSchemaType.OBJECT,
        title: 'MyRequest',
        required: ['name', 'age'],
        additionalProperties: false,
        properties: {
          "name": { type: apigateway.JsonSchemaType.STRING },
          "age": { type: apigateway.JsonSchemaType.INTEGER }
        }
      }
    });

    // Create a request validator
    const requestValidator = new apigateway.RequestValidator(this, 'RequestValidator', {
      restApi: api,
      validateRequestBody: true,
      requestValidatorName: 'request-validator'
    });

    // Add a POST method to the /cars resource
    cars.addMethod('POST', new apigateway.AwsIntegration({
      service: 'dynamodb',
      action: 'PutItem',
      integrationHttpMethod: 'POST',
      options: {
        credentialsRole: new iam.Role(this, 'ApiGatewayDynamoDbRole', {
          assumedBy: new iam.ServicePrincipal('apigateway.amazonaws.com'),
          managedPolicies: [
            iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonDynamoDBFullAccess')
          ]
        }),
        requestTemplates: {
          'application/json': `{"TableName": "${carsTable.tableName}",
            "Item": {
              #foreach($entry in $input.path('$').entrySet())
              #if($entry.value != '')
                "$entry.key": { "S": "$entry.value" },
                #end
    #end
                  "carId": { "S": "$context.requestId" }
            }
        }`,
        },
        integrationResponses: [
          {
            statusCode: '200',
            responseTemplates: {
              'application/json': `{"status": "car added"}`
            },
            selectionPattern: '200',
          },
          {
            statusCode: '400',
            responseTemplates: {
              'application/json': `{"status": "client error"}`
            },
            selectionPattern: '4..',
          }
          ,
          {
            statusCode: '500',
            responseTemplates: {
              'application/json': `{"status": "server error"}`
            }
          }
        ]
      }
    }), {
      methodResponses: [
        {
          statusCode: '200'
        },
        {
          statusCode: '400'
        },
        {
          statusCode: '500'
        }
      ],
      requestModels: {
        'application/json': requestModel
      },
      requestValidator: requestValidator
    });

    // Grant API Gateway permissions to put items into the DynamoDB table
    carsTable.grantWriteData(new iam.ServicePrincipal('apigateway.amazonaws.com'));
  }
}

Validating credit card numbers

scl — Wed, 06 Dec 2023 12:46:45 +0000

A Typescript implementation of the Luhn algorithm in validating credit card numbers.

/**
 * Returns `true` if and only if the given credit card number is valid.
 */
function isLuhnValid2(ccNum: string): boolean {
    const chars = [...ccNum];
    const sumOfAltDoubles: number = chars.reduce((sumSoFar: number, value: string, index: number) => {
        const multiplier = (index % 2 === 0) ? 2 : 1;
        let tempValue: number = parseInt(value) * multiplier;
        if (tempValue >= 10) {
            tempValue = tempValue - 9;
        }

        return sumSoFar += tempValue;
    }, 0);

    return (sumOfAltDoubles % 10 === 0);
}

Source of algorithm: https://www.groundlabs.com/blog/anatomy-of-a-credit-card/

Building on top of DynamoDB API

scl — Mon, 16 Oct 2023 01:49:33 +0000

This is a good article on creating a higher level API over the official DynamoDB API:

https://cloudonaut.io/dynamodb-entity-store-cleaner-typescript-code/

API Gateway's Direct Integrations

scl — Tue, 10 Oct 2023 09:37:57 +0000

(Update on 16th Oct 2023) After working with VTL for a good two weeks or so, I've found it easy to pick up and to be proficient with it. Thus, direct integration will be my default go-to mode if it's CRUD operations without complex business logic.

Recently I've been working knee-deep in getting API Gateway to directly integrate with DynamoDB. While the performance appears to be faster without going through an intermediary Lambda, the trouble of having to be proficient in VTL makes me rethink if direct integration is a wise choice.

I'm just rehashing a few articles that I found useful:

Build different integrations and ran performance tests. Best of all, codes are available too: https://www.readysetcloud.io/blog/allen.helton/serverless-integration-speed-test/
Some common struggles when working with direct integrations and how to overcome them: https://www.readysetcloud.io/blog/allen.helton/direct-integrations-arent-scary/
A fun but meaningful Gopher Holes deployment: https://github.com/allenheltondev/gopher-holes-unlimited/blob/master/openapi.yaml

Converting codes from CJS to ESM

scl — Wed, 04 Oct 2023 07:46:00 +0000

I'm trying to wrap my head around ECMAScript Modules (ESM) and CommonJS (CJS). I understand that the way forward is to use ESM as much as possible because ESM is supported on both browsers and Node while CJS is supported only on Node.

Jotting down how to convert my codes from CJS to ESM. This is based on Typescript.

The files that need to be changed are:

package.json

  "type": "module"

tsconfig.json

  "compilerOptions": {
    "target": "ESNext",
    "module": "ESNext",                       
    "moduleResolution": "Node",
    "declaration": true,
    "esModuleInterop": true, 
    ....
  },
  "ts-node": {
    "esm": true
  }

(Or simply conveniently install Sindre's tsconfig.json)

(My source codes)

When importing relative dependencies, add ".js" to the filename, for example:

import { sayHello } from './mylib/util.js';

(Yes, even though my source code is located in util.ts, the Typescript server should be able to help by suggesting ".js")

Publishing my Typescript package to npm

scl — Wed, 04 Oct 2023 07:33:53 +0000

(Managed to get one working here: https://www.npmjs.com/package/api-gateway-direct-dynamodb)

Documenting the steps to publish a Typescript project to npm.

Check that tsconfig.json is emitting declaration files:

    "declaration": true, 
    "outDir": "./dist",

Compiles all Typescript source files and generates type declarations files:

rm -rf ./dist
npx tsc

Check that ./dist contains the compiled .js and .d.ts files.

Edit package.json to inform users' compilers on where to find the type declarations:

  "main": "dist/index.js",
  "types": "dist/index.d.ts",

This next step is optional, which is to test the library locally before publishing. Create another empty project and try to installing the library:

npm install path/to/my-package

Finally, publish the library:

npm publish

(For subsequent republishing, we would also need to increment the version number in package.json.

Ref: https://medium.com/cameron-nokes/the-30-second-guide-to-publishing-a-typescript-package-to-npm-89d93ff7bccd

Clearing Postman DNS cache

scl — Wed, 04 Oct 2023 07:29:54 +0000

Postman apparently defers to the underlying OS for DNS lookup cache.

Hence in order to clear Postman DNS cache, we need to clear the underlying OS's cache.

This can be done on MacOS:

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Ref: https://mackeeper.com/blog/flush-dns-cache-on-mac/