DEV Community: Sergio Kaz

Storage-first pattern using Serverless

Sergio Kaz — Fri, 07 Jun 2024 15:30:48 +0000

Sometimes we have sensitive requests which we need to prepeare our systems to avoid losing it. Usually losing these requests mean that the company/product loses money or introduce issues in the product/application

Sensitive requests

Payment requests
Delivery requests
Webhooks with external information

Non-sensitive requests

List of users
List of user-roles
Endpoints for internal systems

The main difference between sensitive and non-sensitive request is that sensitive requests in general, you can not re-send it or it's more difficult in comparison with non-sensitive request

The idea behind of storage-first pattern, is basically to store the request information first ( in a high availability storage ) and process it later .

Why serverless is usefull in these cases ?

Using Serverless technologies like s3, sqs, kinesis, dynamodb gives you high availability without needing to pay for the provisioning. Using serverless storage you only pay per use.

we're going to explain here different use-cases and pattern, and talk a bit about pricing

Use cases

Here, there are 3 kind of implementation of storage-first pattern :

SQS-based Queueing

Ideal for: Payment processing, order fulfillment, webhook handling.

How it works:

Request is immediately placed in an SQS queue.
A separate process (Lambda function, ECS task, etc.) continuously polls the queue, processes messages, and performs necessary actions.

Benefits

Decouples request ingestion and processing.
Ensures high availability and fault tolerance.
Scales automatically to handle spikes in requests.
Retry mechanism

Pricing example

Processing per 1 M of requests

SQS : $0.4
API-GATEWAY (http): $1.00
Total : $1.4*
This is an estimate of the pricing because we're not taking care of Data Transfer

DynamoDB

Ideal for: Storing and retrieving sensitive request data quickly (e.g., payment details, delivery addresses)

How it works

Critical request data is stored in DynamoDB with a unique identifier.
Subsequent processes can look up and update data in DynamoDB with low latency using DynamoDB streams.

Benefits

Fast and reliable data access.
Serverless, scales automatically.
Supports fine-grained access control.

Pricing example

Processing per 1 M of requests

DynamoDB (write) : $1.25
API-GATEWAY (http): $1.00
Total : $2.25*
This is an estimate of the pricing because we're not taking care of Data Transfer and Storage

S3

Ideal for: Storing raw request data for auditing, compliance, and analysis.

How it works:

Requests are stored as objects in S3.
S3 triggers events (e.g., object creation) to notify other services for further processing or analysis.

Benefits

Durable and scalable storage.
Cost-effective for long-term storage.
Integrates with other AWS services.

Pricing example

Processing per 1 M of requests

S3 put(standard) : $5
API-GATEWAY (http): $1.00
Total : $6*
This is an estimate of the pricing because we're not taking care of Data Transfer and Storage

If you want to go deeper on AWS storage first pattern I highly recommend the following link

How to process CSV file at scale using Step Function and Lambda function ?

Sergio Kaz — Sat, 11 May 2024 23:44:41 +0000

When you need to process CSV files, you usually write a simple script and it's enough, however everything tends to be more complex when you need to run it at scale.

At scale means, tens or hundreds of millions of rows.

Challenges

When you need to process CSV at scale and repetitively, you need to take of these following things :

Parallel processing. The idea is to split the work in multiple nodes
Storage. Having big CSV you need to have a place to store it.
Output. If you need to store the output in a Database, you need to take care of things like DB ( connections, iops, throughput )

Solution

Using S3 + Lambda + Step Functions + MapState + DynamoDB you can solve the mentioned challenges, creating a Serverless solution and only pay for exactly what you used.

For this example I used CDK V2 to write the IaC, however you can implement using any othe IaC

Architecture overview

Storage

S3 works really well if you need to store files to process later. It's deeply integrated with AWS services and it's cost-effective

CDK code

const bucket = new Bucket(scope, 'Bucket', {
    bucketName: 'csv.data-lake.example.com',
    encryption: BucketEncryption.S3_MANAGED
});

Github Link

Storage output

DynamoDB is a no relational database which has a serverless provisioning method where you pay per request and you don't need to take care of DB connections, iops, throughput

CDK code

 new Table(scope, 'CsvItemTable', {
    tableName: `CsvItems`,
    partitionKey: {
      name: 'pk',
      type: AttributeType.STRING,
    },
    billingMode: BillingMode.PAY_PER_REQUEST
  })

Github Link

Processing

To process the CSV lines, I used a lambda function because it's really simple to step up. Other options could be ECS task using Fargate / or EKS.

Lambda definition

CDK code

new lambda.NodejsFunction(scope, 'ProcessCsvItems', {
    functionName: 'process-csv-items',
    entry: `${__dirname}/handler.ts`,
    handler: 'handler',
    runtime: Runtime.NODEJS_18_X,
    environment: {
      CSV_ITEM_TABLE_NAME: csvItemTable.tableName,
    },
    bundling: {
      minify: true,
      nodeModules: ['uuid'],
    },
  })

Github Link

Lambda code

Here we receive items as an array and we store in DynamoDB using Batch Write.

export const handler: Handler<HandlerEvent> = async ({ Items }) => {
  const command = new BatchWriteCommand({
    RequestItems: {
      [CSV_ITEM_TABLE_NAME]: Items.map((item) => ({
        PutRequest: {
          Item: {
            pk: uuid(),
            ...item,
          },
        },
      })),
    },
  })
  await docClient.send(command)
}

Github Link

Orchestrator

Step function is an orchestrator, where you can create state machines and integrate a wide range of AWS services, like Lambda function, ECS Task, EMR.

This part is a bit challenging to understand because we're not using a CDK Level 3 constructor, we need to write most of the CloudFormation code.

For this specifically case I used Map State which allows you to process a variety kind of files, like CSV, JSON.

Step Function - State defintion

Here you can specify different things like :

CSVHeaderLocation -> Header location
MaxConcurrency -> Max lambdas running at the same time
MaxItemsPerBatch -> The amount of items that our lambda function is going to receive

CDK code

new CustomState(scope, 'CsvToDynamoDBState', {
    stateJson: {
      Type: 'Map',
      ItemReader: {
        Resource: 'arn:aws:states:::s3:getObject',
        ReaderConfig: {
          InputType: 'CSV',
          CSVHeaderLocation: 'FIRST_ROW',
        },
        Parameters: {
          'Bucket.$': '$.bucket',
          'Key.$': '$.key',
        },
      },
      ItemProcessor: {
        ProcessorConfig: {
          Mode: 'DISTRIBUTED',
          ExecutionType: 'STANDARD',
        },
        StartAt: 'Lambda Invoke',
        States: {
          'Lambda Invoke': {
            Type: 'Task',
            Resource: 'arn:aws:states:::lambda:invoke',
            OutputPath: '$.Payload',
            Parameters: {
              'Payload.$': '$',
              FunctionName: lambda.functionArn,
            },
            Retry: [
              {
                ErrorEquals: [
                  'Lambda.ServiceException',
                  'Lambda.AWSLambdaException',
                  'Lambda.SdkClientException',
                  'Lambda.TooManyRequestsException',
                ],
                IntervalSeconds: 2,
                MaxAttempts: 6,
                BackoffRate: 2,
              },
            ],
            End: true,
          },
        },
      },
      MaxConcurrency: 200,
      Label: 'CsvToDynamoDB',
      End: true,
      ItemBatcher: {
        MaxItemsPerBatch: 25,
      },
    },
  })

Github Link

Step Function - definition

In this part we defined the type of StateMachine, we need to define the permissions and the steps.

Here, you need to create a role which can :

Invoke a lambda function
Get an object from S3
Start the Step Function

CDK code

new StateMachine(scope, 'CsvToDynamoDBStateMachine', {
    stateMachineName: 'parallel-processing-csv-to-dynamodb',
    stateMachineType: StateMachineType.STANDARD,
    definitionBody: DefinitionBody.fromChainable(chain),
    role: new Role(scope, 'StateMachineRole', {
      assumedBy: new ServicePrincipal('states.amazonaws.com'),
      roleName: 'parallel-processing-csv-to-dynamodb-role',
      inlinePolicies: {
        LambdaInvokePolicy: new PolicyDocument({
          statements: [
            new PolicyStatement({
              effect: Effect.ALLOW,
              actions: ['lambda:InvokeFunction'],
              resources: [lambda.functionArn],
            }),
          ],
        }),
        S3GetObjectPolicy: new PolicyDocument({
          statements: [
            new PolicyStatement({
              effect: Effect.ALLOW,
              actions: ['s3:GetObject'],
              resources: [bucket.arnForObjects('*')],
            }),
          ],
        }),
        StepFunctionsStartExecutionPolicy: new PolicyDocument({
          statements: [
            new PolicyStatement({
              effect: Effect.ALLOW,
              actions: ['states:StartExecution', 'states:DescribeExecution', 'states:StopExecution'],
              resources: ['*'],
            }),
          ],
        }),
      },
    }),
  })

Github Link

Here I left the repository link with the whole information about how to make the deploy to your own AWS account. Link

Any question or feedback is welcome !

How to get insights from our dataset without writing code?

Sergio Kaz — Fri, 24 Jun 2022 19:56:29 +0000

Data scientist spend most of their time (about 50% to 80%) cleaning, preparing and organizing data.

There are many tools in the market to achieve this, however I'll show you one of the most powerful tools that I've ever seen.

Wellcome AWS Glue DataBrew

AWS Glue DataBrew

AWS Glue DataBrew is a new visual data preparation tool that makes it easy for data analysts and data scientists to clean and normalize data to prepare it for analytics and machine learning.

Why is so powerful ?

Because, you can clean, prepare and organize your data at scale only paying per amount of information and time spending.

Step by step using DataBrew to get insights

Prerequisites

AWS Account

Create a bucket and upload your dataset

You can create a new bucket clicking here.

Once you create the bucket, you must to upload a dataset: this is the dataset which I'm using for this demo. Here

Set up the dataset on [DataBrew](https://us-east-

1.console.aws.amazon.com/databrew/home)

First we need to connect your Dataset to DataBrew

Here, you have different kind of ways to connect to your dataset. For this demo, we use Amazon S3.

Now, you have to select your S3 Bucket (that you created before) and select the dataset.

After that, click on Create

Run data profile

Once, you have your connection, select your dataset and click on Run data profile

There your are going to see, differents options like, number of rows that you want run the job, output file, etc.

At the end of the form, you are going to see a section named Permissions

There you must to select, Create new IAM role, fill the role name and click on Create and run job

Wait until the job finish

In the job section (Profile jobs), you'll see something like that:

When the job finish, click on View data profile and you'll see something like that:

Summary of the dataset and the correlation between variables

Value distribution

and columns summary!!

Well, there are much more insights that you can get with DataBrew, this is a short introduction.

Deploy multiple machine learning models for inference using Lambda and CDK V2

Sergio Kaz — Wed, 22 Jun 2022 14:39:04 +0000

This article was base on this post, however in this case I'm using CDK V2

You can take advantage of Lambda benefits for machine learning model inference with large libraries or pre-trained models.

Solution overview

For the inference Lambda function we use the Docker implementation in order to import the necessary libraries and load the ML Model. We're doing in this way because of the Lambda limits (10 GB for container images and 50 MB for .zip files) in the deployment package size.

We implement Amazon EFS as filesystem of Lambda function for inference, so now it’s even easier and faster to load large models and files to memory for ML inference workloads.

To upload the ML models to your file system, we use a Lambda function that is triggered when you upload the model to your S3 bucket. (in the next picture, you're going to see the architecture)

Architecture overview

The following diagram illustrates the architecture of the solution.

AWS services

Amazon VPC ( required for Amazon EFS )
Amazon S3
Amazon EFS
AWS Lambda
Amazon API Gateway

Steps to create this architecture from scratch

We are using AWS CDK V2 with TypeScript.

The Docker for inference was created for scikit-learn requirement. You can customize the requirements file, to use the machine learning framework you want such as TensorFlow, PyTorch, xgboost, etc.

1) Create VPC, EFS and Access Point

const vpc = new Vpc(this, "MLVpc");

We create a VPC because EFS needs to be inside one.

const fs = new FileSystem(this, "MLFileSystem", {
   vpc,
   removalPolicy: RemovalPolicy.DESTROY,
   throughputMode: ThroughputMode.BURSTING,
   fileSystemName: "ml-models-efs",
});

Here, we create the EFS within our VPC. Something to keep in mind is the throughput mode.

In bursting mode, the throughput of your file system depends on how much data you’re storing in it. The number of burst credits scales with the amount of storage in your file system. Therefore, if you have an ML model that you expect to grow over time and you drive throughput in proportion to the size, you should use burst throughput mode.

const accessPoint = fs.addAccessPoint("LambdaAccessPoint", {
  createAcl: {
    ownerGid: "1001",
    ownerUid: "1001",
    permissions: "750"
  },
  path: "/export/lambda",
  posixUser: {
    gid: "1001",
    uid: "1001"
  }
});

Here, we add an access point that we're going to use in our lambda's functions to connect to EFS.

2) Create S3 Bucket and Lambda function (to upload content from S3 to EFS)

const bucket = new Bucket(this, "MLModelsBucket", {
  encryption: BucketEncryption.S3_MANAGED,
  bucketName: "machine-learning.models",
});

This is a basic configuration of S3 Bucket.

const MODEL_DIR = "/mnt/ml";

const loadFunction = new NodejsFunction(this, "HandleModelUploaded", {
  functionName: "handle-model-uploaded",
  entry: `${__dirname}/functions/model-uploaded/handler.ts`,
  handler: "handler",
  environment: {
    MACHINE_LEARNING_MODELS_BUCKET_NAME: bucket.bucketName,
    MODEL_DIR,
  },
  vpc,
  filesystem: LambdaFileSystem.fromEfsAccessPoint(accessPoint, MODEL_DIR),
});

//Permission settings

bucket.grantRead(loadFunction);

loadFunction.addEventSource(
  new S3EventSource(bucket, {
    events: [EventType.OBJECT_CREATED],
  })
);

Here there are a few stuff to consider:

1) VPC: You must to set up the same VPC of EFS in the lambda.
2) MODEL_DIR: This is the path where you're going to store and load machine learning models.
3) Event source: We're saying that every new object in the bucket is going to execute that lambda function.

3) Lambda function (to make inference) and API Gateway

const inferenceFunction = new DockerImageFunction(this, "InferenceModel", {
  functionName: "inference-model",
  code: DockerImageCode.fromImageAsset(
    `${__dirname}/functions/model-inference`
  ),
  environment: {
    MODEL_DIR,
  },
  memorySize: 10240,
  timeout: Duration.seconds(30),
  vpc,
  filesystem: LambdaFileSystem.fromEfsAccessPoint(accessPoint, MODEL_DIR),
});

We're using DockerImage as lambda function, because of package size. Remember the Lambda limits (10 GB for container images and 50 MB for .zip files) in the deployment package size.

We set up the maximum memory allowed because we're using an internal cache to avoid loading it on every request.

const api = new RestApi(this, "ApiGateway", {
   restApiName: "inference-api",
   defaultCorsPreflightOptions: {
     allowHeaders: ["*"],
     allowMethods: ["*"],
     allowOrigins: ["*"],
   }
});

const todos = api.root.addResource("inference");

todos.addMethod("POST", new LambdaIntegration(inferenceFunction, { proxy: true }));

Finally, we create a Rest Api Gateway. We use ApiGateway V1 because CDK V2 hasn't released Api Gateway V2 as constructor yet.

Github

Let's take a look to our Github

How to create a website using S3 + Certificate Manager + Cloud Front with CDK ?

Sergio Kaz — Wed, 05 May 2021 00:17:20 +0000

Prerequisites

Before start we need a few things installed on our computer.

1) npm
2) NodeJs + TypeScript(2.7 or later)
3) CDK

After that we have to set up your aws credentials, to achieve that you can follow this link

The last one, you should have created a hosted zone in AWS and delegate your domain.

Well, we can start!!

Configure SSL certificate

const hostedZone = HostedZone.fromLookup(this,"myZone", {
    domainName: "example.com"
 });

const certificate = new acm.Certificate(this, `myCertificate`, {
      domainName: "example.com",
      subjectAlternativeNames: ["www.example.com"],
      validation: acm.CertificateValidation.fromDns(hostedZone),
});

On the first part we got the hosted zone related to your domain.
After that we created the SSL certificate using Certificate Manager. Adding the key validation into the CM, AWS automatically add the necessary records to validate your domain.

Set up bucket

const bucket = new Bucket(this, `myBucket`, {
      encryption: BucketEncryption.S3_MANAGED,
      bucketName: `your-bucket-name`,
      cors: [
        {
          allowedMethods: [HttpMethods.GET, HttpMethods.HEAD],
          allowedOrigins: ["*"],
          allowedHeaders: ["*"],
        },
      ],
});

In this way we create the bucket. It's recommended to use the domain's name as bucket's name and use a prefix to difference the environments.

Set up policy and CDNOAI

In this part, I am going to explain step by step, because It could be a bit confuse.

const CDNOAI = new CfnCloudFrontOriginAccessIdentity(
      this, "myCDNOAI", {
        cloudFrontOriginAccessIdentityConfig: {
          comment: "S3 OAI",
        },
      }
);

What is cloud front origin access identity ?

An Origin Access Identity (OAI) is used for sharing private content via CloudFront. The OAI is a virtual user identity that will be used to give your CF distribution permission to fetch a private object from your origin server (e.g. S3 bucket).

const policy = new iam.PolicyStatement({
      actions: ["s3:Get*"],
      effect: Effect.ALLOW,
      resources: [bucket.bucketArn, bucket.arnForObjects("*")],
});

We created a policy to allow get object on the whole bucket.

policy.addCanonicalUserPrincipal(CDNOAI.attrS3CanonicalUserId);

We add our CDNOAI to our policy.

const bucketPolicy = new BucketPolicy(this, `myBucketPolicy`, { 
  bucket,
});

bucketPolicy.document.addStatements(policy);

The last step of this section is create a bucket policy and attach to it our policy.

Set up Cloud front


const cloudFront = new cloudfront.Distribution(
      this,
      `myCloudFront`,
      {
        defaultBehavior: {
          origin: new origins.S3Origin(bucket),
          allowedMethods: AllowedMethods.ALLOW_GET_HEAD,
          viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
        },
        certificate,
        defaultRootObject: "index.html",
        domainNames: ["example.com"],
        enabled: true,
        errorResponses: [
          {
            ttl: Duration.days(1),
            httpStatus: 404,
            responseHttpStatus: 200,
            responsePagePath: "/index.html",
          },
          {
            ttl: Duration.days(1),
            httpStatus: 403,
            responseHttpStatus: 200,
            responsePagePath: "/index.html",
          },
        ],
      }
    );

This is a basic cloudfront configuration.

I want to highlight the certificate variable. It is the same certificate that we previously created.

Create A record

new ARecord(this, `myARecord`, {
      zone: hostedZone,
      ttl: Duration.minutes(5),
      target: RecordTarget.fromAlias(new targets.CloudFrontTarget(cloudFront)),
      recordName: "example.com",
    });

In this part we are going to create an A record within our hosted zone, and reference it with our cloud front.

GITHUB

This is my github, and here you are going to find a full complete example using CDK to create and deploy powerfull webapps!!

https://github.com/skaznowiecki/cdk-s3-cloudfront