DEV Community: efi shtain

CRUD API in 3 lines using NestJS

efi shtain — Sun, 06 Mar 2022 20:37:54 +0000

Photo by Kier In Sight on Unsplash

If it sounds like a clickbait, looks like a clickbait and smells like a clickbait, it must be… well it’s not.

Here are the 3 lines to create a webserver with a crud api for a candy store which sell chocolate

I admit it’s not a full application, but If you look closely, I said an API.

Why did I write this article ?

Working with NestJS is super nice. It has an opinionated structure (that you can like or dislike), but you have to agree, it reduces a lot of the time wasted tinkering on directories structure, setting up tests, routes, database connections and so on, everything is solved for the average use case.

It even comes with a great extendable CLI to facilitate everything using generators.

But even with the CLI, every time I needed some kind of boiler plate code for a crud API I had to do the same stuff. Create a module, controller and a service.

nest g module chocolate

nest g controller chocolate

nest g service chocolate

Then I had to add the crud APIs which are the same for each resource. And finally, If I’m really committed, I had to have some DTOs and Entities definitions.

Even before I wrote any actual logic related to my application, I had to do all of that. So the generators are amazing, but not enough.

But I was wrong! there are more generators, mainly the resource generator

Using the following line of bash

nest g resource chocolate

I get EVERYTHING done for me in one command!

All resources created using generators

References:

Documentation | NestJS - A progressive Node.js framework

Dynamic AWS S3 Notification Configuration using Terraform

efi shtain — Tue, 08 Jun 2021 16:22:09 +0000

It is the simple stuff that you expect to be, well, simple. Until they aren’t.

Photo by James Harrison on Unsplash

So the situation is like this:

I have an aws s3 bucket
I have multiple dynamically created consumers for objectCreated event from this bucket
Each consumer subscribes to specific prefix on this bucket. i.e. service 1 will listen to new files on srv1/ path only. service2 will listen to new files on srv2/ path only and so on.

Since one SNS cannot differentiate between s3 prefixes, I need to have multiple SNS topics — one for each service

Doing it manually — it is super easy, just create another pair of sns/sqs and I’m ready to go. But manual process belongs to the dark ages of 1995, I need to do it automatically using IaaC, Terraform in this case.

Easy, let’s do it!

So there I went, knowing exactly what I need to do. Create a terraform file to manage the s3 bucket, sns topics and their policies, and attach each sns topic to the bucket with correct prefix (/srv1, /srv2, etc…)

Like any good programmer — I wanted to copy paste stuff just to get it going, so I googled “aws s3 sns notification terrafom” and clicked the first link (which super convinently was terraform documentations). This is the link:

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification

A simple note on that page caught my eyes and broke my heart:

What does that all mean??

Not so easy…

well, it means that I can attach one aws_s3_bucket_notification to the bucket. It also means, I have to know in advance all topics I’m going to have since I have to write them as part of the terraform code.

But I said before, I have a dynamic list of consumers, I can’t know all of them in advance, am I stuck?

Creativity time

So after thinking about it for a while, I tried playing with the aws api, fetching configuration and updating it. So if I can do it using the API and get what I want, I surely can automate it!

It took few trials and errors and I wrote a python script which accepts few parameters — bucket name, sns arn, and the relevant prefix. The scripts get the current configuration from aws API, check if the subscription already exists and adds it if it doesn’t.

In case someone needs the python file, I’d be happy to share it

Now I just need to run the script as part of Terraform.

Terraform Null resource and local exec provisioner

The null_resource resource implements the standard resource lifecycle but takes no further action.

The triggers argument allows specifying an arbitrary set of values that, when changed, will cause the resource to be replaced.

Basically null_resource can do anything, it has no backing resource in the cloud, it has a trigger and a provisioner which runs once the trigger changed.

So I’ve added a null resource wherever I have an SNS topic, pass the bucket name, the sns arn and the prefix, and that’s it!

provider "null" {}

resource "null_resource" "s3_object_created_subscription" {

  triggers = {
     prefix = var.prefix
  }

  provisioner "local-exec" {
     command = "python ${var.bucket_name} ${var.sns_arn} ${var.prefix}"
  }
}

This little piece of code can run from many terraform state files and attach an sns to a bucket with whatever prefix and how many prefixes I need. Like any terraform resource it has state, so it will run only once, unless you change the prefix.

Conclusion

I know that’s a hack. It might break since s3 API is eventually consistent. But it does not bother me since I don’t have a massive amount of parallel changes.

The reasoning behind it, vs using multiple buckets, is the bucket limit I tackled. Amazon currently allows only 1000 buckets per account. That’s actually a lot of buckets. But once you have multiple services, let’s say 30, each one requires at least 1 bucket and you have dynamic test environments, you can easily see how you reach this limit.

VSCode find and replace using regex super powers to remove duplicates

efi shtain — Sat, 21 Nov 2020 11:33:06 +0000

Photo by Markus Spiske on Unsplash

It happens sometimes, mainly in documentation or comments, that a word slips twice. Not the end of the world, but if there is an easy and quick solution, why leave the wrongs unfixed?

Lets fix “a a word” to “a word” without even knowing there is a problem

Well, the first problem is identifying such cases. It won’t be easy to spot a a two consecutive words repeating in a long document (did you see the a a?)

Then, if it happen once, it will happen again. What about other occurrences of the problem? or similar ones? It might happen a lot using the dark magic of copy pasting stuff.

Some background knowledge about regex

Regex is kind of powerful (at least in spec). If you never heard of it, there is a notion of grouping and capturing. Every time you wrap part of the expression in (), it is captured, meaning grouped and numbered in increasing order, so you can use it later in the expression (without knowing the value in advance).

So defining our problem, we look for full words (at least one letter), words should be bounded (so a ab won’t count as duplicate a) and there should be at least one space between the words (otherwise it is still the same word of course)

\b(\w+)\s+\1\b

This regex does exactly what we want, \b — words with a boundary, \w+ give us a word, \s+ catches at least one space. But what is the\1? and why \w+ is wrapped in parentheses? Well this are the exact captured groups we talked about before. (\w+) — match the first occurrence of a word and we capture it, \1 match the first captured group, in our case, the exact same word!

VSCode usage

Vscode has a nice feature when using the search tool, it can search using regular expressions. You can click cmd+f (on a Mac, or ctrl+f on windows) to open the search tool, and then click cmd+option+r to enable regex search.

VSCode find tool, regex search activated, duplicate regex inserted

Using this, you can find duplicate consecutive words easily in any document. You can also search across all your documents at once.

Matching all duplicate consecutive words easily

Now, we just need to replace the duplicate with one instance of the word. For that, toggle the replace mode (click the right arrow) and type in $1. The $1 references the same captured group as before.

VSCode find and replace expanded mode

Now click replace and watch the magic happens!

For brevity, of course there are no duplicates.

Firebase Firestore unit testing with jest (and kind of typescript)

efi shtain — Wed, 06 May 2020 11:29:01 +0000

It took some time but I think I got it

Photo by Ferenc Almasi on Unsplash

I was working the other day on a firebase project. The project is written using typescript, which is nice. It is basically a set of cloud functions which interacts with a Firestore database, among other things.

I wanted to write unit tests and I wanted the tests to run as fast as possible without any dependencies on a real Firestore instance — so I had to mock the Firestore calls.

Fair enough, should be easy… Well….

I wasted so much time trying to mock the complex structure of Firestore types, using way too many Firestore internal data structures when I decided that I must be wrong. If it is hard to mock — you are doing it wrong.

Let’s say I want to mock an update call on a document.

await collection('myCollection')
            .doc('myDoc')
            .update({ updated:true});

I have to mock three functions - collection, doc and update on the firestore instance i’m using. So if I’ll try to do something like this:

import \* as admin from 'firebase-admin';

let update = jest.fn()
let doc = jest.fn(()=>({update})
jest.spyOn(admin.firestore(), 'collection').mockReturnValue({ doc });

The compiler does not like it, it says:

Argument of type ‘{ doc: any; }’ is not assignable to parameter of type ‘CollectionReference<DocumentData>’.
 Type ‘{ doc: any; }’ is missing the following properties from type ‘CollectionReference<DocumentData>’: id, parent, path, listDocuments, and 16 more.

Problem is, I don’t really need most of the functionality of a Firestore collection object or a document object, so I don’t want to waste time mocking it.

The solution, is pretty straightforward, don’t mock what you don’t need.

Just make the return value an unknown type, and then make it any type. It’s kind of anti typescript, but for mocking external dependencies I think I can live with that.

import \* as admin from 'firebase-admin';

const update = jest.fn();
const doc = jest.fn(() => ({update}));
const collection = jest.spyOn(admin.firestore(), 'collection').mockReturnValue((({ doc } as unknown) as any);

voilà!

I can now expect what ever I want on these mocks:

expect(collection).toHaveBeenCalledWith('myCollection');

expect(doc).toHaveBeenCalledWith('myPost');

expect(update).toHaveBeenCalledWith({updated:true});

Happy mocking!

Github Actions — ship code to GCP Cloud Run

efi shtain — Wed, 11 Mar 2020 14:35:07 +0000

Github Actions — How to ship code to GCP Cloud Run

Photo by Kinsey on Unsplash

Deploying code to production is both time consuming and a delicate process. Getting things wrong might take your service down and eventually might hurt your users and cost you money.

The solution of course is automating as much as possible.

FYI: the blog post assumes basic knowledge in node.js/docker, a GCP account with permissions to deploy code and to create service accounts and a github account.

What is GCP Cloud run?

In the spirit of managed services, Google Cloud provides us with a service called Cloud Run.

Cloud Run is a fully managed compute platform that automatically scales your stateless containers

What can we understand from that sentence?

Cloud run is a managed service
Cloud run is for running containers
It scales automagiclly
It is designed for stateless services

So basically, anyone in need for running containers can benefit from using cloud run.

We want to run containers!

We can wrap our code in a container, docker for example, push it to GCP’s container registry (or another registry if we want) and use the gcloud cli tool to deploy it to cloud run or do some updates from the gcp console website.

But, that’s a repetitive process, it’s human error prone, and it consumes time. A wonderful oppourtonity for automatation.

There are many CI/CD tools on the market, in this post we’ll focus on the kind of newish Github Actions

What are Github Actions?

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub

Github Actions give us the capability to automate processes, so we can program what we want to do and know that it repeats itself exactly the same when we need it.

An action is basically a yaml file describing a series of instructions (called steps). Each step can be another action. I’m no going to go any deeper, documentation can be found here https://help.github.com/en/actions/getting-started-with-github-actions/core-concepts-for-github-actions

One key point is: Github provides an action’s marketplace, meaning we probably won’t have to invent the wheel unless we’re doing really bleeding edge stuff. The marketplace can be found here: https://github.com/marketplace?type=actions

Alright, What’s the plan?

We’re going to follow the next steps:

Create a new Github repo
Create a super simple node.js server
Create the matching Dockerfile so we can containerize the server
Create Github action yaml to actually build the container and deploy it to cloud run
Create a service account
Give the service account powers to act
Update Github secrets
Commit code and watch the magic happen

Full code can be found here: https://github.com/efiShtain/github-action-cloud-run

Create a new Github repo

If you already have a repo you’re working on, skip this step. Otherwise, go to Github.com, log in to your account a create a new repo, name it what ever you like and clone it to your drive

Create a super simple node.js server

Run the following commands in the repo directory:

// Initialize an npm package
npm init -y

// create source file directory
mkdir src

// create the application server entry point file
touch src/index.js

// vi to src/index.js or edit it how over you like and add the following code to it

const http = require('http');

function requestHandler(req, res) {
    res.write('ok');
    res.end();
}

const server = http.createServer(requestHandler);
const port = process.env.PORT || 8888;
server.listen(port, () => console.log(`listening on port ${port}`));

Create the matching Dockerfile

// create the docker file
touch Dockerfile

// Add the follwing lines to Dockerfile
FROM node:alpine
WORKDIR /usr/app/src
COPY package\*.json ./

RUN npm install --production

COPY src .

CMD ["node", "index.js"]

Create Github action yaml to actually build the container and deploy it to cloud run

Go to:

GoogleCloudPlatform/github-actions

This is the Google cloud platform official cloud run example for github actions. Create the exact same file in your local repo under .github/workflows/cloud-run.yml

We won’t deep dive to the file structure, let’s focus only on few points:

PROJECT\_ID: ${{ secrets.RUN\_PROJECT }}

That is the project id we’re going to use in gcp. To find your project id, go to https://console.cloud.google.com/. Log in if you need to, and open the projects list from the top left menu. In the modal window, you’ll see the list of your projects, their names and ids.

RUN\_REGION: us-central1  
SERVICE\_NAME: helloworld-nodejs

You can control the service name and which region it is going to run.

service\_account\_email: ${{ secrets.SA\_EMAIL }} service\_account\_key: ${{ secrets.GOOGLE\_APPLICATION\_CREDENTIALS}}

We need to provide a service account and credentials so github can control our GCP project.

THIS IS VERY DANGEROUS AND IMPORTANT!!! RESTRICT AS MUCH ACCESS AS POSSIBLE FROM ANY SERVICE ACCOUNT

Go over the file if you want to see the rest of the stuff that’s going on, but it is fairly basic and does exactly what we described in the manual part.

Create a service account and update github secrets

Go to your GCP project console-> IAM & Admin-> Service account (https://console.cloud.google.com/iam-admin/serviceaccounts)

Click on Create Service Account

Fill in service account name, i.e. github-action, and a description

Click Create, and continue and done in the following two screens. You should now see your service account created successfully.

If you remember, we had three secrets in the cloud-run.yml file:

secrets.RUN\_PROJECET
secrets.GOOGLE\_APPLICATION\_CREDENTIALS
secrets.SA\_EMAIL

The SA_EMAIL is: github-action@…iam.gserviceaccount.com

To get the GOOGLE_APPLICATION_CREDENTIALS, click on the right menu and select create key

The following window will show:

Make sure you leave JSON selected and click create. A JSON file will be downloaded to you computer. Any one who has that file has the service account permissions to do what ever he wants. keep it safe, surely not as part of your repository.

The final step, it to base64 the content of the JSON file. On a mac you can use the following in terminal:

cat {PATH\_TO\_DOWNLOADED\_JSON\_FILE} | base64

Save the result, we will use it soon.

Make sure the service account can do what it needs to do

Our service account is worthless right now, it has no roles and it can’t do anything. Let’s give it some powers.

Go to the IAM pages, click Add and write the service account name we just created in the “new member” field — it should auto complete. Add the following roles to the service account:

Cloud builder — allow the service account to build and submit images
Service account user — allow the service account to act as the service
Cloud run admin — allow the service to deploy and start containers
Viewer on the project — allow the service account to list storage objects and use them (this is where our container data is stored)

Update Github secrets

Alright, we’re done with GCP, Open your repo in Github’s website. Go to settings and select the secret tab

Add 3 secrets to the repo:

RUN\_PROJECT - project id
GOOGLE\_APPLICATION\_CREDENTIALS - service account credentials, in base64 format
SA\_EMAIL - the service account email

Commit code and watch the magic happen

That’s it! we’re fully set! you can commit your code and let github do the work. Go to the actions page, there you would see what is going on. Here is an example from some of my playground using github actions.

Conclusion

This was kind of a long process, But you only have to do it once, and i tried to describe the hard parts in details.

You now have a fully functioning Github Action which deploys a cloud run container and hides all the delicate repetitive tasks so you can focus on coding.

With that said, this is only touching the ice tip. This solution is not fully production ready as there are more things to consider like handling failures, rollbacks, canary testing, unit testing and so on.

Let me know if I have made any mistakes or if you can think of a better/easier way of doing the same

How JSON.stringify killed my express server

efi shtain — Mon, 30 Sep 2019 20:09:12 +0000

Getting up to 300% more performance from an express server with a simple change

Photo by Max Chen on Unsplash

Express is one of the most used node.js frameworks out there when you need to create server.

Here is an example of a simple express server with only one endpoint returning a small static JSON response.

Running the code above and benchmarking it using autocannon for multiple times shows the server, running on a 2018 MacBook pro, handles roughly 190K requests in 11 seconds ~= 1900 RPS

➜ autocannon [http://localhost:3000/not\_cached](http://localhost:3000/not_cached)

We can do better!

Since there is so little code in this example, there is only one place where we can look for better performance — express source code, the definition of the json function on the response object (full code found at: https://github.com/expressjs/express/blob/master/lib/response.js)

The most important part happens in line 22, stringily — for every res.json that we use, the returned value is stringifed to be sent as an http response. After stringifing the data, the content type is set and response is sent.

JSON.stringify is a cpu bound operation, not node’s best friend, so let’s try and do it only once.

We can stringify the result and save it to variable, and for each incoming request we can set the content-type to application/json and use the end method to write the string directly to the socket:

Running autocannon again gives us around 350K requests in 11 seconds ~= 3500 RPS. 80% improvement.

But wait you say, you promised me 300% improvement !! and you would be right!

The performance difference depends heavily on the object returned. I wanted to show that even a small change on a small object can be significant.

Try doing the same thing with a big json object (like 500–600 Kb), you’ll get your performance boost. Actually, using res.json, might actually causes your server to crash in a limited environment like a container running on kubernetes.

Conclusions

When using express, if your server performs poorly RPS wise, try cache any shared response and write the string directly to the response stream instead of using res.json which uses JSON.stringify everytime.