DEV Community: Nathaniel

From Jobless to FAANG - My three year journey

Nathaniel — Sat, 16 Oct 2021 18:59:13 +0000

Disclaimer: this post is all about my personal experience, only representing my point of view, take it with a grain of salt.

About three years ago, I was graduating from college with a CS degree. Dreaming of going to U.S. to study, I gave up the opportunity of attending graduate program without entrance exam at my school. Unfortunately, I wasn't able to get into any U.S schools. Not only didn't I prepare for job interviews, I also missed all on-campus hiring event as well as new grad hiring time window of most companies.

At the point, I was fresh out of the college and jobless, possibly the most painful time of my life. I made some changes to my plan and tried to take an exam to get in another graduate school at my country because there's a pandemic going on and studying abroad seems impossible. Somehow I failed the exam. I was hopeless and desperately trying my best to find a job to get by.

After a plethora of interviews, I was fortunate enough to land a job at Ericsson. However, I was hired through a vendor so the pay and benefits are terrible. I had no choice but to learn on the job. I never wrote much javascript at school and it took me a long time to understand the mechanics of async and await.

Once I got the hang of it. I began creating side projects on GitHub. At first the projects are pretty simple: a REST api with jwt authentication and authorization, a single page made from react or vue, or a CMS with some CRUD functions. I was never satisfied with my work and always trying to explore and improve. I began learning typescript, learning advanced react usage, learning graphQL. I was also writing my learnings down as I progress. My most popular blog post had more than 10k views and is still at the first page of migrate to typescript google search result to this day. My toy react hook library also have thousands of download per week. Like all the developers, I'm also obsessed with creating my own blogging system. From markdown engine to database design, I was in from every aspect of my blogging system. My blog have gone through three major version change, reflecting my skill up. Going from ASP.NET Core+ JQuery + Bootstrap to ASP.NET Core + React + Component Library to Node.js + React + components written myself. Looking at my blog now, I can say that it came a long way.

Having a lot of side projects is not you gold ticket to FAANG though, There's also leetcode grind like everyone else. The only difference might be when I was practicing problems, I will first try to understand the mechanism of this type of problems, then I take the problem to strengthen my impression.

Then these's the interview, There's 6 rounds of technical interviews and some of them might even in 1AM due to timezone differences. I would just talk about projects at my previous companies and then solve some leetcode problems, not being able to find the optimal solution is fine, but you have to at least provide a promising solution.

After what feels like a century, I finally got the offer from Microsoft. To be honest, I didn't believe I would got into Microsoft at all and would never dream of getting into FAANG three years ago. Using GitHub, npm, typescript and vscode almost everyday, I'm quite impressed by the products that Microsoft was building. I'm really looking forward to take on new challenges at Microsoft.

How I make memes inside code

Nathaniel — Sun, 01 Aug 2021 06:35:59 +0000

Just another day, I was making memes on reddit, and writing code in the interval. It suddenly struck me, why can't I do both at the same time? Making memes inside code is totally possible, just make ascii based memes inside comments. However, it's very time consuming to

Find the meme template
Download it.
Convert it to ascii format
Paste it into IDE
Make them as comments

After a few tries, I decided to make a vscode extension to speed up this process by combing all those steps into one: Hitting a hotkey on your keyboard.

Extension Link: https://marketplace.visualstudio.com/items?itemName=llldar.vsc-memer

Github link: https://github.com/llldar/vsc-memer

Begin

In order to write a vscode extension, first you need yo and generator-code, install them by

npm install -g yo generator-code

After installation you can

yo code

And the command line will interactively prompt you a number of options and you can follow them one by one.

After the steps, you will have a working hello-word vscode extension.

Implementation

I was planning supporting 4 commands for this extension

insertRandomMeme
copyRandomMeme
insertMeme
copyMeme

In order to achieve those, I need use vscode extension API extensively.

To insert code, you first need to get current selection position, then you can insert text into those places.

editor?.selections.forEach((v) =>
  edit.replace(v, comment(getRandomMeme()))
);

One thing you need to know is that vscode keeps the inserted text selected after insertion, you need to unselect the text by clearing selection

editor.selection = new vscode.Selection(
  editor.selection.end,
  editor.selection.end
);

For copying text to clipboard, it's very easy, just use some simple vscode api, you need to notify users after copy

vscode.env.clipboard.writeText(comment(getRandomMeme()));
vscode.window.showInformationMessage('Random meme copied to clipboard');

Now inserting random memes are easy, while if you want to insert a meme selected by user, you need to have user input. Fortunately, we have QuickPick api that allows user to pick one from our list:

const options = Object.keys(memes).map((label) => ({ label }));
const quickPick = vscode.window.createQuickPick();
quickPick.items = options;

Show the QuickPick on user command:

quickPick.show();

Receive user input from callback, hide the quick pick after selection:

quickPick.onDidChangeSelection(([{ label }]) => {
  vscode.env.clipboard.writeText(
    comment(memes[label as keyof typeof memes])
  );
  vscode.window.showInformationMessage('Random meme copied to clipboard');
  quickPick.hide();
});

One more thing we need to consider, memes need to be wrapped inside comments, different language have different comment styles, we need to allow user to configure those by adding configuration inside pakcage.json like:

"configuration": [
    {
        "title": "Meme maker",
        "properties": {
            "vsc-memer.commentStyle": {
                "type": "string",
                "default": "line",
                "enum": [
                    "line",
                    "block"
                ],
                "enumDescriptions": [
                    "line comment style, like //",
                    "block comment style, like /* */"
                ],
                "description": "Type of comment style"
            },
            "vsc-memer.lineComment": {
                "type": "string",
                "default": "//",
                "description": "Symbol of line comments the generated meme should be wrapped in"
            },
            "vsc-memer.blockCommentLeft": {
                "type": "string",
                "default": "/*",
                "description": "Symbol of left side of block comments the generated meme should be wrapped in"
            },
            "vsc-memer.blockCommentRight": {
                "type": "string",
                "default": "*/",
                "description": "Symbol of right side of block comments the generated meme should be wrapped in"
            }
        }
    }
],

Note they needs to be under contributes block, otherwise they would do nothing.

Adding comment mark is easy, but one more thing we need to consider: windows and macOS/linux have different line endings. Also vscode can override system settings, it becomes more and more complicated now.

First we need to read vscode config files.eol, if it's not set to auto, we need to insert the option value.

If it's auto, we need get operating system's EOL by using node api os.EOL.

After implementing all the features, we need to register all the commands like:

context.subscriptions.push(insertRandomMeme);
context.subscriptions.push(copyRandomMeme);
context.subscriptions.push(insertMeme);
context.subscriptions.push(copyMeme);

Debugging

It's very straight forward, just hit F5 and you can debug in the child vscode window from host vscode window.

Publish

After you finished your code, you need another tool to publish it vsce, install it by:

npm i -g vsce

Also you need to have an Azure devops account, and create a publisher in vscode market place.

You also need to generate a personal access token with permission to manage you vscode marketplace and all accessible organizations from Azure devops.

Read offical documentation for more information.

After you have your personal access token you can

vsce login <your_publisher_name>

Then you can just

vsce publish

Promote

A great extension needs create promotion, since it's meme making extension, I decided to promote it mainly on reddit using memes. If you have discord server, twitter following, YouTube channel, you can always use them to promote.

Sorting 1 billion numbers

Nathaniel — Sun, 25 Jul 2021 14:11:35 +0000

Today we are going to try to sort 1 billion numbers. To be exact, they are 1 billion 32bit signed integers. Which have a size of around 4GB. Loading all of them into memory at once is possible, but not really the point here. I'm trying to practice external sorting, a way of sorting data that are too large to fit inside memory.

Beginning

The easiest way of external sorting is merge sort. We can use the large-capacity external storage to divide the 1 billion numbers into small pieces, loading and sorting them one by one, then write each piece into the external memory after sorting.

After the segmentation is completed, these small pieces are merged and sorted. At the same time, in the process of merging and sorting, the maximum value obtained will be written into the file in real time, so that low memory usage can be guaranteed.

Divide

Suppose we have a file named billion, which contains 1 billion 32 bit signed int binary data. We need to cut it up into a number of pieces. Here I divide it into 100 parts, with each part containing 10 million numbers. Each one of them need to be sorted. Here we use qsort to sort part of the file.

static __int32 piece[PIECESIZE];

int comp( const void *a, const void *b )
{
    return *(__int32*)a - *(__int32*)b;
}

FILE *billion = fopen( "billion", "rb" );
int i = 0;
while( i < TOTAL / PIECESIZE ){  
    fseek( billion, PIECESIZE * i, SEEK_SET );
    fread( piece, sizeof( *piece ), PIECESIZE, billion );

    qsort( piece, PIECESIZE, sizeof( *piece ), comp );

    char fileName[300];
    snprintf( fileName, sizeof(fileName), "pieces/piece%d", i );

    FILE *outFile = fopen( fileName, "wb" );
    fwrite( piece, sizeof( *piece ), PIECESIZE, outFile );
    fclose( outFile );
    ++i;
}

SORT

After 1 billion integers have been divided into 100 blocks, and these blocks were already sorted, we can read these blocks, apply merge sort, and write the results to the file in real time. During this period, memory consumption will remain low.

FILE *outFile = fopen( "out", "wb" );
FILE *fileList[FILEAMOUNT];  
int i = 0;
for( i = 0; i < FILEAMOUNT; ++i ){
    char filePath[200];
    snprintf( fileName, sizeof(fileName), "pieces/piece%d", i );
    fileList[i] = fopen( filePath, "rb" );
}

int numbers[ FILEAMOUNT ];
for( i = 0; i < FILEAMOUNT; ++i ){
    fread( numbers + i, sizeof( __int32 ), 1, fileList[i] );
}

int n = 0;
while( 1 ){
    int minIndex = MinIndex( numbers );
    if( minIndex == -1 )  break; 

    fwrite( numbers + minIndex, sizeof( __int32 ), 1, outFile );
    ++n;

    fread( numbers + minIndex, sizeof( __int32 ), 1, fileList[minIndex] );
    if( feof( fileList[minIndex] ) ){
        numbers[minIndex] = -1; 
    }
}

for( i = 0; i < FILEAMOUNT; ++i ){
    fclose( fileList[i] );
}
fclose( outFile );

Among them, the MinIndex function gets the subscript of the smallest value in the array, and it will skip when it encounters -1, I used number -1 as end of file. If the MinIndex function returns -1, it means that all files have been read.

int MinIndex( int *arr )
{
    int i, index = -1;
    for( i = 0; i < FILEAMOUNT; ++i ){
        if( arr[i] == -1 ) continue;
        if( index == -1 || arr[index] > arr[i] ) index = i;
    }
    return index;
}

Conclusion

According to the timer, it takes 240s to split + sort each block and 433s to merge. We can see that the hard disk IO performance is the bottleneck here.

Building a high concurrency student course selection system

Nathaniel — Sat, 24 Jul 2021 14:48:34 +0000

Intro

Remember those days when your school system suck so much that you can never pick the course you want? Well, those days are no more. After learning how to design a high concurrency student course selection system. You can make it so that everyone can pick their class in the most fair way possible.

Technology

Node.js, Postgresql, Redis, RabbitMQ

Database design

The first step of building a great system is great data modelling and subsequently a great database design. All following database schema was created in Prisma schema format.

Suppose you are in a school, some common entites in the school are teachers, students, courses, classes. So here we have

model Teacher {
  id           String     @id @default(dbgenerated("uuid_generate_v4()")) @db.Uuid
  idNumber     String     @unique
  name         String
  gender       String
  phoneNumber  String     @unique
  email        String     @unique
  passwordHash String
  age          Int
  jobTitle     String
  departmentId String     @db.Uuid
  department   Department @relation(fields: [departmentId], references: [id])
  courses      Course[]

  enrolledAt  DateTime
  createdAt   DateTime      @default(dbgenerated("now()"))
  updatedAt   DateTime      @default(dbgenerated("now()")) @updatedAt
  deletedAt   DateTime?
  courseClass CourseClass[]

  @@index([idNumber])
  @@index([name])
  @@index([email])
  @@index([phoneNumber])
}

Teachers, who are employees at school, who have jobTitle, email, department and teach some classes.

model Student {
  id            String               @id @default(dbgenerated("uuid_generate_v4()")) @db.Uuid
  idNumber      String               @unique
  name          String
  gender        String
  phoneNumber   String               @unique
  email         String               @unique
  passwordHash  String
  age           Int
  departmentId  String               @db.Uuid
  department    Department           @relation(fields: [departmentId], references: [id])
  courseClasses StudentCourseClass[]
  subjects      StudentSubject[]

  enrolledAt DateTime
  createdAt  DateTime  @default(dbgenerated("now()"))
  updatedAt  DateTime  @default(dbgenerated("now()")) @updatedAt
  deletedAt  DateTime?

  @@index([idNumber])
  @@index([name])
  @@index([email])
  @@index([phoneNumber])
}

Students, who study at school, have id number, and take some classes. Majoring some subjects and might minor some other subjects.

model Department {
  id       String    @id @default(dbgenerated("uuid_generate_v4()")) @db.Uuid
  name     String    @unique
  students Student[]
  teachers Teacher[]

  createdAt DateTime  @default(dbgenerated("now()"))
  updatedAt DateTime  @default(dbgenerated("now()")) @updatedAt
  deletedAt DateTime?
}

Departments, which consist of students and teachers, also have unique names.

model Course {
  id                 String           @id @default(dbgenerated("uuid_generate_v4()")) @db.Uuid
  name               String
  credit             Int
  courseClasses      CourseClass[]
  courseCategories   CourseCategory[]
  type               CourseType       @default(PUBLIC)
  selectableSubjects Subject[]

  createdAt DateTime  @default(dbgenerated("now()"))
  updatedAt DateTime  @default(dbgenerated("now()")) @updatedAt
  deletedAt DateTime?

  @@index([name])
}

enum CourseType {
  PUBLIC
  MAJOR_LIMITED
}

model CourseCategory {
  id      String   @id @default(dbgenerated("uuid_generate_v4()")) @db.Uuid
  name    String
  courses Course[]

  createdAt DateTime  @default(dbgenerated("now()"))
  updatedAt DateTime  @default(dbgenerated("now()")) @updatedAt
  deletedAt DateTime?
  @@index([name])
}

Courses, which have names, credits and can be divided into many categories, also a same course can have multiple courseClasses, taught by different teachers. Also there can be different types of courses, ones that can be select by everyone and ones that can only be selected by students of certain subjects.

model Subject {
  id          String           @id @default(dbgenerated("uuid_generate_v4()")) @db.Uuid
  name        String           @unique
  courses     Course[]
  students    StudentSubject[]
  departments Department[]
  teachers    Teacher[]

  createdAt DateTime  @default(dbgenerated("now()"))
  updatedAt DateTime  @default(dbgenerated("now()")) @updatedAt
  deletedAt DateTime?
}

model StudentSubject {
  studentId String  @db.Uuid
  subjectId String  @db.Uuid
  student   Student @relation(fields: [studentId], references: [id])
  subject   Subject @relation(fields: [subjectId], references: [id])

  isMajor Boolean @default(true)

  @@id([studentId, subjectId])
}

Subjects, which are chosen by students when enrolled. Multiple teachers can teach same subject. Student can major them or minor them, identified by isMajor column in relation table StudentSubject.

model CourseClass {
  id        String               @id @default(dbgenerated("uuid_generate_v4()")) @db.Uuid
  name      String
  course    Course               @relation(fields: [courseId], references: [id])
  courseId  String               @db.Uuid
  capacity  Int
  time      String
  location  String
  status    CourseClassStatus    @default(NEW)
  teacher   Teacher              @relation(fields: [teacherId], references: [id])
  teacherId String               @db.Uuid
  students  StudentCourseClass[]

  createdAt DateTime  @default(dbgenerated("now()"))
  updatedAt DateTime  @default(dbgenerated("now()")) @updatedAt
  deletedAt DateTime?

  @@index([name])
}

enum CourseClassStatus {
  NEW
  OPEN
  CONFIRMED
  CANCELED
}

CourseClasses, which are instances of classes for a certain course. They should have time, name, location and can have 1 teacher and many students. The number of students it can hold are indicated by capacity and it should have four statuses:

NEW means this course class is newly created, should only be visible to admin.
OPEN means this course class is open to be selected, all students who met the requirement of course should be able to see it.
CONFIRMED means this course class have all the students it needed and will take place in the future.
CANCELED means this course class might have too few students, therefore it's canceled.

model StudentCourseClass {
  student       Student         @relation(fields: [studentId], references: [id])
  studentId     String          @db.Uuid
  courseClass   CourseClass     @relation(fields: [courseClassId], references: [id])
  courseClassId String          @db.Uuid
  status        SelectionStatus @default(SELECTED)
  selectedAt    DateTime        @default(dbgenerated("now()"))

  @@id([studentId, courseClassId])
}

enum SelectionStatus {
  SELECTED
  CANCELED
}

As for the actual course class selection phase, we need a status to indicate the status of the selection to avoid abuse of the system in the form of SELECTED and CANCELED, if student have more than 3 classes canceled at the same day, we might pose some punishment for this particular student.

Business Scenarios

Preparation for course selection

Admin will first need to create new course classes from courses, filling a form about the course classes including name, location, time, capacities of the class, also assign the teacher of this class.

After creation, admin can setup the start end time when all classes are available to be selected, which can be cancelled at any time before the actual selection begins.

Course Selection

At the beginning of the course selection, students would be waiting in the lobby, waiting for course selection begin countdown.

After count down runs out, the page would automatically refresh, loading all course classes that are available to the student.

After click on the pick button, students will be prompted a modal, to see if student is going to confirm selecting this course class.

After student confirms, the course class status in the list would become SELECTED, and capacity would go up by 1.

Student can cancel courses after selection a course class by click on CANCEL button.

After three cancellations in the same day, student would not be able to cancel selection.

Course Selection Finished

After course selection time ends, student would not be able to see course list. But they should be able to see list of course they selected.

Admin should be able to see the overall selection result of each class, including class participants, and can make adjustments on the selection result.

Admin can also cancel course class if the select count is too low.

Key implementation

Preparation for course selection

Things to do before task selection:

Use timed background worker jobs to open up course selection.
Cache warm-up
Random token for course information requests to avoid people from requesting before selection starts
API gateway rate-limit

Setup redis and background jobs

Here we use bull for timed background jobs.

Admin will use UI to set a start, end time for course class selection.

First we store studentCouseClassBegin and studentCouseClassEnd keys with value of start end time into Redis. We then use bull to create a new queue named CoureSelectionQueue, add a task to remove the key from Redis at end time.

When user request for available course list, we check the Redis for studentCouseClassBegin and studentCouseClassEnd, depending on the result, we can have 4 type of response.

No keys, we can return Course selection not available.
Current time < studentCouseClassBegin, return xxx mintues before course selection begins, admin can call cancel option, which would remove both keys from Redis and remove the task from bull queue.
studentCouseClassBegin < Current time < studentCouseClassEnd, we return xxx minutes before course selection ends.
Current time > studentCouseClassEnd , we return Couse selection have ended.

Cache warm-up

Before course selection begins, we need to warm it up to avoid scenarios where too many people are requesting at the same time and due to cache is mostly empty, most of requests would hit our api server and database, causing cache breakdown, cache penetration and even cache avalanche.

To warm-up the the cache, first we need to know what to cache. Based on our database model, we need to:

Get information about course class and teacher by id
Generate a random uuid as token for request.
Use redis-semaphore to generate a distributed semaphore, the key of semaphore is generated token and value is the capacity for a certain course class.

We use hashes to store the information. For each course class, key is course class id, value is nested course class information read from db and the random token in json form.

We also use the same token for key of semaphore we mentioned before and course capacity as its value.

This combo stores all the information we need before querying and actually selecting course classes.

Course Selection

We then need to store the StudentCourseClass information, with key of couseClass Id + studentId value is the selection status.

Select

To prevent malicious request(by bot or script), api would only send out token after course selection begins. Without token, api would invalidate the request instantly. Also the value inside Redis is tightly coupled with token, so api can only find the data within Redis if it has the correct token.

Considering there might be multiple instances of api, we only want data be cached into Redis once, We will need to check if data already exist inside cache as well as adding mutex lock using redis-semaphore.

Student selecting a course, we will need capacha to filter bots and reduce the peak of request count.
After request hits api, there should be a set of request validation, including whether user is authenticated, capacha, random token and if this student have already selected this class.
If request is valid, we use acquire() function to try to reduce the capacity of this course class.
After reduction success, we will need to read and update course information from Redis. Then we need to generate a course selection data, write it to message queue.

Cancel

Basically the same as the select, after request validation, we would just release 1 semaphore based on the token.

Message queue

After request hit the message queue. There's still a lot going on.

We will need one exchange and a queue for rabbitmq, also an api route that listens on the queue. When the route receives message from message queue, it will then update the information to database and then send ack to message queue.

We also needed to ensure the reliability of message queue.

Message loss:
- Message didn't reach broker: before sending message, store message information to Redis, and remove it once we received confirm from broker. Also we need to resend message inside Redis with a maximum of 3 times. If message failed to send for more than 3 times. we need manual intervention.
- ack didn't reach broker: retry 3 times, then manual intervention.
Message duplication: since we have message retries, there could be message duplication. We need to make sure the operation on the receiving api is Idempotent.
Message backlog: Using dedicated message queue service, Persist all messages into database and process them later.

Ending

At this point, our high concurrency course selection system has been finished. Under high concurrency, our selection and cancel api only have a few Redis IOs and negligible cost of sending message to message queue. The response time is very short, which is enough to achieve high concurrency.

The main reason for the good performance of this api is due to the asynchronous implementation of the message queue. Database operations that might have taken a long time is replaced by sending a message, which would only take a few milliseconds.

The making of a modern javascript library

Nathaniel — Mon, 21 Jun 2021 16:34:13 +0000

Every 60 seconds passed in Africa, a new javascript library is created. There are a plethora of js libraries out there, and today I made another one.

What

Recently, I need to build some function revolving around syncing contacts and calendar events. They were mostly based on an old protocol named WEBDAV. It's so old that it uses convoluted xml structure to represent data. It would be hard for me to handle data since JSON is the first class citizen in javascript while xml is not.

Why

I took a look at all the existing solutions. They either are no longer maintained or lack core functions. Therefore, I've decided create a brand new library in order to fulfil these goals:

Allow using json data. In order to handle data with ease.
Support a wide range of protocols. Not just one of the three DAV families, but all three of them.
Well tested, have little to no bugs.
First class typescript support, well defined type with minimum amount of any
Have an easy to read, searchable documentation.

How

First thing you need to do is understand the problem, it can be done by doing any of the following:

Reading previous project's code, understand the pros and cons of the previous library.
If your target have documented standards (in my case IETF RFC 4918, 4791, 6352), read them throughly, understand the reason behind the standards and the problem it aims to solve.
Write small sandboxed code samples. By starting small, you'll solve the problem piece by piece, eventually allowing the solution to the entire problem.

After making sure that you understand the problem, you can then proceed to start writing code. It's recommended to write unit tests alone the way so that you can have good code quality and be confident about every part of the project as you move.

When you finished writing code, you can setup CI pipeline which allows every commit and pull request is tested and meet the standard of code quality. It can also help automating things like deploying the documentation. I'm not going to explain the details here since they are outside the scope of this post.

Doing all of the above will only make your a decent project. To make your project great (again?), you need to do most, if not all of the following:

Design a cool logo for it, best paired with a slick slogan.
Added badges by using shields.io badges.
Provide a clean, easy to understand README on how to use the project.
Add tags that helps others to discover the great work you've done.

Takeaway

Making a good library is quite an undertaking. It take months and sometimes even years of hardworking for a good library to be made. After making this library, I'm now much more incline to give other devs projects stars. The project itself might not be useful to you, but by the act of starring the project, you appreciate the hard work of maintainers and allows the library to gain more visibility.

How I build an app in 2 hours

Nathaniel — Thu, 18 Feb 2021 14:51:06 +0000

Yesterday I built an android app in 2 hours with kotlin

without even knowing how to write hello world in kotlin before starting to work on it.

Granted, the app is really simple and I've written some demo app at around 2014 when I'm still at school.

The point of this article is not to showing off or anything like that, the point is to share the methods to get things done quickly

Background

My father is a teacher. He always wanted an app on his phone that can do the followings:

Store the information of his students (name, score, student id number, etc)
Standard CRUD operation on the student information.
Customized theme and accent color.
Easily add the score of one of his students by 1.
Sort the students by score or by id.

At first glance, these functions seem to be not that hard. However, my time is valuable and I do not want to spend too much time on this project (for obvious reasons).

Analyze the core

Given the task, I need to find the core part of the application. The essential parts of the system are:

A database, to store
A list view, to show all entities.
A detail view, to inspect, edit or delete a specific entity.
Three events, to link the action of user input to
- Jump from list view to detail view
- Switch between two methods of sorting
- Add the score of entity by 1

Now we have the core function of this app, we need to figure out how to implement them in android, after some quick google search and google documentations. I now know:

Database -> room database
List view -> recycle view
Detail view -> just an activity
Three events -> event listeners
Kotlin is the preferred language for android development

The power of boilerplate

After knowing the core tech I need, I now search for android kotlin room crud recyclerview boilerplate on google. I was very fortunate to find a boilerplate that fits my exact need on github. The tech you might need is different, but the general approach is the same.

With the boilerplate, I now need to identify the changes I need to make:

Customize the name, theme and accent color the app.
Change the data model to be a student
Change all related code to make sure CRUD operations work perfectly.
Add event listeners like the one in the boilerplate for the actions I need.

Knowledge transfer

When making the changes, I encountered two key problems:

There's no built-in double click event, I need to somehow make it myself
Database operation cannot reside in main thread, it must be working with something called suspend functions in kotlin.

I developed a lot of javascript so I applied a few techniques I learned in the javascript world and managed to solve the problems like:

Use a boolean to mimic double click, basically there's a boolean and a timer. The first click reverse the boolean and start the timer, if the second click occured within the timer time frame, it's counted as a double click. After timer expires, the boolean go back to its original value.
suspend is basically async, in javascript we solve this kind of problem with iife : Immediately invoked function expression, maybe there's something like suspend lambda that was invoked immediately in kotlin as well.

By applying what I've learned developing using javascript, I managed to solve my problems and finished the project within 2 hours, I installed it on my dad's phone and he's super happy.

Key takeaway

When time is limited and money is short, the best way to maximize your output is to leverage all the help you can get. Open source is a gift for all developers, there's no shame in using it. All you need to do is see the core of the application and apply knowledges of different domain.

Typescript function return type based on parameters

Nathaniel — Mon, 11 Jan 2021 17:09:41 +0000

Typescript is all fun and games until you want some behaviour based on runtime values, recently I encountered a tricky problem: How do I type a function's return type based on the parameter value?

I know this sound like an anti-pattern but there are many real world use case for it, for example your function have an option field that will determine the type of value it returns:

type User = {
  id: string;
  firstName: string;
  lastName: string;
  profilePicture?: string | ProfilePicture;
};
type ProfilePicture = {
  height: string;
  width: string;
  url: string;
};

const db = {
  findUserById: async (userId: string): Promise<User> => ({
    id: '1',
    firstName: 'Bruce',
    lastName: 'Wayne',
  }),
};

const generateProfilePictureById = async (
  userId: string
): Promise<ProfilePicture> => ({
  height: '20px',
  width: '20px',
  url: `http://example.com/${userId}.png`,
});

const getUserProfile = async (
  userId: string,
  options?: { generateProfilePicture: boolean }
) => {
  const user = await db.findUserById(userId);
  if (options?.generateProfilePicture) {
    return {
      ...user,
      profilePicture: await generateProfilePictureById(userId),
    };
  }
  return { ...user, profilePicture: 'picture not generated' };
};

Now if you want to use getUserProfile like:

(async () => {
  const user = await getUserProfile('1', { generateProfilePicture: true });
  console.log(
    `${user.firstName} ${user.lastName} has profilePicture with height: 
    ${user.profilePicture.height}`
  );
})();

Typescript will complain that height does not exist on user.profilePicture

But you know that if generateProfilePicture option is set to true, user.profilePicture will not be the inferred type string | ProfilePicture

How do we solve this problem then? Typescript have the answer: Function overload

Basically, typescript will map multiple signatures of a function in an order of their appearance in code. It will use the first matching type signature for that function.

Knowing this, let's improve the typing of our function getUserProfile:

interface GetUserProfileType {
  <T extends boolean>(
    userId: string,
    options?: { generateProfilePicture: T }
  ): Promise<
    Omit<User, 'profilePicture'> & {
      profilePicture: T extends true ? ProfilePicture : string;
    }
  >;
  (
    userId: string,
    options?: { generateProfilePicture: boolean }
  ): Promise<User>;
}
const getUserProfile: GetUserProfileType = async (
  userId: string,
  options?: { generateProfilePicture: boolean }
) => {
  const user = await db.findUserById(userId);
  if (options?.generateProfilePicture) {
    return {
      ...user,
      profilePicture: await generateProfilePictureById(userId),
    };
  }
  return { ...user, profilePicture: 'picture not generated' };
};

Now our user.profilePicture will be string when generateProfilePicture is false, and ProfilePicture when generateProfilePicture is true.

But wait, there's more

What if we omit the options entirely and use it like:

(async () => {
  const user = await getUserProfile('1');
  console.log(
    `${user.firstName} ${user.lastName} has profilePicture with height: 
    ${user.profilePicture.length}`
  );
})();

Now for the above code typescript complains: Property 'length' does not exist on type 'ProfilePicture'. Apparently it did not match with any of the two function overloads. Well, guess three time is a charm, let's add the third function overload:

interface GetUserProfileType {
  <T extends { generateProfilePicture: boolean } | undefined>(
    userId: string,
    options?: T
  ): Promise<
    Omit<User, 'profilePicture'> & {
      profilePicture: T extends undefined ? string : never;
    }
  >;
  <T extends boolean>(
    userId: string,
    options?: { generateProfilePicture: T }
  ): Promise<
    Omit<User, 'profilePicture'> & {
      profilePicture: T extends true ? ProfilePicture : string;
    }
  >;
  (
    userId: string,
    options?: { generateProfilePicture: boolean }
  ): Promise<User>;
}

Now the code is working as expected.

Migrate to typescript - the advance guide

Nathaniel — Mon, 07 Dec 2020 19:36:25 +0000

About a year ago I wrote a guide on how to migrate to typescript from javascript on node.js and it got more than 7k views. I did not have much knowledge on javascript nor typescript at the time and might have been focusing too much on certain tools instead of the big picture. And the biggest problem is that I didn't provide a solution to migrating large projects where you obviously not going to rewrite everything in a short time, thus I feel the urge to share the greatest and latest of what I learned on how to migrate to typescript.

The entire process of migrating your mighty thousand-file mono-repo project to typescript is easier than you think. Here's 3 main steps on how to do it.

NOTE: This article assumes you know the basics of typescript and use Visual Studio Code, if not, some details might not apply.

Relevant code for this guide: https://github.com/llldar/migrate-to-typescript-the-advance-guide

Typing Begins

After 10 hours of debugging using console.log, you finally fixed that Cannot read property 'x' of undefined error and turns out it's due to calling some method that might be undefined: what a surprise! You swear to yourself that you are going to migrate the entire project to typescript. But when looking at the lib, util and components folder and those tens of thousands of javascript files in them, you say to yourself: 'Maybe later, maybe when I have time'. Of course that day never come since you always have "cool new features" to add to the app and customers are not going to pay more for typescript anyway.

Now what if I told you that you can migrate to typescript incrementally and start benefiting from it immediately?

Add the magic `d.ts`

d.ts files are type declaration files from typescript, all they do is declaring various types of objects and functions used in your code and does not contain any actual logic.

Now considering you are writing a messaging app:

Assuming you have a constant named user and some arrays of it inside user.js

const user = {
  id: 1234,
  firstname: 'Bruce',
  lastname: 'Wayne',
  status: 'online',
};

const users = [user];

const onlineUsers = users.filter((u) => u.status === 'online');

console.log(
  onlineUsers.map((ou) => `${ou.firstname} ${ou.lastname} is ${ou.status}`)
);

Corresponding user.d.ts would be

export interface User {
  id: number;
  firstname: string;
  lastname: string;
  status: 'online' | 'offline';
}

Then you have this function named sendMessage inside message.js

function sendMessage(from, to, message)

The corresponding interface in message.d.ts should look like:

type sendMessage = (from: string, to: string, message: string) => boolean

However, our sendMessage might not be that simple, maybe we could have used some more complex types as parameter, or it could be an async function

For complex types you can use import to help things out, keep types clean and avoid duplicates.

import { User } from './models/user';
type Message = {
  content: string;
  createAt: Date;
  likes: number;
}
interface MessageResult {
  ok: boolean;
  statusCode: number;
  json: () => Promise<any>;
  text: () => Promise<string>;
}
type sendMessage = (from: User, to: User, message: Message) => Promise<MessageResult>

NOTE: I used both type and interface here to show you how to use them, you should stick to one of them in your project.

Connecting the types

Now that you have the types, how does them work with your js files?

There are generally 2 approaches:

Jsdoc typedef import

assuming user.d.ts are in the same folder, you add the following comments in your user.js:

/**
 * @typedef {import('./user').User} User
 */

/**
 * @type {User}
 */
const user = {
  id: 1234,
  firstname: 'Bruce',
  lastname: 'Wayne',
  status: 'online',
};

/**
 * @type {User[]}
 */
const users = [];

// onlineUser would automatically infer its type to be User[]
const onlineUsers = users.filter((u) => u.status === 'online');

console.log(
  onlineUsers.map((ou) => `${ou.firstname} ${ou.lastname} is ${ou.status}`)
);

To use this approach correctly, you need to keep the import and export inside your d.ts files. Otherwise you would end up getting any type, which is definitely not what you want.

Triple slash directive

Triple slash directive is the "good ol'way" of import in typescript when you are not able to use import in certain situations.

NOTE: you might need to add the following to your eslint config file when deal with triple slash directive to avoid eslint errors.

{
  "rules": {
    "spaced-comment": [
      "error",
      "always",
      {
        "line": {
          "markers": ["/"]
        }
      }
    ]
  }
}

For message function, add the following to your message.js file, assuming message.js and message.d.ts are in the same folder

/// <reference path="./models/user.d.ts" /> (add this only if you use user type)
/// <reference path="./message.d.ts" />

and then add jsDoc comment above sendMessage function

/**
* @type {sendMessage}
*/
function sendMessage(from, to, message)

You would then find out that sendMessage is now correctly typed and you can get auto completion from your IDE when using from , to and message as well as the function return type.

Alternative, you can write them as follows

/**
* @param {User} from
* @param {User} to
* @param {Message} message
* @returns {MessageResult}
*/
function sendMessage(from, to, message)

It's a more of a convention to writing jsDoc function signatures. But definitely more verbose.

When using triple slash directive , you should remove import and export from your d.ts files, otherwise triple slash directive will not work , if you must import something from another file use it like:

type sendMessage = (
  from: import("./models/user").User,
  to: import("./models/user").User,
  message: Message
) => Promise<MessageResult>;

The reason behind all these is that typescript treat d.ts files as ambient module declarations if they don't have any imports or exports. If they do have import or export, they will be treated as a normal module file, not the global one, so using them in triple slash directive or augmenting module definitions will not work.

NOTE: In your actual project, stick to one of import and export or triple slash directive , do not use them both.

Automatically generate `d.ts`

If you already had a lot of jsDoc comments in your javascript code, well you are in luck, with a simple line of

npx typescript src/**/*.js --declaration --allowJs --emitDeclarationOnly --outDir types

Assuming all your js files are inside src folder, your output d.ts files would be in types folder

Babel configuration(optional)

If you have babel setup in your project, you might need to add this to your babelrc

{
  "exclude": ["**/*.d.ts"]
}

To avoid compiling the *.d.ts files into *.d.js , which doesn't make any sense.

Now you should be able to benefit from typescript (autocompletion) with zero configuration and zero logic change in your js code.

The type check

After at least more than 70% of your code base is covered by the aforementioned steps, you now might begin considering switch on the type check, which helps your further eliminate minor errors and bugs inside your code base. Don't worry, you are still going to use javascript for a while, which means no changes in build process nor in library.

The main thing you need to do is add jsconfig.json to your project.

Basically it's a file that define the scope of your project and defines the lib and the tools you are going to work with.

Example jsonconfig.json file:

{
  "compilerOptions": {
    "module": "commonjs",
    "target": "es5",
    "checkJs": true,
    "lib": ["es2015", "dom"]
  },
  "baseUrl": ".",
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

The main point here is that we need checkJs to be true, this way we enable type check for all our js files.

Once it's enabled, expect a large amount of errors, be sure fix them one by one.

Incremental typecheck

// @ts-nocheck

In a file, if you have some js file you would rather fix later , you can // @ts-nocheck at the head of the page and typescript complier would just ignore this file.

// @ts-ignore

What if you just want you ignore 1 line instead of the entire file? Use // @ts-ignore. It will just ignore the line below it.

// @ts-expect-error

It's like @ts-ignore, but better. It allows typescript compiler to complain when there's no longer error somewhere, you'll know to remove this comment.

These three tags combined should allow you fix type check errors in your codebase in a steady manner.

External libraries

Well maintained library

If you are using a popular library, chances are there are already typing for it at DefinitelyTyped , in this case, just run:

yarn add @types/your_lib_name --dev

npm i @types/your_lib_name --save-dev

NOTE: if you are installing a type declaration for an organisational library whose name contains @ and / like @babel/core you should change its name to add __ in the middle and remove the @ and /, resulting in something like babel__core.

Pure Js Library

What if you used a js library that the author archived 10 years ago and did not provide any typescript typing? It's very likely to happen since the majority of the npm models still use javascript. Adding @ts-ignroe doesn't seem like a good idea since you want your type safety as much as possible.

Now you need to augmenting module definitions by creating a d.ts file, preferably in types folder, and add your own type definitions to it. Then you can enjoy the safe type check for your code.

declare module 'some-js-lib' {
  export const sendMessage: (
    from: number,
    to: number,
    message: string
  ) => Promise<MessageResult>;
}

After all these you should a have pretty good way to type check your codebase and avoid minor bugs.

The type check rises

Now after you fixed more than 95% of the type check errors and is sure that every library have corresponding type definitions. You may process to the final move: Officially changing your code base to typescript.

NOTE: I will not cover the details here since they were already covered in my earlier post

Change all files into `.ts` files

Now it's time to merge the d.ts files with you js files. With almost all type check errors fixed and type cover for all your modules. What you do is essentially changing require syntax to import and putting everything into one ts file. The process should be rather easy with all the work you've done prior.

Change jsconfig to tsconfig

Now you need a tsconfig.json instead of jsconfig.json

Example tsconfig.json

Frontend projects

{
  "compilerOptions": {
    "target": "es2015",
    "allowJs": false,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "noImplicitThis": true,
    "strict": true,
    "forceConsistentCasingInFileNames": true,
    "module": "esnext",
    "moduleResolution": "node",
    "resolveJsonModule": true,
    "isolatedModules": true,
    "noEmit": true,
    "jsx": "preserve",
    "lib": ["es2020", "dom"],
    "skipLibCheck": true,
    "typeRoots": ["node_modules/@types", "src/types"],
    "baseUrl": ".",
  },
  "include": ["src"],
  "exclude": ["node_modules"]
}

Backend projects

{
  "compilerOptions": {
      "sourceMap": false,
      "esModuleInterop": true,
      "allowJs": false,
      "noImplicitAny": true,
      "skipLibCheck": true,
      "allowSyntheticDefaultImports": true,
      "preserveConstEnums": true,
      "strictNullChecks": true,
      "resolveJsonModule": true,
      "moduleResolution": "node",
      "lib": ["es2018"],
      "module": "commonjs",
      "target": "es2018",
      "baseUrl": ".",
      "paths": {
          "*": ["node_modules/*", "src/types/*"]
      },
      "typeRoots": ["node_modules/@types", "src/types"],
      "outDir": "./built",
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

Fix any addition type check errors after this change since the type check got even stricter.

Change CI/CD pipeline and build process

Your code now requires a build process to generate to runnable code, usually adding this to your package.json is enough:

{
  "scripts":{
    "build": "tsc"
  }
}

However, for frontend projects you often would need babel and you would setup your project like this:

{
  "scripts": {
    "build": "rimraf dist && tsc --emitDeclarationOnly && babel src --out-dir dist --extensions .ts,.tsx && copyfiles package.json LICENSE.md README.md ./dist"
  }
}

Now make sure your change your entry point in your file like this:

{
  "main": "dist/index.js",
  "module": "dist/index.js",
  "types": "dist/index.d.ts",
}

Then you are all set.

NOTE: change dist to the folder you actually use.

The End

Congratulations, your codebase is now written in typescript and strictly type checked. Now you can enjoy all typescript's benefits like autocomplete, static typing, esnext grammar, great scalability. DX is going sky high while the maintenance cost is minimum. Working on the project is no longer a painful process and you never had that Cannot read property 'x' of undefined error ever again.

Alternative method:

If you want to migrate to typescript with a more "all in" approach, here's a cool guide for that by airbnb team

Use recoil in react custom hooks

Nathaniel — Mon, 13 Jul 2020 16:58:26 +0000

Recoil is a hot new state management tool from Facebook that handles the state sharing problem gracefully with a minimal API. I recommend everyone check it out. It's an amazing tool that feels very react-y and require little to little to no boilerplate code. I'm not going to cover it too much in this article. Check out the official documentation for more information.

As amazing as it is, the library do have its own limitations. One of the biggest limitation is that the useRecoilState and useRecoilValue hook does not work properly in custom hooks. Since recoil only subscribe and update those values inside react component. We will have to update the value by ourselves.

How? You may ask. It's magic time, let's use useRef

Voilà

import { useEffect, useRef } from 'react';
import { useRecoilState } from 'recoil';
import { someAtom } from './recoilstates/someState';

const yourCustomHook = () => {
  const [someState, setSomeState] = useRecoilState(someAtom);
  const latestSomeState = useRef(someState);

  useEffect(()=> {
    latestSomeState.current = someState;
  },[someState]);

  // I use useEffect here as an example,
  // custom hook can contain any React hooks
  useEffect(()=>{
    // your custom hook logic here
    // use latestSomeState.current for latest value
  });
}

By using useRef, and useEffect, the react component which this hook is used in will be subscribed to recoil state changes. Therefore, you can get the latest version of the state value in the hook.

Speed up your build by putting node_modules in ram

Nathaniel — Fri, 03 Apr 2020 07:28:39 +0000

After a day of diligently copying from Stackoverflow and Google searches, you finally ready to build that 10GB Frankenstein project. You type npm run build into the console then went home and enjoy the rest of your day. The next day you back to the office and surprised to find that the build is still in progress. "What should I do to speed this up?" you ask yourself. And then you came to the right place: This article.

Jokes aside, building JavaScript projects is a time consuming process, although we have more advanced build tools than ever before, it never make sense why your simple website take so much time to be build. Turned out there is a way to make it build faster: by putting the node_modules inside your ram.

Why? you may ask, it requires a bit of knowledge of OS, I'll explain the basics here. When building the project, there are a large amount underlying tasks involved, the main ones are parsing, bundling and minifying which are all I/O related. Since I/O is usually the bottleneck of an operating system, putting files that are frequently read/wrote in an extremely fast I/O device is very helpful to reducing the overall build time. Note we only put node_modules because data in ram is very volatile and maybe lost in case of system crash or power outage. Thus putting the code that you wrote in ram is not recommended.

How? It differs between operating system, the basic idea is to build up a ram disk and then link your node_modules there, here's the detail guide on each system.

Linux

Linux is the best suited for this task because of the system built-in tools.

tmpfs is a fantastic way of mounting your data into ram, simply type

sudo mount -t tmpfs -o size=2G tmpfs node_modules

there are a couple of caveats:

if your project is large, increase the size so that node_modules can fit in.
if your node_modules locates somewhere else, you need to change the path to the location.
files in tmpfs will sometimes get swapped out to disk, so the process may be slow down.

ramfs is preferred if you have a lot of ram space (i.e. 16G or more)

sudo mount -t ramfs ramfs node_modules

couple of things to note:

if you are using ramfs and your ram is full, your system will hang
you cannot limit the size of it
both ramfs andtmpfs behaves different from real ext4 filesystem in some way.

To release the ram space, simply use umount

sudo umount node_modules

macOS

macOS is a tricking one, your shining macbook lacks the tools of linux, so you need to manually manage your ram disk and system link.

First you need to create a ram disk

for macOS <= 10.12

diskutil erasevolume HFS+ “ramdisk” hdiutil attach -nomount ram://4194304

for macOS >= 10.13 and you are using APFS

diskutil partitionDisk $(hdiutil attach -nomount ram://4194304) 1 GPTFormat APFS 'ramdisk' '100%'

check device identifier of your ramdisk:

diskutil info /Volumes/ramdisk

here we have /dev/disk1s2, yours might be different.

then you can mount node_modules into the ramdisk

diskutil mount -mountPoint node_modules /dev/disk1s2

Alternatively you can use symbolic link

mv node_modules node_modules_files &&
ln -s /Volumes/ramdisk ./node_modules &&
/bin/cp -r node_modules_files/* node_modules

To free up ram space, use this:

diskutil eject /Volumes/ramdisk

Windows

Windows is not suited for this task, install a linux virtual machine for this.

Well, no that's a joke. There's actually a way, it's just more difficult.

First you need to get ImDisk, since windows have no built-in tool to create a ram disk.

Then let's say the ramdisk is R:

you can create a symbolic link by using mklink

mklink /d C:\path\to\node_modules R:\node_modules

To free up ram space just unmount it in ImDisk, also you can save it as an image file for future use.

Use faster build tool?

All these time we've been trying to make our build faster by going one way. Maybe there is another way of achieving the same goal? Turned out there is: If you want absolute build speed, esbuild might be the tool that you are looking for. It's at least 10 times faster than all the current build tools. The main downside of it is that it's brand new and not a lot of people have used it so It's unwise to use it in production. However I believe it's a great choice for hobby/side projects where risk is low. It maybe the ultimate solution for build speed problems.

credits: the idea of this post came from this tweet.

Migrate to Typescript on Node.js

Nathaniel — Fri, 04 Oct 2019 08:11:43 +0000

Recently I've migrated one of my personal projects from Javascript to Typescript.

The reason for migrating will not be covered here, since it's more of a personal choice.

This guide is for those who know something about Javascript but not much about Typescript and are mainly focus on Node.js applications.

Let's get right into it!

Add tsconfig.json

In order for Typescript to work, the first thing you need is a tsconfig.json

It tells the Typescript compiler on how to process you Typescript code and how to compile them into Javascript.

my config look like this:

{
  "compilerOptions": {
    "sourceMap": true,
    "esModuleInterop": true,
    "allowJs": true,
    "noImplicitAny": true,
    "moduleResolution": "node",
    "lib": ["es2018"],
    "module": "commonjs",
    "target": "es2018",
    "baseUrl": ".",
    "paths": {
      "*": ["node_modules/*", "src/types/*"]
    },
    "typeRoots": ["node_modules/@types", "src/types"],
    "outDir": "./built"
  },
  "include": ["./src/**/*", "jest.config.js"],
  "exclude": ["node_modules"]
}

now let me explain what each line means:

sourceMap Whether or not typescript generate sourceMap files. since sourceMap files help map the generated js file to the ts file, it's recommended to leave this on because it helps debugging.
esModuleInterop Support the libraries that uses commonjs style import exports by generating __importDefault and __importStar functions.
allowJs Allow you to use .js files in your typescript project, great for the beginning of the migration. Once it's done I'd suggest you turn this off.
noImplicitAny Disallow implicit use of any, this allow us to check the types more throughly. If you feel like using any you can always add it where you use them.
moduleResolution Since we are on Node.js here, definitly use node.
lib The libs Typescript would use when compiling, usually determined by the target, since we use Node.js here, there's not really any browser compatibility concerns, so theoretically you can set it to esnext for maximum features, but it all depend on the version of you Node.js and what you team perfer.
module Module style of generated Js, since we use Node here, commonjs is the choice
target Target version of generated Js. Set it to the max version if you can just like lib
baseUrl Base directory, . for current directory.
paths When importing modules, the paths to look at when matching the key. For example you can use "@types": ["src/types"] so that you do not have to type "../../../../src/types" when trying to import something deep.
typeRoots Directories for your type definitions, node_modules/@types is for a popular lib named DefinitelyTyped. It includes all the d.ts files that add types for most of the popular Js libraries.
outDir The output directory of the generated Js files.
include Files to include when compiling.
exclude Files to exclude when compiling.

Restructure the files

Typically you have a node.js project structure like this:

projectRoot
├── folder1
│   ├── file1.js
│   └── file2.js
├── folder2
│   ├── file3.js
│   └── file4.js
├── file5.js
├── config1.js
├── config2.json
└── package.json

With typescript, the structure need to be changed to something like this:

projectRoot
├── src
│   ├── folder1
│   │   └── file1.js
│   │   └── file2.js
│   ├── folder2
│   │   └── file3.js
│   │   └── file4.js
│   └── file5.js
├── config1.js
├── config2.json
├── package.json
├── tsconfig.json
└── built

The reason for this change is that typescript need a folder for generated Js and a way to determine where the typescript code are. It is especially important when you have allowJs on.

The folder names does not have to be src and built , just remember to name them correspondingly to the ones you specified in tsconfig.json.

Install the types

Now after you've done the above, time to install the Typescript and the types for you libraries.

yarn global add typescript

npm install -g typescript

Also for each of your third party libs:

yarn add @types/lib1 @types/lib2 --dev

npm install @types/lib1 @types/lib2 --save-dev

Setup the tools

ESlint

The aslant config you use for Js need to be changed now.

Here's mine:

{
  "env": {
    "es6": true,
    "node": true
  },
  "extends": [
    "airbnb-typescript/base",
    "plugin:@typescript-eslint/recommended",
    "prettier/@typescript-eslint",
    "plugin:prettier/recommended",
    "plugin:jest/recommended"
  ],
  "globals": {
    "Atomics": "readonly",
    "SharedArrayBuffer": "readonly"
  },
  "parser": "@typescript-eslint/parser",
  "parserOptions": {
    "ecmaVersion": 2018,
    "sourceType": "module"
  },
  "rules": {
    "no-plusplus": ["error", { "allowForLoopAfterthoughts": true }]
  }
}

I use ESlint with Prettier and jest. I also use airbnb's eslint config on js and I'd like to keep using them on typescript.

You need to install the new plugins by:

yarn add @typescript-eslint/eslint-plugin @typescript-eslint/parser eslint-config-airbnb-typescript --dev

npm install @typescript-eslint/eslint-plugin @typescript-eslint/parser eslint-config-airbnb-typescript --save-dev

Remember to change your eslint parser to @typescript-eslint/parser so that it can parse typescript.

nodemon

Nodemon is a great tool when you need to save changes and auto restart your program.

For typescript I recommend a new tool ts-node-dev. Because configuring the nodemon is a lot harder, while the ts-node-dev works right out of the box with zero configuration. They basically do the same thing anyway.

yarn add ts-node-dev ts-node --dev

npm install ts-node-dev ts-node --save-dev

Jest

I use jest for testing, the config need to adjust to Typescript as well

module.exports = {
  globals: {
    'ts-jest': {
      tsconfig: 'tsconfig.json'
    }
  },
  moduleFileExtensions: ['ts', 'js'],
  transform: {
    '^.+\\.(ts)$': 'ts-jest'
  },
  testEnvironment: 'node'
};

Apparently you need ts-jest

yarn add ts-jest --dev

npm install ts-jest --save-dev

Then add ts in moduleFileExtensions, since my application is a backend only application, I didn't add jsx or tsx here, you can add them if you need to use react.

Also you need to add

globals: {
  'ts-jest': {
    tsconfig: 'tsconfig.json'
  }
}

to let Jest know what's you Typescript config.

Package.json scripts

The scripts section in your package.json no longer works now, you need to update them:

"scripts": {
  "start": "npm run dev",
  "test": "jest",
  "build": "tsc",
  "lint": "eslint . & echo 'lint complete'",
  "dev": "ts-node-dev --respawn --transpileOnly ./src/app.ts",
  "prod": "tsc && node ./built/src/app.js",
  "debug": "tsc && node --inspect ./built/src/app.js"
},

The commands are mostly self explanatory, just remember to customise them according to your setup.

Then you can start your program by yarn dev or npm start later. But right now the js files haven't been changed yet.

The ignore files

Remember to add built folder in your ignore files like .gitignore and .eslintignore so that they do not generate a ton of errors.

Change the code

Now that we've setup all the things. It's time that we actually change the code itself.

Typescript was built with Javascript in mind, this means you do not have to change most of you code. But you certainly going to spend quite some time changing it.

Rename the files into `.ts`

Rename all your .js files into .ts , except the config files.

The imports and exports

Typescript adopts the es6 import and export syntax, this means you need to change the existing commonjs const a = require('b') and module.exports = c to import a from 'b' and exports default c

See the import and export guide on MDN to have a better understanding on how to use them.

Object property assignment

You may have code like

let a = {};
a.property1 = 'abc';
a.property2 = 123;

It's not legal in Typescript, you need to change it into something like:

let a = {
    property1: 'abc',
    property2: 123
}

But if you have to maintain the original structure for some reason like the property might be dynamic, then use:

let a = {} as any;
a.property1 = 'abc';
a.property2 = 123;

Add type annotations

General functions

If you have a function like this:

const f = (arg1, arg2) => {
    return arg1 + arg2;
}

And they are intended only for number, then you can change it into:

const f = (arg1: number, arg2: number): number => {
    return arg1 + arg2;
}

This way it cannot be used on string or any other type

Express

If you use express, then you must have some middleware function like:

(req, res, next) => {
  if (req.user) {
    next();
  } else {
    res.send('fail');
  }
})

Now you need that req and res to be typed

import { Request, Response, NextFunction } from 'express';

and then change

(req: Request, res: Response, next: NextFunction) => {
  if (req.user) {
    next();
  } else {
    res.send('fail');
  }
})

mongoose

Using Typescript, you want your mongoose model to have a corresponding typescript interface with it.

Suppose you have a mongoose model that goes:

import mongoose, { Schema, model } from 'mongoose';

export const exampleSchema = new Schema(
  {
    name: {
      required: true,
      type: String
    },
    quantity: {
      type: Number
    },
    icon: { type: Schema.Types.ObjectId, ref: 'Image' }
  },
  { timestamps: true, collection: 'Example' }
);

export default model('Example', exampleSchema);

You need add the according Typescript interface like:

export interface exampleInterface extends mongoose.Document {
  name: string;
  quantity: number;
  icon: Schema.Types.ObjectId;
}

Also change the export into:

export default model<exampleInterface>('Example', exampleSchema);

Extend built-in Types

Sometimes you need some custom property on the built-in type, so you need to extend them.

For example, In express, you have req.user as the type Express.User, but if your user will surely different from the default one. Here's how I did it:

import { UserInterface } from '../path/to/yourOwnUserDefinition';

declare module 'express-serve-static-core' {
  interface Request {
    user?: UserInterface;
  }
  interface Response {
    user?: UserInterface;
  }
}

This is called Declaration Merging in Typescript. You can read the official explanation if you want to know more about it.

Note you should name the file with extension of .d.ts and put it in a separate folder and add that folder into the typeRoots in tsconfig.json for it to work globally.

Async functions

For async functions, remember to wrap you return type with Promise<> ,

Dynamic property

If your object have a dynamic property, you need something special union type annotation for it to work.

let a : string;
if (someCondition) {
  a = 'name';
} else {
  a = 'type';
}
const b = { name: 'something', type: 'sometype' };
const c = b[a]; // gets error: Element implicitly has an 'any' type because expression of type 'string' can't be used to index type '{ name: string; }'.

The way to fix it:

let a: 'name' | 'type';
if (someCondition) {
  a = 'name';
} else {
  a = 'type';
}
const b = { name: 'something', type: 'sometype' };
const c = b[a];

Or change the last assignment into const c = b[a as 'name' | 'type'] , but apparently the first one is preferred since it checks if any unexpected value being assigned to the variable. Use this if you do not have control over the definition of the variable.

Sum up

Typescript helps a lot if you have experience in strongly typed language like C++/Java/C#, it checks many of the error at compile time. If you plan on writing an application at scale, I definitely recommend choose Typescript over Javascript.

Build a Javascript async function error handler

Nathaniel — Fri, 13 Sep 2019 04:54:38 +0000

Background

If you are a Js developer, you probably have used async and await, and you love them.

No more callback hell or .then chains. Just like writing a synchronous program.

Util you run into UnhandledPromiseRejectionWarning or Uncaught (in promise) Error

You begin to wrap every piece of code using try.. catch.., but that seems a lot of work.

But I've got good news, you can write a custom async function error handler:

Solution

const asyncHandler = fn => async (...args) => {
  try {
    await fn(...args);
  } catch (err) {
    console.log(err);
  }
};

It takes all types of functions with any number of arguments.

use it like this:

const asyncHandler = fn => async (...args) => {
  try {
    await fn(...args);
  } catch (err) {
    console.log(err);
  }
};

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

const yourFunction = async () => {
  await sleep(2000);
  throw 'something wrong';
  return 'success';
}

(async ()=> {
  await yourFunction(); // will cause Uncaught error
  await asyncHandler(yourFunction)(); // will handle the error properly
})();

Return values

Of course when you need the result of the async function, you can:

const asyncHandler = fn => async (...args) => {
  try {
    return await fn(...args);
  } catch (err) {
    console.log(err);
  }
};

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

const yourFunction = async () => {
  await sleep(2000);
  // throw 'something wrong';
  return 'success';
}

(async ()=> {
  const result = await asyncHandler(yourFunction)(); //will handle the error properly
  console.log(result); // => 'success'
})();

Custom error handling

if you need some custom error handling, for example, to use it in express.js

you can do simply by adding some of your custom logic

const asyncHandler = fn => async (...args) => {
  try {
    await fn(...args);
  } catch (err) {
    logger.error(err);
    const resFn = args.find(arg => arg.name === 'res');
    if (resFn) {
      let errors = {
        message: 'Internal Sever Error',
        error: err
      };

      if (err instanceof mongoose.Error.ValidationError) {
        errors = {
          message: 'Mongoose Model Validation Error',
          error: err
        };
      }
      if (err instanceof mongoose.mongo.MongoError) {
        errors = {
          message: 'MongDB Error',
          error: err
        };
      }

      resFn.status(500).json(errors);
    }
  }
};

later you use it in express routers like this:

router.get('/item',
  asyncHandler(async (req, res) => {
    // router logic here
  })
)

DEV Community: Nathaniel

From Jobless to FAANG - My three year journey

How I make memes inside code

Begin

Implementation

Debugging

Publish

Promote

Sorting 1 billion numbers

Beginning

Divide

SORT

Conclusion

Building a high concurrency student course selection system

Intro

Technology

Database design

Business Scenarios

Preparation for course selection

Course Selection

Course Selection Finished

Key implementation

Preparation for course selection

Setup redis and background jobs

Cache warm-up

Course Selection

Select

Cancel

Message queue

Ending

The making of a modern javascript library

What

Why

How

Takeaway

How I build an app in 2 hours

Background

Analyze the core

The power of boilerplate

Knowledge transfer

Key takeaway

Typescript function return type based on parameters

But wait, there's more

Migrate to typescript - the advance guide

Typing Begins

Add the magic d.ts

Connecting the types

Jsdoc typedef import

Triple slash directive

Automatically generate d.ts

Babel configuration(optional)

The type check

Incremental typecheck

// @ts-nocheck

// @ts-ignore

// @ts-expect-error

External libraries

Well maintained library

Pure Js Library

The type check rises

Change all files into .ts files

Change jsconfig to tsconfig

Frontend projects

Backend projects

Change CI/CD pipeline and build process

The End

Use recoil in react custom hooks

Speed up your build by putting node_modules in ram

Linux

macOS

Windows

Use faster build tool?

Migrate to Typescript on Node.js

Add tsconfig.json

Restructure the files

Install the types

Setup the tools

ESlint

nodemon

Jest

Add the magic `d.ts`

Automatically generate `d.ts`

Change all files into `.ts` files

Rename the files into `.ts`