DEV Community: Dzung Nguyen

Develop Spotify Player web extension

Dzung Nguyen — Sun, 16 Aug 2020 06:55:09 +0000

Motivation

When we are working or just browsing it is inconvenient to have to switch to another window to play / pause or jump to previous tracks or forward to upcoming tracks. Also, we lose focus or get distracted once we navigate away from our current window and open Spotify. With this simple extension we can eliminate the need to navigate away from your current window to control Spotify. This extension has another nifty feature. With a quick right click on the widget, you can search a song by name. So for example if you hear a tune on Youtube you can quickly search it on Spotify without navigating away from your current window. Then you can maybe add it to a playlist later.

UX / UI

How many features does it support?

The goal of Spotify extension is to provide basic features that allow the user to quickly and easily control the Spotify app. Below is the list of basic features:

Play / Pause
Next / Previous
Like
Right click & search song in Spotify

Right click to search a song

Sometimes, we unintentionally find a good melody on youtube or some other websites and we want to find that song in Spotify to add to a playlist. We usually:

highlight & copy song title
go to Spotify desktop app or web player
search by song name
add it to a playlist

As you can see it takes a few steps and we have to switch context (open web player with a new tab or switch to desktop app) to achieve the goal. So finding optimal way to do this is needed to reduce switching context (which can lead to lost focus)

Is it possible to find a song in Spotify while I am on the current page (where I found the song)?
If it is possible, how can I make it natural for users?

Here are the steps that I found optimal & natural:

highlight the song title
right click on the song, select item search in Spotify
navigate to Spotify web player with that song name

From there I can add the song to my playlist. Everything is done in the browser.

What kind of information needs to be displayed in Spotify extension?

Web extensions are small software programs that customize the browsing experience. Therefore only basic information is displayed .
Song title & artist name are the basis of any song.

As a result, I decided to display the song title, artist name, and their links.

But, how does it look ?

The UI of Spotify extension is inspired by built-in “control your music, videos & more” feature of Chrome browser.

Note: Chrome browser has a built-in extension that allows controlling any media playing in the browser.

Here is UI of built-in

Spotify extension UI has 3 cases

User is not logged in

Spotify desktop app or web player is not opened

Happy case

Logo

The logo is very important for any product or business, it helps other people to recognize you. At first, I tried to design one by myself but could not come up with a good design so I asked my friend who is a designer to help. Here is what he came up with:

Spotify Web API

To control the state of the Spotify application via web extension, I need a way to tell the Spotify application that I want to update state, e.g from playing to pause the song. Fortunately, Spotify provides the Web API endpoints to control the state of the application.

https://developer.spotify.com/documentation/web-api/reference/

Tech stack

At first, I used JS, HTML & CSS to set up a simple development. Because I thought that creating the Spotify Player extension would be as simple as building a UI and integrating with the API. It turned out to be more than that. During development I encountered edge cases that forced me to refactor the code. Refactoring code has never been an easy task, especially for JS which does not have strong type support and I think it’s not good in the long run. So I decided to rewrite the whole code base using a new stack and put a lot of effort into development set up. Here is what I used:

Webpack
Typescript (compile to JS)
HTML
Jest
Travis for CI

Why not use a library or framework?

Because libraries and frameworks are for larger scale projects and the JS file size is usually pretty big especially for plugins. So I decided to stick with Typescript with DOM rendering and manage application state on my own. You can check the details here:

https://github.com/davidnguyen179/spotify-extension

Edge cases

After integrating the API with the UI, the extension works pretty well. However, there are some edge cases which took me some time to fix.

What happens if you leave the Spotify related app (desktop app or web player) idle for sometime (without playing any song) and you use Spotify Player extension to play a song?
What happens if you suddenly turn off the Spotify app and you open the Spotify Player extension, how does the UI look?
What if your token expired or you’re not logged in? How does the UI look?
What if you don’t open any Spotify related app on your PC? Then, how does the UI of Spotify Player extension look?

To bring the best UX possible for users, I had to think about all possible cases and fix them all. Whenever I fixed things, I had to update my test suite in code to make sure future fixes or features won’t break the extension.

Cross Browser support

My initial goal was only to create a Chrome extension. Then I realized web extensions are a standard for most browsers. They are all developed using JS, CSS, HTML and manifest file to manage configuration and permission of extension. Knowing this, I decided to support 3 browsers:

Chrome
Firefox
Microsoft Edge (Chromium)

Solution 1 — poly repo

Create 2 more repositories and develop for Firefox & Edge. This solution will work, but there are some downsides:

What if you decided to add more functionality to the extension? Then you have to do it 3 times in 3 different repos.
What if you have bugs? Then you have to fix it 3 times.
How do you release it? You have to assure that 3 repos have the same version number and repeat the same process of publishing 3 times.

Well, switching contexts (switching repo to another repo) multiple times and doing many things can make you feel confused and tired. And with this approach, we violate the DRY design principle. Because the logic and functionality is the same.

Solution 2 — mono repo

With this approach, I keep one source code only and analyze what are differences between Chrome, Firefox, and Microsoft Edge. Here are the differences:

manifest.json
background.ts

How can we make those files dynamic and correspond to each browser?

This can be done at build-time. That means I will have 3 build scripts for each browser.

With this approach, I can only solve the first 2 problems that are listed above. But the core logic is shared between each browser.

Testing

My first purpose is to build this extension to solve my own problem. But things are getting serious, I want to publish it to the web stores, that means Spotify Player Extension will have its own users. So it’s better to make sure the extension has no bugs and works smoothly.

After completing the development, I asked friends who are willing to try it out, but I have to write down detailed documentation on how to install this extension with developer mode in the browser. I tried to gather feedback, bug reports, and filter which ones should be fixed. After fixing those issues, I also updated my test suites to make sure those issues will not happen again.

My lesson is if you ask people to test it for you, you need to make sure that everything is super simple for them to test and you only have 2 – 3 chances to ask them to do it again, because they have jobs to do. Otherwise, it becomes annoying for them.

Publishing

Here are the resources for publishing the extension to web stores

While publishing the web extension, I faced some issues which prevented me from publishing. Specifically, they will send an email to inform you if you have violated some rules.

Chrome

The email is sent by Google support team is done by bot and it is hard to understand because it contains many rules. Here is a resource I found really useful

If you have a problem, you can post it in the google group and provide them your extension ID. https://groups.google.com/a/chromium.org/forum/#!forum/chromium-extensions
Jack and Amy Dev youtube channel, which explains some common mistakes when we develop Chrome extension.

Microsoft Edge

They will require file “PRIVACY POLICY”. Make sure you have it.

Firefox

They might require you to upload your source code if your source code is minified.

Marketing

The “Spotify Player Extension” has been published in 3 web stores. To let everyone know it exists, I used social media:

Reddit
Facebook
Twitter

After I posted it I thought my work was done. However, I received a bunch of feedback from “Reddit”. They suggested a lot of improvements. Here we go again, I have to filter which ones I should implement, which ones are not going to add to a better UX, and just keep fixing and improving until it is stable.

Conclusion

Here is how it looks like

And here is the number of users of Chrome & Firefox browsers:

The Spotify Player extension is available on 3 web stores:

Chrome

https://chrome.google.com/webstore/detail/spotify-player/bhdjjppbnlpjpeicimhemencfgjeldoa

Microsoft Edge

https://microsoftedge.microsoft.com/addons/detail/spotify-player/odplfjpibjdajlmaocmfmlcdidldlmnk

Firefox

https://addons.mozilla.org/en-US/firefox/addon/spotify-player/

Source code

https://github.com/davidnguyen179/spotify-extension

Website

https://spotify-extension.netlify.app/

Boilerplate

https://github.com/davidnguyen179/web-extension-boilerplate

I really appreciate the community that helps me to improve this extension and friends who help me to test it out.

Rewrite commit history

Dzung Nguyen — Sun, 08 Sep 2019 07:54:39 +0000

Overview

As an engineer, we like to keep things organized. Especially, when you are working on the project and the team is growing larger. That means there are a lot of engineers working on the same source code and a bunch of commits are made every day. Keeping the git history clean and linear makes it easier for all of the engineers to understand what is going on in the source code. Doing so requires using a few advanced git features. In this article we will learn how to perform 4 git tasks that I believe are essential to keeping your commit history minimal and concise before merging it with the main branch.

Squash & Reword
Merge base & Reset
Amend
Push force

Before jumping to the detail. Let’s see why we need clean commit history and which scenario should we apply to our work.

Why clean commit history?

Having the clean commit history brings us 4 main benefits:

Documentation. A linear history is typically easier to follow. This is similar to how you want your code to be well structured and documented: whenever someone needs to deal with it later (code or history) it is very valuable to be able to quickly understand what is going on.
Improving code review efficiency. If a topic branch is divided into linear, logical steps, it is much easier to review the changes between the PR and a target branch.
When you need to modify the history at a later time. For instance when reverting or cherry-picking a feature in whole or in part.
Scalability. Unless you strive to keep your history linear when your team grows larger (e.g. hundreds of contributors), your history can become very bloated with cross branch merges, and it can be hard for all the contributors to keep track of what is going on.

However, beside the benefits there are some drawbacks:

The clean commit history is only effective only if all members agree to apply it to their work.
To do this requires some advanced git features, the learning curve costs you some time at first.
And if you don’t know it well, it can cause you some troubles when you apply it to your project.

Scenario

I was assigned the task to develop the React components called “List” and “List Item”. So here is what I did:

Create a new branch called “features/add-list-list-item”

git checkout -b “features/add-list-list-item”

I made a commit for adding the UI of the ListItem component.

git commit -m “add the ui of ListItem”

Then I found that my code was not very clean. So I refactored it.

git commit -m “refactor”

And then, I found some issues and fixed them. Suddenly, I have to go home for some private business, then I decided to commit my work temporarily.

git commit -m “temp”

OK! I completed the “ListItem” component.

git commit -m “completed the ListItem”

I completed the “List” component and with the ListItem I committed above. Now I can make a final commit to mark that I finished my task.

git commit -m “finished task”

Looking back at the commit history, we see the history of the feature we are working on is lacks meaningful descriptions of changes and in general is messy.

Clean commit history

Squash & Reword

Squash: combine commits into 1 commit.
Reword: Edit the commit message.

In order to do that, here is the steps:

1 - Open the commit history

git log --oneline

2 - Identify how many commits we need to squash and reword

In this case, we are going to squash 5 commits in the “red box” and reword the 5th commit message.

3 - Rebase to the commit just before the list of commits that you want to squash and reword

3.1 - Rebase to commit

In this case, it is the sixth commit. Copy the hash string “7c63b5f” and rebase to 6th commit by:

git rebase -i 7c63b5f

3.2 - Select options to modify commits

After running the command at previous step, it takes us to Vim screen.

So here is what we are doing:

We replace the “pick” with “s” to commits in the “red box” to squash.
We replace the “pick” with “r” to commits in the “yellow box” to reword commit message.

To modify Vim screen, you need to do 2 things:

Enter the “Insert mode” by press “i” for editing.
Enter the “Command mode” by press “esc” and press “:wq” to save the changes.

r 1e857cc add the ui of ListItem
s cb0cc38 refactor
s 4a1d911 temp
s 00cde97 completed the ListItem
s 975b990 finished task

3.3 - Reword commit message

After modifying the commits, it takes us to the reword screen.

Because we develop 2 components “List” & “List Item”. So the message should contains 2 components

Change:

add the ui of ListItem

To:

Add List & ListItem components

Then “:wq” (Write & Quit) Vim.

3.4 - Squash commits

After we done editing the commit message. It takes us to the second screen to squash commits.

Delete all lines in the “red box”.
Keep the line in the “green box”. Because this is the final commit message reflects to our work.

To delete lines in the “red box”.

Move your cursor to that line and press “dd”.

Then “:wq” (Write & Quit) Vim. Done!

3.5 - Check if our work was successful or not

git log --oneline

As you can see, we already combined 5 commits into 1 and reworded the commit message.

Merge base & Reset

After squashing commits, I create a PR (Pull Request or Merge Request) and add reviewers. And they give me feedback.

What do you think about splitting the commit into 2 commits? Because 2 components should correspond to 2 commits. One for ListItem component, the other for List component.

Our mission now is:

Splitting the squashed commit into 2 different commits.
Each commit needs to have its own commit message.

Therefore, this section uses the second technique to re-commit and it requires a few steps:

Using “git merge-base ” to get the common ancestors between the main branch and feature branch. The output of this is a hash string represent the commit.
Using “git reset ” to go back when we haven’t committed anything related to List & ListItem components.
Using “git add ” to add files which belong to “List” component.
Commit the “List” component with the message “Add List component”.
Using “git add ” to add files which belong to “ListItem” component.
Commit the “ListItem” component with the message “Add ListItem component”.

Using merge-base

The first thing we need to do is find the most common parts between the main branch and feature branch by running the command below:

git merge-base master features/add-list-list-item

And the output is the hash string points to the commit has the most common parts between 2 branches:

7c63b5ffc4ba24581b624e298dd15a02d52bf2a7

Make sure you copy this hash string before moving to the next step.

Reset commit

In this step, we use the “git reset ” with the hash string from the previous step to go back to the moment before we made any commits.

git reset 7c63b5ffc4ba24581b624e298dd15a02d52bf2a7

As you can see in the “orange box”, our files were back to the original state. Now we can use “git add ” to shape our commit.

Finish move

So we will create 2 commits which correspond to 2 components “List” & “ListItem”.

List: “Add List component”

git add src/components/list/index.js
git add src/components/list/style.css
git commit -m "Add List component"

List Item: “Add List Item component”

git add src/components/list-item/index.js
git add src/components/list-item/style.css
git commit -m "Add List Item component"

Done! Now to check whether or not we successfully re-shaped our commit. Just open the git commit history.

Amend

After we shaped our commits. Suddenly, we forgot we haven’t update document. So we went back and updated the document and this one is just a small change. You will ask yourself the question?

Is there anyway that I commit and don’t have to shape the commits again?

Yes, we have. It’s git commit --amend.

What is git commit --amend?

The git commit --amend command is a convenient way to modify the most recent commit.

How we use that?

We updated the document of the source code and commit with the --amend flag.

git commit --amend

It takes you to Vim screen like this

Then “:wq” (Write & Quit) Vim. Done!

Now your changes were included in the last commit “Add List Item component”.

Push force

After we finished everything and now we have the nice commits structure. The last thing we need to do is “push” code to remote branch.

If your branch is already in “remote branch” and you just rewrite commit history. Therefore, you need to override it in the “remote branch”.

git push -f

Otherwise

git push --set-upstream origin features/add-list-list-item

Conclusion

Having the clean git history in your project is important. Especially, when you are working in the large team (≥ 20 contributors) in which everyday there is a bunch of commits are made. It helps you easy to catch up what’s going on, easy to revert and makes your life easier. But it comes with a price, if you apply it without any experience in the real project, it can cause you some trouble.

Make sure you have tried it in your personal projects multiple times before applying it to the real project.

References

NextJS deployment pipeline on S3 with Gitlab-CI

Dzung Nguyen — Sat, 27 Apr 2019 01:50:23 +0000

Overview

NextJS was released in 2016. Since then it has become one of the largest frameworks that support Server-side Rendering (SSR) with ReactJS & NodeJS. Being one of the first companies in Vietnam which have applied NextJS to develop products, we have encountered interesting challenges and learned throughout development to deployment processes. In today’s topic, I am going to share about how we dealt with the deployment step which helped us achieve 2 big milestones.

Web team and SRE (Site Reliability Engineering) team work independently.
Only one step to production.

Problem

Mapping URL to static resources.

After setting up a brand new project, developers jump into setting up the deployment environment based on the SRE rules to have a smooth workflow. Here are what we usually need:

Identify the available IP & port.
Register the service with the available IP & port.
Ask for the domain to map to the service.

For a web application, some extra items need to be handled. In order to go live, you need to specify some static resources and assets such as CSS files, JS files and images… So we need to list out every resource with a specific URL and then work with SRE team to map with the IP & port.

For example:

A manual approach to do this is simple and easy to setup. However, there would be challenges as follow:

If the page has a lot of resources. We need to manually list out each resource with a specific domain. It takes a lot of time.
And because we have a lot of services and small web apps. Consequently, the mapping config file from the SRE team becomes huge.
Sometimes the mapping config file has conflicts between services, for example, the other team has registered the URL to their own resource.

Depends on SRE team

There were a lot of services of Chợ Tốt which depend on the SRE team. Every time we start a new web app, we need to come to the SRE desk and ask for mapping URLs to static resources and when the configuration becomes more complicated such as this URL was taken by another web app and we did some “hack” to make it work. Consequently, it leads to creating some bugs of other web apps. Then we revert back to the origin to find another solution.

For those reasons, the Web team & SRE team come to the solution which helps the automation CI/CD pipeline of the whole system.

Solution

When the code is merged to the specific branch (master or staging or release). The CI/CD flow will be triggered to execute. The CI/CD has 3 stages.

Dockerize
Converting an application to run within a Docker container.

Uploading static resources to S3
After dockerizing the web app, we do some post processes, then start uploading the whole build directory to S3.

S3 here is actually Chợ Tốt server and we apply the interface of AWS S3. So uploading static resources to S3 means uploading to our own server.

Deployment
Activating the web application to run on the production environment.

Hands-on

Below is the structure of the project. You could find it at cna

my-app
├── deployment/
 └── post-build/
 └──── index.js
 └── s3/
 └──── index.js
├── app/
 └── …
 └── next.config.js
 └── …
├── package.json

In this section, I’ll cover 4 points.

Bucket name
Credential
Configs
Deployment

Bucket name

The first thing we do is to define a good convention name for the S3’s bucket. To do that, we manually create by using commands of AWS S3 SDK. The bucket name is the combination of service name and environment.

<SERVICE_NAME>_<ENVIRONMENT>

For example 1:
The service name “chotot-vehicle”. And the bucket name is

staging: CHOTOT_VEHICLE_STAGING
production: CHOTOT_VEHICLE_PRODUCTION

For example 2:
The service name “chotot-property”. And the bucket name is

staging: CHOTOT_PROPERTY_STAGING
production: CHOTOT_PROPERTY_PRODUCTION

Credential

We make use of Gitlab via “Secret Variables” feature which provides the setting allows adding “Credential Information” as the environment variables.(I believe other CI/CD has the same feature such as TravisCI, CircleCI, …).

By doing this way, we follow the Separation of Concerns (SoC) design principle. The development does not have to care about the deployment or manage the credential.

Configs

.gitlab-ci.yml

stages:
- dockerize
- s3
- deploy
dockerize:
  stage: dockerize
  script:
  - <DOCKERIZE_DO_SOMETHING>
s3:
  stage: s3
  script:
  - yarn install
  - yarn build
  - NODE_ENV=staging CDN_HOST_BUCKET=$CDN_HOST_BUCKET CDN_ACCESS_KEY_ID=$CDN_ACCESS_KEY_ID CDN_SECRET_ACCESS_KEY=$CDN_SECRET_ACCESS_KEY yarn s3:upload
  - NODE_ENV=production CDN_HOST_BUCKET=$CDN_HOST_BUCKET CDN_ACCESS_KEY_ID=$CDN_ACCESS_KEY_ID CDN_SECRET_ACCESS_KEY=$CDN_SECRET_ACCESS_KEY yarn s3:upload
deploy:
  stage: deploy
  script:
  - <DEPLOYMENT_SCRIPTS>

next.config.js

const version = require('package.json').version;

const config = {
  development: {
    assetPrefix: ''
  },
  staging: {
    assetPrefix: `https://static.com/CHOTOT_VEHICLE_STAGING/${version}`
  },
  production: {
    assetPrefix: `https://static.com/CHOTOT_VEHICLE_PRODUCTION/${version}`
  }
};


module.exports = {
  assetPrefix: config.assetPrefix,
  generateBuildId: async () => {
    return 'build';
  },
  webpack(config, options) {
    return config;
  }
};

Deployment

Problem

After building NextJS web app, it has the directory called “.next” and the structure of files is different from the URLs access to the static resources.

As you can see there are 2 different paths.

.next/bundles/pages/index.js

versus

/_next/build/page/index.js

So if we upload the whole built directory to S3 and run our web application. It is absolutely not working.

** Solution

Because of the deployment is separated from the development stage. So that we add one more stage before uploading built directory to S3 called “post-build”.

The “post-build” stage takes care of re-arranging the structure of files to match the structure of URLs which point to the files.

For example:

The post-build script

To create the post build, we created a small script which allow to re-structure the built directory. Each major changes from NextJS which created a new structure of built files. For this reason, each of “post-build” script need to adapt with NextJS major version.

post-build.js - nextjs 6

var fs = require('fs');
var mv = require('mv');
var dirs = ['app/.next/_next', 'app/.next/_next/build', 'app/.next/_next/build/page', 'app/.next/_next/static'];

const PAGE_PATH = {
  old: 'app/.next/bundles/pages',
  new: 'app/.next/_next/build/page'
};

const STATIC_PATH = {
  old: 'app/.next/static',
  new: 'app/.next/_next/static'
};

// create dir
dirs.forEach(item => {
  fs.mkdirSync(item);
});

// move files
mv(PAGE_PATH.old, PAGE_PATH.new, { mkdirp: true }, function(err) {
  if (err) console.log(err);
});

mv(STATIC_PATH.old, STATIC_PATH.new, { mkdirp: true }, function(err) {
  if (err) console.log(err);
});

post-build.js - nextjs 7

var fs = require('fs');
var mv = require('mv');
var dirs = [
  'app/.next/_next',
  'app/.next/_next/static',
];

const STATIC_PATH = {
  old: 'app/.next/static',
  new: 'app/.next/_next/static'
};

// create dir
dirs.forEach(item => {
  fs.mkdirSync(item);
});

mv(STATIC_PATH.old, STATIC_PATH.new, { mkdirp: true }, function(err) {
  if (err) console.log(err);
});

post-build.js - nextjs 8

var fs = require('fs');
var mv = require('mv');

var dirs = ['.next/_next', '.next/_next/static'];

const STATIC_PATH = {
  old: '.next/static',
  new: '.next/_next/static'
};

// create dir
dirs.forEach(item => {
  fs.mkdirSync(item);
});

mv(STATIC_PATH.old, STATIC_PATH.new, { mkdirp: true }, function(err) {
  if (err) console.log(err);
});

Uploading to S3

After re-structuring the built files. Now all we need is some code to upload the whole build directory to S3. To to this, we created a small package called “s3-uploading” which helps us to make it.

var path = require('path');
var version = require('./package.json').version;
var Uploader = require('s3-uploading').default;
var env = (process.env.NODE_ENV || 'staging').toUpperCase();

// CHOTOT_VEHICLE_PRODUCTION/1.0.1
var BUCKET_PATH = `<YOUR_BUCKET_NAME>_${env}/${version}`;

const options = {
  s3: {
    accessKeyId: process.env.CDN_ACCESS_KEY_ID,
    secretAccessKey: process.env.CDN_SECRET_ACCESS_KEY,
    endpoint: process.env.CDN_HOST_BUCKET,
    region: '',
    ACL: 'public-read',
    sslEnabled: false
  },
  upload: {
    directory: path.resolve(__dirname, '../../app/.next'), // path to built directory
    bucket: BUCKET_PATH
  }
};
const job = new Uploader(options);
job.upload();

Register scripts

One last thing we need to do is registering the scripts to execute the “uploading static resources to s3” step to package.json

{
  "scripts": {
    "s3:upload": "node ./deployment/s3",
    "start": "node ./server",
    "build": "next build app",
    "postbuild": "node ./deployment/post-build"
  },
}

Result

Here are the results of staging and production environment.

As you can see, we manage the static resources based on the version from “package.json”. We follow Semantic versioning, more detail here. In each release, we update the version and then upload them to S3.

Conclusion

With the optimization above, we cut down from n (n > 1) steps of the deployment into 1 step. It brings some of the benefits.

Speeding up the development time.
Less depending on the other team.
Caching of static resource versioning is well taken cared of (Because we store all static resources based on the version).
Fully control by the web team.

If you’re interested in it, make sure you check out our repos.

How we built the universal “Header & Footer” using React

Dzung Nguyen — Fri, 01 Feb 2019 14:14:33 +0000

Overview

In 2016, Chợ Tốt rebuilt frontend web application using React & Redux for business expansion. Likewise, we also decided to step by step migrate current services to Microservices Architecture to scale up. To boost the speed of our web app, we split our business into small chunks and each one is an application. Although doing this way reduces the complexity of the business, we faced some challenges:

Consistency: many web apps mean the more work to keep consistency across product features.
Reusability: many web apps mean we need a way to organize and share common components.

One particular feature on our website that has all these challenges is the “Header& Footer”. Typically, the “Header & Footer” is used by all products at Chợ Tốt and it contains links to important pages that potential users will often visit before making a purchase or inquiry.

This topic talks about 3 main points of how we built the “Header & Footer”:

Architecture
CSS
Build process

Before deep dive into the detail, let’s take a look at the principles we keep in mind when we develop this component.

Principles

Usability

We keep in mind the “Simplicity & Efficiency” of the components’ interface. They also help developers integrate easily.

“Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple“ — Steve Jobs.

Let’s look at the example of 2 components code as shown:

It’s easy to see that if we want to use the component on the left side, we have to read documents carefully to understand the meaning of each prop and what if there is no document?. Developers often dive into the code to see how it works and makes assumptions.

On the other hand, the right side there are only 3 props they need to care about and the name of properties are declarative. With that even though the developers don’t read the document, they still understand it.

A library is like a product of developers. If it has a good UX (How easy to use it), other developers will be happy to use it.

Extensibility & Testability
With the business expansion, there are a lot of features integrated into the “app-wrapper”. We follow the “Single Responsibility Principle” to design the code base to make it easy to extend and test.

Less depending on libraries.
Using many libraries to develop is unnecessary. The more libraries we use, the bigger size of JS file is. It inadvertently slows down the web page loading. Because Chợ Tốt products reply on React & Redux, we decided to keep only those libraries to develop “app-wrapper”.

I. Architecture

The app-wrapper divides into 2 zones

Component
Extension

1.1. What is the Component zone?

The component zone contains web components need to render the Header & Footer such as:

1.1a. Problem

The Microservices Architecture is good for reducing the complexity and unnecessary code of the particular feature under construction. However, with many new features are continually released every 1–2 months and each one has its own “entry point” when we release a new feature, we need to add “entry point” to the “app-wrapper” and publish it with a new version. Then we go to each project to upgrade the “app-wrapper” in order to have an entry point link to this feature.

Additionally, we have a lot of web applications and we have to make sure all of them have the latest version of app-wrapper. What happens if we miss one? It could affect user experience.

1.1b. Solution

Because of these reasons, we decided to develop an API to manage entry points, the “app-wrapper” makes a request to fetch a list of menu items from the server and render.

By doing this way, when we have a new entry point, we only update the entry point at API endpoint and do it once.

1.2. What is the Extension zone?

The “app-wrapper” has some social features such as “Receiving Chat’s Notification”, “Displaying Announcements”. Those features require a lot of logic and big libraries such as Socket I/O. If we put all the code inside the “app-wrapper”, here what we are going to deal with:

The code base is going to be huge.
It hurts “Single Responsibility Principle”. Since the “app-wrapper” take care of displaying Header & Footer. It doesn’t need to take care of other business.
It unnecessarily becomes more complex.

1.2a. Solution

We develop an area called “Extension” zone which allows loading asynchronously the third party services.

For example:
We got the CDN links to services “Receiving Chat Notification” & “Receiving Announcements” (each service is a specific project and the output is a CDN link). Then we only need to register the link to “Extension” zone and let the Extension do the rest.

By doing this way, we achieve some benefits:

Delegating all the logic of third-party services to CDN links helps to separate the logic.
Reducing the size of the main JS file.
Simplifying code base which helps other engineers easy to improve.

II. CSS

The “app-wrapper” contains the styles itself. Choosing a way to manage CSS is one of the hardest problems. There are 2 approaches:

CSS-in-JS
JS exports CSS to a JS module. This means we could import CSS directly to JS code.

CSS file

This is the original method. All CSS is bundled to CSS file (style.css).

2.1. Approach

Since all products at Chợ Tốt use JS to develop and the “app-wrapper” is a library which needs to provide less configuration for developers to integrate to main apps. For this reason, we decided to choose “CSS-in-JS” approach to manage styles of “app-wrapper”.

There are some libraries help to apply “CSS-in-JS” approach such as “styled-components”, “JSS”,… However, we have various teams and each team has its own style. Some use “CSS-in-JS”, some use “CSS file” to develop web apps. So the question is “Is there any method that could fit all?”. And we came with 1 solution that instead of using CSS-in-JS frameworks, we choose “Template strings” feature of ES6 to develop CSS.

After using this approach. Everything works pretty well. However, we encountered 2 big problems in the production environment.

2.2. Problem

CSS is not minified.
CSS does not contain prefixes for old browsers.

2.3. Solution

After running the build command to compile ES6 to ES5, we run another script to add prefixes & minify the CSS.

We chose Gulp to customize the build process by adding the post-build stage with 2 tasks:

Minifying
Autoprefix

For example:

It means after we ran the build command successfully

npm run build

The post-build command is automatically executed. Below is the result applied this method.

III. Build process

The build process is the way we convert JS code from ES6 to ES5 by using Babel CLI.

The build process has 2 stages.

Stage 1 (build): It uses babel CLI to compile the ES6 code to ES5 code.
Stage 2 (postbuild): It runs gulp tasks to minify & adds prefixes to CSS string from the build directory of stage 1.

After we finish the build process, we version the package and publish to private npm registry. All the projects only need to install a newer version of the package and enjoy.

Usage

We have just described the detail of “How we built the Header & Footer”. Now let’s have a quick look at the usage of the app-wrapper component.

with next.js

Mobile demo

Desktop demo

Conclusion

Currently, the app-wrapper component is used by all the Chợ Tốt’s products.

and more…

We solved the 2 problems mentioned at the beginning of the post: “Consistency & Reusability”.

Besides that, we could bring the “app-wrapper” to the next level to become a common library.

Apply the Context API which allows easy to change a theme such as color, icon, logo, … based on the business.
Integrate CI/CD for testing, deployment, …
Consider integrating strong type system such as Flow Type, TypeScript

Introducing React Paginating 🎉🎉🎉

Dzung Nguyen — Wed, 07 Feb 2018 15:23:17 +0000

React Paginating component.

There are some popular components which help us to solve pagination problem such as react-paginate, react-pager, … Now there is another one. It’s called react-paginating with a different approach.

How “react-paginating” is different

“react-paginating” uses Render Props pattern which allows a component to publish any variables, states or functions to the outside as input params of a function which is going to be used for handling logic and rendering the UI.

Here are some differences:

Input props.
Controlled props.
Child callback functions.
Flexible UI.

Input props

We minimize the number of props which you pass to “react-paginating” for some several reasons:

Make code more readable.
Easy to remember the props.
Not taking too much time to read a document.
Easy to use

Here is a list of input props:

total

The total records of your data. Your API should include it

I.e:

{
    "total": 50,
    "data": [
        { "id": 1, "name": "foo" },
        { "id": 2, "name": "bar" }
    ]
}

limit

How many pages you want to have based on a limit. The formula to calculate totalPages:

const totalPages = Math.ceil(total / limit);

pageCount

How many pages you want to display.

I.e:

pageCount = 5

pageCount = 9

currentPage

The page currently you are visiting. You can pass it from your “query string” or “state”. You can visit the example here.

Controlled props

After receiving input props. The component calculates and publishes props which allow controlling UI. Here is a list of public props:

pages
currentPage
previousPage
nextPage
totalPages
hasNextPage
hasPreviousPage

Here is how it looks like in code

<Pagination total={total} limit={limit} pageCount={pageCount} currentPage={currentPage}>
  {({
    pages,
    currentPage,
    hasNextPage,
    hasPreviousPage,
    previousPage,
    nextPage,
    totalPages,
    getPageItemProps
  }) => (
    <div>
      <a href="/?page=1">first</a>
      {hasPreviousPage && <a href={`/?page=${previousPage}`}>{'<'}</a>}
      {pages.map(page => {
        return (
          <a
            key={page}
            style={currentPage === page ? { backgroundColor: '#fdce09' } : null}
            href={`/?page=${page}`}
          >
            {page}
          </a>
        );
      })}
      {hasNextPage && <a href={`/?page=${nextPage}`}>{'>'}</a>}
      <a href={`/?page=${totalPages}`}>last</a>
    </div>
  )}
</Pagination>

Child callback functions

If you use paging with state and has no update on your query string. You can use this callback function pass to your control.

{pages.map(page => {
  return (
    <button
      key={page}
      style={currentPage === page ? { backgroundColor: '#fdce09' } : null}
      {...getPageItemProps({
        pageValue: page,
        onPageChange: this.handlePageChange
      })}
    >
      {page}
    </button>
  );
})}

Flexible UI

By using Function as Child Components pattern. We can completely control UI component. Take a look:

{pages.map(page => {
  return (
    <a
      key={page}
      style={currentPage === page ? { backgroundColor: '#fdce09' } : null}
      href={`/?page=${page}`}
    >
      {page}
    </a>
  );
})}

import CustomAnchor from './component/CustomAnchor';

{pages.map(page => {
  return (
    <CustomAnchor
      key={page}
      style={currentPage === page ? { backgroundColor: '#fdce09' } : null}
      href={`/?page=${page}`}
    >
      {page}
    </CustomAnchor>
  );
})}

In the example above shows that we can create component and then replace . After that, you might put your somewhere on your “storybook” or components manager.

Demo

You can check a basic demo: https://codesandbox.io/s/z2rr7z23ol

Conclusion

If you see it is useful for you. Please give react-paginating a star 🌟, a watch 👀, and a try 😎.

Or if you see any issues or improvements. PR is welcomed.

The original article is here.

Thanks!

next.js at Chotot

Dzung Nguyen — Tue, 30 Jan 2018 11:18:25 +0000

Overview

Every day, Chợ tốt (Chotot) receives over 1 million visits cross platforms, most of the traffic comes from mobile devices. It is really important for us to develop products that can run across devices. Last year, we switched to a new stack to rebuild our products.

The technologies that we chose for our new stack are: React, Redux & NodeJS to build the new web app. React makes it easy to build web components that can be reused in other products. Redux is the main channel communicating between the Back End & Front End. NodeJS builds a web server. To support SEO with this stack, we implement the “Universal Web Application” approach.

First boilerplate

Base on a starter boilerplate put together by erikras (link here) to experiment with our new web stack. However, we encountered problems in production.

problem

In production, if the traffic is high, the web server will stop responding to the client. At first, we optimized our codebase but the result was still the same. We resolved to use client-side rendering instead of server-side rendering. But the challenge is if we turn off server rendering, SEO will be affected.

solution 1

Search engines now support Javascript in their crawlers (check here for more detail). Our first solution was to delegate the rendering page to client-side. Here is our analysis:

SEO was affected negatively
Our ranking dropped

The crawlers were still indexing but it was slower than using server-side rendering. So delegating all missions to client-side did not work for us.

solution 2

The next solution was to split the project into 2 and deploy it to 2 different servers. One serves users with client-side rendering. Another one serves crawlers bot from Google, Facebook, … with server-side rendering.

This solution worked fine. However, we were also looking for another boilerplate to replace it.

Why did we want to change?

When deploying code changes we had to deploy the new version to 2 different servers.
The boilerplate was out of date.
The time to rebuild code at development was too slow. It took more than 10 seconds to rebuild every single change.
We wanted to apply new tools to have good experience for Engineers and good performance for products as well: webpack 2 with many improvements like “Tree shaking”, “dynamic import”...

next.js 2 the rescuer

After looking around some repos, we considered next.js as a potential replacement for several reasons:

Supports server rendering natively.
There are many small examples on integrations between next.js and other libraries or technologies (check them out here).
The documentation is very clean and up-to-date.
next.js takes care of all basic configs. Allowing to extend configs of webpack or babel…

I will talk about the way we applied next.js in our products including some issues and solutions. I will provide the sample at the end of this article.

redux

We chose redux to manage application states. next.js provides a redux wrapper to help integrate with it easier and more consistent (You can visit here to see example)

import React from 'react'
import withRedux from 'next-redux-wrapper'
import { initStore } from '@/store'

class ExamplePage extends React.Component {
  render() {
    <h1>Hello Next.js</h1>
  }
}

export default withRedux(initStore, (state) => ({
  newsListing: state.newsListing
}))(ExamplePage)

Here we can see that Redux was wrapped by “next-redux-wrapper” as a Higher Order Component. To use redux, we have to wrap our component with the function ‘withRedux’.

routing

“next.js” provides a very clean directory structure. There is a special directory “pages”. When you put all your React components into that one, next.js automatically executes:

code splitting
routing
hot code reloading and universal (server-side and client-side) rendering.

You just don’t have to worry about configurations of all that stuff, next.js will manage them.

We created 2 directories to place our React component. The first one is pages directory. The second one is the components directory.

pages: we use it for containers in redux.
components: to store stateful & stateless component in react.

static resources

next.js also provides a static directory to serve static resources for the web application. All you have to do is place all resources like images, fonts, stylesheets, … into the ‘static’ directory. Then just use it as follows:

image

<img src='/static/img/logo.png' />

css

<link rel='stylesheet' href='/static/css/app.css' />

import modules

import CSSTag from '../../../components/CSSTag'

If your module contains many nested directories, this option is going to be complicated. There are many ways to solve this:

Alias config of webpack

You can use the alias feature of webpack to define alias names for your modules (You can check out how to extend webpack config in next.js here). And then just import it as follows:

next.config.js

module.exports = {
  webpack: (config, { dev }) => {
    config.alias: {
       components_example: path.resolve(__dirname, '../components'),
    }
    return config
  },
}

and use it like

import CSSTag from 'components_example/CSSTag'

But every time you add a new directory as a module container, you have to define it in your webpack config.

Add NODE_PATH to commands

You can set NODE_PATH into our commands in package.json like:

package.json

{
   "scripts": {
      "dev": "NODE_PATH=./ next"
    }
}

By setting NODE_PATH, our current position is now at the root directory when we run command “npm run dev”. We can import as follows:

import CSSTag from 'components/CSSTag'

However, this will complicate our commands and if the commands require root path, NODE_PATH needs to be added.

babel plugins

We resolved this by using babel plugin provided by next.js. “babel-plugin-root-import” allows us to configure our root path with specified characters in .babelrc (You can learn how to customize babel config here).

{
  "presets": [
    "next/babel"
  ],
  "plugins": [
    ["babel-plugin-root-import", [
      {
        "rootPathPrefix": "@"
      }
    ]]
  ]
}

The root path is “@”. So you can import a module at components:

import CSSTag from '@/components/CSSTag'

CSS development

To develop CSS we use the pre-processor Sass with SCSS syntax. Sass provides many functionalities (check them out here). It allows us to

write functions (@mixin)
define variables
call a function (@include )
We can write CSS with module scope

To integrate SCSS with next.js, we also referenced this example.

For example:

/styles/index.scss

.indexPage {
  .cardItem {
    margin-bottom: 15px;
  }
}

/pages/index.js

import React from 'react'
import withRedux from 'next-redux-wrapper'
import { initStore } from '@/store'

// style
import style from '@/styles/index.scss'

class IndexPage extends React.Component {
  render() {
    <div>
      <div className="indexPage">
        Hello I am Index page!!!
      </div>
      <style dangerouslySetInnerHTML={{ __html: style }} />
    </div>
  }
}

export default withRedux(initStore, (state) => ({
  newsListing: state.newsListing
}))(IndexPage)

problem

In production mode, when you browse your site and use “View Page Source” you will see styles are not minified in the HTML document. If our styles are large, that means the time to ship your page to the client is going to increase.

solution

We use gulp and postCSS CLI to manage CSS in production mode. The output of this step will produce an app.css file which includes all minified styles used in our web application.

The idea is each component will have a style file (*.scss). We divided the way to develop CSS with 2 environments.

** development**

We created a stateless component called CSSTag to manage CSS at development.

import React from 'react'

const dev = process.env.NODE_ENV !== 'production'

// Note
// this component will only work for ENV = development
function CSSTag (props) {
  const { style } = props
  const element = dev && <style dangerouslySetInnerHTML={{ __html: style }} />
  return element
}

export default CSSTag

and we can use it like this:

import style from '@/styles/Example.scss'

<CSSTag style={style} />

import React from 'react'

import CSSTag from '@/components/CSSTag';

// style
import style from '@/styles/Example.scss'

class Example extends React.Component {
  render () {
    return (
      <div>
        <div className='example'>
          <h1>Hello Example Component</h1>
        </div>
        <CSSTag style={style} />
      </div>
    )
  }
}

export default Example

** production**

We created ‘app.scss’ which includes all styles of the web application, then we used gulp to build final CSS based on ‘app.scss’.

The final CSS has to meet some rules:

contains all styles of web application
auto-prefix
minified

app.scss

// components
@import "./variables";
@import "./ultilities";
@import "./global.scss";
@import "./components/ToolBar";

// pages
@import "./index.scss";

gulpfile.js

const gulp = require('gulp')
const sass = require('gulp-sass')
const minifyCSS = require('gulp-csso')

gulp.task('app', () => {
  return gulp.src('./styles/**/app.scss')
             .pipe(sass().on('error', sass.logError))
             .pipe(minifyCSS())
             .pipe(gulp.dest('./static/css'))
})

gulp.task('default', ['app'])

autoprefix with postCSS

The last step to produce final CSS is auto-prefix. We are using postCSS CLI to auto-prefix final CSS after gulp bundled it. Then includes final CSS to our document via “_document.js” (you can check out here to know how to use extend document)

Here is the diagram that shows the way we manage CSS in production mode:

Result

After completed a project with “next.js”, we used lighthouse to audit our web page. Here is a result.

Here is a result of the old one:

You can see here the Critical Rendering Path of the first result we only need 1.5s for first meaningful paint while the second result is 9.5s. We improved a lot of performance with next.js.

Demo

Here is a full demo

https://github.com/davidnguyen179/nextjs-full-demo

What’s next?

Next.js gives us a lot of benefits so far:

Performance
Lightweight
Easy to use
Well document
Strong support from the community

With the new feature of next.js 3, we are so excited to apply “dynamic import” into our product to speed up the performance.

next.js multiple environment builds

Dzung Nguyen — Mon, 29 Jan 2018 13:39:14 +0000

Overview

Next.js is a minimalistic framework for the universal web application using ReactJS. It was open-sourced 25th October 2016. The company behind Next is ZEIT. It has huge contributors out there to maintain and improve. Next provides easy interfaces for web development and has some advantages such as:

Supports server rendering natively.
There are many small examples of integrations between next.js and other libraries or technologies (check them out here).
The documentation is very clean and up-to-date.
Next takes care of all basic configs. Allowing to extend configs of webpack or babel…
It has a huge community to support when we got issues.

Recently, we have used next.js for some experiment projects in our company (You can check here to know why we chose next.js). During development, we met a problem with multiple env builds.

next build — development vs production

development

Next only builds a page that you are visiting.
Every single changed, Next rebuilds your web app into “.next” directory, but only a page that you are visiting for speed up development.

For example:

Our web app has 6 pages: home (index.js), create new (action.js), listing (listing.js) register & login (auth.js), list of accounts (accounts.js), detail account (detail_account.js).
When visiting “home” page, Next will bundle everything belongs to “home” page (index.js).
Similarly with “create new” page, Next bundles everything belongs to “create new” page (action.js) into “.next” directory.

production

Next bundles everything belongs to our web app into “.next” directory.
When you visit a page, next will not rebuild again because it already built in “.next” directory.

Scenario

In our process, we divided a development environment (env) into 3 parts:

Development: for Engineers to develop features.
Staging: for Quality Assurance (QA) to test features before it comes to real production.
Production: for Users to use features.

Each env uses different configs, API endpoints, resources (CPU, RAM, …). But the staging and production have to be as similar as possible.

Below is our package.json scripts to manage to build and run of our web app

{
    "scripts": {
        "build": "next build",
        "start:development": "node server.js",
        "start:staging": "NODE_ENV=staging node server.js",
        "start:production": "NODE_ENV=production node server.js"
    }
}

We have 4 commands. In each environment, we will run with order:
development

npm start

staging

npm run build
npm run start:staging

production

npm run build
npm run start:production

After running commands above, we have the NODE_ENV, then we can get configs of that NODE_ENV to serve our stuff.

const env = process.env.NODE_ENV || 'development';

const configs = {
  development: {
    api: 'https://api.staging.com',
  },
  staging: {
    api: 'https://api.staging.com',
  },
  production: {
    api: 'https://api.production.com',
  },
}[env];

export default configs;

Problem

While development & production work well. We met a problem at “staging” env. Here is what happened when we build & run our web app in staging env:

npm run build: Next bundles everything (page, resource, …) into “.next” directory.
npm run start:staging: Next detects NODE_ENV is not production. It automatically rebuilds our web app with development env everytime you visit a page.

That means in staging env, our web app is treated as development env. So when you visit a page, Next takes time to rebuild it again. It affects our performance and productivity of the team in staging env.

Solution

We found out that Next only support 2 env: development & production in “run command”. So if you pass NODE_ENV=production to “run command”, Next will not treat your app as development env.

So here are new “run commands” in package.json file:

{
    "scripts": {
        "build": "next build",
        "start:development": "node server.js",
        "start:staging": "NODE_ENV=production ENV=staging node server.js",
        "start:production": "NODE_ENV=production ENV=production node server.js"
    }
}

In staging & production run commands, we pass NODE_ENV=production to notify Next that we want to build our web app in production mode. We also pass another argument called ENV to determine the environment value to load env resources from “config.js”.

In config files, we changed to use ENV instead of NODE_ENV.

const env = process.env.ENV || 'development';

const configs = {
  development: {
    api: 'https://api.staging.com',
  },
  staging: {
    api: 'https://api.staging.com',
  },
  production: {
    api: 'https://api.production.com',
  },
}[env];

export default configs;

Conclusion

This solution works well for our projects using Next. Of course, there are many solutions out there. If you have a better solution, you can give me the feedbacks.

Thank you very much!