DEV Community: Jacob Goh

3 Ways You Could Customize 3rd Party React Component

Jacob Goh — Sat, 08 Dec 2018 08:09:08 +0000

Introduction

Component libraries make our life easier.

But as developers, you would often find yourself in situations where 3rd party components don't provide the functionality or customization capability the project needs.

We are left with 2 choices:

Write the component from scratch yourself
Customize the 3rd party components

What to choose depends on the component and the situation that you are in.

Apparently, some components are not customizable, Some feature requirements are not feasible. But most of the time, customizing 3rd party component is the less time-consuming option. Here's how.

Before we start

As example, we are going to customize react-bootstrap-typeahead component.

Here's the starter if you wanna follow along https://stackblitz.com/edit/react-hznpca

1. Overwriting CSS

This is fairly straightforward.

Just find out what's the component's CSS classes and overwrite them with new CSS.

Example

Goal: Add a dropdown icon to the input box, so that it looks like a drop-down.

Just add Font Awesome to index.html

and add these CSS to style.css

Demo: https://stackblitz.com/edit/react-wdjptx

2. Wrapper Component

This is where you could alter the default behavior of the 3rd party component.

Start by creating a wrapper component CustomizedTypeahead and replace Typeahead with it.

https://stackblitz.com/edit/react-rwyjmm

This wrapper component has no effect for now. It's simply passing props down to the Typeahead component.

We are going to customize the component behavior by making changes to props.

Example: Setting Default Props

Goal: Adding default props

Let's start with the simplest customization.

Let say we want all the CustomizedTypeahead to have the clearButton props enabled by default.

We can do so by

This is equilavant to

We create injectedProps and will put all the props modification inside to make the codes manageable.

Demo: https://stackblitz.com/edit/react-tk9pau

Example: Modifying Props

Goal: To sort all options by alphabetic order

We are receiving options, which is an array of objects, and labelKey, which tell us that the option's label should be optionObject[labelKey]. Our goal is to sort optionObject[labelKey] by alphabetic order.

We can do so by using Array.prototype.sort() to sort the options array.
This way, the options in injectedProps will overwrite the original options in props. That's how we can sort all options by alphabetic order by default.

Demo: https://stackblitz.com/edit/react-cqv5vz

Example: Intercepting Event Listeners

Goal: When the user selects an option, if the user has selected both "California" and "Texas" together, alert the user and clear the selection (for no particular reason other than for demo).

This is the fun part where you can do lots of customization.

Basically, this is how it will work,
Note the if(onChange) onChange(selectedOptions);. This makes sure that the original onChange event listener continues to run after we intercept it.

Here's what we did in the code above,

We create an onChange function that is of the same structure of the default onChange function. It's a function that receives an array of selected options.
We scan through the selected options and check if it's valid.
If it's invalid,
- show an alert
- clear the input
Run the original onChange event listener

Demo: https://stackblitz.com/edit/react-ravwmw

3. Modifying the source code

Caution: Don't overuse this! This is your last resort. You should only do this if there is no other choice.

If none of the above works for you, the choices you have are now limited to:

Find another component library
Write your own component from scratch
Modify the component source code

It's actually not uncommon that one would have to modify a package's source code to fit a project's need. Especially if you found a bug in a package and you need it fixed urgently.

But there are a few cons:

Some package uses different languages like CoffeeScript, Typescript. If you don't know the language, you don't know how to edit it.
It can be time-consuming to study the source code and figure out where exactly to put your modification.
You may unintentionally break some part of the package.
When the package updates, you would need to manually apply the update.

If you decide to go ahead and make some modification to the source code, here's how.

1. Fork the Github Repository

In our example case, go to https://github.com/ericgio/react-bootstrap-typeahead and fork the repo to your own GitHub account.

2. Clone the repo to your machine

3. Make the modification

4. Push the repo to your GitHub account

5. Install your repo as a dependency

After you fork the repo, your GitHub repo's URL should be https://github.com/<your GitHub username>/react-bootstrap-typeahead.

You can install this git repo as a dependency by executing this command

npm i https://github.com/<your GitHub username>/react-bootstrap-typeahead

After installation, you should see this in package.json

  "dependencies": {
    "react-bootstrap-typeahead": "git+https://github.com/<your github username>/react-bootstrap-typeahead.git"
  }

Conclusion

We talked about 3 ways to customize 3rd party React component.

Overwriting CSS
Using Wrapper Component
Modifying the source code

Hopefully, this would make your life as a React developer easier.

In the meantime, let's all take a moment and be grateful to all the open source creators/contributors out there. Without these open source packages, we wouldn't be able to move as fast as we do today.

What's your experience with 3rd party component libraries? What other method would you use to customize them? Leave a comment!

Simple & Customizable Web Scraper using RxJS and Node

Jacob Goh — Sat, 10 Nov 2018 17:08:00 +0000

Introduction

After getting to know RxJS (Thanks to Angular!), I realized that it's surprisingly a good fit for handling web scraping operations.

I tried it out in a side project and I would like to share my experience with you. Hopefully, this would open your eyes to how reactive programming can make your life simpler.

The codes can be found at

jacobgoh101 / web-scraping-with-rxjs

Codes for article Simple & Customizable Web Scraper using RxJS and Node

View on GitHub

Requirements

Node
RxJS and intermediate understanding of it
cheerio: it allows you to use jQuery like syntax to extract information out of HTML codes
request-promise-native: for sending HTTP request

Hypothetical Goal

Everybody loves a good comedy movie.

Let's make it our goal to scrape a list of good comedy movies from IMDB.

There are only 3 requirements that the target data needs to fulfill

it is a movie (not TV shows, music videos, etc)
it is a comedy
it has a rating of 7 or higher

Get Started

Let's set our base URL and define a BehaviorSubject allUrl$ that uses the base URL as the initial value.

(A BehaviorSubject is a subject with an initial value.)

const { BehaviorSubject } =  require('rxjs');

const  baseUrl  =  `https://imdb.com`;

const  allUrl$  =  new  BehaviorSubject(baseUrl);

allUrl$ is going to be the starting point of all crawling operation. Every URL will be passed into allUrl$ and be processed on later.

Making sure that we scrape each URL only once

With the help of distinct operators and normalize-url, we can easily make sure that we never scrape the same URL twice.

// ...
const { map, distinct, filter } =  require('rxjs/operators');
const  normalizeUrl  =  require('normalize-url');

// ...

const  uniqueUrl$  =  allUrl$.pipe(
    // only crawl IMDB url
    filter(url  =>  url.includes(baseUrl)),
    // normalize url for comparison
    map(url  =>  normalizeUrl(url, { removeQueryParameters: ['ref', 'ref_']     })),
    // distinct is a RxJS operator that filters out duplicated values
    distinct()
);

It's time to start scraping

We are going to make a request to each unique URL and map the content of each URL into another observable.

To do that, we use mergeMap to map the result of the request to another observable.

const { BehaviorSubject, from } =  require('rxjs');
const { map, distinct, filter, mergeMap } = require('rxjs/operators');
const rp = require('request-promise-native');
const  cheerio  =  require('cheerio');

//...
const urlAndDOM$ = uniqueUrl$.pipe(
  mergeMap(url => {
    return from(rp(url)).pipe(
      // get the cheerio function $
      map(html => cheerio.load(html)),
      // add URL to the result. It will be used later for crawling
      map($ => ({
        $,
        url
      }))
    );
  })
);

urlAndDOM$ will emit an object consist of 2 properties, which are $ and url. $ is a Cheerio function where you can use something like $('div').text() to extract information out of raw HTML codes.

Crawl all the URLs

const { resolve } =  require('url');
//...

// get all the next crawlable URLs
urlAndDOM$.subscribe(({ url, $ }) => {
  $('a').each(function(i, elem) {
    const href = $(this).attr('href');
    if (!href) return;

    // build the absolute url
    const absoluteUrl = resolve(url, href);
    allUrl$.next(absoluteUrl);
  });
});

In the code above, we scrape all the links inside the page and send it to allUrl$ for it to be crawled later.

Scrape and save the movies we want!

const  fs  =  require('fs');
//...

const isMovie = $ =>
  $(`[property='og:type']`).attr('content') === 'video.movie';
const isComedy = $ =>
  $(`.title_wrapper .subtext`)
    .text()
    .includes('Comedy');
const isHighlyRated = $ => +$(`[itemprop="ratingValue"]`).text() > 7;

urlAndDOM$
  .pipe(
    filter(({ $ }) => isMovie($)),
    filter(({ $ }) => isComedy($)),
    filter(({ $ }) => isHighlyRated($))
  )
  .subscribe(({ url, $ }) => {
    // append the data we want to a file named "comedy.txt"
    fs.appendFile('comedy.txt', `${url}, ${$('title').text()}\n`);
  });

Yup, we had just created a web scraper

In around 70 lines of code, we have created a web scraper that

automatically crawled URLs without unnecessary duplicates
automatically scrape and save the info we want in a text file

You may see the code up to this point in https://github.com/jacobgoh101/web-scraping-with-rxjs/blob/86ff05e893dec5f1b39647350cb0f74efe258c86/index.js

If you had tried writing a web scraper from scratch, you should be able to see now how elegant it is to write one with RxJS.

But we are not done yet...

In an ideal world, the code above may work forever without any problem.

But in reality, ~~shits~~ errors happen.

Handling Errors

Limit the number of active concurrent connection

If we send too much request to a server in a short period of time, it's likely that our IP would be temporarily blocked for making any further request, especially for an established website like IMDB.

It's also considered rude/unethical to send to request at once because it would create a heavier load on the server and in some cases, crash the server.

mergeMap has built-in functionality to control concurrency. Simply add a number to the 3rd function argument and it will limit the active concurrent connection automatically. Graceful!

const maxConcurrentReq = 10;
//...
const urlAndDOM$ = uniqueUrl$.pipe(
  mergeMap(
    //...
    null,
    maxConcurrentReq
  )
);

Code Diff: https://github.com/jacobgoh101/web-scraping-with-rxjs/commit/6aaed6dae230d2dde1493f1b6d78282ce2e8f316

Handle and Retry Failed Request

Requests may fail randomly due to dead links or server-side rate limiting. This is crucial for web scrapers.

We can use catchError, retry operators to handle this.

const { BehaviorSubject, from, of } =  require('rxjs');
const {
  // ...
  retry,
  catchError
} = require('rxjs/operators');
//...

const maxRetries = 5;
// ...

const urlAndDOM$ = uniqueUrl$.pipe(
  mergeMap(
    url => {
      return from(rp(url)).pipe(
        retry(maxRetries),
        catchError(error => {
          const { uri } = error.options;
          console.log(`Error requesting ${uri} after ${maxRetries} retries.`);
          // return null on error
          return of(null);
        }),
        // filter out errors
        filter(v => v),
        // ...
      );
    },

Code Diff: https://github.com/jacobgoh101/web-scraping-with-rxjs/commit/3098b48ca91a59aa5171bc2aa9c17801e769fcbb

Improved Retry Failed Request

Using retry operator, the retry would happen immediately after the request failed. This is not ideal.

It's better to retry after a certain amount of delay.

We can use the genericRetryStrategy suggested in learnrxjs to achieve this.

Code Diff: https://github.com/jacobgoh101/web-scraping-with-rxjs/commit/e194f4ff128a573241055ffc0d1969d54ca8c270

Conclusion

To recap, in this post, we discuss

how to crawl a web page using Cheerio
how to avoid duplicated crawl using RxJS operators like filter, distinct
how to use mergeMap to create an observable of request 's response
how to limit concurrency in mergeMap
how to handle error
how to handle retry

I hope this has been helpful to you and has deepened your understanding of RxJs and web scraping.