DEV Community: Badewa kayode

Managing state with binary data & bitwise op

Badewa kayode — Sat, 19 Jun 2021 15:57:51 +0000

Bitwise Op, What is it?

A bitwise operation that operates on a bit string, a bit array, or a binary numeral at the level of individual bits - Wikipedia

Why save state in binary?

Encoding data in a serialized binary format is more efficient than a string format. That's why data encoders like Gob, Protobufs are more efficient than JSON & XML.

They reduce the amount of space required to save data. They don't contain extra rubbish that takes space like field names or whitespaces.

One disadvantage of saving state or serializing data in binary format is rigidness. Since they follow a strict format, both the encoder & decoder must update how they handle new changes to the data format.

Let's say we have a position object that has only three type int8 fields (radius, longitude, and latitude). We can represent the fields in a binary format as {toBinary(rad)}{toBinary(lat)}{toBinary(long)}

If the radius is 30(11110), latitude is 20(10100) and longitude is 50(110010) we save it in our serialized binary format as

{toBinary(30)}{toBinary(20)}{toBinary(50)} = {00011110}{00010100}{00110010} = 000111100001010000110010

It can be represented in Hex as

1e1432

If we are to save this in JSON it will be

{"rad":30,"lat":20, "long": 50}

The JSON format takes more memory than the binary format and requires space for field names, while the binary format knows where the fields belong based on the position of the byte. This shows the space efficiency of storing states with binary.

Let's say we no longer need the radius field in the position object.

If we used a JSON format, removing the radius field won't affect the encoding & decoding rules.

If we used the binary serialized format from earlier, we have to make sure both the encoder & decoder are aware of the changes to the radius field.

When one expects {toBinary(rad)}{toBinary(lat)}{toBinary(long)} and the other is expecting {toBinary(lat)}{toBinary(long)} ****there will be a mismatch of field vaules and data won't be accuratly encoded/decoded.

We have to pass 0 as the first byte, provided the encoder & decoder are not in sync. Doing this keeps the format of the previous position object and can be ignored by the side that is up to date.

A simple use case

We have a notification service that groups notifications by categories and mediums(SMS, Email, and Push).

This notification service has two categories of notification which are deposits and withdrawals.

{
    "deposit": {
        "push": {
            "state": true,
            "can_edit": true
        },
        "email": {
            "state": true,
            "can_edit": false
        },
        "sms": {
            "state": true,
            "can_edit": false
        }
    },
    "withdrawal": {
        "push": {
            "state": true,
            "can_edit": true
        },
        "email": {
            "state": true,
            "can_edit": false
        }
    }
}

So the goal is to come up with a way to save the JSON object in a binary serialized format. Something like this 0001011100000111

Going From Json→Binary

Binary is made up of only 0 & 1, which can represent boolean values. Every bit position in binary is a power of 2, just like each position in decimal is a power of 10(1,10,100).

To format the JSON representation above to binary we first flatten the content of the notification category.

{
  "deposit": {
    "push_state": true,
    "push_can_edit": true,
    "email_state": true,
    "email_can_edit": false,
    "sms_state": true,
    "sms_can_edit": false
  },
  "withdrawal": {
    "push_state": true,
    "push_can_edit": true,
    "email_state": true,
    "email_can_edit": false,
    "sms_state": false, //false cus it's empty
    "sms_can_edit": false//false cus it's empty
  }
}

Now that it's flattened we want to represent each field in a byte. A byte is made up of 8 bits so we can represent the fields as individual bits in the byte.

Since the field type is bool we use 1 to represent true and 0 to represent false.

So we map the fields to bit positions like:

push_state => 1st bit
push_can_edit => 2nd bit
email_state => 3rd bit
email_can_edit => 4th bit
sms_state => 5th bit
sms_can_edit => 6th bit

We can now represent notification setting has

{
  "deposit": "00010111",
  "withdrawal": "00000111"
}

We want to change the JSON above to an array of bytes ([]byte). To do this we have to set index positions on the byte array for the notification categories.

For our byte array of length 2(two notification categories), deposit_vaue will be the 1st value in the array and withdrawal_value the second.

[]byte{deposit_value,withdrawal_value}
[]byte{0b00010111,0b00000111} = 0001011100000111 = 0x1707

Now we can finally represent our Notification Setting as

0001011100000111 #binary format
1707 #hexidecimal format
[]byte{23,7} #in code

The Set, Unset & Check Operators

Now that we have a binary serialized format to save the notification setting object, the next step is updating the binary representation of the notification setting. Let's call it bitNotifcationSetting.

We make use of the Set, Unset & Check bit operators to manipulate the bits of bitNotifcationSetting.

Set Operator: This sets the nth bit to 1.

set_op(data, nth_flag) => data | nth_flag

Unset Operator: This sets the nth bit to 0.

unset_op(data, nth_flag) => data & ^(nth_flag)

Check Bit Operator: this checks if the *n*th bit is set.

check_bit_op(data, nth_flag) => (data & (nth_flag) != 0

golang playground example

Updating A Notification Medium State

Let's say we want to update the SMS state.

Select the byte we want to edit using the index of the category (deposit = 0)
call `check_bit_op(*bitNotifcationSetting,sms_can_edit_flag) to c*heck if the sms_state` bit is editable.
If the `check_bit_op(*bitNotifcationSetting,sms_can_edit) == true`, move to the next step else return the field is not editable.*
call set_op(bitNotifcationSetting, sms_state_flag) to set the sms_state bit to 1/true, or call unset_op(bitNotifcationSetting, sms_state_flag) set sms_state bit to 0/false

Following the steps above we see how the operators are used in the manipulation of bits.

Conclusion

The binary format used here can help reduce memory space used and allows the cache hold more information about the notification setting in memeory without the need to cleanup to accomodate more recent values.

The example I used here is easy because the field type is boolean, which takes up a bit. For more complex types like strings and types with variable lengths, using a binary format will be a little hard to manage on your own without the help of external libraries.

For something more complex look at the way Bitcoin encodes transaction object to binary format.

Look at the way Bitcoin encodes transactions and blocks
Look into some of MongoDB bitwise query operators. It could help with querying documents when using BinData

Using Hashing to prevent file duplicates and save storage

Badewa kayode — Thu, 25 Jun 2020 09:51:06 +0000

This article aims to show a method of preventing duplicate files and saving storage space by using hashing algorithms.

There are cases where a video, image, or media might be trending due to a meme, news, or whatever can make anything trend. In such a scenario, the same file gets uploaded repeatedly by different users. With repeated uploads, the load on the server will increase, and more space used to save duplicates of the same file.

A hash function is any function that can be used to map data of arbitrary size to fixed-size values.
The values returned by a hash function are called hash values, hash codes, digests, or simply hashes. 
The values are used to index a fixed-size table called a hash table

- Wikipedia

How it works

I'm going to use a method that makes use of a file store, a key-value store (hash-storage linker), and a reference object to a file.

The File Store

It is responsible for saving the actual file uploaded by users. Storage platforms or services like S3, Google Storage, Block Storage, for example, can be used as a file store.

Hash Storage Linker

It is a DB that holds the file hash value as a key and the path to the file on the file store (e.g S3) as its value. A KV-store DB is great for storing this type of data.

It serves the middle-man between the file store and the file reference.

Note: The Hash-Storage is not required for this to work. If the server uses a single folder to store
 all the files, it can just hash the value of the file, and save the file using the hash value. 

Using the Hash-Storage makes checking the existence of the file faster since it is a KV-store. 

It can also act as an abstraction for separating files from file reference.
using it allows us to use multiple distributed file stores.

Provided the files exist, a hash-storage linker record must exist.

The Uploader's File Reference Object

It holds the reference to the hash and the contents of the file's metadata. It can be modified by the uploader, which means it can be created, updated, and deleted by the user.

Now That we have seen the entities that play a role in making this work, the next step is to show how they come together to solve the problem.

Upload Flow On The Frontend

The first step is for the frontend application (this could be a browser, mobile app, or another client) uploading the file, is to generate the hash value of the file to be uploaded. After generating the hash value check if a file with the hash value exists on the hash-storage linker(File exist endpoint).

If the file does not exist, the frontend application will send the file's content and the information(metadata) about the file (Upload file endpoint). This info passed with can be the filename, uploader info, time e.t.c.

If the file exists, The frontend will only send the information needed by the file reference and the hash of the file. It reduces latency and load on the frontend and backend applications.

Upload Flow On The Backend Server

The backend server waits for a request from a frontend application. The server handles the upload request based on the content passed in the body of the request.

The server expects the file metadata (information), and a body or hash value. Use body when the file does not exist and hash when the file exists.

If the file does not exist, body must be sent in the request body so that the server knows it's a new file and creates the file on the file storage service. After creating the file the server creates a record on the hash-storage linker links the file path on the file store to reference objects.

The last thing the server does is create a reference to the file for the uploader to use. The data contained in the reference object contains the hash key value from the hash-storage linker and extra information about the uploaded file. Info like time of upload, uploader's id, file_name, access .e.t.c. Just anything the user requires for the application to work.

When using the hash in place of body in the request body, create only the file's reference object.

Get File Flow On The Backend

When getting a file, the backend server provides two endpoints. One endpoint gets the file reference object,
and the other one gets the actual file content.

Fetching the file reference object is of a less task on the server.
The hash returned with the file reference object can be used to fetch the file, but in the case where the storage provider provides something similar to S3 Presigned Link, the server could return a presigned get link. The reason for this is to reduce the load on your server and move it to S3, which is more likely better equipped for pulling files from the internet (just saying).

Get File Flow On The Frontend

The frontend makes use of fetch file reference endpoint to get the file. The hash value extracted from the file reference object data helps identify the path to fetch the actual file on the backend or a locally stored data (this depends on the application).

Not all frontend implementation will have to save files locally, but if your app has that ability to save files, then storing it locally will save time when getting a file that has been downloaded.

WhatsApp's status is an example of a case where saving the file on the device is a cool idea. Sometimes a user sees the same video or image multiple times on different people's status.

Instead of downloading the same image multiple times, using the hash to save the file will reduce the number of times we request the server to get the same media file. The only thing that changes is the metadata (caption, uploaded time, user id).

Deleting Files

Since there is a possibility that different reference objects can depend on the same file, deleting it won't be so straight forward.

When Deleting the reference object, we need to know if other reference objects reference the same hash-storage linker as the soon to be deleted ref object. If the no ref object points to the hash-storage link, we remove the file from the file storage platform, and delete the hash-storage linker record, and the reference file.

If other ref object points to the same hash-storage linker record, then we delete only the ref file and leave the hash-storage linker and file store alone.

How the code will look like

The code in this article will focus on the backend part of the uploader. Postman will be used to represent the frontend service making the request.

Why use postman? It makes me lazy while at the same time doing a good job showing the frontend flow.

Postman Documentation

Hashing Method

The reason for hashing the file is to create a unique key for a file. So for this, we must use a hashing algorithm that has fewer chances of a collision.

In computer science, a collision or clash is a situation that occurs when two distinct 
pieces of data have the same hash value, checksum, fingerprint, or cryptographic digest. 

- Wikipedia

I should not be saying this, but the hashing algorithm on the frontend and backend should be the same.

The hashing algorithm I'll be making use of in the example is md5. It does the job of explaining my point here. For serious projects, use something that has fewer collision conflicts.

Check this github repo for the implementation.

Conclusion

I honestly don't know what to conclude this.

THE END 🤷‍♂. BYE 🤷‍♂.

Handling Slow Third-Party API

Badewa kayode — Sun, 03 May 2020 06:06:56 +0000

Recently, I had to make use of a third-Party API service that takes at least 30 seconds to respond.

The information these API services offer is not frequently requested by users but needed at some point by the user.

I can wait for the response to come back from the third-party service, but are users willing to wait for a response? A user may not be willing to wait for a reply and cancel the request before a response is returned.
If the user can't wait for the response to complete, they won't be able to get the info since it will always take 30 seconds or more to complete.

Every request to the API service costs money. If the user cancels the request before a response comes back, I will still pay for the request 😑. Some users will retry which means more money spent on getting nothing.

I have to come up with a way to reduce the request to the third-party API service, run the operation in the background in case the user cancels the request, and improving response time.

For this, I made use of a function that only calls the API once for a unique info_id. Other requests with the same info_id will wait for the response of the first call to complete. When the response arrives, every request waiting will get the response. The value returned is saved in a DB/Cache to speed up future requests.

This way, the user will wait for how long it takes the first request to complete. Requests made before the response of the first request will wait for the response, while requests made after it completes, get the value from the DB/Cache.

Implemetation

package main

import (
    "log"
    "sync"
    "time"
)

var (
    duration = time.Second * 5
)

type infoModel struct {
    // INFO FIELDS
    Name    string
    Age     int
    Sex     string
    Hobbies []string
    InfoID  string
}

// infoSaver defines the interface required to save infoModel to a db
type infoSaver interface {
    CreateInfo(i *infoModel) error
}

// infoChan the struct passed in the channel
// contains the info model and an error
type infoChan struct {
    info infoModel
    err  error
}

// infoBatchRequester this handles the request to the slow api/operation
// ensures only the first request call the api/op
type infoBatchRequester struct {
    mx      sync.RWMutex
    pending map[string][]chan<- infoChan // this holds pending requests
    infoSaver
}

// Request makes request to the api
func (i *infoBatchRequester) Request(infoID string, result chan<- infoChan) {
    var done chan infoChan

    i.mx.Lock()
    _, ok := i.pending[infoID]
    i.mx.Unlock()

    if !ok {
        // create the first pending request
        done = make(chan infoChan)

        i.mx.Lock()
        i.pending[infoID] = []chan<- infoChan{result}
        i.mx.Unlock()

        // make the request to get the info
        go func() {
            var (
                infoResp infoModel
                err      error
            )
            //pretend to do long running task
            log.Println("DOING LONG RUNNING TASK FOR. CALLED ONCE", infoID)
            time.Sleep(duration)
            infoResp = infoModel{
                Name:    "John Doe",
                Age:     50,
                Sex:     "Male",
                Hobbies: []string{"eating"},
                InfoID:  infoID,
            }

            //send info and error to done chan
            done <- infoChan{info: infoResp, err: err}

        }()

    } else {
        // append info to list of pending requests with the same infoID
        i.mx.Lock()
        i.pending[infoID] = append(i.pending[infoID], result)
        i.mx.Unlock()
        return
    }

    //wait for done to receive a value
    value := <-done

    i.mx.RLock()
    results, ok := i.pending[infoID]
    i.mx.RUnlock()

    if ok {
        for _, r := range results {
            r <- value
            close(r)
        }
    }

    defer func() {
        //delete the key when func execution is complete
        i.mx.Lock()
        delete(i.pending, infoID)
        i.mx.Unlock()
    }()

    if value.err == nil {
        log.Println("SAVED TO DB")
        log.Println(i.infoSaver.CreateInfo(&value.info))
    } else {
        log.Println("Error returned", value.err)
    }
    return
}

Using it within an HTTP handler

package main

import (
    "encoding/json"
    "errors"
    "log"
    "net/http"
)

// mockInfoSaver mock implementation of infoSaver
type mockInfoSaver struct {
}

func (b mockInfoSaver) CreateInfo(i *infoModel) error {
    log.Println("saving to db", i)
    return nil
}

var (
    ErrNotFound = errors.New("Not Found")
    defaultInfoBatchRequester = &infoBatchRequester{
        pending:   map[string][]chan<- infoChan{},
        infoSaver: mockInfoSaver{},
    }
)

// getInfo acts like a failed db get operation
func getInfo(infoID string) (infoModel, error) {
    return infoModel{}, errors.New("info does not exists")
}

// handleSlowThirdParty http handler that makes the call to the third party api
func handleSlowThirdParty(w http.ResponseWriter, r *http.Request) {
    infoID := r.URL.Query().Get("info_id")

    // make a fake request to get the info
    info, err := getInfo(infoID)

    if err != nil {
        // make a request to get the info
        result := make(chan infoChan, 1)
        go defaultInfoBatchRequester.Request(infoID, result)

        select {
        //the request was canceled early
        case <-r.Context().Done():
            log.Println("request terminated")

        //result returned
        case rs := <-result:
            if rs.err != nil {
                http.Error(w, rs.err.Error(), http.StatusBadRequest)
            }

            info = rs.info
        }
    }

    b, err := json.Marshal(info)

    w.Write(b)

}

Test file to check if it works as expected.

package main

import (
    "context"
    "fmt"
    "log"
    "sync"
    "testing"
    "time"
)

var (
    infoBatcher *infoBatchRequester
)

func Test_infoFetchPoll_ResolveINFO(t *testing.T) {
    infoBatcher = &infoBatchRequester{
        pending:   map[string][]chan<- infoChan{},
        infoSaver: mockInfoSaver{},
    }

    var wg sync.WaitGroup

    now := time.Now()

    // make 5 request to the get an info_id with the same id
    for i := 0; i < 5; i++ {
        wg.Add(1)
        go func(i int) {
            defer wg.Done()
            waitTime := time.Second * time.Duration(4+i)
            _, err := run(fmt.Sprintf("same_info"), i, waitTime)

            // not expecting error for the index greater than 1
            if i > 1 && err != nil {
                t.Errorf("not expection error %v on index %d", err, i)
            }
        }(i)
    }

    wg.Wait()
    log.Println("completed in ", time.Since(now).Seconds())

}

// run calls infoBatcher.Request for an infoID
// it takes the infoID, the index of the call, and the duration for the deadline.
// duration t is the user canceling the request after time t
func run(infoID string, id int, t time.Duration) (infoModel, error) {

    ctx, cancel := context.WithDeadline(context.TODO(), time.Now().Add(t))

    defer cancel()
    result := make(chan infoChan, 1) // using a buffered channel to prevent it from blocking if no
    // receiver is available

    infoBatcher.Request(infoID, result)
    select {
    case <-ctx.Done():
        log.Println("CLOSED", id, infoID)
        return infoModel{}, ctx.Err()

    case r := <-result:
        log.Println("PASSED", infoID, id, r)
        return r.info, r.err
    }

}

Running the test file returns the image we see below. The image shows the request was completed in approximately 5 seconds, the time it takes our fake request to complete.

We can see the request was to the slow fake service was called only once, and even when the request that made the call timed out, other requests with the same info_id still got a response.

Money saved for the remaining request to the same info_id. We can now use the extra cash to buy something for ourselves.

Conclusion

I don't know what to conclude with. I have run out of wise words to say 🤔.

Let me just say using the right algorithm can save money, time, and other things good algorithms can save.

The End.

Concurrency Won’t solve Your CPU problem, but it can help with your IO Problem

Badewa kayode — Mon, 20 Apr 2020 07:30:05 +0000

Moving articles from Medium to dev.to

Concurrency won’t solve your CPU bound problems, but it can help with your IO problem.

Many people feel they can improve the speed of their program by adding some concurrency. This is true and not true. concurrency can make your code slower than just running it serially.

What is concurrency to begin with? concurrency is dealing with multiple things at a time. It does not run two tasks simultaneously at the same time, which is parallelism. it runs tasks out of order so they appear to run at the same time.

Concurrency is mostly achieved using system-level threads that are spread across the available processors.

Concurrency with system-level threads has a cost for switching context. Some languages try to avoid it (Node.js), while some reduce it by using user-level threads (Go, Elixir) and some just use it (Java*).

The cost of context switching is a good price to pay for IO tasks, but not a good price for CPU intensive tasks.

Meet our Example functions

We are gonna separate our functions into 2 categories:

CPU functions (serial & concurrent)
IO function (serial & concurrent)

The CPU example functions call a poorly written Fibonacci function to simulate running a CPU intensive task. One function, cpuExampleSequential, runs the Fibonacci function in a loop serially, while cpuExampleConcurrent runs the Fibonacci function in a loop concurrently.

IO example function calls a struct that implements an io.Reader. The implementation of this reader pretends to make a long blocking call by sleeping for some time. A function in this example calls the reader serially, while the other calls it concurrently.

Below is the implementation of the example functions.

package main

import (
    "io"
    "sync"
    "time"
)

// a function that does much cpu bound work
func fibo(n int) int {
    if n < 2 {
        return 1
    }
    return fibo(n-2) + fibo(n-1)
}

// run sequentially
func cpuExampleSequential() {
    for i := 0; i < 10; i++ {
        fibo(40)
    }
}

// run concurrently
func cpuExampleConcurrent() {
    var wg sync.WaitGroup

    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            fibo(40)
        }()
    }

    wg.Wait()
}

type sleepReader struct {
    delay time.Duration
}

func (sr sleepReader) Read(b []byte) (int, error) {
    time.Sleep(sr.delay)
    copy(b, []byte("hello world"))
    return 0, nil
}

func read(r io.Reader) {
    b := make([]byte, 10)
    r.Read(b)
}

func ioExampleSequential() {
    for i := 0; i < 10; i++ {
        read(sleepReader{time.Second * 3})
    }
}

func ioExampleConcurrent() {
    var wg sync.WaitGroup

    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            read(sleepReader{time.Second * 3})
        }()
    }

    wg.Wait()
}

Now for some benchmark test functions

package main

import (
    "runtime"
    "testing"
)

// limit to a single processor
var _ = runtime.GOMAXPROCS(1)

func Benchmark_cpuExampleSequential(b *testing.B) {

    for n := 0; n < b.N; n++ {
        cpuExampleSequential()
    }
}

func Benchmark_cpuExampleConcurrent(b *testing.B) {
    for n := 0; n < b.N; n++ {
        cpuExampleConcurrent()
    }
}

func Benchmark_ioExampleSequential(b *testing.B) {
    for n := 0; n < b.N; n++ {
        ioExampleSequential()
    }
}

func Benchmark_ioExampleConcurrent(b *testing.B) {
    for n := 0; n < b.N; n++ {
        ioExampleConcurrent()
    }
}

The result from the benchmark

The result shows the following:

The difference between serially and concurrent execution of the CPU examples is not much. The concurrent benchmark takes a bit longer to execute.
There is a 10X difference between the serial and concurrent execution of the IO function. the concurrent call completes in approximately 3s, while the serial function call takes 30S.

The Reason

Let’s start with the CPU op. The reason for this is kind of simple. This is because the only processor we have spends most of its time doing work. There is no call to a blocking operation that can run in a background thread. Running the task concurrently won’t improve its performance, but can slow it down due to the cost from context switching. In Go’s case, there is little or no context switching (but there is a tiny cost for switching goroutines). The processor is always busy and we just added more busyness to it.

The IO concurrent call is 10X faster because the read function releases the CPU when a call is made to the reader. This gives other goroutines time with the processor. In the case of the serial call, the function spends most of the time waiting for the reader to complete before calling the next reader. Most of the work by the IO examples are blocking operation, so the CPU is free to perform other tasks.

Conclusion

From the examples above we can say that concurrency does not improve the performance of CPU intensive tasks, but we can use it in improving IO intensive tasks.

One way of solving the CPU intensive task is by making use of multiple processors. This solves the problem by adding more CPUs to run the task.

Note: the algorithm used should be written to support parallel execution. We have had cases where adding more processors slow down the system. Cases where all the processors have to wait on a lock before they can perform a task. Adding more processors, in this case, will slow down the performance.

There are still other ways of improving CPU heavy task. Some are :

Batching calls that perform similar tasks
Caching the response of expensive task
Improving the algorithm

Build a CLI to crawl a web page with web-crawljs

Badewa kayode — Sun, 19 Apr 2020 23:11:24 +0000

Moving Articles from medium to dev.to

In this tutorial, we are going to create a web crawler that scraps information from Wikipedia pages. This web crawler would run
from a command-line interface (e.g. terminal, command prompt).

The code for this article is on github.

An example of the command that would crawl the page would look like

$ node crawl.js -d 3 -x wiki

The command will get a config file named wiki and saves the crawled data to a MongoDB collection called wiki.

Web Crawling

Web crawlers are programs written to get information from a web page.

“A Web crawler, sometimes called a spider, is an Internet bot that systematically 
browses the World Wide Web, typically for the purpose of Web indexing”
 — Wikipedia

What we will be needing

For this project, we will be needing commander, web-crawljs, and mongoose.

Commander

Commander is an npm module that makes working with the command line interface easier. It makes it easy to handle command-line arguments. Check out it’s documentation.

web-crawljs

web-crawljs is an npm module that crawls web pages and extracts information from the page. It makes crawling web pages with Nodejs easy.

The only thing that web-crawljs needs, is a configuration object for it to start crawling.

why web-crawljs

One of the reasons I chose web-crawljs is because of how easy it is to crawl web pages with it. It is also a light weight web crawler, that is, it uses far less CPU and RAM compared to using a headless browser (e.g Phanthomjs).

Due to the less CPU and RAM usage, it can not render a SPA (single page application) pages. And also because I built It :).

All that is required to run it is Nodejs, no need for installing Phanthomjs on your machine; so far you have node installed, you are good to go.

mongoose

Mongoose is a MongoDB object modeling tool designed to work in an asynchronous environment. It’s an Object Data Modeling library that gives a modeling environment for MongoDB and it enforces a more structured data model.

Mongoose gives us the ability to create MongoDB data Models and Schemas.

We are going to use mongoose to save the information extracted from a page to the MongoDB database.

Project Structure

The structure of this project would look like this.

    ├── config
    │   ├── db.js
    │   └── wiki.js
    ├── crawl.js
    ├── package.json
    ├── package-lock.json
    └── readme.md

crawler/config

The main file in the crawler/config folder is the db.js. This file contains the configuration for our database. The wiki.js is the javascript file that will hold the configuration for web-crawljs.

Apart from db.js, all other files are configurations for web-crawljs.

crawler/config

The main file in the crawler/config folder is the db.js. This file contains the configuration for our database. The wiki.js is the javascript file that will hold the configuration for web-crawljs.

Apart from db.js, all other files are configurations for web-crawljs.

What we will crawl

In this article, we are going to extract some information out of Wikipedia and save it to a MongoDB database. The information we want to extract from the page are:

title of the wiki content
content of the wiki page
all the reference link

Requirements

For this tutorial, Nodejs and MongoDB must be installed on your machine. And I’ll be making use of node 7.8.0 and MongoDB version 2.6.10. I am also making use of ES6 syntax (arrow function, destructuring e.t.c).

node >=v7.0.0
mongodb

Let’s get started

Now let’s go straight to business. we will start by creating a new folder called crawler

$ mkdir crawler
$ cd crawler #move into the folder

Now that it is done, let’s create the config directory inside the crawler directory

$ mkdir config
#create the config files
$ touch config/wiki.js config/db.js
#create the crawl.js file
$ touch crawl.js

time to create the package.json file. use the npm init -y command to create the package.json (using it because it’s easy).

$ npm init -y

Installing the dependencies

We are making use of only three dependencies in this project, the mongoose, commander, and web-crawljs module. To install this module we will use our good friend npm. run npm install --save web-crawljs mongoose to install the dependencies.

$ npm install --save web-crawljs mongoose commander

Now that it is installed let's move to next stuff

config/db.js

This file holds the configuration details of our MongoDB database

/**
 * Created by kayslay on 6/3/17.
 */
module.exports = {
    dbName: "crawl",
    dbHost: "localhost",
};

config/wiki.js

The config/wiki.js file holds the configuration we will use to crawl our Wikipedia page.

/**
 * Created by kayslay on 6/3/17.
 */
const mongoose = require('mongoose');
const dbConfig = require('../config/db');
//mongoose configs
const Schema = mongoose.Schema;
//creating a schema for the extracted data
const wikiSchema = new Schema({
    title: String,
    body: String,
    references: [String]
});
//connect to mongo db
mongoose.connect(`mongodb://${dbConfig.dbHost}/${dbConfig.dbName}`);
//create the model
const wikiModel = mongoose.model('Wiki', wikiSchema);

//crawl config
module.exports = {
    //the selectors on page we want to select
    //here we are selecting the title, a div with an id of mw-content-text and links with a
    //class name of external and text
    fetchSelector: {title: "title", body: "div#mw-content-text",references: 'a.external.text'},
    //what we want to select from the selector
    //for the title and body we want the text
    //for the references we want to get the href of the links
    fetchSelectBy: {title: "text", body: "text",references:['attr','href']},
    // the same rules apply to the nextSelector and nextSelectBy
    //but this is used to get the links of the page to crawl next
    nextSelector: {links: 'a[href^="/wiki"]'},
    nextSelectBy: {links: ['attr','href']},
    //this changes the next selector when the links match .svg
    dynamicSchemas:{
        nextSelector:[{url:/\.svg/,schema:{links:""}}]
    },
    //formats the url
    formatUrl: function (url) {
        if((/\.svg?/.test(url) || /[A-Z]\w+:\w+?/.test(url))){
            //returning a visited string so that it does not visit the link
            //when the url ends with `.svg` or something like `Wikipedia:About`
        return 'https://en.wikipedia.org/wiki/Web_crawler/'
        }
        return url;
    },
    //what we want to do with the data extracted from the page
    //we want to save it to a mongodb database
    fetchFn: (err, data, url) => {

        if (err) {
            return console.error(err.message);
        }
        let {title, body, references} = data;
        let wikiData = {title: title[0], body: body[0], references};
        wikiModel.create(wikiData, function (err, wiki) {
            console.log(`page with a title ${wiki.title}, has been saved to the database`);
        });
    },
    //called at the end of the whole crawl
    finalFn: function () {
        console.log('finished crawling wiki');
    },
    depth: 3, //how deep the crawl should go
    limitNextLinks: 10,// limit the amount of links we get from wikipedia to 10. this helps when you dont want to get all the links
    urls: ['https://en.wikipedia.org/wiki/Web_crawler/'] //the default urls to crawl if one is not specified
};

crawl.js

#!/usr/bin/env node
/**
 * Created by kayslay on 5/31/17.
 */
const crawler = require('web-crawljs');
const program = require('commander');

//commander configuration
function list(val) {
    "use strict";
    return val.split(',');
}

program
    .option('-x --execute <string>', 'the configurtion to execute')
    .option('-d --depth [number]', 'the depth of the crawl')
    .option('-u --urls [items]', 'change the urls',list)
    .parse(process.argv);

//throw an error if the execute flag is not used
if (!program.execute) {
    throw new Error('the configuration to use must be set use the -x flag to define configuration;' +
        ' use the --help for help')
}
//holds the additional configuration that will be added to crawlConfig
const additionalConfig = {};

//set the object that will override the default crawlConfig
(function (config) {
    //depth
    if (program.depth) config['depth'] = program.depth;
    if(!!program.urls) config['urls'] = program.urls

})(additionalConfig);

//the action is the file name that holds the crawlConfig
let action = program.execute;


try {
    //set the crawlConfig 
    //adds the additional config if need
    let crawlConfig = Object.assign(require(`./config/${action}`), additionalConfig);
    const Crawler = crawler(crawlConfig);
    Crawler.CrawlAllUrl()
} catch (err) {
    console.error(`An Error occurred: ${err.message}`);
}

The crawl.js file is the main file of this project. This file is what we will run using the node command. It’s our entry point.

It depends on two packages: web-crawljs and commander. Which were imported on lines 5 and 6.

From line 9 to line 18 we set up the flags needed to be used by our CLI.

Thanks to commander this is very easy to achieve. Check its documentation for more.

Line 21 all the way down to line 37, configures the values gotten from the CLI.

The comment in the file should explain what's going on.

The lines that follow just performs the web crawl operation.

Let’s test our crawler

Now that's all the code has been written, it’s time to test the crawler.

Type the following in your terminal

$ node crawl.js -x wiki

When we check our MongoDB collection we will see the title, body and reference added to it.

Instead of using the default Wikipedia URL, we are going to use our own wiki page URL.

$ node crawl -u https://en.wikipedia.org/wiki/Web_crawler -x wiki

This will not start crawling from the default https://en.wikipedia.org/ but would start crawling from https://en.wikipedia.org/wiki/Web_crawler.
To add more URLs, separate the URLs by commas.

Conclusion

We now know how to create a web crawler using web-crawljs, commander and mongoose :).

And to those who don’t know how easy it is to create a Command Line Interface with Nodejs is; Now you know.

This is at least one more thing you know.

Thanks for reading and please recommend this post.

Running CPU Intensive task in Nodejs

Badewa kayode — Sun, 19 Apr 2020 22:16:58 +0000

Moving my articles from Medium to Dev.to

This article was originally posted here:

The code for the article can be found here.

Nodejs is good for IO intensive tasks but bad for CPU intensive tasks. The reason Nodejs is bad for CPU intensive task is that it runs on the event loop, which runs on a single thread.

The event loop is responsible for everything that runs on the user-land of Nodejs. This event loop runs on a single thread. When this thread is blocked all other tasks would have to wait for the thread to be unlocked before they can be executed.

I am not an expert on this issue, I am only giving a way in which I achieved this, so if anyone has something to add or some corrections to make about the post I’m open to advice.

Running Fibonacci

In this article, I would be using Fibonacci as my CPU intensive task (it takes time to get the Fibonacci number of numbers above 45). I am going to create a server that serves
a simple response for any URL that does not match /fibo, and when the URL matches /fibo I will serve a Fibonacci result.

In this article I will not use any npm module; I will just be using core node modules in this article.

The Server

The server for this article would only return two types of response:

A Fibonacci number for the req.headers.fibo value when the URL route is equal to fibo
A hello world string for any URL route that does not equal fibo

Lets run the fibo normally

First to show how Fibonacci blocks the event loop, I will create a server that serves a Fibonacci that runs on the same process as the simple hello world response.

Create a file called fibo_in_server.js. This file would return the Fibonacci number of a number passed into the
req.headers.fibo when the URL route is equal to the /fibo and return’s hello world for any other URL match.

        const http = require("http");

        function fibo(n) { 

            if (n < 2)
                return 1;
            else   return fibo(n - 2) + fibo(n - 1);
        }

        const server = http.createServer((req, res) => {
            "use strict";
            if (req.url == '/fibo') {
                let num = parseInt(req.headers.fibo); 
                console.log(num)
                res.end(`${fibo(num)}`) 
            } else {
                res.end('hello world'); 
            }
        });

        server.listen(8000, () => console.log("running on port 8000"));

We can run the above code and check the response. When the req.url is not /fibo the response is hello world and the Fibonacci number of the number passed into the header fibo field for a req.url that is equal to /fibo.

I’m using the Postman Chrome extension for requesting the server.

If we send a number like 45 to the server, the request would block the event loop until it is done getting the Fibonacci number. Any request to get the hello world string would have to wait until the long-running Fibonacci is done.

This is not good for users who want to get only a simple response, because they have to wait for the Fibonacci response to be completed.

In this article, what I am going to do is look at some ways to fix this problem. I am not a Pro Super NodeJs Guru User, but I can give some methods of dealing with this problem.

Methods of dealing with this problem

running Fibonacci in another Nodejs process
using method 1 with a batch queue to process the Fibonacci
using method 2 with a pool to manage the processes

Method 1: Running in another process

What we can do is run the Fibonacci function in another Nodejs process. This would prevent the event loop from getting blocked by the Fibonacci function.

To create another process we use the [child_process]() module. I am going to create a file, fibonacci_runner.js, that runs as the child
process, and another file called server_method1.js, the parent process.

The server_method1.js serves the response to the client. When a request to the /fibo is made the server gives the work to its child process fibo_runner.js to
handle. This prevents the event loop on the server from getting blocked, making it easier for a smaller request to be handled.

Here is the code for fibonacci_runner.js

process.on("message", (msg) => {
    "use strict";
    process.send({value: fibo(parseInt(msg.num)),event:msg.event})
});

function fibo(n) { // 1
    if (n < 2)
        return 1;
    else   return fibo(n - 2) + fibo(n - 1)
}

And here is the code for server_method1.js:

const http = require("http");
const {fork} = require('child_process');
const child = fork(`${__dirname}/fibonacci_runner.js`);
let {EventEmitter} = require('events');

let event = new EventEmitter();


const server = http.createServer(function(req, res){

    if (req.url == '/fibo') {
        let rand = Math.random() * 100; //generate a random number

        child.send({num:req.headers.fibo,event:rand});  //send the number to fibonacci_running

        event.once(rand, (value) => { //when the event is called
            res.end(`${value}`)
        })
    } else {
        res.end('hello world');
    }
});

child.on("message",(msg)=> event.emit(msg.event,msg.value)); //emit the event event sent

server.listen(8000, () => console.log("running on port 8000"));

Now if we visit the URL route /fibo with a value >= 45 in the req.headers.fibo value, it won’t block the request for the hello world. Better than what we had before.

The next step is to reduce the amount of computation the fibonacci_runner does. One way of reducing this is by using a batch queue with/or a cache (Note:
there are still other methods of doing this).

In this article, I am going to discuss the batch queue alone.

You can check out these articles to know more about the cache :

https://community.risingstack.com/redis-node-js-introduction-to-caching/amp/
https://goenning.net/2016/02/10/simple-server-side-cache-for-expressjs/

Method 2: Batching queue

When dealing with asynchronous operations, the most basic level of caching can be achieved by batching together a set of invocations to the same API. The idea is very simple: 
if I am invoking an asynchronous function while there is still another one pending, we can attach the callback to the already running operation, instead of Creating a brand new request. — “Nodejs Design Patterns”

From the definition above, we want to batch requests with the same req.headers.fibo value together, Instead of calling a new Fibonacci call while one with the same req.headers.fibo value
is still pending.

I am still going to use the fibonacci_runner.js to run the Fibonacci operation, but I’m going to create a new file, server_method2.js, that has
an asyncBatching function that sits between the fibonacci_runner.js and the call to process the req.headers.fibo.

Here is the code for server_method2.js

const http = require("http");
const {fork} = require('child_process');
const child = fork(`${__dirname}/fibonacci_runner.js`);
let Queue = {}//1

function asyncBatching(num, cb) {
    if (Queue[num]) {
        Queue[num].push(cb) //2
    } else {
        Queue[num] = [cb]; //3
        child.send({num: num, event: num})//4
    }
}

const server = http.createServer(function (req, res) {

    if (req.url == '/fibo') {
        const num = parseInt(req.headers.fibo)
        asyncBatching(num,(value)=>res.end(`${value}`))
    } else {
        res.end('hello world');
    }
});

child.on("message", (msg) =>{
    "use strict";
    let queue = [...Queue[msg.event]];
    Queue[msg.event] = null;  //empty the Queue
    queue.forEach(cb=>cb(msg.value))
    console.log(`done with ${msg.event}`)
});

server.listen(8000, () => console.log("running on port 8000"));

I would use the Apache benchmark to run this test

$ ab -n 10 -c 10 -H 'fibo: 39' http://localhost:8000/fibo

It takes 3.196 on my machine for method2,and 32.161 for method1. This means method2 responds n times faster than method1
(number of concurrent users sending the same req.headers.fibo value).

To improve method2 further we can use a cache to save the value of the Fibonacci but am not going to touch caching in
this article :(.

What is going to do here is improve on method2 by increasing the number of child processes. I am going to use a pool that
would manage the distribution of work among the child processes.

Method 3: Pooling and managing multiple processes

Creating multiple child processes to handle the Fibonacci operation would make it respond faster and better. You have to know that running many processes is making
use of system resources. Creating too many processes is bad; Just create enough.

The Pool is responsible for handling child processes. First, let’s create a Pool file, Pool.js, that exports a Pool class.

Code for Pool.js file:

const child = require('child_process');

class Pool {
    constructor(file, maxPool, messageCb) {
        this.pool = [];
        this.active = [];
        this.waiting = [];
        this.maxPool = maxPool;

        let releaseWorker = (function (worker) {
            //move the worker back to the pool array
            this.active = this.active.filter(w => worker !== w);
            this.pool.push(worker);
            //if there is work to be done, assign it
            if (this.waiting.length > 0) {
                this.assignWork(this.waiting.shift())
            }
        }).bind(this);

        for (let i = 0; i < maxPool; i++) {
            let worker = child.fork(file);
            worker.on("message", (...param) => {
                messageCb(...param);
                releaseWorker(worker)
            });
            this.pool.push(worker)

        }
    }

    assignWork(msg) {

        if (this.active.length >= this.maxPool) {
            this.waiting.push(msg);
            console.log(this.waiting)
        }

        if (this.pool.length > 0) {
            let worker = this.pool.pop();
            worker.send(msg);
            this.active.push(worker)
        }
    }

}

module.exports = Pool;

The Pool class

As said before, the Pool is responsible for handling the child process. It has only one method, the assignWorker method. The assignWorker method
assigns work to a worker (child process) to handle. If all the workers are busy the work would be done as soon as one is free.

The Pool Object takes three parameters on creation. These arguments are :

the file to run as the child process
the number of processes to create
the function to call when the workers send a message back

Now let’s create server_method3.js file that makes use of the Pool Object.

The code for server_method3.js:

const http = require("http");
let Queue = {};
const Pool = require("./Pool");

let Pooler = new Pool(`${__dirname}/fibonacci_runner.js`,2, (msg) => {
    "use strict";
    let queue = [...Queue[msg.event]];
    Queue[msg.event] = null;  //empty the Queue
    queue.forEach(cb => cb(msg.value));
    console.log(`done with ${msg.event}`)
});

//responsible for batching
function asyncBatching(num, cb) {
    if (Queue[num]) {
        Queue[num].push(cb)
    } else {
        Queue[num] = [cb];
        Pooler.assignWork({num: num, event: num})
    }
}

const server = http.createServer(function (req, res) {

    if (req.url == '/fibo') {
        const num = parseInt(req.headers.fibo);
        asyncBatching(num, (value) => res.end(`${value}`)) // 
    } else {
        res.end('hello world');
    }
});


server.listen(8000, () => console.log("running on port 8000"));

server_methodw3.js runs more than one child process, so we can run multiple Fibonacci operations at the same time,
instead of waiting for the one to finish.

The number of Fibonacci we can run at the same time depends on the number passed as the second parameter to the Pool
constructor.

Note: limit the number of processes you spawn ups.

Conclusion

Running heavy task on node event loop is a bad idea, and remember to pass the task to another process to handle, be it Nodejs or not (you can start a C++ to handle
very heavy operations).

Remember to always keep the event loop from getting blocked by any operation.

Read this article for more about the Event Loop.

Badewa Kayode, peace out :).

DEV Community: Badewa kayode

Managing state with binary data & bitwise op

Bitwise Op, What is it?

Why save state in binary?

A simple use case

Going From Json→Binary

The Set, Unset & Check Operators

Updating A Notification Medium State

Conclusion

Read More

Using Hashing to prevent file duplicates and save storage

How it works

The File Store

Hash Storage Linker

The Uploader's File Reference Object

Upload Flow On The Frontend

Upload Flow On The Backend Server

Get File Flow On The Backend

Get File Flow On The Frontend

Deleting Files

How the code will look like

Hashing Method

Conclusion

Handling Slow Third-Party API

Implemetation

Conclusion

Concurrency Won’t solve Your CPU problem, but it can help with your IO Problem

Meet our Example functions

The Reason

Conclusion

Build a CLI to crawl a web page with web-crawljs

Web Crawling

What we will be needing

Commander

web-crawljs

why web-crawljs

mongoose

Project Structure

crawler/config

crawler/config

What we will crawl

Requirements

Let’s get started

Installing the dependencies

config/db.js

config/wiki.js

crawl.js

Let’s test our crawler

Conclusion

Running CPU Intensive task in Nodejs

Running Fibonacci

The Server

Lets run the fibo normally

Methods of dealing with this problem

Method 1: Running in another process

Method 2: Batching queue

Method 3: Pooling and managing multiple processes

The Pool class

Conclusion