Add -i/--ignore option to insane-cli(Finding broken URL CLI program)

getting start

insane-cli is a great tool for checking if there is any broken URLs in your files. However, there is still a room for enhancement. This week I have decided to add a new feature so that the URLs can be excluded by passing ignore.txt.

New feature

The --ignore, -i, \i flag is optional, and if present should take a filepath and use it to load and parse an ignored URLs text file. All URLs should be checked against this list to see if they need to be ignored.

For example,

# Ignore All Seneca College URLs
https://senecacollege.ca
# Ignore all Learn@Seneca URLs
https://learn.senecacollege.ca/

Any text following # must not be read!!

Coding 1 -ignore.txt files-

first to test the feature I have created ignore.txt file like this

# 2. File with a comment and single URL that matches any URL
# that starts with https://www.google.com, for example:
#
# https://www.google.com/index.html would match
# https://www.google.com/ would match
# https://www.google.ca would NOT match
https://www.google.com
https://stackoverflow.com/
https://wiki.cdot.senecacollege.ca

Coding 2 -add -i/--ignore flag-

insane-cli program check every single argument that would be passed. So if certain flag matches the condition, each task will be exectued.

let argv3 = process.argv[2]
let argv4 = process.argv[3]
...
if (argv3 != null) {
    if (argv4 != null) { // for URL process
        if (argv3 === '-url' || argv3 === '/url') {
        ...
        } else if (argv3 === '-w' || argv3 === '/w') {
        ...
        } else {
          console.log('Command not found!');
    ...
}

To add a new flag -i, I need to add one more condition

...
else if (argv4 === '-i' || argv4 === '--ignore'){
...
} else{
   console.log('Command not found!');
}

So, if the process.argv3 is -i or --ignore, it would run the else if (argv4 === '-i' || argv4 === '--ignore') block.

Coding 3 -read ignore.txt-

fs.readFile is basically read whole txt at once. So it doesn't read the txt line by line. All we want to do in this feature is checking line by line, so that if the URL is following '#', it would be ignore.

To do that, I have added new library readline. This makes the program read the file line by line. First we need to read stream const fileStream = fs.createReadStream(process.argv[4]);
then, create the readline interface

    const fileStream = fs.createReadStream(process.argv[4]);
    const rl = readline.createInterface({
        input: fileStream,
    });

Now, we read the text file line by line,

If there is '#' ignore the line

    let ignoreURL = []
    ...
    for await (const line of rl) {
        if (!line.startsWith('#')) {
            ignoreURL.push(line)
        }
    }
    ...

So, the readIgnoreFile() function looks like:

const readIgnoreFile = async () => {
    let ignoreURL = []
    const fileStream = fs.createReadStream(process.argv[4]);
    const rl = readline.createInterface({
        input: fileStream,
    });

    for await (const line of rl) {
        if (!line.startsWith('#')) {
            ignoreURL.push(line)
        }
    }
    return ignoreURL;
}

Put altogether

} else if (argv4 === '-i' || argv4 === '--ignore') {
            readIgnoreFile().then(lists => {
                let filename = argv3;

                try {
                    if (fs.existsSync(filename)) { // check if file exist
                        (async () => {
                            await fs.promises.readFile(filename)
                                .then(function (data) {
                                    let finalURLs = getURLs(data)
                                    finalURLs = finalURLs.filter(url => {
                                        let flag = true;
                                        for (let i = 0; i < lists.length; i++) {
                                            if (url.includes(lists[i])){
                                                flag = false;
                                            } 
                                        }
                                        return flag
                                    })
                                    printURLStatus(finalURLs)

                                })
                                .catch(function (error) {
                                    console.log(error);
                                })
                        })()
                    }

PR

I let the owner of insane-cli know about this enhancement and make a PR on my fork repo. After he check the PR, he will remotely merge my branch commit to his master branch