Filtering Googlebot IPs from a list of IP addresses

Mert Yazıcıoğlu — Mon, 25 Jun 2018 15:05:10 +0000

Hey there dev.to community!

I recently needed a simple script to filter Googlebot IPs from a list of IP addresses to be able to extract actual Googlebot visits from an access log.

Thankfully, Google provides a method to make sure a visitor is actually Googlebot.

If you have a similar need, there you go:

#/bin/bash
#
# Performs reverse and forward DNS lookups to list Googlebot's IPs, given a list
# of IP addresses as a file. Useful for filtering access logs to find out actual
# Googlebot visits.
#
# An implementation of https://support.google.com/webmasters/answer/80553?hl=en

while IFS='' read -r IP_ADDRESS || [[ -n "$IP_ADDRESS" ]];
do
    IS_GOOGLEBOT=0
    REVERSE_LOOKUP="$(host $IP_ADDRESS)"
    echo "$REVERSE_LOOKUP" | grep -E "google.com.$|googlebot.com.$" > /dev/null && IS_GOOGLEBOT=1

    if [[ IS_GOOGLEBOT -eq 1 ]]; then
        FORWARD_LOOKUP="$(host $(echo "$REVERSE_LOOKUP" | cut -d " " -f 5) | cut -d " " -f 4)"
        if [[ "$FORWARD_LOOKUP" = "$IP_ADDRESS" ]];
        then
            echo $IP_ADDRESS
        fi
    fi
done < "$1"

You may save it as something like filter-googlebot-ips.sh and provide a file with a list of IP addresses to filter (each on a single line), as an argument. Like so:

$ ./filter-googlebot-ips.sh access-log-ips.txt > googlebot-ips.txt

This will perform reverse and forward DNS lookups for each of the IP addresses and print out the verified Googlebot IPs to STDOUT, which you can write to a file like in the example above.

Hope it helps someone out there! 🙌

PS: Here is a GitHub Gist if you prefer that.

JavaScript Linting Tools

Mert Yazıcıoğlu — Fri, 25 Aug 2017 14:23:09 +0000

Linting is statically checking the source code of a program to find code that does not adhere to a certain code style and discover problematic patters in the code. As a dynamic and loosely-typed language, JavaScript benefits greatly from a linting tool.

Below are the most popular linting tools for JavaScript, listed and briefly explained, in the order they are released.

For more information, please visit projects’ own websites.

JSLint

JSLint is a tool created by Douglas Crockford that aims to ensure high quality JavaScript code by checking the code against the rules defined by Douglas Crockford who detailed them in his book, JavaScript: The Good Parts.

Although JSLint has a few options to “tolerate certain things, it is not a configurable linting tool. The reason why is explained as follows at jslint.com:

JSLint was designed to reject code that some would consider to be perfectly fine. The reason for this is that JSLint’s purpose is to help produce programs that are free of error. That is difficult in any language and is especially hard in JavaScript. JSLint attempts to help you increase the visual distance between correct programs and incorrect programs, making the remaining errors more obvious. JSLint will give warnings about things that are not necessarily wrong in the current situation, but which have been observed to mask or obscure errors. Avoid those things when there are better options available.

It is dangerous out there. JSLint is here to help.

You can either use it online following the link above or integrate it to your development and/or deployment workflow:

Command-Line Tool: https://github.com/reid/node-jslint

Grunt Plugin: https://www.npmjs.com/package/grunt-jslint

Gulp Plugin: https://www.npmjs.com/package/gulp-jslint

The source code is also available on GitHub:

https://github.com/douglascrockford/JSLint

If you wish more control over the linting criteria, check out JSHint, which is explained right below.

JSHint

JSHint is a community-driven, flexible, open source linting tool that does not try to enforce certain coding preferences and allows developers to configure it according to their coding conventions. JSHint simply does its job by checking the code against the given style and detects errors and potential problems in the code.

You can either use it online following the link above or integrate it to your development and/or deployment workflow:

Command-Line Tool: https://www.npmjs.com/package/jshint
Grunt Plugin: https://www.npmjs.com/package/grunt-contrib-jshint
Gulp Plugin: https://www.npmjs.com/package/gulp-jshint
More Information: http://jshint.com/install/

ESLint

ESLint is the next-generation linter that is highly configurable to the point that it allows custom parsers and plugins to extend its linting process. The customizability of ESLint should not overwhelm you, you can still simply use it the way you use JSHint.

To fully grasp what ESLint is and what it isn’t, here is the philosophy behind ESLint, from its About page:

Everything is pluggable:

Rule API is used both by bundled and custom rules

Formatter API is used both by bundled and custom formatters

Additional rules and formatters can be specified at runtime

Rules and formatters don’t have to be bundled to be used

Every rule:

Is standalone

Can be able to be turned off or on (nothing can be deemed “too important to turn off”)

Can be set to be a warning or error individually

Additionally:

Rules are “agenda free” – ESLint does not promote any particular coding style

Any bundled rules are generalizable

For more information, please follow the instructions on Getting Started with ESLint:

If you are looking for more advanced features mentioned above, please follow the detailed instructions on Configuring ESLint:

But, which one to choose?

At this day and age, ESLint is hands down the best choice with its extensibility and out-of-the-box support for many of your modern day needs.

Get started with ESLint following the basic instructions linked above and dive deeper into its capabilities as your needs arise.