Abstraction for the sake of Abstraction

I wanted to talk a bit about task runners and automation. Personally, I've utilized NPM and GruntJS to automate repetitive tasks on multiple projects. That includes file compression, JavaScript/PHP linting, Sass compilation, script/style minification, amd running PHP/JS unit tests. I've even used it to keep mirrored SVN and Git repos in sync.

Having just started a new project, I went back to my previous projects for a refresher. I didn't want to reinvent the wheel, so I was going to scavenge whatever I needed from the old projects. Looking back, I used the following NPM packages:

1. grunt
2. grunt-cli
3. grunt-concurrent
4. grunt-contrib-concat
5. grunt-contrib-cssmin
6. grunt-contrib-jshint
7. grunt-contrib-sass
8. grunt-contrib-uglify
9. grunt-contrib-watch
10. grunt-exec
11. grunt-mocha-test
12. load-grunt-tasks
13. time-grunt
14. sinon
15. jsdom
16. mocha
17. chai

That didn't seem too bad, but then I looked at the Gruntfile. It was 195 lines long! One of my other projects used less packages, but the Gruntfile was 425 lines long...Yeah.

I came to the sudden realization that I had put a ton of effort into utilizing Grunt for these projects. The reason being that I was learning to use NPM and Grunt and the like. Although it was useful for learning automation and how to write unit tests, it was otherwise unnecessary. Nearly all the tasks that I needed to accomplish could be accomplished without Grunt. I realized that I was adding abstraction for the sake of abstraction.

When that bubble popped, I remembered the NPM has built-in functionality for running commands using npm run-script. Yup, I totally forgot about that!

So, I thought, "how could this be done with NPM only?"

Here's what I've come up with, for the above project, with comparisons between using Grunt and the default NPM run-script feature.

Sass Compilation and Minification

Grunt setup:

require( 'load-grunt-tasks' )( grunt );

grunt.initConfig({
    paths: {
        sass: {
            dir: 'styles',
            files: '<%= paths.sass.dir %>/**/*.scss'
        },
        css: {
            dir: 'public/styles'
        },
    },
    sass: {
        options: {
            style: 'expanded'
        },
        dist: {
            files: [{
                expand: true,
                cwd: '<%= paths.sass.dir %>',
                src: ['**/*.scss'],
                dest: '<%= paths.css.dir %>',
                ext: '.css'
            }]
        }
    }
});

grunt.registerTask( 'scss', 'sass' );

Grunt usage:

$ grunt scss

NPM setup:

"scripts": {
    "sass": "sass --scss -t compressed styles/*.scss public/styles/style.min.css"
}

NPM usage:

$ npm run sass

JavaScript Linting, Compression, and Minification

Grunt setup:

require( 'load-grunt-tasks' )( grunt );

grunt.initConfig({
    paths: {
        js: {
            source: 'scripts/*.js',
            public_dir: 'public/scripts/',
            public_dest: '<%= paths.js.public_dir %>main.js',
            public_ugly: '<%= paths.js.public_dir %>main.min.js',
            files: [
                '<%= paths.js.source %>',
                'Gruntfile.js',
                'test/**/*.js',
                '!test/utils/**/*.js'
            ]
        },
    },
    concat: {
        js: {
            src: '<%= paths.js.source %>',
            dest: '<%= paths.js.public_dest %>'
        }
    },
    uglify: {
        options: {
            mangle: {
                except: ['jQuery']
            }
        },
        target: {
            files: {
                '<%= paths.js.public_ugly %>': ['<%= paths.js.public_dest %>']
            }
        }
    },
    jshint: {
        options: {
            curly: true,
            eqeqeq: true,
            browser: true,
            devel: true,
            undef: true,
            unused: false,
            mocha: true,
            globals: {
                'jQuery': true,
                'module': true,
                'require': true,
                'window': true,
                'global': true
            }
        },
        dist: '<%= paths.js.files %>'
    }
});

grunt.registerTask( 'js', [ 'jshint', 'uglify', 'concat' ] );

Grunt usage:

$ grunt js

NPM setup:

"scripts": {
    "js": "jshint scripts/*.js test/*.test.js && uglifyjs scripts/*.js -cmo public/scripts/word_search.min.js"
}

NPM usage:

$ npm run js

Mocha Unit Tests

Grunt setup:

require( 'load-grunt-tasks' )( grunt );

grunt.initConfig({
    paths: {
        test: {
            files: 'test/**/*.test.js'
        },
    },
    mochaTest: {
        test: {
            options: {
                reporter: 'spec',
                require: 'test/utils/jsdom-config.js'
            },
            src: '<%= paths.test.files %>'
        }
    }
});

grunt.registerTask( 'mocha', 'mochaTest' );

Grunt usage:

$ grunt mocha

NPM setup:

"scripts": {
    "test": "mocha -R spec -r test/utils/jsdom-config.js test/*.test.js"
}

NPM usage:

Since npm run-script accepts test as a predefined task, the command is even shorter!

$ npm test

Deployment

My source directory is right next to my distribution directory. The Grunt task for deployment requires the grunt-exec dependency, which allows running command line expressions. The deploy task runs all the linting, uglification, and sass compilation before copying the public/ directory to the distribution directory. For the sake of brevity, I'm not going to list all the tasks out again, just the key ones.

Grunt setup:

require( 'load-grunt-tasks' )( grunt );

grunt.initConfig({
    pkg: grunt.file.readJSON( 'package.json' ),
    paths: {
        host: {
            dir: '../foresthoffman.github.io/<%= pkg.name %>/'
        },
        source: {
            dir: 'public/'
        }
    },
    /* Other task definitions */
    exec: {
        copy: {
            cmd: function () {
                var host_dir_path = grunt.template.process( '<%= paths.host.dir %>' );
                var source_dir_path = grunt.template.process( '<%= paths.source.dir %>' );

                var copy_command = 'cp -r ' + source_dir_path + ' ' + host_dir_path;

                return copy_command;
            }
        }
    },
});

grunt.registerTask( 'build', [
    'jshint', 
    'sass', 
    'cssmin', 
    'mochaTest', 
    'concat', 
    'uglify', 
    'exec:zip'
]);
grunt.registerTask( 'deploy', [ 'build', 'exec:copy' ] );

Grunt usage:

$ grunt deploy

NPM setup:

I made a simple bash script, so that I could completely drop Grunt for this task.

##
# cp_public
##

if [ ! $# == 2 ] || [ ! -d $1 ] || [ ! -d $2 ]; then
    echo "cp_public /path/to/source /path/to/dist"
    exit
fi

# the name property line from the package.json file in the current directory
name_line=$(grep -Ei "^\s*\"name\":\s\".*\",$" package.json | grep -oEi "[^\",]+")

# the package name by itself
name=$(echo $name_line | cut -d " " -f 3)

path_reg="\/"

source_path="$(echo ${1%$path_reg})/"
dist_path="$(echo ${2%$path_reg})/$name"

cp -r $source_path $dist_path

"scripts": {
    "deploy": "./cp_public public/ ../foresthoffman.github.io/"
}

NPM usage:

$ npm run deploy

Watchers

Rather than using Grunt to run concurrent file watchers I opted to use the npm-watch package. This allows me to indicate X number of scripts (from the scripts property of my package.json file) and the files that should trigger the scripts while the watcher is active. I only really needed to have the watcher handle js files changing, since the sass command has it's own --watch argument.

Then the configuration for the watch statement looks like this:

"watch": {
    "js": "scripts/*.js"
},
"script": {
    "js": "uglifyjs scripts/*.js -cmo public/scripts/main.min.js",
    "watch": "npm-watch"
}

Which can then be run via, npm run watch.

While writing this I actually implemented the changes that I've mentioned here. I've deleted my Gruntfile and am left with the configurations in my package.json and my new bash script in cp_public. Everything is working as intended. Woot!

package.json...

"devDependencies": {
    "chai": "^3.4.1",
    "jsdom": "^7.2.2",
    "mocha": "^2.3.4",
    "npm-watch": "^0.1.7",
    "sinon": "^1.17.2"
}
...
"watch": {
    "js": "scripts/*.js"
},
"scripts": {
    "deploy": "./cp_public public/ ../foresthoffman.github.io/ || true",
    "sass": "sass --scss -t compressed styles/*.scss public/styles/style.min.css",
    "sassWatch": "sass --watch --scss -t compressed styles/style.scss:public/styles/style.min.css",
    "js": "uglifyjs scripts/*.js -cmo public/scripts/main.min.js",
    "test": "mocha -R spec -r test/utils/jsdom-config.js test/*.test.js || true",
    "testWatch": "mocha -w -R spec -r test/utils/jsdom-config.js test/*.test.js || true",
    "watch": "npm-watch"
}

and cp_public...

#!/bin/bash

##
# cp_public
#
# Copies the source directory to the target directory.
#
# Usage: cp_public /path/to/source /path/to/dist
# 
# The package.json file, from which the package name is collected is relative to where this script
# is executed. It uses the current directory. That is why this script should be placed next to the
# package.json file in the project's hierarchy.
##

if [ ! $# == 2 ] || [ ! -d $1 ] || [ ! -d $2 ]; then
    echo "cp_public /path/to/source /path/to/dist"
    exit
fi

# the name property line from the package.json file in the current directory
name_line=$(grep -Ei "^\s*\"name\":\s\".*\",$" package.json | grep -oEi "[^\",]+")

# the package name by itself
name=$(echo $name_line | cut -d " " -f 3)

path_reg="\/"

source_path="$(echo ${1%$path_reg})/"
dist_path="$(echo ${2%$path_reg})/$name"

cp -r $source_path $dist_path

That's all folks!

I will admit that there aren't many good packages for concurrency without having to take on a lot more dependencies. In that way using Grunt (or another runner) is beneficial. Personally, I can no longer justify adding that complexity for what seems like a relatively minute benefit.

I like tooling and automation. I think writing unit tests is a freaking fantastic and very rewarding endeavor. However, I feel that there is a point at which the tools we use to automate rudimentary tasks add too much complexity. I'm seeing more and more of this everyday, thanks to the speed at which the web development ecosystem is progressing.

Anyone else feel the same? Anyone stuck in situation where they can't actually justify removing these kinds of dependencies? Anyone still a huge fan of runners, and will use them in future projects?

Obligatory xkcd comic plug:

Comments

[Josh Cheek]

Love your point, here! Do note, in. your bash script, several of the vars in there should be quoted to avoid them getting broken into multiple tokens. Also, jq is the king of command-line JSON, if you have it installed, you can change the code to get the program name to jq -r .name package.json

While I generally find them inflexible and opaque, sometimes runners can be useful. Eg, for complex dependency graphs, make and rake are honestly pretty good. I've not done enough js to know how they compare to something like webpack. I do like the idea that with the import keyword, those dependency graphs can be automatically detected, but reading the web pack file, I had no clue what any of it did or how any of it fit together. Make and Rake are nice and simple: "this file depends on those files, and is built with this command-line invocation".

I similarly spent 3 hours trying to get gulp to do the fkn simplest thing. Finally got fed-up, b/c my goal was to play w/ React, not Gulp, finally just removed it and wrote the build script in Ruby in ~20 min.

[Forest Hoffman]

Oh okay, thank you for the feedback. Gulp threw me in a similar loop, so I can definitely empathize.

[Forest Hoffman]

That's exactly what my 425 line Gruntfile looks like. I'm definitely not going to be taking that one and removing Grunt from the picture!

The major issue I've had with mixing bash scripts and Grunt is that it requires the grunt-exec package. It's a nice concept, but the implementation requires more setup for basic commands compared to simply running the command via an alias.

Thanks for the feedback! Happy coding. :D

[Jason Crist]

Just made the same switch. I was in the process of transitioning from grunt to gulp and realized everything I wanted to do was easier as a script. The amount of dev dependencies and configuration cruft went down dramatically. It was a good journey for me to take but glad to have ended up in a simple spot.

[Games For Girls]

I like the idea but I usually chain my grunt tasks together and so at that point if I pull them out of grunt I'm writing a shell script which might as well be nearly as long as my grunt file. I find I usually have some combination of the two in the end.