Using npx and npm scripts to Reduce the Burden of Developer Tools

On Friday, I was working on a workshop-ified version of Step by Step Express for Flawless Hacks in Brooklyn.

In the workshop-ified version, I've modified each step so there is an app.js and an app.complete.js so that attendees could start with a clean slate from the previous step and know what they're aiming for to complete in the step that they're working on.

I figured there was probably a tool on npm that would allow me to lower the barrier even further to let attendees/users know what they need to do to complete the step and check their code against app.complete.js if their code isn't doing what they think it is.

After a bit of searching, I was able to find Pretty Diff, which exposes a CLI tool that allows you to diff two files, and show the difference in the console.

I tested the CLI a bit after globally installing it and was able to figure out that because of how I structured the changes (app.js and app.complete.js in each folder in addition to each folder having its own package.json), I was able to use the same command for every single step:

diff source:"app.js" diff:"app.complete.js"

Awesome! I found a tool that will hopefully lower the barrier to finding success with the code that they'll be writing. There's only one problem: they'll need the CLI to be installed for the above command to work. That sounds like a fantastic way to increase the barrier to entry and waste time on tooling that was intended to improve the experience, not take away from it 😱

Enter npx

Luckily, there's this excellent tool that everyone who has a modern version of npm installed already has: npx.

In case you're not familiar, npx is a CLI that the npm team ships which automagically executes a CLI from a module on the npm registry. Ideally, most modules will only ship one top-level command – for those modules, you can simply run npx <module name> and the command will be executed.

In our case, we're looking to run the prettydiff module and pass the diff command. Luckily, npx makes this super easy:

npx prettydiff diff source:"app.js" diff:"app.complete.js"

Workshop attendees can 100% run this in any of the steps' directories and they'll be able to see a diff of the before/after!

Awesome! I've found a workable solution... that is 58 characters long and has some weird syntax that may be difficult to remember for everyone. It works, but it's not necessarily ideal. Luckily, we can make it even easier.

Using npm scripts

npm scripts are a super handy utility in our toolbelt that makes repetitive tasks and long commands easy. Awesomely, you can use npx inside of npm scripts – meaning you can use any CLI on npm to do work in your project without ever needing to actually install it. This is fantastic for build steps, productivity tools, and (in our case) diffing code.

In my case, I added a diff command to my package.json:

  "scripts": {
    "lint": "standard",
    "diff": "npx prettydiff diff source:\"app.js\" diff:\"app.complete.js\""
  },

Now, instead of needing to write out npx prettydiff diff source:"app.js" diff:"app.complete.js" attendees of my workshop can just type npm run diff and the command will execute 🤗

Fin

I love npx and think npx + npm scripts is a super powerful combination that allows end-users of our JavaScript code to reduce the cognitive load of repetitive commands that aid their workflow. I wanted to share this quick example of how I've used it in hopes of enabling others to focus more on code rather than getting caught up on workflows ✨

If you've got any questions about npx, npm scripts, or anything else I've talked about in this article, please don't hesitate to ask in the comments – more than happy to answer any questions you may have!

Comments

[Bryan Hughes]

Cool article! 💜 npx!

I thought I'd throw out that in certain situations, there is another solution: install the tool as a dev depedency locally and call it from a run script.

npm does this trick that I don't think many folks are aware of: whenever you install a module with a "bin" entry in it's package.json, it installs a link to that bin entry in node_modules/.bin. Then, whenever you call a run script, npm will automatically append node_modules/.bin to your path before executing any commands. The upshot is that you don't have to require your users to do a manual step, you don't have to wait for npx to download the dependency on the fly, and you can control the version of prettydiff like you would for any other dependency.

Your package.json would effectively look like this:

{
  "devDependencies": {
    "prettydiff": "^101.0.0"
  },
  "scripts": {
    "diff": "prettydiff diff source:\"app.js\" diff:\"app.complete.js\""
  }
}

This does assume that you want this dependency tracked in package.json though, which isn't always the case, and I do think npx is a little more flexible in the end. Still a good trick though.

[Tierney Cyren]

I never knew about this! I've seen many projects suggest using this, but was always incredibly frustrated because I could never quite get it to work and always ended up installing globally. My assumption was that the authors were making an assumption that I already have the module globally installed.

Thanks for sharing this, Bryan! 💖

[dAVE Inden]

This is really cool! I didn't know about this. Being able to execute the package without having to download it is pretty nice. Thanks for sharing this!

[dAVE Inden]

Great article! I love npx. I recently setup a new machine and as I was getting stuff installed I realized there are multiple CLIs I would install globally, but now don't have to and can just use npx when I need them. Now, I'm thinking about creating bash aliases to have the commands ready and waiting.

[Tierney Cyren]

I've very much considered doing this as well, but I'm still a total bash newbie and every time I've ended up setting up aliases they somehow get blown away.

I've absolutely considered shipping an additional command to my bitandbang module that just creates a child process and does all the commands I want to run (specifically for project initialization, which actually ended up documenting for the workshop here!).

There's some npm-y things I learned about recently that are launching in the not too distant future that I'm super excited about and am going to use to do this instead.

[dAVE Inden]

I'm not great with bash either. I have to search for everything whenever I need to make adjustments. I've been successful at adding aliases to my .bashrc file, but if it got blown away somehow I'm not sure I'd be able to debug that and would just have to recreate it. Thankfully that hasn't happened. It does remind me that I need to have a better place online to save things like this for later use.
That's awesome there is new features coming to npm to help your process. I'm excited to hear about them when they come out! I really like your workshop too. It's a very cool way to structure things for teaching folks.

[Ben Sinclair]

I think npx is a handy wrapper for giving a one-off demo in your project's README, but it quickly becomes a chore - at a talk the other day someone kept using it instead of installing the apps locally and it added about an extra 30 seconds to every single time they ran the build command. In other words, if you're going to use the command on the regular, don't use npx.

[Tierney Cyren]

I definitely agree that heavier CLIs may not be perfect. Specifically, I've noticed this problem with a lot of "modern" CLIs that have absolutely massive dependency trees to accomplish scaffolding tasks (think create-react-app).

That said, npm still caches the modules so after the first run (outside of situations like CI, where you should be using npm ci) this realistically shouldn't be a problem and now I'm super curious about why it was taking so long to run those commands 🤔

[Ben Sinclair]

I'm guessing that it was because the project was running BackstopJS (a visual regression testing tool) and in order to work cross platform, that runs in a docker container. If the npx command was running inside a container that didn't persist, then it would have to download everything every time.
TBH, I wasn't paying enough attention to that side of things to be able to give you an accurate description. I just assumed that it was npx that didn't cache anything!

[Tierney Cyren]

Ah yeah, a Docker container would have absolutely stripped any benefits there. In that case, you're 100% right that a normal global install would be best.

Ideally, pre-baking that into the image would be the best case scenario... but if they couldn't have done that taking Bryan Hughes advice from his comment on this post and putting it into devDependencies. Combining that with npm ci usage would be great too, since npm ci just reads the predetermined dependencies in the package-lock.json and doesn't take compute time to resolve the dependency structure.