Tool installation and initial configuration - Part 3 of Implementing a RESTful API on AWS

Quick recap: this is a soup to nuts series covering implementing a full-featured REST API on AWS from the perspective of a long-time infrastructure engineer with a historical preference for Java-based solutions.

Table of contents

1. Preliminaries - Unavoidable Drawbacks
   1. Dependency Creep
   2. Disparate Licensing Terms
   3. Goodbye Non-corporate Open-Source, Freeware Build Tools
2. Tool Installation
   1. Node.js - Runtime Engine
   2. NPM - Build Automation + Package Manager
   3. VSCode - (Sorta) Integrated Development Environment
   4. Serverless Framework - Vendor-Agnostic Cloud CLI
3. Conclusion

---

Preliminaries - Unavoidable Drawbacks

I come from a Eclipse/Maven/Apache-based world of:

1. structured development tools;
2. version control;
3. debuggers; and
4. distrust of libraries that pull in dependencies from all over the Internet.

In moving to a Javascript-based solution, we're going to have to accept a number of compromises on these fronts.

Dependency Creep

For example, when it comes to dependencies, Javascript packages have a tendency to create cascading dependency creep, raising the question of: do I REALLY trust not only the library I am incorporating into my code, but also the cascading libraries that it relies on? Moreover, what license terms have been tacked onto any of these various libraries?

This has always been one of the benefits of Java-based development and the various standards-based libraries. I could have some naive-yet-reasonable expectation that anything bundled by Apache/Sun/Oracle/BEA/RedHat/Spring would be covered by specific licensing terms (let's ignore everything after Java 1.8/8) and wouldn't include some chunk of malware pulled in by a random home-brewed regex library.

Unfortunately, we are about to move into a world where incorporating 4-5 libraries results in 150+ random sub-projects being introduced into your code.

For example, one of the major Javascript testing frameworks--Jest--lists 3 runtime library dependencies. These then rely on a number of random dependencies, which rely on more random dependencies, etc., etc., until you find yourself including a package called "path-exists"--which I am sure is a wonderful library, but nevertheless is just one more of the 800+ modules that are eventually pulled into my codebase by my handful of initial dependencies.

There are tools out there to help deal with this, but a new developer should be aware that this is going to be one of the drawbacks of this stack.

Disparate Licensing Terms

Aside from the security implications, each of these is a library you will eventually need to check from a licensing standpoint. Although there are yet again tools to aide you in this task, it can be time-consuming (albeit nowhere near as time-consuming as writing your own code in place of these libraries).

Goodbye Non-corporate Open-Source, Freeware Build Tools

Finally, one last point before moving onto setting up our build environment: we're leaving behind the open-source build environment of the 1980s-2000s and entering the world of corporate-controlled open-source.

For example, the two primary package managers used with Javascript modules are npm and yarn. Although both of these products walk and talk like Apache's Maven, NPM is part of Microsoft's Github subsidiary, while Yarn (like Jest) originates with Facebook. Likewise, when it comes to IDEs, chances are you are going to be using Microsoft Visual Studio--goodbye Eclipse Software Foundation.

Obviously, being corporate projects doesn't necessarily make them bad (see Redhat's Wildfly), but those of us who have been disappointed with the evolution of Java after Oracle's acquisition of Sun should be forgiven for wanting to see their build tools originate from an entity like Apache.

Fortunately, Node.js itself--our back-end runtime environment--is part of the OpenJS Foundation.

---

Tool Installation

All of the following instructions assume you are running MacOS. Also, although I prefer installing most everything inside of a Docker image for security and consistency, we are going to skip that as it adds too much complexity for this tutorial.

---

Node.js - Runtime Engine

First, we'll need to install Node.js. Node.js is the back-end runtime engine for our Javascript code, somewhat like the actual java command.

Check to see if it is installed. We want at least version 12, preferably version 14. In early 2021, AWS claimed it added support for version 14, but as of February 2021, certain features, like nullish coalescing, seemed to be broken. Either way, check to see if it is installed:

> node --version
v15.2.1

If you get:

> node --version
-bash: node: command not found

it is available available here.

---

NPM - Build Automation + Package Manager

In the Java world, we relied on build tools like Maven to handle our builds and package management (or, to really date myself, Ant build.xml files and Make Makefiles).

With Javascript, it is a two horse race between npm and yarn. For all intents and purposes, they are interchangeable. Because I generally avoid Facebook, we will be using npm (and yes, I will later hypocritically recommend using jest over mocha as a testing framework). Also, npm is included with Node.js.

You can verify npm is installed:

> npm --version
v7.0.8

Whereas maven uses POM manifest files, npm and yarn use a JSON-based package.json manifest file to track information about your dependencies and build. As should be self-evident from the filename, Javascript tools tend to use JSON (or YAML) instead of the more structured (and verbose) XML.

Example package.json file:

{
  "name": "my app",
  "version": "1.0.0",
  "description": "This is my app",
  "main": "index.js",
  "scripts": {
    "lint": "tslint -p tsconfig.json -c tslint.json",
    "local": "serverless offline",
    "deploy": "serverless deploy",
    "test": "jest",
    "coverage": "jest --coverage"
  },
  "devDependencies": {
    "@babel/core": "^7.13.10",
    "@babel/preset-env": "^7.13.12",
    "@babel/preset-typescript": "^7.13.0",
    "@shelf/jest-dynamodb": "github:shelfio/jest-dynamodb",
    "@types/aws-lambda": "^8.10.51",
    "@types/aws-sdk": "^2.7.0",
    "@types/jest": "^26.0.21",
    "@types/node": "^14.0.23",
    "@types/uuid": "^8.3.0",
    "babel-jest": "^26.6.3",
    "jest": "^26.6.3",
    "serverless": "^2.30.3",
    "serverless-offline": "^6.8.0",
    "serverless-plugin-typescript": "^1.1.9",
    "typescript": "^3.8.3"
  },
  "dependencies": {
    "dynamoose": "^2.7.3",
    "uuid": "^8.3.2"
  }
}

Moreover, because the dependency creep inherent in Javascript modules, npm and yarn users discovered it can be difficult to ensure the exact versions they are relying on in dependencies of dependencies (package.json only lists your direct dependencies).

To solve this problem, npm and yarn litter your directory with yet another file: package-lock.json. This file endeavors to list the exact version of every dependency in your dependency tree.

The ~30 line package.json file listed above results in a 33,000+ line package-lock.json along the lines of:

{
.
.
.
    "node_modules/@babel/core/node_modules/ms": {
      "version": "2.1.2",
      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz",
      "integrity": "sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==",
      "dev": true
    },
    "node_modules/@babel/core/node_modules/semver": {
      "version": "6.3.0",
      "resolved": "https://registry.npmjs.org/semver/-/semver-6.3.0.tgz",
      "integrity": "sha512-b39TBaTSfV6yBrapU89p5fKekE2m/NwnDocOVruQFS1/veMgdzuPcnOM34M6CwxW8jH/lxEa5rBoDeUwu5HHTw==",
      "dev": true,
      "bin": {
        "semver": "bin/semver.js"
      }
    },
    "node_modules/@babel/generator": {
      "version": "7.13.9",
      "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.13.9.tgz",
      "integrity": "sha512-mHOOmY0Axl/JCTkxTU6Lf5sWOg/v8nUa+Xkt4zMTftX0wqmb6Sh7J8gvcehBw7q0AhrhAR+FDacKjCZ2X8K+Sw==",
      "dev": true,
      "dependencies": {
        "@babel/types": "^7.13.0",
        "jsesc": "^2.5.1",
        "source-map": "^0.5.0"
      }
    },
.
.
.
}

Note: although both yarn and npm use package.json and package-lock.json, it generally advisable to stick to one package manager, rather than switching back and forth between the two.

---

VS Code - (Sorta) Integrated Development Environment

For all its drawbacks, the Eclipse IDE provides a robust enterprise development environment with a variety of following invaluable features.

Some features that I find invaluable

• code writing with syntax highlighting
• function and method code completion
• easy refactoring of classes, methods, and functions
• synchronization with git version control
• unit testing, coverage, and quality control via integration with, e.g., testng and sonarqube
• robust debugging with tracing and breakpoints
• server management tools for, e.g., Spring Boot, WildFly, JBoss, etc.

Unfortunately, in our transition to Javascript and AWS-based serverless, we are going to be jettisoning Eclipse and shifting over to a more watered-down development environment: Visual Studio Code. (Note that there are other options out there, such as Webstorm from JetBrains, but Visual Studio is arguably the most widely used).

Go ahead and grab a copy from: https://code.visualstudio.com. After installation, you should have a screen like this:

We'll get into the details later, but one of the first things you'll notice is that the pulldown menus are not as robust as in Eclipse. Rather, VS Code has gone the way of vi, with a large toolbox of hotkeys and commands hidden from view. A list of various keyboard shortcuts can be found off the menu:

as well as online. The primary keystroke to remember is either FN1 or ↑⌘P to open a command search pulldown that Visual Code calls the "Command Palette" (never mind that this 'palette' is neither a board nor artistic).

For now, we are going to want to install the code shell command so we can launch VSCode from our working directory in terminal. So go ahead and type shell in the "Command Palette" and select it, which will install the command in your path:

In the bottom right corner, you should receive a popup:

If you pop open terminal (⌘-Space then terminal), you should now be able to execute the code command.

One of VSCode's strengths is a robust selection of extensions, which you can access from the 4 "cubes" icon on the left "Activity Bar". I run with the following basic extensions amongst others:

They provide functionality like (a) code cleanup; (b) improved readability; (c) more legible icons in file lists; (d) autocompletion; and (e) code quality checks.

---

Serverless Framework

Next we need to get the tools necessary to start working with Amazon Web Services. As I noted earlier, I like a formal development environment that includes (a) version control; (b) automated testing; (c) debuggers; and (d) consistent deployment. Accordingly, I want my deployments done according to a repeatable script that (1) I can check into my version control system; and (2) can be placed into a debug harness.

Unfortunately, AWS historically was poorly designed from this standpoint. Rather than focus on a command-line driven deployment, application configuration is instead web page driven (an issue now somewhat addressed with the aws CLI).

Even now, much of its documentation and tutorials have the user step through web page after web page of deployment, with the underlying configurations created by these pages often hidden from view. Moreover, it generally lacks the offline lambda function debugger functionality one would want in a true enterprise development environment.

This is where the Serverless Framework comes in. The Serverless Framework provides:

• a level of cloud-server abstraction giving one's code the veneer of avoiding vendor lock-in;
• parameter-based naming and references of resources;
• a robust offline mock server testing environment;
• ability to run attach a debugger to the server; and
• fully scripted deployment of resources

Serverless has both an open-source and a commercial product. We are only going to cover the open-source solution. Serverless also provides a web-based monitoring dashboard, but we won't be using this for the time being.

Serverless includes a relatively decent installation instructions on their website. Keep in mind, however, that it appears to be a smaller company. As such, its tutorials seem to be running a few versions behind and are outdated at times with respect to variable names and template structures.

We start off by grabbing and installing the latest version of Serverless (2.37.0 as of April 2021):

curl -o- -L https://slss.io/install | bash

which will result in modifications to our login scripts and installation of serverless into $HOME/.serverless with output similar to:

Note we can uninstall (assuming we are not using Docker) by running serverless uninstall. As your current shell likely won't pick up these changes, you'll need to launch a new terminal before continuing.

Switch to your desired base workspace directory. Go ahead and run serverless --version:

$ serverless --version
Framework Core: 2.37.0 (standalone)
Plugin: 4.5.3
SDK: 4.2.2
Components: 3.8.3

Conclusion

And that's it for the time being.

Next time, we'll get our AWS account up and running, along with various NPM modules that we will be using for our builds.