DEV Community: Martin McKeaveney

Deno: A Simple Guide

Martin McKeaveney — Sat, 23 May 2020 09:22:59 +0000

On the 13th of May 2020, a very promising step forward was taken in the JavaScript community. The first stable release of Deno, which has been in development for over 2 years, was released. Deno has been touted as a fresh approach to some of the mistakes and issues that occurred in the original development of NodeJS back in 2009.

This blog post aims to cut through a lot of the hype you may see around Deno in an attempt to present the facts - so even a complete beginner to the JavaScript or NodeJS ecosystem can understand Deno and its impact.

What is Deno?

Deno is a JavaScript runtime created by Ryan Dahl, the original creator of NodeJS. It was originally announced by Ryan at JSConf EU 2018, born out of a desire by Dahl to improve and rethink certain aspects of the JavaScript runtime he originally created.

On the deno website, the tagline reads:

Deno is a simple, modern and secure runtime for JavaScript and TypeScript that uses V8 and is built in Rust.

What does this actually mean? Let's break this down, and clear up some of the jargon.

A JavaScript runtime is an environment which contains everything you need to execute a program written in JavaScript. In the case of NodeJS, the runtime is made up of 2 high level concepts:

The V8 JavaScript engine. A JavaScript engine compiles and executes your JavaScript code. V8 is a specific JavaScript engine written in C++ created and maintained by Google and used to power NodeJS, Chrome and Chromium.
APIs and modules. These include modules for interacting with the file system, making HTTP requests, timers etc.

So with this in mind, we are in a much better position to understand Deno and how it is different to Node.

Deno is a runtime that still uses the V8 engine to run JavaScript code. Deno differs because it's written in rust rather than C++. This includes the CLI and all the nice tooling that Deno provides, which come packaged up in a single, executable file. The rust code interfaces with the V8 engine through the rusty_v8 rust package (or crate, as they are known in the rust ecosystem). The rust crate Tokio is used heavily in deno under the hood for asynchronous behaviour and scheduling.

The modules and APIs in Deno are written in TypeScript which is a first class citizen of the runtime. TypeScript code is compiled into JavaScript internally in Deno. This allows developers to write TypeScript and have it compiled and run as if it was just JavaScript, without installing any additional tooling.

Why Deno?

So we now know how the actual runtimes differ. But why? What was wrong with the current one in Node? Why did Ryan Dahl and the rest of the deno team spend 2 years developing this? Ryan Dahl himself goes into detail about this here. He details the regrets he had about building Node in the first place, which laid the groundwork for deno. Ryan announces Deno in this talk, albeit in it's very early stages.

If you don't want to watch the video, or haven't got time. I respect that. Let's summarise the points.

Regret: Not Sticking with Promises

NodeJS is built on the concept of callbacks. Callbacks are functions that are run when a task has finished executing. For example, when you read a file using the fs module:

const fs = require("fs");

fs.readFile("index.html", function(err, data) {
    if (err) {
        console.log("Error Reading File!", err);
    }
    console.log(data);
});

The function passed as the second argument to fs.readFile here is the callback. This function will be run whenever node has finished reading your file and will return either the data (if successful) or an error (if something went wrong). Callbacks have since been eclipsed by promises, which provide a simpler and more powerful API for dealing with asynchronous behaviour in JavaScript.

Later, it became possible to convert functions with callbacks to use promises using the promisify function on the util module in Node. Here's the promisified version of the above example.

const util = require("util");
const fs = require("fs");

const readFilePromise = util.promisify(fs.readFile);

readFilePromise("index.html").then(data => {
    console.log(data);
})
.catch(err => console.error("Error Reading File!", err))

A little verbose, but it does the job. We can further improve this code by using the async/await feature, which aimed to make asynchronous JavaScript code read like synchronous code. Here's the above example rewritten to use async/await.

const util = require("util");
const fs = require("fs");

const readFilePromise = util.promisify(fs.readFile);
async function readFile() {
    try {
        const data = await readFilePromise("index.html");
    } catch (err) {
        console.error("Error reading file!", err);
    }
}

readFile();

Promises were actually added to Node in June 2009, but removed again in February 2010 to keep things simple in the NodeJS codebase. This was a regret of Ryans, as the unified usage of promises would have sped up the delivery and standardisation or async/await.

Deno supports promises out of the box and each of it's asynchronous APIs were written with Promises in mind. Not having to mix and match promises and callbacks provides a simpler, more consistent asynchronous programming experience. Let's see how we perform the same task as above using the deno standard library.

const file = await Deno.open("index.html");

try {   
  await Deno.readFile(file, Deno.stdout);
} catch (err) {
    console.error("Error reading file!", err);
}

file.close();

No callbacks. All the deno async APIs will return a promise to us.

Regret: Security

Node has access to several things by default that could be considered sensitive. This includes the file system, network and system calls. Dahl wishes that he had put more thought and effort into security in NodeJS. The example he gave was running a linter. Linters like ESLint probably should not have full access to your computer and network. Unfortunately, they do.

Deno, however, is secure by default. The runtime will not allow you to access the file system, or perform network calls unless you explicitly give it permissions. We'll get into some examples of this later on.

Regret: The Build System (GYP)

According to Ryan Dahl, the NodeJS build system is his biggest regret.

As much as we'd all like to, we cannot write everything in JavaScript. Sometimes we need much better performance than JavaScript can provide. This is where we need to compile native modules written in lower level languages like C++.

If you are writing a NodeJS module that interfaces to a C++ libary, your code will make use of GYP (Generate Your Projects). GYP and more specifically node-gyp is a tool that is used to compile native nodeJS modules written in C or C++ so they can be accessed with require, like any other JS module.

Node originally used GYP because Google Chrome used to use GYP heavily. The chrome team later switched over to GN. GN aims to be a more readable and maintainable file format than GYP. It also offers significantly better performance.

This left Node in limbo as the sole user of GYP. Deno seeks to solve this by using a mixture of GN and cargo, the build system for Rust. This provides much better build performance and more friendly APIs for compiling native modules.

Regret: Package.json

package.json is the cornerstone of modern JavaScript. It includes the dependencies for your project and tells Node what to install when you run an npm install. NPM is a centralised repository of JavaScript packages and it is where you will download a vast majority of JS packages from. As JavaScript developers, we generally do not go for long without having to interact with NPM in one way or another.

Dahl regrets package.json for several reasons. He does not like the fact that you have to define your dependencies in 2 places - package.json and also inside your code when you require the module. He also regrets the fact that this is not an abstraction that exists on the web, since you can just add a <script /> tag with a URL to your HTML.

package.json also contains a bunch of other information, such as the name, description and repository of your project. Dahl calls this "noise" and believes that it is unneccessary.

Deno looks to solve this by removing the concept of package.json altogether. Deno allows you to require modules directly from URLs, drastically simplifying the whole concept of requiring external modules in your scripts and applications.

Regret: node_modules

So package.json says which modules you need, but the modules themselves are installed in your node_modules directory inside your project. Every project has it's own node_modules folder, requiring you to duplicate dependencies across projects. This concept deviates a lot from the way dependencies work in the browser.

Deno takes a fresh approach to this. node_modules does not exist in Deno. When you require an external module in Deno, it is downloaded and your dependencies are stored in a specific directory defined by you. Your program will not fetch the same modules again however, as deno caches dependencies for you.

Regret: require("module") without extension (".js", ".ts")

In Node, you can require files without the .js extension. This creates the need to query the file system and check what you actually meant when the node module loader is resolving your modules. Ryan argues that this is less explicit and deviates from the way it's done in the browser, which looks something like the following:

<script src="myscript.js" type="text/javascript"></script>

Deno enforces the .js or .ts extension on imports to alleviate this.

Regret: index.js

In Node, if you require a module, index.js is generally used as an "entry" file to a module or application. This is similar to how index.html is resolved by default in a web server. This means you can require a folder like this.

const mymodule = require("./my-cool-module")

Node will look for a index.js file inside the my-cool-module directory by default to resolve the module. The same is true for modules imported from node_modules. Dahl regrets this because it is not explicit, but more practically, it complicated the node module loading system by introducing another implicit step to check for the existence of index.js.

Installing Deno

It's pretty simple to get up and running with deno. On OSX, simply run:

brew install deno

This command should install the single deno executable on your machine. You can check that worked by running:

deno --version

If that didn't work, or you are on another OS, check out the deno installation guide.

Features

Now that we have deno up and running, let's dive into some of the shiny new features.

TypeScript Support

TypeScript is a first class citizen of Deno. This means that we can write our code in TypeScript without having to wory about adding any tooling for our compilation step. When we run TypeScript code with deno, it will type check and compile it for us and run it as if it were normal JavaScript. This completely removes the overhead of setting up the TypeScript compiler in your project.

No NPM - Deno Packaging

Deno takes a fresh and different approach to importing external code. It has no package manager. You can run some external code by simply passing a URL to the deno run command.

deno run https://deno.land/std/examples/welcome.ts

This command will pull down the module from the internet, compile it, cache it indefinitely and run it, all without NPM. If you want to import a module and use it in your code, you don't have to install a module from NPM and require it. Let's say we create the following file, test.ts.

import { assertEquals } from "https://deno.land/std/testing/asserts.ts";

assertEquals(2, 2);

We can then run this with:

deno run test.ts

Our module will resolve modules from the web and run the code. The next time you do this, deno doesn't have to fetch the module again as it is cached. We can run code from any publicly accessible URL. For example, if you want to run some code directly from github, you could use the following command:

deno run https://github.com/shogunpurple/deno-test/blob/master/src/hello.ts

Secure Runtime Permissions

Deno is a secure runtime. This means that it operates on the principle of only giving permissions that you supply explicitly. This differs to NodeJS in that node automatically gives permissions to things like your file system and network. Some things that deno will not let you do without explicitly enabling permissions are:

Read/Write from the File System
Make HTTP requests and access the network
Run subprocesses

To demonstrate this, let's work through an example. We are going to run a script that reads a file on the file system.

$ deno run https://deno.land/std/examples/cat.ts /etc/passwd

We get the following output:

$ deno run https://deno.land/std/examples/cat.ts /etc/passwd  
error: Uncaught PermissionDenied: read access to "/etc/passwd", run again with the --allow-read flag
    at unwrapResponse ($deno$/ops/dispatch_json.ts:43:11)
    at Object.sendAsync ($deno$/ops/dispatch_json.ts:98:10)
    at async Object.open ($deno$/files.ts:37:15)
    at async https://deno.land/std/examples/cat.ts:4:16

Whoops. This happens because deno doesn't allow us to read from the file system unless we specify file reading permissions in our deno run command.

$ deno run --allow-read=/etc https://deno.land/std/examples/cat.ts /etc/passwd

This works! You will now see the contents of the /etc/passwd file in your terminal output. This provides a powerful layer of security that allows us to configure only the exact permissions we want our program to have when it's run. Some other examples of things you may want to give your program permissions for when running are:

--allow-net - allow HTTP requests from your program
--allow-env - allow access to environment variables.
--allow-run - allow your program to run sub-processes.

Most of these flags take arguments, allowing you to restrict access to particular resources. For example, if you only want to allow your program to send HTTP requests to https://myapi.com:

deno run --allow-net=https://myapi.com my-program.ts

Standard Library

The limited standard libary in Node has always been a bit of a sticking point for developers. There is heavy reliance on external modules to perform tasks in Node that are included in the standard library of many other programming languages. UUID generation is an example of this, where the uuid library is the de facto solution for Node developers, however it is not part of the node standard library. Deno provides an extensive standard library based on that of the go programming language that includes some nice functionality such as:

date/time functions
http servers
logging
permissions
testing
uuid generation
websockets
file system utilities (fs)
hashing and cryptography
parsing command line flags

Being able to do this stuff without external modules gives deno the ability to build a myriad of different applications with just the standard library.

Browser Compatible APIs

Also included in the deno standard library are an impressive set of APIs that adhere to web standards and therefore can be run in the browser, allowing code written and compiled with deno to be run on the client as well as the server!

One of the most important ones that deno includes is an implemention of fetch, a browser API used for making HTTP requests. In node, you need to import an external module for this like node-fetch, or use the native http module in node, which is a little clunky and verbose. If we wanted to use fetch to make a call to google.com with deno, we could use the following code:

const response = await fetch("http://www.google.com");

console.log(response);

Also notice how we are able to use a top level await here - another feature deno supports out of the box. Let's run our code above with:

deno run --allow-net fetch_google.ts

The result of our HTTP call to google will be shown in the console. Notice how we specified the allow-net permissions to allow our code to make an HTTP request. Here are some other common browser APIs that deno supports.

addEventListener
removeEventListener
setInterval
clearInterval
dispatchEvent

You can see the full list of web compliant APIs in deno here.

Complying to web standards will make the deno API much more future proof and provides utility for front-end developers.

Deno Tools

As well as the actual language features above, deno gives us additional tooling that performs tasks currently carried out by the likes of webpack, rollup, and prettier. The difference is that deno includes these tools out of the box.

Bundling

Bundling is the process of taking your application and dependencies, and outputting it into a single JavaScript file that can be executed. This job is generally performed by module bundlers such as rollup, webpack and parcel. Deno gives us a simple approach for bundling code with the deno bundle command. If we want to bundle some code, we can do the following with deno.

$ deno bundle https://deno.land/std/examples/echo_server.ts server.bundle.js
Bundling https://deno.land/std/examples/echo_server.ts
Download https://deno.land/std/examples/echo_server.ts
Warning Implicitly using master branch https://deno.land/std/examples/echo_server.ts
Emitting bundle to "server.bundle.js"
2661 bytes emmited.

We can now run our bundle like any other normal script.

$ deno run --allow-net server.bundle.js
Listening on 0.0.0.0:8080

Built in Testing

Deno has a built in test runner that allows us to test our JavaScript and TypeScript code. If you are familiar with JavaScript testing libraries such as Jest or Jasmine, this syntax will look familiar.

Deno.test("deno test", () => {
     const name = "John";
     const surname = "Wick";
     const fullname = `${name} ${surname}`;
   assertEquals(fullname, "John Wick");
});

We use the test functionality on the Deno namespace to create a test. We can then run our tests with the deno test command:

$ deno test test.ts
Compile file:///Users/martinmckeaveney/Development/deno-test/.deno.test.ts
running 1 tests
test deno test ... ok (4ms)

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out (5ms)

This is a powerful feature, as you could just run a full test suite by pointing your deno test command to a test suite hosted somewhere online, without having to pull down and run the test suite yourself.

Script Installation

Packaging your scripts up into a single executable is a very useful thing when you want someone to be able to run your script on their machine without installing Node. Currently if you want to do this with a node script, this can be done with the fantastic pkg module by vercel, which requires you to install that external module through NPM.

Deno provides a built in script installer, allowing you to distribute your scripts as a single executable. Let's see how it works. We will install the basic deno "Hello World" script from the deno.land website as an executable script.

$ deno install https://deno.land/std/examples/welcome.ts
Download https://deno.land/std/examples/welcome.ts
Warning Implicitly using master branch https://deno.land/std/examples/welcome.ts
Compile https://deno.land/std/examples/welcome.ts
✅ Successfully installed welcome
/Users/martinmckeaveney/.deno/bin/welcome
ℹ️  Add /Users/martin/.deno/bin to PATH
    export PATH="/Users/martin/.deno/bin:$PATH"

Scripts are saved by default to the .deno/bin folder located in our home directory. We can execute our script directly:

$ ./.deno/bin/welcome 
Welcome to Deno 🦕

As mentioned in the message above, if we want to execute this script from anywhere on our system, we need to add the .deno directory to our PATH. The PATH variable tells our machine where to look for scripts when we execute them from the terminal. Once we add the .deno directory to our path, we can run the script from anywhere!

$ welcome
Welcome to Deno 🦕

Formatting

Prettier is the de facto formatter for JavaScript code. Deno provides a built in formatter (that actually uses prettier under the hood) through the deno fmt command.

We can format some ugly looking code in this test.ts script by running deno fmt on it.

import { assertEquals } from "https://deno.land/std/testing/asserts.ts";

Deno.test("deno test", () => { const name = "John"; const surname = "Wick"; 
const fullname = `${ name} ${surname}`;
   assertEquals(fullname, "John Wick"); }
    )
  ;

$ deno fmt

Our result is nicely formatted code.

import { assertEquals } from "https://deno.land/std/testing/asserts.ts";

Deno.test("deno test", () => {
  const name = "John";
  const surname = "Wick";
  const fullname = `${name} ${surname}`;
  assertEquals(fullname, "John Wick");
});

deno fmt provides a nice utility by default that can be a little bit of effort to set up in standard JavaScript projects. Some other commands supports by deno fmt include:

deno fmt --check - check if files are already formatted
deno fmt file1.ts - format specific files

Downsides

A radical simplification and completely different approach to a JavaScript runtime is not without its limitations.

Backwards Compatibility with Node

Deno is not the "next version" of node. It is a brand new implementation that is no longer compatible with NPM packages and existing node modules. The https://deno.land/std/node/ API is being built to allow for this, but at the time of writing this is not complete. There are quite a few ports of existing node modules on the deno website that you can use. Check them out on the deno.land website. As Deno matures, this list of libraries ported over for deno support will grow exponentially.

TypeScript Compiler

Deno uses the TypeScript compiler to check types and compile your code into JavaScript. This provides a bottleneck where your code must go through that extra compilation and type checking step before it runs. The TypeScript compiler is written in TypeScript, which means type checking performance in TypeScript will never match the raw performance of native languages like C++ or Rust. There are potential plans to implement the type checking for deno in rust, which is a huge project and won't be ready for quite a long time.

This limitation primarily applies to much larger projects, where the time compiling TypeScript code becomes more of an issue as the codebase grows. For most small to medium projects, this will not be an issue for most developers.

Conclusion

The internet loves dichotomies.

A lot of people want to know if they should drop the thing they are learning to move to the hot new one on an almost daily basis, for fear of being left behind. You can see evidence of this with the regular reddit posts and tweets screaming the title "is dead?!", or the commonly thrown around quip - "You shouldn't use (mature and well established technology), (shiny new thing) is better, because of (feature)".

Deno is not a replacement for NodeJS. There are too many node programs that currently exist, with many more being written every hour of every single day. Deno is a fresh take and new ethos towards building, packaging and distributing scripts and applications in the JS ecosystem, built on modern technologies with a particular focus on providing a powerful scripting environment with the tooling you need built in.

Deno still has a long way to go. In order for Deno to become a success, the community must use it and build modules for it. Deno must find it's place in the daily workflow of JavaScript developers.

One of the problems up to this point with modern JavaScript is the heavy reliance on tooling. Although JS tooling has opened doors that were never thought possible in the past, it can be overwhelming for beginners and even experienced developers who aren't familiar with the JS ecosystem. Deno hides a lot of this away and makes those decisions for you. This is a huge plus for a lot of engineers.

I hope you learned something. Thanks for reading!

Feel free to reach out to me or follow me on Twitter, where I tweet and blog about JavaScript, Python, AWS, automation and no-code development.

ES2020: Everything You Need to Know

Martin McKeaveney — Tue, 05 May 2020 11:10:14 +0000

Another Year

Another year has passed, and we now know the latest features that have been finalised to go into ES2020, the latest specification of JavaScript/ECMAScript. The finalised proposals can be found here on the TC39 proposals GitHub repository. You may be wondering why or how these proposals are put together, and the journey of a JavaScript proposal before it goes into the language itself. Well, you are in luck.

How Features are added to JavaScript

No, it's not Brendan Eich, or Google, or some supreme being. The JavaScript specification is managed and iterated by a committee called TC39 (Technical Committee 39). TC39 is made up of various developers, people from academia and platform enthusiasts.

TC39 meet around 6 times a year, mostly in the US but also in Europe. They work with the community to accept proposals for new JavaScript features and take them through the 4 "stages" of a JavaScript language proposal. The 4 stages are as follows:

Stage 0: Strawperson

You have made a proposal to the language. This is generally done by raising a PR against the [TC39 ECMAScript GitHub repository](https://github.com/tc39/ecma262).

Stage 1: Proposal

A TC39 member has been identified as a "champion" who is on board with the idea of introducing your feature. A public GitHub repository is created around your proposal, with examples, high-level API, rationale and potential issues.

Stage 2: Draft

Things are starting to get real. The Draft stage now means that all of the syntax and semantics of your proposal need to be nailed down. This involves describing your proposal feature using the formal specification language you will see in the JavaScript spec itself.

Stage 3: Candidate

Stage 3 is when your proposal is pretty much ready to go - TC39 reviewers have signed off on the specification text. Responsibility falls to the community at this stage. Developers are expected to use the feature and provide feedback that is only possible through actually using it in their software. The only changes made to your proposal here will be critical bugs and problems that are uncovered by the community.

Stage 4: Finished

Done and dusted. Your proposal has been well tested in the community in real implementations. Your proposal will be included in the next version of the ECMAScript standard and will be used by millions.

In this blog post, we are going to dive into each of the confirmed, stage 4 features that are being added to ES2020, as well as some examples.

String.prototype.matchAll

String.prototype.matchAll is a utility function that's used to get all the matches for a particular regular expression (including capturing groups, which will be explained later). How was this problem being solved before ES2020? Let's take a simple example and iterate. Say we have the following string:

const test = "climbing, oranges, jumping, flying, carrot";

We want to get the verb from each of the words that end with ing. For example climb from "climbing" and jump from "jumping". This is a perfect job for regex. Lets walk through the steps.

Search the string for any words that end in "ing" (eg. "climbing")
Capture all the letters that come before "ing" in the word (eg. "climb")
Return them

Okay, so to do that we might use the following regex.

const regex = /([a-z]*)ing/g;

Regex is hard. Let's break it down so we understand how this works.

([a-z]*) - match any string which contains the letters a-z in a row. We wrap this in parens () to make it a "capturing group". A capturing group is as the name suggests - we want to "capture" the group of characters that match this particular part. In our example, we want to match all words ending in "ing", but what we really want are the letters before that, hence using a capturing group.
ing - only match strings that end in "ing"
/g - global. Search the whole input string. Don't stop at the first match.

`String.prototype.match`

Let's use our regex to find our verbs. One option in JavaScript is to use the match function on a string, which lets you pass in a regex expression.

const test = "climbing, oranges, jumping, flying, carrot";
const regex = /([a-z]*)ing/g;

test.match(regex);

// ["climbing", "jumping", "flying"]

No good. It's returning the full words rather than just the verbs! This is happening because match does not support capturing groups with the /g flag - meaning that we can't use this to solve our problem. match would be fine if we didn't need to use capturing groups, but in this case we do. Let's try something else.

`RegExp.prototype.exec`

The exec method is executed on a regex itself, rather than the string like match. exec has support for capturing groups, but is a slightly more clunky API to use. You must keep calling exec over and over on the regex to get the next match. This requires us to create an infinite loop and continue calling exec until there are no matches left.

const regex = /([a-z]*)ing/g;

const matches = [];

while (true) {
  const match = regex.exec(test);
  if (match === null) break;
  matches.push(match[1]);
}

matches
// ["climb", "jump", "fly"]

This approach works fine, but it is a little confusing and unintuitive. There are two primary reasons for this:

It only performs the intended operation if the /g flag is set at the end. This can get a little confusing if you are passing the regex around as a variable or parameter.
When using the /g flag, RegExp objects are stateful and store a reference to their last match. This can cause strange bugs if you are re-using the same regex over and over with multiple calls to exec().

Using String.prototype.matchAll

Finally - we have arrived. (If you skipped to this part, I don't judge you.) String.prototype.matchAll will make our lives much easier here and provide a simple solution that supports capturing groups, returning an iterable that we can spread into an array. Let's refactor our code above to use matchAll.

const test = "climbing, oranges, jumping, flying, carrot";

const regex = /([a-z]*)ing/g;

const matches = [...test.matchAll(regex)];

const result = matches.map(match => match[1]);

result

// ["climb", "jump", "fly"]

We get a 2 dimensional array with the full word match in the first element ("climbing") as well as the captured group in the second element ("climb"). By iterating over and pulling out the second element, we get the results we want. Great!

Dynamic import()

This is one you may be already familiar with. It's been supported by webpack since v2 and is common in production JavaScript applications for "code splitting". Code splitting is very powerful in a single page application - in a lot of cases speeding up initial page load times significantly.

The dynamic import syntax allows us to call import as a function that returns a promise. This becomes useful for dynamically loading modules at runtime. For example, you may want to load a certain component or module based on some logic in your code.

// JavaScript for side panel is loaded
  const sidePanel = await import("components/SidePanel");
  sidePanel.open();

Interpolation is also supported.

async function openSidePanel(type = "desktop") {
    // JavaScript for desktop side panel is loaded
    const sidePanel = await import(`components/${type}/SidePanel`);
    sidePanel.open();
}

This feature enhances the performance of our applications. We don't have to load all the JavaScript up front. Dynamic imports give us the control to only load exactly as much JS as we need to.

BigInt

The largest number JavaScript can handle is 2^53. That's 9007199254740991, or you can use the slightly more memorable Number.MAX_SAFE_INTEGER.

What happens when you go beyond MAX_SAFE_INTEGER? Well, it's not so SAFE anymore.

console.log(Number.MAX_SAFE_INTEGER); // 9007199254740991
console.log(Number.MAX_SAFE_INTEGER + 1); // 9007199254740992
console.log(Number.MAX_SAFE_INTEGER + 2); // 9007199254740992 - wut
console.log(Number.MAX_SAFE_INTEGER + 3); // 9007199254740994 - WUT

The BigInt type in ES2020 solves this. To convert a number literal to a BigInt, you can use the BigInt constructor, or you simply add a n to the end of it. So to fix our example above where we got the same value after adding 2 to Number.MAX_SAFE_INTEGER:

BigInt(Number.MAX_SAFE_INTEGER) + 2n; // 9007199254740993n ✅

Who needs these numbers?

You might be surprised to hear that it's quite common to have numbers this large in software development. Timestamps and unique identifiers can be numbers this large.

For example, Twitter use integers this large as unique keys for tweets. You would see weird bugs in your JavaScript application if you tried to store these as numbers without BigInt. You would have to use a community package, or store them as a string instead - which is a common solution that JavaScript developers were using to solve this problem in environments where BigInt isn't supported.

Promise.allSettled

Let's say you are sitting an exam. When you receive your results, you find out you got 99% of the questions correct. In most walks of life, you would have passed with flying colours. In this case, though - you receive a big red stamp on your results letter telling you that you failed.

This is how Promise.all works. Promise.all takes an array of promises, and concurrently fetches their results. If they all succeed, your Promise.all succeeds. If one or more fail, your promise rejects. In some cases you may want this behaviour - but not always.

Enter Promise.allSettled

Promise.allSettled of ES2020 is much kinder when it comes to your exam. It will give you a pat on the back and tell you not to worry about that 1% of promises that failed.

A promise is regarded as "settled" when it comes back - pass or fail. Promise.allSettled allows us to pass an array of promises and it will resolve when they are all settled. The return value of the promise is the array of results. Let's look at an example.

const promises = [
  fetch('/api1'),
  fetch('/api2'),
  fetch('/api3'),
];

Promise.allSettled(promises).
  then((results) => results.forEach((result) => console.log(result.status)));

// "fulfilled"
// "fulfilled"
// "rejected"

globalThis

We live in a world where "universal" JavaScript is common. This means that the same JavaScript code could be running on the client and on the server in NodeJS. This presents a particular set of challenges.

One is the global object, accessible from any piece of running code. This is window in the browser, but global in Node. Writing universal code that accesses this global object relies on some conditional logic, that may look something like this (cover your eyes).

(typeof window !== "undefined"
? window
: (typeof process === 'object' &&
   typeof require === 'function' &&
   typeof global === 'object')
    ? global
    : this);

Thankfully, ES2020 brings with it the addition of the globalThis global variable. This will do the heavy lifting above for you and means you can now relax when it comes to accessing window or global in either front-end or back-end code.

globalThis.something = "Hello"; // Works in Browser and Node.

for-in mechanics

for (x in obj) ... is a super useful syntax for many things - mainly iterating over the keys of an object.

for (let key in obj) {
  console.log(key);                      
}

This proposal is related to the order and semantics of which elements are iterated in a for..in loop. Before this proposal, most JavaScript engines had already applied common sense - currently all major browsers loop over the properties of an object in the order in which they were defined. There were some nuances, however. These mainly involved more advanced features like proxies. for..in loop semantics have historically been left out the JavaScript spec, but this proposal ensures everyone has a consistent point of reference as to how for..in should work.

Optional Chaining

Optional chaining is probably one of the most highly anticipated features to come to JavaScript in quite some time. In terms of impact on cleaner JavaScript code, this one scores very highly.

When checking for a property deep inside a nested object, you often have to check for the existence of intermediate objects. Let's work through an example.

name: "foo",
  age: 25,
  address: {
    number: 44,
    street: "Sesame Street",
    city: {
      name: "Fake City",
      lat: 40,
      lon: 74
    }
  }
}

// when we want to check for the name of the city
if (test.address.city.name) {
  console.log("City name exists!");
}

// City Name exists!

This works fine! But in software, we can't always rely on the happy path. Sometimes intermediate values will not exist. Let's look at the same example, but with no city value defined.

const test = {
  name: "foo",
  age: 25,
  address: {
    number: 44,
    street: "Sesame Street"
  }
}

if (test.address.city.name) {
  console.log("City name exists!");
}

// TypeError: Cannot read property 'name' of undefined

Our code is broken. This happens because we are trying to access name on test.address.city which is undefined. When you attempt to read a property on undefined, the above TypeError will be thrown. How do we fix this? In a lot of JavaScript code, you will see the following solution.

const test = {
  name: "foo",
  age: 25,
  address: {
    number: 44,
    street: "Sesame Street"
  },

}

if (test.address && test.address.city && test.address.city.name) {
  console.log("City name exists!");
}

// no TypeError thrown!

Our code now runs, but we've had to write quite a bit of code there to solve the problem. We can do better. The Optional Chaining operator of ES2020 allows you to check if a value exists deep inside an object using the new ? syntax. Here is the above example rewritten using the optional chaining operator.

const test = {
  name: "foo",
  age: 25,
  address: {
    number: 44,
    street: "Sesame Street"
  },

}

// much cleaner.
if (test?.address?.city?.name) {
  console.log("City name exists!");
}

// no TypeError thrown!

Looking good. We have condensed the long && chains into our much more succinct and readable optional chaining operator. If any of the values along the chain are null or undefined, the expression simply returns undefined.

The optional chaining operator is very powerful. Have a look at the following examples to see other ways it can be used.

const nestedProp = obj?.['prop' + 'Name']; // computed properties

const result = obj.customMethod?.(); // functions

const arrayItem = arr?.[42]; // arrays

Nullish Coalescing Operator (`null` or `undefined`)

The Nullish Coalescing Operator is a very fancy sounding name for something very simple. What this feature gives us is the ability to check if a value is null or undefined and default to another value if so - nothing more, nothing less.

Why is this useful? Let's take a step back. There are five "falsy" values in JavaScript.

null
undefined
empty string ("")
0
Not a number -NaN

We may have some code where we want to check against a numeric value. We want to assign a squad number to players in a team. If they already have a squad number, we keep that. Otherwise, we give them the value of "unassigned".

const person = {
  name: "John",
  age: 20,
  squadNumber: 100
};

const squadNumber = person.squadNumber || "unassigned";

console.log(`${person.name}s squad number is ${squadNumber}`);


// "Johns squad number is 100"

This code works fine. However let's think about this from a slightly different angle. What if our person had a slightly obscure squad number, like zero?

const person = {
  name: "Dave",
  age: 30,
  squadNumber: 0
};

const squadNumber = person.squadNumber || "unassigned";

console.log(`${person.name}s squad number is ${squadNumber}`);


// "Daves squad number is unassigned"

This isn't right. Dave has been playing for the team for years. Our code has a bug. This happens because 0 is falsy, causing the false condition of our || to be invoked. This example is where the standard check for a value can fall short. You can of course, solve this by doing the following:

const person = {
  name: "Dave",
  age: 30,
  squadNumber: 0
};

const squadNumber = person.squadNumber >= 0 ? person.squadNumber : "unassigned";

console.log(`${person.name}s squad number is ${squadNumber}`);


// "Daves squad number is 0"

Not a bad solution - but we can do better using the Nullish Coalescing operator (??) to ensure that our value is null or undefined, before we say someones squad number is unassigned.

const person = {
  name: "Dave",
  age: 30,
  squadNumber: 0
};

// Nullish Coalescing Operator
// If person.squadNumber is null or undefined
// set squadNumber to unassigned
const squadNumber = person.squadNumber ?? "unassigned";

console.log(`${person.name}s squad number is ${squadNumber}`);


// "Daves squad number is 0"

Nothing wrong with that little bit more type safety and explicitness in our JavaScript.

import.meta

import.meta is a convenience property that provides an object containing the base URL of the currently running module. If you are familiar with node, this functionality is available out of the box with CommonJS through the __dirname or __filename properties.

const fs = require("fs");
const path = require("path");
// resolves data.bin relative to the directory of this module
const bytes = fs.readFileSync(path.resolve(__dirname, "data.bin"));

What about the browser though? This is where import.meta becomes useful. If you want to import a relative path from a JavaScript module running in the browser, you can use import.meta to do so.

// Will import cool-image relative to where this module is running.
const response = await fetch(new URL("../cool-image.jpg", import.meta.url));

This feature is very useful for library authors - as they do not know how and where you will run your code.

Conclusion

All in all, the newest features added to the ECMAScript specification add even more utility, flexibility and power to the constantly evolving and developing ecosystem that is JavaScript. It's encouraging and exciting to see the community continue to thrive and improve at such a rapid pace.

You may be thinking - "That all sounds great.. but how do I get started using ES2020 features?"

When/how can I use this stuff?

You can use it now! In the later versions of most modern browsers and Node, you will have support for all of these features. caniuse.com is another great resource to check levels of compatibility for ES2020 features across browsers and node.

If you need to use these features on older browsers or versions of node, you will need babel/typescript.

Enabling ES2020 support

Using Babel 7.8.0 or TypeScript 3.7

Babel 7.8.0 and above as well as TypeScript 3.7 and above support ES2020 features out of the box.

Using babel plugins

If you can't upgrade your babel setup, you will have to install the ES2020 features through a multitude of plugins. Here's an example .babelrc config of how you may do that.

{
  "plugins": [
    "@babel/plugin-proposal-nullish-coalescing-operator",
    "@babel/plugin-proposal-optional-chaining",
    "@babel/plugin-proposal-class-properties",
    "@babel/plugin-syntax-bigint"
  ]
}

Feel free to reach out to me or follow me on Twitter, where I tweet and blog about JavaScript, Python, AWS, automation and no-code development.

I hope you learned something. Thanks for reading!

Convert a Website into an API: Building a Serverless Web Scraper with the AWS Cloud Development Kit

Martin McKeaveney — Sun, 26 Apr 2020 10:49:44 +0000

Imagine you are looking at a website and you see some really nice data that you would love to use to build an app or integration for. After furiously googling for an API providing this data, you come up short. This is when you need to make use of a web scraper.

A web scraper is an application or program that fetches HTML from a web page, parses it, and stores the data in a human readable format for future reference.

Web scrapers are very useful for things like automation, social media scraping, statistics and monitoring for changes on a web page such as prices or products.

Today, I’m going to show you how to convert a website into a serverless API using NodeJS running in Amazon Web Services. We’re going to scrape the UFC Stat Leaders Site, which contains statistics for the top 10 fighters of all time in a whole host of different categories.

We want to scrape the results for each fighter in every section and set up a REST API that allows us to query for a fighter by their name. This will return the list of statistics for which that fighter is in the top 10 of all time, including:

The statistic (eg. Total fights, wins)
The value for that statistic (eg. 12)
The position in the leaderboard they sit (eg. 2)

Let’s get into it.

How?

At a high level, here are the main problems we need to solve to complete our web scraper.

Trigger a scrape the UFC website by performing an HTTP request
Store the data from the scrape in a database
Provide a way to query data out of the database via an HTTP request
Deploy everything above to the cloud in a simple, self-contained way.

AWS Cloud Development Kit

In order to perform the above steps, we are going to make use of the AWS Cloud Development Kit.

The AWS Cloud Development Kit (AWS CDK) is an open source software development framework to model and provision your cloud application resources using familiar programming languages.

The AWS CDK lets us describe all of our cloud resources using code, and deploy them to AWS easily. This is useful for developers because it allows us to group our resources together. This enables us to tear down or update everything related to the scraper (database, API) at once, reducing the chance of potential mistakes. Another benefit of the CDK is the ability to version control your infrastructure and roll back to an old version if something goes wrong. This makes it a powerful tool for teams.

Creating your AWS Account and IAM User

If you haven’t already got an AWS account, you will need to create one and configure your security credentials.

Getting Started with the AWS CDK

Now we are all set up with our AWS credentials, we can get set up with the CDK. The AWS CDK is developed at Amazon with TypeScript, so we can install it globally from NPM.

npm install -g aws-cdk

You can then create a new AWS CDK project by invoking cdk init in an empty directory. Since we will be writing our Lambdas in JavaScript, we’ll use the JavaScript CDK language option as well. The AWS CDK also supports several other languages such as Python, Java and C#. Let’s run the following to set up our project.

mkdir aws-cdk-scraper
cd aws-cdk-scraper
cdk init app —language javascript
cdk bootstrap

That’s it! The CDK will create some files and directories for us, as well as set up the relevant config in AWS and we are ready to start building. We’ll come back to the CDK whenever we are ready to start deploying our resources in the cloud, but now it’s time to..

Design our Scraper

When programming any web scraper, it is imperative that you know the structure of the HTML of the webpage that you are trying to scrape. You can see this by using the developer tools in your preferred browser.

If you look at http://statleaders.ufc.com/ you will see it looks like this:

We want to scrape each section of this site and populate our database with the data inside each section. Let’s break it down:

Get the title of the section (Total Fights, Wins, etc)
- Iterate through the top 10 fighters for that section
- Write each fighter into the database with their name, statistic, leaderboard position, and value for that statistic. For example:

{ name: “Donald Cerrone”, statistic: “Wins”, value: 23, position: 1 }

We know our steps. Let’s write our scraper!

Coding the Scraper

In order to build our scraper, we will need to install some modules from NPM.

cheerio

Cheerio is a library that parses markup and exposes a simple, jQuery-like API. It’s a very popular choice for web scraping with JavaScript. We will be using it to parse the HTML we fetch from the UFC website.

node-fetch

We need to make an HTTP GET request to fetch the HTML from the UFC website so we can parse it. Feel free to use your library of preference here, or even the native node http module. We will be using node-fetch in this tutorial for our code examples.

We will store our scraper code in a src directory in the root of our project. Let’s create that and install the modules above. We need to install these modules in the src directory so that they will be available in the AWS Lambda environment when we deploy our code to the cloud.

mkdir src
cd src
npm init -y
npm i —-save cheerio node-fetch

Next up, we can create a scraper by defining the src/scraper.js file with the code below. Please take some time to read the comments and understand the control flow.

// src/scraper.js
const cheerio = require("cheerio");
const fetch = require("node-fetch");

const TABLE_NAME = process.env.TABLE_NAME;
const SCAPER_URL = "http://statleaders.ufc.com/";

exports.scrape = async function (event, context) {
    // fetch the HTML from the UFC website
  const response = await fetch(SCAPER_URL);
  const html = await response.text();
  const $ = cheerio.load(html);

    // get each section
  const resultsGroup = $(".results-group");

  const fighterStatistics = [];

     // iterate over the sections
  resultsGroup.each(function (i, result) {
      // get the name of the statistic (“Total Fights”, etc)
    const statistic = $(result).find("header > h3").text();

      // find each row in the table for that statistic
    $(result)
      .find(".results-table--tr")
      .each(function (i, row) {
        const result = $(row)
          .find("span")
          .map((i, el) => $(el).text())
          .get();

          // get the data from each table row
        const [position, name, value] = result;
        const isHeaderRow = result.length > 3;

        if (!isHeaderRow) {
          fighterStatistics.push({
            name,
            statistic,
            value,
            position
          });
        }
      });
  });

    // let’s print the results for debugging purposes
  fighterStatistics.forEach(console.log);
};

Let’s test it out.

node src/scraper

Looking good! The scraper has scraped the UFC stats website and printed out a list of our results.

Deploying Our Scraper to AWS

Our scraper is now working locally, but we want to get it up and running in the cloud. As mentioned before, we are using the AWS CDK to create the cloud resources we need for our scraper. Let’s dive in.

Creating a Serverless App with the CDK

The scraper will be composed of several components and AWS services:

AWS Lambda functions for the scraper and our API written in NodeJS
An API to expose our lambda functions over HTTP using AWS API Gateway
A DynamoDB database to store and query our data

Let’s start by installing all the necessary dependencies in the root of our project so we can create these resources with the AWS CDK.

npm install @aws-cdk/aws-apigateway @aws-cdk/aws-lambda @aws-cdk/aws-s3 @aws-cdk/aws-dynamodb

Creating Our DynamoDB Table and Serverless REST API

We can now create our dynamoDB table. Update /lib/aws-cdk-scraper-stack.js with the following code.

const cdk = require("@aws-cdk/core");
const dynamodb = require("@aws-cdk/aws-dynamodb");
const lambda = require("@aws-cdk/aws-lambda");
const apiGateway = require("@aws-cdk/aws-apigateway");

class AwsCdkScraperStack extends cdk.Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create our DynamoDB table
    const fighterStatsTable = new dynamodb.Table(this, "FighterStatsTable", {
      partitionKey: {
        name: "name",
        type: dynamodb.AttributeType.STRING
      },
      sortKey: {
        name: "statistic",
        type: dynamodb.AttributeType.STRING
      },
    });

    // Create our lambda
    const scraperLambda = new lambda.Function(this, "scraper", {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: lambda.Code.asset("src"),
      handler: "scraper.scrape",
      timeout: cdk.Duration.seconds(20),
      environment: {
        TABLE_NAME: fighterStatsTable.tableName,
      },
    });

    // Create API Gateway API
    const api = new apiGateway.RestApi(this, "ufc-scraper-api", {
      restApiName: "UFC Stats Scraper",
      description: "UFC Stats Scraper API.",
    });

    // call our lambda when someone makes a GET request to /scrape
    const scrape = api.root.addResource("scrape");
    const scraperIntegration = new apiGateway.LambdaIntegration(scraperLambda);
    scrape.addMethod("GET", scraperIntegration);

    // allow the scraper lambda to write to dynamoDB
    fighterStatsTable.grantWriteData(scraperLambda);
  }
}

module.exports = { AwsCdkScraperStack };

As with all software development, it’s usually a good idea to test incrementally. Let’s deploy these resources to the cloud and test that our scraper and shiny new REST API are working by sending an HTTP request to the /scrape endpoint we’ve just created. Deploying with the CDK couldn’t be simpler - just run

cdk deploy

AWS will do its thing - creating your Lambda, DynamoDB table, and API Gateway REST API. When done, you should receive some output in your console with the URL for your new API! Copy the URL to your clipboard.

Before we move on - let’s test our new scraper API with curl.

curl -H ‘Content-Type: application/json’ -H ‘Accept: application/json’ <your-api-url>

You should now see the results of the scrape in your terminal. 🥳

Interacting with DynamoDB

Okay, our scraper is working, but it isn’t really that useful yet. We need to store the data from our scrapes in our database, and provide a way to query this data.

We will be using dynamoDB to store each of the fighters statistics keyed by their name so we can fetch all of the areas where a fighter ranks in the UFC top ten by passing their name to our API. This is not recommended in production apps because a persons name is not always unique.

Writing the Scrape Result into Dynamo

To write the array of results from our scrape into DynamoDB, we must use the AWS SDK for node. We are going to use the DynamoDB DocumentClient which is a nice wrapper for interacting with Dynamo.

We will need to update src/scraper.js to write our scrape results to the FighterStatsTable in DynamoDB.

const AWS = require(“aws-sdk”); 
const cheerio = require(“cheerio”); 
const fetch = require(“node-fetch”);

const TABLE_NAME = process.env.TABLE_NAME;
const SCAPER_URL = “http://statleaders.ufc.com”;

const client = new AWS.DynamoDB.DocumentClient();

exports.scrape = async function (event, context) {
  // fetch the HTML from the UFC website
  const response = await fetch(SCAPER_URL);
  const html = await response.text();
  const $ = cheerio.load(html);

  // get each section
  const resultsGroup = $(“.results-group”);

  const fighterStatistics = [];

  // iterate over the sections
  resultsGroup.each(function (i, result) {
    const statistic = $(result).find(“header > h3”).text();


    // find each row in the table for that statistic
    $(result)
      .find(“.results-table—tr”)
      .each(function (i, row) {
        const result = $(row)
          .find("span")
          .map((I, el) => $(el).text())
          .get();

        // get the data from each table row
        const [position, name, value] = result;
        const isHeaderRow = result.length > 3;

        if (!isHeaderRow) {
          fighterStatistics.push({
            name,
            statistic,
            value,
            position
          });
        }
      });
  });


  try {
    // Write all the fighter statistics into DynamoDB
    await Promise.all(fighterStatistics.map((result) =>
      client
        .put({
          TableName: TABLE_NAME,
          Item: result,
        })
        .promise()
    ));

    return {
      statusCode: 200,
      headers: {
        “Content-Type”: “application/json”,
      },
      body: JSON.stringify({
        results: fighterStatistics,
      }),
    };
  } catch (error) {
    return {
      statusCode: 500,
      body: JSON.stringify(error),
    };
  }
};

Let’s test this out.

curl -H ‘Content-Type: application/json’ -H ‘Accept: application/json’ <your-api-url>

If you check your DynamoDB console in AWS , you should be able to see the results from the updated scraper which is now successfully persisting to our database!

Querying DynamoDB

Now that we have results in Dynamo, we now want to query these results from our Fighter Statistics Table. Since we've used a composite key, this enables us to query using a fighters name, and get back all the statistics for which they are in the top 10 of all time in the UFC. Let’s start by creating a lambda at src/queryStats.js that will query the results from Dynamo.

const AWS = require("aws-sdk");

const TABLE_NAME = process.env.TABLE_NAME;

const client = new AWS.DynamoDB.DocumentClient();

exports.query = async function (event) {
  if (event.body) {
    const body = JSON.parse(event.body);
    const result = await client
      .query({
        TableName: TABLE_NAME,
        ExpressionAttributeNames: { "#column": "name" },
        KeyConditionExpression: "#column = :name",
        ExpressionAttributeValues: { ":name": body.name, },
      })
      .promise();

    return {
      statusCode: 200,
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(result.Items),
    };
  } else {
    return {
      statusCode: 400,
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        message: "No Fighter Name Provided.",
      }),
    };
  }
};

Creating our new API in AWS

Now that we've written the code to actually query the database, we need to deploy it to AWS and put it behind API Gateway so that we can send HTTP requests to it. Update your lib/aws-cdk-scraper-stack.js to create our lambda and attach it to API Gateway at the /fighter path, meaning we can POST to that endpoint to query our data in Dynamo.

const cdk = require("@aws-cdk/core");
const dynamodb = require("@aws-cdk/aws-dynamodb");
const lambda = require("@aws-cdk/aws-lambda");
const apiGateway = require("@aws-cdk/aws-apigateway");

class AwsCdkScraperStack extends cdk.Stack {
  /**
   *
   * @param {cdk.Construct} scope
   * @param {string} id
   * @param {cdk.StackProps=} props
   */
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create our DynamoDB table
    const fighterStatsTable = new dynamodb.Table(this, "FighterStatsTable", {
      partitionKey: {
        name: "name",
        type: dynamodb.AttributeType.STRING,
      },
      sortKey: {
        name: "statistic",
        type: dynamodb.AttributeType.STRING,
      },
    });

    // Create our lambda
    const scraperLambda = new lambda.Function(this, "scraper", {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: lambda.Code.asset("src"),
      handler: "scraper.scrape",
      timeout: cdk.Duration.seconds(20),
      environment: {
        TABLE_NAME: fighterStatsTable.tableName,
      },
    });

    const fighterApiLambda = new lambda.Function(this, "fighterApi", {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: lambda.Code.asset("src"),
      handler: "queryStats.query",
      environment: {
        TABLE_NAME: fighterStatsTable.tableName,
      },
    });

    // Create API Gateway API
    const api = new apiGateway.RestApi(this, "ufc-scraper-api", {
      restApiName: "UFC Stats Scraper",
      description: "UFC Stats Scraper API.",
    });

    // call our lambda when someone makes a GET request to /scrape
    const scrape = api.root.addResource("scrape");
    const scraperIntegration = new apiGateway.LambdaIntegration(scraperLambda);
    scrape.addMethod("GET", scraperIntegration);

    const fighterApi = api.root.addResource("fighter");
    const apiIntegration = new apiGateway.LambdaIntegration(fighterApiLambda);
    fighterApi.addMethod("POST", apiIntegration);

    // allow the scraper lambda to write to dynamoDB
    fighterStatsTable.grantWriteData(scraperLambda);
    // allow the api lambda to read from dynamoDB
    fighterStatsTable.grantReadData(fighterApiLambda);
  }
}

module.exports = { AwsCdkScraperStack };

Now run cdk deploy again to deploy this latest stack.

Querying our API

We can query our new API using an HTTP POST request, with a request body containing the name of the fighter we want to query for. For example, if you wanted to see all the statistics where Jon Jones ranks in the top 10 of all time, you would perform the following curl request.

curl <your-api-url>/fighter -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{ "name": "Jon Jones" }'

You should get back the following results:

[
    {
        "value": "0.08%",
        "statistic": "Bottom Position Percentage",
        "name": "Jon Jones",
        "position": "3"
    },
    {
        "value": "10",
        "statistic": "Decision Wins",
        "name": "Jon Jones",
        "position": "4"
    },
    {
        "value": "1463",
        "statistic": "Sig Strikes Landed",
        "name": "Jon Jones",
        "position": "6"
    },
    {
        "value": "95.0%",
        "statistic": "Takedown Defense",
        "name": "Jon Jones",
        "position": "2"
    },
    {
        "value": "14",
        "statistic": "Title Fight Wins",
        "name": "Jon Jones",
        "position": "1"
    },
    {
        "value": "5:40:15",
        "statistic": "Total Fight Time",
        "name": "Jon Jones",
        "position": "9"
    },
    {
        "value": "13",
        "statistic": "Win Streak",
        "name": "Jon Jones",
        "position": "2"
    },
    {
        "value": "20",
        "statistic": "Wins",
        "name": "Jon Jones",
        "position": "3"
    }
]

Nice one! Looks like everything is working as expected 😃

Wrapping Up

We scraped the UFC website in this particular case, but these concepts are applicable to any web scraping you may want to do. Feel free to check out the full code for this tutorial here on GitHub or reach out to me on Twitter.

Thanks for reading!

clinews - read the news from the CLI!

Martin McKeaveney — Tue, 18 Sep 2018 13:44:00 +0000

A CLI for reading the news and getting the latest headlines including search functionality. Supports over 70 sources. Check it out on Github!

DEV Community: Martin McKeaveney

Deno: A Simple Guide

What is Deno?

Why Deno?

Regret: Not Sticking with Promises

Regret: Security

Regret: The Build System (GYP)

Regret: Package.json

Regret: node_modules

Regret: require("module") without extension (".js", ".ts")

Regret: index.js

Installing Deno

Features

TypeScript Support

No NPM - Deno Packaging

Secure Runtime Permissions

Standard Library

Browser Compatible APIs

Deno Tools

Bundling

Built in Testing

Script Installation

Formatting

Downsides

Backwards Compatibility with Node

TypeScript Compiler

Conclusion

ES2020: Everything You Need to Know

Another Year

How Features are added to JavaScript

String.prototype.matchAll

String.prototype.match

RegExp.prototype.exec

Using String.prototype.matchAll

Dynamic import()

BigInt

Who needs these numbers?

Promise.allSettled

Enter Promise.allSettled

globalThis

for-in mechanics

Optional Chaining

Nullish Coalescing Operator (null or undefined)

import.meta

Conclusion

When/how can I use this stuff?

Enabling ES2020 support

Using Babel 7.8.0 or TypeScript 3.7

Using babel plugins

Convert a Website into an API: Building a Serverless Web Scraper with the AWS Cloud Development Kit

How?

AWS Cloud Development Kit

Creating your AWS Account and IAM User

Getting Started with the AWS CDK

Design our Scraper

Coding the Scraper

cheerio

node-fetch

Deploying Our Scraper to AWS

Creating a Serverless App with the CDK

Creating Our DynamoDB Table and Serverless REST API

Interacting with DynamoDB

Writing the Scrape Result into Dynamo

Querying DynamoDB

Creating our new API in AWS

Querying our API

Wrapping Up

clinews - read the news from the CLI!

`String.prototype.match`

`RegExp.prototype.exec`

Nullish Coalescing Operator (`null` or `undefined`)