DEV Community: Valentino Gagliardi

Understanding OAuth 2 with PKCE in Single-Page Applications (2020)

Valentino Gagliardi — Tue, 29 Sep 2020 10:11:39 +0000

Token-based authentication on the web is one of those things you know exists, but sometimes you're too scared to implement in your projects.

Authentication for SPA (single page applications) is even more scary, but willing or not you should know a bit of theory to work proficiently with any SDK when the time comes.

In this post I hope to clarify for you the current recommended OAuth 2 flow for single-page applications: authorization code grant with PKCE.

Who should read this post

Frontend developers who already worked with OAuth 2 in the past, and want to learn more about what's behind authorization code grant with PKCE.

Requirements: you know that OAuth 2 is a security protocol for authorization delegation, and you know what an OAuth application is.

If you need a refresh on the basics instead, check out OAuth 2.0 Simplified by Aaron Parecki.

A word of warning

Security on the web is a minefield. First of all, do not use my examples for production code. They serve just as a pointer for helping you understand OAuth 2. I'm not suggesting you write your own OAuth client. Also:

Don't roll out your own crypto.
Don't roll out your own authorization servers by hand. Use solid libraries instead.
Don't use localStorage for saving authorization codes, or worst, tokens.

Enjoy!

Terminology

Albeit boring, to start off we need to outline OAuth 2.0 terminology.

Resource owner: (simply user from now on) the user who owns permissions on a third-party service. Example: any user who has an account on Github, or Twitter, or on a generic service exposing an API.
Authorization server: also called AS, is the service which grants access to a client on a resource server on behalf of the user.
Resource server: the third-party service that the user want to delegate access on. Example: Github, Twitter, Google Drive, or any protected API. Sometimes the resource server and the authorization server run on the same server.
Client: a single-page application or some leaner JS frontend which needs access to the resource server. Example: a single-page app to display your Github repos.
Scope: it answers the question "what the client can access on the resource server"?

For simplicity, in the following example we condense resource server and authorization server in a single place.

Let these terms sink in, and once confident head over the next section.

The flow

We are developing a single-page application that eventually will be distributed, but for now should only access a resource on a third-party service on which we have an account.

Here are the steps we need to take to abide to the OAuth 2.0 flow for single-page applications:

0: User registers and logins on the third-party service, creates a new OAuth application. Users obtains client_id.
1: User visits the client, and clicks a link which takes it to the authorization server.
2: The authorization server asks the user which permissions (scope) should delegate to the client.
3: Once the user accepts, the authorization server redirects the user back to the client, with an authorized code attached in the URL.
4: Client uses the authorization code to request an access token from the authorization server.
5: Client sends back the token on each subsequent request to a protected resource on the resource server, until the access token expires.
6: Eventually, client uses a refresh token to request a new access token.

Not covered in this post: scopes, the logout phase, handling of the refresh token.

Let's now see the flow in more detail.

0: User registers and logins to the service

As a prerequisite for the OAuth2.0 flow, the user has to have a registered account on the authorization server. She must also create an OAuth application on the authorization server.

Suppose users want to delegate access to Github to a single-page application she's building. She first registers with Github, maybe she creates a couple of repos, and then she creates a new OAuth application in Settings -> Developer settings -> OAuth apps.

If the user/developer controls also an API, say a Django REST API, with a package like django-auth-toolkit she can configure an OAuth 2.0 provider in a few minutes. Here's for example the admin interface for adding a new OAuth application in Django:

In this phase she also provides one or more redirect_uri, where the authorization server will redirect the user. Once done, the service gives back a client_id code, and optionally a client_secret.

Single-page applications can't safely store a client_secret, so they rely only on client_id and PKCE (more on this later).

Next step: user visits the client, and clicks a link which takes it to the authorization server.

1: User -> Client -> Authorization server

This is the one most important part of the flow.

In this part of the flow, the user visits the client, normally a webpage or a single-page application. There she finds a link to follow, which takes her to the authorization server.

Alongside with the request, the client should pass a bunch of query parameters in the URL. Those are at least:

client_id: the id obtained during the OAuth application registration.
response_type: the grant type. In our case it's authorization code grant type so we use response_type=code.
state: a random string used for CSRF protection.
code_challenge: part of PKCE, Proof Key for Code Exchange. More on this later.
code_challenge_method: part of PKCE, the hashing algorithm.
redirect_uri

We could also use a scope parameter to narrow down the permissions for the client, not used in this example.

Now assuming:

https://client.example/ is the client
https://client.example/auth is the client redirect URI
https://auth.example/oauth/authorize is the authorization server

The developer can craft the following link in the page:

https://auth.example/oauth/authorize
?client_id=ERPn7zeLvPCnZ1OtF7jxlTiZz0uQjpiWQEdeAUva
&response_type=code
&state=nMdR7oTxgXYr0INLytRi9XH7InnbrNbg
&code_challenge=M5oPBP6RmJPh5QsGtx6ptVH7S2yjqt8sum96jBCyhZg
&code_challenge_method=S256
&redirect_uri=https://client.example/auth

(Note: the client id here has nothing to do with the previous image).

state is a random string, you can generate it with any JavaScript library:

import randomstring from "randomstring";

const state = randomstring.generate();

For Node.js there is randomstring, but on the frontend you need to find something lighter or tree-shakeable.

Next up, where do we get code_challenge? To generate code_challenge we:

Generate first a code_verifier. This should be a high entropy string. The longer the string, the better (maximum 128 characters as per spec):

import randomstring from "randomstring";

const state = randomstring.generate();
const code_verifier = randomstring.generate(128);

It's important to remember that code_verifier and state need to persist somewhere in the browser because we need them in the next steps. Persisting these two values is harmless.

You can put them in sessionStorage:

import randomstring from "randomstring";

const state = randomstring.generate();
const code_verifier = randomstring.generate(128);

function saveStateAndVerifier() {
  /*
  Don't overwrite our saved state if location has the state parameter.
  This means we got authorization from the AS, and we need to compare them later.
 */
  if (window.location.search.includes("state")) return;
  const storage = window.sessionStorage;
  storage.clear();
  storage.setItem("state", state);
  storage.setItem("code_verifier", code_verifier);
}

saveStateAndVerifier();

Then we compute code_challenge starting from code_verifier:

import randomstring from "randomstring";
import { encode as base64encode } from "base64-arraybuffer";

const state = randomstring.generate();
const code_verifier = randomstring.generate(128);

function saveStateAndVerifier() {
  /*
  Don't overwrite our saved state if location has the state parameter.
  This means we got authorization from the AS, and we need to compare them later.
 */
  if (window.location.search.includes("state")) return;
  const storage = window.sessionStorage;
  storage.clear();
  storage.setItem("state", state);
  storage.setItem("code_verifier", code_verifier);
}

saveStateAndVerifier();

async function generateCodeChallenge(codeVerifier) {
  const encoder = new TextEncoder();
  const data = encoder.encode(codeVerifier);
  const digest = await window.crypto.subtle.digest("SHA-256", data);
  const toBase64 = base64encode(digest);
  // you can extract this replacing code to a function
  return base64Digest
    .replace(/\+/g, "-")
    .replace(/\//g, "_")
    .replace(/=/g, "");
}

generateCodeChallenge(code_verifier).then((challenge) => {
  console.log(challenge);
  // whatever you generated, example:
  // M5oPBP6RmJPh5QsGtx6ptVH7S2yjqt8sum96jBCyhZg
});

This is probably the most difficult part. For a complete explanation see "Generating the code challenge for PKCE in OAuth 2".

What matters here is that you're left with a code_challenge, a string like M5oPBP6RmJPh5QsGtx6ptVH7S2yjqt8sum96jBCyhZg. This will be sent later to the authorization server.

Once you have everything you can build the URL:

// These should go in environment variables injected at build time
// I put it here to keep things simple
const client_id = "ERPn7zeLvPCnZ1OtF7jxlTiZz0uQjpiWQEdeAUva"
const redirect_uri = "https://client.example/auth";

// omitted the rest for brevity

generateCodeChallenge(code_verifier).then((challenge) => {
  buildLoginUrl(challenge);
});

// Proto-code. Don't use for production
function buildLoginUrl(challenge) {
  const link = document.querySelector("a");
  const linkValue = new URL("https://auth.example/oauth/authorize");

  const queryParams = {
    client_id, 
    response_type: "code",
    state: state,
    code_challenge: challenge,
    code_challenge_method: "S256",
    redirect_uri
  };

  for (const param in queryParams) {
    linkValue.searchParams.append(param, queryParams[param]);
  }

  link.setAttribute("href", linkValue);
}

At the end it should translate to something along these lines:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>OAuth 2.0 in Single-Page Applications for Mere Mortals</title>
</head>
<body>
<a href="https://auth.example/oauth/authorize
?client_id=ERPn7zeLvPCnZ1OtF7jxlTiZz0uQjpiWQEdeAUva
&response_type=code
&state=nMdR7oTxgXYr0INLytRi9XH7InnbrNbg
&code_challenge=M5oPBP6RmJPh5QsGtx6ptVH7S2yjqt8sum96jBCyhZg
&code_challenge_method=S256
&redirect_uri=https://client.example/auth">LOGIN</a>
</body>
</html>

You can see an example of this first step any time you're about to log in on a service with "Login with GitHub" or "Login with Google". In most frameworks this is called social authentication.

Who codes this part? This part of the flow is a responsibility for the frontend developer. In most cases, it is recommended to use the SDK provided by your authentication provider.

Next step: authorization server asks the user for permissions.

2. Authorization server asks the user for permissions

As a prerequisite for this step, the user must be logged in the third-party service. If that's the case, the user is presented with a screen which asks for permissions on the resource server:

This is again an example with django-oauth-toolkit.

Remember: here we condense resource server and authorization server in a single place, but in the real world this is not always the case.

If the user gives consent, the authorization server redirects the user to the redirect_uri specified in step 0.

Who codes this part? This part of the flow is a responsibility for the backend developer who should prepare an OAuth 2.0 capable server using proven and tested packages.

Next step: authorization server redirects the user back to the client.

3. Authorization server -> Client -> User

Once the user authorizes the client, the authorization server redirects the user to the redirect_uri provided in step 0 during the OAuth application registration, in our example https://client.example/auth.

The redirect URL will carry in addition an authorization code as a query parameter, as well as state:

https://client.example/auth
?code=l2Z1DNMaiJWZEa3iZJsChdUeKraG3k
&state=nMdR7oTxgXYr0INLytRi9XH7InnbrNbg

A couple of important points:

Don't save the authorization code in localStorage, there's no need.
You can notice how state from this request is the same state we computed in step 1.

In this step we need to compare the state we get from the authorization server, and the state we saved in sessionStorage in step 1.

If they are not the same, we must alt the flow:

const currentLocation = new URL(window.location.href);
const stateFromLocation = currentLocation.searchParams.get("state");

if (window.sessionStorage.getItem("state") !== stateFromLocation){
    throw Error("Probable session hijacking attack!");
}

Who codes this part? This part of the flow is a responsibility for the frontend developer who ensures that the redirect URI is accessible.

Next step: client uses the authorization code to request an access token from the authorization server.

4. Client uses the authorization code to request access token

At this point the JavaScript client has everything in place for requesting an access token.

This request should reach the authorization server at https://auth.example/oauth/token by the means of a application/x-www-form-urlencoded POST request.

There are a lot of things that need to go in the request body. Namely:

client_id: we have this from the beginning.
grant_type: this should be authorization_code.
state: we saved this in sessionStorage, step 1.
code: we get this from the current location.
code_verifier: : we saved this in sessionStorage, step 1.
redirect_uri: we have this from the beginning.

To get the access token, the developer crafts a function which runs right after the redirect flow. The following code is a loose idea of how it should look like:

// Proto-code. Don't use for production
function getToken() {
  const tokenEndpoint = new URL(
    "https://auth.example/oauth/token"
  );

  const currentLocation = new URL(window.location.href);
  const authorizationCode = currentLocation.searchParams.get("code");
  const stateFromLocation = currentLocation.searchParams.get("state");
  const initialCodeVerifier = window.sessionStorage.getItem("code_verifier");

  // This is a good place for checking the state too  
  if (window.sessionStorage.getItem("state") !== stateFromLocation){
    throw Error("Probable session hijacking attack!");
  }

  const queryParams = {
    client_id,
    grant_type: "authorization_code",
    state: stateFromLocation,
    code: authorizationCode,
    code_verifier: initialCodeVerifier,
    redirect_uri
  };

  for (const param in queryParams) {
    tokenEndpoint.searchParams.append(param, queryParams[param]);
  }

  fetch(`${tokenEndpoint.origin}${tokenEndpoint.pathname}`, {
    method: "POST",
    body: tokenEndpoint.searchParams,
    headers: {
      "Content-Type": "application/x-www-form-urlencoded",
      Accept: "application/json"
    }
  }).then(/* more later */)
}

Here we get state and code from the current location, we check that the previous state and the state from the authorization server match.

Next up, we build the query parameters, and we send the POST request. The authorization server must allow CORS for this to work.

If everything goes well, the authorization server responds with a Bearer access token, and a refresh token. Here's an example of response:

Who codes this part? The frontend developer puts together the logic for building the POST request. In most cases, it is recommended to use the SDK provided by your authentication provider. The backend developer ensures that the authorization server exposes the appropriate CORS headers.

Next step: Client uses the access token to access a protected resource.

5. Client sends the token to access a protected resource

This is probably the "easiest" part. In the getToken() function we add a minimal logic for saving the JSON response, which now holds access and refresh token, as well as the expiration. Here's only the relevant snippet:

let tokens = null;

function getToken() {
// omit

  fetch(`${tokenEndpoint.origin}${tokenEndpoint.pathname}`, {
    method: "POST",
    body: tokenEndpoint.searchParams,
    headers: {
      "Content-Type": "application/x-www-form-urlencoded",
      Accept: "application/json"
    }
  })
    .then(response => {
      if (!response.ok) throw Error(response.statusText);
      return response.json();
    })
    .then(json => {
      // save the tokens
      tokens = json;
    });
}

In $insertYourFrameworkHere you will save these tokens in the ephemeral state of the application.

Next up, we send the access token on each subsequent request to the resource server (a REST API for example), until the token expires.

To do so, we pass an Authorization header with the bearer token. For example, we might want to fetch a list of resources on a button click:

const button = document.querySelector("button");
button.addEventListener("click", fetchData);

function fetchData() {
  fetch("https://auth.example/api/customers/", {
    headers: {
      Authorization: `Bearer ${tokens.access_token}`
    }
  }).then(/* do stuff with the response */);
}

We imagined from the beginning that in our example the resource server, and the OAuth 2 authorization server live on the same machine. Hence, we call the API at https://auth.example/api/customers/.

Who codes this part? The frontend developer puts together the logic for handling the access token. The backend developer ensures that the REST API (resource server in our case) exposes the appropriate CORS headers.

What's next?

At this point we have a minimal working implementation of a rudimentary Auth 2 client. But we lack logout, scope handling, and the logic for using the refresh token.

Authorization providers like AWS or Okta have already solved this problem for us, and that's what most companies use in the real world.

If anything, hopefully you learned something new, and the theory behind authorization code grant with PKCE.

Wrapping up

OAuth 2 is hard, there are a lot of ways you can shoot yourself in the foot. In the frontend, always program defensively. In particular:

Use the OAuth SDK from your authorization provider.
Don't roll out your own crypto.
Don't use localStorage for saving the authorization code, even if it's temporary.
Don't save the access token, or the refresh token in localStorage!
Again, use the OAuth SDK from your authorization provider.
Again, don't roll out your own crypto.

On the backend, don't venture in reinventing the wheel. There are solid, battle-tested libraries for building OAuth 2 authorization servers.

Have fun!

Further resources

OAuth 2.0 Simplified by Aaron Parecki covers ins and outs of OAuth 2.0 in clear, simple steps.

My book Decoupled Django, (Apress, May 2021) covers OAuth 2 for Django APIs with Single-page applications. Stay tuned on my Telegram channel for previews and excerpts from the book!

A practical, Complete Tutorial on HTTP cookies

Valentino Gagliardi — Wed, 16 Sep 2020 06:41:35 +0000

What are cookies in web development?

Cookies are tiny pieces of data that the backend can store in the user's browsers. User tracking, personalization, and most important, authentication, are the most common use cases for cookies.

Cookies have a lot of privacy concerns, and have been subject to strict regulation over the years.

In this post I'll focus mainly on the technical side: you'll learn how to create, use, and work with HTTP cookies, on the frontend, and on the backend.

What you will learn

In the following guide you'll learn:

how to work with cookies, backend and frontend
cookie security and permissions
interaction between cookies, AJAX, and CORS

Setting up the backend

The examples for the backend are in Python with Flask. If you want to follow along, create a new Python virtual environment, move into it, and install Flask:

mkdir cookies && cd $_

python3 -m venv venv
source venv/bin/activate

pip install Flask

In the project folder create a new file named flask_app.py, and use my examples to experiment locally.

Who creates cookies?

First things first, where does cookies come from? Who creates cookies?

While it's possible to create cookies in the browser with document.cookie, most of the times it's responsibility of the backend to set cookies in the response before sending it to the client.

By backend here we mean that cookies can be created by:

the actual application's code on the backend (Python, JavaScript, PHP, Java)
a webserver responding to requests (Nginx, Apache)

For doing so the backend sets in the response an HTTP header named Set-Cookie with a corresponding string made of a key/value pair, plus optional attributes:

Set-Cookie: myfirstcookie=somecookievalue

When and where to create these cookies depends on the requirements.

So, cookies are simple strings. Consider this example in Python with Flask. Create a Python file named flask_app.py in the project folder with the following code:

from flask import Flask, make_response

app = Flask(__name__)


@app.route("/index/", methods=["GET"])
def index():
    response = make_response("Here, take some cookie!")
    response.headers["Set-Cookie"] = "myfirstcookie=somecookievalue"
    return response

Then run the app:

FLASK_ENV=development FLASK_APP=flask_app.py flask run

When this application is running, and the user visits http://127.0.0.1:5000/index/ the backend sets a response header named Set-Cookie with a key/value pair.

(127.0.0.1:5000 is the default listening address/port for Flask applications in development).

The Set-Cookie header is the key to understand how to create cookies:

response.headers["Set-Cookie"] = "myfirstcookie=somecookievalue"

On the right side you can see the actual cookie "myfirstcookie=somecookievalue".

Most frameworks have their own utility functions for setting cookies programmatically, like Flask's set_cookie().

Under the hood they simply set a header in the response with Set-Cookie.

How to see cookies?

Consider again the previous example with Flask. Once you visit http://127.0.0.1:5000/index/, the backend sets a cookie in the browser. To see this cookie you can either call document.cookie from the browser's console:

Or you can check the Storage tab in the developer tools. Click on Cookies, and you should see the cookie there:

On a command line you can use also curl to see what cookies the backend sets:

curl -I http://127.0.0.1:5000/index/

To save cookies to a file for later use:

curl -I http://127.0.0.1:5000/index/ --cookie-jar mycookies

To display cookies on stdout:

curl -I http://127.0.0.1:5000/index/ --cookie-jar -

Note that cookies without the HttpOnly attribute are accessible on document.cookie from JavaScript in the browser. On the other hand a cookie marked as HttpOnly cannot be accessed from JavaScript.

To mark a cookie as HttpOnly pass the attribute in the cookie:

Set-Cookie: myfirstcookie=somecookievalue; HttpOnly

Now the cookie will still appear in the Cookie Storage tab, but document.cookie will return an empty string.

From this point on for convenience I'll use Flask's response.set_cookie() to create cookies on the backend.

To inspect cookies along the way in this guide we'll use alternatively:

curl
Firefox developer tools
Chrome developer tools

I've got a cookie, what now?

Your browser gets a cookie. Now what? Once you have a cookie, the browser can send back the cookie to the backend.

This could have a number of applications: user tracking, personalization, and most important, authentication.

For example, once you log in in a website the backend can give you a cookie:

Set-Cookie: userid=sup3r4n0m-us3r-1d3nt1f13r

To properly identify you on each subsequent request, the backend checks the cookie coming from the browser in the request.

To send the cookie, the browser appends a Cookie header in the request:

Cookie: userid=sup3r4n0m-us3r-1d3nt1f13r

How, when, and why the browser sends back cookies is the topic for the next sections.

Cookies can expire: Max-Age and expires

By default, cookies expire when the user closes the session, that is, when she closes the browser. To persist a cookie we can pass expires or Max-Age attributes:

Set-Cookie: myfirstcookie=somecookievalue; expires=Tue, 09 Jun 2020 15:46:52 GMT; Max-Age=1209600

When bot attributes are present, Max-Age has precedence over expires.

Cookies are scoped by path: the Path attribute

Consider this backend which sets a new cookie for its frontend when visiting http://127.0.0.1:5000/. On the other two routes instead we print the request's cookies:

from flask import Flask, make_response, request

app = Flask(__name__)


@app.route("/", methods=["GET"])
def index():
    response = make_response("Here, take some cookie!")
    response.set_cookie(key="id", value="3db4adj3d", path="/about/")
    return response


@app.route("/about/", methods=["GET"])
def about():
    print(request.cookies)
    return "Hello world!"


@app.route("/contact/", methods=["GET"])
def contact():
    print(request.cookies)
    return "Hello world!"

To run the app:

FLASK_ENV=development FLASK_APP=flask_app.py flask run

In another terminal, if we make connection with the root route we can see the cookie in Set-Cookie:

curl -I http://127.0.0.1:5000/ --cookie-jar cookies

HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 23
Set-Cookie: id=3db4adj3d; Path=/about/
Server: Werkzeug/1.0.1 Python/3.8.3
Date: Wed, 27 May 2020 09:21:37 GMT

Notice how the cookies has a Path attribute:

Set-Cookie: id=3db4adj3d; Path=/about/

Let's now visit the /about/ route by sending the cookie we saved in the first visit:

curl -I http://127.0.0.1:5000/about/ --cookie cookies

In the terminal where the Flask app is running you should see:

ImmutableMultiDict([('id', '3db4adj3d')])
127.0.0.1 - - [27/May/2020 11:27:55] "HEAD /about/ HTTP/1.1" 200 -

As expected the cookie goes back to the backend. Now try to visit the /contact/ route:

curl -I http://127.0.0.1:5000/contact/ --cookie cookies

This time in the terminal where the Flask app is running you should see:

ImmutableMultiDict([])
127.0.0.1 - - [27/May/2020 11:29:00] "HEAD /contact/ HTTP/1.1" 200 -

What that means? Cookies are scoped by path. A cookie with a given Path attribute cannot be sent to another, unrelated path, even if both path live on the same domain.

This is the first layer of permissions for cookies.

When Path is omitted during cookie creation, the browsers defaults to /.

Cookies are scoped by domain: the Domain attribute

The value for the Domain attribute of a cookie controls whether the browser should accept it or not and where the cookie goes back.

Let's see some examples.

NOTE: the following URL are on free Heroku instances. Give it a second to spin up. Open up a browser's console before opening the links to see the result in the network tab.

Non matching host (wrong host)

Consider the following cookie set by https://serene-bastion-01422.herokuapp.com/get-wrong-domain-cookie/:

Set-Cookie: coookiename=wr0ng-d0m41n-c00k13; Domain=api.valentinog.com

Here the cookie originates from serene-bastion-01422.herokuapp.com, but the Domain attribute has api.valentinog.com.

There's no other choice for the browser to reject this cookie. Chrome for example gives a warning (Firefox does not):

Non matching host (subdomain)

Consider the following cookie set by https://serene-bastion-01422.herokuapp.com/get-wrong-subdomain-cookie/:

Set-Cookie: coookiename=wr0ng-subd0m41n-c00k13; Domain=secure-brushlands-44802.herokuapp.com

Here the cookie originates from serene-bastion-01422.herokuapp.com, but the Domain attribute is secure-brushlands-44802.herokuapp.com.

They are on the same domain, but the subdomain is different. Again, the browser rejects this cookie as well:

Matching host (whole domain)

Consider now the following cookie set by visiting https://www.valentinog.com/get-domain-cookie.html:

set-cookie: cookiename=d0m41n-c00k13; Domain=valentinog.com

This cookie is set at the web server level with Nginx add_header:

add_header Set-Cookie "cookiename=d0m41n-c00k13; Domain=valentinog.com";

I used Nginx here to show you there are various ways to set a cookie. The fact that a cookie is set by a web server or by the application's code doesn't matter much for the browser.

What matters is the domain the cookie is coming from.

Here the browser will happily accept the cookie because the host in Domain includes the host from which the cookie came.

In other words, valentinog.com includes the subdomain www.valentinog.com.

Also, the cookie travels back with any new request against valentinog.com, as well as any request to subdomains on valentinog.com.

Here's a request to the www subdomain with the cookie attached:

Here's a request to another subdomain with the cookie automatically attached:

Cookies and the Public Suffix List

Now consider the following cookie set by https://serene-bastion-01422.herokuapp.com/get-domain-cookie/:

Set-Cookie: coookiename=d0m41n-c00k13; Domain=herokuapp.com

Here the cookie comes from serene-bastion-01422.herokuapp.com, and the Domain attribute is herokuapp.com. What should the browser do here?

You might think that serene-bastion-01422.herokuapp.com is included in the domain herokuapp.com, so the browser should accept the cookie.

Instead, it rejects the cookie because it comes from a domain included in the Public Suffix List.

The Public Suffix List is a list maintained by Mozilla, used by all browsers to restrict who can set cookies on behalf of other domains.

Resources:

Matching host (subdomain)

Consider now the following cookie set by https://serene-bastion-01422.herokuapp.com/get-subdomain-cookie/:

Set-Cookie: coookiename=subd0m41n-c00k13

When Domain is omitted during cookie creation, the browsers defaults to the originating host in the address bar, in this case my code does:

response.set_cookie(key="coookiename", value="subd0m41n-c00k13")

When the cookie lands in the browser's cookie storage we see the Domain applied:

So we have this cookie from serene-bastion-01422.herokuapp.com. Where this cookie should be sent now?.

If you visit https://serene-bastion-01422.herokuapp.com/ the cookie goes with the request:

But, if you visit herokuapp.com the cookie does not leave the browser at all:

(It doesn't matter that herokuapp.com later redirects to heroku.com).

To recap, the browser uses the following heuristics to decide what to do with cookies (by sender host here I mean the actual URL you visit):

Reject the cookie altogether if either the domain or the subdomain in Domain don't match the sender host
Reject the cookie if the value of Domain is included in the Public suffix list
Accept the cookie if the domain or the subdomain in Domain matches the sender host

Once the browsers accepts the cookie, and it's about to make a request it says:

Send it back the cookie if the request host matches exactly the value I saw in Domain
Send it back the cookie if the request host is a subdomain matching exactly the value I saw in Domain
Send it back the cookie if the request host is a subdomain like sub.example.dev included in a Domain like example.dev
Don't send it back the cookie if the request host is a main domain like example.dev and Domain was sub.example.dev

Takeaway: Domain is the second layer of permissions for cookies, alongside with the Path attribute.

Cookies can travel over AJAX requests

Cookies can travel over AJAX requests. AJAX requests are asynchronous HTTP requests made with JavaScript (XMLHttpRequest or Fetch) to get and send back data to a backend.

Consider another example with Flask where we have a template, which in turn loads a JavaScript file. Here's the Flask app:

from flask import Flask, make_response, render_template

app = Flask(__name__)


@app.route("/", methods=["GET"])
def index():
    return render_template("index.html")


@app.route("/get-cookie/", methods=["GET"])
def get_cookie():
    response = make_response("Here, take some cookie!")
    response.set_cookie(key="id", value="3db4adj3d")
    return response

Here's the template in templates/index.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
<button>FETCH</button>
</body>
<script src="{{ url_for('static', filename='index.js') }}"></script>
</html>

Here's the JavaScript code in static/index.js:

const button = document.getElementsByTagName("button")[0];

button.addEventListener("click", function() {
  getACookie();
});

function getACookie() {
  fetch("/get-cookie/")
    .then(response => {
      // make sure to check response.ok in the real world!
      return response.text();
    })
    .then(text => console.log(text));
}

When visiting http://127.0.0.1:5000/ we see a button. By clicking the button we make a Fetch request to /get-cookie/ to obtain a cookie back. As expected the cookie lands in the browser's Cookie storage.

Now let's change a bit our Flask app to expose another endpoint:

from flask import Flask, make_response, request, render_template, jsonify

app = Flask(__name__)


@app.route("/", methods=["GET"])
def index():
    return render_template("index.html")


@app.route("/get-cookie/", methods=["GET"])
def get_cookie():
    response = make_response("Here, take some cookie!")
    response.set_cookie(key="id", value="3db4adj3d")
    return response


@app.route("/api/cities/", methods=["GET"])
def cities():
    if request.cookies["id"] == "3db4adj3d":
        cities = [{"name": "Rome", "id": 1}, {"name": "Siena", "id": 2}]
        return jsonify(cities)
    return jsonify(msg="Ops!")

Also, let's tweak our JavaScript code so that we make another Fetch request after getting the cookie:

const button = document.getElementsByTagName("button")[0];

button.addEventListener("click", function() {
  getACookie().then(() => getData());
});

function getACookie() {
  return fetch("/get-cookie/").then(response => {
    // make sure to check response.ok in the real world!
    return Promise.resolve("All good, fetch the data");
  });
}

function getData() {
  fetch("/api/cities/")
    .then(response => {
      // make sure to check response.ok in the real world!
      return response.json();
    })
    .then(json => console.log(json));
}

When visiting http://127.0.0.1:5000/ we see a button. By clicking the button we make a Fetch request to /get-cookie/ to obtain a cookie back. As soon as the cookie comes, we make another Fetch request to /api/cities/.

In the browser's console you should see an array of cities. Also, in the Network tab of the developer tool you should see a header named Cookie, transmitted to the backend over the AJAX request:

This cookie exchange back and forth between frontend and backend works fine as long as the frontend is in the same context of the backend: we say that they're on the same origin.

That's because by default, Fetch sends credentials, i.e. cookies only when the request hits the same origin from which the request fires.

Here, JavaScript is served by a Flask template on http://127.0.0.1:5000/.

Let's see instead what happens for different origins.

Cookies cannot always travel over AJAX requests

Consider a different situation where the backend runs stand-alone, so you have this Flask app running:

FLASK_ENV=development FLASK_APP=flask_app.py flask run

Now in a different folder, outside of the Flask app, create an index.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
<button>FETCH</button>
</body>
<script src="index.js"></script>
</html>

Create in the same folder a JavaScript file named index.js with the following code:

const button = document.getElementsByTagName("button")[0];

button.addEventListener("click", function() {
  getACookie().then(() => getData());
});

function getACookie() {
  return fetch("http://localhost:5000/get-cookie/").then(response => {
    // make sure to check response.ok in the real world!
    return Promise.resolve("All good, fetch the data");
  });
}

function getData() {
  fetch("http://localhost:5000/api/cities/")
    .then(response => {
      // make sure to check response.ok in the real world!
      return response.json();
    })
    .then(json => console.log(json));
}

In the same folder, from the terminal run:

npx serve

This command gives you a local address/port to connect to, like http://localhost:42091/. Visit the page and try to click the button with the browser's console open. In the console you should see:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:5000/get-cookie/. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)

Now, http://localhost:5000/ is not the same as http://localhost:42091/. They're different origins, hence CORS kick ins.

An origin consists of a scheme, domain, and port number. That means http://localhost:5000/ is a different origin from http://localhost:42091/.

Dealing with CORS

CORS, acronym for Cross-Origin Resource Sharing, is a way for servers to control access to resources on a given origin, when JavaScript code running on a different origin requests these resources.

By default, browsers block AJAX requests to remote resources which are not on the same origin, unless a specific HTTP header named Access-Control-Allow-Origin is exposed by the server.

To fix this first error we need to configure CORS for Flask:

pip install flask-cors

Then apply CORS to Flask:

from flask import Flask, make_response, request, render_template, jsonify
from flask_cors import CORS

app = Flask(__name__)
CORS(app=app)


@app.route("/", methods=["GET"])
def index():
    return render_template("index.html")


@app.route("/get-cookie/", methods=["GET"])
def get_cookie():
    response = make_response("Here, take some cookie!")
    response.set_cookie(key="id", value="3db4adj3d")
    return response


@app.route("/api/cities/", methods=["GET"])
def cities():
    if request.cookies["id"] == "3db4adj3d":
        cities = [{"name": "Rome", "id": 1}, {"name": "Siena", "id": 2}]
        return jsonify(cities)
    return jsonify(msg="Ops!")

Now try to click again the button with the browser's console open. In the console you should see:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:5000/api/cities/. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)

Despite we got the same error, this time the culprit lies in the second route.

There's no such cookie named "id" attached to the request, so Flask crashes and no Access-Control-Allow-Origin gets set.

You can confirm this by looking at the request in the Network tab. No such Cookie is sent:

To include cookies in a Fetch requests across different origins we must provide the credentials flag (by default it's same origin).

Without this flag Fetch simply ignores cookies. To fix our example:

const button = document.getElementsByTagName("button")[0];

button.addEventListener("click", function() {
  getACookie().then(() => getData());
});

function getACookie() {
  return fetch("http://localhost:5000/get-cookie/", {
    credentials: "include"
  }).then(response => {
    // make sure to check response.ok in the real world!
    return Promise.resolve("All good, fetch the data");
  });
}

function getData() {
  fetch("http://localhost:5000/api/cities/", {
    credentials: "include"
  })
    .then(response => {
      // make sure to check response.ok in the real world!
      return response.json();
    })
    .then(json => console.log(json));
}

credentials: "include" has to be present on the first Fetch request, to save the cookie in the browser's Cookie storage:

fetch("http://localhost:5000/get-cookie/", {
    credentials: "include"
  })

It has also to be present on the second request to allow transmitting cookies back to the backend:

  fetch("http://localhost:5000/api/cities/", {
    credentials: "include"
  })

Try again, and you'll see we need to fix another error on the backend:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:5000/get-cookie/. (Reason: expected ‘true’ in CORS header ‘Access-Control-Allow-Credentials’).

To allow cookie transmission in CORS requests, the backend needs to expose the Access-Control-Allow-Credentials header as well. Easy fix:

CORS(app=app, supports_credentials=True)

Now you should see the expected array of cities in the browser's console.

Takeaways: to make cookies travel over AJAX requests between different origins provide:

credentials: "include" on the frontend for Fetch
Access-Control-Allow-Credentials and Access-Control-Allow-Origin on the backend.

Cookies can travel over AJAX requests, but they have to respect the domain rules we described earlier.

Resources:

A concrete example

Our previous example uses localhost to keep things simple and replicable on your local machine.

To imagine cookie exchange over AJAX requests in the real world you can think of the following scenario:

a user visits https://www.a-example.dev
she clicks a button or makes some action which triggers a Fetch request to https://api.b-example.dev
https://api.b-example.dev sets a cookie with Domain=api.b-example.dev
on subsequent Fetch requests to https://api.b-example.dev the cookie is sent back

Cookies can be kind of secret: the Secure attribute

But not so secret after all.

The Secure attribute for a cookie ensures that the cookie is never accepted over HTTP, that is, the browser rejects secure cookies unless the connection happens over HTTPS.

To mark a cookie as Secure pass the attribute in the cookie:

Set-Cookie: "id=3db4adj3d; Secure"

In Flask:

response.set_cookie(key="id", value="3db4adj3d", secure=True)

If you want to try against a live environment, run the following command on the console and note how curl here does not save the cookie over HTTP:

curl -I http://serene-bastion-01422.herokuapp.com/get-secure-cookie/ --cookie-jar -

Note: this will work only in curl 7.64.0 >= which implements rfc6265bis. Older versions of curl implement RCF6265. See

Over HTTPS instead, the cookie appears in the cookie jar:

curl -I https://serene-bastion-01422.herokuapp.com/get-secure-cookie/ --cookie-jar -

Here's the jar:

serene-bastion-01422.herokuapp.com      FALSE   /       TRUE    0       id      3db4adj3d

To try the cookie in a browser visit both versions of the url above and check out the Cookie storage in the developer tool.

Don't get fooled by Secure: browsers accept the cookie over HTTPS, but there's no protection for the cookie once it lands in the browser.

For this reason a Secure cookie, like any cookie, is not intended for transmission of sensitive data, even if the name would suggest the opposite.

Don't touch my cookie: the HttpOnly attribute

The HttpOnly attribute for a cookie ensures that the cookie is not accessible by JavaScript code. This is the most important form of protection against XSS attacks

However, it is sent on each subsequent HTTP request, with respect of any permission enforced by Domain and Path.

To mark a cookie as HttpOnly pass the attribute in the cookie:

Set-Cookie: "id=3db4adj3d; HttpOnly"

In Flask:

response.set_cookie(key="id", value="3db4adj3d", httponly=True)

A cookie marked as HttpOnly cannot be accessed from JavaScript: if inspected in the console, document.cookie returns an empty string.

However, Fetch can get, and send back HttpOnly cookies when credentials is set to include, again, with respect of any permission enforced by Domain and Path:

fetch(/* url */, {
  credentials: "include"
})

When to use HttpOnly? Whenever you can. Cookies should always be HttpOnly, unless there's a specific requirement for exposing them to runtime JavaScript.

Resources:

The dreaded SameSite attribute

First and third-party cookie

Consider a cookie acquired by visiting https://serene-bastion-01422.herokuapp.com/get-cookie/:

Set-Cookie: simplecookiename=c00l-c00k13; Path=/

We refer to this kind of cookies as first-party. That is, I visit that URL in the browser, and if I visit the same URL, or another path of that site (provided that Path is /) the browser sends the cookie back to the website. Normal cookie stuff.

Now consider another web page at https://serene-bastion-01422.herokuapp.com/get-frog/. This page sets a cookie as well, and in addition it loads an image from a remote resource hosted at https://www.valentinog.com/cookie-frog.jpg.

This remote resource in turns sets a cookie on its own. You can see the actual scenario in this picture:

Note: If you're on Chrome 85 you won't see this cookie. Starting from this version Chrome rejects it.

We refer to this kind of cookies as third-party. Another example of third-party cookie:

a user visits https://www.a-example.dev
she clicks a button or makes some action which triggers a Fetch request to https://api.b-example.dev
https://api.b-example.dev sets a cookie with Domain=api.b-example.dev
now the page at https://www.a-example.dev holds a third-party cookie from https://api.b-example.dev

Working with SameSite

At the time of writing, third-party cookies causes a warning to pop up in the Chrome console:

"A cookie associated with a cross-site resource at http://www.valentinog.com/ was set without the SameSite attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with SameSite=None and Secure.

What the browser is trying to say is that third-party cookies must have the new SameSite attribute. But why?

The SameSite attribute is a new feature aimed at improving cookie security to: prevent Cross Site Request Forgery attacks, avoid privacy leaks.

SameSite can be assigned one of these three values:

Strict
Lax
None

If we are a service providing embeddable widgets (iframes), or we need to put cookies in remote websites (for a good reason and not for wild tracking), these cookies must be marked as SameSite=None, and Secure:

Set-Cookie: frogcookie=fr0g-c00k13; SameSite=None; Secure

Failing to do so will make the browser reject the third-party cookie. Here's what browsers are going to do in the near future:

A cookie associated with a cross-site resource at http://www.valentinog.com/ was set without the SameSite attribute. It has been blocked, as Chrome now only delivers cookies with cross-site requests if they are set with SameSite=None and Secure.

In other words SameSite=None; Secure will make third-party cookies work as they work today, the only difference being that they must be transmitted only over HTTPS.

A cookie configured this way is sent alongside each request if domain and path matches. This is the normal behaviour.

Worth noting, SameSite does not concern only third-party cookies.

By default, browsers will enforce SameSite=Lax on all cookies, both first-party and third-party, if the attribute is missing. Here's Firefox Nightly on a first-party cookie:

Cookie "get_frog_simplecookiename" has "sameSite" policy set to "lax" because it is missing a "sameSite" attribute, and "sameSite=lax" is the default value for this attribute.

A SameSite=Lax cookie is sent back with safe HTTP methods, namely GET, HEAD, OPTIONS, and TRACE. POST requests instead won't carry the cookie.

Third-party cookies with SameSite=Strict instead will be rejected altogether by the browser.

To recap, here's the browser's behaviour for the different values of SameSite:

VALUE	INCOMING COOKIE	OUTGOING COOKIE
Strict	Reject	-
Lax	Accept	Send with safe HTTP methods
None + Secure	Accept	Send

To learn more about SameSite and to understand in detail all the use cases for this attribute, go read these fantastic resources:

Cookies and authentication

Authentication is one of the most challenging tasks in web development. There seems to be so much confusion around this topic, as token based authentication with JWT seems to supersede "old", solid patterns like session based authentication.

Let's see what role cookies play here.

Session based authentication

Authentication is one of the most common use case for cookies.

When you visit a website that requests authentication, on credential submit (through a form for example) the backend sends under the hood a Set-Cookie header to the frontend.

A typical session cookie looks like the following:

Set-Cookie: sessionid=sty1z3kz11mpqxjv648mqwlx4ginpt6c; expires=Tue, 09 Jun 2020 15:46:52 GMT; HttpOnly; Max-Age=1209600; Path=/; SameSite=Lax

In this Set-Cookie header the server may include a cookie named session, session id, or similar.

This is the only identifier that the browser can see in the clear. Any time the authenticated user requests a new page to the backend, the browser sends back the session cookie.

At this point the backend pairs the session id with the session stored on a storage behind the scenes to properly identify the user.

Session based authentication is know as stateful because the backend has to keep track of sessions for each user. The storage for these sessions might be:

a database
a key/value store like Redis
the filesystem

Of these three session storages, Redis or the like should be preferred over database or filesystem.

Note that session based authentication has nothing to do with the browser's Session Storage.

It's called session based only because the relevant data for user identification lives in the backend's session storage, which is not the same thing as a browser's Session Storage.

When to use session based authentication?

Use it whenever you can. Session based authentication is one of the simplest, secure, and straightforward form of authentication for websites. It's available by default on all the most popular web frameworks like Django.

But, its stateful nature is also its main drawback, especially when a website is served by a load balancer. In this case, techniques like sticky sessions, or storing sessions on a centralized Redis storage can help.

A note on JWT

JWT, short for JSON Web Tokens, is an authentication mechanism, rising in popularity in recent years.

JWT is well suited for single page and mobile applications, but it presents a new set of challenges. The typical flow for a frontend application wanting to authenticate against an API is the following:

Frontend sends credentials to the backend
Backend checks credentials and sends back a token
Frontend sends the token on each subsequent request

The main question which comes up with this approach is: where do I store this token in the frontend for keeping the user logged in?

The most natural thing to do for someone who writes JavaScript is to save the token in localStorage. This is bad for so many reasons.

localStorage is easily accessible from JavaScript code, and it's an easy target for XSS attacks.

To overcome this issue, most developers resort to save the JWT token in a cookie thinking that HttpOnly and Secure can protect the cookie, at least from XSS attacks.

The new SameSite attribute, set to SameSite=Strict would also protect your "cookified " JWT from CSRF attacks. But, is also completely invalidates the use case for JWT in first instance because SameSite=Strict does not sends cookies on cross-origin requests!

How about SameSite=Lax then? This mode allows sending cookies back with safe HTTP methods, namely GET, HEAD, OPTIONS, and TRACE. POST requests won't transmit the cookie either way.

Really, storing a JWT token in a cookie or in localStorage are both bad ideas.

If you really want to use JWT instead of sticking with session based auth, and scaling your session storage, you might want to use JWT with refresh tokens to keep the user logged in.

Resources:

Wrapping up

HTTP cookies have been there since 1994. They're everywhere.

Cookies are simple text strings, but they can be fine tuned for permissions, with Domain and Path, transmitted only over HTTPS with Secure, hide from JavaScript with HttpOnly.

A cookie might be used for personalization of the user's experience, user authentication, or shady purposes like tracking.

But, for all the intended uses, cookies can expose users to attacks and vulnerabilities.

Browser's vendors and the Internet Engineering Task Force have worked year after year to improve cookie security, the last recent step being SameSite.

So what makes a secure cookie? There isn't such a thing. We could consider relatively secure a cookie that:

travels only over HTTPS, that is, has Secure
has HttpOnly whenever possible
has the proper SameSite configuration
does not carry sensitive data

Thanks for reading!

Further resources

Icon in the featured picture by freepik.

var, let, and const in JavaScript: a cheatsheet

Valentino Gagliardi — Tue, 15 Sep 2020 15:01:54 +0000

var

The var statement declares a variable in JavaScript which abides to the following rules:

is function-scoped or globally-scoped.
is not subject to the temporal dead zone.
it creates a global property on window with the same name.
is reassignable.
is redeclarable.

Function-scoped or globally-scoped

var creates a global variable when appears in the global scope. In addition, it creates a global property on window with the same name:

var city = "Florence";

console.log(window.city); // "Florence"

When declared inside a function, the variable is scoped to that function:

var city = "Florence";

function bubble() {
  var city = "Siena";
  console.log(city);
}

bubble(); // "Siena"

console.log(city); // "Florence"

var declarations are subject to hoisting:

function bubble() {
  city = "Siena";
  console.log(city);
  var city; // hoists
}

bubble(); // "Siena"

Accidental global variables

Variables assigned without any statement, be it var, let, or const, become global variables by default:

function bubble() {
  city = "Siena";
  console.log(window.city);
}

bubble(); // "Siena"

To neutralize this behaviour we use strict mode:

"use strict";

function bubble() {
  city = "Siena";
  console.log(window.city);
}

bubble(); // ReferenceError: assignment to undeclared variable city

reassignable and redeclarable

Any variable declared with var can be later reassigned, or redeclared. An example of redeclaration:

function bubble() {
  var city = "Florence";
  var city = "Siena";
  console.log(city);
}

bubble(); // "Siena"

An example of reassignation:

function bubble() {
  var city = "Siena";
  city = "Florence";
  console.log(city);
}

bubble(); // "Florence"

let

The let statement declares a variable in JavaScript which abides to the following rules:

is block scoped.
is subject to the temporal dead zone.
it does not create any global property on window.
is reassignable.
is not redeclarable.

Block scoped

A variable declared with let does not create any global property on window:

let city = "Florence";

console.log(window.city); // undefined

When declared inside a function, the variable is scoped to that function:

let city = "Florence";

function bubble() {
  let city = "Siena";
  console.log(city);
}

bubble(); // "Siena"

console.log(city); // "Florence"

When declared inside a block, the variable is scoped to that block. An example with the block statement:

let city = "Florence";

{
  let city = "Siena";
  console.log(city); // "Siena";
}

console.log(city); // "Florence"

An example with an if block:

let city = "Florence";

if (true) {
  let city = "Siena";
  console.log(city); // "Siena";
}

console.log(city); // "Florence"

var instead, doesn't care about blocks:

var city = "Florence";

{
  var city = "Siena";
  console.log(city); // "Siena";
}

console.log(window.city); // "Siena"

Temporal dead zone

let declarations are subject to hoisting, but temporal dead zone kicks in:

function bubble() {
  city = "Siena";
  console.log(city); // TDZ
  let city;
}

bubble();

// ReferenceError: can't access lexical declaration 'city' before initialization

Temporal dead zone prevents access to let declarations before their initialization. Another example:

function bubble() {
  console.log(city); // TDZ
  let city = "Siena";
}

bubble();

// ReferenceError: can't access lexical declaration 'city' before initialization

We can see that the exceptions are the same in both examples: the proof that temporal dead zone kicked in.

Further resources on the topic: Temporal dead zone demystified.

Reassignable, not redeclarable

Any variable declared with let can't be redeclared. An example of redeclaration which throws:

function bubble() {
  let city = "Siena";
  let city = "Florence";
  console.log(city);
}

bubble(); // SyntaxError: redeclaration of let city

An example of valid reassignation:

function bubble() {
  let city = "Siena";
  city = "Florence";
  console.log(city);
}

bubble(); // "Florence"

const

The const statement declares a variable in JavaScript which abides to the following rules:

is block scoped.
is subject to the temporal dead zone.
it does not create any global property on window.
is not reassignable.
is not redeclarable.

Block scoped

A variable declared with const does not create any global property on window:

const city = "Florence";

console.log(window.city); // undefined

When declared inside a function, the variable is scoped to that function:

const city = "Florence";

function bubble() {
  const city = "Siena";
  console.log(city);
}

bubble(); // "Siena"

console.log(city); // "Florence"

When declared inside a block, the variable is scoped to that block. An example with the block statement {}:

const city = "Florence";

{
  const city = "Siena";
  console.log(city); // "Siena";
}

console.log(city); // "Florence"

An example with an if block:

const city = "Florence";

if (true) {
  const city = "Siena";
  console.log(city); // "Siena";
}

console.log(city); // "Florence"

Temporal dead zone

const declarations are subject to hoisting, but temporal dead zone kicks in:

function bubble() {
  console.log(city);
  const city = "Siena";
}

bubble();

// ReferenceError: can't access lexical declaration 'city' before initialization

Not reassignable, not redeclarable

Any variable declared with const can't be redeclared, nor reassigned. An example of redeclaration which throws:

function bubble() {
  const city = "Siena";
  const city = "Florence";
  console.log(city);
}

bubble(); // SyntaxError: redeclaration of const city

An example of reassignation which trows as well:

function bubble() {
  const city = "Siena";
  city = "Florence";
  console.log(city);
}

bubble(); // TypeError: invalid assignment to const 'city'

A mostly complete guide to error handling in JavaScript

Valentino Gagliardi — Mon, 14 Sep 2020 09:24:03 +0000

What is an error in programming?

Things don't go always well in our programs.

In particular, there are situations where we may want to stop the program or inform the user if something bad happens.

For example:

the program tried to open a non-existent file.
the network connection is broken.
the user entered invalid input.

In all these cases we as programmers, create errors, or we let the programming engine create some for us.

After creating the error we can inform the user with a message, or we can halt the execution altogether.

What is an error in JavaScript?

An error in JavaScript is an object, which is later thrown to halt the program.

To create a new error in JavaScript we call the appropriate constructor function. For example, to create a new, generic error we can do:

const err = new Error("Something bad happened!");

When creating an error object it's also possible to omit the new keyword:

const err = Error("Something bad happened!");

Once created, the error object presents three properties:

message: a string with the error message.
name: the error's type.
stack: a stack trace of functions execution.

For example, if we create a new TypeError object with the appropriate message, the message will carry the actual error string, while name will be "TypeError":

const wrongType = TypeError("Wrong type given, expected number");

wrongType.message; // "Wrong type given, expected number"
wrongType.name; // "TypeError"

Firefox also implements a bunch on non-standard property like columnNumber, filename, and lineNumber.

Many types of errors in JavaScript

There are many types of errors in JavaScript, namely:

Error
EvalError
InternalError
RangeError
ReferenceError
SyntaxError
TypeError
URIError

Remember, all these error types are actual constructor functions meant to return a new error object.

In your code you'll mostly use Error and TypeError, two of the most common types, to create your own error object.

Most of the times, the majority of errors will come directly from the JavaScript engine, like InternalError or SyntaxError.

An example of TypeError occurs when you try to reassign const:

const name = "Jules";
name = "Caty";

// TypeError: Assignment to constant variable.

An example of SyntaxError is when you misspell language keywords:

va x = '33';
// SyntaxError: Unexpected identifier

Or when you use reserved keywords in wrong places, like await outside of an async function:

function wrong(){
    await 99;
}

wrong();

// SyntaxError: await is only valid in async function

Another example of TypeError occurs when we select non-existent HTML elements in the page:

Uncaught TypeError: button is null

In addition to these traditional error objects, an AggregateError object is going to land soon in JavaScript. AggregateError is convenient for wrapping multiple errors together, as we'll see later.

Besides these built-in errors, in the browser we can find also:

DOMException.
DOMError, deprecated and no longer used today.

DOMException is a family of errors related to Web APIs. They are thrown when we do silly things in the browser, like:

document.body.appendChild(document.cloneNode(true));

The result:

Uncaught DOMException: Node.appendChild: May not add a Document as a child

For a complete list see this page on MDN.

What is an exception?

Most developers think that error and exceptions are the same thing. In reality, an error object becomes an exception only when it's thrown.

To throw an exception in JavaScript we use throw, followed by the error object:

const wrongType = TypeError("Wrong type given, expected number");

throw wrongType;

The short form is more common, in most code bases you'll find:

throw TypeError("Wrong type given, expected number");

throw new TypeError("Wrong type given, expected number");

It's unlikely to throw exceptions outside of a function or a conditional block. Instead, consider the following example:

function toUppercase(string) {
  if (typeof string !== "string") {
    throw TypeError("Wrong type given, expected a string");
  }

  return string.toUpperCase();
}

Here we check if the function argument is a string. If it's not we throw an exception.

Technically, you could throw anything in JavaScript, not only error objects:

throw Symbol();
throw 33;
throw "Error!";
throw null;

However, it's better to avoid these things: always throw proper error objects, not primitives.

By doing so you keep error handling consistent through the codebase. Other team members can always expect to access error.message or error.stack on the error object.

What happens when we throw an exception?

Exceptions are like an elevator going up: once you throw one, it bubbles up in the program stack, unless it is caught somewhere.

Consider the following code:

function toUppercase(string) {
  if (typeof string !== "string") {
    throw TypeError("Wrong type given, expected a string");
  }

  return string.toUpperCase();
}

toUppercase(4);

If you run this code in a browser or in Node.js, the program stops and reports the error:

Uncaught TypeError: Wrong type given, expected a string
    toUppercase http://localhost:5000/index.js:3
    <anonymous> http://localhost:5000/index.js:9

In addition, you can see the exact line where the error happened.

This report is a stack trace, and it's helpful for tracking down problems in your code.

The stack trace goes from bottom to top. So here:

    toUppercase http://localhost:5000/index.js:3
    <anonymous> http://localhost:5000/index.js:9

We can say:

something in the program at line 9 called toUppercase
toUppercase blew up at line 3

In addition to seeing this stack trace in the browser's console, you can access it on the stack property of the error object.

If the exception is uncaught, that is, nothing is done by the programmer to catch it, the program will crash.

When, and where you catch an exception in your code depends on the specific use case.

For example you may want to propagate an exception up in the stack to crash the program altogether. This could happen for fatal errors, when it's safer to stop the program rather than working with invalid data.

Having introduced the basics let's now turn our attention to error and exception handling in both synchronous and asynchronous JavaScript code.

Synchronous error handling

Synchronous code is most of the times straightforward, and so its error handling.

Error handling for regular functions

Synchronous code is executed in the same order in which is written. Let's take again the previous example:

function toUppercase(string) {
  if (typeof string !== "string") {
    throw TypeError("Wrong type given, expected a string");
  }

  return string.toUpperCase();
}

toUppercase(4);

Here the engine calls and executes toUppercase. All happens synchronously. To catch an exception originating by such synchronous function we can use try/catch/finally:

try {
  toUppercase(4);
} catch (error) {
  console.error(error.message);
  // or log remotely
} finally {
  // clean up
}

Usually, try deals with the happy path, or with the function call that could potentially throw.

catch instead, captures the actual exception. It receives the error object, which we can inspect (and send remotely to some logger in production).

The finally statement on the other hand runs regardless of the function's outcome: whether it failed or succeeded, any code inside finally will run.

Remember: try/catch/finally is a synchronous construct: it has now way to catch exceptions coming from asynchronous code.

Error handling for generator functions

A generator function in JavaScript is a special type of function.

It can be paused and resumed at will, other than providing a bi-directional communication channel between its inner scope and the consumer.

To create a generator function we put a star * after the function keyword:

function* generate() {
//
}

Once inside the function we can use yield to return values:

function* generate() {
  yield 33;
  yield 99;
}

The return value from a generator function is an iterator object. To pull values out a generator we can use two approaches:

calling next() on the iterator object.
iteration with for...of.

If we take our example, to get values from the generator we can do:

function* generate() {
  yield 33;
  yield 99;
}

const go = generate();

Here go becomes our iterator object when we call the generator function.

From now on we can call go.next() to advance the execution:

function* generate() {
  yield 33;
  yield 99;
}

const go = generate();

const firstStep = go.next().value; // 33
const secondStep = go.next().value; // 99

Generators also work the other way around: they can accept values and exceptions back from the caller.

In addition to next(), iterator objects returned from generators have a throw() method.

With this method we can halt the program by injecting an exception into the generator:

function* generate() {
  yield 33;
  yield 99;
}

const go = generate();

const firstStep = go.next().value; // 33

go.throw(Error("Tired of iterating!"));

const secondStep = go.next().value; // never reached

To catch such error you would wrap your code inside the generator with try/catch (and finally if needed):

function* generate() {
  try {
    yield 33;
    yield 99;
  } catch (error) {
    console.error(error.message);
  }
}

Generator functions can also throw exceptions to the outside. The mechanism for catching these exceptions is the same for catching synchronous exceptions: try/catch/finally.

Here's an example of a generator function consumed from the outside with for...of:

function* generate() {
  yield 33;
  yield 99;
  throw Error("Tired of iterating!");
}

try {
  for (const value of generate()) {
    console.log(value);
  }
} catch (error) {
  console.error(error.message);
}

/* Output:
33
99
Tired of iterating!
*/

Here we iterate the happy path inside a try block. If any exceptions occurs we stop it with catch.

Asynchronous error handling

JavaScript is synchronous by nature, being a single-threaded language.

Host environments like browsers engines augment JavaScript with a number of Web API for interacting with external systems, and for dealing with I/O bound operations.

Examples of asynchronicity in the browser are timeouts, events, Promise.

Error handling in the asynchronous world is distinct from its synchronous counterpart.

Let's see some examples.

Error handling for timers

In the beginning of your explorations with JavaScript, after learning about try/catch/finally, you might be tempted to put it around any block of code.

Consider the following snippet:

function failAfterOneSecond() {
  setTimeout(() => {
    throw Error("Something went wrong!");
  }, 1000);
}

This function throws after roughly 1 second. What's the right way to handle this exception?

The following example does not work:

function failAfterOneSecond() {
  setTimeout(() => {
    throw Error("Something went wrong!");
  }, 1000);
}

try {
  failAfterOneSecond();
} catch (error) {
  console.error(error.message);
}

As we said, try/catch is synchronous. On the other hand we have setTimeout, a browser API for timers.

By the time the callback passed to setTimeout runs, our try/catch is long gone. The program will crash because we failed to capture the exception.

They travel on two different tracks:

Track A: --> try/catch
Track B: --> setTimeout --> callback --> throw

If we don't want to crash the program, to handle the error correctly we must move try/catch inside the callback for setTimeout.

But, this approach doesn't make much sense most of the times. As we'll see later, asynchronous error handling with Promises provides a better ergonomic.

Error handling for events

HTML nodes in the Document Object Model are connected to EventTarget, the common ancestor for any event emitter in the browser.

That means we can listen for events on any HTML element in the page.

(Node.js will support EventTarget in a future release).

The error handling mechanics for DOM events follows the same scheme of any asynchronous Web API.

Consider the following example:

const button = document.querySelector("button");

button.addEventListener("click", function() {
  throw Error("Can't touch this button!");
});

Here we throw an exception as soon as the button is clicked. How do we catch it? This pattern does not work, and won't prevent the program from crashing:

const button = document.querySelector("button");

try {
  button.addEventListener("click", function() {
    throw Error("Can't touch this button!");
  });
} catch (error) {
  console.error(error.message);
}

As with the previous example with setTimeout, any callback passed to addEventListener is executed asynchronously:

Track A: --> try/catch
Track B: --> addEventListener --> callback --> throw

If we don't want to crash the program, to handle the error correctly we must move try/catch inside the callback for addEventListener.

But again, there's little of no value in doing this.

As with setTimeout, exception thrown by an asynchronous code path are un-catchable from the outside, and will crash your program.

In the next sections we'll see how Promises and async/await can ease error handling for asynchronous code.

How about onerror?

HTML elements have a number of event handlers like onclick, onmouseenter, onchange to name a few.

There is also onerror, but it has nothing to do with throw and friends.

The onerror event handler fires any time an HTML element like an <img> tag or a <script> hits a non-existent resource.

Consider the following example:

// omitted
<body>
<img src="nowhere-to-be-found.png" alt="So empty!">
</body>
// omitted

When visiting an HTML document with a missing or non-existent resource the browser's console records the error:

GET http://localhost:5000/nowhere-to-be-found.png
[HTTP/1.1 404 Not Found 3ms]

In our JavaScript we have the chance to "catch" this error with the appropriate event handler:

const image = document.querySelector("img");

image.onerror = function(event) {
  console.log(event);
};

Or better:

const image = document.querySelector("img");

image.addEventListener("error", function(event) {
  console.log(event);
});

This pattern is useful for loading alternate resources in place of missing images or scripts.

But remember: onerror, has nothing to do with throw or try/catch.

Error handling with Promise

To illustrate error handling with Promise we'll "promisify" one of our original examples. We tweak the following function:

function toUppercase(string) {
  if (typeof string !== "string") {
    throw TypeError("Wrong type given, expected a string");
  }

  return string.toUpperCase();
}

toUppercase(4);

Instead of returning a simple string, or an exception, we use respectively Promise.reject and Promise.resolve to handle error and success:

function toUppercase(string) {
  if (typeof string !== "string") {
    return Promise.reject(TypeError("Wrong type given, expected a string"));
  }

  const result = string.toUpperCase();

  return Promise.resolve(result);
}

(Technically there's nothing asynchronous in this code, but it serves well to illustrate the point).

Now that the function is "promisified" we can attach then for consuming the result, and catch for handling the rejected Promise:

toUppercase(99)
  .then(result => result)
  .catch(error => console.error(error.message));

This code will log:

Wrong type given, expected a string

In the realm of Promise, catch is the construct for handling errors.

In addition to catch and then we have also finally, similar to the finally in try/catch.

As its synchronous "relative", Promise's finally runs regardless of the Promise outcome:

toUppercase(99)
  .then(result => result)
  .catch(error => console.error(error.message))
  .finally(() => console.log("Run baby, run"));

Always keep in mind that any callback passed to then/catch/finally is handled asynchronously by the Microtask Queue. They're micro tasks with precedence over macro tasks such as events and timers.

Promise, error, and throw

As a best practice when rejecting a Promise it's convenient to provide an error object:

Promise.reject(TypeError("Wrong type given, expected a string"));

By doing so you keep error handling consistent through the codebase. Other team members can always expect to access error.message, and more important you can inspect stack traces.

In addition to Promise.reject, we can exit from a Promise chain by throwing an exception.

Consider the following example:

Promise.resolve("A string").then(value => {
  if (typeof value === "string") {
    throw TypeError("Expected a number!");
  }
});

We resolve a Promise with a string, and then the chain is immediately broken with throw.

To stop the exception propagation we use catch, as usual:

Promise.resolve("A string")
  .then(value => {
    if (typeof value === "string") {
      throw TypeError("Expected a number!");
    }
  })
  .catch(reason => console.log(reason.message));

This pattern is common in fetch, where we check the response object in search for errors:

fetch("https://example-dev/api/")
  .then(response => {
    if (!response.ok) {
      throw Error(response.statusText);
    }

    return response.json();
  })
  .then(json => console.log(json));

Here the exception can be intercepted with catch. If we fail, or decide to not catch it there, the exception is free to bubble up in the stack.

This is not bad per-se, but different environments react differently to uncaught rejections.

Node.js for example in the future will let crash any program where Promise rejections are un-handled:

DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Better you catch them!

Error handling for "promisified" timers

With timers or events it's not possible to catch exceptions thrown from a callback. We saw an example in the previous section:

function failAfterOneSecond() {
  setTimeout(() => {
    throw Error("Something went wrong!");
  }, 1000);
}

// DOES NOT WORK
try {
  failAfterOneSecond();
} catch (error) {
  console.error(error.message);
}

A solution offered by Promise consists in the "promisification" of our code. Basically, we wrap our timer with a Promise:

function failAfterOneSecond() {
  return new Promise((_, reject) => {
    setTimeout(() => {
      reject(Error("Something went wrong!"));
    }, 1000);
  });
}

With reject we set off a Promise rejection, which carries an error object.

At this point we can handle the exception with catch:

failAfterOneSecond().catch(reason => console.error(reason.message));

Note: it's common to use value as the returning value from a Promise, and reason as the returning object from a rejection.

Node.js has a utility called promisify to ease the "promisification" of old-style callback APIs.

Error handling in Promise.all

The static method Promise.all accepts an array of Promise, and returns an array of results from all resolving Promise:

const promise1 = Promise.resolve("All good!");
const promise2 = Promise.resolve("All good here too!");

Promise.all([promise1, promise2]).then((results) => console.log(results));

// [ 'All good!', 'All good here too!' ]

If any of these Promise rejects, Promise.all rejects with the error from the first rejected Promise.

To handle these situations in Promise.all we use catch, as we did in the previous section:

const promise1 = Promise.resolve("All good!");
const promise2 = Promise.reject(Error("No good, sorry!"));
const promise3 = Promise.reject(Error("Bad day ..."));

Promise.all([promise1, promise2, promise3])
  .then(results => console.log(results))
  .catch(error => console.error(error.message));

To run a function regardless of the outcome of Promise.all, again, we can use finally:

Promise.all([promise1, promise2, promise3])
  .then(results => console.log(results))
  .catch(error => console.error(error.message))
  .finally(() => console.log("Always runs!"));

Error handling in Promise.any

We can consider Promise.any (Firefox > 79, Chrome > 85) as the opposite of Promise.all.

Whereas Promise.all returns a failure even if a single Promise in the array rejects, Promise.any gives us always the first resolved Promise (if present in the array) regardless of any rejection occurred.

In case instead all the Promise passed to Promise.any reject, the resulting error is an AggregateError. Consider the following example:

const promise1 = Promise.reject(Error("No good, sorry!"));
const promise2 = Promise.reject(Error("Bad day ..."));

Promise.any([promise1, promise2])
  .then(result => console.log(result))
  .catch(error => console.error(error))
  .finally(() => console.log("Always runs!"));

Here we handle the error with catch. The output of this code is:

AggregateError: No Promise in Promise.any was resolved
Always runs!

The AggregateError object has the same properties of a basic Error, plus an errors property:

//
  .catch(error => console.error(error.errors))
//

This property is an array of each individual error produced by the reject:

[Error: "No good, sorry!, Error: "Bad day ..."]

Error handling in Promise.race

The static method Promise.race accepts an array of Promise:

const promise1 = Promise.resolve("The first!");
const promise2 = Promise.resolve("The second!");

Promise.race([promise1, promise2]).then(result => console.log(result));

// The first!

The result is the first Promise who wins the "race".

How about rejections then? If the rejecting Promise is not the first to appear in the input array, Promise.race resolves:

const promise1 = Promise.resolve("The first!");
const rejection = Promise.reject(Error("Ouch!"));
const promise2 = Promise.resolve("The second!");

Promise.race([promise1, rejection, promise2]).then(result =>
  console.log(result)
);

// The first!

If the rejection instead appears as the first element of the array, Promise.race rejects, and we must catch the rejection:

const promise1 = Promise.resolve("The first!");
const rejection = Promise.reject(Error("Ouch!"));
const promise2 = Promise.resolve("The second!");

Promise.race([rejection, promise1, promise2])
  .then(result => console.log(result))
  .catch(error => console.error(error.message));

// Ouch!

Error handling in Promise.allSettled

Promise.allSettled is an ECMAScript 2020 addition to the language.

There is not so much to handle with this static method since the result will always be a resolved Promise, even if one or more input Promise rejects.

Consider the following example:

const promise1 = Promise.resolve("Good!");
const promise2 = Promise.reject(Error("No good, sorry!"));

Promise.allSettled([promise1, promise2])
  .then(results => console.log(results))
  .catch(error => console.error(error))
  .finally(() => console.log("Always runs!"));

We pass to Promise.allSettled an array consisting of two Promise: one resolved and another rejected.

In this case catch will never be hit. finally instead runs.

The result of this code, logged in then is:

[
  { status: 'fulfilled', value: 'Good!' },
  {
    status: 'rejected',
    reason: Error: No good, sorry!
  }
]

Error handling for async/await

async/await in JavaScript denotes asynchronous functions, but from a reader standpoint they benefit from all the readability of synchronous functions.

To keep things simple we'll take our previous synchronous function toUppercase, and we transform it to an asynchronous function by putting async before the function keyword:

async function toUppercase(string) {
  if (typeof string !== "string") {
    throw TypeError("Wrong type given, expected a string");
  }

  return string.toUpperCase();
}

Just by prefixing a function with async we cause the function to return a Promise. That means we can chain then, catch, and finally after the function call:

async function toUppercase(string) {
  if (typeof string !== "string") {
    throw TypeError("Wrong type given, expected a string");
  }

  return string.toUpperCase();
}

toUppercase("abc")
  .then(result => console.log(result))
  .catch(error => console.error(error.message))
  .finally(() => console.log("Always runs!"));

When we throw from an async function the exception becomes cause of rejection for the underlying Promise.

Any error can be intercepted with catch from the outside.

Most important, in addition to this style we can use try/catch/finally, much as we would do with a synchronous function.

In the following example we call toUppercase from another function, consumer, which conveniently wraps the function call with try/catch/finally:

async function toUppercase(string) {
  if (typeof string !== "string") {
    throw TypeError("Wrong type given, expected a string");
  }

  return string.toUpperCase();
}

async function consumer() {
  try {
    await toUppercase(98);
  } catch (error) {
    console.error(error.message);
  } finally {
    console.log("Always runs!");
  }
}

consumer(); // Returning Promise ignored

The output is:

Wrong type given, expected a string
Always runs!

On the same topic: How to Throw Errors From Async Functions in JavaScript?

Error handling for async generators

Async generators in JavaScript are generator functions capable of yielding Promises instead of simple values.

They combine generator functions with async. The result is a generator function whose iterator objects expose a Promise to the consumer.

To create an async generator we declare a generator function with the star *, prefixed with async:

async function* asyncGenerator() {
  yield 33;
  yield 99;
  throw Error("Something went wrong!"); // Promise.reject
}

Being based on Promise, the same rules for error handling apply here. throw inside an async generator causes a Promise rejection, which we intercept with catch.

To pull Promises out an async generators we can use two approaches:

then handlers.
async iteration.

From the above example we know for sure there will be an exception after the first two yield. This means we can do:

const go = asyncGenerator();

go.next().then(value => console.log(value));
go.next().then(value => console.log(value));
go.next().catch(reason => console.error(reason.message));

The output from this code is:

{ value: 33, done: false }
{ value: 99, done: false }
Something went wrong!

The other approach uses async iteration with for await...of. To use async iteration we need to wrap the consumer with an async function.

Here's the complete example:

async function* asyncGenerator() {
  yield 33;
  yield 99;
  throw Error("Something went wrong!"); // Promise.reject
}

async function consumer() {
  for await (const value of asyncGenerator()) {
    console.log(value);
  }
}

consumer();

And as with async/await we handle any potential exception with try/catch:

async function* asyncGenerator() {
  yield 33;
  yield 99;
  throw Error("Something went wrong!"); // Promise.reject
}

async function consumer() {
  try {
    for await (const value of asyncGenerator()) {
      console.log(value);
    }
  } catch (error) {
    console.error(error.message);
  }
}

consumer();

The output of this code is:

33
99
Something went wrong!

The iterator object returned from an asynchronous generator function has also a throw() method, much like its synchronous counterpart.

Calling throw() on the iterator object here won't throw an exception, but a Promise rejection:

async function* asyncGenerator() {
  yield 33;
  yield 99;
  yield 11;
}

const go = asyncGenerator();

go.next().then(value => console.log(value));
go.next().then(value => console.log(value));

go.throw(Error("Let's reject!"));

go.next().then(value => console.log(value)); // value is undefined

To handle this situation from the outside we can do:

go.throw(Error("Let's reject!")).catch(reason => console.error(reason.message));

But let's not forget that iterator objects throw() send the exception inside the generator. This means we can also apply the following pattern:

async function* asyncGenerator() {
  try {
    yield 33;
    yield 99;
    yield 11;
  } catch (error) {
    console.error(error.message);
  }
}

const go = asyncGenerator();

go.next().then(value => console.log(value));
go.next().then(value => console.log(value));

go.throw(Error("Let's reject!"));

go.next().then(value => console.log(value)); // value is undefined

Error handling in Node.js

Synchronous error handling in Node.js

Synchronous error handling in Node.js does not differ too much from what we saw so far.

For synchronous code, try/catch/finally works fine.

However, things get interesting if we take a glance at the asynchronous world.

Asynchronous error handling in Node.js: the callback pattern

For asynchronous code, Node.js strongly relies on two idioms:

the callback pattern.
event emitters.

In the callback pattern, asynchronous Node.js APIs accept a function which is handled through the event loop and executed as soon as the call stack is empty.

Consider the following code:

const { readFile } = require("fs");

function readDataset(path) {
  readFile(path, { encoding: "utf8" }, function(error, data) {
    if (error) console.error(error);
    // do stuff with the data
  });
}

If we extract the callback from this listing, we can see how it is supposed to deal with errors:

//
function(error, data) {
    if (error) console.error(error);
    // do stuff with the data
  }
//

If any errors arises from reading the given path with fs.readFile, we get an error object.

At this point we can:

simply log the error object as we did.
throw an exception.
pass the error to another callback.

To throw an exception we can do:

const { readFile } = require("fs");

function readDataset(path) {
  readFile(path, { encoding: "utf8" }, function(error, data) {
    if (error) throw Error(error.message);
    // do stuff with the data
  });
}

However, as with events and timers in the DOM, this exception will crash the program. The following attempt to stop it with try/catch won't work:

const { readFile } = require("fs");

function readDataset(path) {
  readFile(path, { encoding: "utf8" }, function(error, data) {
    if (error) throw Error(error.message);
    // do stuff with the data
  });
}

try {
  readDataset("not-here.txt");
} catch (error) {
  console.error(error.message);
}

Passing the error to another callback is the preferred option, if we don't want to crash the program:

const { readFile } = require("fs");

function readDataset(path) {
  readFile(path, { encoding: "utf8" }, function(error, data) {
    if (error) return errorHandler(error);
    // do stuff with the data
  });
}

Here errorHandler is what the name suggests, a simple function for error handling:

function errorHandler(error) {
  console.error(error.message);
  // do something with the error:
  // - write to a log.
  // - send to an external logger.
}

Asynchronous error handling in Node.js: event emitters

Much of what you do in Node.js is based on events. Most of the times you interact with an emitter object and some observers listening for messages.

Any event-driven module (like net for example) in Node.js extends a root class named EventEmitter.

EventEmitter in Node.js has two fundamental methods: on and emit.

Consider this simple HTTP server:

const net = require("net");

const server = net.createServer().listen(8081, "127.0.0.1");

server.on("listening", function () {
  console.log("Server listening!");
});

server.on("connection", function (socket) {
  console.log("Client connected!");
  socket.end("Hello client!");
});

Here we listen for two events: listening and connection.

In addition to these events, event emitters expose also an error event, fired in case of errors.

If you run this code listening on port 80 instead of the previous example, you'll get an exception:

const net = require("net");

const server = net.createServer().listen(80, "127.0.0.1");

server.on("listening", function () {
  console.log("Server listening!");
});

server.on("connection", function (socket) {
  console.log("Client connected!");
  socket.end("Hello client!");
});

Output:

events.js:291
      throw er; // Unhandled 'error' event
      ^

Error: listen EACCES: permission denied 127.0.0.1:80
Emitted 'error' event on Server instance at: ...

To catch it we can register an event handler for error:

server.on("error", function(error) {
  console.error(error.message);
});

This will print:

listen EACCES: permission denied 127.0.0.1:80

In addition, the program won't crash.

To learn more on the topic, consider also reading "Error handling in Node.js".

Wrapping up

In this guide we covered error handling in JavaScript for the whole spectrum, from simple synchronous code, to advanced asynchronous primitives.

There are many ways in which an exception can manifest in our JavaScript programs.

Exceptions from synchronous code are the most straightforward to catch. Exceptions from asynchronous code paths instead can be tricky to deal with.

In the meantime, new JavaScript APIs in the browser are almost all headed toward Promise. This pervasive pattern makes easier to handle exceptions with then/catch/finally, or with try/catch for async/await.

After reading this guide you should be able to recognize all the different situations that may arise in your programs, and catch your exceptions correctly.

Thanks for reading and stay tuned!

Redux course for beginners, lesson 1: introduction, history, and architecture

Valentino Gagliardi — Wed, 04 Mar 2020 12:05:02 +0000

Redux Visualized

Valentino Gagliardi — Tue, 28 Jan 2020 16:53:08 +0000

While revamping my slides for a Redux workshop I came up with this animation:

If you think about it, Redux is not so hard to grasp :-)

For the occasion I updated this React Redux tutorial as well.

Happy coding!

Understanding the Firefox Dev Tools Architecture, and How to Start Contributing to Firefox

Valentino Gagliardi — Tue, 28 Jan 2020 08:25:15 +0000

This year I decided to contribute more to open source projects and since I've made the resolution to switch to Firefox from Chrome I've also got the chance to find an interesting bug to work on.

In this post I want to share my learnings in the hope that more people could start contributing to Firefox.

Contributing to Firefox: a bit of backstory

In Firefox Dev Tools I wanted a "Store as global variable" option in the Scopes panel. The Scopes panel appears in the debugger tab when you stop the script with a breakpoint, much like Chrome Dev Tools.

As of now the only thing that you can do on an object in the panel is adding a watchpoint, that is, a watcher which fires on properties get and set. You can read more about watchpoints here.

With the idea in mind I reached the Firefox Dev Tools account on Twitter and the team replied promptly with a workaround:

𝕡𝕒𝕥𝕣𝕚𝕔𝕜

@patrickbrosset

@gagliardi_vale @FirefoxDevTools This should work:
- open the console while paused
- type and execute inputList
- right-click on the returned value and choose "store as global variable"

12:26 PM - 08 Jan 2020

But it turns out there was already a bug filled for the feature I wanted: Variables/values in Scopes should offer reps context menu.

So I jumped in and I tried to work on the bug. But before going into details let's see how to get started in contributing to Firefox Dev Tools.

Enjoy!

Contributing to Firefox: how to get started

If you want to start contributing to Firefox your first stop must be this website: Firefox Developer Tools. Also, join the Slack channel.

Read every section of the guide carefully and try to not skip anything. There are great insights into the Firefox Developer Tools architecture, but I know, you want to get your hands dirty.

Clone the repo (you'll need Mercurial) and make yourself comfortable with the codebase. It is big, and you'll need a lot of reading through the source code before starting to grasp anything.

Don't try to rush it, don't be like me. Your first patch will be likely need changes because you don't know how Firefox Dev Tools works under the hood.

But that's a good thing because learning from such a big codebase and interacting with more expert developers is invaluable.

After cloning the repo and glancing through the code it will come the time to create your first patch. But first, let's demystify the architecture of Firefox Dev Tools.

Firefox Dev Tools frontend: the architecture

The Firefox Dev Tools frontend is a big React/Redux app made of many little apps. For that reason you'll need a good understanding of Redux, namely:

reducers
actions
middlewares
store enhancers
selectors

Every tab in the developer tool is a React/Redux app which lives in a separate folder in mozilla-central/devtools/client. Here's a breakdown:

$ tree -d -L 1 devtools/client/
devtools/client/
├── aboutdebugging
├── accessibility
├── application
├── bin
├── debugger
├── dom
├── framework
├── inspector
├── jsonview
├── locales
├── memory
├── netmonitor
├── performance
├── performance-new
├── preferences
├── responsive
├── shared
├── storage
├── styleeditor
├── themes
├── webconsole
├── webreplay
└── whats-new

You can recognize each tab: debugger, inspector, memory, netmonitor, performance, and so on.

As you'll find out by going through, each application follows more or less the same directory structure. Inside each app you'll find:

React components
Redux actions, reducers, and so on
tests
utilities

There are also a lot of other folders which I won't cover here, but you can learn more by checking out this link.

Worth noting, each tab in the developer tool is also a panel. The debugger for example is defined in devtools/client/debugger/panel.js.

On top of that, the entirety of each panel grouped together is called toolbox and is defined in devtools/client/framework/toolbox.js.

The toolbox is in practice the entire dev tool window. Each panel gets access to the toolbox, which is injected at runtime. The debugger panel for example takes toolbox as a parameter:

// devtools/client/debugger/panel.js
function DebuggerPanel(iframeWindow, toolbox) {
  this.panelWin = iframeWindow;
  this.panelWin.L10N = L10N;
  this.toolbox = toolbox;
}

Same is true for the inspector panel:

// devtools/client/inspector/panel.js
function InspectorPanel(iframeWindow, toolbox) {
  this._inspector = new iframeWindow.Inspector(toolbox);
}

Turns out, this frontend is only the tip of the iceberg. There is also a Firefox Dev Tools backend. But we'll get to it later: let me show you my first attempt to implement "Store as global variable".

TIP: to browse the codebase you can also use Searchfox.

Store as global variable: a first attempt

After building Firefox (instructions here) I started off by looking around in devtools/client/debugger.

I found out my entry point in devtools/client/debugger/src/components/SecondaryPanes/Scopes.js.

This component is responsible for displaying Scopes, and has already a context menu for adding watchpoints.

I started off by adding a new entry named "Store as global variable" to the menu. Here's the relevant part:

  onContextMenu = (event: any, item: any) => {
    const {
      addWatchpoint,
      removeWatchpoint,
      storeAsGlobalVariable,
    } = this.props;

    // omitted for brevity

    const storeAsGlobalVariableItem = {
      id: "node-menu-store-as-global",
      label: storeAsGlobalLabel,
      disable: false,
      click: () => storeAsGlobalVariable(item),
    };

    const menuItems = [storeAsGlobalVariableItem, watchpointsSubmenuItem];
    showMenu(event, menuItems);
  };

As you can see the menu is shown in response to the contextmenu event.

The Scopes component is connected to the Redux store, that means it has access to actions.

So I've added a new action in devtools/client/debugger/packages/devtools-reps/src/object-inspector/actions.js:

function storeAsGlobalVariable(item) {
  return async function({ panel }) {

  // omit for brevity

    await panel.storeAsGlobalVariable(selectedObjectActor);
  }
}

With the help of Nicolas from the Firefox team I've got a first proof of concept. There is a lot more going on in this action and I've omitted some code for brevity's sake, but if you're familiar with Redux you can recognize a thunk.

Thunk is a middleware, and as for the project is configured each thunk has access to the panel.

If you remember from the previous section a panel is a dev tools tab. In this case panel is the debugger tab.

What happens in this action is that it takes item as a parameter and computes a selectedObjectActor. In brief, the item is the actual thing that has been clicked in the Scopes panel. selectedObjectActor instead is a reference (a string) that we need to pass to the server.

Yes, there is a server. In the next section I'll describe what I learned about it, while trying to explain how the entire architecture works.

Worth noting, most of the codebase is type annotated with Flow. I'm not familiar with it, but it's not so different from TypeScript, which I use most of the times.

By the way, since it's all JavaScript don't forget to install the dependencies for any given panel you're working on (you'll need to install Yarn). In my case:

cd ~/Code/mozilla-central/devtools/client/debugger
yarn install

After making changes don't forget to build the bundle and to rebuild Firefox as well, again in my case:

cd ~/Code/mozilla-central/devtools/client/debugger
node bin/bundle.js
~/Code/mozilla-central/mach build

Now you can run Firefox with:

~/Code/mozilla-central/mach run
# or simply ./mach run if you're in the same folder

The build produces a version of Firefox called Nightly. I use Firefox Developer Edition for development, Nightly is similar, but includes all the newest features. To learn more about all the Firefox versions check out out this video by Jen Simmons.

Firefox Dev Tools server: the architecture

As I said, the Firefox Dev Tools frontend is only the tip of the iceberg. Each panel in fact is a client, and has a corresponding actor on the server.

That means there is the webconsole actor, the inspector actor, the network actor, and so on. It's not a 1:1 match, the debugger panel for example talks to the thread actor, but, you got the point.

Actors are defined in devtools/server/actors. The term actor refers also to any object that is being debugged (the debuggee).

In my action for example I passed a selectedObjectActor:

function storeAsGlobalVariable(item) {
  return async function({ panel }) {

  // omit for brevity

    await panel.storeAsGlobalVariable(selectedObjectActor);
  }
}

Here selectedObjectActor will be a string like:

"server0.conn0.child1/obj464"

This string is used by the client to tell the server "ehi, please, tell me about this object and give me back the result".

The thing to keep in mind then is that it's not the Dev Tool frontend that evaluates the actual JavaScript code.

Everything lives on the server-side and client panels talk to the server via JSON objects. Then it's the server-side of Dev Tools that talks to the Firefox JavaScript engine (SpiderMonkey) and reports back to the client panel if asked.

You can read more about the Debugging protocol architecture here.

Now after introducing server and panels it's time to see how all these components interact.

Firefox Dev Tools: from the toolbox to the server

Let's take a look again at the action:

function storeAsGlobalVariable(item) {
  return async function({ panel }) {

  // omit for brevity

    await panel.storeAsGlobalVariable(selectedObjectActor);
  }
}

panel is a thunk argument and maps to devtools/client/debugger/panel.js, the actual debugger panel.

If you look at the source code you can see that the constructor function for DebuggerPanel takes toolbox among the parameters. Toolbox is at devtools/client/framework/toolbox.js and has an openSplitConsole method.

That was exactly what I was searching for. Also, by looking at how the "Use in console" feature works in the inspector I tried to replicate the same feature. That lead me to create a new method in devtools/client/debugger/panel.js, storeAsGlobalVariable:

  storeAsGlobalVariable: async function(selectedObjectActor) {
    await this.toolbox.openSplitConsole();
    const { hud } = this.toolbox.getPanel("webconsole");

    const script = `{ let i = 0;
      while (this.hasOwnProperty("temp" + i) && i < 1000) {
        i++;
      }
      this["temp" + i] = _self;
      "temp" + i;
    }`;

    const thread = this._selectors.getCurrentThread(this._getState());
    const { id } = this._selectors.getSelectedFrame(this._getState(), thread);

    const options = {
      selectedObjectActor,
      frameActor: id,
    };

    const res = await hud.evaluateJSAsync(script, options);
    hud.setInputValue(res.result);
  },

This method bears a bit of explanation. First, it receives selectedObjectActor from the Redux action. Next up it opens the split console, that is, the little console you can see some times in the Dev Tool.

It also hooks the webconsole panel which returns itself as hud.

You can also notice the script for creating a new global variable. Before forwarding the script for evaluation to the server this method computes also the frameActor id.

We already saw what is selectedObjectActor, a reference to an object in the Scope panel. The frameActor id is another string:

server0.conn0.child1/frame26

In particular it is a reference to the context where we are currently in. In other words if I pause a script with the debugger, that context where the script is paused is the actual frame.

These two references, selectedObjectActor and frameActor are needed by an actor which lives in the server. More on this in the next section.

Store as global variable: the working POC

This is the part that required the help of Nicolas. Let's look again at the previous method:

  storeAsGlobalVariable: async function(selectedObjectActor) {
    // omitted for brevity
    const options = {
      selectedObjectActor,
      frameActor: id,
    };

    const res = await hud.evaluateJSAsync(script, options);
    hud.setInputValue(res.result);
  },

Long story short, the call to hud.evaluateJSAsync is forwarded to the server and lands on evalWithDebugger defined in devtools/server/actors/webconsole/eval-with-debugger.js.

This function required a tweak because it prevented _self from being defined. _self is used by the server as a placeholder during evaluation. Here _self is replaced by the actual object:

    const script = `{ let i = 0;
      while (this.hasOwnProperty("temp" + i) && i < 1000) {
        i++;
      }
      this["temp" + i] = _self;
      "temp" + i;
    }`;

When the evaluation is done the appropriate object is assigned to a temporary variable named temp0, temp1, and so on.

And with the fix in place (you can see the history here) I've got a working proof of concept:

When the script is paused and the Scopes panel is opened I can (in my build for now) save an object in a global variable after the split panel opens.

Conclusions

Firefox is a fantastic browser and the developers tools are second to none.

At the time of writing the feature is far from finished, there are some edge cases to cover. But I was eager to share this little open source story to clarify my understanding about such a big system.

Acknowledgments

Thanks to Jason Laster and Nicolas Chevobbe for answering to my questions and providing guidance on my first patch for Firefox.

Cypress Tutorial for Beginners: Getting started with End to End Testing

Valentino Gagliardi — Mon, 27 Jan 2020 10:14:54 +0000

End to End testing can be slow and boring. Cypress has changed the way we test. Learn how to test your applications in this tutorial.

Cypress Tutorial for Beginners: requirements

To follow along you need a working installation of Node.js on your system. Also, a basic understanding of a newer version of JavaScript is a plus.

What is Cypress? What is End to End Testing?

End to End Testing, or UI testing is one the many approaches for testing a web application.

An end to end test is supposed to check whether a web application works as expected or not, by testing the so called user flow.

Is End to End Testing important? Yes it is. But nobody likes E2E tests. They can be slow, cumbersome and expensive to write.

On the other hand testing gives you confidence. Would you ship a broken product to your users on purpose?

Enter Cypress: a Javascript End to End testing framework. It will make your life easier.

Disclaimer

Before purists yell at me: I know the subtle line between End to End Testing, UI testing, integration testing, and so on.

And for you dear reader: testing terminology is so blurry that I can't even .... If it's your first time with JavaScript testing I recommend reading Jest Tutorial for Beginners: Getting Started With Jest for JavaScript Testing for an introduction on unit testing and terminology.

When you're done come back here for End to End Testing.

Cypress tutorial for beginners: setting up the project

To start off create a new folder, I called mine cypress-tutorial, move into it an initialize a new JavaScript project:

mkdir cypress-tutorial && cd $_
npm init -y

Inside this folder create two new files. An HTML document in index.html:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>Cypress tutorial for beginners</title>
  </head>
  <body>
    <main>
      <form>
        <div>
          <label for="name">Name</label>
          <input type="name" required name="name" id="name" />
        </div>
        <div>
          <label for="email">Email</label>
          <input type="email" required name="email" id="email" />
        </div>
        <div>
          <label for="message">Your message</label>
          <textarea id="message" name="message" required></textarea>
        </div>
        <div>
          <button type="submit">SEND</button>
        </div>
      </form>
    </main>
  </body>
  <script src="form.js"></script>
</html>

It is an HTML form with a bunch on inputs and a textarea. Next up create a JavaScript file in form.js with a minimal logic for handling form submission:

const form = document.forms[0];

form.addEventListener("submit", event => {
  event.preventDefault();
});

Note that I won't add styles to keep things simple. And with this simple project in place we're ready to install Cypress.

Installing Cypress

To install Cypress, still in the project folder, run:

npm i cypress --save-dev

Give it a minute (it needs to download the binary) and then run:

node_modules/.bin/cypress open

Cypress will start for the first time and a bunch of new folder will appear in your project. You can remove the example folder, but look a moment into it because there are some nice examples.

Close the window for now and head over the next section.

Starting the project

To serve the project on the local machine make sure to have a newer version of Node.js installed and then run:

npx serve

This will spin a development server at http://localhost:5000/. Head over the link and you should see our form:

serve is a nice NPM package for development. And now time to write our first test!

Cypress tutorial for beginners: writing your first test

Create a new file in cypress/integration/form.spec.js and write your first block:

describe("Form test", () => {
  //
});

describe is a Cypress method (borrowed from Mocha) for containing one or more related tests. Every time you start writing a new suite of tests for a functionality wrap it in a describe block.

As you can see it takes two arguments: a string for describing the test suite and a callback function for wrapping the actual test.

Next up we're going to meet another function called it which is the actual test block:

describe("Form test", () => {
  it("Can fill the form", () => {
    //
  });
});

If you know already Jest you may recall that it takes it or test interchangeably. That's not the case with Cypress. it is the only accepted block.

And now time for a smoke test! In the it block write:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");
  });
});

Here cy is Cypress itself and visit is a Cypress method for browsing to a given path.

get instead is a method for selecting elements in the page. With this code we tell Cypress "go grab the form in the page".

In a minute we'll see Cypress in action, but first, a bit of configuration!

Configuring Cypress

To streamline things a bit we're going to configure Cypress. To begin with open up package.json and create a script named e2e pointing to the Cypress binary:

  "scripts": {
    "e2e": "cypress open"
  },

Next up open cypress.json and configure the base url:

{
  "baseUrl": "http://localhost:5000"
}

With this option we tell Cypress to visit our development url. (5000 is the default port for the serve package).

And now we're ready to launch your first test!

Cypress tutorial for beginners: running the test

Ready? With the development server still running in a terminal:

npx serve

open up another terminal and run:

npm run e2e

You should see Cypress opening a browser and going through the page:

That's your first test passing! Both visit and get are Cypress commands which act also as implicit assertions, that is, if the element is in the page Cypress will consider the test passed.

Now let's continue extending your test to see if the user can fill the form:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");

    cy.get('input[name="name"]').type("Molly");
  });
});

Here's another Cypress command: type, which unsurprisingly types into our first text input. Also notice the CSS selector for getting the input element.

While there let's also add another command: should. This command creates an assertion and is used for example to check if an input is updating its state as expected:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");

    cy.get('input[name="name"]')
      .type("Molly")
      .should("have.value", "Molly");
  });
});

Notice have.value. If you're new to this concept you can learn more about assertions here.

And with a minimal test in place let's continue in the next section.

Cypress tutorial for beginners: more tests and a submit

To continue our test we can check the email input:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");

    cy.get('input[name="name"]')
      .type("Molly")
      .should("have.value", "Molly");

    cy.get('input[name="email"]')
      .type("molly@dev.dev")
      .should("have.value", "molly@dev.dev");
  });
});

Also we can type into the textarea:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");

    cy.get('input[name="name"]')
      .type("Molly")
      .should("have.value", "Molly");

    cy.get('input[name="email"]')
      .type("molly@dev.dev")
      .should("have.value", "molly@dev.dev");

    cy.get("textarea")
      .type("Mind you if I ask some silly question?")
      .should("have.value", "Mind you if I ask some silly question?");
  });
});

If you left Cypress open the test should watch your changes and run automatically:

How nice! As icing on the cake let's test form submission with submit:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");

    cy.get('input[name="name"]')
      .type("Molly")
      .should("have.value", "Molly");

    cy.get('input[name="email"]')
      .type("molly@dev.dev")
      .should("have.value", "molly@dev.dev");

    cy.get("textarea")
      .type("Mind you if I ask some silly question?")
      .should("have.value", "Mind you if I ask some silly question?");

    cy.get("form").submit();
  });
});

The test should keep passing without any problem. One thing you can notice are these self-describing commands: type, submit. It's plain english.

Now let's go a bit fancy in the next section with XHR request testing.

Stubbing XHR requests with Cypress

Among all the things Cypress is also able to intercept AJAX request. This approach is called stubbing.

Stubbing is convenient when working in development where you can choose to return fake response to your AJAX requests.

To demonstrate this feature let's add a new piece of code inside our test:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");

    // omitted for brevity

    cy.server();
    cy.route({
      url: "/users/**",
      method: "POST",
      response: { status: "Saved", code: 201 }
    });

    cy.get("form").submit();
  });
});

Here cy.server spins up a "virtual" server while cy.route is used for configuring a fake API endpoint.

Now let's add another test to check things out: after the user submits the form we want to test that the fake API is responding. Why so?

Stubbing is useful because we can bypass the real API altogether in development. Let's extend the test with cy.contains:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");

    // omitted for brevity

    cy.server();
    cy.route({
      url: "/users/**",
      method: "POST",
      response: { status: "Form saved!", code: 201 }
    });

    cy.get("form").submit();

    cy.contains("Form saved!");
  });
});

The test is expected to fail because there is no logic in place for sending the form to an API. In the next section we'll make the test pass.

Sending form data to an API

Bear with me in this section! At the time of writing Cypress couldn't intercept Fetch requests. We're going to use XMLHttpRequest instead. (Check this out for an introduction).

Open up form.js and implement the logic:

const form = document.forms[0];

form.addEventListener("submit", event => {
  event.preventDefault();
  new FormData(form);
});

document.addEventListener("formdata", event => {
  const body = Object.fromEntries(event.formData.entries());
  const jsonBody = JSON.stringify(body);
  const request = new XMLHttpRequest();
  request.open("POST", "https://jsonplaceholder.typicode.com/users/");
  request.send(jsonBody);
});

In this snippet I'm using the formdata event. The event is dispatched when we call new FormData.

In the event listener we build an object with fromEntries (ECMAScript 2019). The we send the data to an API.

To make the test pass we also need to get the response back from the API and save it to the document. For doing so we can listen on the onload event of XMLHttpRequest:

// omit
document.addEventListener("formdata", event => {
  const body = Object.fromEntries(event.formData.entries());
  const jsonBody = JSON.stringify(body);
  const request = new XMLHttpRequest();
  request.open("POST", "https://jsonplaceholder.typicode.com/users/");
  request.send(jsonBody);
  // get the response
  request.onload = function() {
    const jsonResponse = JSON.parse(this.response);
  };
});

Finally we can dangerously (just for keeping things simple) save the response in the page (please don't never ever do this in a serious codebase):

// omit
  request.onload = function() {
    const jsonResponse = JSON.parse(this.response);
    document.body.innerHTML += `Response from the server: ${jsonResponse.status}`;
  };

And now it's time to see the test passing!

Stubbing XHR requests with Cypress: a passing test

To recap here's the complete test in cypress/integration/form.spec.js:

describe("Form test", () => {
  it("Can fill the form", () => {
    cy.visit("/");
    cy.get("form");

    cy.get('input[name="name"]')
      .type("Molly")
      .should("have.value", "Molly");

    cy.get('input[name="email"]')
      .type("molly@dev.dev")
      .should("have.value", "molly@dev.dev");

    cy.get("textarea")
      .type("Mind you if I ask some silly question?")
      .should("have.value", "Mind you if I ask some silly question?");

    cy.server();
    cy.route({
      url: "/users/**",
      method: "POST",
      response: { status: "Form saved!", code: 201 }
    });

    cy.get("form").submit();

    cy.contains("Form saved!");
  });
});

And here's the complete code for form.js:

const form = document.forms[0];

form.addEventListener("submit", event => {
  event.preventDefault();
  new FormData(form);
});

document.addEventListener("formdata", event => {
  const body = Object.fromEntries(event.formData.entries());
  const jsonBody = JSON.stringify(body);
  const request = new XMLHttpRequest();
  request.open("POST", "https://jsonplaceholder.typicode.com/users/");
  request.send(jsonBody);
  // get the response
  request.onload = function() {
    const jsonResponse = JSON.parse(this.response);
    document.body.innerHTML += `Response from the server: ${jsonResponse.status}`;
  };
});

One thing to keep in mind is that the real API would not likely return the same shape of our fake stub. When developing a real app you do need to adapt your tests to the real system.

But for now we're good and if you kept Cypress opened you should already see the test passing:

You can see a routes section at the top left and XHR stub in the test output, sign that Cypress has intercepted the POST request.

That's one of the best features of Cypress, not counting the dozens of commands and assertions ready to use.

With stubbing we can conclude the tutorial. Great job!

Cypress tutorial for beginners: conclusions

I hope you learned something new with this tutorial and you'll apply these concepts to your next project! Testing is important!

End to end testing shouldn't be hard: Cypress makes it pleasing and enjoyable. The team at Cypress really nailed it.

Plus, the documentation is pure gold: Cypress Docs are filled up with best practices and examples.

Thanks for reading!

Originally published on my blog

Resources

Stefano Magni has a lot of nice testing tips in his UI testing best practices.

Asynchronous tasks in Django with Django Q

Valentino Gagliardi — Thu, 16 Jan 2020 23:35:28 +0000

Learn how to use Django Q, the task queue, with the Redis broker for offloading long running tasks in your Django applications.

Requirements

To follow along you'll need:

an Heroku account if you want to use their Redis add-on
the Heroku CLI installed on your system
a newer version of Python, ideally 3.6 or 3.7
Git

Deployment on Heroku is optional, and you can use your own Redis instance if you've already got one locally.

Setting up the project

And now let's get to work! To start off we're going to create a new Python virtual environment alongside with a Django installation:

mkdir django-q-django && cd $_
python3 -m venv venv
source venv/bin/activate
pip install django

Next up we're going to create a new Django project from a template:

django-admin startproject \
    --template https://github.com/valentinogagliardi/ponee/archive/master.zip \
    --name=Procfile \
    --extension=py,example django_q_django .

If you're wondering what I'm doing here, this is a template of mine. I've got a link in the resources with a tutorial for creating your own Django project template.

Now let's install the dependencies with pip:

pip install -r ./requirements/dev.txt

We need also to provide some environment variables for our project:

mv .env.example .env

and finally we're going to run Django migrations:

python manage.py makemigrations
python manage.py migrate

at this point you should be able to run the development server:

python manage.py runserver

Now before moving to Django Q let's see what problem is it meant to solve.

Asynchronous tasks in Django with Django Q: the problem with synchronous code

The main issue for Python and Django is that they're synchronous. It's not a bad thing per se, and there are a lot of ways to circumvent it.

Python, on which Django builds on, is single threaded by nature. Single threaded means that the language interpreter can only run your code in sequence.

The practical implication is that any view in a Django application can get stuck if one or more operations take too much to complete.

To demonstrate the concept let's create a new Django application inside our project:

django-admin startapp demo_app

In this app we're going to define a view which returns a simple JSON response:

# demo_app/views.py

from django.http import JsonResponse

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    return JsonResponse(json_payload)

And let's also create the corresponding url:

# demo_app/urls.py

from django.urls import path
from .views import index

urlpatterns = [
    path("demo-app/", index)
]

Don't forget to wire up the url for the new app:

# django_q_django/urls.py

from django.contrib import admin
from django.urls import path, include
from .settings.base import ADMIN_URL

urlpatterns = [
    path(f'{ADMIN_URL}/', admin.site.urls),
    # the new url
    path("", include("demo_app.urls"))
]

And finally activate the app:

# django_q_django/settings/base.py

INSTALLED_APPS = [
    # omitted for brevity
    'demo_app.apps.DemoAppConfig'
]

Now to simulate a blocking event in the view we're going to use sleep from the time module, part of the Python standard library:

from django.http import JsonResponse
from time import sleep

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    sleep(10)
    return JsonResponse(json_payload)

Run the development server, head over http://127.0.0.1:8000/demo-app/ and you can see the view hanging for 10 seconds before returning to the user.

Now, this is a delay created on purpose, but in a real application the block could happen for a number of reasons:

I/O operations taking too long
network delay
interactions with the file system

Even if it's a contrived example you can see why it's crucial to offload long running tasks in a web application.

Django Q was born with this goal in mind. In the next sections we'll finally put our hands on it.

Wait, how about asynchronous Django?

We're left with a Django project, a Django application, and a view that remains stuck for 10 seconds!

It's not that Django and Python don't scale. There are a number of ways to get around single threading.

For Python there is asyncio. Django instead moved to async only recently, the implementation is in its infancy and still there's no support for async views.

Things are going to change in the future, I suggest staying tuned on Andrew Godwin because it's the lead person for the async story in Django.

But the need for third party queues won't go away anytime soon even when Django will go 100% async. Still a nice skill to have.

Preparing the Heroku app and the Redis instance

In this section we'll prepare the Heroku project. I'm using Heroku here because you may want to deploy to production later, and also because they offer the Redis add-on for free.

If you're new to Redis, it's an in-memory database, can be used as a cache and as a message broker.

A message broker is more or less like a post office box: it takes messages, holds them in a queue, and folks from around the city can retrieve these messages later.

If you're interested in how Django Q uses brokers check out this page.

Still in the project folder initialize a Git repo:

git init

Then create a new Heroku app. I'm going to add two add-ons:

heroku-postgresql which is more robust than the default sqlite for production
heroku-redis which will give us the Redis instance

If you haven't got the Heroku CLI and an Heroku account go create one, install the CLI and come back later.

Otherwise follow along with me and create the app:

heroku create --addons=heroku-postgresql,heroku-redis

Once done give Heroku a couple of minutes and then run:

heroku config:get REDIS_URL

This command will reveal REDIS_URL, an environment variable with the credentials for the Redis instance.

Take note of it and head over the next section!

Asynchronous tasks in Django with Django Q: installing and running Django Q

Let's install Django Q and the Redis client library (the client is needed by the Redis broker for Django Q):

pip install django-q redis

Once done activate Django Q in the list of installed apps:

INSTALLED_APPS = [
    # omit
    # add Django Q
    'django_q'
]

Now reveal the Redis Heroku credentials:

heroku config:get REDIS_URL

You should see a string like this:

redis://h:p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf@ec2-52-18-11-1.eu-west-1.compute.amazonaws.com:9059

Before the @ you'll find the password:

p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf

After the @ there's the host:

ec2-52-18-11-1.eu-west-1.compute.amazonaws.com

And 9059 is the port. Note that the credentials will be different for you, don't use mine!

(Needless to say, by the time you read this article these credentials will be gone.)

Now configure Django Q in django_q_django/settings/base.py. Fill host, port, and password with your credentials:

Q_CLUSTER = {
    'name': 'django_q_django',
    'workers': 8,
    'recycle': 500,
    'timeout': 60,
    'compress': True,
    'save_limit': 250,
    'queue_limit': 500,
    'cpu_affinity': 1,
    'label': 'Django Q',
    'redis': {
        'host': 'ec2-52-18-11-1.eu-west-1.compute.amazonaws.com',
        'port': 9059,
        'password': 'p948710311f252a334c3b21cabe0bd63f943f68f0824cd41932781e7793c785bf',
        'db': 0, }
}

You might wonder why I'm not using REDIS_URL as it is. The reason is that Django Q wants credentials in a dictionary.

~~I didn't have time to check if is the Python Redis client imposing this limitation, maybe I'll write a patch for both in the future.~~ It was a limitation of Django Q, ~~hope I'll have time to open a PR~~ I opened a pull request which got merged, and now you can use a Redis url:

Q_CLUSTER = {
    'name': 'django_q_django',
    # omitted for brevity  
    'label': 'Django Q',
    'redis': 'redis://h:asdfqwer1234asdf@ec2-111-1-1-1.compute-1.amazonaws.com:111'
}

(When running the project in production you may want to switch to using environment variables. See the base configuration for learning how to use env).

Once you're done run the migrations (Django Q needs to create its tables in the database):

python manage.py migrate

At this point you're ready to run the Django Q cluster with:

python manage.py qcluster

If everything goes well you should see this:

Well done! In the next section we'll create our first asynchronous task.

What is the Django Q cluster? Check this out.

Asynchronous tasks in Django with Django Q: async_task

Worth doing a quick recap of what we covered so far:

we created a Django project
we created a Django application
we installed Django Q and the Redis client
we created an Heroku project and a Redis instance
finally we configured Django Q

To test that Django Q could connect to Redis I launched:

python manage.py qcluster

With the project in place let's finally see an example of Django Q in action. Remember your view?

# demo_app/views.py

from django.http import JsonResponse
from time import sleep

def index(request):
    json_payload = {
        "message": "Hello world!"
    }
    sleep(10)
    return JsonResponse(json_payload)

Remove the time import and create a new file in demo_app/services.py (the name of this file is totally up to you).

In this new module we're going to define a function, sleep_and_print:

# demo_app/services.py

from time import sleep

def sleep_and_print(secs):
    sleep(secs)
    print("Task ran!")

In the view instead we'll borrow async_task from Django Q:

from django.http import JsonResponse
from django_q.tasks import async_task


def index(request):
    json_payload = {
        "message": "hello world!"
    }
    """
    TODO
    """
    return JsonResponse(json_payload)

async_task is the principal function you'll use with Django Q. It takes at least one argument, the function's module that you want to enqueue:

# example

async_task("demo_app.services.sleep_and_print")

The second group of arguments instead is any argument that the function is supposed to take. sleep_and_print in our example takes one argument, the seconds to wait before printing. That means for async_task:

# example

async_task("demo_app.services.sleep_and_print", 10)

That's enough to enqueue a task. Let's now mix our view with async_task.

Asynchronous tasks in Django with Django Q: enqueue your first task

Back to our view, with async_task imported, call it right after the return statement:

from django.http import JsonResponse
from django_q.tasks import async_task


def index(request):
    json_payload = {"message": "hello world!"}
    # enqueue the task
    async_task("demo_app.services.sleep_and_print", 10)
    #
    return JsonResponse(json_payload)

Now run the cluster:

python manage.py qcluster

Run the Django server:

python manage.py runserver

And finally make a call to your view, either from http://127.0.0.1:8000/demo-app/ or from the terminal:

curl http://127.0.0.1:8000/demo-app/

Now you should notice a couple of things. The Django dev server should log:

13:55:42 [Q] INFO Enqueued 1

The Django Q cluster should log something along these lines:

13:55:42 [Q] INFO Process-1:1 processing [juliet-mountain-august-alaska]

And after that you should see:

Task ran!

Here's my terminal:

What happened here is that:

the Django view responded immediately to the request
Django Q saved the task (just a reference) in Redis
Django Q ran the task

With this "architecture" the view does not remain stuck anymore. Brilliant.

Think about the use cases for this pattern. You can:

safely interact with the I/O
crunch data in the background
safely move out API calls from your views

and much more.

Asynchronous tasks in Django with Django Q: what's next?

In addition to async_task Django Q has the ability to schedule a task. A practical use case is do X every X days, much like a cron job. Check the documentation to learn more.

Django Q supports other brokers in addition to Redis. Again, the docs are your friend.

If you don't need other brokers than Redis, django-rq might be a lightweight alternative to Django Q.

Asynchronous tasks in Django with Django Q: why not Celery?

Fun fact: Celery was created by a friend of mine. We were in high school together. Despite that I don't have much experience with Celery itself, but I always heard a lot of people complaining about it.

Check this out for a better perspective.

Thanks for reading and stay tuned!

Resources

Originally published on my blog

JavaScript private class fields and the TypeScript private modifier

Valentino Gagliardi — Tue, 14 Jan 2020 09:26:15 +0000

In this post we'll shed some light on JavaScript private class fields and see how they compare to the TypeScript private modifier.

JavaScript private class fields and the need for privacy

Historically JavaScript had no native mechanism for protecting variables from access, unless of course, the quintessential closure.

Closures are the foundation for a lot of private-like patterns in JavaScript like the popular module pattern. But after ECMAScript 2015 classes took over in recent years, developers felt the need for more control over classes member privacy.

The class field proposal (at the time of writing in stage 3) tries to solve the problem with the introduction of private class fields.

Let's see how they look.

JavaScript private class fields, an example

Here's a JavaScript class with private fields, note that unlike "public" members every private field must be declared before access:

class Person {
  #age;
  #name;
  #surname;

  constructor(name, surname, age) {
    this.#name = name;
    this.#surname = surname;
    this.#age = age;
  }

  getFullName() {
    return `${this.#name} + ${this.#surname}`;
  }
}

Private class fields are not accessible from outside the class:

class Person {
  #age;
  #name;
  #surname;

  constructor(name, surname, age) {
    this.#name = name;
    this.#surname = surname;
    this.#age = age;
  }

  getFullName() {
    return `${this.#name} + ${this.#surname}`;
  }
}

const marta = new Person("Marta", "Cantrell", 33);
console.log(marta.#age); // SyntaxError

This is true "privacy". At this point if you now a bit of TypeScript you might ask what "native" private fields have in common with the private modifier in TypeScript.

Well, the answer is: nothing. But why?

The private modifier in TypeScript

The private modifier in TypeScript should be familiar to developers coming from more traditional backgrounds. In brief, the keyword is meant to deny class members access from outside the class.

But let's not forget, TypeScript is a layer on top of JavaScript and the TypeScript compiler is supposed to strip away any fancy TypeScript annotation, including private.

That means the following class doesn't do what you think it does:

class Person {
  private age: number;
  private name: string;
  private surname: string;

  constructor(name: string, surname: string, age: number) {
    this.name = name;
    this.surname = surname;
    this.age = age;
  }

  getFullName() {
    return `${this.name} + ${this.surname}`;
  }
}

const liz = new Person("Liz", "Cantrill", 31);
// @ts-ignore
console.log(liz.age);

Without // @ts-ignore, accessing liz.age throws an error only in TypeScript, yet after the compilation you end up with the following JavaScript code:

"use strict";
var Person = /** @class */ (function () {
    function Person(name, surname, age) {
        this.name = name;
        this.surname = surname;
        this.age = age;
    }
    Person.prototype.getFullName = function () {
        return this.name + " + " + this.surname;
    };
    return Person;
}());

var liz = new Person("Liz", "Cantrill", 31);
console.log(liz.age); // 31

As expected we're free to print liz.age. The main take here is that private in TypeScript is not so private, and it feels convenient only at the TypeScript level, not for "real privacy".

And now let's get to the point: "native" private class fields in TypeScript.

Private class fields in TypeScript

TypeScript 3.8 will support ECMAScript private fields, not to be confused with the TypeScript private modifier.

Here's a class with private class fields in TypeScript:

class Person {
    #age: number;
    #name: string;
    #surname: string;

    constructor(name:string, surname:string, age:number) {
        this.#name = name;
        this.#surname = surname;
        this.#age = age;
    }

    getFullName() {
        return `${this.#name} + ${this.#surname}`;
    }
}

Not so different from vanilla JavaScript, besides type annotations. Members cannot be accessed from the outside. But the real problem with private fields in TypeScript is that they use WeakMap under the hood.

To compile this code we need to adjust the target compilation version in tsconfig.json, which must be at least ECMAScript 2015:

{
  "compilerOptions": {
    "target": "es2015",
    "strict": true,
    "lib": ["dom","es2015"]
  }
}

This could be a problem depending on the target browser, unless you intend to ship a polyfill for WeakMap, which at that point becomes too much work if it's just for the sake of writing fancy new syntax.

There is always this tension in JavaScript, where you really want to use the new syntax, but on the other hand you don't want to let the UX down with a gazillion polyfills.

On the flip side I don't think you should worry too much about private class fields, even if you want to ship to newer browsers. At least for now. The support for private fields is almost non-existent. Not even Firefox has implemented the proposal.

Let's see what the future holds.

Conclusions

Still a proposal at the time of writing, JavaScript class fields are interesting, but the support among browser vendors is poor. What's your take on this feature?

Here's mine:

I like ES private class fields (though I dislike the #)
I never relied to much on private in TypeScript, useful only at the TS level
I'll wait until private class fields land in browsers
I wouldn't use private class fields in TS today

To learn more about TypeScript classes check this out.

The official announcement for TypeScript 3.8 private fields.

[Video] Asynchronous tasks in Django with Django Q (Part 3)

Valentino Gagliardi — Tue, 14 Jan 2020 08:29:14 +0000

Django Q is a scheduler and a task queue for Django. In this series you'll learn how to use Django Q in your Django applications for queuing long-running tasks.

In this episode:

hands on Django Q
using async_task

Enjoy!

FormData, the new formdata event, and HTML forms

Valentino Gagliardi — Fri, 10 Jan 2020 09:31:38 +0000

Have you heard about the new formdata event? It's just a new DOM event, but every new addition to the web platform makes me always excited.

What is FormData?

Let's clarify what is FormData before getting into more detail. For those new to web development, HTML forms are able to emit events. This is a feature of almost any HTML element.

Consider the following form:



<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>HTML forms and JavaScript</title>
</head>
<body>
<form>
    <label for="name">Name</label>
    <input type="text" id="name" name="name" required>

    <label for="description">Short description</label>
    <input type="text" id="description" name="description" required>

    <label for="task">Task</label>
    <textarea id="task" name="task" required></textarea>

    <button type="submit">Submit</button>
</form>
</body>
<script src="form.js"></script>
</html>

To follow along you can create an HTML file, copy over my example. When the form is submitted, that is, when the user fills the fields and clicks the "Submit" button, an event named submit is dispatched.

That means we can listen to the submit event with an event listener:



// form.js
const form = document.forms[0];

form.addEventListener("submit", function(event) {
  event.preventDefault();
});

Calling preventDefault() prevents a page refresh, convenient when you don't want to send the form fields on the backend.

Now, there are a couple ways to get the actual data from the form. You could inspect event.target.elements which in this case yields all the form elements.

Or even better you can use FormData, a DOM interface, as a constructor. It needs the form as an argument:



const form = document.forms[0];

form.addEventListener("submit", function(event) {
  event.preventDefault();
  const data = new FormData(form);
});

From now on you can do all sort of things on the FormData object. More on this later. Now let's explore the formdata event.

Getting to know the formdata event

The formdata event is a newer, nice addition to the web platform. As a boost to FormData the event is fired any time you call new FormData(). Now consider the following example:



const form = document.forms[0];

form.addEventListener("submit", function(event) {
  event.preventDefault();
  new FormData(form);
});

form.addEventListener("formdata", event => {
  // event.formData grabs the object
  console.log(event.formData);
});

In the first event listener we build a new FormData from the form. This time there's no need to save the result in a variable.

In response to this call the new object fires up the formdata event, on which we register another listener.

In this second listener we can grab the actual data from event.formData.

This pattern helps decoupling the first event listeners from any other callback that was supposed to handle the actual form data (making API calls and stuff).

In the next section we'll see how to get data from a FormData object.

Grabbing data from a FormData object

If you want to snitch into a FormData object visit the example HTML form in a browser and place a breakpoint on console.log(event.formData).

Fill and submit the form with the browser's console opened and save the object as a global variable:

You should be able to access the object (temp1) in Chrome:



console.log(temp1)
// FormData {append: ƒ, delete: ƒ, get: ƒ, getAll: ƒ, has: ƒ, …}

As you can see it has a bunch of methods. To extract an array of values run:



const values = [...temp1.values()]
// sample output
// ["Liz", "Trip to London", "Trip to London"]

To extract an array of entries run:



const entries = [...temp1.entries()]

In our example we can get all the data in various shapes from the FormData object:



const form = document.forms[0];

form.addEventListener("submit", function(event) {
  event.preventDefault();
  new FormData(form);
});

form.addEventListener("formdata", event => {
  const data = event.formData;

  // get the data
  const entries = [...data.entries()];
  console.log(entries);

  const values = [...data.values()];
  console.log(values);
});

A word of warning: FormData relies on form fields name attributes to build the mapping between fields and values. That means the following elements won't yield anything:



<!-- bad -->
<input type="text" id="name" required>
<input type="text" id="description" required>

Always provide a name for your fields:



<!-- good -->
<input type="text" id="name" name="name" required>
<input type="text" id="description" name="description" required>

Adding data to a FormData object

It's common practice to have hidden inputs in HTML forms for saving additional data in the submission phase:



<!-- no more hidden fields -->
<input type="hidden" id="someInfo" name="someSecretInfo" value="someSecreteInfo">

Hidden inputs don't show up in the rendered HTML (but are still accessible from the developer console).

In addition to reading form fields from a FormData object, it's also possible to add new key/value pair with append:



// omitted for brevity

form.addEventListener("formdata", event => {
  const data = event.formData;

  // append more fields
  data.append("anHiddenKeyA", "anHiddenValueA");
  data.append("anHiddenKeyB", "anHiddenValueB");

  // get the data
  const entries = [...data.entries()];
  console.log(entries);

  const values = [...data.values()];
  console.log(values);
});

This is convenient for avoiding hidden input fields in HTML forms.

Being a newer addition to the platform the formdata event is not available in older browsers, use it with caution.

Thanks for reading and stay tuned for more!

Resources

FormData objects on MDN