Full Stack To Do list, a step-by-step tutorial

Introduction

The To Do list has become the "Hello, World!" example to compare and demonstrate new JS frontend frameworks. But this tutorial is different as it does not involve a frontend framework, instead it will use vanilla web technologies for the user interface (UI). This is done so we can focus on the stack as a whole and the interfaces between tiers, which I consider to be the most important element of a stack.

Another reason why the To Do application is a good example, is it includes the four basic data manipulation (or CRUD) operations common to many business systems. The CRUD acronym stands for:

• Create: adding a new item to storage.
• Read: retrieving one or more items from storage.
• Update: changing the data of one or more stored items.
• Delete: removing one or more items from storage.

In our example the Update and Delete operations will only be applied to one ToDo item at a time and the Read operation will be used to retrieve all the ToDo items in storage.

Another common acronym in such systems is REST or RESTful, which describes an approach used to communicate between the web browser and the server (or API). There is more to REST than we need to discuss in this post but it is worth knowing how the CRUD operations align to the REST strategy. REST uses HTTP (Hypertext Transfer Protocol) to send requests to the server and receive responses in the client/browser (typically in that order). There are a variety of request types used to instruct the server of what is required, known as the HTTP Verbs, or Methods.

This is how CRUD operations can be mapped to the HTTP Verbs:

Create -> POST
Read -> GET
Update -> PATCH (PUT can also be used but they mean slightly different things)
Delete -> DELETE, well that is obvious.

Here is a link to an MDN page if you would like to learn more about HTTP Verbs/Methods.

Developing inside-out

By defining and even implementing the interface between two subsystems early, a more firm foundation is established for the construction of both sides either-side of the interface. As will be demonstrated, in a three-tier architecture (frontend (FE), backend (BE) and database (DB)) there are two subsystem boundaries; FE/BE and BE/DB.

Our backend will be little more than a two-way translation layer between the database and the user interface (UI). Later in this post we will identify other responsibilities of a backend but our implementation will be kept simple to demonstrate the fundamental machinery and concepts. It is worth noting the backend comes in two parts, web server and application server. Both json-server and Express are able to facilitate these roles from the same URL. This is very useful for our tutorial because we do not have to configure the server to manage Cross-Origin Resource Sharing (CORS). It is quite typical for production systems to separate these server roles for all sorts of good reasons but for now it would just create an additional complication.

Source code (caveat emptor)

I have published the source code for this article in a GitHub repo, but I have to issue a word of caution. THIS IS NOT PRODUCTION CODE. For reasons discussed towards the end of this post, the code is for educational purposes only and should not be used in production.

A step-by-step tutorial

Our example application will be based on the ME*N stack where M is a MongoDB (but it could be MySQL, or any other) database. E is for Express, which is a backend framework that sits on top of Node.JS. There are others but Express is extremely common and will meet our needs well. N is for the JavaScript runtime Node.js but Deno and Bun are possible alternatives. Finally the * is for the frontend framework such as Angular (MEAN), React (MERN) or Vue (MEVN), to name a few. However, in our example we will not be using a framework (to keep it agnostic and besides it is not that complicated), instead we will be using the native web technologies (HTML, CSS and JS).

There are six steps to the development in this tutorial, each building on the previous:

1. Simple array-based interface module - to establish the requirement but with volatile (lost on page refresh) storage.
2. Revised web-storage based interface - to provide local data persistence (during the browser session but could be longer).
3. User Interface (UI) with an API module revised to interact with a backend based on json-server - to establish the final version of the FE/BE interface, but with a mock backend and storage.
4. UI connecting to a Node-Express backend with the same FE/BE interface module as previously and reusing the first interface module (array-based) as temporary backend storage. There is a slight alteration to make it work BE but it is minor.
5. Evolution of the previous step to again employ json-server but for BE data storage only. This will reuse the FE/BE interface module from step three, again with a slight alteration.
6. Finally we swap out json-server for a MongoDB database, provided by MongoDB Atlas, which will require a new interface module based on the json-server version.

In the first two steps we will provide a unit test using Jest and a simple integration test (harness) using an HTML file. You will observe the UI is primitive and uses the development tools (browser console) to present the test results. The UI developed for steps three onwards is a little better and enables user interaction to exercise the CRUD operations in a more realistic manner.

Incremental development

This tutorial revolves around the development and evolution of the interfaces between FE/BE and BE/DB (crud-interface.js). The interface employs the revealing module pattern (RMP) to expose the functions:

• createToDo
• readToDoList
• updateToDo
• deleteToDo

The RMP also enables provision of dependencies such as data sets, functions and other modules in a form of dependency injections; a mechanism we will use to mock external facilities.

Step One: The first interface module - keeping it simple

To commence the development, we will adopt a Test-Driven Development (TDD) approach by creating the unit test first using the Jest test framework. However, our FE/BE interface will use the ES6 module (ESM) syntax instead of the Common JS (CJS) syntax default to Node.JS. Therefore, we need (as of Node 19.7.0) we need to use a command-line switch to enable ESM as follows.

node --experimental-vm-modules node_modules/jest/bin/jest.js

The crud-interface unit test will first import the interface module and a dataset in the form of the following JSON document.

{
  "toDos": [
    {
      "text": "Task One",
      "done": true,
      "id": "1"
    },
    {
      "text": "Task Two",
      "done": false,
      "id": "2"
    }
  ]
}

The unit test will perform four positive test cases, one per exposed method, plus three negative test cases for when the list is empty and then no toDo item is found when updating or deleting. The initial implementation of the crud-interface module is based around a simple array, which can be supplied when the module is instantiated. This means the unit test can pass in the toDos property of the above test data when we execute the default method of the module. The array CRUD operations are simple but we cannot replace the array itself otherwise the unit test will lose reference, instead we have to replace its content.

In addition to the unit test we also have an integration/functional test capability in the form of an index.html test harness. Instead of using the Jest framework, this test is exercised in the web browser in much the same way the module would be expected to operate. Therefore, we will be using the json-server package as a proxy web server to make all the resources available to the browser “online”; we will be using a similar strategy for the first three steps.

Step Two: Minor enhancement to improve data persistence

In step two we revise the internal workings of the interface module to swap out the array with the windows session web storage. The external/public interface will remain the same so the unit test will largely remain the same. The two key differences are:

• The interface module dependency will be the window.sessionStorage namespace.
• The unit test will mock the dependency, which enables the test cases to confirm actions from both sides of the interface.

As before, the implementation of the module will use the ESM syntax and there will be an HTML-based test harness to provide greater context and exclude the jest framework from the test environment, but will again use json-server to deliver content to the browser.

Step Three: Adding a User Interface with a proxy backend

We continue with the two-tier test approach in step three, but this time we will use json-server more extensively. Json-server will provide both a web server to deliver the page and an application server for access to data storage (in the form of a json file). In order to execute both unit and integration tests we need to run the server using the ‘start’ script. Once the server has completed loading, the ‘test’ script will perform the unit test of the revised interface module. Once complete, the integration test can be performed by accessing the test harness from the browser by loading ‘http://localhost:3000’ as before.

The crucial difference with the step-three version of the interface module, is that it uses the fetch API to interact with the server. The module’s only dependency is the URL of the API, but it could be extended to include the fetch API. This would make it possible to swap out the actual fetch mechanism for a mock equivalent in the unit test and avoid the need for the actual server to be running. Instead, the start script will reset the source data and start the server. The integration test is also extended to provide a more interactive (and realistic) user interface.

The UI presents a list of stored tasks and an input field to type the name of a new task. As might be expected, the Add button creates a new task in storage, Done updates the associated task to mark it ready for deletion, and the Delete button removes the associated task from storage. When the input field is empty the Add button will be disabled.

This exact version of the module will be used for the FE/BE interface of the next three steps.

Step Four: Connecting the UI to a production-grade backend with simple storage

In this and the following steps we will be using an interface module in the backend as well as the frontend. Where possible the BE/DB interface will be heavily based on a previously developed FE/BE implementation with one minor difference. The FE/BE modules use the standardised ECMAScript Module (ESM) syntax, which requires some configuration of NodeJS to facilitate testing. The BE/DB modules will employ the Common JS (CSJ) module system that is native to the NodeJS environment.

In this step, and from here on, the middle tier (backend) will be implemented using the ExpressJS framework on top of NodeJS. The Express component exposes the RESTful that maps requests from the frontend to the BE/DB interface. Step four uses a BE/DB interface module based on the very first implementation (step one). The only difference between the two implementations is the change from the ESM syntax,

export default function (toDoList = []) {

and the CJS module syntax

module.exports = function (toDoList = []) {

both expect the toDoList array as a dependency.

Step Five: Adapting the json-server FE/BE interface module for BE/DB use

Similar to the previous step, the next step repurposes an earlier implementation of the FE/BD interface module (from step 3) for use as a BE/DB interface using the json-server as a proxy database.

Again the only real difference is the change in module syntax.

From export default function (endpointUrl = 'http://localhost:3000/toDos'),
to module.exports = function (endpointUrl = 'http://localhost:3300/toDos').

You may also notice the only other difference is the change in port number from 3000 to 3300. This reflects the fact we are communicating directly with the database (proxy) instead of the backend server.

Step Six: Replacing the proxy storage with a production-grade cloud server

Finally, we replace the BE/DB interface and proxy database with the actual things.

In our example we establish a connection with a MongoDB database hosted by their Atlas service. The connection needs to be secured so parameters have been stored in a .env file of the following structure.

USER_ID=userId
USER_PWD=userPassword
DB_NAME=databaseName
DB_COLLECTION=databaseCollection

Making a production-grade version of the example stack

As is this implementation of the ME*N stack is exceptionally vulnerable to attack. There is no protection from error or misuse through the FE, which would be greatly improved by using a FE framework such as React, Angular, Vue etc. The FE/BE interface is also wide open to malicious actors. This can be improved by employing HTTPS to encrypt the communication path and implead “man-in-the-middle” attack. JWT can also be used to establish user authentication. The stack can also be made more robust and maintainable through the use of two Express middleware packages (Helmet and Mongoose).

Helmet helps “sanitise” the input, which might not have come from the UI directly. Mongoose is what is known as an Object Document Modelling (ODM), which defines a structure (schema) for the stored data, making it easier to manage in Express. These additions have been omitted from our example stack purely to simplify the tutorial and focus on the fundamental tiers and interfaces.

Comments

[Tracy Gilmore]

In the example source code I have use the Fetch API, granted in a rather naïve manner. An alternative option would be to use the AXIOS library.
Another point about the source code, the crud-interface implementations could have been more agnostic to the data/endpoint they were supporting. Given they CRUD operations are generated in the context of a given endpoint URL, when instantiating the interfaces like,

const crudInterface = CrudInterface(ENDPOINT_URL);

the name of the interface instance could better reflect the purpose of the interface/ the resource the interface is manipulating. E.g.

const toDoInterface = CrudInterface(TODO_ENDPOINT_URL);

Finally, on the topic of PUT vs PATCH: Patch is typically used to perform an partial modification of an existing record whereas Put is used to completely replace an existing record with a new one. In most use cases Patch is preferable because only the affected properties/fields have to be send to the server. There are use cases when a complete replacement is superior such as when multiple versions of the record are being retained.

[Andy Eder]

Great article! I feel this structured approach may prove useful for the re-working (rewrite) of an existing project...