DEV Community: e4c5Nf3d6

Handing Your Database Over to Users

e4c5Nf3d6 — Tue, 07 May 2024 18:46:45 +0000

When designing an application, it is crucial to keep its intended users in mind. An average user probably will not understand the complications that can arise from invalid or incorrect data.

There are several ways to help maintain your data's integrity while making things as simple as possible for the application's users. I will be addressing four of these methods.

Server validation
Front-end validation
Tooltips
Useful error messages

I will be using an example application with a backend written in Python and utilizing Flask and a front-end built with React. In this example, a user can submit up to two product usage reports from different tracking systems and have a product order created for them. The two tracking systems have disparate ways of referencing the same products: different names for the products and different categories. The products currently in the database are all matched correctly, but new products will inevitably be added. When that happens, a user must be able to add that product to the database correctly. It is our responsibility as developers to make that as easy as possible for the user.

1. Server Validations

Database Constraints and Validations

We are able to validate data within our model classes using validations and constraints. Validations will not allow instantiation if the data does not meet certain requirements. Constraints will not allow the data to be added to the database if it is invalid.

An example of a constraint is unique=True, which will not allow the addition of a row to a table if the same value already exists in the column. In this example, we need the names of each product to be different in order to avoid confusion when a product order is actually generated. The unique constraint is perfect for this situation.

# server/models.py

class Product(db.Model, SerializerMixin):
    __tablename__ = 'products'

    ...

    id = db.Column(db.Integer, primary_key=True)
    # Add the unique constraint.
    name = db.Column(db.String, unique=True)
    ...

When a POST request is sent to the server, an instance of the Product class is created. If the name is not unique, the instance will not be added to the database and an error will be returned to the client.

# server/app.py

class Products(Resource):

    def get(self):
        ...

    def post(self):
        ...

        # Create a Product instance.
        product = Product(
            name = name,
            category_id = category_id,
            phorest_name = phorest_name,
            vish_name = vish_name
        )

        # Try to add the instance to products table.
        try:

            db.session.add(product)
            db.session.commit()

            return make_response(product.to_dict(), 201)

        # Return an error if the name is not unique.
        except IntegrityError:

            return make_response({'error': '422 Unprocessable Entity'}, 422)

Response Error Messages

In some cases, errors need to be caught even before an instance is created.

In this example, we need to catch if a user submits the wrong report while creating an order. A user is only able to upload .xls and .xlsx files, but they could accidentaly upload files that have nothing to do with the products they are trying to order. We need to make sure that products are actually detected in the files before instantiating the order and return an error if they are not.

# server/app.py

class Orders(Resource):

    def get(self):
        ...

    def post(self):
        ...

        phorest_products = []

        try:
            # See if a file was submitted
            phorest_file = request.files['phorest_file']

            try:
                # Read the file.
                phorest_data = pd.read_excel(phorest_file, engine='xlrd') 

                ...

                # Identify which rows have data that should be added.
                for i in range(phorest_data.shape[0]):
                    if name == 'Wella': 
                        if phorest_data.iloc[i, 0] == 'PROFESSIONAL':
                            name = ''
                            pass

                        if phorest_data.iloc[i, 0] == 'RETAIL':
                            break

                        # Add the product data to phorest_products.
                        try:
                            phorest_products.append((phorest_data.iloc[i, 0], int(phorest_data.iloc[i, 1])))

                        except:
                            pass

                    else:
                        name = phorest_data.iloc[i, 0]

            # Return an error if a file was submitted but cannot be read.
            except:
                return make_response({'error': '422 Unprocessable Entity', 'file': 'phorest_file'}, 422)

            # Return an error if no products were found in the file.
            if phorest_file and not phorest_products:
                return make_response({'error': '422 Unprocessable Entity', 'file': 'phorest_file'}, 422)

        # Move on if no file was submitted           
        except:
            pass

        ...

This process can be repeated for the other file. An error specifying which of the two files is invalid will be returned to the client.

2. Client Side Validations

When possible, invalid data should be detected before a request is even made. Tools like Yup and Formik allow us to define a form schema that specifies what type of data is allowed and whether a field is required, among other things.

// client/src/components/CreateOrder.js

import React from "react";
import * as yup from "yup";

function CreateOrder() {
    ...

    const formSchema = yup.object().shape({
        // Place restrictions on data type and require certain data.
        location_id: yup.number()
            .required("Please choose a location"),
    });

    ...
}

export default CreateOrder;

3. Tooltips

So, we have done our best not to allow invalid data into our database. However, there is still the issue of data that is valid but incorrect.

In this example, a user could easily choose the wrong category for a product. This would pose a problem when an order is being created and the server compares the products in the database to the data in the report files. The user needs to be informed of how to know which product category they should choose.

Tooltips are an ideal way to provide instruction to your users without cluttering up the user interface.

// client/src/components/Add Product.js

import React from "react";

function AddProduct() {
    ...

    return (
        ...
            <label htmlFor="category_id">Category</label>
            <abbr>
                ?
                <span>
                    This can be found in Vish.
                    <br />
                    <strong>Products → Wella</strong>
                    <img src="category_screenshot.png" className="tooltip-img" alt="Category Screenshot" />
                </span>
            </abbr>
        ...
    );
}

export default AddProduct;

Of course, some users might ignore the help provided. It is always useful to give users a way to fix any mistakes, such as an option to edit the product category.

4. Useful Errors

Finally, error messages displayed to the client should be useful. They do not need to see that the error sent back from the server was 422 Unprocessable Entity. They need to see that Quantity must be greater than 0.

Conclusion

The thought of handing your database over to users can be an intimidating one. However, if you have implemented methods to disallow invalid data and given users to tools to submit the correct data, you can rest easy knowing that you have done your job.

Using React Select with Formik

e4c5Nf3d6 — Sun, 17 Mar 2024 23:30:17 +0000

I was recently building an application that, among other features, allows a user to submit chess players and chess games to a database. I was utilizing Yup for form schema and Formik for error handling, validation, and form submission.

In order to submit a chess game, the user must provide the game's PGN and choose the game's players from players that exist in the database.

I decided that I wanted to use React Select for the player input, specifically React Select Creatable, which allows users to create a new option if the one that they are looking for does not exist. With this approach, if the player that the user is trying to add as one of the game's players is not already in the database, they can add the player directly from the React Select component instead of having to do this though the seperate form for adding a new player.

The Problem

The React Select options used to display the choices to the user in the component need to be in a specific format. For instance, the option for the player Mikhail Tal would need to be in the following format:
{ value: 'Mikhail Tal', label: 'Mikhail Tal' }

However, for validation purposes, the white-player and black_player values used with Yup and Formik and ultimately submitted using Formik would need to be in string format, such as 'Mikhail Tal'. I considered trying to store these values in Formik in the format requred for React Select and simply extracting the necessary value when the form is submitted. However, doing so would cause Formik to throw errors since the Formik field values would not be in the expected format when the form was submitted.

This meant a couple of things for my code:

I would not be able to use Formik to keep track of the values of the React Select fields. They would need to be tracked another way.
I would need a way of setting the Formik values when the values of the React Select fields changed as well as when a new option was created.

The Solution

First, I set up the state that would be needed to control my React Select input values.

// imports

function AddGame(props) {
    const [white, setWhite] = useState(null);
    const [black, setBlack] = useState(null);

    // the rest of the code
}

// exports

Here is the general setup of the CreatableSelect components rendered in the AddGame component:

<CreatableSelect
    isClearable
    onChange={(player) => setWhite(player)}
    options={options}
    placeholder="Choose the player with the white pieces"
    value={white}
/>
<CreatableSelect
    isClearable
    onChange={(player) => setBlack(player)}
    options={options}
    placeholder="Choose the player with the black pieces"
    value={black}
/>

The values are determined by the white or black states and the onChange props use setWhite or setBlack to set said states.

In order for the Formik values to reflect the change in state, I needed to call a more robust function in the onChange prop. I needed to be able to manually set the Formik values, something that is usually handled by Formik but can also be accomplished with formik.setFieldValue(). For example, if I wanted to set the white_player value to 'Mikhail Tal', I could use the following statement:
formik.setFieldValue(white_player, 'Mikhail Tal')

(It was at this point in the process that I set up a custom hook to handle the logic, since I would need this functionality in a few different places in my code. However, it can definitely be accomplished in one file as well.)

I created a handleSelect function that would update both the state that controlled the React Select fields and the corresponding Formik values.

function handleSelect(color, player) {
    if (player === null) {
        formik.setFieldValue(color, '');   
    } else {
        formik.setFieldValue(color, player["value"]);
    }

    if (color === "white_player") {
        setWhite(player);
    } else if (color === "black_player") {
        setBlack(player);
    }
}

The CreatableSelect components' onChange functions need to be updated accordingly. For example:
onChange={(player) => setBlack(player)}
becomes
onChange={(player) => handleSelect('white_player', player)}

Finally, I needed to implement the creatable aspect of React Select. The CreatableSelect components have the onCreateOption prop to help with this.

I made a handleCreate function to handle the creation of a new player. It makes a POST request to the "/players" endpoint of my API and then, assuming it recieves a 201 (created) status code, sets the corresponding Formik field value and player state using the response data.

function handleCreate(color, newPlayer) {
    fetch("/players", {
        method: "POST",
        headers: {
            "Content-Type": "application/json"
        },
        body: JSON.stringify({name: newPlayer}, null, 2)
    }).then((r) => {
        if (r.status === 201) {
            r.json()
            .then((player) => {
                onSetPlayers([...players, player]);
                formik.setFieldValue(color, player.name);

                if (color === "white_player") {
                    setWhite({ value: player.name, label: player.name });
                } else if (color === "black_player") {
                    setBlack({ value: player.name, label: player.name });
                }
            });
        }
    });
}

The CreatableSelect components end up looking like this:

<CreatableSelect
    isClearable
    onChange={(player) => handleSelect('black_player', player)}
    options={options}
    placeholder="Choose the player with the black pieces"
    value={black}
    onCreateOption={(newPlayer) => handleCreate('black_player', newPlayer)}
/>

Conclusion

Yup and Formik are extremely useful tools that help handle many of the more frustrating aspects of setting up forms in React, such as data validation and error handling. React Select has a number of useful features for creating select inputs in React, including built-in styling as well as Creatable components that allow for the creation of new options. Utilizing these tools together can greatly improve the functionality of your form. It is also relatively simple to use them in concert with each other - once you know how.

Full Solution Code

import React, { useState } from "react";
import { useFormik } from "formik";
import * as yup from "yup";
import CreatableSelect from 'react-select/creatable';

import useSelectData from "../hooks/useSelectData";

function AddGame({ games, onSetGames, players, onSetPlayers }) {
    const [isEditing, setIsEditing] = useState(false);
    const [showError, setShowError] = useState(false);
    const [white, setWhite] = useState(null);
    const [black, setBlack] = useState(null);

    const options = [
        ...players.map((player) => ({ value: player.name, label: player.name }))
    ];

    function handleClose() {
        formik.resetForm();
        setShowError(false);
        setWhite(null);
        setBlack(null);
        setIsEditing(false);
    }

    const formSchema = yup.object().shape({
        pgn: yup.string()
            .required("Please enter the game PGN")
            .matches(/^(\s*(?:\[\s*(\w+)\s*"([^"]*)"\s*\]\s*)*(?:(\d+)(\.|\.{3})\s*((?:[PNBRQK]?[a-h]?[1-8]?x?[a-h][1-8](?:\=[PNBRQK])?|O(-?O){1,2})[\+#]?(\s*[\!\?]+)?)(?:\s*(?:\{([^\}]*?)\}\s*)?((?:[PNBRQK]?[a-h]?[1-8]?x?[a-h][1-8](?:\=[PNBRQK])?|O(-?O){1,2})[\+#]?(\s*[\!\?]+)?))?\s*(?:\(\s*((?:(\d+)(\.|\.{3})\s*((?:[PNBRQK]?[a-h]?[1-8]?x?[a-h][1-8](?:\=[PNBRQK])?|O(-?O){1,2})[\+#]?(\s*[\!\?]+)?)(?:\s*(?:\{([^\}]*?)\}\s*)?((?:[PNBRQK]a-h]?[1-8]?x?[a-h][1-8](?:\=[PNBRQK])?|O(-?O){1,2})[\+#]?(\s*[\!\?]+)?))'?\s*(?:\((.*)\)\s*)?(?:\{([^\}]*?)\}\s*)?)*)\s*\)\s*)*(?:\{([^\}]*?)\}\s*)?)*(1\-?0|0\-?1|1\/2\-?1\/2|\*)?\s*)$/ , 'Invalid PGN format'),
        white_player: yup.string()
            .required("Please choose the player with the white pieces"),
        black_player: yup.string()
            .required("Please choose the player with the black pieces"),
    });

    const formik = useFormik({
        initialValues: {
            pgn: "",
            white_player: "",
            black_player: ""
        },
        validateOnChange: false,
        validationSchema: formSchema,
        onSubmit: (values, { resetForm }) => {
            fetch("/games", {
                method: "POST",
                headers: {
                    "Content-Type": "application/json"
                },
                body: JSON.stringify(values, null, 2)
            })
            .then((r) => {
                if (r.status === 201) {
                    r.json()
                    .then((game) => {
                        onSetGames([...games, game]);
                        handleClose();
                    });
                } else if (r.status === 422) {
                    setShowError(true);
                    resetForm();
                    formik.setFieldValue('white_player', white['value']);
                    formik.setFieldValue('black_player', black['value']);
                }
            });
        }
    });

    const { handleSelect, handleCreate } = useSelectData(formik, setWhite, setBlack, players, onSetPlayers);

    return (
        <div>
            {isEditing ? 
                <div className="add">
                    <h3>Add a Game</h3>
                    {showError ? <p style={{ color: "red" }}>Failed PGN Validation</p> : null}
                    <form onSubmit={formik.handleSubmit}>
                        <textarea 
                            type="text"
                            id="pgn"
                            name="pgn"
                            placeholder="Game PGN"
                            value={formik.values.pgn}
                            onChange={formik.handleChange}
                        />
                        {formik.errors.pgn ? <p style={{ color: "red" }}>{formik.errors.pgn}</p> : null}
                        <div className="select">
                            <CreatableSelect
                                isClearable
                                onChange={(player) => handleSelect('white_player', player)}
                                options={options}
                                placeholder="Choose the player with the white pieces"
                                value={white}
                                onCreateOption={(newPlayer) => handleCreate('white_player', newPlayer)}
                                menuPortalTarget={document.body} 
                                styles={{ menuPortal: base => ({ ...base, zIndex: 9999 }) }}
                            />
                            {formik.errors.white_player ? <p style={{ color: "red" }}>{formik.errors.white_player}</p> : null}
                        </div>
                        <div className="select">
                            <CreatableSelect
                                isClearable
                                onChange={(player) => handleSelect('black_player', player)}
                                options={options}
                                placeholder="Choose the player with the black pieces"
                                value={black}
                                onCreateOption={(newPlayer) => handleCreate('black_player', newPlayer)}
                                menuPortalTarget={document.body} 
                                styles={{ menuPortal: base => ({ ...base, zIndex: 9999 }) }}
                            />
                            {formik.errors.black_player ? <p style={{ color: "red" }}>{formik.errors.black_player}</p> : null}
                        </div>
                        <button className="submit-button" type="submit">Submit</button>
                        <button type="reset" onClick={handleClose}>Close</button>
                    </form>
                </div> 
                :
                <button onClick={() => setIsEditing(!isEditing)}>Add Game</button>
            }
        </div>
    );
}

export default AddGame;

function useSelectData(formik, setWhite, setBlack, players, onSetPlayers) {

    function handleSelect(color, player) {
        if (player === null) {
            formik.setFieldValue(color, '');   
        } else {
            formik.setFieldValue(color, player["value"]);
        }

        if (color === "white_player") {
            setWhite(player);
        } else if (color === "black_player") {
            setBlack(player);
        }
    }

    function handleCreate(color, newPlayer) {
        fetch("/players", {
            method: "POST",
            headers: {
                "Content-Type": "application/json"
            },
            body: JSON.stringify({name: newPlayer}, null, 2)
        }).then((r) => {
            if (r.status === 201) {
                r.json()
                .then((player) => {
                    onSetPlayers([...players, player]);
                    formik.setFieldValue(color, player.name);

                    if (color === "white_player") {
                        setWhite({ value: player.name, label: player.name });
                    } else if (color === "black_player") {
                        setBlack({ value: player.name, label: player.name });
                    }
                });
            }
        });
    }

    return { handleSelect, handleCreate };
}

export default useSelectData;

Dynamic, Nested Menus in a Python CLI

e4c5Nf3d6 — Mon, 29 Jan 2024 03:02:51 +0000

The Problem

Recently, I was working on a Python Command Line Interface that allowed a user to explore different art movements, artists, and paintings stored in a database. Movements to artists was a one-to-many relationship, as was artists to paintings.

I wanted a the main menu of the CLI to have three top level menu items that each led to submenus: movements, artists, and paintings.

Within each of these menus, I wanted an option to explore a particular row of the corresponding table.

Choosing option 3 would list all available options and open a menu based on the user's choice.

I needed a way to make these 'exploration' menus. In addition to accessing these dynamic menus from the top level menus, a user would also be able to access them in the following ways:

Within the 'explore movement' menu, there would be an option to explore an artist in that movement.
Within the 'explore artist' menu, there would be an option to explore a painting by that artist.
Creating a new movement, artist, or painting would automatically enter the exploration menu for the newly created table row.

I started with an outline of how I needed the menu navigation to work.

Main Menu
1. Movements Menu -> Explore Movement -> Explore Artist -> Explore Painting
2. Artists Menu -> Explore Artist -> Explore Painting
3. Paintings Menu -> Explore Painting

I wanted the user to be able to move down these lines and back up to the main menu.

I was unsure where to begin, seeing as the only CLI menu with which I was familiar had the following structure:



def main():
    while True:
        menu()
        choice = input("> ")
        if choice == "0":
            exit_program()
        elif choice == "1":
            function_one()
        elif choice == "2":
            function_two()
        else:
            print("Invalid choice")

with menu() calling a function that prints out options for the user. I knew nothing about nested menus or about making menus dynamic.

The Process

Looking at the basic menu function, I realized that I already had a hint as to how to make these menus nested: the while loop. The while loop in the basic menu serves the purpose of keeping the user in the menu until they choose to exit the program. A choice can be made, the function that corresponds to that choice will be executed, and then the options will be reprinted and the user will again be prompted to make a choice.

So what if the function that corresponded to the user's choice contained its own while loop?

The Navigation

In order to move from one menu to the next, calling the function for the secondary menu would suffice. The while loop inside of the function for the secondary menu would keep the user there until they chose to exit.

I initially thought that the backwards navigation would be more complicated. However, I quickly realized that the only thing needed to return to the previous menu was terminating the current while loop.



def artists():
    m = "artists"
    while m == "artists":

        # print user options
        artists_menu()

        choice = input("Enter your choice: ")
        if choice == "0":
            m = ""

        # other functions

        else:
            print("Invalid choice")

Making It Dynamic

Beyond what I already had, I knew that each of the 'exploration' menus would need to take in an argument in order to dynamically display information based on one of the following: the option that was chosen upon entering the menu or a newly created row of data.

To handle the first situation, I wrote helper functions that would print the available options and return row of the table that corresponded to the user's choice.

To handle the second situation, I simply had any create functions return the newly created object.

I could then save these return values to variables and pass them variable into the explore menu functions.



elif choice == "2":
    artist = create_artist()
    if artist:
        explore_artist(artist)
elif choice == "3":
    artist = choose_artist()
    if artist:
        explore_artist(artist)

I could then use the data passed in to these functions to display specific information and options to the user based on their choice.



def explore_artist(artist):
    m = "explore artist"
    while m == "explore artist":

        # print user options
        artist_options_menu(artist)

        choice = input("Enter your choice: ")

        if choice == "0":
            m = ""
        elif choice == "1":
            list_paintings_by_artist(artist)
        elif choice == "2":
            painting = choose_painting_by_artist(artist)
            if painting:
                explore_painting(painting) 
        elif choice == "3":
            painting = create_painting(artist.id)
            if painting:
                explore_painting(painting)
        elif choice == "4":
            update_artist(artist)
        elif choice == "5":
            result = delete_artist(artist)
            if result == "deleted":
                m = ""
        else:
            print("Invalid choice")

I could now allow some of my functions, like create_painting, take in an optional parameter: in this case, an artist's ID. When creating a new painting from the main paintings menu, nothing would be passed in and the user would be asked to enter a painting name, choose an artist, enter a year, and choose a medium. However, when creating a new painting from the explore artist menu, the artist's ID would already be provided, eliminating the step of choosing a artist.

Conclusion

This solution will probably seem obvious, even simplistic, to some of you. However, as someone just beginning to learn about Python and CLIs, the path to solving this problem was not at all clear at first. I hope that my solution will be able to help someone in their software engineering journey!

Multi-Selects with React Select

e4c5Nf3d6 — Thu, 28 Sep 2023 03:09:14 +0000

React Select is a library that can be very helpful when creating select elements in your React application.

It offers a variety of useful features; the one I will be going over is the multi-select option, in particular its use with controlled components.

A multi-select allows a user to choose more than one of the options available to them. Although it is definitely possible to create a regular <select> element that allows multiple options to be selected, doing so can be complicated and cumbersome. (For instance, adding the attribute multiple={true} to <select> does allow a user to select multiple options, but only if they hold down Ctrl or CMD and click, which is decidedly not user friendly.)

React Select simplifies this process, giving you access to a fully functional, already styled multi-select component that is ready to be used in your application.

Here is an example of what a multi-select looks like without any additional styling:

Installation

To install react-select, run npm i --save react-select or yarn add react-select.

Implementation

import React from 'react'
import Select from 'react-select'

You can use the default export, Select, in any of your components after importing it from react-select. If you are going to need several similar selects in your application, it can be simpler and cleaner to create your own select component and reuse that, as I have done in the example below. (Although the following is a very basic implementation of Select, using Select in your own reusable component like this gives you the flexibilty to modify your component later and have these modifications apply to every Select in your application.)

function SelectOption({ options }) {
    return(
        <Select 
            isMulti
            options={options}
        />
    )
}

export default SelectOption

The options passed into Select as props are the options that a user is able to select. The options must have the following format with both a value and a label specified:

options={[
    { value: 'black', label: 'Black' },
    { value: 'blue', label: 'Blue' },
    { value: 'green', label: 'Green' },
    { value: 'red', label: 'Red' },
    { value: 'white', label: 'White' }
]}

The label is what the user will actually see when choosing an option.

A variety of other props can also be specified in order to further customize Select. One you want to be sure to include if you are creating a multi-select is isMulti. This is a boolean that has a default value of true, but can also be specified as either true or false. (If you do not need a multi-select, you can simply omit use of isMulti.)

Other useful props that can be specified include:

isClearable: allows a user to clear the input field with one click instead of removing each element individually
placeholder: sets text to display before any option is selected
isSearchable: allows a user to search for options
className: sets the outer component's className attribute
classNamePrefix: gives any inner components a prefixed className attribute, which is useful when styling with CSS
required: makes the selection of an option a requirement for form submission

There are many more props that you can explore if you have the chance. Check out the React Select documentation to see a full list of props and proper usage.

Using Multi-Select in a Controlled Component

When using Select, you are able to specify the props value and onChange, which allows you to use Select as part of a controlled component.

First, you will need to import the useState hook and initialize state within your component.

It is important to note that the format of the value of Select when using multi-select will be an array of objects.

You should also note that you do not need to use e.target.value to access the value of Select in your onChange handler. Instead of an event, the new value is passed in to the funtion provided. (React Select also allows you to pass in a second argument that specifies what kind of change event has been fired, allowing for more control over how these events are handled.)

Here is a basic example:

import React, { useState } from "react"
import Select from "react-select"

function Form({ options }) {
    const [selectedOptions, setSelectedOptions] = useState(null)

    return(
        <Select 
            isMulti
            options={options}
            value={selectedOptions}
            onChange={setSelectedOptions}
        />
    )
}

export default Form

If you are using a multi-select in a form, you might need to reformat your data. Often, you will need an array of the values instead of an array of objects that inclue a value and a label. If you only need the values, the map() method is an easy and convenient way to access the data that you need.

selectedOptions.map(color => color.value)

It is important to note that you cannot change the format in the state that actually controls the component, since the value that Select accepts as props must be an array of objects. You will either need to use a separate piece of state for the modified data or modify the data when the form is submitted.

Conclusion

The features of that I have gone over here are merely the tip of the iceberg that is React Select. There is also the Async component that can load options from an outside source, as well as the Creatable component that allows users to create new options. I strongly encourage you to explore all that this library has to offer.

Complications and Simplifications: Creating a Chess Application

e4c5Nf3d6 — Fri, 26 May 2023 00:16:08 +0000

The idea that I had for my Flatiron School Phase 1 Project seemed simple enough: a Single Page Application that would allow a user to search for an online chess player by username and would display their stats and recent games. The user would also be able to view each game through an iframe that would be embedded in the page.

As I took the first steps to create this program, I quickly discovered that it was more complicated than it initially seemed.

I started out by looking into the API for Chess.com, the most widely used online chess platform. However, I almost immediately ran into a big problem: The ID for a game's iframe did not match the game ID. As far as I could tell, the iframe ID was nowhere to be found in any of the API's responses, and without that ID, there would be no way to dynamically imbed a game's iframe.

I moved on to Lichess, the second most popular online chess platform. Luckily, Lichess's iframe ID and game ID matched. Problem solved, right?

Wrong.

As I looked over the Lichess API, I noticed something unfamiliar to me. A fetch request to https://lichess.org/api/games/user/{username} returned content in NDJSON (Newline Delimited JSON) format instead of in JSON format - the only format with which I was really familiar. Requests to other endpoints that I would be using - such as player information and games filtered by the opening played in the game, all returned data in JSON format - but I still needed a way to access and list a player's recent games.

Unsurprisingly, ignoring the offending 'Accept': "application/x-ndjson" and hoping to convert to JavaScript using .then(res => res.json()) did not work.

I then realized I had to figure out what NDJSON was before I could proceed. According to ndjson.org, NDJSON is "a convenient format for storing or streaming structured data that may be processed one record at a time." The main difference between the NDJSON and JSON formats seemed to be that NDJSON data is separated into separate lines using \n, while JSON data is not. Each separate line of NDJSON is valid JSON. In my case, each line of NDJSON would represent a chess game.

Here was my general idea for the function that would get a list of a user's games:

function listUserGames(e) {
    let username = e.target.querySelector('#username').value

    fetch(`https://lichess.org/api/games/user/${username}?max=8`, {
        method: 'GET',
        headers: {
            'Accept': "application/x-ndjson"
        }
    })
    .then(res => 
        // somehow convert res from NDJSON to JavaScript
    )
    .then(games => {
        if (games.length > 0) {
            games.forEach(game => displayGame(game))
        } else {
            // indicate that the user has never played a game
        }
    })
}

Google helped me find a working solution based on this repo. After some trial and error, I was left with the following code:

.then(res => res.text())
.then(body => {
    let str = "[" + body.replace(/\r?\n/g, ",").replace(/,\s*$/, "") + "]"
    return JSON.parse(str)
})

This worked, and I (mostly) understood why. It was converting the response to text, adding brackets to make the text like an array, replacing newline characters with commas to make each line like an array element, and removing whitespace before converting it to a JavaScript object. However, I felt uncomfortable using this. I didn't feel like I completely understood the syntax and I was not quite sure why whitespace was being removed. I couldn't have reproduced it. I moved on, eventually completing the application, but this block of code kept bothering me, even though it worked just fine.

I returned to this code block later and broke down what needed to be done.

Convert the response to text. (Done with .then(res => res.text()))
Make each line into an element in an array.
Convert each line to a JavaScript object

I knew how to split a string into an array, so I started there, using data.split('\n').

Now I had an array of strings. How could I go about converting each one to an object? I knew that each one was valid JSON, as stated in the NDJSON documentation, and I realized that the map() method would be perfect for this situation.

After implementing this method, I had the following code:

.then(res => res.text())
.then(data => {
    return data.split('\n').map(game => JSON.parse(game))
})

I tried it out, and... it didn't work. This error was displayed in my console when I ran my code:

Upon closer inspection, I found the problem. Running console.log(data.split('\n')) displayed an array with nine elements. I only asked for 8 games.

I finally found out why the whitespace needed to be removed.

It was time to try again. I used the filter() method to remove the empty string:

.then(res => res.text())
.then(data => {
    return data.split('\n').filter(item => item.length > 0).map(game => JSON.parse(game))
})

This still felt a bit inelegant, but it worked!

In the process of writing this blog, I realized I could make the code simpler by trimming the whitespace at the end of the string before splitting it into an array.

This left me with the following code:

.then(res => res.text())
.then(data => {
    return data.trim().split('\n').map(game => JSON.parse(game))
})

This is what I wanted: a way to convert NDJSON into JavaScript that I fully understood. This block of code turns the NDJSON to text, trims the whitespace, and makes each line of an element in an array. It then returns an array of JavaScript objects.

In retrospect, this solution seems obvious. However, as someone new to programming, a new format seemed much more intimidating than it actually was. I appreciate the process I went through now, as I have a much better understanding of what happens when my code is executed. A lot of debugging and examination of the network tab in Chrome DevTools allowed me see every change that my data underwent in the reformatting process. After this project, I can confidently say that I understand this process and can reproduce it. My takeaway is simple, and fittingly so: things got complicated, but they simplified beautifully.