Flask series part 13: Moving our recipes data source to the server

Introduction

In the previous part of the series, in the process of allowing user recipes to be displayed in the main page, we realized a big limitation of our application:

With the data source for the ingredients being hardcoded in the front-end, we can't index "user-entered ingredients" in our list

This is a huge limitation since a user can add an ingredient that is very regional or unknown to most people, and, we won't be able to index it since our data source for the dropdown is in the front-end.

We will see how to address this problem and improve our application further.

Moving the data source away from the UI component

The idea we will be exploring, is that keeping the data source as close to where we are creating it, reduces the coupling of the data to a specific component, and it allows for a more fine grained control of the actual data that will be used to populate the component.

If we have a dropdown with an hardcoded set of options, we see that it is very static, as we can't control how the actual options that comprise the component data are set. In pseudo-code:

//we create and hardcode the options here.

dropdown = Dropdown([a,b,c])

//now the dropdown is fixed with these options. Static, and not flexible
someExternalService(dropdown)

when someExternalService will be using the dropdown, the options will be fixed, since the component was created with those options.

A good approach, which, is what we did with our component, is to pass the data that will be used to populate the component, as a parameter for the external service that will be using the component: like this, we are putting the data away from the component instantiation and instead, we inject it via the external service that will be consuming the component:

//we create and hardcode the options here.

dropdown = Dropdown([a,b,c])

//now we inject the component data via the service that will consume the component
someExternalService(dropdown, [a, b, c, d])

(...)

someExternalService(Dropdown d, Data opts):
     d.setOptions(opts)

So, this is the same approach we will take with our component, the data will be coming from the back-end, from a dedicated endpoint, and the long array of ingredients hardcoded on the JS side will be removed. Like this, we gain a lot of flexibility, we can add data from a different API, we can include ingredients added by users on their own recipes, read it from a file, etc, the possibilities are endless now!

Preparing the autocomplete endpoint

The endpoint we will design now, will be responsible for creating the JSON we will send to the front-end, containing a list of ingredients that match user input as a prefix: when a user presses "a", we need to send this as a parameter to the back-end, and, based on this letter:

• we need to filter all ingredients that start with it, from our long list of ingredients;

• we also need to get all the user added recipes, and, for those, we need to ensure that we only get ingredients with the matching prefix;

Let's see the code:

@db_session
@app.route('/autocomplete/<inp>', methods=['GET'])
def autocomplete(inp):
    ingredients = ["avocado", "apple", ....]
    user_recipes = select(recipe for recipe in UserCreatedRecipe)
    for recipe in user_recipes:
        ingredients.extend(map(lambda st: st.strip(), map(lambda s: str(s), recipe.ingredients.split(','))))
        filtered=filter(lambda ing: ing.startswith(inp),set(ingredients))
    return make_response({"listaing":filtered}, 200)

As described above, the ingredients array is exactly the same as we had in the front-end previously, and it contains all the ingredients we had as input for the autocomplete component.

Then, we get all the user entered recipes, and, for those, we get all the ingredients (which are entered as a comma-separated string by the user), then we split them by the commas and strip any additional spaces.

We then extend the original ingredients list with all the user entered ingredients. Extend simply allows us to add a collection to another.

Then, our prefix filter is quite simple:

filtered=filter(lambda ing: ing.startswith(inp),set(ingredients))

So, we transform our ingredients list into a set to remove all the duplicated, and check which ingredients start with the user entered input string, and, we build a JSON response with this filtered list, which we then send to the front-end.

With our logic in place for the back-end, we can now start looking into how to re-architecture the front-end.

Re-architecturing the front-end

The re-architecturing of our front-end code will be the most significant work we will do, and, along the way, we will learn about how to work with asynchronous code and how to write and use JavaScript constructs to deal with asynchronicity. These constructs will require ECMA6 (which we were already using).

Currently, in our autocomplete.js javascript file, we have all the code responsible for handling the autocomplete logic, including the data for the component.

As a recap, let's look at the code again:

function autocomplete(inp, arr) {
    /*the autocomplete function takes two arguments,
    the text field element and an array of possible autocompleted values:*/
    var currentFocus;
    /*execute a function when someone writes in the text field:*/
    inp.addEventListener("input", function () {
        let a, b, i, val = this.value;
(...) //rest of listener code
 b.addEventListener("click", function () {
        inp.value = this.getElementsByTagName("input")[0].value;
                    $.ajax({
                        dataType: "json",
                        contentType: "application/json; charset=utf-8",
                        type: 'POST',
                        url: '/addIngredient',
                        data: JSON.stringify({"ingredient": inp.value}),
                        success: function () {
                        console.log("success");
                            $('#searchBtn').trigger("click");
                        },
                        fail: function (data) {
                        }
                    });
                    closeAllExceptCurrent();

});
function closeAllExceptCurrent(element) {
        const x = document.getElementsByClassName("autocomplete-items");
        for (let i = 0; i < x.length; i++) {
            if (element !== x[i] && element !== inp) {
                x[i].parentNode.removeChild(x[i]);
            }
        }
    }
    document.addEventListener("click", function (e) {
        closeAllExceptCurrent(e.target);
    });

let ingredients = ["avocado", "apple", ... ];

autocomplete(document.getElementById("autocomplete"), ingredients);

As we can see, the ingredients list is hardcoded here in the front-end and we just invoke the autocomplete function in the end of the file, so, on the HTML side, after the JavaScript is loaded, we get the input listener on our input field and it all works from there.

This approach however, is not suitable for further development of our feature, as we also want to show the user ingredients in our dropdown.

As a first attempt, we might be tempted to write this incorrect code:

(...)
 let ingredients = []
 inp.addEventListener("input", function () {
        let a, b, i, val = this.value;
        (...)
        $.ajax({
        type: 'GET',
        url: '/autocomplete/' + enteredByUser
    }).then(function (result) {
        ingredients = result['listaing'];
        console.log("Ings are " + ingredients);
    }, function (err) {
        console.log(err); // Error: "It broke"
    });
(...)

The reason why this seems logical is that, in a very basic sense, it does what "we want": based on the input value that the user enters in the search bar, we use that value as a parameter for our endpoint, retrieve the list of ingredients, assign it to the ingredients array on the Javascript side, and then, call the function with our newly set array of ingredients.

However, this will not work as expected, because the code flow execution is asynchronous. What this means, is that the code doesn't execute as sequentially as expected. So, before the result of the AJAX request is received, the attempt to get the ingredients list into our function will be executed and we will have a call to the autocomplete with an empty ingredients list and we won't see any autocomplete results.

Making code sequential with Promises

What we want is to ensure that we can still have a guarantee regarding the execution to be predictable. We want to be able to confidently say: Once I receive the list of ingredients from the backend, I want to populate an array with such a list, and, once the array is populated, call our standard autocomplete function, with the certainty that we have the values we need.

We will use a construct called Promise in JS, which is basically used to control execution flows for asynchronous operations in a more synchronous manner. In other words, it allows us to do things in the natural order.

But, before we do, we can clean up the code itself, by getting rid of the input listener added inside the autocomplete function. We can replace it by binding the input listener on the HTML of the component via the oninput property and passing the value as input to a JS function. Let's see how:

 <label for="autocomplete"></label><input oninput="dropdown(this.value)" id="autocomplete" class='form-control' type='text' name='restaurant_name'
placeholder='Enter ingredients separated by commas...'/>

By passing the value as a parameter and calling the dropdown function on input change, we have "pushed back" the effect of the listener on the element to be done at the HTML level. Our new function will look like this:

function dropdown(input) {
    $.ajax({
        type: 'GET',
        url: '/autocomplete/' + input
    }).then(function (result) {
        ingredients = result['listaing'];
        autocomplete(document.getElementById("autocomplete"), ingredients, enteredByUser);
    }, function (err) {
        console.log(err);
    });
}

This is very compact, easy to read and to the point: we perform a get request to our newly created autocomplete endpoint using AJAX, which will fetch us the ingredients list from our backend that match the input by prefix, as entered by the user.

However, this is where things get interesting. We call the then function which in JS can only be called on the context of what is known as a Promise. Let's inspect the actual jQuery code itself for the then reference:

promise = {
    state: function() {
        return state;
        },
    always: function() {
        deferred.done( arguments ).fail( arguments );
            return this;
                },
    then: function( /* fnDone, fnFail, fnProgress */ ) {
        var fns = arguments;
        return jQuery.Deferred( function( newDefer ) {
        jQuery.each( tuples, function( i, tuple ) {
    var fn = jQuery.isFunction( fns[ i ] ) && fns[ i ];
// deferred[ done | fail | progress ] for forwarding actions to newDefer
deferred[ tuple[ 1 ] ]( function() {
    var returned = fn && fn.apply( this, arguments );
    if ( returned && jQuery.isFunction( returned.promise ) ) {                              returned.promise()                                      .progress( newDefer.notify )                                        .done( newDefer.resolve )                                       .fail( newDefer.reject );                               } else {                                    newDefer[ tuple[ 0 ] + "With" ](                                        this === promise ? newDefer.promise() : this,                                       fn ? [ returned ] : arguments);
        }
    } );
    } );
    fns = null;} ).promise();},
    // Get a promise for this deferred
// If obj is provided, the promise aspect is added to the object
    promise: function( obj ) {
return obj != null ? jQuery.extend( obj, promise ) : promise;
                }
            },

The idea behind promises is simple: they allow for things to be executed in the natural order in which they are expected to happen.

In terms of asynchronous code, like an AJAX call, as the name indicates, the code flow is not blocked during execution, so, if we set our array of ingredients inside an AJAX call success callback and after (outside of the asynchronous code block) we try to reference it, we may find that it might be empty. This can happen because the code flow keeps executing and a callback will be executed "in-between" the normal code flow when the asynchronous call completes, in other words, there is no guarantee of order.

By using the then chained call, what we are saying essentially is:

"I will wait for my AJAX call to complete and then I will execute the next actions"

Like this, with our previous restructure of the input listener, we achieve what we need:

We ensure that we only call our autocomplete function when we have all the data we need for it

The last thing we need to see, is how we can then connect the ingredients that appear in the dropdown with our normal code flow as we had before.

Connecting the ingredients in the dropdown to Flask

The last step we need to take as a measure to re-architecture this feature, is to ensure that when clicking on an ingredient in the dropdown, it gets sent to the back-end so that it can be added to our chips list and a search can be performed.

This is done again, using AJAX, and, we can use promises again:

let ingredientsPromise = new Promise(() => $.ajax({
                    dataType: "json",
                    contentType: "application/json; charset=utf-8",
                    type: 'POST',
                    url: '/addIngredient',
                    data: JSON.stringify({"ingredient": inp.value}),
                }), () => {console.log("Rejected")});

                function triggerSearch() {
                    $("#searchBtn").trigger('click');
                }

                ingredientsPromise.thenResolve(triggerSearch());

Here we wrap the AJAX call in a promise, in a more explicit way, using the native ES6 syntax.

The most important takeaway here is that the call to triggerSearch() is outside the AJAX code and inside a thenResolve call. This ensures that we will only "click" the search button when we are sure that our request succeeded.

Conclusion

Like this, we managed to re-engineer our autocomplete component to be wired to our flask backend and to ensure that we can populate it with data coming from the server-side while retaining all existing functionality.