MVC, AJAX and REST - Breaking out of the sandbox, part 1

The modern browser is the tool of choice for millions of computer users around the world. Many people live in their browser and never install or use any other application, and the power and breadth of the features offered are unequalled. So if you're thinking of creating a new software product with a GUI you may have considered building it as a web application, which among other things immediately confers cross-platform capability, transcending vast hardware differences as well as providing a very capable and fully programmable graphical environment.

Let's take an example. I have a web application, all written in JavaScript, that I would like to enhance with the ability to read and write files on my local system. I'll call this part of it the "File Manager". It's a nice idea but there's a problem. Browsers are intimately connected to the Internet, which is crawling with Bad Guys, so browser scripts run in a "sandbox" that denies them access to the local system and its hardware. They're not allowed to talk to the operating system, only to servers, so this will kill the idea before I even start. Right?

Let's turn the telescope round and look through the other end. As an application developer I have an impressive choice of languages; C/C++, Java, Python, even JavaScript, plus loads more. Maybe I'll rewrite my application in Python as it's among the most popular. There are a lot of features to rewrite so it'll keep me busy full-time, and that's just for the main program logic. Replacing the GUI is another whole heap of work, the effort of building it in Python is going to be considerable and the results will struggle to match what is currently done in the browser. Wouldn't it be great if I could just generate the same HTML as I do now and throw it at the browser, like what happens when I access a server?

Ah, is there a clue here? Maybe if I add a "localhost" webserver to my Python code I can get my local browser to act as the UI?

The answer is "Yes, I can". But before starting to code, let's look at how such a system might be organized.

Model-View-Controller

During the development of Smalltalk in the 1970s the notion of a "design pattern" came up, meaning a general, reusable solution to a commonly occurring problem. Among the first of these to be identified was Model-View-Controller. The 3 components of MVC are

Model

The part of the system that deals with data, whether in files, in a database or fetched remotely.

View

The part of the system the user interacts with; nowadays usually a visual display, though other paradigms such as buttons and lamps are not excluded.

Controller

The part where we keep the system logic and implement our "business rules".

The three parts of MVC are not always kept separate. Although Model and View are usually quite distinct, sometimes the Controller is closely integrated with the Model, sometimes with the View. I prefer not to be dogmatic about these things; if it makes sense to keep 3 distinct components then do so, but in general do whatever seems more natural.

In the case of my web app with its shiny new file manager, this is a client-server solution that looks to the user like a regular self-contained application. The MVC components are simple to identify; the Model is in the Python code, doing stuff with the system. The View is up in the browser. Which only leaves the Controller, which could actually be in either place, or even split between the two.

In a traditional client-server arrangement the server generates markup and sends it to the client for display. It does this for any number of simultaneous clients, which represents rather a waste of processing capability since the clients' power is largely unused. If some of the heavy lifting can be off-loaded to the client then the server can handle more clients.

In a situation where both the client and server are running on the same computer there is no need to delegate work to the client. It makes little difference where the work is done. If my preference is to write it in Javacript it goes in the browser; if instead I prefer Python then it's in the server.

It is useful, however, to make the communications between client and server as efficient as possible. So the Controller should probably be located close to whichever of the Model and View it has the simpler interface to. And the messages that flow across the interface between the browser and our embedded webserver are best implemented using REST.

REpresentational State Transfer

REST is a software architectural style that provides interoperability between processing units, typically computers on the Internet. This definition also covers multiple processes running on a single computer and communicating with each other.

REST does not mandate a specific command format; it just advocates guidelines that aim to make communications easy for humans to follow while still being efficient for machines. Everything is done with URLs and standard HTTP methods.

In the context of my file manager, this has some major benefits during development, the main one being that the two parts of the system can be developed separately. The server code can be tested with a tool like Postman, where you construct HTTP requests, send them off and examine what comes back. And the browser code can be tested with a stripped-down server that just returns known test values, or with a custom-built test rig.

AJAX

AJAX is short for Asynchronous JavaScript and XML. With AJAX, web applications can send and retrieve data from a server asynchronously (in the background) without interfering with the display or the behavior of the existing page. When AJAX arrived as part of "Web 2.0" it permitted web pages to behave more like regular applications, able to update parts of themselves without the need to reload the entire page, thus avoiding the long pause and the annoying flash while the data downloads and the browser redraws.

So now it's time to start coding.

The embedded server

The world of Python is not short of embedded webservers, the best-known of which is probably Flask, a powerful component that does far more than is required for my rather basic needs.

So instead I've chosen the less well-known Bottle, a simpler yet perfectly adequate product that can be installed with the usual one-liner:

sudo pip install bottle

Bottle comes with extensive documentation, and if that doesn't provide the answers, Stack Overflow holds a good deal more. Here's the very minimum need to get it up and running:

import bottle
from bottle import Bottle, run, get

app = Bottle()

# Endpoint: GET localhost:8080
# Gets the default home page
@app.get('/')
def home():
    return 'hello'

if __name__ == '__main__':
    app.run()

This server does just one thing: it says 'hello'. The syntax @app.get('/') is called a decorator and here it defines an endpoint, meaning one end of a communications channel. Here it tells Bottle that when a request is received for just the basic URL it should be handled by the function that follows the decorator. The name of the function can be anything you like, so choose something that makes its purpose clear, and the function can do anything as long as it returns some text (or nothing).

When the script is run it sets up a server on port 8080 and waits for requests. You can test it by calling it from your browser:

http://localhost:8080

which returns "hello".

Often you'll want to get the contents of a file, so here's the same server with a second endpoint:

import bottle
from bottle import Bottle, run, get, post, request, response, static_file

app = Bottle()

# Endpoint: GET localhost:8080
# Gets the default home page
@app.get('/')
def home():
    return getFile('home.html')

# Endpoint: GET localhost:8080/<filename>
# Returns the content of any file in the server space
@app.get('/<filename:path>')
def getFile(filename):
    response = bottle.static_file(filename, root='.')
    response.set_header("Cache-Control", "public, max-age=0")
    return response

if __name__ == '__main__':
    app.run()

Here I've modified the home() endpoint to call for a specific HTML file to be delivered, in this case home.html (which you'll have to provide yourself).

I've also added a second endpoint that returns the contents of any named file. Note how the filename parameter passed to getFile() matches the same parameter in the decorator. The <filename:path> filter syntax tells Bottle that filename can be a simple name or a path, and it returns any file from within the server folder tree.

For example, if you have a file called testfile.txt in a folder called static you can retrieve its contents:

http://localhost:8080/static/testfile.txt

The getFile() function sets a cache control header in its response, that governs how long a file will be cached after it's read. During development you will probably want to disable caching, as I've done here by setting max-age to 0 seconds.

Making it into an application

With just what we have so far we can set up a simple static website, adding folders for CSS and JavaScript, but we still have to fire up our browser and point it to the server port. That's not how a regular application behaves, so we'll add a little more code to make it happen automatically when the server starts:

import bottle, subprocess
from bottle import Bottle, run, get, post, request, response, static_file

# start up the browser (version for Linux/Chromium)
cmd = "chromium-browser http://localhost:8080"
subprocess.call(cmd, shell=True)

app = Bottle()

# Endpoint: GET localhost:8080
# Gets the default home page
@app.get('/')
def home():
    return getFile('home.html')

# Endpoint: GET localhost:8080/<filename>
# Returns the content of any file in the server space
@app.get('/<filename:path>')
def getFile(filename):
    response = bottle.static_file(filename, root='.')
    response.set_header("Cache-Control", "public, max-age=0")
    return response

if __name__ == '__main__':
    app.run()

The subprocess.call() command causes the browser to open a tab and put the requested page into it. The code here is for Linux, using the Chromium browser; you will need to adjust this for your own system.

There's just one more thing I want to say at this stage. Let's suppose you want to add an endpoint that does a specific job. As a simple example here's one that doubles the value given:

# Endpoint: GET localhost:8080/double/<value>
# Doubles the value given
@app.get('/double/<value:int>')
def double(value):
    return f'{value * 2}'

A couple of points to note here. One is that a number of filters are available; see the Bottle documentation. Here we're using one that expects an int.

The next point is that all endpoint functions must return a String value (or a null). So numbers have to be converted by some means such as shown.

The final point is that this endpoint has a "signature" (a pattern, in other words) that is functionally identical to that of the more generic getFile() we already have. This is fine as long as the function names are different and the more specific endpoint is given first. Once Bottle has found a match it doesn't look any further. In this case our double() endpoint must be placed before getFile().

To recap

So in outline, this is how to use your browser as the UI for a Python program. You can arrange to load up HTML files as needed, or you can put some JavaScript in the HEAD of a document that makes calls to the server after the page has loaded. I'll cover these things later on in this series.

Another point to consider relates to a comment I made earlier about choosing the most efficient means of communicating between the client and the server. Yes, you can generate in Python all the HTML you may need, but if you have some intelligence in the client you may have to pass far less data. For example, your UI may feature custom widgets that each require a big heap of HTML to implement. You could end up with huge amounts of data going to the client every time a simple action takes place. It might be better to send a template at the start, then each time it's needed you just supply the data to populate it. Or you could let the UI generate the whole thing each time, so rather than saying "Here's an ACME widget" you have instead "Make me an ACME widget". Sure, it means coding in Python AND JavaScript, but that's good as it widens your experience base.

In Part 2 we'll implement the File Manager in Python, using Bottle endpoints to interface to the UI on a browser.

Photo by Markus Spiske on Unsplash

Comments

[Anwar]

Great introduction to these building blocks that construct the web as we now it!

I am personally a very, very, big fan of a separation of concern, and naturally of client /server separation.

I like the idea of the client as a self autonomous UI entity that we feed some data (via our server, or an external API,...). I like this pattern a lot because it creates a modular system where you could change the client or the server independently without harming each other, and that means multiple person can work on it.

[Graham Trott]

With hindsight, the 'browser' doesn't actually have to be Firefox, Chrome etc. Any component able to render HTML to a UI, execute logical operations and communicate via HTTP will do just as well as far as the rest of the system is concerned. As you say, the client can be substituted without the server having to alter its behavior, or even to know about the change. It would also be free of the sandbox and could be written in any system language. Just thinking out loud - I have no idea where I'm going with this...

[Anwar]

I am actually making an app for my mum that holds a shoe store, and I used this pattern. It is the first time I am doing anything else than classical PHP backend serving the front end all it needs.

This time for this app, I used PHP but only to act as an API, serving just the data, and my front-end in Javascript is generating the HTML and with the help of some AJAX call, compiles the templates into visual components (I use the help of Vue.js to do so).

So far I am liking it, the web experience results in a smooth navigation, and no flash of loading. I just need to work out the display of the loading indicators, but else this is a very pleasant navigation experience. I recommend it.

[Graham Trott]

I came up mostly the same way. Over the years I added bits of JS and the bits gradually got bigger until the point was reached where I decided to roll them all up into a proper client-side programming language. Now I code as little as possible on the server, leaving it to just handle the system data and the comms to all the connected users.

Just a thought: When you need to poll the server for changes, a key decision is how often to poll. If you have many users the polling overhead can get quite large. Though I've never actually done it, I would add JS code to assess when the user is active, and increase the polling frequency so the system feels more responsive. Then when the user leaves the screen untouched for a while the polling frequency can be reduced. JavaScript has a rich set of features for detecting user activity.

[Anwar]

Absolutely! I was thinking the same, until I came up to this article (I cannot find it right now) where the author state that long polling can quickly exhaust the server, and advices to use a bi-directional web socket connection instead.

I personally use Odoo at my work (which is a Python based framework for rapidly prototyping UI for ERPs), and it uses internally long polling (for users to users chat, notifications, ...). So far, no downsides regarding using it, and this framework is widely used, so I guess polls are okay.

That is great to have this conversation, because your answers are actually what I was thinking of the future of web development: we will tend to create more and more responsive web app, until eventually reaching real-time capabability web app, using intuitive event driven features (like we can already with the Javascript Notification API).

The future looks bright, but I think the patterns are still to be reinforced, because for the moment frameworks that propose embeded long polling features are not legion.

[Graham Trott]

I just Googled bidirectional web sockets (which I've never come across before) and they seem to be well worth looking into. For the scenario in this article series, both ends will be on the same machine and interrupted connections are unlikely to happen. If that configuration is supported (I can see no reason for it not to be) then they greatly add to the benefits of using a browser as a GUI for a local application. If I get time I'll try them out; maybe I'll then need to write an extra part for the series!

[Anwar]

I actually just finished to follow this YouTube video in which a presenter demonstrate how websockets can be implemented in PHP by showing a plain PHP example followed by a simple chat app using the Ratchet frameworks :

youtu.be/Q7Us_DjMbXU

He also give his feedback on some things to have in mind, like maximum file opened (because web sockets are using file descriptors behind the scene). Very instructive.