Story Driven Design

Articles about building interactive, content-rich websites usually focus on the software structure and demonstrate the use of advanced JavaScript combined with a modern framework such as Angular, React or Vue. This article isn't one of them. Instead, I'd like to step back from the technology and look at the bigger picture.

Websites don't in general start as technical projects but as stories; descriptions, sometimes accompanied by pictures, of what the site should do. Stories are written by domain experts, not programmers. These are the customers who commission us to meet their real-world needs by constructing a new website, and success depends on how well we match our solutions to the expectations expressed in their stories.

To explain this I have an example called Here On The Map, which you can see working at https://hereonthemap.com. It's a demo website built to illustrate some specific technical features rather than meet any particular need. In design terms it's simple; what matters most is its functionality and the stories that describe it.

The 'story' of Here On The Map

Here On The Map (HOTM) has quite a simple user story, which goes something like this. We have a live map on which there are colored pins. The user can zoom and pan to reveal any part of the Earth's surface, and when they click a pin a panel appears containing information about the location of that pin. Users can register with the site; they are then able to add their own pins and write about them using a rich text editor. Images can be accessed by URL or uploaded to the system using the file manager provided. Here's what it looks like when viewing:

and while editing:

The above is just one of several stories that together constitute a full specification of the website's functionality from the point of view of its users. Most projects start with a similar outline description; a 'project brief' that can be taken to an engineering team to implement. But before we jump into the coding, here are some general issues.

General implementation issues

To start with, the platform used here is WordPress. Why? Because it's the simplest way to get basic functionality in place, on top of which we can build our website. A load of useful functionality is provided, that we can use or ignore as we please. The theme is fairly irrelevant; I chose one called No Header, a very basic theme that provides little more than a sidebar leaving two-thirds of the screen for my own content. It also provides some responsiveness for mobile compatibility by adjusting font sizes and moving the sidebar. The HOTM application knows nothing about the theme and in fact doesn't interact with WordPress at all.

The website is front-end driven. This is partly preference and partly practical. It's not impossible to deliver a similar site with traditional server-side code, but good responsiveness to user actions would be harder to achieve. Furthermore, the visual structure is pretty basic so it's quite feasible - maybe even preferable - to implement it entirely in browser code. There's a rather fuzzy line between a web page and a browser application; the former suggests more emphasis on content and the latter on function. Here we're rather leaning towards a browser application.

One thing that stands out when looking at the website is the way it's made up of functional blocks. Quite big ones, in fact. In particular:

• A Google Map
• A user registration/login module
• A Rich Text editor component
• Picture upload and storage

The usability of the website is crucially dependent upon the ways these blocks interact with each other, but it's hard to predict all of the details without first building an outline prototype of the site. Once it's running, the nature of some of the interactions will become clearer. There's a Catch-22 here, that without knowing all the interactions it's hard to set up a framework to encompass them all, but until it's built you won't know what all the interactions might be, and you might need to go back to square 1 again. The solution - and the whole point of this article, as you'll see - is in the use of APIs (Application Programming Interfaces) to separate stories from components.

Here are some of the interactions, i.e. the Business Logic:

"When you click a pin you see the corresponding article. You also get a link to the author and a list of tags. When any of these are clicked, the map updates to show only the pins matching what was selected."

"A line of text appears under the map to tell you what filter you currently have in place."

"As you zoom the map the current zoom level is shown at the end of a line of text below the map."

"When you log out by clicking the link in the line under the map, the viewer or editor panels disappear, just in case you were viewing something that should only be seen by registered users."

"If you are viewing a pin you created yourself it will be colored green. When you log out it goes yellow."

There are also various rules, such as what state information gets persisted as the application runs. We preserve the current latitude, longitude and zoom, also the name of the pin we're viewing. In the File Manager we hold onto the directory path so the user can return there next time. And so on.

These interactions and rules together form stories that belong to the domain expert. They tend to be rather fluid and subject to unpredictable change at short notice as a result of factors such as responding to competition, the introduction of new product types or the need to meet changing regulations.

Stories rarely get into the details of how a component works, only of how it interacts with other components. They should be expressed in such a way as to ensure quick understanding by both domain experts and programmers and kept where they can easily be accessed to make changes, not buried in arcane code. Ideally they should coded so as to bear at least a passing resemblance to the original project proposals or the discussions that lead up to a change request. Think long-term and think of the site maintainer, who will often be someone other than the original site builder.

There are two different kinds of programming going on here. Components are highly optimised, contained functionality with well-known, unchanging interfaces, but stories are random "glue" logic that expresses frequently-changing relationships between components. A component can be as complex as you like because few are ever going to see inside it, but a story should strive for clarity at all costs. These two aims are directly contradictory when applied to the same entity but not if the entities themselves are separated.

Stories and Components

From the above we can see a structure begin to emerge, one of functional components joined together by stories. If we can find a way to manage these two things independently the website will be much easier to work with than if we jumble everything up together. With interactive websites, maintenance is a serious issue. If the stories are not easy to find in the code, a future maintainer will have trouble understanding what should be happening and is likely to break things.

Some components are just large third-party JavaScript libraries. Here we have two in particular; Google Maps and CKEditor, the component that provides our rich text editor. Each of these is freely available from a CDN or can be installed along with the rest of the website. The key thing is they are both black boxes. We have no interest in how they are implemented, only in the interfaces they present to the world.

The rest of the components might be provided as standard libraries, WordPress plugins, one-off code specifically written for this project or some combination of code and library. Whichever is the case, we should design our application to use components in a clean and consistent manner.

That just leaves the stories. As I've already explained, these should remain visible in a form that's as close to the original as it's possible to get, and a good precedent for this exists in the form of SQL. Database applications may do all sorts of things with data, whether transforming it for display or using it to control layout, but the actual process of extracting it from a database is done in a way that is readily understood by database domain experts as well as by programmers. Can we express our stories in a similar way?

Levels of coding

General-purpose computer languages are for programmers. The rest of us speak and write in English or another human language, and we do it entirely using words. Human languages look nothing like JavaScript code; there are no curly braces, fat arrows or triple-equals, yet we humans process them pretty well and even manage to organize whole societies with them. Another clear difference between human and computer languages is that one you can speak; the other you can't. Not without sounding daft, that is. This may appear to be a trivial point, but most human communication is verbal, not written, and in most languages the way we write closely mirrors the way we think and speak. As soon as you move away from this principle you lose parts of your audience, communications start to suffer and projects fail. Many programmers assume that anyone can follow their work by "reading the code", but in fact this is a rare skill, even among programmers. It's very unwise to assume someone else - or even your future self - will be able to make sense of what you write today without a great deal of effort.

When we come across an unfamiliar entity in daily life, rather than spell it out longwindedly every time we encounter it we give it a name such as a Gazebo or a Defibrillator. Or, to return to the context of HOTM, a Map and an RTF Editor. We don't concern ourselves with how the item works, only with what it does, that is, the interfaces it presents to us. This works in the computer world too. You can draw a map with just 3 items of information, by saying

"draw a map in the box, with latitude 45.234, longitude 5.82 and zoom 5.5"

The one and only absolute requirement for a viable computer language is that it must be unambiguous, so the instruction above qualifies as a valid computer language statement while remaining perfectly readable English.

Of course, we'll need a compiler and runtime. HOTM uses tools that already exist in the WordPress ecosystem, but they may not suit everyone. A competent programmer will need only a few months to build a basic high-level script processor using a conventional system programming language. Many companies build custom tools to help with their own processes; this is just another example. You get maximum bang for your buck by having the compiler run in the browser, so JavaScript would be the obvious choice, but some might prefer a stand-alone application written in Python or Java, where only the compiled code is sent to the browser. Any programmer capable of mastering React is more than able to tackle the job, and once it's done it can be leveraged over and over again, saving huge amounts of time on future projects.

None of this is visible to the people implementing the stories. All they see is a syntax that lets them express all the features they want to use and that provides access to the APIs of the major components.

By layering our application in this way we can maintain high readability, which gives us some confidence that future modifications can be made quickly and safely by whoever is available to do the work.

How to do it

Here On The Map was built to demonstrate the principles outlined above. It's written entirely in high-level script, using a syntax derived from English and a vocabulary that encompasses all the concepts and activities required. The compiler and runtime package are a WordPress plugin called EasyCoder (https://easycoder.software) that's available from the WordPress plugin library. To illustrate the general flavor, here's a simple fragment from a user story:

"When given the id of a particular pin record in the database, read the record, set up the map and display the article for that pin"

and here's the corresponding section of EasyCoder script:

rest get Record from `_/ec_markers/id/` cat RequestedID or
begin
    print `Failed to load the requested pin.  Error: ` cat the error
    stop
end
if property `id` of Record is not 0
begin
    set the latitude of Map to property `latitude` of Record
    set the longitude of Map to property `longitude` of Record
    set the zoom of Map to property `zoom` of Record
    update Map
    fork to ShowStory
end

As you can see, there is no computer code to learn since everything is expressed in English words. Admittedly, the syntax is a bit clunky, but it's still understandable. Names of things are camel-cased while language commands remain in all lower case.

The first command requests a record from the database by calling a REST server. There's a basic one included with the plugin and it has an extension facility to enable the provision of the extra code needed to handle the specific commands associated with maps. In the event of an error a report is logged but nothing else happens and the screen remains unchanged.

Assuming a valid record came back it will have a non-zero id, so we can extract the 3 map parameters and update the map. Here we're using the Google Maps module, wrapped in simple commands that let us access its API without worrying about the details.

And finally we go off to show the article for the pin. The 'fork' command merely ensures that the story is shown in parallel while processing continues in this thread. (No, we haven't discovered a hitherto-unknown threading feature in JavaScript; threads in this context are provided by cooperative multitasking inside the language engine.)

The HOTM website documents all the EasyCoder scripts used for the application, plus the PHP REST server extension. The total amount of code is a lot less than the equivalent JavaScript would be, with or without a React framework to help it. Of course, driving all this is a compiler/runtime JavaScript engine of about 50kB and further components that bring the total close to 200kB if they are all used. Plus the scripts themselves, loaded from GitHub where they are included with the Open Source EasyCoder project. The site loads in under 3 seconds, but over half of this is just getting the initial WordPress page from the server, after which things fly along. Scripts compile in a few tens of milliseconds and compilation usually happens in parallel with other things like image downloads.

A website can be built very quickly using EasyCoder, partly owing to the way each command does a lot of work and partly because the scripts express concepts that directly map to what you see on the screen, but also because so much of the usual infrastructure needed by websites is already provided. The bulk of HOTM took less than 2 weeks from an initial idea to a fully working website. Errors are rare because the internal functions are used over and over again, leaving bugs with few places to hide. However, it has to be acknowledged that this is a fairly small website using readily available technologies, so what if you need to build some really complex functionality?

Mixing technologies

People have a tendency to favor particular ways of working, and programmers are no exception. The current hot subject is JavaScript frameworks, with React leading the pack, and its adherents often suggest it should be used for everything. I disagree. There's nothing wrong with React but I don't want it in any code that is likely to be modified - or maybe even seen - by my customer. All that complex stuff belongs at the component level.

The reason is my wish to keep the stories visible. JavaScript code, however carefully written, usually fails to reveal user stories in a form that can be even found, let alone easily understood by anyone other than the original programmer. Without understanding, many more problems soon follow. I've demonstrated above that functionality can be split into one level for the stories and another for the infrastructure, and it's in the latter area that React and JavaScript belong.

A key feature of React is that it's a component technology. This works well in the environment described here, where major units of functionality are built as components and joined together by the random glue of stories. Anything with a well-defined API can be a component, including complex visual elements built using a modern framework.

As for the glue, the EasyCoder high-level scripting language used by HOTM is built out of individual plugin modules that each perform a bounded range of duties, either for a specific domain or for a particular technology. Examples are DOM and Google Map plugins or commands that interface to DropBox. The core of it just provides basic language features needed anywhere. Writing new plugins is simple; you code them to handle the syntax chosen to deal with functionality exposed by a component through its API, which itself will often be delivered as minimized JavaScript from a CDN. How these components are engineered is entirely down to their authors, and if they are most comfortable using React then that's what it will be.

Remember at all times that what we're doing is creating a top-level syntax that's readily understandable by English speakers who are also domain experts. The complexity needed to implement this syntax is handled by the programmer, who is then free to pick the best tools for the job. If you wish, you can construct a language that has just one word; "doit", everything else being hidden inside that one command. That's how websites are in effect built today, but I don't advocate doing it that way. We can do better.

To a certain extent, adopting this strategy tends to focus the mind on building better reusable components. When you build a component alongside the code that uses it there's a tendency for ad-hoc interfaces to be added to solve short-term needs, making it progressively more difficult for the component to be used outside of the context for which it was originally built. The discipline of having to build the component in such a way that it can be accessed solely through published interfaces forces a reappraisal of just what the component is offering to the world, and in the process makes it a better component. And better components lead to better, more reliable and less costly websites.

Title photo by Simon Daoudi on Unsplash