(In a hurry? Jump to the conclusion!)
This is an attempt to measure how good are front-end frameworks in dealing with simple errors.
I took the implementations of TodoMVC made in 5 different frameworks: VanillaJS, React, Vue, Svelte and Elm.
Then I injected 7 different errors and I analyzed the behavior.
After injecting error in the 5 implementations, I measured the outcome using this scale (ordered from best to worse):
- A. Impossible to introduce such error
- B. Compiler error
- C.
- Nothing render on the screen and there is an error in the console at start
- The app render and there are no bugs
- D. The app render on the screen, it has a bug and an error in console at start
- E. The app render on the screen, it has a bug and an error in console during execution
- F. The app render on the screen but it has a bug, end there are no errors in the console
They are ordered in a way that I find more desirable. For example, having the app not showing at all (rank C) is better than having it appear on the screen but with bugs (ranks D, E, F) because in the first case it's easier to note that there is something wrong.
The best case is when errors are not even possible to be added at all (rank A) or are intercepted by the compiler (rank B). The compiler-error case is only possible for Elm, Svelte and React JSX. Everything else is not compiled.
The worst case is when the app renders on the screen and there are no errors in the console whatsoever (case F). It means that if we don't test our application carefully we are likely to deploy a bug in production.
I also rated the accuracy of the errors:
-
Very accurate, errors that possibly suggest the solution to the problem (add a
+to the score). - Regular, errors that give a good explanation and position of the error (no changes to the score).
-
Unclear, error with bad explanation or wrong indication about the error position, (add a
-to the score).
Disclaimers
- My bias: I work primarily with Elm and I am an advocate for functional programming.
- I use the word "front-end frameworks" in a very loose way here, so that it includes all of them because some of them are languages with a compiler, others are just Javascript libraries.
- I changed the code with a plain text editor. No linters or any other plugin.
- I didn't run any automatic tests. A good development environment should be able to detect all these issues through the assistance of the IDE or the test suite. Here I am judging the naked framework.
- I didn't use Typescript or Babel to transpile the code. I only used the Svelte and Elm compiler as these are integrated parts of the language.
- Using the code of the TodoMVC application. The result may be affected by the quality of that particular implementation. Different implementations written using the same framework could get different scores.
- Sometimes it is not simple to introduce exactly the same error in different languages/implementations so sometimes I had to modify the code slightly.
- All the code modifications are available at github.com/lucamug/framework-resilience. Each of the seven errors has its own branch so you can verify how these errors were implemented.
Let's get started!
1. Misspelling one HTML element
I changed the opening of a <span> element to <spam>, leaving the closure intact, in the footer of the application: <spam>...</span>
Outcome
Vanillajs
The application renders but the footer is not there. There is an error in the console: Cannot set property 'className' of null.
This is a typical example of an unhelpful error, the text does not contain any hint on what could be the cause and the file where the error happens (view.js) is different from where the error is (index.html).
Rank: D-
React
The application doesn't render and shows an error in the console at start:
Uncaught Error: Parse Error: Line 33: Expected corresponding JSX closing tag for spam
This would be intercepted by the JSX precompiler.
Rank: B
Vue
The application renders, but the footer has a wrong layout.
There is a clear error in the console:
vue.js:525 [Vue warn]: Unknown custom element: <spam> - did you register the component correctly?
Rank: D
Svelte
Nice compiler error
ParseError: </span> attempted to close an element that was not open
Rank: B+
Elm
Changing only one element is not possible in Elm. A span element, for example, is created with span [] [ text "content" ].
Rank: A
2. Misspelling two HTML elements
This time we use spam both in the opening and in the closing element. This is a subtle mistake because we could have wanted to use a custom HTML type. But let's assume that is a mistake and see which framework detects it.
Outcome
Elm in the only framework that has a compiler error:
In Elm, to create a spam element you need to write
node "spam" [] [ text "content" ]
instead of
spam [] [ text "content" ]
Other frameworks either just work or they have a warning:
This is the case of Vue that has this in the console:
vue.js:525 [Vue warn]: Unknown custom element: <spam> - did you register the component correctly? For recursive components, make sure to provide the "name" option.
- Rank Elm: B+
- Rank Vue: C+
- Everything else: C
3. Forgetting a space between a and href
Just replace <a href="..."> with <ahref="...">. I did this in the footer button "Active" that displays the active todos.
Outcome
VanillaJS
No errors at all for VanillaJS, but the button "Active" doesn't work.
Rank: F
React
React return the error Unexpected token <ahref="#/" and it doesn't render the application. Another error that will be detected by the JSX compiler.
Rank: B
Vue
No errors at all, the same as VanillaJS, and the footer is also broken:
Rank: F
Svelte
Compiler error ParseError: Expected >
Rank: B
Elm
Is difficult to add this error as a and href are separated by a [ in Elm:
a [ href = "url" ] [ text = "label" ]
so I added the error this way
a [ ahref = "url" ] [ text = "label" ]
That generate this compiler error I cannot find a 'ahref' variable: These names seem close though: href
Rank: B+
4. Misspelling the content of a href
Remove the content of href in the "Active" button. From href="#/active" to href="#/activa"
Outcome
Changing the content of a string, as expected, goes undetected by all frameworks except VanillaJS and makes the link "Active" not working anymore.
VanillaJS give an error at runtime:
controller.js:254 Uncaught TypeError: this[("show" + activeRoute)] is not a function.
Strangely enough, the Elm application kept working also with this modification!
I investigated the reason and I found out that
- The Elm application apply changes "onClick" instead of waiting for the url changing
- The Elm application store the entire model in the local storage while all other applications only store the list of todo items
As I consider this some kind of "cheating", I downgraded Elm to the same rank of all other applications.
This error is telling us that having strings in the code is usually a bad idea. This is why in Elm, and probably in other frameworks too, we usually write the routing differently.
Specifically
- We create a custom type containing all possible routes
- We create a function
routeToStringthat convert such type into a string containing the url of the route
So, using the function routeToString as href when we create links, assure that this type of errors cannot happen. It also has the nice side effect of making the code more maintainable if, in the future, we decide to change the url format or name.
Rank: VanillaJS get an E-, all other applications get an F.
5. Corrupting the initial state
We change the code where the state is initialized, changing the name of an object key or a variable.
Outcome
Vanillajs
There is an error in the console:
store.js:21 Uncaught ReferenceError: todos is not defined
and the app is not working. The error happens only if the local storage is missing.
Rank: D
React
Unclear error in the console:
app.jsx:96 Uncaught TypeError: Cannot read property 'filter' of undefined
but the file with the error is todoModel.js:18
The app doesn't render.
Rank: C-
Vue
Unclear error in the console:
vue.js:525 [Vue warn]: Property or method "todos" is not defined on the instance but referenced during render. Make sure to declare reactive data properties in the data option.
Rank: C-
Svelte
The compiler gives a warning
Plugin svelte: 'items' is not defined
but not an error, so it is still building the application.
The application doesn't render and in the console there is the same error Plugin svelte: 'items' is not defined
Rank: Between B- and C-. I will go with B- as maybe is possible to configure the compiler so that it will stop the compilation, instead of just giving a warning.
Elm
In Elm we change a key of a record from entries to entriez. This is the compiler error:
Rank: B+
6. Corrupting the function that add a new todo item
Here I mostly change the variable name containing the title of the new todo.
The interesting part of this error is that it reveals itself only when users are adding a new item, so it is harder to detect compared to the others.
Outcome
Vanillajs
Error when adding a new todo:
controller.js:17 Uncaught ReferenceError: title is not defined
Rank: E
React
Error when adding a new todo:
todoModel.js:36 Uncaught ReferenceError: title is not defined
Rank: E
Vue
Changed "value" to "valua". Error when adding a new todo:
app.js:72 Uncaught ReferenceError: value is not defined
Rank: E
Svelte
Changed "description" to "descriptiom". Error when adding a new todo:
bundle.js:1031 Uncaught ReferenceError: description is not defined
Rank: E-
Elm
Changed "desc" to "descr". Compiler error:
I cannot find a 'descr' variable. These names seem close though: 'desc'
Rank B+
7. Forgetting to add the method .length to calculate the active todos
Usually the total quantity of active todo is calculated by filtering all todos and counting the remaining items with length.
I remove ,length where possible. If the implementation was using a different system for counting active todos, I refacto that a bit so that I could introduce the same mistake.
Outcome
Vanillajs
The app output [object Object] instead of the number of completed todos:
Rank: F
React
The app output the entire list of items instead of the number of completed todos:
Rank: F
Vue
The app output the entire list of items instead of the number of completed todos:
Rank: F
Svelte
The app output [object Object] instead of the number of completed todos:
Rank: F
Elm
Changing from
entriesCompleted = List.length (List.filter .completed entries)
to
entriesCompleted = List.filter .completed entries
Generate this compiler error:
Rank: B
Conclusion
This is the summary
Error n. 1 2 3 4 5 6 7
--------------------------------------
VanillaJS D- C F E- D E F
React B C B F C- E F
Vue D C+ F F C- E F
Svelte B+ C B F B- E- F
Elm A B+ B+ F B+ B+ B
Let's try to quantify the result giving some number:
A: 10, B: 8, C: 6, D: 4, E: 2, F: 0
For the errors we add 1 when there is a + and subtract 1 when there is a -.
Error n. 1 2 3 4 5 6 7
---------------------------------------------
Elm 10 8+ 8+ 0 8+ 8+ 8 = 50++++ = 54
Svelte 8+ 6 8 0 8- 2- 0 = 32+-- = 31
React 8 6 8 0 6- 2 0 = 30- = 29
Vue 4 6+ 0 0 6- 2 0 = 18+- = 18
VanillaJS 4- 6 0 2- 4 2 0 = 18-- = 16
It seems evident from the result that a compiler helps to detect these type of errors (Elm, Svelte and React).
Elm leads the rank mostly because it is strictly typed and the fact that everything in Elm is code, including HTML.
And also thanks to the high quality of its error messages.
Further developments
- Is it possible to have a more objective analysis?
- Was it fair to mix framework with compilers and framework without?
- How different would the result be with Babel and/or Typescript?
Thank you for reading and let me know what you think in the comments below.
Illustrations by Katerina Limpitsouni, from https://undraw.co/
Top comments (1)
Nice job. I will show this to those who don't understand the difference of type systems effect on error detection.