DEV Community: Jonas Liljegren

Getting started with web development in 2024

Jonas Liljegren — Wed, 20 Dec 2023 11:56:04 +0000

Short history

1994: Netscape Navigator made the web popular.

2004: Internet Explorer by Microsoft destroyed Netscape, dominating with 94% market share.

Then started the browser war, with Firefox, Chrome and Safari. It became a real pain trying to create websites that would look and work as intended in all popular browsers. All the browsers had numerous differences in implementation and bugs. Libraries like jQuery were created for helping building sites that would work on most versions of most browsers. Many features could not be used since they would not work the same on all browsers. New features could not be used since you would still need to support all the older browsers used by people.

All this changed in 2014. HTML5 would revitalize the web after many years of XHTML. ES5 would finally get full support in all the browsers. But more important than this was the introduction of Babel.

NodeJS is a javascript engine that lets you run programs outside the browser. Babel was one such program that can rewrite other javascript programs so that they can run on older versions of browsers. That would let you write javascript using the latest version ES6, and still having it work on older versions of all the popular browsers. ES6 introduced a lot of new things including class declarations.

The power of Babel allowed for programs to use packages from the NodeJS ecosystem. Babel would transform them for compatibility with the web and tools like Webpack would optimize them by minimizing the size and bundle all the files together for faster load times. This led to an explosion in popularity and the rise of React.

The evolution of the web platform

Around the same time in 2014, all the major browsers would start to use automatic updates. Instead of many browsers being 5 or more years old, most users would have the latest browser while using the web. This was the rise of the evergreen browsers, that would eventually shorten the time for new features being available for almost all of the users.

Every year there would be lots of new functionality introduced. The last couple of years would see major improvements every single month. Popular libraries and frameworks would serve as inspiration for new features added to the browser.

This means that popular libraries with much needed features bit by bit would be made obsolete by similar features being built into the language and web api in the browser. Popular frameworks like React, Angular and Vue have evolved but their core architecture is based on the capabilities of the web at the time of their creation. They are still using build tools with transpilation and bundling even though this is no longer necessary and makes the whole framework much more complicated than it has to be.

Every added library comes with a cost. Updates in one part can force an upgrade of other parts. There may be security vulnerabilities or bugs. Every added interdependent component makes it harder to replace or fix functions that don't do what you want. The framework will limit what you can do, making many custom solutions complicated and suboptimal. More time will go to adapting what you want to do to the capabilities and architecture of the chosen framework. Every library and every part of the toolchain and build process is another potential point of failure. Installing the example project with create-react-app will download 1500 package dependencies totaling over 2 million lines of javascript code. At the time of writing, they include 6 high level vulnerabilities. How long will it take you to understand what's happening across all those lines of code?

Since frameworks like React and Angular have cemented its position as a de facto webstandard, there is a huge ecosystem of existing sites, apps, libraries and people. Mastery in these prevailing toolchains are required by more than 95% of existing jobs. But this article is not about culture or popularity. It’s about how to do better. And regardless of what framework you end up using, you should still learn the foundation and capabilities of the modern web. And be prepared to re-learn every year.

The modern web

There are three main browser engines today. These are the ones used by Chrome, Safari and Firefox. Most other browsers are using Chromium, which is the engine known for Chrome. About 95% of all users have a browser version that is less than two months old.

The core features of the modern web includes ES6 (2017), HTTP/2 (2017), CSS Grid (2017), Shadow DOM (2018), CSS Container Query (2023), importmap (2023), Declarative Shadow DOM (2024) among many other Web APIs, javascript features and CSS modules. Many of these features have conditional polyfills that is javascript code that will make the feature work even in older browsers. The years given is the time it was supported in all three engines.

This article will start from the beginning for how to develop web applications that use npm packages. I may continue in later articles with more considerations often covered by popular frameworks, like routing, user sessions, database access, state management with reactivity, Server Side rendering, testing, documentation and deployment.

VS Code

I started out with Emacs and still use it daily. You can use whatever text editor you like. But it should include syntax highlighting, indentation and some sort of integrated documentation like code autocompletion. Microsoft Visual Studio Code is very popular, especially for web development.

You will need a web server for serving the website. The easiest to start with might be the ritwickdey.LiveServer extension for VS Code. Open your project folder, create a html file and then Go Live using the button at the bottom right.

I hate all distracting elements including all the helpful popups that VS Code throws at you. The editor has thousands of things to configure to your liking. I disabled most things related to quickSuggestions, suggestOnTriggerCharacters, cursorBlinking, inlineSuggest, parameterHints, suggest, snippetSuggestions, closeEmptyGroups, autoClosingQuotes, autoClosingBrackets, autoClosingTags and lightbulb. I also removed some of the columns in the editor like minimap, glyphMargin and showFoldingControls. I’m also using the extension garaemon.vscode-emacs-tab and bound it to shift enter for indenting the current line.
I also have a personal “Cheat sheet” for VS Code.

ES modules

NodeJS introduced CommonJS modules (cjs), but those only work on the server. You will recognize them by their use of require for importing libraries. ES6 introduced platform support for modules (mjs) using import and export syntax, which will also work in the browser.

Most examples of importing libraries are using bare module specifiers. Those assume you are using nodeJS or that you are transpiling the code before running them on the browser. You don’t need a compilation step. Just provide the actual path to the module you want to require, using a relative or absolute path. The mjs file extension may not be associated with the mime type text/javascript in older web servers. You could use js instead, but I prefer mjs since that is suggested by NodeJS and I use it as a way to declare them as real modules and not something that has to be transpiled. You can add the mime type to the web server if needed.

Here is a complete example of module use for remembering form input, in case the user accidentally closes the page before submitting.

index.html

<!DOCTYPE html>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
    textarea { width: 100%; min-height: 10rem; box-sizing: border-box; }
</style>
<body>
    <textarea disabled id="notes"></textarea>
</body>
<script type="module">
    import {persist_input} from "/utils.mjs";
    persist_input( document.getElementById('notes') );
</script>

utils.mjs

export function persist_input($el){
  $el.value = localStorage.getItem( $el.id );
  $el.disabled = false;
  $el.addEventListener('input', on_input);
  function on_input() {
    localStorage.setItem($el.id, $el.value);
  }
}

A larger project will usually use several layers of nested folders. For those projects, it’s usually easier to use absolute paths. But while using absolute paths, you must keep track of what folder to use as the web root. It’s usually named public. You can tell the Live Server extension to use that folder by creating a file .vscode/settings.json with the content

{
  "liveServer.settings.root": "/public"
}

Web components

Web components are the standard for reusable encapsulated components. Not to be confused with the non-standard framework-specific component systems such as react components and the like. Components lets you create your own html tags with any functionality you like. Here we encapsulate the persist_input function as a component:

persist-input.mjs

import {persist_input} from "/utils.mjs";
class El extends HTMLElement {
  connectedCallback(){
    const $inputs = this.querySelectorAll("input,textarea");
    for( const $inp of $inputs ) persist_input( $inp );
  }
}
customElements.define("persist-input", El);

And now we can use it for persisting all input elements inside the tag:

<!DOCTYPE html>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
    textarea { width: 100%; min-height: 10rem; box-sizing: border-box; }
</style>
<body>
  <persist-input>
    <textarea disabled id="notes"></textarea>
  </persist-input>
</body>
<script type="module">
  import "/persist-input.mjs";
</script>

NodeJS

NodeJS is the server side javascript engine used by most frontend frameworks. It can run under Windows, but will work better on a unix-like system like MacOS or Linux or the Windows Subsystem for Linux (WSL). You will be using a terminal with the Bash shell for command line interface (CLI) tools. Start by installing the Node Version Manager (nvm). With that, you can install the latest stable version of node.

Each project can have its own installed CLI tools, located in ./node_modules/bin. Those can be run by using npx, the relative path, or with a tool that checks the project bin dir using bash command_not_found_handle(). Keeping the tools with the project makes it easier to share the code and make sure that the project works with the version of the tool installed.

The modules we created don't need to be compiled. But many modules from npm do need a compilation step. We often need to change the module paths used inside the imported modules so that they can work in the browser. We can use rollup for repairing the paths in the module. We will have to tell rollup to not bundle or mangle the files. We only need the path translation.

An alternative to repairing the import paths is to use the <script type="importmap"> feature for mapping directories or bare module specifiers to their actual location. For this, you can use a tool that finds all the module paths and places them in an importmap.

Lit is a library that can simplify writing web components. During rollup, we can tell it to use the uncompressed development version of the files. In most cases, there is no need to use minified files. I believe in giving every user the freedom of seeing what's happening in the pages they visit. We can always optimize later if it's truly needed.

Standing in your project directory, install lit and rollup with npm i -D lit rollup @rollup/plugin-node-resolve. This will create a manifest in package.json and install the modules in node_modules.

Here is the rollup.config.mjs I use for repairing the files. Use the config by running npx rollup -c.

import resolve from "@rollup/plugin-node-resolve";
export default {
 input: [
   "lit",
   "lit/decorators.js",
 ],
 output: {
   dir: "public/x",
   format: "es",
   preserveModules: true,
   minifyInternalExports: false,
   entryFileNames: (chunkInfo) => {
     return (
       chunkInfo.name
         .replace("node_modules", "")
         .replace("/index", "") +
       ".mjs"
     )
   },
 },
 plugins: [
   resolve({ exportConditions: ["development"] }),
 ],
}

The result is put in the folder x. I like short names, but you may want to call it external or vendor to show that it is an external dependency.

Now we can use the library for creating components, like this minimal

say-hello.mjs

import {html, LitElement} from '/x/lit.mjs';
class El extends LitElement {
 render() { return html`<p>Hello</p>` }
}
customElements.define('say-hello', El);

And now it can be used as a component with

<!DOCTYPE html>
<body>
 <say-hello></say-hello>
</body>
<script type="module">
 import "/say-hello.mjs";
</script>

npmjs.com has millions of packages. And this is how you can use them for building web applications.

Optimized delivery

Once upon a time we used HTTP/1 for delivering files. That protocol had an overhead for each file, so we used bundlers to minimize the number of files the browser needed to fetch. This is no longer the case with HTTP/2 and HTTP/3. It doesn’t make a difference if the content resides in one large file or 1000 small ones. In fact, it’s much better to keep them as many small files, since the browser will only fetch the files that are needed for the current page, and only if the files have updated since the last fetch.

For a tree of dependencies where module 1 imports module 2 that imports module 3, the browser will only start to download the dependent modules after the previous ones have been loaded and parsed. This is called the network waterfall. For optimizing the load, you can just declare what modules to load directly from the html page. One way to do that is to use <link rel="modulepreload" href="/my-module.mjs"> tags at the beginning of the page. You can use the modulepreload tool from npmjs to automatically look up all nested dependencies of the page and inject the modulepreload tags that will eliminate the network waterfall. In our example, this will add

<link rel="modulepreload" href="/x/lit.mjs">
<link rel="modulepreload" href="/x/@lit/reactive-element/development/reactive-element.mjs">
<link rel="modulepreload" href="/x/@lit/reactive-element/development/css-tag.mjs">
<link rel="modulepreload" href="/x/lit-html/development/lit-html.mjs">
<link rel="modulepreload" href="/x/lit-element/development/lit-element.mjs">
<link rel="modulepreload" href="/x/lit-html/development/is-server.mjs">

I did my own tool for generating preloads before found this newly published tool.

30 years dreaming of the semantic web

Jonas Liljegren — Fri, 01 Dec 2023 07:11:13 +0000

I started writing HTML in 1995 after enrolling at the university informatics department. That was the year when Netscape just started adding new capabilities to the web, like background images, colors, tables, javascript and animated gifs.

The first parts of what would become paranormal.se were created at that time. In those days, a homepage was your home on the web. It was a place for your notes, bookmarks and contact information. I have basically kept my homepage from that time, with just some minor updates to keep it usable and somewhat up to date over the years.

I soon found my home at the student computer club. Spending all nights learning programming in Perl on the Solaris, while listening on MP3s of Belinda Carlisle, Tracy Chapman or anything else I found on the server. Installing qtest1 on all the lab computers for checking it out.

I used Perl for creating the first online facebook for the students at the informatics department. Also created the student newspaper CMS (Snutten), and the computer club website (SKIP) along with the growing paranormal site.

I have always preferred dynamic languages that cover the space between low level functionality and high level concepts, from ARexx to Perl to Javascript. Living on the edge, submitting bug reports and patches to the language or libraries since around 1996.

Paranormal.se was a project to create a semantic knowledge base with a hypertext frontend. The idea was to discover and surface new knowledge by integrating a large body of research and existing theories. I was hoping to create a sort of discussion forum where each new discussion would help to organize all existing information, making the whole continually more connected and refined. So many algorithms today work by surfacing the most popular and generally understood thinking. I had an idea of how to nurture ideas and perspectives and let them grow until more people understand them. Unlocking the secrets of the universe.

Dreaming of Xanadu. I love the original vision of Ted Nelson, who coined the terms hypertext and hypermedia. This would later inspire Tim Berners-Lee creation of the Web. I was deeply involved in the semantic web during the introduction of RDF in 1999. The dream of connecting all the world's knowledge with machine-readable semantic markup and a universal API for all public databases. The ontologies would allow for artificial intelligence to answer any question using deduction with symbolic logic. This is still very much a central part of knowledge representation today and used in combination with machine learning and large language models.

Changing workplace, I picked up my personal GNU/Linux server with all my sites and mail, to find a new home at the next company. This would be the place where I fully developed my hypertext dream, using the technology both for the company and the personal project.

Paranormal.se version 3 (P3) consists of about 30.000 topics and 16.000 users. It was a very active community around 1998 - 2008. This was before Wikipedia and Facebook. In 2007, I started developing version 4 (P4) with a much better system for version management, graph database flexibility and deduction capabilities. By 2009, I was burned out.

P4 was supposed to use my semantic database library RDF::Base (RB), which was created to run on my application framework Para::Frame (PF), built with Perl. PF and RB have been continually developed and refined since 2003. It's powerful and versatile, including multisite frontends, content management, request tracker, email handing, translations, external API and load balancing. All of it built on a semantic database inspired by RDF with typed literals, reified statements, class schemas and syllogism inference. Since it's a temporal database, parts or whole of queries can be constrained to a selected date.

PF and RB was the foundation for Hotellguiden.se from 2003 to 2009. After that, the system would continue to serve as the main customer management database, handling bookings and statistics, newsletters and even the internal call center administration, integrating all the business systems.

GOV, a web voting system for liquid democracy, was built in 2009 by my brother, for Aktiv Demokrati. It was built using RDF::Base and included ranked pair condorcet method for voting on lists. The liquid democracy was implemented by letting users delegate voting to other users. If you did not vote, the vote of your delegate would be counted. This also handled prioritized lists of delegates, voting jurisdiction and the temporal information of membership and voting history. I took over the development in 2010 and continued its development until 2017.

I have administered a lot of sites for forums, blogs or facebook pages related to groups I participated in. This includes Perl Mongers, Aktiv Demokrati, Occupy Göteborg, Zeitgeist Movement Sweden, Direktdemokraterna, in addition to many more personal sites. I also produced a lot of IRL events for Zeitgeist and Direktdemokraterna, including several Z-day conferences and election campaigns. This included a lot of marketing on the streets, public speaking and also online campaigns using ads, facebook and emails. Also did a lot of printing material, recording and editing of audio and video.

All of this ended in 2015. Burned out again, but now with health problems. I started investigating what I could do better for recovering my health, including diet, exercise and supplements. Eventually learned what to do to stay productive. I also took more time for rest and relaxation with playstation gaming, renewing my interest in game design. Especially systemic story games.

A new generation for the booking system started development in 2015, now based on Javascript, NodeJS and the graph database Neo4J. I made the jump from Perl to Javascript and took the opportunity to jump to the bleeding edge, with ES modules, web components and low complexity. That which later would be called Modern Web.

I see so many problems in apps from Facebook, Youtube and others. Especially for making systems that support you switching devices or working with other people, with all information kept up to date regardless of their source. I did a lot of work to make it easy to build large interconnected semantic systems with guaranteed consistency in all async scenarios. More on that later.

By 2020, the project was halted. My workplace decided to simplify and outsource all IT. The world-leading system I had developed was not the right match for the company.

Now, I’m looking for my dream job, advancing the state of the web using bleeding edge web technologies like buildless web components and reactive state semantic graphs, connected to the latest of LLM AI using embeddings for ML vector databases.