DEV Community: Thomas Broyer

How do HTML event handlers work?

Thomas Broyer — Sun, 10 Nov 2024 22:54:02 +0000

HTML event handlers are those onxxx attributes and properties many of us are used to, but do you know how they actually work? If you're writing custom elements and would like them to have such event handlers, what would you have to do? And what would you possibly be unable to implement? What differences would there be from native event handlers?

Before diving in: if you just want something usable, I wrote a library that implements all this (and more) but first jump to the conclusion for the limitations; otherwise, read on.

High-level overview

Before all, an event handler is a property on an object whose name starts with on (followed by the event type) and whose value is a JS function (or null). When that object is an element, the element also has a similarly-named attribute whose value will be parsed as JavaScript, with a variable named event whose value will be the current event being handled, and that can return false to cancel the event (how many times have we seen those infamous oncontextmenu="return false" to disable right click?)

Setting an event handler is equivalent to adding a listener (removing the previous one if any) for the corresponding event type.

Quite simple, right? but the devil lies in the details!

(fwiw, there are two special kinds of event handlers, onerror and onbeforeunload, that I won't talk about the here.)

In details

Let's go through those details the devil hides in (in no particular order).

Globality

All built-in event handlers on elements are global, and available on every element (actually, every HTMLElement; that excludes SVG and MathML elements). This include custom elements so you won't need to implement, e.g., an onclick yourself, it's built into every element. This also implies that as new event handlers are added to HTML in the future, they might conflict with your own event handlers for a custom event (this is also true of properties and methods that could later be added to the Node, Element and HTMLElement interfaces though).

Custom elements already have all "native" event handlers built-in.

Conversely, this globality isn't something you'll be able to implement for a custom event: you can create an onfoo event handler on your custom element, but you won't be able to put an onfoo on a <div> element and expect it to do anything useful.
(Technically, you possibly could monkey-patch the HTMLElement.prototype and use a MutationObserver to detect the attribute, but you'll still miss attributes on detached elements and, well, monkey-patching… do I need to say more?)

To avoid forward-incompatibility (be future-proof) you might want to name your event handler with a dash or other non-ASCII character in its attribute name, and maybe an uppercase character in its property name. When custom attributes are a thing, then maybe this will also allow having such an attribute globally available on all elements. Not sure it's a good idea, if you ask me I think I'd just use a simple name and hope HTML won't add a conflicting one in the future.

Return value

We briefly talked about the return value of the event handler function above: if it returns false then the event will be cancelled.

It happens that we're talking about the exact false value here, not just any falsy value.

Fwiw, by cancelled here, we mean just as if the event handler function had called event.preventDefault().

Listener ordering

When you set an event handler, it adds an event listener for the corresponding event, so if you set it in between two element.addEventListener(), it'll be called in between the event listeners.

Now if you set it to another value later on, it won't actually remove the listener for the previous value and add one for the new value; it will actually reuse the existing listener!
This was likely some kind of optimization in browser engines in the past (from the time of Internet Explorer or even Netscape I suppose), but as websites relied on it it's now part of the spec.

const events = [];
element.addEventListener("click", () => events.push("click 1"));
element.onclick = () => "replaced below"; // starts listening
element.addEventListener("click", () => events.push("click 3"));
element.onclick = () => events.push("click 2"); // doesn't reorder the listeners
element.click();
console.log(events);
// → ["click 1", "click 2", "click 3"]

If you remove an event handler (set the property to null –wait, there's more about it, see below– or remove the attribute), the listener will be removed though.
So if for any reason you want to make sure an event handler is added to the end of the listeners list, then first remove any previous value then set your own.

Non-function property values

We talked about setting an event handler and removing an event handler already, but even there there are small details to account for.

When you set an event handler's property, any object value (which include functions) will set the event handler (and possibly add an event listener). When an event is dispatched, only function values will have any useful effect, but any object can be used to activate the corresponding event listener (and possibly later be replaced with a function value without reordering the listeners).

Conversely, any non-object, non-function value will be coerced to null and will remove the event handler.

This means that element.onclick = new Number(42) sets the event handler (to some useless value, but still starts listening to the event), and element.onclick = 42 removes it (and element.onclick then returns null).

Invalid attribute values, lazy evaluation

Attribute values are never null, so they always set an event handler (to remove it, remove the attribute). They're also evaluated lazily: invalid values (that can't be parsed as JavaScript) will be stored internally until they're needed (either the property is read, or an event is dispatched that should execute the event handler), at which point they'll be tentatively evaluated.

When the value cannot be parsed as JavaScript, an error is reported (to window.onerror among others) and the event handler is replaced with null but won't remove the event handler!
(so yes, you can have an event handler property returning null while having it listen to the event, and not have the listener be reordered when set to another value)

const events = [];
element.addEventListener("click", () => events.push("click 1"));
element.setAttribute("onclick", "}"); // invalid, but starts listening
console.log(element.onclick); // reports an error and logs null, but doesn't stop listening
element.addEventListener("click", () => events.push("click 3"));
element.onclick = () => events.push("click 2"); // doesn't reorder the listeners
element.click();
console.log(events);
// → ["click 1", "click 2", "click 3"]

The error reports the original location of the value, that is the setAttribute() call in a script, or even the attribute in the HTML, even though the value is actually evaluated much later.
This is something that I don't think could be implemented in userland.

Scope

We've said above that an event variable is available in the script set as an attribute value, but that's not the only variable in scope: every property of the current element is directly readable as a variable as well. Also in scope are properties of the associated form element if the element is form-associated, and properties of the document.

This means that <a onclick="alert(href)" will show the link's target URL, <button onclick="alert(action)"> will show the form's target URL (as a side effect, you can also refer to other form elements by name), and <span onclick="alert(location)"> will show the document's URL.

This is more or less equivalent to evaluating the attribute value inside this:

with (document) {
  with (element.form) {
    with (element) {
      // evaluate attribute value here
    }
  }
}

Related to scope too is the script's base URL that would be used when import()ing modules with a relative URL.
Browsers seem to behave differently already on that: Firefox resolves the path relative to the document URL, whereas Chrome and Safari fail to resolve the path to a URL (as if there was no base URL at all). I don't think anything can be done here in a userland implementation.

Function source text

When the event handler has been set through an attribute, the function returned by the event handler property has a very specific source text (which is exposed by its .toString()), which is close to, but not exactly the same as what new Function("event", attrValue) would do (declaring a function with an event argument and the attribute's value as its body).

You couldn't directly use new Function("event", attrValue) anyway due to the scope you need to setup, but there's a trick to control the exact source text of a function so this isn't insurmoutable:

const handlerName = "onclick"
const attrValue = "return false;"
const fn = new Function(`return function ${handlerName}(event) {\n${attrValue}\n}`)()
console.log(fn.toString())
// → "function onclick(event) {\nreturn false;\n}"

Content Security Policy

Last, but not least, event handler attribute values are rejected early by a Content Security Policy (CSP): the violation will be reported as soon as the attribute is tentatively set, and this won't have any effect on the state of the event handler (that could have been set through the property).

The CSP directive that controls event handler attributes is script-src-attr (which falls back to script-src if not set, or to default-src). When implementing an event handler for a custom event in a custom element, the attribute value will have to be evaluated by scripting though (through new Function() most likely) so it will be controlled by script-src that will have to include either an appropriate hash source, or 'unsafe-eval' (notice the difference from native event handlers that would use 'unsafe-inline', not 'unsafe-eval'). Hash sources will be a problem though, because you'll have to evaluate not just the attribute's value, but a script that embeds the attribute's value (to set up the scope and source text). And you'd have to actually evaluate both to make sure the attribute value doesn't mess with your evaluated script (think SQL injection but on JavaScript syntax). This would mean that each event handler attribute would have to have two hash sources allowed in the script-src CSP directive, one of them being dependent on the custom element's implementation of the event handler.

An alternative would be to use a native event handler for parsing, but then the function would have that native event handler as its function name, and you'd have to make sure to use an element associated with the same form (if not using the custom element directly because e.g. you don't want to trigger mutation observers) to get the appropriate variables in scope.

Recap: What does it mean for custom event handlers?

As seen above, it's not possible to fully implement event handlers for a custom event in a way that would make it indistinguishable from native event handlers:

they won't be globally available on every element (except maybe in the future with custom attributes)
a Content Security Policy won't be able to use script-src-attr on those custom event handlers, and if it uses hash sources, chances are that 2 hash sources will be need for each attribute value (one of them being dependent on the custom event handler implementation details)
errors emitted by the scripts used as event handler attribute values won't point to the source of the attribute value
an import() with a relative URL, inside an event handler attribute value, won't behave the same as in a native event handler

The first point alone (or the first two) might make one reevaluate the need for adding such event handlers at all.
And if you're thinking about only implementing the property, think about what it brings compared to just having users call addEventListener().

That being said, I did the work (more as an exercise than anything else), so feel free to go ahead a implement event handlers for your custom elements.

Making Web Component properties behave closer to the platform

Thomas Broyer — Sun, 21 Jan 2024 23:16:59 +0000

Built-in HTML elements' properties all share similar behaviors, that don't come for free when you write your own custom elements. Let's see what those behaviors are, why you'd want to implement them in your web components, and how to do it, including how some web component libraries actually don't allow you to mimic those behaviors.

Built-in elements' behaviors

I said it already: built-in elements' properties all share similar behaviors, but there are actually several different such shared behaviors. First, there are properties (known as IDL attributes in the HTML specification) that reflect attributes (also known as content attributes); then there are other properties that are unrelated to attributes. One thing you won't find in built-in elements are properties whose value will change if an attribute change, but that won't update the attribute value when they are changed themselves (in case you immediately thought of value or checked as counter-examples, the situation is actually a bit more complex: those attributes are reflected by the defaultValue and defaultChecked properties respectively, and the value and checked properties are based on an internal state and behave differently depending on whether the user already interacted with the element or not).

Type coercion

But I'll start with another aspect that is shared by all of them, whether reflected or not: typing. DOM interfaces are defined using WebIDL, that has types and extended annotations, and defines mapping of those to JavaScript. Types in JavaScript are rather limited: null, undefined, booleans, IEEE-754 floating-point numbers, big integers, strings, symbols, and objects (including errors, functions, promises, arrays, and typed arrays). WebIDL on the other hand defines, among others, 13 different numeric types (9 integer types and 4 floating point ones) that can be further annotated to change their overflowing behavior, and several string types (including enumerations).

The way those types are experienced by developers is that getting the property will always return a value of the defined type (that's easy, the element owns the value), and setting it (if not read-only) will coerce the assigned value to the defined type. So if you want your custom element to feel like a built-in one, you'll have to define a setter to coerce the value to some specific type. The underlying question is what should happen if someone assigns a value of an unexpected type or outside the expected value space?

Convert and validate the new value in a property custom setter.

You probably don't want to use the exact WebIDL coercion rules though, but similar, approximated, rules that will behave the same most of the time and only diverge on some edge cases. The reason is that WebIDL is really weird: for instance, by default, numeric values overflow by wrapping around, so assigning 130 to a byte (whose value space ranges from -128 to 127) will coerce it to… -126! (128 wraps to -128, 129 to -127, and 130 to -126; and by the way 256 wraps to 0; for the curious, BigInt.asIntN and BigInt.asUintN will do such wrapping in JS, but you'll have to convert numbers to BigInt and back); non-integer values assigned to integer types are truncated by default, except when the type is annotated with [Clamp], in which case they're rounded, with half-way values rounded towards even values (something that only happens natively in JS when setting such non-integer values to typed arrays: Math.round(2.5) is 3, but Int8Array.of(2.5)[0] is 2).

Overall, I feel like, as far as primitive/simple types are concerned, boolean, integers, double (not float), string (WebIDL's DOMString), and enumerations are all that's needed; truncating (or rounding, but with JavaScript rules), and clamping or enforcing ranges for integers. In other words, wrapping integers around is just weird, and what matters is coercing to the appropriate type and value space. Regarding enumerations, they're probably best handled by the reflection rules though (see below), and treated only as strings: no single built-in element has a property of a type that's a WebIDL enum.

Reflected properties

Now let's get back to reflected properties: most properties of built-in elements reflect attributes or similarly (but with specific rules) correspond to an attribute and change its value when set; non-reflected properties are those that either expose some internal state (e.g. the current value or validation state of a form field), computed value (from the DOM, such as the selectedIndex of a select, or the cellIndex of a table cell) or direct access to DOM elements (elements of a form, rows of a table, etc.), or that access other reflected properties with a transformed value (such as the valueAsDate and valueAsNumber of input). So if you want your custom element to feel like a built-in one, you'll want to use similar reflection wherever appropriate.

Have your properties reflect attributes by default.

The way reflection is defined is that the source of truth is the attribute value: getting the property will actually parse the attribute value, and setting the property will stringify the value into the attribute. Note that this means possibly setting the attribute to an invalid value that will be corrected by the getter. An example of this is setting the type property of an input element to an unknown value: it will be reflected in the attribute as-is, but the getter will correct it text. Another example where this is required behavior is with dependent attributes like those of progress or meter elements: without this you'd have to be very careful setting properties in the right order to avoid invalid combinations and having your set value immediately rewritten, but this behavior makes it possible to update properties in any order as the interaction between them are resolved internally and exposed by the getters: you can for example set the value to a value upper than max (on getting, value would be normalized to its default value) and then update the max (on getting, value could now return the value you previously set, because it wasn't actually rewritten on setting). Actually, these are not technically reflected then as they have specific rules, but at least they're consistent with actual reflected properties; for the purpose of this article, I'll consider them as reflected properties though.

This is at least how it theoretically works; in practice, the parsed value can be cached to avoid parsing every time the property is read; but note that there can be several properties reflecting the same attribute (the most known one probably being className and classList both reflecting the class attribute). Reflected properties can also have additional options, depending on their type, that will change the behavior of the getter and setter, not unlike WebIDL extended attributes.

Also note that HTML only defines reflection for a limited set of types (if looking only at primitive/simple types, only non-nullable and nullable strings and enumerations, long, unsigned long, and double are covered, and none of the narrower integer types, big integers, or the unrestricted double that allows NaN and infinity).

You can see how Mozilla tests the compliance of their built-in elements
in the Gecko repository (the ok and is assertions are defined in their SimpleTest testing framework). And here's the Web Platform Tests' reflection harness, with data for each built-in element in sibling files, that almost every browser pass.

Events

Most direct changes to properties and attributes don't fire events: user actions or method calls will both update a property and fire an event, but changing a property programmatically generally won't fire any event. There are a few exceptions though: the events of type ToggleEvent fired by changes to the popover attribute or the open attribute of details elements, or the select event when changing the selectionStart, selectionEnd or selectionDirection properties of input and textarea elements (if you know of others, let me know); but notably changing the value of a form element programmatically won't fire a change or input event. So if you want your custom element to feel like a built-in one, don't fire events from your property setters or other attribute changed callbacks, but fire an event when (just after) you programmatically change them.

Don't fire events from your property setters or other attribute changed callbacks.

Why you'd want to implement those

If you're you (your team, your company) are the only users of the web components (e.g. building an application out of web components, or an internal library of reusable components), then OK, don't use reflection if you don't need it, you'll be the only user anyway so nobody will complain. If you're publicly sharing those components, then my opinion is that, following the principle of least astonishment, you should aim at behaving more like built-in elements, and reflect attributes.

Similarly, for type coercions, if you're the only users of the web components, it's ok to only rely on TypeScript (or Flow or whichever type-checker) to make sure you always pass values of the appropriate type to your properties (and methods), but if you share them publicly then you should in my opinion coerce or validate inputs, in which case you'd want to follow the principe of least astonishment as well, and thus use rules similar to WebIDL and reflection behaviors. This is particularly true for a library that can be used without specific tooling, which is generally the case for custom elements.

For example, all the following design systems can be used without tooling (some of them provide ready-to-use bundles, others can be used through import maps): Google's Material Web, Microsoft's Fluent UI, IBM's Carbon, Adobe's Spectrum, Nordhealth's Nord, Shoelace, etc.

How to implement them

Now that we've seen what we'd want to implement, and why we'd want to implement it, let's see how to do it. First without, and then with libraries.

I started collecting implementations that strictly follow (as an exercise, not as a goal) the above rules in a GitHub repository (strictly because it directly reuses the above-mentioned Gecko and Web Platform Tests harnesses).

Vanilla implementation

In a vanilla custom element, things are rather straightforward:

class MyElement extends HTMLElement {
  get reflected() {
    const strVal = this.getAttribute("reflected");
    return parseValue(strVal);
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.setAttribute("reflected", stringifyValue(newValue));
  }
}

or with intermediate caching (note that the setter is identical, setting the attribute will trigger the attributeChangedCallack which will close the loop):

class MyElement extends HTMLElement {
  #reflected;

  get reflected() {
    return this.#reflected;
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.setAttribute("reflected", stringifyValue(newValue));
  }

  static get observedAttributes() {
    return [ "reflected" ];
  }
  attributeChangedCallback(name, oldValue, newValue) {
    // Note: in this case, we know it can only be the attribute named "reflected"
    this.#reflected = parseValue(newValue);
  }
}

And for a non-reflected property (here, a read-write property representing an internal state):

class MyElement extends HTMLElement {
  #nonReflected;
  get nonReflected() {
    return this.#nonReflected;
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.#nonReflected = newValue;
  }
}

Because many rules are common to many attributes (the coerceType operation is defined by WebIDL, or using similar rules, and the HTML specification defines a handful of microsyntaxes for the parseValue and stringifyValue operations), those could be packaged up in a helper library. And with decorators coming to ECMAScript (and already available in TypeScript), those could be greatly simplified:

class MyElement extends HTMLElement {
  @reflectInt accessor reflected;
  @int accessor nonReflected;
}

I actually built such a library, mostly as an exercise (and I already learned a lot, most of the above details actually). It's currently not published on NPM but you can find it on Github.

With a library

Surprisingly, web component libraries don't really help us here.

First, like many libraries nowadays, most expect people to just pass values of the appropriate types (relying on type checking through TypeScript) and basically leave you handling everything including how to behave in the presence of unexpected values. While it's OK, as we've seen above, in a range of situations, there are limits to this approach and it's unfortunate that they don't provide tools to make it easier at least coercing types.

Regarding reflected properties, most libraries tend to discourage you from doing it, while (fortunately!) supporting it, if only minimally.

All libraries (that I've looked at) support observed attributes though (changing the attribute value updates the property, but not the other way around), and most default to this behavior.

Now let's dive into the how-to with Lit, FAST, and then Stencil (other libraries left as a so-called exercise for the reader).

With Lit

By default, Lit reactive properties (annotated with @property()) observe the attribute of the same (or configured) name, using a converter to parse the value if needed (by default only handling numbers through a plain JavaScript number coercion, booleans, strings, or possibly objects or arrays through JSON.parse(); but a custom converter can be given). If your property is not associated to any attribute (but needs to be reactive to trigger a render when changed), then you can annotate it with @property({ attribute: false }) or @state() (the latter is meant for internal state though, i.e. private properties).

To make a reactive property reflect an attribute, you'll add reflect: true to the @property() options, and Lit will use the converter to stringify the value too. This won't be done immediately though, but only as part of Lit's reactive update cycle. This timing is a slight deviation compared to built-in elements that's probably acceptable, but it makes it harder to implement some reflection rules (those that set the attribute to a different value than the one returned by the getter) as the converter will always be called with the property value (returned by the getter, so after normalization). For a component similar to progress or meter with dependent properties, Lit recommends correcting the values in a willUpdate callback (this is where you'd check whether the value is valid with respect to the max for instance, and possibly overwrite its value to bring it in-range); this means that attributes will have the corrected value, and this requires users to update all properties in the same event loop (which will most likely be the case anyway).

It should be noted that, surprisingly, Lit actively discourages reflecting attributes:

Attributes should generally be considered input to the element from its owner, rather than under control of the element itself, so reflecting properties to attributes should be done sparingly. It's necessary today for cases like styling and accessibility, but this is likely to change as the platform adds features like the :state pseudo selector and the Accessibility Object Model, which fill these gaps.

No need to say I disagree.

For type coercion and validation, Lit allows you to have your own accessors (and version 3 makes it even easier), so everything's ok here, particularly for non-reflected properties:

class MyElement extends LitElement {
  #nonReflected;
  get nonReflected() {
    return this.#nonReflected;
  }
  @state()
  set nonReflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.#nonReflected = newValue;
  }
}

For those cases where you'd want the attribute to possibly have an invalid value (to be corrected by the property getter), it would mean using a non-reactive property wrapping a private reactive property (this assumes Lit won't flag them as errors in future versions), and parsing the value in its getter:

class MyElement extends LitElement {
  @property({ attribute: "reflected", reflect: true })
  accessor #reflected = "";

  get reflected() {
    return parseValue(this.#reflected);
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.#reflected = stringifyValue(newValue);
  }
}

or with intermediate caching (note that the setter is identical):

class MyElement extends LitElement {
  @property({ attribute: "reflected", reflect: true })
  accessor #reflected = "";

  #parsedReflected = "";
  get reflected() {
    return this.#parsedReflected;
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.#reflected = stringifyValue(newValue);
  }

  willUpdate(changedProperties) {
    if (changedProperties.has("#reflected")) {
      this.#parsedReflected = parseValue(this.#reflected);
    }
  }
}

It might actually be easier to directly set the attribute from the setter (and as a bonus behaving closer to built-in elements) and only rely on an observed property from Lit's point of view (setting the attribute will trigger attributeChangedCallback and thus Lit's observation code that will use the converter and then set the property):

class MyElement extends LitElement {
  @property({
    attribute: "reflected",
    converter: (value) => parseValue(value),
  })
  accessor #reflected = "";

  get reflected() {
    return this.#reflected;
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.setAttribute("reflected", stringifyValue(newValue));
  }
}

Note that this is actually very similar to the approach in the vanilla implementation above but using Lit's own lifecycle hooks. It should also be noted that for a USVString that contains a URL (where the attribute value is resolved to a URL relative to the document base URI) the value needs to be processed in the getter (as it depends on an external state –the document base URI– that could change independently from the element).

A previous version of this article contained a different implementation that happened to be broken.

class MyElement extends LitElement {
  #reflected = "";
  get reflected() {
    return this.#reflected;
  }
  @property()
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    const stringValue = stringifyValue(newValue);
    // XXX: there might be a more optimized way
    // than stringifying and then parsing
    this.#reflected = parseValue(stringValue);
    // Avoid unnecessarily triggering attributeChangedCallback
    // that would reenter that setter.
    if (this.getAttribute("reflected") !== stringValue) {
      this.setAttribute("reflected", stringValue);
    }
  }
}

This implementation would for instance have the setter called with null when the attribute is removed, which actually needs to behave differently than user code calling the setter with null: in the former case the property should revert to its default value, in the latter case that null would be coerced to the string "null" or the numeric value 0 and the attribute would be added back with that value.

If we're OK only reflecting valid values to attributes, then we can fully use converters but things aren't necessarily simpler (we still need the custom setter for type coercion and validation, and marking the internal property as reactive to avoid triggering the custom setter when the attribute changes; we don't directly deal with the attribute but we now have to normalize the value in the setter in the same way as stringifying it to the attribute and parsing it back, to have the getter return the appropriate value):

const customConverter = {
  fromAttribute(value) {
    return parseValue(value);
  },
  toAttribute(value) {
    return stringifyValue(value);
  },
};

class MyElement extends LitElement {
  @property({ reflect: true, converter: customConverter })
  accessor #reflected = "";
  get reflected() {
    return this.#reflected;
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    // XXX: this should use a more optimized conversion/validation
    this.#reflected = parseValue(stringifyValue(newValue));
  }
}

With FAST

I know FAST is not used that much but I wanted to cover it as it seems to be the only library that reflects attributes by default. By default it won't do any type coercion unless you use the mode: "boolean", which works almost like an HTML boolean attribute, except an attribute present but with the value "false" will coerce to a property value of false!

Otherwise, it works more or less like Lit, with one big difference: the converter's fromView is also called when setting the property (this means that fromView receives any external value, not just string values from the attribute). But unfortunately this doesn't really help us as most coercion rules need to throw at one point and we want to do it only in the property setters, never when parsing attribute values; and those rules that don't throw will have possibly different values between the attribute and the property getter (push invalid value to the attribute, sanitize it on the property getter), or just behave differently between the property (e.g. turning a null into 0 or "null") and the attribute (where null means the attribute is not set, and the property should then have its default value which could be different from 0, and will likely be different from "null").

This means that in the end the solutions are almost identical to the Lit ones (here using TypeScript's legacy decorators though; and applying the annotation on the private property to avoid triggering the custom setter on attribute change):

class MyElement extends FASTElement {
  @attr({ attribute: "reflected" })
  private _reflected = "";

  get reflected() {
    return parseValue(this._reflected);
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this._reflected = stringifyValue(newValue);
  }
}

or with intermediate caching (note that the setter is identical):

class MyElement extends FASTElement {
  @attr({ attribute: "reflected" })
  private _reflected = "";

  private _reflectedChanged(oldValue, newValue) {
    this._parsedReflected = parseValue(newValue);
  }

  private _parsedReflected;
  get reflected() {
    return this._parsedReflected;
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.__reflected = stringifyValue(newValue);
  }
}

Or if you want immediate reflection to the attribute (the internal property can now be used to store the parsed value):

class MyElement extends FASTElement {
  @attr({
    attribute: "reflected",
    mode: "fromView",
    converter: {
      fromView(value) {
        return parseValue(value);
      },
      toView(value) {
        // mandatory in the converter type
        throw new Error("should never be called");
      }
    }
  })
  private _reflected;

  get reflected() {
    return this._reflected ?? "";
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this.setAttribute("reflected", stringifyValue(newValue));
  }
}

Note that the internal property is not initialized, to avoid calling the converter's fromView, and handled in the getter instead (our fromView expects a string or null coming from the attribute, so we'd have to initialize the property with such a string value which would hurt readability of the code as that could be a value different from the one actually stored in the property and returned by the pblic property getter).

If we're OK only reflecting valid values to attributes, then we can fully use converters but things aren't necessarily simpler (we still need the custom setter for type coercion and validation, and marking the internal property as reactive to avoid triggering the custom setter when the attribute changes; we don't directly deal with the attribute but we still need to call stringifyValue as we know the converter's fromView will receive the new value):

const customConverter = {
  fromView(value) {
    return parseValue(value);
  },
  toView(value) {
    return stringifyValue(value);
  },
};

class MyElement extends FASTElement {
  @attr({ attribute: "reflected ", converter: customConverter })
  private _reflected;

  get reflected() {
    return this._reflected ?? "";
  }
  set reflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this._reflected = stringifyValue(newValue);
  }
}

For non-reflected properties, you'd want to use @observable instead of @attr, except that it doesn't work on custom accessors, so you'd have to do it manually:

class MyElement extends FASTElement {
  private _nonReflected = "";
  get nonReflected() {
    Observable.track(this, 'nonReflected');
    return this._nonReflected;
  }
  set nonReflected(value) {
    const newValue = coerceType(value);
    // …there might be additional validations here…
    this._nonReflected = newValue;
    Observable.notify(this, 'nonReflected');
  }
}

With Stencil

First a disclosure: I never actually used Stencil, only played with it a bit locally in a hello-world project while writing this post.

Stencil is kind of special. It supports observable attributes through the @Prop() decorator, and reflected ones through @Prop({ reflect: true }). It will however reflect default values to attributes when the component initializes, doesn't support custom converters, and like FAST will convert an attribute value of "false" to a boolean false. You also have to add mutable: true to the @Prop() if the component modifies its value (Stencil assumes properties and attributes are inputs to the component, not state of the component).

A @Prop() must be public too, and cannot have custom accessors. You can use a @Watch() method to do some validation, but throwing from there won't prevent the property value from being updated; you can revert the property to the old value from the watch method, but other watch methods for the same property will then be called twice, and not necessarily in the correct order (depending on declaration order).

You cannot expose properties on the element's API if they are not annotated with @Prop(), making them at a minimum observe an attribute.

In other words, a Stencil component cannot, by design, feel like a built-in custom element (another thing specific to Stencil: besides @Prop() properties, you can expose methods through @Method but they must be async).

Improving a web component, one step at a time

Thomas Broyer — Sat, 16 Dec 2023 19:12:52 +0000

Earlier this month, Stefan Judis published a small web component that makes your text sparkle.

In the spirit of so-called HTML web components which apparently often comes with some sort of aversion for the shadow DOM, the element directly manipulates the light DOM. As a developer of web apps with heavy DOM manipulations, and lover of the platform, this feels weird to me as it could possibly break so many things: other code that manipulates the DOM and now sees new elements and could also change them, handling of disconnection and reconnection of the element (as most such elements modify their children in the connectedCallback without checking whether it had already been done), MutationObserver, etc.

The first thing that came to my mind was that shadow DOM, for all its drawbacks and bugs, was the perfect fit for such an element, and I wanted to update Stefan's element to use the shadow DOM instead. Then a couple days ago, Zach Leatherman published a similar element that makes it snow on its content, and I was pleased to see he used shadow DOM to encapsulate (hide) the snowflakes. That was the trigger for me to actually take the time to revisit Stefan's <sparkle-text> element, so here's a step by step of various improvements (in my opinion) I made.

Disclaimer before I begin: this not in any way a criticism of Stefan's work! On the contrary actually, it wouldn't have been possible without this prior work. I just want to show things that I think could be improved, and this is all very much subjective.

I'll link to commits in my fork without any (intermediate) demo, as all those changes don't have much impact on the element's behavior, as seen by a reader of the web page (if you're interested in what it changes when looked at through the DevTools, then clone the repository, run npm install, npm run start, then checkout each commit in turn), except in some specific situations. The final state is available here if you want to play with it in your DevTools.

Using shadow DOM

The first step was moving the sparkles to shadow DOM, to avoid touching the light DOM. This involves of course attaching shadow DOM, with a <slot> to let the light DOM show, and then changing where the sparkles are added, but also changing how CSS is handled!

Abridged diff of the changes (notably excluding CSS)

@@ -66,16 +62,21 @@ class SparklyText extends HTMLElement {
 `;
     let sheet = new CSSStyleSheet();
     sheet.replaceSync(css);
-    document.adoptedStyleSheets = [...document.adoptedStyleSheets, sheet];
-    _needsStyles = false;
+    this.shadowRoot.adoptedStyleSheets = [sheet];
   }

   connectedCallback() {
+    if (this.shadowRoot) {
+      return;
+    }
+
     this.#numberOfSparkles = parseInt(
       this.getAttribute("number-of-sparkles") || `${this.#numberOfSparkles}`,
       10
     );

+    this.attachShadow({ mode: "open" });
+    this.shadowRoot.append(document.createElement("slot"));
     this.generateCss();
     this.addSparkles();
   }
@@ -99,7 +100,7 @@ class SparklyText extends HTMLElement {
       Math.random() * 110 - 5
     }% - var(--_sparkle-base-size) / 2)`;

-    this.appendChild(sparkleWrapper);
+    this.shadowRoot.appendChild(sparkleWrapper);
     sparkleWrapper.addEventListener("animationend", () =&gt; {
       sparkleWrapper.remove();
     });

In Stefan's version, CSS is injected to the document, with a boolean to make sure it's done only once, and styles are scoped to .sparkle-wrapper descendants of the sparkle-text elements. With shadow DOM, we gain style encapsulation, so no need for that scoping, we can directly target .sparkle-wrapper and svg as they're in the shadow DOM, clearly separate from the HTML that had been authored. We need to do it for each element though (we'll improve that later), but we now need to make sure we initialize the shadow DOM only once instead (I'm going step by step, so leaving this in the connectedCallback).

As a side effect, this also fixes some edge-case bug where the CSS would apply styles to any descendant SVG of the element, whether a sparkle or not (this could have been fixed by only targetting SVG inside .sparkle-wrapper actually); and of course with shadow DOM encapsulation, page author styles won't affect the sparkles either.

Small performance improvements

Those are really small, and probably negligible, but I feel like they're good practice anyway so I didn't even bother measuring actually.

First, as said above, the CSS needs to be somehow injected into each element's shadow DOM, but the constructible stylesheet can actually be shared between all of them. I've thus split construction of the stylesheet with its adoption in the shadow DOM, and made sure construction was only made once. Again, to limit the changes, everything's still in the same method, just move inside an if (I think I would have personally constructed the stylesheet early, as soon as the script is loaded, rather than waiting for the element to actually be used; it probably doesn't make a huge difference).

   generateCss() {
-    const css = `…`;
-    let sheet = new CSSStyleSheet();
-    sheet.replaceSync(css);
+    if (!sheet) {
+      const css = `…`;
+      sheet = new CSSStyleSheet();
+      sheet.replaceSync(css);
+    }
     this.shadowRoot.adoptedStyleSheets = [sheet];
   }

Similarly, sparkles were created by innerHTML the SVG into each. I changed that to using cloneNode(true) on an element prepared only once.

   addSparkle() {
-    const sparkleWrapper = document.createElement("span");
-    sparkleWrapper.classList.add("sparkle-wrapper");
-    sparkleWrapper.innerHTML = this.#sparkleSvg;
+    if (!sparkleTemplate) {
+      sparkleTemplate = document.createElement("span");
+      sparkleTemplate.classList.add("sparkle-wrapper");
+      sparkleTemplate.innerHTML = this.#sparkleSvg;
+    }
+
+    const sparkleWrapper = sparkleTemplate.cloneNode(true);

We actually don't even need the wrapper element, we could directly use the SVG without wrapper.

Handling disconnection

The element uses chained timers (a setTimeout callback that itself ends up calling setTimeout with the same callback, again and again) to re-add sparkles at random intervals (removing the sparkles is done as soon as the animation ends; and all of this is done only if the user didn't configure their browser to prefer reduced motion).

If the element is removed from the DOM, this unnecessarily continues in the background and could create memory leaks (in addition to just doing unnecessary work). I started with a very small change: check whether the element is still connected to the DOM before calling adding the sparkle (and calling setTimeout again). It could have been better (for some definition of better) to track the timer IDs so we could call clearTimeout in disconnectedCallback, but I feel like that would be unnecessarily complex.

       const {matches:motionOK} = window.matchMedia('(prefers-reduced-motion: no-preference)');
-      if (motionOK) this.addSparkle();
+      if (motionOK && this.isConnected) this.addSparkle();

This handles disconnection (as could be done by any destructive change to the DOM, like navigating with Turbo or htmx, I'm not even talking about using the element in a JavaScript-heavy web app) but not reconnection though, and we've exited early from the connectedCallback to avoid initializing the element twice, so this change actually broke our component in these situations where it's moved around, or stashed and then reinserted. To fix that, we need to always call addSparkles in connectedCallback, so move all the rest into an if, that's actually as simple as that… except that when the user prefers reduced motion, sparkles are never removed, so they keep piling in each time the element is connected again. One way to handle that, without introducing our housekeeping of individual timers, is to just remove all sparkles on disconnection. Either that or conditionally add them in connectedCallback if either we're initializing the element (including attaching the shadow DOM) or the user doesn't prefer reduced motion. The difference between both approaches is in whether we want the small animation when the sparkles appear (and appearing at new random locations). I went with the latter.

This still doesn't handle the situation where prefers-reduced-motion changes while the element is displayed though: if it turns to no-preference, then sparkles will start animating (due to CSS) then disappear at the end of their animation (due to JS listening to the animationend event), and no other sparkle will be added (because the setTimeout chain would have been broken earlier). I don't feel like it's worthy enough of a fix for such an element but it's also rather easy to handle so let's do it: listen to the media query change and start the timers whenever the user no longer prefers reduced motion.

@@ -94,6 +94,19 @@ connectedCallback() {
       );
       this.addSparkles();
     }
+
+    motionOK.addEventListener("change", this.motionOkChange);
+  }
+
+  disconnectedCallback() {
+    motionOK.removeEventListener("change", this.motionOkChange);
+  }
+
+  // Declare as an arrow function to get the appropriate 'this'
+  motionOkChange = () => {
+    if (motionOK.matches) {
+      this.addSparkles();
+    }
   }

Browser compatibility

Constructible stylesheets aren't supported in Safari 16.3 and earlier (and possibly other browsers). To avoid the code failing and strange things (probably, I haven't tested) happening, I started by bailing out early if the browser doesn't support constructible stylesheets (the element would then just do nothing; I could have actually even avoided registering it at all). Fwiw, I borrowed the check from Zach's <snow-fall> which works this way already (thanks Zach). As an aside, it's a bit strange that the code assumed construtible stylesheets were available, but tested for the availability of the custom element registry 🤷

   connectedCallback() {
-    if (this.shadowRoot) {
+    // https://caniuse.com/mdn-api_cssstylesheet_replacesync
+    if (this.shadowRoot || !("replaceSync" in CSSStyleSheet.prototype)) {
       return;
     }

But Safari 16.3 and earlier still represent more than a third of users on macOS, and more than a quarter of users on iOS! (according to CanIUse) To widen browser support, I therefore added a workaround, which consists of injecting a <style> element in the shadow DOM. Contrary to the constructible stylesheet, styles cannot be shared by all elements though, as we've seen above, so we only conditionally fallback to that approach, and continue using a constructible stylesheet everywhere it's supported.

-      sheet = new CSSStyleSheet();
-      sheet.replaceSync(css);
+      if (supportsConstructibleStylesheets) {
+        sheet = new CSSStyleSheet();
+        sheet.replaceSync(css);
+      } else {
+        sheet = document.createElement("style");
+        sheet.textContent = css;
+      }
     }

-    this.shadowRoot.adoptedStyleSheets = [sheet];```


+    if (supportsConstructibleStylesheets) {
+      this.shadowRoot.adoptedStyleSheets = [sheet];
+    } else {
+      this.shadowRoot.append(sheet.cloneNode(true));
+    }

Other possible improvements

I stopped there but there's still room for improvement.

For instance, the number-of-sparkles attribute is read once when the element is connected, so changing the attribute afterwards won't have any effect (but will have if you disconnect and then reconnect the element). To handle that situation (if only because you don't control the order of initialization when that element is used within a JavaScript-heavy application with frameworks like React, Vue or Angular), one would have to listen to the attribute change and update the number of sparkles dynamically. This could be done either by removing all sparkles and recreating the correct number of them (with addSparkles()), but this would be a bit abrupt, or by reworking entirely how sparkles are managed so they could adapt dynamically (don't recreate a sparkle, let it expire, when changing the number of sparkles down, or create just as many sparkles as necessary when changing it up). I feel like this would bump complexity by an order of magnitude, so it's probably not worth it for such an element.

The number of sparkles could also be controlled by a property reflecting the attribute; that would make the element more similar to built-in elements. Once the above is in place, this hopefully shouldn't be too hard.

That number of sparkles is expected to be, well, a number, and is currently parsed with parseInt, but the code doesn't handle parsing errors and could set the number of sparkles to NaN. Maybe we'd prefer using the default value in this case, and similarly for a zero or negative value; basically defining the attribute as a number limited to only positive numbers with fallback.

All this added complexity is, to me, what separates so-called HTML web components from others: they're designed to be used from HTML markup and not (or rarely) manipulated afterwards, so shortcuts can be taken to keep them simple.

Still speaking of that number of sparkles, the timers that create new sparkles are entirely disconnected from the animation that also makes them disappear. The animation length is actually configurable through the --sparkly-text-animation-length CSS custom property, but the timers delay is not configurable (a random value between 2 and 3 seconds). This means that if we set the animation length to a higher value than 3 seconds, there will actually be more sparkles than the configured number, as new sparkles will be added before the previous one has disappeared. There are several ways to fix this (if we think it's a bug –this is debatable!– and is worth fixing): for instance we could use the Web Animations API to read the computed timing of the animation and compute the timer's delay based on this value. Or we could let the animation repeat and move the element on animationiteration, rather than remove it and add another, and to add some randomness it could be temporarily paused and then restarted if we wanted (with a timer of some random delay). The code would be much different, but not necessarily more complex.

Regarding the animation events (whether animationend like it is now, or possibly animationiteration), given that they bubble, they could be listened to on a single parent (the element itself –filtering out possible animations on light DOM children– or an intermediate element inserted to contain all sparkles). This could hopefully simplify the code handling each sparkle.

Last, but not least, the addSparkles and addSparkle methods could be made private, as there's no reason to expose them in the element's API.

Final words

Had I started from scratch, I probably wouldn't have written the element the same way. I tried to keep the changes small, one step at a time, rather than doing a big refactoring, or starting from scratch and comparing the outcome to the original, as my goal was to specifically show what I think could be improved and how it wouldn't necessarily involve big changes. Going farther, and/or possibly using a helper library (I have written earlier about their added value), is left as an exercise for the reader.

Beyond the login page

Thomas Broyer — Wed, 29 Nov 2023 19:54:40 +0000

There are many blog posts floating around about “adding authentication to your application”, be it written in Node.js, ASP.NET, Java with Spring Boot, JS in the browser talking to a JSON-based Web API on the server, etc. Most of them handle the login page and password storage, and sometimes logout and a user registration page. But authentication is actually much more than that!

Don't get me wrong, it's great that we can describe in a single blog post how to do such things, but everyone should be aware that this is actually just the beginning of the journey, and most of the time those blog posts don't have any such warnings.

So here are some things to think about when “adding authentication to your application”:

are you sure you store passwords securely? and verify them securely?
is your logout secure? (ideally cannot be abused by tricking you just clicking a link on a mail or random site)
are passwords robust?
- put a lower bound on password length (NIST recommends a minimum of 8 characters); don't set an upper bound, or if you really want to make sure it's high enough (NIST recommends accepting at least 64 characters)
- if possible, check passwords (at registration or change) against known compromised passwords (use Pwned Passwords or similar)
how well do you handle non-ASCII characters? For example, macOS and Windows encode diacritics differently, so make sure that someone who signed up on one device will be able to sign in on another (put differently, use Unicode normalization on inputs; NIST recommends using NFKC or NFKD)
do you have a form to securely change the password? (when already authenticated)
are your forms actually compatible with password managers?
do you protect against brute-force attacks? if you do (e.g. by locking out accounts, or even just throttling), do you somehow protect legitimate users against DDoS?
once authenticated, how do you maintain the authenticated state (sessions; btw don't use JWTs)? and is this secure? (in other words, do you protect against session fixation? cross-site request forgery?)
how long are your sessions? There's a balance between short and long sessions regarding security and convenience, but a choice needs to be made.
do you have a mechanism to ask for re-authentication before sensitive actions?
what do you do if a user forgot their password? Password recovery generally requires an email address, do you have one? how can you make sure that the user didn't mistype it and you will actually be able to use it when they need it? Put differently: you need a secure email verification process before you can have a secure password reset process. Implementing those processes securely go beyond the scope of this post, but let's just say we've just come from one single blog post explaining how to “add authentication to your application” to a series of blog posts.
by the way, now that you store an email address for password reset purpose, how can the user securely update it? and by that I also mean, how do you handle the case where the account got breached and the attacker changes the email address? There's unfortunately no simple answer to that, because there are a handful of cases to handle: the user may have lost access to the previous email, an attacker may have gained access to the previous email, the user may still have access to the previous email but have mistyped the new email, etc.
speaking of changing passwords, do you make it easier for password managers? (spoiler: through a /.well-known/change-password URL)
do you handle multi-factor authentication? do you plan on handling it in the future? If you use SMS to send one-time codes, can the device autofill the form?
how about passkeys?

That being said, I don't think I ever implemented all of the above perfectly. There are always tradeoffs. But these are things to think about and make choices, and sometimes deliberate choices to postpone things (or just not implement them, after pondering the risks). Unfortunately, I did however see big mistakes in implementations of the various processes hinted above.

Most of the time nowadays, I prefer offloading this to an identity provider, using OpenID Connect or soon Federated Credential Management (FedCM), even if that means shipping an identity provider as part of the deliverables (I generally go with Keycloak, with keycloak-config-cli to provision its configuration). I'm obviously biased though as I work in IT services, developping software mainly for intranets/extranets, and companies now increasingly have their own identity providers or at a minimum have that in their roadmap. So YMMV.

And we've only talked about authentication, not even authorization!

Some resources to go farther:

What are JWT?

Thomas Broyer — Wed, 29 Nov 2023 19:50:38 +0000

This is a translation of an article I wrote for our internal knowledge base at work, and that we later decided to publish (in French).

This article's goal is to present what JWTs are, whenever you face them. As we'll see, you won't deliberately choose to use JWTs in a project, and more importantly: you won't use JWTs as session tokens.

What is it?

JSON Web Token (JWT) is a compact, URL-safe means of representing data to be transferred between two parties. The data is encoded as a JSON object that can be signed and/or encrypted.

This is, paraphrased, the definition from the IETF standard that defines it (RFC 7519).

What's the point? What's the use case?

So the goal is to transfer data, with some guarantees (or none, by the way): authenticity, integrity, even possibly confidentiality (if the message is encrypted). There are therefore many possible uses.

JWT is thus used in OpenID Connect to encode the ID Token that forwards to the application information on the authentication process that took place at the identity server. OpenID Connect also uses JWT to encode aggregated claims: information from other identity servers, for which we'll want to verify the authenticity and integrity.

A JWT might be used to authenticate to a server, such as with the OAuth 2 JWT Bearer (RFC 7523).

Still in OAuth 2 land, access tokens could themselves be JWTs (RFC 9068), authorization request parameters could be encoded as a JWT (RFC 9101), as well as token introspection responses (IETF draft: JWT Response for OAuth Token Introspection), and finally dynamic client registration uses a JWT to identify the software of which an instance attempts to register (so-called software statements of RFC 7591).

How does it work?

A JWT is composed of at least 2 parts, separated with a . (dot), the first one always being the header. Each part is always encoded as base64url, a variant of Base 64 with the + and / characters (that have special meaning in URLs) replaced with - and _ respectively, and without the trailing =.

There are two types of JWTs: JSON Web Signature (JWS, defined by RFC 7515), and JSON Web Encryption (JWE, defined by RFC 7516). The most common case is the JWS, composed of 2 or 3 parts: the header, the payload, and optionally the signature. JWEs are rarer (and more complex) so I won't talk about them here.

The header, common to both types, describes the type of JWT (JWS or JWE) as well as the different signature, MAC, or encryption algorithms being used (codified by RFC 7518), along with other useful information, as a JSON object.

In the case of JWS, we'll find the signature or MAC algorithm, possibly a key identifier (whenever multiple keys can be used, e.g. to allow for key rotation), or even a URL pointing to information about the keys (in JWKS format, defined by RFC 7517), etc.

In the case of JWS, the payload will generally be a JSON object with the transfered data (but technically could be another JWT).

The third part is the signature or MAC. This part is absent if the header says the JWT is unprotected ("alg":"none").

For debugging, one can use the JWT Debugger by Auth0 to decode JWTs (beware not to use it with sensitive data, only on JWTs coming from test servers).

⚠️ JWT being almost always used in security-related contexts, handle them with care, specifically when it comes to their cryptographical components.

One MUST use dedicated libraries to manipulate JWTs, and be careful to use them correctly to avoid introducing vulnerabilities.

RFC 8725 has a set of best practices when manipulating and using JWTs.

Criticism

Numerous security experts, among them cryptographers, vehemently criticize JWTs and advise against their use.

The main criticism relates to its complexity, even though it could look simple to developers:

first, you need to know how to decode UTF-8 and JSON ; that's as many sources of bugs (and potential vulnerabilities).
and of course because it's a generic format capable of signing and/or encrypting, or even not protecting anything at all ("alg":"none"), with a list of supported algorithms as long as your arm, you have to handle many cases (even if only to reject them).

As a result, a number of vulnerabilities have been identified; among them (identified as soon as March 2015):

As the JWT itself declares the algorithm used to sign or encrypt it, software that receives it needs to partly trust it, or correctly check the used algorithm against a list of authorized algorithms. Because of its apparent simplicity, many libraries came out that didn't do those necessary checks and readily accepted unprotected JWTs ("alg":"none"), allowing an attacker to use any JWT, without authenticity or integrity check. And as incredible as it may seem, we still find vulnerable applications nowadays!

Note: in the same way, the header can directly include the public key to be used to verify the signature. Using it will prove the integrity of the JWT, but not its authenticity as the signature could have been generated by anyone.

Another attack involes using the public key intended to verify an asymmetric signature ("alg":"RS256" or "alg":"ES256") as a MAC key ("alg":"HS256"): the application receiving the JWT could then mistakenly validate the MAC and allow the JWT in. Anybody could then create a JWS that would be accepted by the application, when that one thinks it's verifying an asymmetric signature.

This vulnerability could be due to a misuse of the library used to verify JWTs, but also in some cases directly to its API that cannot tell between a public key and a shared secret (generally for the sake of making it easy to use).

Aside: despite ID Tokens in OpenID Connect being JWTs, you won't actually need to verify their signature as you generally get them through HTTPS, that already guarantees authenticity and integrity (and confidentiality), which saves us from a whole class of vulnerabilities.

Another criticism is due to the misuse of JWT, most often by ignorance or lack of expertise in software security: validity of a JWT is directly verifiable, without the need for a database of valid tokens or a validation service (authenticity and integrity are verifiable, so the validity period contained within in the JWT are reliable), but it makes the JWT impossible to revoke (unless you add such a mecanism –possibly based on the jti claim, initially designed to protect against replay attacks– going against the whole reason for which JWT was chosen in the first place). If a JWT is used as a session token for example, it then becomes impossible to sign out or terminate a session. In most use cases (in the specifications), a JWT is validated and used as soon as it's received from the issuer, so revocation is not even an issue. It's when a JWT is stored by the receiver for a later use that the problem arises (such as with a session token or an access token).

Some articles critical of JWT:

JWT should not be your default for sessions (by Evert Pot, developper)
Why JWTs Suck as Session Tokens (by Okta, vendor of an identity management platform)
section "JSON Web Tokens" of "API Tokens: A Tedious Survey" (by Thomas H. Ptacek, security researcher)
Alternatives to JWTs (by Scott Brady, engineering manager specializing in identity management systems)
Stop using JWTs for sessions and Stop using JWT for sessions, part 2: Why your solution doesn’t work (on a web site surprisingly without HTTPS)
No Way, JOSE! Javascript Object Signing and Encryption is a Bad Standard That Everyone Should Avoid (by Scott “CiPHPerCoder” Arciszewski, cryptographer)
How to Write a Secure JWT Library If You Absolutely Must (by the same author)

How I teach Git

Thomas Broyer — Sun, 26 Nov 2023 19:29:47 +0000

I've been using Git for a dozen years. Eight years ago, I had to give a training session on Git (and GitHub) to a partner company about to create an open source project, and I'm going to tell you here about the way I taught it. Incidentally, we created internal training sessions at work since then that use the same (or similar) approach. That being said, I didn't invent anything: this is heavily inspired by what others wrote before, including the Pro Git book, though not in the same order, and that IMO can make a difference.

The reason I'm writing this post is because over the years, I've kept seeing people actually use Git without really understanding what they're doing; they'd either be locked into a very specific workflow they were told to follow, and unable to adapt to another that, say, an open source project is using (this also applies to open source maintainers not really understanding how external contributors use Git themselves), or they'd be totally lost if anything doesn't behave the way they thought it would, or if they made a mistake invoking Git commands. I've been inspired to write it down by Julia Evans' (renewed) interest in Git, as she sometimes ask for comments on social networks.

My goal is not to actually teach you about Git, but more about sharing my approach to teaching Git, for others who will teach to possibly take inspiration. So if you're learning Git, this post was not written with you in mind (sorry), and as such might not be self-sufficient, but hopefully the links to other learning resources will be enough to fill the blanks are make it a helpful learning resource as well. If you're a visual learner, those external learning resources are illustrated, or even oriented towards visual learning.

Mental model

Once we're clear why we use a VCS (Version Control System) where we record changes inside commits (or in other words we commit our changes to the history; I'm assuming some familiarity with this terminology), let's look at Git more specifically.

One thing I think is crucial to understand Git, is getting an accurate mental model of the concepts behind it.

First, that's not really important, but Git doesn't actually record changes, but rather snapshots of our files (at least conceptually; it will use packfiles to store things efficiently and will actually store changes –diffs– in some cases), and will generate diffs on-demand. This sometimes shows in the result of some commands though (like why some commands show one file removed and another added, while other commands show a file being renamed).

Now let's dive into some Git concepts, or how Git implements some common VCS concepts.

Commit

A Git commit is:

one or more parent commit(s), or none for the very first commit (root)
a commit message
an author and an author date (actually a timestamp with timezone offset)
a committer and commit date
and our files: their pathname relative to the repository root, their mode (UNIX file-system permissions), and their content

Each commit is given an identifier determined by computing the SHA1 hash of this information: change a comma and you get a different SHA1, a different commit object. (Fwiw, Git is slowly moving to SHA-256 as the hashing function).

Aside: how's the SHA1 computed?

Git's storage is content-adressed, meaning that each object is stored with a name that's directly derived from its content, in the form of its SHA1 hash.

Historically, Git stored everything in files, and we can still reason that way. A file's content is store as a blob, a directory is stored as tree (a text file that lists files in the directory with their name, mode, and the SHA1 of the blob representing their content, and their subdirectories with their name and the SHA1 their tree)

If you want the details, Julia Evans wrote an amazing (again) blog post; or you can read it from the Pro Git book.

A commit and its tree (source: Pro Git)

The parent commit(s) in a commit create a directed acyclic graph that represents our history: a directed acyclic graph is made of nodes (our commits) linked together with directed edges (each commit links to its parent(s) commit(s), there's a direction, hence directed) and cannot have loops/cycles (a commit will never be its own ancestor, none of its ancestor commits will link to it as a parent commit).

Commits and their parents (source: Pro Git)

References, branches and tags

Now SHA1 hashes are impractical to work with as humans, and while Git allows us to work with unique SHA1 prefixes instead of the full SHA1 hash, we'd need simpler names to refer to our commits: enter references. Those are labels for our commits that we chose (rather than Git).

There are several kinds of references:

branches are moving references (note that main or master aren't special in any way, their name is only a convention)
tags are immutable references
HEAD is a special reference that points to the current commit. It generally points to a branch rather than directly to a commit (we'll see why later). When a reference points to another reference, this is called a symbolic reference.
there are other special references (FETCH_HEAD, ORIG_HEAD, etc.) that Git will setup for you during some operations

A branch and its commit history (source: Pro Git)

The three states

When you work in a Git repository, the files that you manipulate and record in the Git history are in your working directory. To create commits, you'll stage files in the index or staging area. When that's done you attach a commit message and move your staged files to the history.

And to close the loop, the working directory is initialized from a given commit from your history.

Working tree, staging area, and Git directory (source: Pro Git)

Aside: ignoring files

Not all files need to have their history tracked: those generated by your build system (if any), those specific to your editor, and those specific to your operating system or other work environment.

Git allows defining naming patterns of files or directories to ignore. This does not actually mean that Git will ignore them and they cannot be tracked, but that if they're not tracked, several Git operations won't show them to you or manipulate them (but you can manually add them to your history, and from then on they'll no longer be ignored).

Ignoring files is done by putting their pathname (possibly using globs) in ignore files:

.gitignore files anywhere in your repository define ignore patterns for the containing directory; those ignore files are tracked in history as a mean to share them between developers; this is where you'll ignore those files generated by your build system (build/ for Gradle projects, _site/ for an Eleventy website, etc.)
.git/info/excludes is local to the repository on your machine; rarely used but sometimes useful so good to know about
and finally ~/.config/git/ignore is global to the machine (for your user); this is where you'll ignore files that are specific to your machine, such as those specific to the editors you use, or those specific to your operating system (e.g. the .DS_Store on macOS, or Thumbs.db on Windows)

Summing up

Here's another representation of all those concepts:

Commits, references, and areas (source: A Visual Git Reference, Mark Lodato)

Basic operations

This is where we start talking about Git commands, and how they interact with the graph:

git init to initialize a new repository
git status to get a summary of your files' state
git diff to show changes between any two of your working directory, the index, the HEAD, or actually between any commit
git log to show and search into your history
creating commits
- git add to add files to the index
- git commit to transform the index into a commit (with an added commit message)
- git add -p to add files interactively to the index: pick which changes to add and which ones to leave only in your working directory, on a file-by-file, part-by-part (called hunk) basis
managing branches
- git branch to show branches, or create a branch
- git switch (also git checkout) to check out a branch (or any commit, any tree, actually) to your working directory
- git switch -b (also git checkout -b) as a shortcut for git branch and git switch
git grep to search into your working directory, index, or any commit; this is kind of an enhanced grep -R that's aware of Git
git blame to know the last commit that changed each line of a given file (so, who to blame for a bug)
git stash to put uncommitted changes aside (this includes staged files, as well as tracked files from the working directory), and later unstash them.

Commit, branch switching, and HEAD

When you create a commit (with git commit), Git not only creates the commit object, it also moves the HEAD to point to it. If the HEAD actually points to a branch, as is generally the case, Git will move that branch to the new commit (and HEAD will continue to point to the branch). Whenever the current branch is an ancestor of another branch (the commit pointed by the branch is also part of another branch), committing will move HEAD the same, and branches will diverge.

When you switch to another branch (with git switch or git checkout), HEAD moves to the new current branch, and your working directory and index are setup to ressemble the state of that commit (uncommitted changes are tentatively kept; if Git is unable to do it, it will refuse the switch).

For more details, and visual representations, see the commit and checkout sections of Mark Lotato's A Visual Git Reference (be aware that this reference was written years ago, when git switch and git restore didn't exist and git checkout was all we had; so the checkout section covers a bit more than git switch as a result).
Of course, the Pro Git book is also a good reference with visual representations; the Branches in a Nutshell subchapter covers a big part of all of the above.

Aside: Git is conservative

As we've seen above, due to its content-addressed storage, any “change” to a commit (with git commit --amend for instance) will actually result in a different commit (different SHA1). The old commit won't disappear immediately: Git uses garbage collection to eventually delete commits that aren't reachable from any reference. This means that many mistakes can be recovered if you manage to find the commit SHA1 back (git reflog can help here, or the notation <branch-name>@{<n>}, e.g. main@{1} for the last commit that main pointed to before it changed).

Working with branches

We've seen above how branches can diverge.
But diverging calls for eventually merging changes back (with git merge). Git is very good at that (as we'll see later).

A special case of merging is when the current branch is an ancestor of the branch to merge into. In this case, Git can do a fast-forward merge.

Because operations between two branches will likely always target the same pair of branches, Git allows you to setup a branch to track another branch. That other branch with be called the upstream of the branch that tracks it. When setup, git status will, for example, tell you how much the two branches have diverged from one another: is the current branch up to date with its upstream branch, behind it and can be fast-forwarded, ahead by a number of commits, or have they diverged, each by some number of commits. Other commands will use that information to provide good default values for parameters so they can be omitted.

To integrate changes from another branch, rather than merging, another option is to cherry-pick (with the same-named command) a single commit, without its history: Git will compute the changes brought in by that commit and apply the same changes to the current branch, creating a new commit similar to the original one (if you to know more about how Git actually does it, see Julia Evans' How git cherry-pick and revert use 3-way merge).

Finally, another command in your toolbelt is rebase.
You can see it as a way to do many cherry-picks at once but it's actually much more powerful (as we'll see below). In its basic use though, it's just that: you give it a range of commits (between any commit as the starting point and an existing branch as the end point, defaulting to the current one) and a target, and it cherry-picks all those commits on top of the target and finally updates the branch used as the end point. The command here is of the form git rebase --onto=<target> <start> <end>. As with many Git commands, arguments can be omitted and will have default values and/or specific meanings: thus, git rebase is a shorthand for git rebase --fork-point upstream where upstream is the upstream of the current branch (I'll ignore --fork-point here, its effect is subtle and not that important in every-day use), which itself is a shorthand for git rebase upstream HEAD (where HEAD must point to a branch), itself a shorthand for git rebase --onto=upstream upstream HEAD, a shorthand for git rebase --onto=upstream $(git merge-base upstream HEAD) HEAD, and will rebase all commits between the last common ancestor of upstream and the current branch on one hand and the current branch (i.e. all commits since they diverged) on the other hand, and will reapply them on top of upstream, then update the current branch to point to the new commits. Explicit use of --onto (with a value different from the starting point) is rare actually, see my previous post for one use case.

We cannot present git rebase without its interactive variant git rebase -i: it starts with exactly the same behavior as the non-interactive variant, but after computing what needs to be done, it'll allow you to edit it (as a text file in an editor, one action per line). By default, all selected commits are cherry-picked, but you'll be able to reorder them, to skip some commit(s), or even combine some into a single commit. You can actually cherry-pick a commit that was not initially selected, and even create merge commits, thus entirely rewriting the whole history! Finally, you can also stop on a commit to edit it (using git commit --amend then, and/or possibly create new commits before continuing with the rebase), and/or run a given command between two commits. This last option is so useful (to e.g. validate that you didn't break your project at each point of the history) that you can pass that command in an --exec option and Git will execute it between each rebased commit (this works with non-interactive rebase too; in interactive mode you'll see execution lines inserted between each cherry-pick line when given the ability to edit the rebase scenario).

For more details, and visual representations, see the merge, cherry pick, and rebase sections of Mark Lodato's A Visual Git Reference, and the Basic Branching and Merging, Rebasing, and Rewriting History subchapters of the Pro Git book.
You can also look at the “branching and merging” diagrams from David Drysdale's Git Visual Reference.

Working with others

For now, we've only ever worked locally in our repository.
But Git was specifically built to work with others.

Let me introduce remotes.

Remotes

When you clone a repository, that repository becomes a remote of your local repository, named origin (just like with the main branch, this is just the default value and the name in itself has nothing special, besides sometimes being used as the default value when an command argument is omitted). You'll then start working, creating local commits and branches (therefore forking from the remote), and the remote will probably get some more commits and branches from its author in the mean time. You'll thus want to synchronize those remote changes into your local repository, and want to quickly know what changes you made locally compared to the remote. The way Git handles this is by recording the state of the remote it knows about (the branches, mainly) in a special namespace: refs/remote/. Those are known as remote-tracking branches. Fwiw, local branches are stored in the refs/heads/ namespace, and tags in refs/tags/ (tags from remotes are generally imported right into refs/tags/, so for instance you lose the information of where they came from). You can have as many remotes as needed, each with a name. (Note that remotes don't necessarily live on other machines, they can actually be on the same machine, accessed directly from the filesystem, so you can play with remotes without having to setup anything.)

Fetching

Whenever you fetch from a remote (using git fetch, git pull, or git remote update), Git will talk to it to download the commits it doesn't yet know about, and will update the remote-tracking branches for the remote. The exact set of references to be fetched, and where they're fetched, is passed to the git fetch command (as refspecs) and the default value defined in your repository's .git/config, and configured by default by git clone or git remote add to taking all branches (everything in refs/heads/ on the remote) and putting them in refs/remote/<remote> (so refs/remote/origin/ for the origin remote), with the same name (so refs/heads/main on the remote becomes refs/remote/origin/main locally).

Remotes and remote-tracking branches (source: Pro Git)

You'll then use branch-related commands to get changes from a remote-tracking branch to your local branch (git merge or git rebase), or git pull which is hardly more than a shorthand for git fetch followed by a git merge or git rebase. BTW, in a number of situations, Git will automatically setup a remote-tracking branch to be the upstream of a local branch when you create it (it will tell you about it when that happens).

Pushing

To share your changes with others, they can either add your repository as a remote and pull from it (implying accessing your machine across the network), or you can push to a remote. (If you ask someone to pull changes from your remote, this is called a… pull request, a term you'll have probably heard of from GitHub or similar services.)

Pushing is similar to fetching, in reverse: you'll send your commits to the remote and update its branch to point to the new commits. As a safety measure, Git only allows remote branches to be fast-forwarded; if you want to push changes that would update the remote branch in a non-fast-forward way, you'll have to force it, using git push --force-with-lease (or git push --force, but be careful: --force-with-lease will first ensure your remote-tracking branch is up-to-date with the remote's branch, to make sure nobody pushed changes to the branch since the last time you fetched; --force won't do that check, doing what you're telling it to do, at your own risks).

As with git fetch, you pass the branches to update to the git push command, but Git provides a good default behavior if you don't. If you don't specify anything, Git will infer the remote from the upstream of the current branch, so most of the time git push is equivalent to git push origin. This actually is a shorthand to git push origin main (assuming the current branch is main), itself a shorthand for git push origin main:main, shorthand for git push origin refs/heads/main:refs/heads/main, meaning to push the local refs/heads/main to the origin remote's refs/heads/main. See my previous post for some use cases of specifying refspecs with differing source and destination.

git push (source: Git Visual Reference, David Drysdale)

For more details, and visual representations, see the Remote Branches, Working with Remotes, and Contributing to a Project subchapters of the Pro Git book, and the “dealing with remote repositories” diagrams from David Drysdale's Git Visual Reference.
The Contributing to a Project chapter of Pro Git also touches about contributing to open source projects on platforms like GitHub, where you have to first fork the repository, and contribute through pull requests (or merge requests).

Best practices

Those are directed towards beginners, and hopefully not too controversial.

Try to keep a clean history:

use merge commits wisely
clear and high-quality commit messages (see the commit guidelines in Pro Git)
make atomic commits: each commit should be compile and run independently of the commits following it in the history

This only applies to the history you share with others.
Locally, do however you want. For beginners, I'd give the following advices though:

don't work directly on main (or master, or any branch that you don't specifically own on the remote as well), create local branches instead; it helps decoupling work on different tasks: about to start working on another bug or feature while waiting for additional details on instructions on the current one? switch to another branch, you'll get back to that later by switching back; it also makes it easier to update from the remote as you're sure you won't have conflicts if your local branches are simply copies of the remote ones of the same name, without any local change (except when you want to push those changes to that branch)
don't hesitate to rewrite your commit history (git commit --amend and/or git rebase -i), but don't do it too early; its more than OK to stack many small commits while working, and only rewrite/cleanup the history before you share it
similarly, don't hesitate to rebase your local branches to integrate upstream changes (until you shared that branch, at which point you'll follow the project's how branching workflow)

In case of any problem and you're lost, my advice is to use gitk or gitk HEAD @{1}, also possibly gitk --all (I'm using gitk here but use whichever tool you prefer), to visualize your Git history and try to understand what happened. From this, you can rollback to the previous state (git reset @{1}) or try to fix things (cherry-picking a commit, etc.) And if you're in the middle of a rebase, or possibly a failed merge, you can abort and rollback to the previous state with commands like git rebase --abort or git merge --abort.

To make things even easier, don't hesitate, before any possibly destructive command (git rebase), to create a branch or a tag as a "bookmark" you can easily reset to if things don't go as expected. And of course, inspect the history and files after such a command to make sure the outcome is the one you expected.

Advanced concepts

Only a few of them, there are many more to explore!

Detached HEAD: the git checkout manpage has a good section on the topic, also see my previous post, and for a good visual representation, see the Committing with a Detached HEAD section of Mark Lodato's A Visual Git Reference.
Hooks: those are executables (shell scripts most of the time) that Git will run in reaction to operations on a repository; people use them to lint the code before each commit (aborting the commit if that fails), generate or post-process commit messages, or trigger actions on the server after someone pushes to the repository (trigger builds and/or deployments).
A couple rarely needed commands that can save you hours when you actually need them:
- git bisect: an advanced command to help you pinpoint which commit introduced a bug, by testing several commits (manually or through scripting); with a linear history, this is using bisection and could be done manually, but as soon as you have many merge commits this becomes much more complex and it's good to have git bisect do the heavy lifting.
- git filter-repo: a third-party command actually, as a replacement to Git's own filter-branch, that allows rewriting the whole history of a repository to remove a mistakenly added file, or help extract part of the repository to another.

We're done.

With this knowledge, one should be able to map any Git command to how it will modify the directed acyclic graph of commits, and understand how to fix mistakes (ran a merge on the wrong branch? rebased on the wrong branch?) I'm not saying understanding such things will be easy, but should at least be possible.

Climate-friendly software: don't fight the wrong battle

Thomas Broyer — Mon, 01 May 2023 00:05:08 +0000

When talking about software ecodesign, green IT, climate-friendly software, the carbon footprint of software, or however you name it, most of the time people focus on energy efficiency and server-side code, sometimes going to great length measuring and monitoring it. But what if all this was misguided?

Ok, this is a bit of a bold statement, but don't get me wrong:I'm not saying you shouldn't care about this. Let's look at one of the most recent examples I've seen: GitHub's ReadME Project Q&A: Slash your code's carbon footprint newsletter issue. It's good and I agree with many things in there (go read it if you haven't already), but it talks almost exclusively about energy efficiency and server-side code, or in other words it limits actions to the scope 2 of the GHG Protocol.

So let's first understand which impacts we're talking about before I give you my opinion on the low-hanging fruits.

Disclaimer: people regarded as experts in green IT trusted me enough to have me contribute to a book on the subject but I'm not myself an expert in the field.

Note: this post is written for developers and software architects; there are other actions to lower the climate impact of the digital world that won't be covered here.

Stepping back

Most software nowadays is client-server: whether web-based or mobile, more and more end-user software talk to servers. This means there's a huge asymmetry in usage: even for small-scale professional software the end users generally vastly outnumber the servers. And this implies the impacts of the individual clients need to be much lower than those of the servers.

Data Table

	Manufacturing	Use	Total
User equipment	40%	26%	66%
Networks	3%	17%	19%
Data centers	1%	14%	15%
Total	44%	56%

Greenhouse gas emissions balance (source, PDF, 533 KB)

What life-cycle assessments (LCA) for end-users' devices tell us is that manufacturing, transport and disposal summed up immensely outweighs use, ranging from 65% up to nearly 98% of the global warming potential (GWP). Of course, this depends where the device was manufactured and where it's being used, with the use location's biggest impact being related to the carbon footprint of the electric system, as the use phase is all about charging or powering our smartphones, laptops and desktops.

Estimated Greenhouse Gas (GHG) emissions for a Google Pixel 7 (source, PDF, 224 KB)

Estimated carbon footprint allocation for my Dell Precision 3520, assuming 4 years of use (I've had mine for more than 5.5 years already): 304 kg CO₂e ± 68 kg CO₂e (source, PDF, 557 KB)

I am French, working mainly for French companies with most of their users in France, so I'm ready to admit I'm biased towards a very low use phase weight compared to other regions: go explore data for your users on Electricity Map and Our World in Data. And yet, that doesn't change the fact that the use phase has a much lower carbon footprint than all three of manufacturing, transport, and disposal as a whole.

What we can infer from this, is that keeping our devices longer will increase the share of use in the whole life-cycle impacts. Fairphone measured that extending the lifespan of their phones from 3 to 5 years helps reduce the yearly emissions on global warming by 31%, while a further extension to 7 years of use helps reduce the yearly impact by 44%.

Fairphone 4: comparative of yearly emissions per baseline scenario (source, PDF, 1.1 MB)

Extending the lifespan of a smartphone from 3 to 5 years can reduce its yearly global warming impacts by almost a third.

Things are different for servers though, where the use phase's share varies much more depending on use location: from 4% up to 85%! As noted in the ReadME Project Q&A linked above, big companies' datacenters are for the most part net-neutral in carbon emissions, so not only the geographic regions of your servers matter, but also the actual datacenters in those regions. This implies that whatever you do on the server side, its impact will likely be limited (remember what I was saying in the introduction?) Of course there are exceptions, and there will always be, so please look at this through the prism of your own workloads.

Estimated carbon footprint for a Dell PowerEdge R640 server, assuming 4 years of use: 7730 kg CO₂e (source, PDF, 514 KB)

Keep in mind the orders of magnitude though: 70 kg CO₂e for a single Pixel 7 (on 3 years) vs. 7730 kg CO₂e for a Dell PowerEdge R640 server (on 4 years), that's 110 smartphones for a server (or a 83:1 ratio when considering yearly emissions): chances are that you'll have much more users than that. The ratio for laptops (304 kg CO₂e on 4 years for a Dell Precision 3520) would be 25 laptops for a server. But as seen previously the actual carbon footprint will vary a lot depending on the location; you can explore some data in the Boavizta data visualization tool that compiles dozens of LCAs of various manufacturers. The Dell PowerEdge R640 in France would actually emit 1701 kg CO₂e rather than 7730 kg CO₂e: that's a 4.5:1 ratio! Comparatively, my Dell Precision 3520 would fall from 304 kg CO₂e to 261 kg CO₂e, only a 1.16:1 ratio. The laptop to server ratio would thus fall from 25 down to 7.9:1, which makes the laptops' impacts comparatively much bigger than the server compared to other regions.

Note that there are three tiers: end-users, datacenters, and networks. Network energy consumption however doesn't vary proportionally to the amount of data transferred, which means we as users of those networks don't have much levers on their footprint. That being said, data transmission is among the things that will drain the batteries of mobile devices, so reducing the amount of data you exchange on the network could have a more direct impact on the battery life of end-users' smartphones (even though what will drain the battery the most will more likely be the screen).

Taking action

So, what have we learned so far?

It's important that end users keep their devices longer,
we can't do much about networks,
the location (geographic region and datacenter) of servers matter a lot, more so than how and how much we use them.

Now, what can we do about it?

For servers, it's relatively simple: if you can, rent servers in energy efficient datacenters, and/or countries with low-carbon electricity; in addition, or otherwise, then of course optimize your server-side architecture and code. If you manage your own servers, avoid buying machines to let them sit idle: maximize their utilization.

Pick servers in carbon-neutral or low-carbon datacenters first, then optimize your architecture and code.

For the networks, our actions are probably limited to reducing data usage, not because it reduces immediate emissions (it doesn't), but to avoid the need for rapid expansion of the network infrastructure (I'm quoting Wim Vanderbauwhede here, from a private conversation).

For the end-users' devices, it's more complicated, but not out of reach: we want users to keep their devices as long as possible so, put differently, we must not be responsible for them to change their devices. There will always be people changing devices "for the hype" or on some scheduled basis (or just because the vendor stopped pushing security updates, some form of planned obsolescence, or can't be repaired; two things laws could alleviate), but there are also many people who keep them as long as possible (because they're eco-conscious or can't afford purchasing a new device, or simply because they don't feel the need for changing something that's still fully functioning.) For those people, don't be the one to make them change their mind and cross the line.

Don't be the one that will make your users change their device.

This is something we won't ever be able to measure, as it depends on how people perceive the overall experience on their device, but it boils down to perceived performance. So by all means, optimize your mobile apps and web frontends, test on old devices and slow networks (even if only emulated), and monitor their real-user performance (e.g. through Web Vitals). As part of performance testing, have a look at electricity use, as it will both be directly associated with emissions to produce that electricity, and be perceptible by the user (battery drain). And don't forget to account for the app downloads as part of the overall perceived performance: light mobile apps that don't need to be updated every other day, frontend JS and CSS that can be cached and won't update several times a day either (defeating the cache).

Optimize for the perceived performance and battery life.

Don't forget about the space taken by your app on the user's device too: users shouldn't have to make a choice between apps due to no space left on device, so when possible prefer a website or progressive web app (PWA) to a native application (you can still publish them to application stores if required, through tiny wrapper native apps).

When possible, prefer a website or PWA to a native application.

A note to product managers

The above advices were mostly technical, answering the question What can I do as an architect or developer?
but product managers have their share, and they're actually the ones in power here: they can choose which features to build or not build, they can shape the features, they can reduce software complexity by limiting the number of features and of levers and knobs. This will undoubtedly avoid bloat and help you make things leaner and faster.

Avoid feature creep and beware of Wirth's law.

Refrain from adding features, reduce software complexity.

Last, but not least, make sure you really need software! Sometimes you should embrace low-tech. For example, instead of developing a mobile app with accounts to identify the user so you can notify them, then maybe you could simply use SMS (assuming you have some out-of-band means of knowing their phone number, and the latency of distribution is acceptable). And sometimes what you're trying to address with software just isn't worth it, particularly if it involves IoT (remember that we should strive for fewer devices that we keep longer, not more).

Sometimes, ideas aren't even worth their impacts.

Conversely, as we'll need to electrify parts of our economy to reduce their carbon footprint, software is one of the few sectors to start with a head-starts: we get greener at the same rate as the grid without other work needed (I'm quoting Alex Russell here, from a private conversation), so please do use software to digitalize and replace more carbon-intensive activities.

Other pitfalls

Besides only evaluating electricity consumption on your servers, another pitfall is trying to attribute emissions to each user or request: when you have dozens, hundreds or even thousands of concurrent requests, how do you distribute electricity consumption among them? There's an IETF proposal for a HTTP response header exposing such information, and while it's a commendable idea I doubt it's realistic. My personal belief is that display of such information is often a sign of greenwashing. To my knowledge, data can only be accurate in aggregates.

If you really do want to show how green you are, conduct a life-cycle assessment (LCA): take all three scopes into account, all three tiers, evaluating impacts over more criterias than the global warming potential (GWP) alone.

Here are a couple resources if you want to go farther:

Thanks to Alex Russell and Wim Vanderbauwhede for their feedback.

Naming things is hard, SPA edition

Thomas Broyer — Tue, 28 Mar 2023 16:09:10 +0000

During the past few months, social networks have been shaken by a single-page vs multi-page applications (SPA vs MPA) battle, more specifically related to Next.js and React, following, among other things, a tweet by Guillermo Rauch and a GitHub comment by Dan Abramov.

I've read a few articles and been involved in a few discussions about those and it appeared that we apparently don't all have the same definitions, so I'll give mine here and hope people rally behind them.

SPA vs MPA: it's about navigation

It's not that hard: a single-page application means that you load a page (HTML) once, and then do everything in there by manipulating its DOM and browser history, fetching data as needed.
This is the exact same thing as client-side navigation and requires some form of client-side routing to handle navigation (particularly from history, i.e. using the back and forward browser buttons).

Conversely, a multi-page application means that each navigation involves loading a new page.

This by itself is a controversial topic: despite SPAs having lots of problems (user experience –aborting navigation, focus management, timing of when to update the URL bar–, accessibility, performance even by not being able to leverage streaming) due to taking responsibility and having to reimplement many things from the browser (loading feedback, error handling, focus management, scrolling), some people strongly believe this is “one of the first interesting optimizations” and they “can’t really seriously consider websites that reload page on every click good UX”
(I've only quoted Dan Abramov from the React team here, but I don't want to single him out: he's far from being alone with this view; others are in denial thinking that “this is the strategy used by most of the industry today”).
Some of those issues are supposedly (and hopefully) fixed by the new navigation API that's currently only implemented in Chromium browsers.
And despite their many advantages, MPAs aren't free from limitations too, otherwise we probably wouldn't have had SPAs to being with.

My opinion? There's no one-size-fits-all: most sites and apps could (and probably should) be MPAs, and an SPA is a good (and better) fit for others.
It's also OK to use both MPA and SPA in a single application depending on the needs.
Jason Miller published a rather good article 4 years ago (I don't agree with everything in there though).
Nolan Lawson also has written a good and balanced series on MPAs vs SPAs.

And we haven't even talked about where the rendering is done yet!

Rendering: SSR, ESR, SWSR, and CSR

Before diving into where it's done, we first need to define what rendering is.

My definition of rendering is applying some form of templating to some data.
This means that getting some HTML fragment from the network and putting it into the page with some form of innerHTML is not rendering.
Conversely, getting some virtual DOM as JSON for example and reconstructing the equivalent DOM from it would qualify as rendering.

Now that we've defined what rendering is, let's see where it can be done: basically at each and any stage of delivery: the origin server (SSR), edge (ESR), service-worker (SWSR), or client (CSR).

There's also a whole bunch of prerendering techniques: static site generation (SSG), on-demand generation, distributed persistent rendering (DPR), etc.

All these rendering stages, except client-side rendering (CSR), generate HTML to be delivered to the browser engine.
CSR will however directly manipulate the DOM most of the time, but sometimes will also generate HTML to be used with some form of innerHTML; the details here don't really matter.

Rendering at the origin server or at the edge (Cloudflare Workers, Netlify Functions, etc.) can be encompassed under the name server-side rendering (SSR), but depending on the context SSR can refer to the origin server only.
Similarly, rendering in a service worker could be included in client-side rendering (CSR), but most of the time CSR is only about rendering in a browsing context.
I suppose we could use browser-side rendering (BSR) to encompass CSR and SWSR.

As noted by Jason Miller and Addy Osmani in their Rendering on the Web blog post, applications can leverage several stages of rendering (SSR used in the broader sense here), but like many they conflate SPA and CSR.
Eleventy (and possibly others) also allows rendering a given page at different stages, with parts of the page prerendered at build-time or rendered on the origin server, while other parts will be rendered at the edge.

What does that imply?

My main point is that rendering is almost orthogonal to single-page vs multi-page: an SPA doesn't imply CSR.

Most web sites are MPAs with SSR, sometimes ESR.
Most React/Vue/Angular applications are SPAs with CSR: the HTML page is mostly empty, generally the same for every URL, and the page loads data on boot and renders it (at the time of writing, the Angular website is such an SPA+CSR).
Next.js/Gatsy/Remix/Nuxt/Angular Universal/Svelte Kit/Solid Start/îles applications are SPAs with SSR and CSR: data is present as HTML in the page, but navigations then use CSR staying on the same page (and actually, despite the content being present in the HTML page, those frameworks will discard and re-render it client-side on boot).
Qwik City/Astro/Deno Fresh/Enhance/Marko Run applications are MPAs with SSR (and CSR as needed through islands of interactivity); Qwik City provides an easy way to switch to an SPA with SSR and CSR (though contrary to the above-mentioned frameworks, Qwik City won't re-render on page load).
Hotwire Turbo Drive (literally HTML over the wire; formerly Turbolinks) and htmx applications are SPAs with SSR.
GitHub is known for its use of Turbolinks and is actually both MPA and SPA, depending on pages and sometimes navigation (going from a user profile to a repository loads a new page, but the reverse is a client-side navigation).

Some combinations aren't really useful: an MPA with CSR (and without SSR) would mean loading an almost empty HTML page at each navigation to then fetch data (or possibly getting it right from HTML page) and do the rendering. Imagine the Angular website (which already makes a dubious choice of not including the content in the HTML page, for a documentation site) but where all navigations would load a new (almost empty) page.

Similarly, if you're doing a SPA, there's no real point in doing rendering in a service worker as it could just as well be done in the browsing context; unless maybe you're doing SPA navigation only on some pages/situations (video playing?) and want to leverage SWSR for all pages including MPAs?

Other considerations

In an application architecture, navigation and rendering locality aren't the only considerations.

Inline updates

Not every interaction has to be a navigation:
there are many cases where a form submission would return to the same page (reacting to an article on Dev.to, posting a comment, updating your shopping cart), in which case progressive enhancement could be used to do an inline update without a full page refresh.

Those are independent from SPAs: you can very well have an MPA and use such inline updates.
Believe it or not, this is exactly what Dev.to does for their comment form (most other features like following the author, reacting to the post or a comment, or replying to a comment however won't work at all if JavaScript is somehow broken).

Concatenation and Includes

Long before we had capable enough JavaScript in the browser to build full-blown applications (in the old times of DHTML, before AJAX), there already were optimization techniques on the servers to help build an HTML page from different pieces, some of which could have been prerendered and/or cached.
Those were server-side includes and edge-side includes.

While they are associated with specific syntaxes, the concepts can be used today in edge functions or even in service workers.

The different parts being concatenated/included this way can be themselves static or prerendered, or rendered on-demand.
Actually the above-mentioned feature of Eleventy where parts of a page are server-rendered or prerendered and other parts are rendered at the edge is very similar to those includes as well.

Migrating from Jekyll to Eleventy

Thomas Broyer — Mon, 13 Mar 2023 01:12:34 +0000

Yes, this is going to be yet another one of those articles explaining how I migrated my blog from Jekyll to Eleventy. You've been warned.

Why?

I don't really have issues with Jekyll and I've been using it for 10 years now here, but I haven't really chosen Jekyll: it's been more-or-less imposed on me by GitHub Pages.
But GitHub now has added the possibility to deploy using a custom GitHub Actions workflow, and this is game-changer!

I could have kept using Jekyll with unlocked possibilities, but I'm not a Rubyist, that's just not a language I'm comfortable with, and I know almost nothing about Gems, so definitely not something I'd be comfortable maintaining going forward.

I also could have just kept using the built-in Jekyll Pages integration, and this is what I would have done if I hadn't found any satisfying alternative. I'm not forced to change, so at least I have a fallback in the form of the status quo.

So what would replace it? Let's evaluate my requirements.

The Requirements

I have articles written in HTML (exports from Posterous) and Markdown, using a bit of Liquid to link to other articles (with the post_url Jekyll tag). The Markdown articles use GitHub Flavored Markdown, including syntax-highlighted fenced code blocks, with embedded HTML. Ideally I shouldn't have to update the articles at all.
I only have 4 templates only (index.html, rss.xml, and default and post layouts) so migrating to another templating engine wouldn't really be a problem. The index.html uses pagination (even though I still only have a single page). The default layout builds a Content Security Policy using flags from the articles' front matter.
I also have a few static files: CSS, JS, and images (and a file to verify ownership for the Google Search Console).
Of course, because cool URIs don't change, the permalinks have to be ported to the new solution.
I hadn't identified it at first, but I actually have an old article that's not published, through Jekyll's published: false in the front matter. In the worst case, I'd just delete it (it'd still be there in the Git history).
Nice to have: I kinda like Jekyll's _drafts folder using the file's last modified date, and _posts folder with the publication date as part of the file name. (I don't commit my drafts, and yes that means I don't have backups; I don't have many drafts, and I'll probably never finish and publish them so 🤷)
Of course I want something I'm comfortable using for the next 10 years, in terms of technology and ecosystem. This means essentially that I'd like a Node-based solution.
Last, but not least, I want the output to be (almost) identical (for now at least) to the Jekyll site: must be static HTML, with <script>s added by the layouts and possibly right from the articles, no client-side hydration and upgrading to a Single Page Application.

The choice

The HTML-first approach rules out (a priori, correct me if I'm wrong) every React or Vue based approach, or similar.

I've quickly evaluated a couple alternatives, namely Astro and Eleventy.

Astro is fun, but I must say it doesn't really look content oriented, relegating the content into its src/pages, or worse, a subfolder inside src/content/.
I really like the typesafe nature of content collections, but moving everything down to src/content/blog really hides the content away IMO.
Extracting the publication date from the file name is possible, but it looks more and more like a development project rather than a content project.
It's great, but not what I'm looking for here.

I then looked at Eleventy. I have to admit my first contacts with the Eleventy documentation months ago left me with a bitter taste as I couldn't really figure out how collections worked and how you were supposed (or not) to organize your files. Looking at tweetback more recently didn't really help: absolutely everything is JS, loading content from a SQLite database.

I decided to give it a chance: maybe I misunderstood the documentation the last time(s) I read it.
And indeed it was the case: moving from Jekyll to Eleventy probably couldn't be easier.

How?

I felt my way a bit, so I'll summarize here what I ended up doing, also describing some things I tried along the way.

Getting Started

Removing Jekyll consists in deleting the _config.yml and possibly Gemfile (I didn't have one).
Adding Eleventy means initializing a new NPM packaging and adding the @11ty/eleventy dependency (and of course adding node_modules to the .gitignore), and creating a configuration file (I chose eleventy.config.cjs rather than the .eleventy.js hidden file).

Because the deployment workflow is different, the CNAME file becomes useless and can be deleted.
A new GitHub Actions workflow also has to be created, using the actions/configure-pages, actions/upload-pages-artifact, and actions/deploy-pages actions. I took inspiration from the Astro starter workflow and updated it for Eleventy.

Markdown

Eleventy supports Markdown out of the box, with all the options I needed, except syntax highlighting and heading anchors for deep linking.
It also automatically extracts the date from the file name.

Syntax highlighting is as easy as using the official plugin, but then the generated HTML markup is different than with the Rouge highlighter in Jekyll, so I had to change the CSS accordingly.
I ended up importing an existing theme: display would be slightly different than before, but actually probably better looking.

Deep linking requires using the markdown-it-anchor plugin, and to make sure existing deep links wouldn't break I provided my own slugify function mimicking the way CommonMarkGhPages computes the slug from the heading text (I happen to have a few headings with <code> in them, and CommonMarkGhPages would compute the slug from the rendered HTML leading to things like codejavaccode; I chose to break those few links in favor of better-looking anchor slugs).
I also disabled tabIndex to keep the same rendering as previously (I'll read more on the accessibility implications and possibly revert that choice later.)

I reimplemented the post_url first as a custom short code but that meant updating all articles to quote the argument (due to how Eleventy wires things up), so I ended up using a custom tag; that's specific to the Liquid template engine (in case I would want to change later on) but at least I don't have to update the articles.

In terms of rendering, besides syntax highlighting, the only difference is the <br> which are now rendered that way rather than <br /> (there's an option in markdown-it but I'll keep the less XHTML-y, more HTML-y syntax).

The rss.xml file wouldn't be treated as a template by default, so I aliased the xml extension to the Liquid engine, and added an explicit permalink: to avoid Eleventy creating an rss.xml/index.html file.
I did the same with the css extension so I could use an include to bring in the syntax-highlighting theme in my style.css.

Liquid Templating

I had to rename my layout files to use a .liquid extension rather than .html.
I didn't want to move them though, so I configured a layouts directory instead.

I also had to handle all the Jekyll-specific things I was using: xml_escape, date_to_xmlschema, date_to_string, and date_to_long_string filters, and the site.time and site.github.url variables (we already handled the post_url tag above).

At first, I tried to recreate them in Eleventy (which is easy with custom shortcodes and global data files), but finally decided that I could replace most with more standard Liquid that would be compatible right-away with LiquidJS: xml_escape becomes escape, date_* become date: with the appropriate format (this made it possible to fix my <time> elements erroneously including the time), and site.time becomes "now" or "today" with the date filter.
I put that in a separate commit as that's compatible with Jekyll Liquid as well.
And all that's left is therefore site.github.url that can be put in a global data file (a JS file getting the value out of an environment variable, fed by the actions/configure-pages output in the GitHub Actions workflow).

Finally, I actually had to update all templates to use Eleventy's way of handling pagination, and looping over collections.

Speaking of collections, I initially used directory data files to assign a post tag to all posts in _posts and _drafts.
This didn't handle the published: false, so I used a custom collection in the configuration file instead.
I probably could have also used a computed eleventyExcludeFromCollections to exclude it, but this also helped fix an issue with the sort order and apparently a bug in LiquidJS's for loop with both reversed and limit: where it would limit before reversing whichever way I wrote things, contrary to what the doc says.

One last change I made: update the Content Security Policy to account for the Eleventy dev mode autoreload; I used eleventy.env.runMode != "build" to detect when run with autoreload.

Static Files

Contrary to Jekyll where any file without front matter is simply copied, static files have to be explicitly declared with Eleventy.
I also had to ignore those HTML files I needed to just copy without processing.

Permalinks

Permalinks for the rss.xml and style.css are defined right in those files' front matter.
The index.html uses pagination so I declared a mapping there as well.

Finally I decided to compute the permalink for posts right in the front matter of the post layout, using the page.fileSlug gives me exactly what I want (the date part has already been removed by Eleventy).
Using a JS front matter allowed me to filter out the published: false article so it wouldn't ever be rendered to disk (I already excluded it from the posts collection, but Eleventy would still process and render it).

Drafts

To handle drafts, I'm using the getFilteredByGlob function when declaring the posts collection, so I can decide whether to include the _drafts folder depending on an environment variable.
This would include the drafts in the posts collection so they would appear in the index.html and rss.xml.

More importantly though, when not including drafts, I have to ignore the _drafts folder, otherwise the drafts are still processed and generated (despite not being linked to as they don't appear in the posts collection).
This is actually not really a problem given that I don't commit drafts to my Git repository, so I would observe this behavior only locally.

Comparing the results

To make sure the output was identical to the Jekyll-based version, I built the site once with Jekyll before any modification and backed up the _site folder; then compared it with the output of Eleventy to make sure everything was OK.

Conclusion

As I felt my way and learned about Eleventy, this took me nearly two weekends to complete (not full time, don't worry!)
What took me the most time actually was probably finding (and deciding on) the new syntax-highlighting theme!
Otherwise, things went really smoothly.

I'm very happy with the outcome, so I switched over.
And now that I control the build workflow, I know I could setup an asset pipeline, minify the generated HTML, bring in more Eleventy plugins to split the syntax-highlighting theme out and only send it when there's a code block on the page, etc.

A big would recommend!

The benefits of Web Component Libraries

Thomas Broyer — Mon, 27 Feb 2023 17:27:07 +0000

Web component browser APIs aren't that many, and not that hard to grasp (if you don't know about them, have a look at Google's Learn HTML section and MDN's Web Components guide);
but creating a web component actually requires taking care of many small things.
This is where web component libraries come in very handy, freeing us of having to think about some of those things by taking care of them for us.
Most of the things I'll mention here are handled one way of another by other libraries (GitHub's Catalyst, Haunted, Hybrids, Salesforce's LWC, Slim.JS, Ionic's Stencil) but I'll focus on Google's Lit and Microsoft's FAST here as they probably are the most used web component libraries out there (ok, I lied, Lit definitely is, FAST not that much, far behind Lit and Stencil; but Lit and FAST have many things in common, starting with the fact that they are just native web components, contrary to Stencil that compiles to a web component).
Both Lit and FAST leverage TypeScript decorators to simplify the code even further so I'll use that in examples,
even though they can also be used in pure JS (decorators are coming to JS soon BTW). I'll also leave the most apparent yet most complex aspect for the end.

Let's dive in!

Registration

It's a small detail, and I wouldn't even consider it a benefit;
it's mostly syntactic sugar, but with FAST at least it also does a few other things that we'll see later: registering the custom element.

In vanilla JS, it goes like this:

class MyElement extends HTMLElement {}

customElements.define('my-element', MyElement);

With Lit:

@customElement('my-element')
class MyElement extends LitElement {}

And with FAST:

@customElement({
    name: 'my-element',
    // other properties will come here later
})
class MyElement extends FASTElement {}

Attributes and Properties

With native HTML elements, we're used to accessing attribute (also known as content attributes in the HTML spec) values as properties (aka IDL attributes) on the DOM element (think id, name, checked, disabled, tabIndex, etc. even className and htmlFor although they use sligthly different names to avoid conflicting with JS keywords) with sometimes some specificities: the value attribute of <input> elements is accessible through the defaultValue property, the value property giving access to the actual value of the element (along with valueAsDate and valueAsNumber that additionally convert the value).

Custom elements have to implement this themselves if they want it, and web component libraries make it a breeze.
They help us reflect properties to attributes when they are modified (if that's what we want; and all while avoiding infinite loops), convert attribute values (always strings) to/from property values, or handle boolean attributes (where we're only interested in their absence or presence, not their actual value: think checked and disabled).

Let's compare some of these cases, first without library:

class MyElement extends HTMLElement {
    // Attributes have to be explicitly observed
    static get observedAttributes() {
        return [ 'reflected-converted', 'reflected', 'non-reflected', 'bool' ];
    }

    #reflectedConverted;

    // In a real element, you'd probably use a getter and setter
    // to somehow update the element when the property is set.
    nonReflected;

    attributeChangedCallback(name, oldValue, newValue) {
        switch (name) {
        case 'reflected-converted':
            // Convert the attribute value
            this.reflectedConverted = newValue != null ? Number(newValue) : null;
            break;
        case 'non-reflected':
            this.nonReflected = newValue;
            break;
        // other attributes handled in accessors below
        }
    }

    get reflectedConverted() {
        return this.#reflectedConverted;
    }
    set reflectedConverted(newValue) {
        // avoid infinite loop with attributeChangedCallback
        if (newValue !== this.#reflectedConverted) {
            this.#reflectedConverted = newValue;
            if (newValue == null) {
                this.removeAttribute('reflected-converted');
            } else {
                // Here we let the browser automatically convert to string
                this.setAttribute('reflected-converted', newValue);
            }
        }
    }

    get reflected() {
        return this.getAttribute('reflected');
    }
    set reflected(newValue) {
        if (newValue == null) {
            this.removeAttribute('reflected');
        } else {
            this.setAttribute('reflected', newValue);
        }
    }

    get bool() {
        return this.hasAttribute('bool');
    }
    set bool(newValue) {
        this.toggleAttribute('bool', newValue);
    }
}

Now with Lit:

class MyElement extends LitElement {
    @property({ attribute: 'reflected-converted', type: Number, reflect: true})
    reflectedConverted?: number;

    @property({ reflect: true })
    reflected?: string;

    @property({ attribute: 'non-reflected' })
    nonReflected?: string;

    @property({ type: Boolean })
    bool: boolean = false;
}

And with FAST:

class MyElement extends FASTElement {
    @attr({ attribute: 'reflected-converted', converter: nullableNumberConverter })
    reflectedConverted?: number;

    @attr reflected?: string;

    @attr({ attribute 'non-reflected', mode: 'fromView' })
    nonReflected?: string;

    @attr({ mode: 'boolean' })
    bool: boolean = false;
}

Early-initialized properties

Another thing with properties is that they could be set on a DOM element even before it's upgraded:
the script that defines the custom element does not need to be loaded by the time the browser parses the custom tag in the HTML,
and some script might access that element in the DOM before the script that defines it has loaded;
only then will the element be upgraded: the class instantiated to take control of the custom element.

When that happens, you wouldn't want properties that would have been set on the element earlier to be overwritten by the upgrade process.

Without a library, you would have to take care of it yourself with code similar to the following:

class MyElement extends HTMLElement {
    constructor() {
        super();
        // "upgrade" properties
        for (const propName of ['reflectedConverted', 'reflected', 'nonReflected', 'bool']) {
            if (this.hasOwnProperty(propName)) {
                let value = this[propName];
                delete this[propName];
                this[propName] = value;
            }
        }
    }
}

Again, libraries do that for you, automatically, based on the previously seen declarations.

Responding to property changes

The common way to respond to property changes is to implement a setter. This requires also implementing a getter though, as well as storing the value in a private field. When changing the value from attributeChangedCallback, make sure to also use the setter and not assign directly to the backing field.

To respond to changes to the nonReflected property in the above example, one would have to write it like so:

#nonReflected;

get nonReflected() {
    return this.#nonReflected;
}
set nonReflected(newValue) {
    this.#nonReflected = newValue;
    // respond to change here
}

Both Lit and FAST provide their own way of doing this, though most of the time this is not really needed given that most reaction to change is to update the shadow tree, and Lit and FAST have their own ways of doing this (see below for more about rendering).

With Lit, you listen to changes to any property and have to tell them apart by name, a bit similar to attributeChangedCallback but batched for several properties at a time:

@property({ attribute: 'non-reflected' })
nonReflected?: string;

// You could also use updated(changedProperties), depending on your needs
willUpdate(changedProperties: PropertyValues<this>) {
    if (changedProperties.has("nonReflected")) {
        // respond to change here
    }
}

With FAST, you can implement a method with a Changed suffix appended to the property name:

@attr({ attribute 'non-reflected', mode: 'fromView' })
nonReflected?: string;

nonReflectedChanged(oldValue?: string, newValue?: string) {
    // respond to change here
}

Shadow DOM and CSS stylesheets

The most efficient way to manage CSS stylesheets in Shadow DOM is to use so-called constructable stylesheets: construct a CSSStyleSheet instance once (or soon import a CSS file), then reuse it in each element's shadow tree through adoptedStyleSheets:

const sheet = new CSSStyleSheet();
sheet.replaceSync(`
    :host { display: block; }
    :host([hidden]) { display: none; }
`);

class MyElement extends HTMLElement {
    constructor() {
        super();
        this.attachShadow({ mode: 'open' });
        this.shadowRoot.adoptedStyleSheets = [ sheet ];
    }
}

With Lit, you'd rather use this more declarative syntax:

class MyElement extends HTMLElement {
    static styles = css`
        :host { display: block; }
        :host([hidden]) { display: none; }
    `;
}

and similarly with FAST:

const styles = css`
    :host { display: block; }
    :host([hidden]) { display: none; }
`;

@customElement({
    name: 'my-element',
    styles,
})
class MyElement extends FASTElement {}

Constructable stylesheets currently still require a polyfill in Safari though (this is being added in Safari 16.4), but both Lit and FAST take care of this for you.

Rendering and templating

The most efficient way to populate the shadow tree of a custom element is by cloning a template that has been initialized once.
That template could be any document fragment but the <template> element was made specifically for these use-cases.
You would then retrieve nodes inside the shadow tree to add event listeners and/or manipulate it in response to those inside events or to attribute and property changes (see above) from the outside.

const template = document.createElement('template');
template.innerHTML = `
    <button>Add</button>
    <output></output>
`;

class MyElement extends HTMLElement {
    #output;

    constructor() {
        super();
        // … (upgrade properties as seen above) …
        this.attachShadow({ model: 'open' });
        this.shadowRoot.append(template.content.cloneNode(true));
        // Using an <output> element makes it easier
        // We could also create a text node and append it ourselves
        this.#output = this.shadowRoot.querySelector('output');
        this.shadowRoot.querySelector('button').addEventListener('click', () => this.count++);
        this.#output.value = this.count;
    }

    // … (count property, with attribute changed callback and converter) …
    set count(value) {
        // … (reflect to attribute or whatever) …
        this.#output.value = value;
    }
}

customElements.define('my-element', MyElement);

Things get much more complex when you want to conditionally render some subtrees (the easiest probably being to toggle their hidden attribute), or render and update a list of elements.

This is where Lit and FAST (and a bunch of other libraries) work much differently from the above, introducing the concept of reactive or observable properties and a render lifecycle based on a specific syntax for templates allowing placeholders for dynamic values, a syntax to register event listeners right from the template, and composability.

With Lit, that could look like:

@customElement('my-element')
class MyElement extends LitElement {
    @property count: number = 0;

    render() {
        // No need for an <output> element here, though we could
        // (and it would possibly even be better for accessibility)
        return html`
            <button @click=${this.#increment}>Add</button>
            ${this.count}
        `;
    }

    #increment() {
        this.count++;
    }
}

and with FAST:

// No need for an <output> element here, though we could
// (and it would possibly even be better for accessibility)
const template = html<MyElement>`
    <button @click=${x => x.count++}>Add</button>
    ${x => x.count}
`;

@customElement({
    name: 'my-element',
    template,
})
class MyElement extends FASTElement {
    @attr({ converter: numberConverter }) count: number = 0;
}

The way Lit and FAST work is by observing changes to the properties and scheduling an update everytime it happens.

With Lit, the update (also called rerender) will call the render() method of the component and then process the template. The rerender is scheduled using a microtask such that it can batch changes to multiple properties into a single rerender.

With FAST, the update is instead scheduled using requestAnimationFrame (achieving the same batching as Lit) and will call every lambda of the template that needs to be: FAST tracks which dynamic part uses which properties to only reevaluate those parts when a given property changes.

In the examples above, any change to the count property, either from the outside or in response to the click of the button, schedules a update.
And in FAST's case, only the x => x.count lambda is called and the corresponding DOM node updated.
In Lit's case, the button's click listener would also be evaluated, but determined to be the same as before so no change would be performed.

The html tagged template literal (both in Lit and FAST, also in other libraries such as @github/jtml) will use a <template> under the hood to parse the HTML once and reuse it later, just like the css seen earlier uses a constructable stylesheet. It puts markers in the HTML (special comments, elements or attributes) in place of the dynamic parts so it can find them back to attach event listeners and inject values, making it possible to surgically update only the nodes that need it, without touching anything else (FAST actually being even more surgical by tracking the properties used in each dynamic part).

With Lit's render() method returning such a template and called each time a property or internal state changed, its programming model looks a bit like React, rerendering and returning a new JSX each time a prop or state changed;
while FAST's approach looks a bit more like Angular (or Vue or Svelte) where each component is associated with a single template at definition time.

Other niceties

Lit and FAST also provide some helpers to peek into the shadow tree: to get a direct reference to some node, or the <slot>s' assigned nodes.

Lit also pioneers reactive controllers that allow code reuse between components through composition, where the controller can hook into the render lifecycle (i.e. trigger a rerender of the component that uses the controller).
The goal is ultimately to make them reusable across frameworks too.
Some have already embraced it, like Haunted with its useController() that allows using reactive controllers in Haunted components, or Apollo Elements that's built around reactive controllers. Lit also provides a useController() React hook as part of its @lit-labs/react package (that also makes it easier to use a Lit component in a React application by wrapping it as a React component), and there are prototypes for several frameworks such as Angular or Svelte, or even native web components through a mixin.
FAST developers are interested in supporting reactive controllers too, but those currently don't quite match with the way FAST updates the rendering.

The Lit team provides reactive controllers to manage contextual values, easily wire asynchronous tasks to rendering, wrap a bunch of native observers (mutation, resize, intersection, performance), or handle routing. Others are embracing them too: Apollo Elements for GraphQL already cited above; James Garbutt has a collection of utilities for Lit, many of them being reactive controllers usable outside Lit; Nano Stores provide reactive controllers; Guillem Cordoba has controllers for Svelte Stores; etc.

Conclusion

Web component libraries are really helpful to streamline the development of your components.
While you could develop custom elements without library, chances are you'd be eventually creating your own set of helpers, reinventing the wheel (there are more than enough ways to build web components already).
Understand what each library brings and pick one.

The Javax to Jakarta mess, it's even worse than I thought

Thomas Broyer — Fri, 22 Apr 2022 15:52:36 +0000

In the previous post, I described how the Javax to Jakarta migration was a mess, but doing more research on the subject I discovered that it's actually way worse than that.

EDIT(2022-11-18): There's a new/updated Gradle plugin to help us, and Spring 6 uses Jakarta EE 9 as a baseline. Continue reading for details.

EDIT(2023-05-15): Guice has made the switch to jakarta in version 7, with a version 6 that stays on javax but also supports jakarta.inject to help in the transition.

Wait, how could it be worse‽

Until now, I focused on APIs, but what I forgot about were implementations. I didn't really forgot about them, as they were partly what led me to do the research to begin with, but I forgot about how you may want to have implementations for both Java EE and Jakarta EE 9+ at the same time.

The case that got me looking at the subject was a combination of Resteasy, jOOQ, and Sentry-Java. Resteasy and Sentry-Java both require a Servlet implementation (I'm using an embedded Jetty server, but I could have picked Tomcat or Undertow). Resteasy and jOOQ both depend on XML Binding. Resteasy and jOOQ (and Pac4j and Jackson, that I also use) actually maintain two parallel versions, comptible with either Java EE 8 or Jakarta EE 9 (and in the case of jOOQ at least, they're not really parallel, it's more that the previous version is kept perfused with all the bugfixes being backported, but the featureset of the Java EE 8 compatible version is not the same as of the Jakarta EE 9 compatible one). Sentry-Java on the other side is only compatible with the javax. flavor of Servlets. What this means is that I had to choose between using the older versions of Resteasy and jOOQ, or forking Sentry-Java to bring it to Jakarta EE 9 (this is what I ended up doing, as it's only 3 classes). And of course I discovered the problem totally by accident, looking the results of a ./gradlew dependencies where I had the older version of Resteasy but the latest version of jOOQ, and noting that the jakarta.xml.bind:jakarta.xml.bind-api transitive dependency of Resteasy was being upgraded to 3.0.0 because of jOOQ.

If it was only a matter of waiting for everyone to provide a Jakarta EE compatible version, it could possibly be workable, although it would take years and some companies don't have incentives in such upgrades (anyone knows when ~~Guice, or~~ Dagger, will move to jakarta.inject? fortunately I don't use anything that depends on Dependency Injection, so I can live very well with javax.inject alongside everything jakarta.*, and that's indeed what I'm currently doing. UPDATE(2023-05-15): Guice made the switch in version 7, so I can finally move on).

No, what I had totally forgotten actually were reference implementations, and specifically (in my case) for Mail. Many EE APIs are frameworks (Servlets, REST) where you have to pick one flavor and one implementation anyway: you wouldn't use Java EE Servlets within a Jakarta EE Servlet container, just like you wouldn't use Vert.x handlers in such container either, or a Spring Web resource within Resteasy or Jersey, and you wouldn't use Resteasy and Jersey at the same time either. But Mail is different, it's a library that you use to send mails, and because it knows how to parse multipart content, it can be used transitively by other libraries that you'll depend on (honestly, I don't think that will be the case for me, fortunately).

So you may want or need to use both JavaMail and Jakarta Mail at the same time. And it happens that this is not possible, because Jakarta EE decided to keep using the com.sun.mail.* package names while migrating to Jakarta EE 9 APIs. Note that those exact same class names are actually published at separate Maven coordinates, similar to the Java EE 8 vs. Jakarta EE 8, but this time it extends to the whole Jakarta EE irrespective of the version (and to make the matter worse, they changed the Maven coordinates again for Jakarta EE 10: from com.sun.mail:javax.mail to com.sun.mail:jakarta.mail to org.eclipse.angus:jakarta.mail ; oh, and they reset the versioning scheme, so you also have to somehow guess that Angus Mail 1.0.0 is Jakarta Mail 2.1)

It's as if everything was deliberately made to make the migration as painful as possible, and specifically make it impossible (or at the very least make zero effort to make it possible) to have both Java EE and Jakarta EE in the same project, for no technical reason (besides we'll only have to search/replace javax. with jakarta., and nothing else, which you'll convene is rather weak an argument).

Another thing I had totally forgotten, was Java EE's tradition to publish API jars that are only good to compile against, and then reference implementations that also contain the API classes, but this time with actual code in the methods (this is also why you have EE vendor flavors of them by the way, because they have to change one constant somewhere to point to their actual implementation class).

This means that com.sun.mail:javax.mail contains the same classes as javax.mail:javax.mail-api. And because they apparently always have to complicate things, this JAR is also a bundle of mailapi (that still also contains the API classes), smtp, imap and pop3. Jakarta EE 9 followed the exact same layout, except for the javax to jakarta renaming (but keeping the com.sun.mail package names though, remember?) Angus Mail (the Jakarta EE 10 reference implementation) doesn't depart from that tradition and also provides org.eclipse.angus:jakarta.mail as a bundle of both org.eclipse.angus:angus-mail and jakarta.mail:jakarta.mail-api (but this time it seems like jakarta.mail:jakarta.mail-api contains normal classes, not a version stripped out of the methods' code to keep only the ABI), and org.eclipse.angus:angus-mail is a bundle of angus-core, image, smtp, pop3, and logging-mailhandler.

It's also been a tradition to have JARs bundling all the EE APIs together: javax.javaee-web-api and javax:javaee-api, respectively jakarta.platform:jakartaee-web-api and jakarta.platform:jakartaee-api (fortunately, they now also publish a jakarta.platform:jakartaee-bom that simply references the other Maven artifacts rather than bundling them in a fat jar).

Don't get me wrong, it's OK to build such JARs for people who don't use Maven or Maven-compatible dependency resolvers ; what is not OK is to publish them to the Central Repository with their own Maven coordinates (did you keep count of how many JARs contained javax.mail.* or jakarta.mail.* classes and how many contained com.sun.mail.* classes?)

OK, so it's indeed way worse, but we have Gradle our savior, right?

Gradle can help indeed: we can teach it to fail the build if we ever have conflicting JARs in our classpaths, but there are cases that don't have a clean solution besides picking one side and rewriting everything to either javax or jakarta.

Detecting more conflicts

Using the same kind of component metadata rules as in our previous installment, we'd be able to teach Gradle that:

javax.mail:javax.mail-api, com.sun.mail:javax.mail, and com.sun.mail:mailapi (in version 1) are incompatible as they all contain javax.mail.* classes
com.sun.mail:javax.mail is incompatible with com.sun.mail:mailapi, com.sun.mail:smtp, com.sun.mail:imap, and com.sun.mail:pop3 as it's a bundle of those 4 JARs (it provides all the same capabilities than each one of them taken individually), same for com.sun.mail:jakarta.mail
jakarta.mail:jakarta.mail-api, com.sun.mail:jakarta.mail, com.sun.mail:mailapi (in version 2), and org.eclipse.angus:jakarta.mail are incompatible as they all contain jakarta.mail.* classes (and note that Angus Mail doesn't use the same versioning scheme, because it wasn't complicated enough)
org.eclipse.angus:jakarta.mail and org.eclipse.angus:angus-mail are incompatible with their subsets
org.eclipse.angus artifacts are incompatible with their com.sun.mail counterparts (and again beware of the new versioning scheme)
com.sun.mail artifacts in version 1 shouldn't be upgraded to version 2

(and I hope I haven't missed more cases‼)

Rewriting things?

There are tools to rewrite JARs and classes, the most complete probably being Eclipse Transformer, that is apparently used by the Payara application server to rewrite EARs dynamically at deployment time.

There's a Gradle plugin by Hibernate, but it's not much documented and it looks like Hibernate didn't even use it in their migration. The plugin seems to be tailored to creating JARs to be deployed (historically, the *-jakarta Hibernate JARS, even though there were made without the plugin as far as I can tell), not for rewriting those dependencies that you might be using.

To rewrite your dependencies, you could theoretically use an artifact transform. Registering it would probably require either identifying the dependencies that need rewriting (by way of attributes), or apply it to each and every dependency (by adding said attribute to all variants of all components). That mess was decided years ago, we should be passed the point where someone already packaged it so you only have to apply one Gradle plugin and it Just Works™, but the community seems to have decided that we should all suffer this mess and wait for every library you depend on to have migrated, and in the mean time be locked with older versions, possibly unmaintained, of your other dependencies.

Note that this rewriting wouldn't magically solve all your problems: you'd still have to have capabilities rule to prevent having several components with different Maven coordinates provide the same package names (including after the rewriting), but this time maybe resolve the conflicts automatically to pick the highest Jakarta EE version.

So what now?

Honestly, I'm fed up.

Maybe I'll try to make a plugin with all these rules (or contribute them to Jendrik Johannes' [Java Ecosystem Capabilities Gradle Plugin](https://github.com/jjohannes/java-ecosystem-capabilities)) so at least I can verify that I don't have issues. If anyone would like to help create a list of all the conflicts (including the vendor libraries), get in touch (but I don't promise anything).

UPDATE(2022-11-18): the GradleX plugin now has most of those rules, at least for the official Java EE/Jakarta EE artifacts, i.e. not the Jetty, Tomcat or Glassfish flavors. See also this comment for current limitations; specifically it won't downgrade Jakarta EE 8 to Java EE 8, so Jakarta EE 9 dependencies might upgrade them and break things at runtime.

But for the rest, the best thing to do is probably to poke at project maintainers so they do the upgrade and/or provide parallel flavors (possibly helped by the Eclipse Transformer, and by yourself: please don't be assholes with open source maintainers, lend them a hand or sponsor them).

UPDATE(2022-11-18): Spring Framework 6 has been released that uses Jakarta EE 9 as a baseline. This will undoubtedly drive adoption of the jakarta.* namespace, but not all projects have an interest in such combination (e.g. ~~Guice [tracking issue] and~~ Dagger [tracking issue] will likely stay with javax.inject for a good while~~, and Guice Servlets with javax.servlet [tracking issue]~~).

UPDATE(2023-05-15): Guice made the switch to jakarta.inject, jakarta.servlet, and jakarta.persistence in version 7, and simultaneously published version 6 that's still on javax but also supports jakarta.inject, to help in the transition. Note that to support both namespaces, Guice 6 must depend on the javax.inject:javax.inject:1 artifact, which makes it incompatible with JPMS.

And maybe in the future I won't use "standard APIs" as often as I used to: I'd rather have libraries that don't know how to talk to each other, and write some glue code, than libraries you cannot even put in the same classpath. So maybe I'll try alternatives to Servlets and/or Jakarta RS, trying to minimize my dependency on them through clear segregation (already what I'm doing mostly, where the Jakarta RS endpoints only translate the HTTP request to a business-oriented service, and translate the result back to an HTTP response), such that rewriting the Web layer/adapter would indeed be costly, but entirely doable. I'm glad I never actually tried to use javax.json for instance, similar to how I already ditched javax.ws.rs.client for OkHttp a few years ago.

Theoretically, I could also embrace the Java Module System (JPMS), as it's good to detect duplicate packages and missing dependencies (e.g. when Jakarta EE 8 is upgraded to Jakarta EE 9, assuming a change in module name), but it's a whole other mess as it adds yet another naming scheme. Just as an example, the JBoss version of Jakarta RS 3.0 is still using the java.ws.rs module name rather than jakarta.ws.rs, and Jetty's version of Jakarta Servlet 5.0 is using jetty.servlet.api rather than jakarta.servlet, Jakarta EE 8 dependencies themselves have sometimes switched module name during patch releases; this means that you cannot just swap one JAR for another as the JVM will then complain that some requires is not fulfilled. Not to mention that many JARs still aren't compatible with JPMS, with not even an Automatic-Module-Name: Resteasy 6 and Sentry-Java for instance, and some Java EE dependencies too (e.g. javax.inject:javax.inject) so downgrading Jakarta EE 8 to Java EE 8 is not without problems either. That would probably deserve a third post, but for my sanity I'd rather wait a couple years 😁

The Javax to Jakarta mess, and a Gradle solution

Thomas Broyer — Mon, 18 Apr 2022 19:09:22 +0000

EDIT(2022-11-18): There's a new Gradle plugin to help us. Continue reading for details.

Nearly five years ago, Oracle was preparing the release of Java EE 8 and announced that it would move it to an open source foundation. Less a month later, they announced they selected the Eclipse Foundation for that work. Two years later, Jakarta EE 8 was released as fully compatible version of Java EE 8. The only thing that changed in that period was about migrating the process to the Eclipse Foundation and Jakarta EE Working Group.

According to this article, the source code was exactly the same, except for some additional commits maybe because Oracle transfered the head of the master branches of Git repositories, and the artifacts got released twice as milestones of the transfer process: first proving that the code could be built, then a less technical but more procedural release where the Java EE name was replaced by Jakarta EE in javadocs (of course this also means different terms and conditions).

Then fifty months after that, Jakarta EE 9 was released with only two major changes:

the package names were all migrated from javax.* to jakarta.*
some older specifications were pruned

But you talked about a mess‽

Yes, Oracle being Oracle, they transfered the technology and documentation, but not the name and trademark. Indeed, Java EE was renamed to Jakarta EE. But that's not all, they also prohibited any modification to the javax.* packages, so everything would eventually be moved new packages. The Eclipse Foundation presented it as what Eclipse and Oracle had agreed on but let's not be fooled by that PR wording: what would you expect from the company that almost ruined our whole industry with the trial against Google over Android?

That's only part of the problem though. We could have had javax.* artifacts (I'm talking about Maven coordinates here) using the javax.* package names, and jakarta.* artifacts using the jakarta.* package names, and while this big breaking change would have had a years-long impact, it would have been somewhat manageable (we're right into it now actually, with many projects maintaining two branches, one for each package namespace).

But the Jakarta Working Group decided to publish Jakarta EE 8 under the same Maven coordinates as what they expected to publish later Jakarta EE versions. I have no idea if they were somehow forced to publish those at all (or could have possibly kept them into their own repository), but they could have at least used specific Maven coordinates, as they already knew at that point that this would happen: Oracle froze the javax.* package name in May 2019, while Jakarta EE 8 came out four months later in September 2019. I have no idea how much Jakarta Working Group members were aware of this decision by Square, Inc. to version their Maven coordinates and package names when releasing versions with major breaking changes, back in December 2015 (but the problem itself was nothing new); but the decision to publish javax.* and later jakarta.* packages under the same Maven coordinates was hugely misguided, and possibly the worst mistake in all this story.

What does this mean in practice?

First, when there was only Jakarta EE 8, you could have had (transitive) dependencies on both Java EE 8 (or earlier) and Jakarta EE 8. This would cause duplicates of the javax.* classes in the classpath, because dependency managers aren't told that those artifacts are actually the same but renamed. If you had an older Java EE artifact on the classpath, it could also shadow the newer Jakarta EE, causing breakages at compile-time or, worse, at runtime.

But now that there's also Jakarta EE 9 (and 9.1, and very soon Jakarta EE 10), you could have Jakarta EE 8 artifacts being upgraded to Jakarta EE 9 despite being a completely incompatible API.

In their newsletter just before the Jakarta EE 9 release, Eclipse acknowledged the practical issues it caused, inviting people to actually depend on Java EE 8 artifacts rather than Jakarta EE 8 ones, but this came way too late, the harm had been done already.

In retrospect, we could say that nobody should have dependended upon Jakarta EE 8 artifacts, or they should have used version ranges to exclude the next major version (but version ranges are bad for build reproducibility, unless you use a dependency manager that somehow supports version locking).

Oh wow! Ok, but you hinted at a Gradle solution?

Yes, this is where Gradle really shines compared to many other dependency managers: it lets you hook into the dependency resolution process and fix many things.

UPDATE(2022-11-18): there's a new plugin in town that applies some of the solutions mentionned below (see this comment for the limitations; specifically it won't downgrade Jakarta EE 8 to Java EE 8, so Jakarta EE 9 dependencies might upgrade them and break things at runtime)

With Maven for instance, you could use the Maven Enforcer Plugin with Mojohaus "ban duplicate classes" rule to detect the Java EE vs Jakarta EE 8 issue, or the built-in "dependency convergence" rule and you would have to resolve it yourself through dependency exclusions everywhere needed, (and because the "dependency convergence" rule isn't configurable, you cannot enable it only for the Jakarta EE dependencies, so it has a huge impact). By the way, if you look at how the "dependency convergence" rule is implemented, you'll see that it will actually resolve the dependency once again so it can get to the dependency details, I don't think you can peek into, and influence the dependency resolution process itself.

With Gradle, we can do two things:

declare that Java EE and Jakarta EE 8 are actually the same thing under a different name
fix third-parties' dependencies on Jakarta EE 8 to reject any upgrade to Jakarta EE 9+ (Gradle also has the equivalent to Maven's "dependency convergence" rule through failOnVersionConflict())

This would fail the build the same way as with Maven, but Gradle also allows us to automatically fix those: because Jakarta EE 8 artifacts are fully compatible with Java EE 8 ones, we could rewrite third-parties' dependencies to actually use Java EE 8 rather than Jakarta EE 8.

Declaring that Java EE and Jakarta EE 8 are equivalent

To declare that Jakarta EE 8 replaced Java EE, we could use a module replacement rule, but this would also say that Jakarta EE 9 replaced Java EE, which is not actually true: there's no real problem having both Java EE 8 (javax.* package) and Jakarta EE 9 (jakarta.* package) in the same classpath.

So we'd rather declare that Jakarta EE 8 artifacts (and only those) provide the same capabilities as their Java EE counterparts.

For XML Binding for instance (which also changed name from JAXB), this would look something like this:

components {
    withModule("jakarta.xml.bind:jakarta.xml.bind-api") {
        if (id.version.startsWith("2.")) {
            allVariants {
                withCapabilities {
                    addCapability("javax.xml.bind", "jaxb-api", id.version)
                }
            }
        }
    }
}

With that rule,

you can have javax.xml.bind:jaxb-api in any version (Java EE) and jakarta.xml.bind:jakarta-xml.bind-api:3.0.0 or a later version (Jakarta EE 9+)) without error, but
you cannot have javax.xml.bind:jaxb-api and jakarta.xml.bind:jakarta-xml.bind-api:2.3.3 (Jakarta EE 8).

If we have such a conflict, Gradle will allow us to resolve it using a rule too, rather than having to play with exclusions which are a PITA to maintain in the long run. Let's first continue to detect the problems.

Reject upgrades of Jakarta EE 8 to Jakarta EE 9+

Rejecting such upgrades, within the same coordinates, is not as easy.

As we've seen above, we could retrospectively say that the dependencies should have originally been declared to reject them, but that's never the case in practice. Fortunately, Gradle allows us to fix those declarations at resolution time:

components {
    all {
        allVariants {
            withDependencies {
                val dep = find {
                    it.group == "jakarta.xml.bind" &&
                    it.name == "jakarta.xml.bind-api" &&
                    it.versionConstraint.includesMajor("2")
                }
                if (dep != null) {
                    dep.version { reject("[3,)") }
                }
            }
        }
    }
}

The includesMajor here would be a Kotlin extension function doing the check on the version, which has to deal with Gradle's rich versions including version ranges for instance. The simplest implementation would only look at the required version, which is what a simple version in a Maven POM would map to:

fun VersionConstraint.includesMajor(major: String) =
    requiredVersion.startsWith("${major}.")

Resolving the Java EE / Jakarta EE 8 conflicts

We've used a capability to make them incompatible with one another, but this will only fail the build if such a thing arise. We then have to use a capabilities resolution rule to select between them.

Because we declared the capability only on Jakarta EE 8, we can safely use selectHighestVersion() to pick the Jakarta EE 8.

Why safely? Because this is evaluated after versions are mediated. What this means is that if you have all three of javax.xml.bind:jaxb-api:2.3.1 (Java EE 8), jakarta.xml.bind:jakarta.xml.bind-api:2.3.3 (Jakarta EE 8), and jakarta.xml.bind:jakarta.xml.bind-api:3.0.0 (Jakarta EE 9), then Gradle will first upgrade the Jakarta EE dependency to Jakarta EE 9, which will remove Jakarta EE 8 from the equation, and at the same time the capability conflict, leaving the Java EE 8 and Jakarta EE 9. In this case, because Java EE 8 and Jakarta EE 8 are fully compatible, this is not a problem at all; it would be though if we had a dependency on an older version of Java EE, as this could break compatibility for the library that depends on Jakarta EE 8. Anyway, all of this won't happen because we also made it so that the Jakarta EE 8 dependency won't be upgraded to Jakarta EE 9 or later.

To look at the whole picture, if you only had the Java EE (any version) and Jakarta EE 8 dependencies, then the conflict would be on the javax.xml.bind:jaxb-api:2.3.1 and javax.xml.bind:jaxb-api:2.3.3 capabilities (Gradle capabilities follow the same naming rules as Maven coordinates, with a group, a name and a version) so with selectHighestVersion() the 2.3.3 capability would be selected, hence the Jakarta EE artifact. In other words, the Java EE 8 would be upgraded to Jakarta EE 8. This works because they kept Jakarta EE versions increasing after Java EE ones, and we declared the capability using that version.

Resolving the Jakarta EE 8 / Jakarta EE 9+ conflicts

Because we know that Java EE 8 and Jakarta EE 8 are fully compatible with one another, the solution here will be to actually downgrade Jakarta EE 8 to Java EE 8.

We'll thus replace the rule we added above that would reject the upgrade to Jakarta EE 9+, with a similar one that actually downgrades to Java EE 8.

components {
    all {
        allVariants {
            withDependencies {
                val found = removeIf {
                    it.group == "jakarta.xml.bind" &&
                    it.name == "jakarta.xml.bind-api" &&
                    it.versionConstraint.includesMajor("2")
                }
                if (found) {
                    add("javax.xml.bind:jaxb-api:2.3.1")
                }
            }
        }
    }
}

Note that we might want to actually conserve the attributes when replacing the dependency that way (or maybe not, I haven't thought about it much yet).

Such a rule actually trumps all our previous attempts:

if we replace Jakarta EE 8 with Java EE 8, we no longer have to detect when both are present, even less so resolve any such conflict; and
because Jakarta EE 8 has been replaced, we no longer have to reject upgrading it to Jakarta EE 9 either.

We can still keep those rules just in case, e.g. if Jakarta EE 8 is added as a direct dependency.

Problem solved then?

Well, for some definition of solved, yes.

Ideally, one would have to go through all the Java EE and Jakarta EE dependencies (fortunately, they can all be found in javax:javaee-api and jakarta.platform:jakartaee-bom POMs) and package all those rules into a plugin.

To that, one would have to add all the artifacts from EE vendors (e.g. for JAX-RS alone: javax:javaee-api, javax:javaee-web-api, org.jboss.spec.javax.ws.rs:jboss-jaxrs-api_2.1_spec, org.jboss.resteasy:jaxrs-api, org.apache.servicemix.specs:org.apache.servicemix.specs.jaxrs-api-2.1, org.apache.aries.spec:org.apache.aries.javax.jax.rs-api, org.apache.geronimo.specs:geronimo-jaxrs_2.1_spec, org.apache.tomee:javaee-api, and additionally for Jakarta RS: jakarta.platform:jakartaee-api, jakarta.platform:jakartaee-web-api, org.jboss.spec.javax.ws.rs:jboss-jaxrs-api_3.0_spec, org.apache.tomee:jakartaee-api, and I'm probably missing some) to declare them as providing the same Java EE or Jakarta EE capability (and note that versions do not match).

Then there needs to be extensive testing with various combinations of Java EE 7, Java EE 8, Jakarta EE 8 and Jakarta EE 9, including the includesMajor version check hinted above.

And of course, because we tap into each and every component, looking at all their dependencies, this needs to be optimized so as to not slow down all your builds.

Easier said than done.

Many thanks to Björn Kautler (Vampire) and Jendrik Johannes for the discussion and ideas.