DEV Community: Paulo Santos

Reactive Programming 🌫️ - Demystified using RxJS

Paulo Santos — Fri, 23 Jul 2021 22:38:48 +0000

If you're looking for a RxJS quick start then this article is not for you!

Here, I will be tackling Reactive Programming with the goal of shedding some light over its unreasonably illusive world using RxJS as an example.

I will be explaining the core Reactive Programming concepts, relating them to RxJS and how they work in practice. Hopefully, by the end of the read, you'll have a truer understanding of RxJS and should be able to quickly pick up any Rx Implementation to start coding with.

Statement

RxJS is an API for asynchronous programming
with observable streams.

To understand what this means we need to define what is meant by asynchronous programming and observable streams.

The best starting point can only be Reactive Programming itself!

Reactive Programming

Reactive Programming (not to be confused with Functional Reactive Programming!!) is a subset of Asynchronous Programming and a paradigm where the availability of new information drives the logic forward rather than having control flow driven by a thread-of-execution.

Asynchronous Programming is a means of parallel programming in which a unit of work runs separately from the main application thread. Generally, this is achieved via a messaging system where threads of execution competing for a shared resource don’t need to wait by blocking (preventing the thread of execution from performing other work until current work is done), and can as such perform other useful work while the resource is occupied. This concept is vital for Reactive Programming because it allows for non-blocking code to be written. Bellow a visualisation of the process:

Asynchronous Message System

Synchronous, blocking communication (left) is resource inefficient and easily bottlenecked. The Reactive approach (right) reduces risk, conserves valuable resources, and requires less hardware/infrastructure.

Messages vs Events

Reactive Programming is generally Event-driven. Events are simply undirected messages. At their core, they are (for all intents and purposes) an extension of an event.

The Application Program Interface (API) for Reactive Programming libraries are generally either:

Callback-based: where anonymous, side-effecting callbacks are attached to event sources and are being invoked when events pass through the dataflow chain.
Declarative: through functional composition, usually using well-established combinators like map, filter, fold etc.

Reactive Systems

The Reactive Manifesto defines that reactive systems are:

Responsive: responsive systems focus on providing rapid and consistent response times.
Resilient: resilient systems handle problems as they occur and stay responsive in the face of failure.
Elastic: elastic systems stays responsive under the varying workload and ergo have the ability to scale.
Message Driven: message-driven systems rely on asynchronous message-passing to establish to ensure that change is propagated between components without interruptions.

Principles of Reactive Systems

Reactive Programming & Reactive Systems

How do these two relate? In summary:

Reactive Programming is a technique for managing internal logic and dataflow transformation within components of a system. It is a way of providing clarity, performance and resource efficiency of code.
Reactive Systems is a set of architectural principles. It puts emphasis on distributed communication and gives us tools to tackle resilience and elasticity in distributed systems.

Reactive Programming should be used as one of the tools to construct a Reactive System.

Defining the Paradigm in Practice

Right, so, what is exactly is Reactive Programming? There are many definitions out there... some of which I think not even their authors understand what they mean. In the wise words of @andrestaltz - "Lets cut the bullshit"

Reactive programming is programming with asynchronous data streams.

Beautiful, concise and above all explainable! In fact, this definition is almost the same as the statement about RxJS I presented before. That is because RxJS is within the Reactive Programming paradigm.

From now on we assume Reactive programming with streams. There are other types of implementations that can also be considered within the paradigm, such as: Promises/Futures and Dataflow variables.

Now, as promised, I'll be showing you guys what is meant by 'asynchronous data streams'.

Streams

The key idea in Reactive Programming is that everything (for the most part) can be a stream. Streams are cheap and ubiquitous.

A stream is a sequence of ongoing events ordered in time. It can only emit 3 things: a data typed value, an error, or a termination signal.

This definition is important to remember since it remains the same no matter the implementation of the paradigm.

The way I like to think about streams is by visualising a water pipe with a closing mechanism where each water molecule (or set of) is an emitted value.

Water pipe stream analogy

The closing mechanism can be triggered manually by turning the tap, representing a termination signal, or implicitly, if the pipe fails to do its function, representing an error. A closed pipe can no longer push out water and we call it a completed stream.

Now, let us focus on the first sentence of our definition: 'A stream is a sequence of ongoing events ordered in time.'

Capturing values analogy

In other words, water droplets (data) are being pushed out of the pipe (stream) as time (program execution) passes. How do we capture these droplets to act on them?

In most implementations of Reactive Programming, we capture these emitted events only asynchronously, by defining functions that are called and passed one of the three appropriate outputs as a parameter:

On value emission: Each time a value is pushed through the stream it will be emitted and captured here. Can happen multiple times.
On error emission: When the stream error it will be captured here and the stream terminates. Happens only once.
On termination: When the stream is terminated it will be captured here. Happens only once.

That covers capturing. It's time to move into the manipulation of streams themselves. We do this via Operators.

Operators

Operators offer a way to manipulate streams by transforming them. A transformation, in our context, is simply a function f that maps a stream into another stream, i.e. f: S1 → S2 This function we call an operator.

To visualise this simple imagine placing one or more appliances within the pipeline of our stream. These appliances could have filters in them or could modify the contents of the water (or other transformations) thereby transforming our stream into a new stream.

Water pipe stream analogy with operators

In the image above, our initial stream of type 'Unpurified Water' was transformed into a stream of type 'Purified Water' transforming the data that gets observed at the end of the pipeline from its original form.

To explain operators and their effects on real streams of data we will have to dive into the world of Marble Diagrams.

Marble Diagrams

Before explaining marble diagrams we need to improve our terminology a little bit.

Redefining Some Terms

Now, because we will be dealing with ReactiveX in the next chapter, it's time to introduce some of the required terminologies. Don't worry, for now, I will only be giving abstracted definitions to a few terms that map to terms I've already covered. Below, the same diagrams as before, but with the new terminology included.

Water pipe stream analogy (w/terminology)

and for the operator diagram,

Water pipe stream analogy with operators (w/terminology)

Simple definitions for these terms are:

Stream -> Observable: A structure representing a stream of values over time.
Tap -> Subscriber: Sometimes called the consumer, the code that calls the subscription process on an observable structure.
Turning the tap -> Subscription: The method that opens the stream for the observer.
Closing the tap -> Completing: The action of marking the stream as completed meaning it is terminated.
Bucket -> Observer: The structure that captures our pushed values allowing us to act on them.
Appliances -> Operators: Functions that transform the stream.

We will be returning to more precise definitions later since they are pretty much required to read any sort of RX documentation without inducing a headache. So don't worry if you don't quite understand what these mean yet.

However, we will be using this new terminology from now on so I recommend keeping the term mapping in your head.

Marble Diagrams - The Observable

Okay, time for actual marble diagrams!

Learning Reactive Programming can be a daunting task so the Rx team came up with the concept of marble diagrams to help with visualising observables and their operators. These diagrams are incredibly intuitive and are commonly found in any Rx Operator documentation. They allow for an easy understanding of the operators without having to read much else. A good alternative to a chunky wall of text filled with terminology! I'll try to explain how to read them as best I can:

Overview of a Marble Diagram

Okay... my bad haha, sorry! Let's go step by step.

Marble diagrams describe observables. Observables are streams of values through time. So, we need a time axis!

Marble Diagram: time axis

Now that we have a time axis we need to represent our observable outputs. If you recall our earlier definition, an observable can only output a value, a termination signal or an error.

Let's start with the easy one, the termination signal:

Marble Diagram: termination signal

In similar fashion, we have our error output:

Marble Diagram: error

Finally, lets represent our emitted value:

Marble Diagram: value push/emission

There can be multiple values across the time axis as long as there is no termination or error output behind them as those will unsubscribe from the observable.

Done, simple right? On to the next part: operators in marble diagrams!

Marble Diagrams - The Operators

As previously mentioned, operators are functions that transform observables. That means they take as input one or more observables and output a new observable. We can represent them in a marble diagram like so:

Marble diagram of a filter operator

The block in between is our operator function, taking in an observable and returning another. So, our function is filtering the input observable by taking the modulus 2 to determine whether a pushed value is even and if it is it allows that push value to pass through, essentially, filtering the stream.

As mentioned before, operators can have more than one observable as input, such as the case for operators such as switchMap

Marble diagram of a switch map operator

The switchMap operator is a very popular one that has a handful of practical applications. It is generally used to implement a discard action between the input streams which can save a lot of trouble and computation in practice.

In summary, every time the Input Observable 1 emits a value, Input Observable 2 emits all of its values unless Input Observable 1 emits a new value before the Input Observable 2 completes. If you look at the output observable you will notice that there are only two 30's. This is because Input Observable 2 could not be complete before Input Observable 1 emitted the value 5. You easily confirm this because the space between 3 and 5 is much less than the size of the axis for Input Observable 2, suggesting there was only time to emit the first two values.

In Practice - RxJS

Overview of RxJS

RxJS is a library extending ReactiveX for composing asynchronous and event-based programs by using observable sequences with JavaScript. It provides one core type, the Observable, satellite types (Observer, Schedulers, Subjects) and operators (map, filter, reduce, every, etc) to allow the manipulation of the observable streams with easy and significantly reducing the amount of code needed to solve asynchronous problems.

Advantages VS Disadvantages

Advantages

Growing very rapidly.
- RxJs alone has 25mil weekly downloads.
Provides a very high-quality asynchronous API.
Lightweight & memory optimised.
Easy error handling.
Makes asynchronous programming much faster in most applications.

Disadvantages

Relatively steep learning curve.
Implies a functional programming style (data immutability).
Testing/debugging can be a learning process.

RxJS Glossary

In RxJS some arguably established definitions are:

Entities

Observable: represents the idea of an invokable collection of future values or events.
Observer: is a collection of callbacks that knows how to listen to values delivered by the Observable.
Subscription: represents the execution of an Observable, which is primarily useful for cancelling the execution.
Operators: are pure functions that enable a functional programming style of dealing with collections with operations like map, filter, concat, reduce, etc.
Subject: is equivalent to an EventEmitter, and the only way of multicasting a value or event to multiple Observers.
Schedulers: are centralized dispatchers to control concurrency, allowing us to coordinate when computation happens on e.g. setTimeout or requestAnimationFrame or others.
Producer: The code that is subscribing to the observable. This is whoever is being notified of nexted values, and errors or completions.
Consumer: Any system or thing that is the source of values that are being pushed out of the observable subscription to the consumer.

Concepts

Unicast: The act of one producer being observed only one consumer. An observable is "unicast" when it only connects one producer to one consumer. Unicast doesn't necessarily mean "cold".
Multicast: The act of one producer being observed by many consumers.
Cold: An observable is "cold" when it creates a new producer during subscribe for every new subscription. As a result, a "cold" observables are always unicast, being one producer observed by one consumer. Cold observables can be made hot but not the other way around.
Hot: An observable is "hot", when its producer was created outside of the context of the subscribe action. This means that the "hot" observable is almost always multicast. It is possible that a "hot" observable is still technically unicast if it is engineered to only allow one subscription at a time, however, there is no straightforward mechanism for this in RxJS, and the scenario is unlikely. For the purposes of discussion, all "hot" observables can be assumed to be multicast. Hot observables cannot be made cold.
Push: Observables are a push-based type. That means rather than having the consumer call a function or perform some other action to get a value, the consumer receives values as soon as the producer has produced them, via a registered next handler.
Pull: Pull-based systems are the opposite of push-based. In a pull-based type or system, the consumer must request each value the producer has produced manually, perhaps long after the producer has actually done so. Examples of such systems are Functions and Iterators

Observables & Subscriptions

By now we should agree that observables are simply structures that lazy push collections of multiple values. Subscriptions are the resulting structure representing a disposable resource, usually the execution of an Observable.

Heres how we code them in RxJS:

import { Observable } from 'rxjs';

/* Instantiate an observable */
const observable = new Observable(subscriber => {
  subscriber.next(1); // pushes a value
  subscriber.next(2); // pushes another value synchronously
  setTimeout(() => {
    subscriber.next(3); // pushes last value after a wait of 1s
    subscriber.complete(); // terminates observable stream
  }, 1000);
});

/* Subscribing to an observable */
console.log('just before subscribe');
const subscription = observable.subscribe({
  // The three possible output captures:
  next(x) { console.log('got value ' + x); },
  error(err) { console.error('something wrong occurred: ' + err); },
  complete() { console.log('done'); }
}); // creates subscription object
console.log('just after subscribe');

/* Unsubscribing to an observable using subscription */
setTimeout(() => {
  subscription.unsubscribe();
}, 500);

// Logs:
// just before subscribe 
// got value 1 
// got value 2 
// just after subscribe

Notice how we never see the value 3 logged because we cancel our subscription before it is emitted through the closure function passed to setTimeout.

However, this does not mean the value was not emitted, it was, we just don't see it because we stopped subscribing. The stream was not terminated via the act of unsubscribing.

Hot vs Cold Observables

A cold observable starts producing data when some code invokes a subscribe() function on it.

A cold observable:

import { Observable } from "rxjs";

// Creating a cold observable
const observable = Observable.create((observer) => {
    observer.next(Math.random()); // We explicitly push the value to the stream
});

// Subscription 1
observable.subscribe((data) => {
  console.log(data); // 0.24957144215097515 (random number)
});

// Subscription 2
observable.subscribe((data) => {
   console.log(data); // 0.004617340049055896 (random number)
});

A hot observable produces data even if no subscribers are interested in the data.

A hot observable:

import { Observable } from "rxjs";

// Coming from an event which is constantly emmit values
const observable = Observable.fromEvent(document, 'click');

// Subscription 1
observable.subscribe((event) => {
  console.log(event.clientX); // x position of click
});

// Subscription 2
observable.subscribe((event) => {
   console.log(event.clientY); // y position of click
});

Promises vs Observables

The main differences are:

Promises are eager. Observables are lazy.
Promises are single-value emissions. Observables are multi-value streams.
Promises have no cancelling or operator APIs. Observables do.

Promise vs Observable

A stackblitz example of RxJS vs Promises: https://stackblitz.com/edit/classicjs-vs-rxjs

Observables can be Promises

Although observables are not an extension of the Promise/A+ specification, RxJS still provides means to transform an observable into a true Promise. An example follows:

import { Observable } from "rxjs";

// Return a basic observable
const simpleObservable = val => Observable.of(val).delay(5000);

// Convert basic observable to promise
const example = sample('First Example')
  .toPromise() // Now its a promise
  .then(result => {
    console.log('From Promise:', result); // After 500ms, output 'First Example'
  });

With the use of RxJS's toPromise method, any observable can be converted to a promise. Note that because it returns a true JS Promise, toPromise is not a pipable operator, as it does not return an observable.

Observer

In practice, an Observer is a consumer of values delivered by an Observable. Observers are simply a set of callbacks, one for each type of notification delivered by the Observable: next, error, and complete. The following is an example of a typical Observer object:

const observer = {
  next: x => console.log('Observer got a next value: ' + x),
  error: err => console.error('Observer got an error: ' + err),
  complete: () => console.log('Observer got a complete notification'),
};

// To use it, pass it to a subscribe
observable.subscribe(observer);

That's it for observers, really!

Operators

RxJS is mostly useful for its operators, even though the Observable is the foundation. Previously we studied operators as functions that transformed streams. Nothing changes here, just terminology!

RxJS has a (very) vast library of operators. We will only be touching on a few simple ones to cover what we've talked about already:

import { from } from "rxjs";
import { filter } from "rxjs/operators";

from([1, 2, 3, 4, 5]).pipe(
  filter((x) => (x % 2) === 0)
).subscribe(console.log); // [2, 4]

If you remember our filter example from before this should be fairly simple to understand!

Pipeline

A pipeline is simply a series of operators that get executed in order. Something obvious but that people forget, every pipeline operator must return an observable.

The same exmaple as before but with chaining operators:

import { from } from "rxjs";
import { filter, take, map } from "rxjs/operators";

from([1, 2, 3, 4, 5]).pipe(
  filter((x) => (x % 2) === 0),
  take(1),
  map((firstValue) => "The first even number was " + firstValue)
).subscribe(console.log);

There are a ton more operators that do vastly different things in categories such as: Creation, Filtering, Combination, Error Handling, Transformation, Multicasting, etc. I encourage you to try a few from each of the categories out. This is the power of RxJS, there's a lot already done for you!

Subjects

A Subject is like an Observable, but can multicast to many Observers. Subjects are like EventEmitters: they maintain a registry of many listeners. In fact, part of a subject is literally an observable and you can get a reference to that observable.

The easiest way to think of a subject is quite literally:

Subject = Observer + Observable

Example:

import { Subject, from } from 'rxjs';

const subject = new Subject<number>();

subject.subscribe({
  next: (v) => console.log(`observerA: ${v}`)
});
subject.subscribe({
  next: (v) => console.log(`observerB: ${v}`)
});

subject.next(1);
subject.next(2);

// Logs:
// observerA: 1
// observerB: 1
// observerA: 2
// observerB: 2

const observable = from([1, 2, 3]);

observable.subscribe(subject); // You can subscribe providing a Subject

// Logs:
// observerA: 1
// observerB: 1
// observerA: 2
// observerB: 2
// observerA: 3
// observerB: 3

IMO, the best use case for Subjects is when the code it is referenced in is the one that is producing the observable data. You can easily let your consumers subscribe to the Subject and then call the .next() function to push data into the pipeline. Be wary of overusing them since most problems are solvable with only data transformation and Observables.

Schedulers

Finally, schedulers! They might seem hard to understand but are quite simple at a surface level which is more than enough for us to know about. In essence, schedulers control the order of tasks for Observables. There are only a few of them and they won't be changing anytime soon, here they are:

Table of the types of schedulers

You can use schedulers by passing them to observables through a handful of operators (usually of the creation category) as arguments. The most basic example, forcing a synchronous observable to behave asynchronously:

import { Observable, asyncScheduler } from 'rxjs';
import { observeOn } from 'rxjs/operators';

const observable = new Observable((observer) => {
  observer.next(1);
  observer.next(2);
  observer.next(3);
  observer.complete();
}).pipe(
  observeOn(asyncScheduler)
);

console.log('just before subscribe');
observable.subscribe({
  next(x) {
    console.log('got value ' + x)
  },
  error(err) {
    console.error('something wrong occurred: ' + err);
  },
  complete() {
     console.log('done');
  }
});
console.log('just after subscribe');

// Logs
// just before subscribe
// just after subscribe
// got value 1
// got value 2
// got value 3
// done

Notice how the notifications got value... were delivered after just after subscription. This is because observeOn(asyncScheduler) introduces a proxy Observer between the new Observable and the final Observer.

Other schedulers can be used for different timings. We are done!

Amazing RxJS Resources

RxJS visualizer: https://rxviz.com/
Instant marble diagrams: https://thinkrx.io/
Docs with marble diagrams: https://rxmarbles.com/
Operator decision tree: https://rxjs.dev/operator-decision-tree

Reference

Journey of a web page 🛣️ - How browsers work

Paulo Santos — Mon, 19 Jul 2021 09:04:55 +0000

This article covers the processes that happen in-between a user entering a web address in a browser and a web page showing up as a result.

Overview

There are a handful of steps that occur betiwx the time a user requests a web page and the time it displays in their browser. To better cover each of them, I've divided those steps into the following sections:

Navigation
- Resolving the web address (DNS Lookup)
- Establishing a connection to the server (TCP 3-way handshake)
- Establishing a security protocol (TLS negotiation)
Fetching
- Browser Caching
- HTTP request
- HTTP response
- Streaming and Progressive Parsing
Parsing
- Building the DOM tree
- Building the CSSOM tree
- Combining the trees into the render tree
- Preload Scanner
- JavaScript Compilation
- Building the Accessibility Tree
Rendering
- Critical Rendering Path
- Layout
- Paint
- Compositing
- Dynamic Rendering and Updates
Finalising
- JavaScript Occupied.
- Document Lifecycle Events
- User can now browse the page!

Use this index to jump to any particular section of interest. If you're new to some of these concepts, I suggest a linear read.

It is worth noting that browsers perform many of these steps concurrently, and some may be skipped entirely thanks to caching or connection reuse. For simplicity, we'll follow the general journey of a page from navigation to rendering.

It is time for a quick background lesson!

Background

This section contains a very quick and general overview of the some of the core background concepts that are required to understand the later parts of the article.

Networking models

Models exist to explain how data is transmitted through a network. In particular, there exists a network model so widely known that even those that do not partake in hackerman things might've heard of! We call it the Open Systems Interconnected (OSI) model.

The OSI Model

The Open Systems Interconnection (OSI) model describes seven layers that computer systems use to communicate over a network. Each upper layer is one level of abstraction higher than the previous, all the way up to the application (browser) layer which we will be talking about.

It’s important to understand that the OSI model is a “conceptual model” for how applications communicate over a network. It is not a protocol. Do not get the two confused. Protocols are strict sets of rules that may live within these layers.

An older and very similar model, possibly more relevant towards the article, is the TCP/IP model. This network model is used for both modelling current Internet architecture and providing a set of rules (concrete protocols) that are followed by all forms of transmission over that network.

OSI vs TCP/IP

I will be using and referring to this model with its associated protocols throughout the article.

Any data sent from an application to another will have to travel up and down through these layers a few numbers of times (depending on how many middlemen there are). Of course, this happens incredibly fast nowadays, however, it still does happen and understanding the overview of the process is something that every developer should know. The following is an image representation of network process between a server and a client application:

TCP/IP model data path between applications

Take this example - a user requesting to browse a page using their browser:

The request is first sent to the application layer, where it is processed from layer to layer down with each layer performing its designated functions.
The data is then transmitted over the physical layer of the network until the destination server or another device receives it.
At this point the data is passed up through the layers again, each layer performing its assigned operations until the data is used by the web server software.
This process is repeated again for the response of the server.

In general, this is how machines communicate over the network!

High-level abstraction of a browser

The later sections of the article will be covering how a typical browser displays the content of a page on screen. A high-level understanding of the browser is important when reading those sections. I will be referencing some of the following browser components:

The user interface: this includes the address bar, back/forward button, bookmarking menu, etc. Every part of the browser display except the window where you see the requested page.
The browser engine: marshals actions between the UI and the rendering engine.
The rendering engine: responsible for displaying requested content. For example, if the requested content is HTML, the rendering engine parses HTML and CSS and displays the parsed content on the screen.
Networking: for network calls such as HTTP requests, using different implementations for different platforms behind a platform-independent interface.
UI backend: used for drawing basic widgets like combo boxes and windows. This backend exposes a generic interface that is not platform-specific. Underneath it uses operating system user interface methods.
JavaScript interpreter: Used to parse and execute JavaScript code.
Data storage: This is a persistence layer. The browser may need to save all sorts of data locally, such as cookies. Browsers also support storage mechanisms such as localStorage, IndexedDB, WebSQL and FileSystem.

Browser Components

It is important to note that browsers such as Chrome have a multi-process approach for performance and security reasons. This means that they run instances of some of these components, such as the rendering engine, in separate processes. A tab is not necessarily a single process: Chrome may use multiple renderer processes within a tab, or share one across tabs. You can see this by checking Chrome's processes in its Task Manager.

Screenshot of Chrome's Task Manager

As you can observe from the screenshot above, Chrome's Task Manager shows separate tasks with CPU/Network/GPU statistics. Some of these correspond to renderer processes, while others belong to shared browser services. You can further confirm this by listing your OS's processes.

To conclude this background section, it is important to note that what you just read so far is a very high-level generalisation and abstraction of how networks and browsers actually work. Not all networks strictly abide by the OSI/TCP IP models and the main browsers in use today are all different in their own ways but share a common base of concepts that allows us to abstract them to what you've just read.

For browsers, as an example, they all follow (to some degree) the specifications maintained by the W3C (World Wide Web Consortium) organization, which is the standards organization for the web. However, the actual implementation of each component varies quite significantly. e.g. rendering engines are different, Internet Explorer uses Trident, Firefox uses Gecko, Safari uses WebKit. Chrome, Edge and Opera use Blink, a fork of WebKit etc. And the same goes for javascript engine pipelines!

Journey of a page

You open up a browser and type in www.google.com. This is what happens.

Navigation

The first step is navigating to the correct place. Navigating to a web page is finding where the assets for that page are located. To us, web pages are simply domain names, but to computers, they resolve into IP addresses.

www.google.com, the page resources will be located on servers reachable through the IP addresses returned by DNS. If the address is not already cached, a Domain Name System (DNS) lookup is usually required.

Round Trip Time

Round-trip time (RTT) is the duration, measured in milliseconds, from when a browser sends a request to when it receives a response from a server. It's a key performance metric for web applications and one of the main factors, along with Time to First Byte (TTFB), when measuring page load time and network latency.

I will be annotating each network process with their corresponding RTT.

Resolving a web address - The DNS Process (Variable RTT)

An overview of the DNS process for www.google.com is as follows:

Check browser & OS cache, return if IP found.
The browser or operating system sends a DNS query to its configured resolver.
1. DNS resolver checks its cache, return if IP is found.
DNS resolver sends out a request asking root nameservers.
A root nameserver responds to the DNS resolver with an IP address to a TLD nameserver (in this case the TLD for extensions of .com).
DNS resolver sends out another request, now to the TLD nameserver, asking if they know what the IP is.
TLD nameserver responds back to the DNS resolver with an IP of the authoritative nameserver.
DNS resolver sends out the final request to the authoritative nameserver asking for the IP.
The authoritative nameserver will scan zones files to find the domain name:ipaddress mapping and will return whether it exists or not to the DNS resolver.
Finally, the DNS resolver will now respond back to the browser with the IP of the server the browser is trying to communicate.

Visualisation of the DNS process

Note that this process is usually incredibly fast and rarely follows through in its entirety because of all the layers of caching. It was designed to be fast!

Establishing a connection to the server - TCP Handshake (1 RTT)

Now that the IP address is known, the browser sets up a connection to the server via a TCP three-way handshake. This applies to HTTP/1.1 and HTTP/2 connections; HTTP/3 uses a different transport, as we will see below.

TCP uses a three-way handshake to establish a reliable connection. The connection is full-duplex, and both sides synchronize (SYN) and acknowledge (ACK) each other. The exchange of these four flags is performed in three steps—SYN, SYN-ACK, and ACK—as shown in

TCP 3-way handshake

The client chooses an initial sequence number, set in the first SYN packet.
The server also chooses its own initial sequence number.
Each side acknowledges the other's sequence number by incrementing it; this is the acknowledgement number.

Once a connection is established, ACKs typically follow for each segment. The connection will eventually end with an RST (reset or tear down the connection) or FIN (gracefully end the connection).

This mechanism is designed so that two entities attempting to communicate, in this case, the browser and web server, can negotiate the parameters of the network TCP socket connection before transmitting data, in our case, it will be over HTTPS; the secure version of HTTP.

HTTPS is HTTP with encryption. The only difference between the two protocols is that HTTPS uses TLS (SSL) to encrypt normal HTTP requests and responses. As a result, HTTPS provides a solid layer of security over HTTP. A website that uses HTTP has HTTP:// in its URL, while a website that uses HTTPS has HTTPS://.

Establishing a security protocol - TLS Negotiation (typically 1 RTT with TLS 1.3)

For secure connections established over HTTPS, another "handshake" is required. This handshake, or rather the TLS negotiation, determines which cipher will be used to encrypt the communication, verifies the server, and establishes that a secure connection is in place before beginning the actual transfer of data.

Modern day TLS establishing

While making the connection secure adds time to the page load, a secure connection is worth the latency expense, as the data transmitted between the browser and the web server cannot normally be decrypted by a third party. TLS has come a long way: TLS 1.2 typically requires 2 RTT for a full handshake, while TLS 1.3 typically requires just 1. TLS 1.3 also supports 0-RTT early data on some resumed connections, although this has security trade-offs and is not always used.

For a new HTTP/2 connection using TLS 1.3, we can roughly count 1 RTT for TCP, 1 for TLS, and another for the HTTP request and the start of its response, plus DNS and server processing time. These are simplified figures, not fixed totals; existing connections can skip the handshake entirely.

It is worth noting that HTTP/3 takes a different approach. It uses QUIC over UDP instead of TCP, integrating TLS 1.3 into the connection handshake and reducing the round trips needed to establish a secure connection.

Fetching

Now that we have a TCP connection setup and the TLS exchange has been completed, the browser can now begin to fetch the page's resources. It starts by fetching the markup document for the page. It does this by using the HTTP protocol. In our HTTP/2 example, HTTP requests are sent over TCP and encrypted with Transport Layer Security (TLS); since google uses HTTPS.

Browser Caching

Before making a request, however, there is another important optimization worth mentioning: HTTP caching. The browser doesn't necessarily need to download a resource every time it is requested. If it has a reusable response stored in its cache, it may be able to use that instead. This applies to things like HTML documents, CSS, JavaScript, images, and other resources.

How does the browser know when it can reuse something? The server can provide instructions through HTTP response headers. Take this example:

cache-control: public, max-age=3600
etag: "v1"

Here, max-age=3600 tells caches that the response can be considered fresh for 3600 seconds (one hour). The public directive also allows shared caches to store it. The ETag is an identifier for a particular version of the response.

Once the response becomes stale, the browser may need to check with the server again. Instead of downloading the whole resource, it can send a conditional request using an If-None-Match header with the previously received ETag. If the resource hasn't changed, the server can respond with 304 Not Modified, allowing the browser to reuse the cached body. If it has changed, the server sends an updated response.

It's also worth distinguishing no-cache, which requires validation before reuse, from no-store, which instructs caches not to store the response. And while we won't be going into detail on them here, service workers can also intercept requests, while the back/forward cache can preserve an entire page for history navigation. They are different mechanisms from regular HTTP caching.

For the rest of our example, let's assume the browser needs to fetch the page from the server.

HTTP Request

To fetch a page a safe and idempotent request is made. We use the HTTP GET request.

GET—Requests information from a given server using a Uniform Resource Identifier (URI). A correct implementation of the GET method is intended to retrieve data, not request changes to the server's state. It is also idempotent, meaning that repeating the same request has the same intended effect as making it once.

There are many other types of HTTP methods:

HTTP Methods as per sepecification

For fetching our page we are only interested in GET. A human-readable representation of an HTTP/2 request could look like this (HTTP/2 actually transmits headers in binary frames):

:method: GET
:scheme: https
:authority: www.google.com
:path: /
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
accept-language: en-GB,en;q=0.5
accept-encoding: gzip, deflate, br
upgrade-insecure-requests: 1
cache-control: max-age=0
te: trailers

HTTP Response

Once the webserver receives the request. It will parse the request and try to fulfil it. We assume the request is valid and the files are available. It will reply with an HTTP response, attaching the relevant headers and the contents of the requested HTML document to the body of that response structure.

:status: 200
date: Sun, 18 Jul 2021 00:26:11 GMT
expires: -1
cache-control: private, max-age=0
content-type: text/html; charset=UTF-8
strict-transport-security: max-age=31536000
content-encoding: br
server: gws
content-length: 37418
x-xss-protection: 0
x-frame-options: SAMEORIGIN

*the source code of the HTML document will be in the body of the response.

HTTP Redirects

Of course, not every HTTP response contains the page we requested. Sometimes the server responds with a redirect, telling the browser to request another URL instead.

For example, a response may look like this:

:status: 302
location: https://www.example.com/new-page

The browser reads the Location header and makes another request to that address. Depending on where the new URL points, this might involve another DNS lookup or a new connection to a server.

Some common redirect status codes are 301 and 308 (permanent), and 302 and 307 (temporary). An important distinction is that 307 and 308 preserve the original request method, while 301 and 302 may change a POST into a GET when followed by a browser.

Streaming and Progressive Parsing

So far, we've talked about the HTTP response as if the browser waits for the entire document to arrive before doing anything with it. That isn't quite what happens!

The response body can arrive in pieces over the network. Once the browser has enough data, it can start processing the HTML while the rest of the document is still downloading. If the response uses compression, such as br (Brotli), the browser first decompresses the incoming data and decodes it into characters for the HTML parser.

As those characters become available, the browser can begin tokenizing the HTML and building the DOM tree. Meanwhile, additional parts of the document are still arriving, and the preload scanner may already have discovered other resources such as CSS or JavaScript files.

This means that by the time the last part of the document arrives, some of the DOM may already be constructed, and the browser may even have displayed part of the page. This isn't always the case, as resources such as stylesheets and blocking scripts can still delay rendering. But it is one of the reasons why browsers can begin displaying a page before everything has finished downloading.

Parsing

Once the browser receives the response, it can begin parsing the information received. Parsing is the step the browser takes to turn the data it receives over the network into the DOM and CSSOM, which is used by the renderer to paint a page to the screen.

The Document Object Model (DOM) is an internal representation of the objects that comprise the structure and content of the markup (HTML in this case) document the browser just received. It represents the page so that programs can change the document structure, style, and content.

The DOM represents the document as nodes and objects. That way, programming languages can connect to the page. There are many different types of nodes in a DOM tree. An example part of the specification for the DOM Node Interface looks as follows:

Node.ELEMENT_NODE
Node.ATTRIBUTE_NODE
Node.TEXT_NODE
Node.CDATA_SECTION_NODE
Node.PROCESSING_INSTRUCTION_NODE
Node.COMMENT_NODE
Node.DOCUMENT_NODE
Node.DOCUMENT_TYPE_NODE
Node.DOCUMENT_FRAGMENT_NODE
Node.NOTATION_NODE

These node types cover all possible nodes in the document. Such representations help programming languages connect to the page so that they can manipulate it.

To conclude our parsing preface, we need to talk about the CSSOM.

The CSS Object Model (CSSOM) is a set of APIs allowing the manipulation of CSS from JavaScript. It is much like the DOM, but for the CSS rather than the HTML. It allows users to read and modify CSS style dynamically. It is represented very similarly to the DOM, as a tree, and it will be used along with the DOM to form a render tree so that the browser can begin its rendering process. Let's find out how by going through the entire process.

Building the DOM tree

The first step is processing the HTML markup and building the DOM tree. HTML parsing involves tokenization and tree construction.

HTML may come as a surprise in the world of parsing as it cannot be parsed through conventional means. Unlike XML, modern HTML is not defined by a Document Type Definition (DTD). Instead, the HTML specification defines its own parsing algorithm. I will not be going too much into detail about how this is done but the main reasons for this are:

The forgiving nature of the language.
The fact that browsers have traditional error tolerance to support well-known cases of invalid HTML.
The parsing process is reentrant. For other languages, the source doesn't change during parsing, but in HTML, dynamic code (such as script elements containing document.write() calls) can add extra tokens, so the parsing process actually modifies the input.

Unable to use the regular parsing techniques, browsers implement the HTML parsing algorithm defined by the specification.

The parsing algorithm is described in detail by the HTML Living Standard. As previously mentioned, the algorithm consists of two stages: tokenization and tree construction.

Tokenization is the lexical analysis, parsing the input into tokens. Among HTML tokens are start tags, end tags, attribute names and attribute values.
Tree construction is essentially creating a tree-based of the parsed tokens and what we will be focusing on—The DOM tree.

The DOM tree describes the content of the document. The <html> element is the first tag and root node of the document tree. The tree reflects the relationships and hierarchies between different tags. Tags nested within other tags are child nodes. The greater the number of DOM nodes, the longer it takes to construct the DOM tree. Below you can find a visual representation of the DOM tree—The output of the parser:

Part of the DOM Tree

When the parser finds non-blocking resources, such as an image, the browser will request those resources and continue parsing. Parsing can continue when a CSS file is encountered, but <script> tags—particularly those without an async or defer attribute—block rendering, and pause the parsing of HTML. Though the browser's preload scanner hastens this process, excessive scripts can still be a significant bottleneck.

Building the CSSOM tree

The second step is processing CSS and building the CSSOM tree. In a similar fashion to the DOM parsing phase, the browser goes through each rule set in the CSS, creating a tree of nodes with parent, child, and sibling relationships based on the CSS selectors.

In regards to parsing, well, unlike HTML, CSS is a context-free grammar and can be parsed using regular CFG parsing techniques. In fact the CSS specification defines CSS lexical and syntax grammar.

As with HTML, the browser needs to convert the received CSS rules into something it can work with. From there, it repeats the HTML-to-object process, but for the CSS.

Combining the trees into the render tree

The CSSOM and DOM trees are combined into a render tree, which is then used to compute the layout of each visible element and serves as an input to the paint process that renders the pixels to the screen.

Combining the CSSOM + DOM trees to make the render tree

To construct the render tree, the browser roughly does the following:

Starting at the root of the DOM tree, traverse each visible node.
- Some nodes are not visible (for example, script tags, meta tags, and so on), and are omitted since they are not reflected in the rendered output.
- Some nodes are hidden via CSS and are also omitted from the render tree; for example, the span node—in the example above—is missing from the render tree because we have an explicit rule that sets the "display: none" property on it.
For each visible node, find the appropriate matching CSSOM rules and apply them.
Emit visible nodes with content and their computed styles.

The final output is a render that contains both the content and style information of all the visible content on the screen. With the render tree in place, we can proceed to the "layout" stage.

Preload Scanner

While the browser's main thread is busing building the DOM tree it has a helper worker scan through the content available. This helper is the preload scanner and will prepare high priority fetch request for resources like CSS, JavaScript, and web fonts. This is an optimization added over the parsing stage as it would take far too long to make these requests as the parser finds references to them.

<link rel="stylesheet" href="styles.css"/>
<script src="myscript.js" async></script>
<img src="myimage.jpg" alt="image description"/>
<script src="anotherscript.js" async></script>

Taking the example above, the preload scanner will try to find the scripts and images, and start downloading them. The async and defer attributes affect when scripts execute:

async: When present, it specifies that the script will be executed asynchronously as soon as it is available.
defer: When present, it specifies that the script is executed when the page has finished parsing, before the DOMContentLoaded event.

Another important part of this optimization is fetch priority. Not all resources are equally important! The browser assigns priorities to requests so that resources needed early, such as render-blocking stylesheets, can be fetched ahead of less important ones.

We can also provide a hint using the fetchpriority attribute. For example, <img src="hero.jpg" fetchpriority="high" alt="..."> tells the browser that this image is particularly important. It is only a hint though; the browser still makes the final decision. Unlike async and defer, this attribute affects fetching priority, not when a script executes.

Waiting to obtain CSS doesn't block HTML parsing or downloading, but it does block JavaScript because JavaScript is often used to query CSS properties’ impact on elements.

JavaScript Compilation

While the CSS is being parsed and the CSSOM created, other assets, including JavaScript files, are downloading (thanks to the preload scanner). JavaScript is interpreted, compiled, parsed and executed.

Overview of the JS compilation process

The scripts are parsed into abstract syntax trees. Like in the above diagram (from V8's engine blog), some browser engines take the Abstract Syntax Tree and pass it into an interpreter, outputting bytecode which is executed on the main thread. This is known as JavaScript compilation.

Building the Accessibility Tree

The browser also builds an accessibility tree that assistive devices use to parse and interpret content. The accessibility object model (AOM) is like a semantic version of the DOM. The browser updates the accessibility tree when the DOM is updated. The accessibility tree is not modifiable by assistive technologies themselves.

Build and use process of the AOM

Until the AOM is built, the content is not accessible to screen readers.

Rendering

Now that the information has been parsed the browser can begin to display it. To achieve this the browser will now use the render tree to produce a visual representation of the document. Rendering steps include layout, paint and, in some cases, compositing.

Critical Rendering Path

Now is a good time to introduce the notion of the Critical Rendering Path. The best way to visualise it is with a diagram:

The Critical Rendering Path

Optimizing the critical rendering path improves the time to first render. It is important to ensure reflows and repaints can happen at the goal of 60 frames per second (for performant user interactions).

We won't be going into detail on how to optimise the CRP but the general gist relies on improving page load speed by prioritizing which resources get loaded, controlling the order in which they are loaded, and reducing the file sizes of those resources.

Now let's move on to the rendering stages.

Layout

Layout is the first rendering stage, where the geometry and positioning of the render tree nodes are determined. Once the render tree is built, the layout commences.

Layout is a recursive process. It begins at the root renderer, which corresponds to the <html> element of the HTML document. Layout continues recursively through some or all of the frame hierarchy, computing geometric information for each renderer that requires it.

Layout stage

At the end of the layout stage, a tree similar to the one above is generated with what we call blocks/Coxes as nodes. Blocks/Boxes hold the geometric information of DOM objects and nodes.

Dirty bit system

In order not to do a full layout for every small change, browsers use a "dirty bit" system. A renderer that is changed or added marks itself and its children as "dirty": needing layout.

There are two flags:
- dirty: node needs a layout.
- children are dirty: node has at least one child that needs a layout.

Layout Algorithm

Using the dirty bit system browsers can now perform an algorithm to generate the layout. A high-level abstraction of that algorithm is as folows:

Parent node determines its own width.
Parent goes over children and:
1. Calculate child render's size
2. Calls child layout if they have a dirty descendant
Parent uses children's accumulative heights and the heights of margins and padding to set its own height–this will be used by the parent renderer's parent.
Sets its dirty bit to false.

An added important concept is that of reflow. As previously mentioned, the first time the size and position of nodes are determined is called layout. Subsequent recalculations of node size and locations are called reflows. As an example, suppose the initial layout occurs before the image is returned. Since we didn't declare the size of our image, there will be a reflow once the image size is known!

Paint

The third and last stage of rendering. In this painting phase, the browser converts each box calculated in the layout phase to actual pixels on the screen. Painting involves drawing every visual part of an element to the screen, including text, colours, borders, shadows, and replaced elements like buttons and images. The browser needs to do this super quickly.

Painting order

CSS2 defines the order of the painting process. This is actually the order in which the elements are stacked in the stacking contexts. This order affects painting since the stacks are painted from back to front. The stacking order of a block renderer is:

background colour
background image
border
children
outline

Paint Layers

Painting can break the elements in the layout tree into layers. Promoting content into layers on the GPU (instead of the main thread on the CPU) improves paint and repaint performance. There are specific properties and elements that instantiate a layer, including <video> and <canvas>, and any element which has the CSS properties of opacity, a 3D transform, will-change, and a few others. These nodes will be painted onto their own layer, along with their descendants, unless a descendant necessitates its own layer for one (or more) of the above reasons.

Layers do improve performance but are expensive when it comes to memory management, so should not be overused as part of web performance optimization strategies.

Compositing

When sections of the document are drawn in different layers, overlapping each other, compositing is necessary to ensure they are drawn to the screen in the right order and the content is rendered correctly.

Compositing stage

As the page continues to load assets, reflows can happen (recall our example image that arrived late). A reflow sparks a repaint and a re-composite. Had we defined the size of our image, no-reflow would have been necessary, and only the layer that needed to be repainted would be repainted, and composited if necessary. But we didn't include the image size! When the image is obtained from the server, the rendering process goes back to the layout steps and restarts from there.

Dynamic Rendering and Updates

By now, we've seen how the browser turns HTML and CSS into pixels on our screen. But rendering isn't something that only happens once! The page can continue changing as JavaScript modifies the DOM, images load, the user resizes their window, or animations update the appearance of elements.

When something changes, the browser may need to go through parts of the rendering process again. It doesn't necessarily repeat every step. As an overview:

Style recalculation: If the styles affecting an element change, the browser may need to calculate its styles again.
Layout: Changes affecting size or position can require a new layout (reflow) for the affected elements.
Paint: Changes to visual properties, such as a background colour, may need a repaint without requiring a new layout.
Compositing: Some changes, such as transform or opacity, can often be handled by the compositor without a new layout or paint, provided the element is on a suitable layer.

This is why changing an element's width repeatedly can be more expensive than moving it using a transform. It depends on which stages the change causes the browser to repeat.

For animations, we can use requestAnimationFrame() to request a JavaScript callback before the browser's next repaint. This helps schedule visual updates around the browser's rendering cycle, although lengthy JavaScript work can still block the main thread.

In general, the browser tries to avoid repeating unnecessary work to keep the page responsive and animations smooth.

Finalising

Once the main thread is done painting the page, you would think we would be "all set." That isn't necessarily the case. If the load includes JavaScript, which was correctly deferred and executed after the HTML document has finished parsing, the main thread might still be busy, and not available for scrolling, touch, and other interactions.

JavaScript Occupied

Time to Interactive (TTI) was historically used to measure when a page becomes interactive, but was removed from Lighthouse 10 due to its variability. Metrics such as Total Blocking Time (TBT) and Interaction to Next Paint (INP) now provide better insight into different aspects of responsiveness. If the main thread is occupied parsing, compiling, and executing JavaScript, it is not available and therefore not able to respond to user interactions in a timely fashion.

In our example, maybe the image loaded quickly, but perhaps the anotherscript.js file was 2MB and contained JavaScript that took a long time to execute. In this case, the user could see the page super quickly, but wouldn't be able to scroll without jank while the script occupies the main thread. That is not a good user experience. Avoid occupying the main thread, as demonstrated in this WebPageTest example:

Example - JavaScript hogging main thread

In this example, the DOM content load process took over 1.5 seconds, and the main thread was fully occupied that entire time, unresponsive to click events or screen taps.

One more concept worth mentioning here is the JavaScript event loop. In a browser, JavaScript running on the main thread typically executes one task at a time. These tasks may include running a script, handling a click, or executing a setTimeout() callback. While a task is running, another main-thread task cannot run, so a long-running script can delay user interactions.

Once a task finishes, the browser processes queued microtasks, such as callbacks from resolved promises, before moving on to another task. The browser also gets opportunities to render between tasks, but not while the main thread is busy executing a long task. This is one of the reasons why breaking lengthy JavaScript work into smaller tasks can help improve responsiveness.

Document Lifecycle Events

Before we conclude, there are a few events and states worth mentioning. The browser exposes these so that JavaScript can tell how far the document has progressed through loading. You may have come across DOMContentLoaded and load before, but they don't mean quite the same thing.

document.readyState: A property that can be loading while the HTML is being parsed, interactive once parsing is complete, or complete once the document and its dependent resources have finished loading.
DOMContentLoaded: An event that fires after the HTML document has been parsed and deferred scripts (including module scripts) have executed. It does not wait for images and other non-blocking resources to finish loading.
load: An event that fires when the page and its dependent resources, such as images and stylesheets, have finished loading, excluding resources that are lazy-loaded.

An important distinction is that these aren't necessarily the moments when the page first becomes visible or responsive. The browser may have already painted content before the load event, and JavaScript can still occupy the main thread afterwards.

The user can now browse the page! 🎉

Yes, after all these stages the user can now both see and interact with the page!

Summary

For a simple page to be displayed in our browser window it may need to go through:

DNS Lookup: To find out the IP of the web address.
TCP Handshake: To set up TCP communication between the client and server for HTTP/1.1 and HTTP/2.
TLS Handshake: To safeguard the information that will be sent via encryption (integrated into QUIC for HTTP/3).
Browser Caching: To reuse resources already stored locally when possible.
HTTP communication: To establish a method of communication that browsers can understand, including redirects and streaming the response.
Browser Parsing: To parse the HTTP's response—the document of the page to be displayed.
Browser Rendering: To render the document on the browsers window and update it when something changes.
Javascript Occupied: To wait for JS to compile and execute as it may hog the main thread.
Document Lifecycle Events: To signal the progress of document parsing and loading.

It is incredible the number of things that happen for this seemingly simple task to be accomplished. Truly an impressive journey for our little page :)

There is more...

In a real-world scenario, our little page may have to tackle even more obstacles before reaching its destination things such as: load balancers, proxies, multiple layers of firewalls, etc...

These topics are beasts of their own and if you're curious about how they work I recommend looking them up! Perhaps I'll write something for them in the future.

Thank you for reading,
Paulo 😎

DEV Community: Paulo Santos

Reactive Programming 🌫️ - Demystified using RxJS

Statement

Reactive Programming

Messages vs Events

Reactive Systems

Reactive Programming & Reactive Systems

Defining the Paradigm in Practice

Streams

Operators

Marble Diagrams

Redefining Some Terms

Marble Diagrams - The Observable

Marble Diagrams - The Operators

In Practice - RxJS

Overview of RxJS

Advantages VS Disadvantages

Advantages​

Disadvantages​

RxJS Glossary

Entities

Concepts

Observables & Subscriptions

Hot vs Cold Observables

Promises vs Observables

Observables can be Promises

Observer

Operators

Pipeline

Subjects

Schedulers

Amazing RxJS Resources

Reference

Journey of a web page 🛣️ - How browsers work

Overview

Background

Networking models

High-level abstraction of a browser

Journey of a page

Navigation

Round Trip Time

Resolving a web address - The DNS Process (Variable RTT)

Establishing a connection to the server - TCP Handshake (1 RTT)

Establishing a security protocol - TLS Negotiation (typically 1 RTT with TLS 1.3)

Fetching

Browser Caching

HTTP Request

HTTP Response

HTTP Redirects

Streaming and Progressive Parsing

Parsing

Building the DOM tree

Building the CSSOM tree

Combining the trees into the render tree

Preload Scanner

JavaScript Compilation

Building the Accessibility Tree

Rendering

Critical Rendering Path

Layout

Dirty bit system

Layout Algorithm

Paint

Painting order

Paint Layers

Compositing

Dynamic Rendering and Updates

Finalising

JavaScript Occupied

Document Lifecycle Events

The user can now browse the page! 🎉

Summary

There is more...

Reference

Advantages

Disadvantages