DEV Community: Benny Guo

Browser Working Principle - HTML Parsing

Benny Guo — Wed, 16 Jun 2021 03:22:53 +0000

Introduction

In the previous article, we discussed how HTTP requests are sent and how responses are received. Additionally, we parsed the text context from the response body.

In this part, we will discuss how HTML is parsed and how CSS is computed.

Base on the diagram above, the blue part represents what we had done so far.

Refactoring file structure

To improve the management of our files, we will break down the code into JavaScript files based on the purpose of each.

We are going to take out the part that is responsible for parsing into a different file called parser.js, the rest of the code would be our client.js.

Adding in HTML Parser

We received a Response object at the end of our last article, and we are going to use the body content from it
Pass the body content to a parser and run it's parseHTML() method to parse the content.
A real browser passes the context to the parser in a stream (meaning that data is passed chunk by chunk and parsing is also chunk by chunk).
Instead, we will pass the entire content to the parser in our mini-browser. In this way, it can be better understood.

File: client.js

// client.js

 // 1. require parser.js
const parser = require('./parser.js');

// ...
// Copy all the code here, 
// from want we have before
// ...

let response = await request.send();

// 2. Get the response body from the request and pass it to the HMTL parser.
// -> At the end we should have a DOM object returning.
let dom = parser.parseHTML(response.body);

File: parser.js

/**
 * Parser
 * @filename parser.js
 * @author TriDiamond
 * @version v1.0.0
 */

module.exports.parseHTML = function (html) {
  // Here we just print out the html content for now.
  console.log(html); 
};

Implementing HTML parser

We are going to use the Finite State Machine (FSM) to implement our HTML parser.

Using the FSM to analyze the HTML context
In the HTML standards, there are states rules we can follow
In our mini-browser we are only going choose one part of it to implement

There is a very complete state design made in the HTML standards, these states are exactly what our FSM need to use. You can jump to the "Tokenization" part of the document to look at all the state.

Looking at this standard may cause you to feel confused and lost. That's perfectly normal. The browser engineers are the target audience of this standard.

Once we had implemented our own mini-browser, we should be able to read and understand these standards, and you will notice what we implement is very similar to what is stated in the standards.

HTML standards have 80 states, we are here to understand the browser working flow, so we will not implement all of them. Instead, we will implement only a portion of it, just enough for us to understand how it works!

Initializing the FSM

let's begin by initializing our parseHTML FSM, we will start adding code from what we have above.

/**
   * Parser
   * @filename parser.js
   * @author TriDiamond
   * @version v1.0.0
 */

const EOF = Symbol('EOF'); // EOF: end of file

function data(char) {}

/**
   * HTTP Parser
   * @param {string} html HTML context
 */
module.exports.parseHTML = function (html) {
  let state = data;

  // Iterate the HTML text content by 
  // each character of the string
  for (let char of html) {

    // Pass each string to the state machie
    state = state(char);

  }

  // When it reach the EOF string, means
  // it's the end of the content.
  state = state(EOF);
};

We used a little trick in the above example. There is always an end to every file. We need a string character to represent the end of the file in our FSM. Upon reaching this string character, we will know that we have reached the end of this HTML content.

In this case, the EOF character serves as a pointer to the end of a file, which is a meaningless character. The EOF character string is created by using a Symbol.

Parsing HTML tags

HTML has three types of tags:

Opening tag
Closing tag
Self-closing tag

We will ignore the attributes for now, just focus on parsing the tag itself first.

File: parser.js

/**
   * Parser
   * @filename parser.js
   * @author TriDiamond
   * @version v1.0.0
 */

const EOF = Symbol('EOF'); // EOF: end of file


// STATE: Start reading HTML content
// --------------------------------
// 1. If `<` is found - means start of a tag
// 2. If `EOF` is found - means end of HTML content
// 3. Other characters - continue searching
function data(char) {
  if (char === '<') {
    // Start of a tag
    return tagOpen;
  } else if (char === EOF) {
    // End of HTML content
    // Exit out of the FSM
    return;
  } else {
    // Continue searching
    return data;
  }
}

// STATE: Start of a tag
// ----------------------------------
// 1. If `/` is found - means it's a self-closing tag
// 2. If a-Z is found - means it's the tag name
// 3. Other characters - continue searching
function tagOpen(char) {
  if (char === '/') {
    // self-closing tag
    return endTagOpen;
  } else if (char.match(/^[a-zA-Z]$/)) {
    // tag name
    return tagName(char);
  } else {
    // continue searching
    return;
  }
}

// STATE: End of a tag
// --------------------------------
// 1. If a-Z is found - means it's still tag name
// 2. If `>` is found - means syntax error
// 3. If `EOF` is found - means syntax error
function endTagOpen(char) {
  if (char.match(/^[a-zA-Z]$/)) {
    return tagName(char);
  } else if (char === '>') {
    // syntax error —— Tag is not closed
  } else if (char === EOF) {
    // syntax error —— End tag is invalid
  }
}

// STATE: Tag name
// --------------------------------
// 1. If `\t`(Tab), `\n`(Space), `\f`(Stop) or space
//    are found - means attributes property detected
// 2. If `/` is found - means self-closing tag
// 3. If a-Z character found - means still is tag name
// 4. If `>` is found - means start of end tag
// 5. Other characters - continue searching 
//    for tag name
function tagName(char) {
  if (c.match(/^[\t\n\f ]$/)) {
    return beforeAttributeName;
  } else if (char === '/') {
    return selfClosingStartTag;
  } else if (c.match(/^[a-zA-Z]$/)) {
    return tagName;
  } else if (char === '>') {
    return data;
  } else {
    return tagName;
  }
}

// STATE: Tag attributes and properties
// --------------------------------
// 1. If `/` is found - means sel-closing tag
// 2. If a-Z is found - means attribute name
// 3. If `>` is found - means tag ending
// 4. If `=` is found - means attribute value start
// 5. Other cases - means attribute value
function beforeAttributeName(char) {
  if (char === '/') {
    return selfClosingStartTag;
  } else if (char.match(/^[\t\n\f ]$/)) {
    return beforeAttributeName;
  } else if (char === '>') {
    return data;
  } else if (char === '=') {
    return beforeAttributeName;
  } else {
    return beforeAttributeName;
  }
}

// STATE: Self-closing tag
// --------------------------------
// 1. If `>` found - means self-closing tag ends
// 2. if `EOF` found - syntax error
// 3. Other cases are also syntax error
function selfClosingStartTag(char) {
  if (char === '>') {
    return data;
  } else if (char === 'EOF') {
  } else {
  }
}

/**
* HTTP Parser
* @param {string} html HTML context
*/
module.exports.parseHTML = function (html) {
  let state = data;
  for (let char of html) {
    state = state(char);
  }
  state = state(EOF);
};

This isn't done yet! Hang in there buddy!, this part we only written the state changing flow. All the tag information isn't being saved.

Next we will look at how to create Element Tokens using the states we have now.

Creating Element Tokens

Right now in our FSM all we have is a switching process of each state. We need to save the information somewhere for us to use to create our DOM object later on.

In a DOM object all the HTML information are saved in Element Tokens, so we will also use that structure, at each state we will create a respective Element Token and fill in the tag's information.

Let's first look at how are we going to tackle this:

First we need to define a currentToken variable to store our current Token (This token is used to store the start and end tag information)
Then create an emit() method to receive the token (It will generate the DOM tree at the end.)

Implementation Logic of Each Method

Start reading HTML content - data()

If EOF is found

set token with {type: 'EOF'}

If text content is found

set token with {type: 'text', content: char}

Start of a tag - tagOpen()

If text content is found - means start of a tag

set token with {type: 'startTag', tagName: > ''}

In the tagName() state method, we will finish filling up the tagName value

End of a tag - endTagOpen()

If text content is found - means found end tag name emit a token with {type: 'endTag', tagName: ''}

Same as before, the tagName value will be filled in tagName() state method.

Tag name - tagName()

Here contain the core logic of HTML parsing

We will fill out the tagName value in the currentTag in this method

When we found > that means it's the end of the tag, now we can emit the currentToken and let the emit() method to hook it into our DOM tree.

Tag attributes and properties - beforeAttributeName()

If > is found - means tag ended, we can emit the currentToken

Self-closing tag - selfClosingStartTag

If > string is found - means closing of the self-closing tag

Here we need to set an extra property to our currentToken, give it a isSelfClosing: true

Then we can emit the currentToken

Now let's look at how we implement all these logic into our code.

File: parser.js

/**
   * Parser
   * @filename parser.js
   * @author TriDiamond
   * @version v1.0.0
 */

let currentToken = null;

/**
 * Emitting HTML token
 * @param {*} token
 */
function emit(token) {
  console.log(token);
}

const EOF = Symbol('EOF'); // EOF: end of file

// STATE: Start reading HTML content
// --------------------------------
// 1. If `<` is found - means start of a tag
// 2. If `EOF` is found - means end of HTML content
// 3. Other characters - continue searching
function data(char) {
  if (char === '<') {
    // Start of a tag
    return tagOpen;
  } else if (char === EOF) {
    // End of HTML content
    // Emit token
    emit({
      type: 'EOF',
    });
    return;
  } else {
    // Text content
    emit({
      type: 'text',
      content: char,
    });
    return data;
  }
}

// STATE: Start of a tag
// ----------------------------------
// 1. If `/` is found - means it's a self-closing tag
// 2. If a-Z is found - means it's the tag name
// 3. Other characters - continue searching
function tagOpen(char) {
  if (char === '/') {
    // self-closing tag
    return endTagOpen;
  } else if (char.match(/^[a-zA-Z]$/)) {
    // tag name
    currentToken = {
      type: 'startTag',
      tagName: '',
    };
    return tagName(char);
  } else {
    // continue searching
    return;
  }
}

// STATE: End of a tag
// --------------------------------
// 1. If a-Z is found - means it's still tag name
// 2. If `>` is found - means syntax error
// 3. If `EOF` is found - means syntax error
function endTagOpen(char) {
  if (char.match(/^[a-zA-Z]$/)) {
    currentToken = {
      type: 'endTag',
      tagName: '',
    };
    return tagName(char);
  } else if (char === '>') {
    // syntax error —— Tag is not closed
  } else if (char === EOF) {
    // syntax error —— End tag is invalid
  }
}

// STATE: Tag name
// --------------------------------
// 1. If `\t`(Tab), `\n`(Space), `\f`(Stop) or space
//    are found - means attributes property detected
// 2. If `/` is found - means self-closing tag
// 3. If a-Z character found - means still is tag name
// 4. If `>` is found - means start of end tag
// 5. Other characters - continue searching 
//    for tag name
function tagName(char) {
  if (char.match(/^[\t\n\f ]$/)) {
    return beforeAttributeName;
  } else if (char === '/') {
    return selfClosingStartTag;
  } else if (char.match(/^[a-zA-Z]$/)) {
    currentToken.tagName += char;
    return tagName;
  } else if (char === '>') {
    emit(currentToken);
    return data;
  } else {
    return tagName;
  }
}

// STATE: Tag attributes and properties
// --------------------------------
// 1. If `/` is found - means sel-closing tag
// 2. If a-Z is found - means attribute name
// 3. If `>` is found - means tag ending
// 4. If `=` is found - means attribute value start
// 5. Other cases - means attribute value
function beforeAttributeName(char) {
  if (char === '/') {
    return selfClosingStartTag;
  } else if (char.match(/^[\t\n\f ]$/)) {
    return beforeAttributeName;
  } else if (char === '>') {
    emit(currentToken);
    return data;
  } else if (char === '=') {
    return beforeAttributeName;
  } else {
    return beforeAttributeName;
  }
}

// STATE: Self-closing tag
// --------------------------------
// 1. If `>` found - means self-closing tag ends
// 2. if `EOF` found - syntax error
// 3. Other cases are also syntax error
function selfClosingStartTag(char) {
  if (char === '>') {
    currentToken.isSelfClosing = true;
    emit(currentToken);
    return data;
  } else if (char === 'EOF') {
  } else {
  }
}

/**
* HTTP Parser
* @param {string} html HTML context
*/
module.exports.parseHTML = function (html) {
  let state = data;
  for (let char of html) {
    state = state(char);
  }
  state = state(EOF);
};

I know you can make it through! Here we have generated the Tokens for each of our HTML tags.

However, in this section we ignored the parsing of the HTML attributes, next let's add in the part where we parse the attributes.

Parsing HTML Attributes

First let's analyze the syntax of HTML attributes, there are 3 types of syntax:

Single quote - ''
Double quote - ""
No quote

The parsing process is similar to tags, at the end of the parsing we will add these attributes onto our Token and emit it out.

Implementation Logic of Attributes

First we need to define a currentAttributes variable to store the attributes names and values.
After parsing all the attributes of a HTML tag, we will add them all into the currentToken object.

beforeAttributeName

If space, new line or return character is found, we can continue to search for attribute content

If / or > is found, we go straight to tag end state

if = or EOF is found, this is a HTML syntax error

Other situations, means start of attribute

create currentAtrribute object with {name: '', value: ''}

return attributeName state method

attributeName

If SPACE, NEW LINE, RETURN, /, > or EOF string is found, means this attribute is ending, return afterAttributeName state method.

If = string is found, means attribute name is done, attribute value is coming up next.

If \u0000 is found, throw Parse error

Any other situations means still reading the attribute name value, continue moving to the next character.

beforeAttributeValue

If SPACE, NEW LINE, RETURN, /, > or EOF string is found, we can continue reading for attribute value. Return beforeAttributeValue state method.

If " is found, means it's a double quoted attribute value, go to doubleQuotedAttributeValue state method next.

If ' is found, means it's a single quoted attribute value, go to singleQuotedAttributeValue state method next.

All other situations means no quote attribute value, to to unquotedAttributeValue(char) state method.

doubleQuotedAttributeValue

Just keep waiting for the " string, when this string is found we know it's end of reading for the property. Then we can add this attribute name and value pair into our currentAttribute token.

If \u0000 or EOF is found, this is a syntax error. Throw Parse Error.

All other situation means continue reading for the attribute value, return doubleQuotedAttributeValue state method.

singleQuotedAttributeValue

Just keep waiting for the ' string, when this string is found we know it's end of reading for the property. Then we can add this attribute name and value pair into our currentAttribute token.

If \u0000 or EOF is found, this is a syntax error. Throw Parse Error.

All other situation means continue reading for the attribute value, return singleQuotedAttributeValue state method.

afterQuotedAttributeValue

If SPACE, NEW LINE, RETURN string is found, means there are still more attributes to be read. Therefore we need to return beforeAttributeName state method.

If / string is found mean we had hit the self-closing tag ending state. Return selfClosingStartTag state method.

If > string is found, means tag is ending. At this point we can add all our formatted attribute array into currentToken and emit the token.

If EOF is found, means it's a HTML syntax error. This is a missing-whitespace-between-attributes error.

unquotedAttributeValue

If SPACE, NEW LINE, RETURN string is found, means it's the end of this attribute, add the current attribute to the currentToken. But there may have more attributes coming after this, so we need to go to beforeAttributeName state method next.

If / string is found mean we had hit the self-closing tag ending state. Return selfClosingStartTag state method.

If > string is found, means tag is ending. At this point we can add all our formatted attribute array into currentToken and emit the token.

All other situation means continue reading for the attribute value, return unquotedAttributeValue state method.

afterAttributeName

If SPACE, NEW LINE, RETURN string is found, means still haven't found the ending character of our attribute name, continue to look for attribute name characters. Return afterAttributeName state method.

If / string is found mean we had hit the self-closing tag ending state. Return selfClosingStartTag state method.

If = string is found, means attribute name is done, attribute value is coming up next.

If > string is found, means tag is ending. At this point we can add all our formatted attribute array into currentToken and emit the token.

If EOF is found, means HTML syntax error, HTML ended unexpectedly.

All other situation means attribute name started again, add the current attribute name and value key into the currentAttributes and continue reading the next attribute.

That's all for the logic, now let's look at how are we going to implement these logic into our previous code.

File: parser.js

/**
   * Parser
   * @filename parser.js
   * @author TriDiamond
   * @version v1.0.0
 */

let currentToken = null;
let currentAttribute = null;

/**
 * Emitting HTML token
 * @param {*} token
 */
function emit(token) {
  console.log(token);
}

const EOF = Symbol('EOF'); // EOF: end of file

// STATE: Start reading HTML content
// --------------------------------
// 1. If `<` is found - means start of a tag
// 2. If `EOF` is found - means end of HTML content
// 3. Other characters - continue searching
function data(char) {
  if (char === '<') {
    // Tag starting
    return tagOpen;
  } else if (char === EOF) {
    // Context ended
    emit({
      type: 'EOF',
    });
    return;
  } else {
    // Text
    emit({
      type: 'text',
      content: char,
    });
    return data;
  }
}

// STATE: Start of a tag
// ----------------------------------
// 1. If `/` is found - means it's a self-closing tag
// 2. If a-Z is found - means it's the tag name
// 3. Other characters - continue searching
function tagOpen(char) {
  if (char === '/') {
    // 自关闭标签
    return endTagOpen;
  } else if (char.match(/^[a-zA-Z]$/)) {
    // 标签名
    currentToken = {
      type: 'startTag',
      tagName: '',
    };
    return tagName(char);
  } else {
    return;
  }
}

// STATE: End of a tag
// --------------------------------
// 1. If a-Z is found - means it's still tag name
// 2. If `>` is found - means syntax error
// 3. If `EOF` is found - means syntax error
function endTagOpen(char) {
  if (char.match(/^[a-zA-Z]$/)) {
    currentToken = {
      type: 'endTag',
      tagName: '',
    };
    return tagName(char);
  } else if (char === '>') {
    // // syntax error —— Tag is not closed
  } else if (char === EOF) {
    // syntax error —— End tag is invalid
  }
}

// STATE: Tag name
// --------------------------------
// 1. If `\t`(Tab), `\n`(Space), `\f`(Stop) or space
//    are found - means attributes property detected
// 2. If `/` is found - means self-closing tag
// 3. If a-Z character found - means still is tag name
// 4. If `>` is found - means start of end tag
// 5. Other characters - continue searching 
//    for tag name
function tagName(char) {
  if (char.match(/^[\t\n\f ]$/)) {
    return beforeAttributeName;
  } else if (char === '/') {
    return selfClosingStartTag;
  } else if (char.match(/^[a-zA-Z]$/)) {
    currentToken.tagName += char;
    return tagName;
  } else if (char === '>') {
    emit(currentToken);
    return data;
  } else {
    return tagName;
  }
}

// STATE: Tag attributes and properties
// --------------------------------
// 1. If `/` is found - means sel-closing tag
// 2. If a-Z is found - means attribute name
// 3. If `>` is found - means tag ending
// 4. If `=` is found - means attribute value start
// 5. Other cases - means attribute value
function beforeAttributeName(char) {
  if (char.match(/^[\t\n\f ]$/)) {
    return beforeAttributeName;
  } else if (char === '/' || char === '>') {
    return afterAttributeName(char);
  } else if (char === '=' || char === EOF) {
    throw new Error('Parse error');
  } else {
    currentAttribute = {
      name: '',
      value: '',
    };
    return attributeName(char);
  }
}

// STATE: Attribute Name
function attributeName(char) {
  if (char.match(/^[\t\n\f ]$/) || char === '/' || char === '>' || char === EOF) {
    return afterAttributeName(char);
  } else if (char === '=') {
    return beforeAttributeValue;
  } else if (char === '\u0000') {
    throw new Error('Parse error');
  } else {
    currentAttribute.name += char;
    return attributeName;
  }
}

// STATE: Before Attribute Value
function beforeAttributeValue(char) {
  if (char.match(/^[\t\n\f ]$/) || char === '/' || char === '>' || char === EOF) {
    return beforeAttributeValue;
  } else if (char === '"') {
    return doubleQuotedAttributeValue;
  } else if (char === "'") {
    return singleQuotedAttributeValue;
  } else if (char === '>') {
    // return data;
  } else {
    return unquotedAttributeValue(char);
  }
}

// STATE: Doube Quoted Attribute Value
function doubleQuotedAttributeValue(char) {
  if (char === '"') {
    currentToken[currentAttribute.name] = currentAttribute.value;
    return afterQuotedAttributeValue;
  } else if (char === '\u0000') {
    throw new Error('Parse error');
  } else if (char === EOF) {
    throw new Error('Parse error');
  } else {
    currentAttribute.value += char;
    return doubleQuotedAttributeValue;
  }
}

// STATE: Single QUoted Attribute Value
function singleQuotedAttributeValue(char) {
  if (char === "'") {
    currentToken[currentAttribute.name] = currentAttribute.value;
    return afterQuotedAttributeValue;
  } else if (char === '\u0000') {
    throw new Error('Parse error');
  } else if (char === EOF) {
    throw new Error('Parse error');
  } else {
    currentAttribute.value += char;
    return singleQuotedAttributeValue;
  }
}

// STATE: After QUoted Attribute Value
function afterQuotedAttributeValue(char) {
  if (char.match(/^[\t\n\f ]$/)) {
    return beforeAttributeName;
  } else if (char === '/') {
    return selfClosingStartTag;
  } else if (char === '>') {
    currentToken[currentAttribute.name] = currentAttribute.value;
    emit(currentToken);
    return data;
  } else if (char === EOF) {
    throw new Error('Parse error: eof-in-tag');
  } else {
    throw new Error('Parse error: missing-whitespace-between-attributes');
  }
}

// STATE: Unquoted Attribute Value
function unquotedAttributeValue(char) {
  if (char.match(/^[\t\n\f ]$/)) {
    currentToken[currentAttribute.name] = currentAttribute.value;
    return beforeAttributeName;
  } else if (char === '/') {
    currentToken[currentAttribute.name] = currentAttribute.value;
    return selfClosingStartTag;
  } else if (char === '>') {
    currentToken[currentAttribute.name] = currentAttribute.value;
    emit(currentToken);
    return data;
  } else if (char === '\u0000') {
    throw new Error('Parse error');
  } else if (char === '"' || char === "'" || char === '<' || char === '=' || char === '`') {
    throw new Error('Parse error');
  } else if (char === EOF) {
    throw new Error('Parse error');
  } else {
    currentAttribute.value += char;
    return unquotedAttributeValue;
  }
}

// STATE: After Attribute Name
function afterAttributeName(char) {
  if (char.match(/^[\t\n\f ]$/)) {
    return afterAttributeName;
  } else if (char === '/') {
    return selfClosingStartTag;
  } else if (char === '=') {
    return beforeAttributeValue;
  } else if (char === '>') {
    currentToken[currentAttribute.name] = currentAttribute.value;
    emit(currentToken);
    return data;
  } else if (char === EOF) {
    throw new Error('Parse error');
  } else {
    currentToken[currentAttribute.name] = currentAttribute.value;
    currentAttribute = {
      name: '',
      value: '',
    };
    return attributeName(char);
  }
}

// STATE: Self-closing tag
// --------------------------------
// 1. If `>` found - means self-closing tag ends
// 2. if `EOF` found - syntax error
// 3. Other cases are also syntax error
function selfClosingStartTag(char) {
  if (char === '>') {
    currentToken.isSelfClosing = true;
    emit(currentToken);
    return data;
  } else if (char === 'EOF') {
  } else {
  }
}

/**
* HTTP Parser
* @param {string} html HTML context
*/
module.exports.parseHTML = function (html) {
  let state = data;
  for (let char of html) {
    state = state(char);
  }
  state = state(EOF);
};

Up to this point, we are finally done with parsing the HTML tag and its attribute values. Isn't it easy? 👻

Where do we go from here?! Before we wrap up this part of the code, there is one more thing to do. All of these tokens must be used to create the DOM tree object.

Building DOM Tree with Tokens

Compare to the complicated JavaScript syntax parsing, HTML syntax parsing is relatively easier already. Up until this point, we have all the HTML tag and attribute information saved inside tokens, but with these tokens laying around won't be enough for our browser to use to render our web pages.

Side note: A real browser will find certain HTML syntax errors and auto-fix them. Since we are building a mini-browser, not a complete and fully featured browser, we will not consider these features and functions.

If we going to make a complete mini-browser, we will need to use these tokens and create a DOM tree object. So, the question is "How are we going to use all these tokens to create our DOM tree?""

Let's flatten out our logic:

To create a DOM tree, the basic trick is by using a Stack Data Structure
When we bump into a starting tag, we create the Stack and push it into the stack, when we bump into the ending tag, we pop everything out of the stack, at the end we will have a full HTML tag information in the right order.
Self-closing tags will push in the stack and out of the stack when it closes, because there are not contents in between the opening and closing tags (well there is no closing tag to between with right?)

Still confused? Let's look at this:

<div>
    <p>JavaScript</p>
    <span> is fun!</span>
</div>

In a Stack Data Structure, it will look like this:

. <- Here is top of the stack
├── <div>
├── <p>
├── JavaScript
├── </p>
├── <span>
├── is fun!
├── </span>
└── </div>

For a self-closing tag:

<img src="https://example.com/img.png" />

. <- Here is top of the stack
└── <img src="https://example.com/img.png" />

From the look of it, it will go in the stack and come right back out right?

After this basic understanding of how HTML DOM is stacked and formed, let's go look at how are we going to implement this in our code.

Let's start simply by ignoring Text Nodes inside our HTML tags first.

// Default root node `document`
// All HTML start with the `document` node
let stack = [{ type: 'document', children: [] }];

// Emitting HTML token
function emit(token) {
  if (token.type === 'text') return;

  // Record the previous element - Top of Stack
  let top = stack[stack.length - 1];

  // If it's starting tag
  if (token.type == 'startTag') {
    let element = {
      type: 'element',
      children: [],
      attributes: [],
    };

    element.tagName = token.tagName;

    for (let prop in token) {
      if (prop !== 'type' && prop != 'tagName') {
        element.attributes.push({
          name: prop,
          value: token[prop],
        });
      }
    }

    // Find the matching closing tag
    top.children.push(element);
    element.parent = top;

    if (!token.isSelfClosing) stack.push(element);

    currentTextNode = null;
  } else if (token.type == 'endTag') {
    if (top.tagName !== token.tagName) {
      throw new Error('Parse error: Tag start end not matched');
    } else {
      stack.pop();
    }

    currentTextNode = null;
  }
}

That's it, now you will build a DOM tree looking like this:

.
├── `<div>`
│   ├── `<p>`
│   ├── `</p>`
│   ├── `<span>`
│   └── `</span>`
└── `</div>`

However we are still missing the Text Element inside the HTML tags, that's what we are going to do next.

Adding Text Node to DOM Tree

This is the last section of the HTML parsing, we need to add Text Node into our DOM tree object. Here are two things we need to note:

Processing a Text Node is the same as Self-Closing Tag.
Multiple Text Nodes need to be combined.

For this part let's talk less and let our code speak the truth.

File: emit() method inside parser.js

let currentToken = null;
let currentAttribute = null;
let currentTextNode = null;

// Default root node `document`
// All HTML start with the `document` node
let stack = [{ type: 'document', children: [] }];

// Emitting HTML token
function emit(token) {
  // Record the previous element - Top of Stack
  let top = stack[stack.length - 1];

  // If it's starting tag
  if (token.type == 'startTag') {
    let element = {
      type: 'element',
      children: [],
      attributes: [],
    };

    element.tagName = token.tagName;

    for (let prop in token) {
      if (prop !== 'type' && prop != 'tagName') {
        element.attributes.push({
          name: prop,
          value: token[prop],
        });
      }
    }

    // Find the matching closing tag
    top.children.push(element);
    element.parent = top;

    if (!token.isSelfClosing) stack.push(element);

    currentTextNode = null;
  } else if (token.type == 'endTag') {
    if (top.tagName !== token.tagName) {
      throw new Error('Parse error: Tag start end not matched');
    } else {
      stack.pop();
    }

    currentTextNode = null;
  } else if (token.type === 'text') {
    // Text Node processing
    if (currentTextNode === null) {
      currentTextNode = {
        type: 'text',
        content: '',
      };
      top.children.push(currentTextNode);
    }

    currentTextNode.content += token.content;
  }
}

That's it! You made it!

That's all the content for HTML parsing, next article we will talk about how CSS computes the style properties and creating the CSSOM.

Recommended Open Source Projects

Hexo Theme Aurora

Hexo Aurora Theme released today!

Benny Guo for Auroral UI ・ Mar 30 '21

#vue #javascript #typescript #hexo

Usage Document

VSCode Aurora Future theme

A futuristic dark theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 17 '21

#vscode #design #webdev #javascript

Firefox Aurora Future

Firefox theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 25 '21

#firefox #aurora #theme #webdev

Theme page

Browser Working Principles - HTTP request and parsing

Benny Guo — Thu, 13 May 2021 13:17:44 +0000

Introduction

Browser working principle is a piece of very important knowledge for frontend developers. We often use some knowledge of browser working principles to explain and understand the concept of repaint, reflow or CSS properties.

Trying to figure out how the browser works by going through all the theory is rather ineffective and it's just too boring.

Here we will start from scratch and develop a simple browser using JavaScript. By creating a simple browser on our own, we will gain a deeper understanding of the browser working principles.

Less talk more code! Let's go!

Browser rendering process

General understanding of this process:

First of all, the browser content is rendered in 5 different steps.
When we access a web page from a URL, the page is parsed by the browser and rendered as a Bitmap.
Last but not least, our graphics card renders the page so we can view it visually.

This is a browser's basic rendering process.

This part of the Frontend Advancement Series is only going to implement the basic functionality of the browser. For a real browser, it would include many more features, such as history, bookmarks management, user accounts, data syncing and many more.

Therefore the main goal for this part of the series is to have a good understanding of the browser's working principle.

In order to do that, we need to implement the entire process from URL request to Bitmap rendering.

Understanding the process of a browser

To understand the process a little deeper, we should go through each step of the process with more details:

After a URL is entered into the browser, an HTTP request is sent. The browser then parses the returned content and extracts the HTML.
After getting the HTML content, the browser will parse it and turn it into a DOM tree.
The DOM is basically nake at this time. The next step is to perform a CSS computation to mount the CSS properties onto the DOM tree. At the end, we will get a styled DOM tree.
The styled DOM tree we get after the computation is then useful to start forming out your page layout.
Each DOM will get a calculated box. (Of course, in the real browser, every CSS will generate a box, but for simplicity, we only need to calculate one box per DOM.)
Finally, we can start rendering the DOM tree, which should render CSS properties like the background-image or the background-color onto an image. Next, the user will be able to see it through the API interface provided by the operating system and the hardware driver.

Use Finite-state Machine to parse character strings

It is important to understand one more thing before we dive into some coding.

A character string parser is required in many places throughout the browser's process. We will have a tough time implementing the code if we do not have a good "logic" management system to manage these different character string's parsing processes.

Therefore we need to use a state management system called "Finite-state Machine".

So what is Finite-state Machine (FSM)?

A Finite State Machine is a model of computation based on a hypothetical machine made of one or more states. Only one single state of this machine can be active at the same time. It means the machine has to transition from one state to another to perform different actions.

A Finite State Machine is any device storing the state of something at a given time. The state will change based on inputs, providing the resulting output for the implemented changes.

The important points here are the following:

Every state is a machine
- Every machine is decoupled from each other, it is a powerful abstract mechanism
- In each machine, we can do calculations, storage, output and etc.
- All these machines receive the same input
- Each state machine itself should have no state. If we express it as a pure function, it should have no side effects.
Every machine knows the next state
- Every machine has a definite next state (Moore state machine)
- Each machine determines the next state based on input (Mealy state machine)

For an in-depth explanation of Finite-state Machine, check out the article here.

How to implement FSM in JavaScript?

Mealy state machine:

// Every function is a state
// Function's parameter is an input
function state (input) { 
  // Inside the function, we can write our code
  // for the current state

  // Return the next State function
  return state2;
}

/** ========= 
    * To run the state matching
    * ========= */
while (input) {
  state = state(input);
}

In the above code, we see that each function is a state
Then the parameter of the function is input
The return value of this function is the next state, which implies that the next return value must be a state function.
n ideal implementation of a state machine is: "A series of state functions that return a batch of state functions."
When state functions are invoked, a loop is often used to obtain the input, then state = state(input) is used to let the state machine receive input to complete the state switch.
Mealy type state machine's return value must be based on the input to return the next state.
Moore type state machine's return value is not related to input, instead, it returns a fixed state.

What if we don't want to use FSM?

Let's take a look at what we can use if we don't want to use FSM to process the character strings in our simple browser.

In the same way that you would perform a science experience, you would attempt to defend your solution by using a different method or theory. It turns out that you already have the right solution.

What we do here is the same, let's look at how to implement the parse character strings without using a state machine.

We will learn this by going through a few challenges:

Challenge 1: Find the character "a" in a character string.

function match(string) {
  for (let letter of string) {
    if (letter == 'a') return true;
  }
  return false;
}

console.log(match('I am TriDiamond'));

Easy, isn't it?

Challenge 2: Find the character "ab" in a character string without using regular expression. Try implementing it with just pure JavaScript.

Tip: Look for a and b directly, return when both are found

function matchAB(string) {
  let hasA = false;
  for (let letter of string) {
    if (letter == 'a') {
      hasA = true;
    } else if (hasA && letter == 'b') {
      return true;
    } else {
      hasA = false;
    }
  }
  return false;
}

console.log( matchAB('hello abert'));

Challange 3: Find the character "abcdef" in a character string without using regular expression. Again try implementing it with just pure JavaScript.

There are 3 ways to approach this challenge.

Method1: By using storage space and move the key pointer to find our target.

/**
 * @param {*} match String that you need to match
 * @param {*} string String that you are matching against
 */
function matchString(match, string) {
  // Break up matching string characters into an array
  const resultLetters = match.split('');
  // Break up the string characters that you are matching against into an array
  const stringArray = string.split('');
  let index = 0; // The matching index

  for (let i = 0; i <= stringArray.length; i++) {
    // Make sure the strings are absolutely matched
    // eg. "abc" and "ab" should not be matched.

    // Therefore we require the string characters have
    // the correct orders
    if (stringArray[i] == resultLetters[index]) {
      // If one matching character is found
      // index + 1 to move to the next character
      index++;
    } else {
      // If the next character is not matched
      // reset the index and match all over again
      index = 0;
    }
    // If all characters of the string is matched
    // return true immediately, which means
    // `match` string is founded in our `string`
    if (index > resultLetters.length - 1) return true;
  }
  return false;
}

console.log('Method 1', matchString('abcdef', 'hello abert abcdef'));

Method2: Using the substring function to intercept the matching string characters to check whether they are equal to the answer.

function matchWithSubstring(match, string) {
  for (let i = 0; i &lt; string.length - 1; i++) {
    if (string.substring(i, i + match.length) === match) {
      return true;
    }
  }
  return false;
}

console.log('Method 2', matchWithSubstring('abcdef', 'hello abert abcdef'));

Method 3: Search the characters one by one until you find the final result.

function match(string) {
  let matchStatus = [false, false, false, false, false, false];
  let matchLetters = ['a', 'b', 'c', 'd', 'e', 'f'];
  let statusIndex = 0;

  for (let letter of string) {
    if (letter == matchLetters[0]) {
      matchStatus[0] = true;
      statusIndex++;
    } else if (matchStatus[statusIndex - 1] && letter == matchLetters[statusIndex]) {
      matchStatus[statusIndex] = true;
      statusIndex++;
    } else {
      matchStatus = [false, false, false, false, false, false];
      statusIndex = 0;
    }

    if (statusIndex > matchLetters.length - 1) return true;
  }
  return false;
}

console.log('Method 3', match('hello abert abcdef'));

Parsing characters using a state machine

Now let's look at how we process the characters by using a state machine.

To demonstrate how to process characters using a state machine, we going to solve the 3rd challenge using state machine:

Challange 3: Find the character "abcdef" in a character string without using regular expression. Again try implementing it with just pure JavaScript.

First, let's think about how are we going to do it with state machine:

First of all, every state is a state function
We should have a start state and an end state function, which we would call them starting and ending respectively
Every state function's name represents the previous matched state of a specific character
- Eg. matchedA means the a character is being matched in the previous state function.
The logic in each state matches the next character
- Therefore the current state function is processing the next state logic.
- Eg. If the current function name is matchedA, the logic inside it is to process when the character is equal to b
If the match fails, return the start state
Because the last of the characters is an f, therefore after matchedE succeeds, we can directly return to the end state
The End state is also known as the 'Trap method' since the state transition is finished, we can let the state stay here until the loop is finished.

/**
 * Character matching state machine
 * @param {*} string
 */
function match(string) {
  let state = start;

  for (let letter of string) {
    state = state(letter); // Switch state
  }

  // If the ending state is `end` return true
  return state === end; 
}

function start(letter) {
  if (letter === 'a') return matchedA;
  return start;
}

function end(letter) {
  return end;
}

function matchedA(letter) {
  if (letter === 'b') return matchedB;
  return start(letter);
}

function matchedB(letter) {
  if (letter === 'c') return matchedC;
  return start(letter);
}

function matchedC(letter) {
  if (letter === 'd') return matchedD;
  return start(letter);
}

function matchedD(letter) {
  if (letter === 'e') return matchedE;
  return start(letter);
}

function matchedE(letter) {
  if (letter === 'f') return end(letter);
  return start(letter);
}

console.log(match('I am abcdef'));

Escalation of the challenge: Parsing of the character string "abcabx" with a state machine.

The main difference in this challenge is that the letters "ab" appears twice.
So the logic of our analysis should be:
- The first "b" is followed by a "c", while the second "b" should be followed by an "x"
- Go back to the previous state function if the character after the second "b" isn't an "x"

function match(string) {
  let state = start;

  for (let letter of string) {
    state = state(letter);
  }

  return state === end;
}

function start(letter) {
  if (letter === 'a') return matchedA;
  return start;
}

function end(letter) {
  return end;
}

function matchedA(letter) {
  if (letter === 'b') return matchedB;
  return start(letter);
}

function matchedB(letter) {
  if (letter === 'c') return matchedC;
  return start(letter);
}

function matchedC(letter) {
  if (letter === 'a') return matchedA2;
  return start(letter);
}

function matchedA2(letter) {
  if (letter === 'b') return matchedB2;
  return start(letter);
}

function matchedB2(letter) {
  if (letter === 'x') return end;
  return matchedB(letter);
}

console.log('result: ', match('abcabcabx'));

That's it!

After we had compared the parsing of a character string with and without a state machine. There is an obvious difference that we can observe.

When parsing with a state machine, the logic is much more manageable, while without a state machine it can be confusing and hard to understand.

The basics of HTTP protocol parsing

To understand the basic of the HTTP protocol, first we need to know what is the OSI Model.

The OSI Model (Open Systems Interconnection Model) is a conceptual framework used to describe the functions of a networking system.

ISO-OSI 7 layer model

HTTP

Composition:
- Application
- Representation
- Conversation

In node.js, we have a very familiar package called http

TCP

Composition:
- Network
There are two meanings for the term "internet"
- Protocol (extranet) of the application layer where the web page is located —— it is the internet that is responsible for data transmission
- Company intranet —— it's the local network build inside a company.

4G/5G/Wi-Fi

Composition:
- Data link
- Physical layer
In order to complete an accurate transmission of data
Transmissions are all done by point-to-point
There must be a direct connection for transmissions

TCP and IP

Stream
- Stream is the main concept of transmitting data in the TCP layer
- A stream is a unit that has no apparent division
- It only guarantees that the order before and after is consistent
Port
- The TCP protocol is used by the software inside the computer
- Every piece of software get the data from the network card
- The port identifies which data is allocated to which software
- Just like the net package in node.js
Package
- Packages in TCP is transported one after another
- Each package can be large or small
- The size of each package is depended on the transmission capacity of your network intermediate equipment
IP Address
- An IP address is used to locate where the package should go.
- The connection relationship on the internet is very complicated, and there will be some large routing nodes in the middle.
- When we connected to an IP address, it first connects to the address of our house cable, then goes to the telecommunication company's cable.
- If you are visiting a foreign country's IP address, you will go to the main international address
- Every IP address is a unique identifier that connects to every device on the internet
- So the IP packet find out where it needs to be transmitted through the IP address
Libnet/libpcap
- The IP protocol needs to call these two libraries in C++
- Libnet is responsible for constructing IP packets and sending them out
- Labpcap is responsible for grabbing all IP packets flowing through the network card.
- If we use switches instead of routers to build our network, we can use the labpcap package to catch many IP packages that do not belong to us

HTTP

Composition
- Request
- Response
HTTP works as a full-duplex channel, which means it can do both sending and receiving, and there is no priority relationship between them.
In particular, HTTP must first be initiated by the client with a request
Then the server comes back with a response
So every request must have a response

Implement HTTP request

HTTP requests - server-side environment preparation

Before we write our own browser, we need to set up a node.js server.

First by writing the following node.js script:

const http = require('http');

http
  .createServer((request, response) =&gt; {
    let body = [];
    request
      .on('error', err => {
        console.error(err);
      })
      .on('data', chunk => {
        body.push(chunk.toString());
      })
      .on('end', () => {
        body = Buffer.concat(body).toString();
        console.log('body', body);
        response.writeHead(200, { 'Content-Type': 'text/html' });
        response.end(' Hello World\n');
      });
  })
  .listen(8080);

console.log('server started');

Understanding HTTP Request Protocol

Before writing our client code, we need to understand the HTTP request protocol.

Let's first look at the request section of the HTTP protocol

POST/HTTP/1.1

Host: 127.0.0.1

Content-Type: application/x-www-form-urlencoded

field1=aaa&code=x%3D1

The HTTP protocol is a text type protocol, text type protocol is generally relative to the binary protocol. In another word, means that all the contents of this protocol are character strings and each byte is part of the character string.

The first line: request line and contains three parts
- Method: Eg. POST, GET
- Path: default is "/"
- HTTP and HTTP version: HTTP/1.1
Follow by headers
- Each row is split with a colon in key: value format
- Headers end with a blank line
Last part is body
- The content of this section is determined by Content-Type
- The body's content format is based on Content-Type specify,

Implement HTTP Requests

Goal:

Design an HTTP request class
Content-type is a required field with a default value
Body is in key-value format
Different Content-type affect body formatting

Request Class

class Request {
  constructor(options) {
    // Fill in the default values
    this.method = options.method || 'GET';
    this.host = options.host;
    this.port = options.port || 80;
    this.path = options.path || '/';
    this.body = options.body || {};
    this.headers = options.headers || {};

    if (!this.headers['Content-Type']) {
      this.headers['Content-Type'] = 'application/x-www-form-urlencoded';
    }
    // Convert the body format base on Content-Type
    if (this.headers['Content-Type'] === 'application/json') {
      this.bodyText = JSON.stringify(this.body);
    } else if (this.headers['Content-Type'] === 'application/x-www-form-urlencoded') {
      this.bodyText = Object.keys(this.body)
        .map(key => `${key}=${encodeURIComponent(this.body[key])}`)
        .join('&');
    }
    // Auto calculate body content length, if the length isn't valid, meaning it's an invalid request
    this.headers['Content-Length'] = this.bodyText.length;
  }
  // Sending request, return Promise object
  send() {
    return new Promise((resolve, reject) => {
      //......
    });
  }
}

Request Method

/**
 * Request method using the Request Class
 */
void (async function () {
  let request = new Request({
    method: 'POST',
    host: '127.0.0.1',
    port: '8080',
    path: '/',
    headers: {
      ['X-Foo2']: 'custom',
    },
    body: {
      name: 'tridiamond',
    },
  });

  let response = await request.end();

  console.log(response);
})();

Implement the send function

The logic of our send function:

Send function is in a form of Promise
The response content will be gradually received during the sending process
Construct the response and let the Promise resolve
Because the process receives information one by one, we need to design a ResponseParser
In this way, the parser can construct different parts of the response object while gradually receiving the response information

  send() {
    return new Promise((resolve, reject) => {
      const parser = new ResponseParser();
      resolve('');
    });
  }

Implement HTTP response

Design the `ResponseParser`

The logic of our ResponseParser:

Need a receive function that collects the character string
Then use the state machine to process the string character by character
So we need to loop each character string and then add the recieveChar function to process each of them

class ResponseParser {
  constructor() {}
  receive(string) {
    for (let i = 0; i &lt; string.length; i++) {
      this.receiveChar(string.charAt(i));
    }
  }
  receiveChar(char) {}
}

This is the basic structure of our ResponseParser.

Understanding HTTP Response Protocol

In this section, we need to parse the contents in the HTTP response. So we will first analyse the HTTP response content.

HTTP / 1.1 200 OK

Content-Type: text/html
Date: Mon, 23 Dec 2019 06:46:19 GMT
Connection: keep-alive

26
<html><body> Hello World <body></html>
0

The status line in the first line is opposite to the request line
- The first part is the version of the HTTP protocol: HTTP/1.1
- The second part is the HTTP status code: 200 (We can mark the state other than 200 as an error in our browser implementation to make it easier.)
- The third part is HTTP status: OK
Follow by the header section
- HTML requests and responses contain headers
- Its format is exactly the same as the request
- The final line of this section will be a blank line, used to divide the headers and the body content
Body part:
- The format of the body here is also determined by Content-Type
- Here is a typical format called chunked body (A default format returned by Node)
- The chunked body will start with a line with a hexadecimal number
- Follow by the content section
- Finally ended with a hexadecimal 0, this is the end of the whole body

Implement the logic of send request

After we have a good understanding of the response protocol, we need a working send request to test and implement our Response Parser.

Design thoughts:

Supports existing connections or adding new connections
Passing the received data to the parser
Resolve the Promise base on the parser's status

Let's see how we implement this.

  send(connection) {
    return new Promise((resolve, reject) =&gt; {
      const parser = new ResponseParser();
      // First check if connection is avaliable
      // If not use Host and Port to create a TCP connection
      // `toString` is used to build our HTTP Request
      if (connection) {
        connection.write(this.toString());
      } else {
        connection = net.createConnection(
          {
            host: this.host,
            port: this.port,
          },
          () => {
            connection.write(this.toString());
          }
        );
      }
      // Listen to connection's data
      // Pass the data to the parser
      // If parser had finished, we can start the resolve
      // Then break off the connection
      connection.on('data', data => {
        console.log(data.toString());
        parser.receive(data.toString());

        if (parser.isFinished) {
          resolve(parser.response);
          connection.end();
        }
      });
      // Listen to connection's error
      // If the request had an error,
      // first reject this Promise
      // Then break off the connection
      connection.on('error', err => {
        reject(err);
        connection.end();
      });
    });
  }
  /**
   * Building HTTP Request text content
   */
  toString() {
    return `${this.method} ${this.path} HTTP/1.1\r
      ${Object.keys(this.headers)
        .map(key =&gt; `${key}: ${this.headers[key]}`)
        .join('\r\n')}\r\r
      ${this.bodyText}`;
  }

Implement the RequestParser Class

Now let's implement the logic for our RequestParser Class.

Logic:

Response must be constructed by sections, so we are going to use Response Parser to assemble it.
Use a state machine to analyze the text structure

Parsing the header

class ResponseParser {
  constructor() {
    this.state = this.waitingStatusLine;
    this.statusLine = '';
    this.headers = {};
    this.headerName = '';
    this.headerValue = '';
    this.bodyParser = null;
  }

  receive(string) {
    for (let i = 0; i &lt; string.length; i++) {
      this.state = this.state(string.charAt(i));
    }
  }

  receiveEnd(char) {
    return receiveEnd;
  }

  /**
   * Waiting status line context
   * @param {*} char
   */
  waitingStatusLine(char) {
    if (char === '\r') return this.waitingStatusLineEnd;
    this.statusLine += char;
    return this.waitingStatusLine;
  }

  /**
   * Waiting for status line ends
   * @param {*} char
   */
  waitingStatusLineEnd(char) {
    if (char === '\n') return this.waitingHeaderName;
    return this.waitingStatusLineEnd;
  }

  /**
   * Waiting for the Header name
   * @param {*} char
   */
  waitingHeaderName(char) {
    if (char === ':') return this.waitingHeaderSpace;
    if (char === '\r') return this.waitingHeaderBlockEnd;
    this.headerName += char;
    return this.waitingHeaderName;
  }

  /**
   * Waiting for Header empty space
   * @param {*} char
   */
  waitingHeaderSpace(char) {
    if (char === ' ') return this.waitingHeaderValue;
    return this.waitingHeaderSpace;
  }

  /**
   * Waiting for the Header value
   * @param {*} char
   */
  waitingHeaderValue(char) {
    if (char === '\r') {
      this.headers[this.headerName] = this.headerValue;
      this.headerName = '';
      this.headerValue = '';
      return this.waitingHeaderLineEnd;
    }
    this.headerValue += char;
    return this.waitingHeaderValue;
  }

  /**
   * Waiting for the Header ending line
   * @param {*} char
   */
  waitingHeaderLineEnd(char) {
    if (char === '\n') return this.waitingHeaderName;
    return this.waitingHeaderLineEnd;
  }

  /**
   * Waiting for Header content end
   * @param {*} char
   */
  waitingHeaderBlockEnd(char) {
    if (char === '\n') return this.waitingBody;
    return this.waitingHeaderBlockEnd;
  }
}

Parsing the body content

Logic:

Response body may have a different structure depending on the Content-Type, so we will use the structure of the sub-parser to solve this problem
Take ChunkedBodyParser as an example, we also use a state machine to deal with the format of the body

Adding a state function for body parsing:

/**
 * Response 解析器
 */
class ResponseParser {
  constructor() {
    this.state = this.waitingStatusLine;
    this.statusLine = '';
    this.headers = {};
    this.headerName = '';
    this.headerValue = '';
    this.bodyParser = null;
  }

  /** ... Previous codes ... **/

  /**
   * Waiting for Header content end
   * @param {*} char
   */
  waitingHeaderBlockEnd(char) {
    if (char === '\n') return this.waitingBody;
    return this.waitingHeaderBlockEnd;
  }

  /** Adding a state function for body parsing **/

  /**
   * Waiting for body content
   * @param {*} char
   */
  waitingBody(char) {
    this.bodyParser.receiveChar(char);
    return this.waitingBody;
  }
}

Adding ChunkedBodyParser class:

class ChunkedBodyParser {
  constructor() {
    this.state = this.waitingLength;
    this.length = 0;
    this.content = [];
    this.isFinished = false;
  }

  receiveChar(char) {
    this.state = this.state(char);
  }

  /**
   * Waiting for Body length
   * @param {*} char
   */
  waitingLength(char) {
    if (char === '\r') {
      if (this.length === 0) this.isFinished = true;
      return this.waitingLengthLineEnd;
    } else {
      // Convert the hexdecimal number
      this.length *= 16;
      this.length += parseInt(char, 16);
    }
    return this.waitingLength;
  }

  /**
   * Waiting for Body line end
   * @param {*} char
   */
  waitingLengthLineEnd(char) {
    if (char === '\n') return this.readingTrunk;
    return this.waitingLengthLineEnd;
  }

  /**
   * Reading Trunk content
   * @param {*} char
   */
  readingTrunk(char) {
    this.content.push(char);
    this.length--;
    if (this.length === 0) return this.waitingNewLine;
    return this.readingTrunk;
  }

  /**
   * Waiting for a new line
   * @param {*} char
   */
  waitingNewLine(char) {
    if (char === '\r') return this.waitingNewLineEnd;
    return this.waitingNewLine;
  }

  /**
   * Waiting for line end
   * @param {*} char
   */
  waitingNewLineEnd(char) {
    if (char === '\n') return this.waitingLength;
    return this.waitingNewLineEnd;
  }
}

Finally

In this section of the Frontend Advancement Series, we have implemented the browser HTTP Request, HTTP Response parser.

In the next section, we will talk about how to use the parsed HTTP to build a DOM tree.

Happy coding!~

Recommended Open Source Projects

Hexo Theme Aurora

Hexo Aurora Theme released today!

Benny Guo for Auroral UI ・ Mar 30 '21

#vue #javascript #typescript #hexo

Usage Document

VSCode Aurora Future theme

A futuristic dark theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 17 '21

#vscode #design #webdev #javascript

Firefox Aurora Future

Firefox theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 25 '21

#firefox #aurora #theme #webdev

Theme page

Font-end learning and knowledge system

Benny Guo — Wed, 05 May 2021 18:50:57 +0000

「0」Intro

Many people feel very confused when they face higher knowledge mountains in their front-end career.

Many of my fans had texted me asking "What should I learn?" and "When will I be finished with this front-end learning loop?"

Don't worry, buddy! Once you read this article, you'll know a bit more about what makes a great front-end engineer. Creating a knowledge management system and building a career development plan will make sense to you.

「1」How do you define a good developer?

Think about the following question:
"What does an excellent front-end developer possess?"

After interviewing many front-end engineers, here is what I got:

Familiarized with frameworks such as Angular, React, Vue.
Knowing the web standards is essential.
Experience with Webpack and Babel
Ten years of experience in development
Performance tuning and optimization
Object-oriented programming and closures
A NodeJS expert.

A good front-end developer will have more important qualities than these.

So how to determine the quality of a developer?

🔭 Knowledge of the field

Different type of engineer has different field knowledge
Different fields of knowledge are interrelated

🧬 Abilities and potentials

As long as the ability of the individual matches the requirements of the role, you will be promoted
Potential is the upper limit of our abilities, anyone with the same potential should be able to write the same quality code.
The difference is the ability to come up with solutions to problems and solve them independently
Each engineer will be assigned with a KPI (Eg. 30% - outstanding, 60% - normal, 10% - unqualified)
Each individual's potential will be assessed and a prediction will be given as to when they will be promoted.

🗓 Career planning

According to Alibaba's engineer ranking

Mid-level Engineer (P5) —— Ability to work independently
Senior Engineer (P6) —— Core developer, work as a team leader, who can lead the team to break through difficult problems.
Expert (P7) —— A field expert. Become an authority in one area, drive development, and apply common solutions to solve problems in this area.
Manager (P8)

📈 How to get promoted?

Continuously improve technical capabilities
Precipitation is occurring in your area of speciality

🏆 Achievements

Your achievements prove your abilities and potentials.
A reflection of what you have contributed and took on in your previous work experiences

「2」How to become a good developer?

Career Planning

You are the owner of your career.

Many thinks career planning is the responsibility of their "boss", their "parents" or their "teachers". The reality is, it is you who is responsible for your career.
Many get out of shape when implementing their career planning. They think that career promotion is the consciousness of the boss.
Do we just give up if our boss doesn't promote us? No! we are the boss of our career, we need to take full responsibility for our career growth. So we have to act for ourselves!
A career plan is the most important aspect of a successful professional life, therefore we should act as if we are in charge of it, and we have to come up with a solution to ensure our results.
You need to know when and where you want to be in the company or your role, and if not, how to get there!
So we should always keep a sense of urgency and crisis in the workplace
No one will be responsible for your career success. This is why you need to exert enough effort. The same applies to career development.

Advancements in Career

Growth —— Develop your skills through continuous learning
Achievement —— As you advance in skill, you will acquire achievements that can be used to prove your technical abilities
Promotion —— You can be promoted to the next level by demonstrating your achievements.

Hence, achievements are paramount to career advancement. What achievements can we acquire in the course of our daily routine?

There are three main types of achievements we can obtain!

🎡 Achieve Project Objectives

Business objectives
- Understand the core objectives of your company's business
- Turn objectives into metrics (UV, PV, page load time, profits)
Technical solutions
- Conversion of the business target to technical target
- Put the plan to work and complete small size trials
Implementing the plan
- Determine implementation objectives and participants
- Manage the implementation progress
Result evaluation
- Collecting data and form into a data report
- Then report to a superior
- No promotion is possible without positive results, else all you get is a "diligent and hard-working" reputation

A study case:

Business goals & metrics:

Improve click-through rate

Technical solution:

Add gesture operation to TAB component

Implementation 1:

Add corresponding feature

Push to production*

Result:

Click-through rate increased 3 times

Implementation 2:

Compile general TAB components

Helping making a good base UI/UX guideline

⚙️ Achieve Development Automation

Target
- Improved code quality
- Improved development efficiency
Plan and implementation
- Rules and regulations
- Repo / Pacakges
- Tools / CLI
- Systems
End result
- Ability to monitor production

A case study: Prevention of XSS attacks

Target:

XSS attack white hat feedback vulnerability

Technical solution:

Organize the security manual

Review the historical code

Building a code scan tool

Implementation:

Change the code release process

Review the entire code

Preach to all the front-end developers

Result:

XSS vulnerabilities significantly reduced

🪄 Resolving technical difficulties

Target a:
- Recognized technical difficulties
- One that can affect the business
Planning and implementation
- Rely on solid programming skills

A case study:

Case: Crawling commodity prices

Background:

In a browser plug-in project, you need to crawl the price comparison of each website

But each website will use image prices and other means to defend

Solution:

Introduce Javascript terminal digital recognition technology

Rely on AI technology to solve the problem

「3」Front-end and learning methods

Being a good developer requires us to know how to study as a front-end developer.

🔮 A better approach to learning

No pressure should be put on learning the knowledge system, instead one should try to understand the integrity and the framework of the knowledge system.

A knowledge system should be accomplished by analyzing things from multiple perspectives.

Many developers are questioning whether what they have learned is useful.

The following tips can help you approach learning better:

Although some knowledge is useless, some small points may be useful during the course of a working day

Most of the time there's no way to find out for certain how effective a piece of knowledge is

In order to extract the most meaningful knowledge, we must have a knowledge system to sort them.

We need to have a good study habit, whenever we see a new the knowledge we should try to understand it regardless of its usefulness

The fact is that we do not have to fully comprehend new knowledge; it becomes clearer once you are faced with it again and again from different situations or environments.

Conclusion is:
"Combining systematic learning with random learning"

🏗 Front-end skill model

Once we have a better understanding of the learning approach, then we must then decide which categories to focus our learning efforts on.

Domain Knowledge
- Different fields will have different knowledge
- Learning by practice
Frontend knowledge
- Establish a knowledge system
- The focus is to mastery the knowledge not the magnitude of the knowledge system
Abilities —— deliberate practice
- Programming ability
- Architecture capability
- Engineering capability

"Frontend technology is not a magic spell book,
the real ability is obtained by practice."

📚 Learning method

Relating pieces of knowledge

To build our knowledge system, we want to pursue completeness but not perfection.

The first key goal is to find the key logical relationship between all your knowledge.

Sequential relation
- Record the relationship between processes in a system.
Combination relationship
- Find the relationship between combinations
Perspective relationship
- When you look at knowledge, Each perspective you see is a complete thing on its own
- Understand a complete thing by looking at a piece of knowledge from various angles
Classification relationship
- Sort knowledge by category

Organizing and sorting

You can find clues by searching or by reading manuals or articles.
Find multiple sources of information.
Make sure your knowledge is complete by comparing and strengthening it.

Finding the source

Source of knowledge
- Establish the knowledge base from the source and enhance the credibility
- Find the earliest published papers or magazines
- Study the initial implementation case
Standards and documents
- Read the ones with high authorities
- Make a good comparison to ensure accuracy

「4」Knowldege System

Following all the tips in this article, you will able to build the above knowledge system.

Building a complete knowledge system by filling in missing pieces of knowledge while learning. In the end, this knowledge system will be carved into your head, not just in your computer.

Recommended Open Source Projects

Hexo Theme Aurora

Hexo Aurora Theme released today!

Benny Guo for Auroral UI ・ Mar 30 '21

#vue #javascript #typescript #hexo

Usage Document

VSCode Aurora Future theme

A futuristic dark theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 17 '21

#vscode #design #webdev #javascript

Firefox Aurora Future

Firefox theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 25 '21

#firefox #aurora #theme #webdev

Theme page

Firefox theme `Aurora Future`

Benny Guo — Sun, 25 Apr 2021 13:19:31 +0000

Recently I had switched back to Firefox for web development.

Thinking back, I had left Firefox since 4-5 years ago, I remember Firefox had the best layout debug tool called firebug.

Not sure how many of the web developer now days hear of this tool.

Therefore I decided to make a Aurora Theme for Firefox as well and here it is!

(Side note I still love Chrome, I still switch between the two from time to time, but a lot more on Firefox now days, because for some reason Chrome is very laggy for my MacBookPro. Also I am thinking of making a chrome theme as well~)

Here is place to download this theme: https://addons.mozilla.org/en-US/firefox/addon/aurora-future/

A futuristic dark theme `Aurora Future`

Benny Guo — Sat, 17 Apr 2021 16:32:34 +0000

Aurora Future

This theme is inspired by Adam Argyle's Kung-Fury-Theme.

Purple, pink and blue are the main colors of this theme. These colors combine give us a futuristic feel. The color scheme is inspired by Adam Argyle's theme.

However I have change the colors, to follow the colors used in Auroral-UI, basically the same colors used in my Hexo Theme Aurora.

Why make this theme?

First is because I fall in love with Adam Argyle's sublime3 theme when I watched his Youtube videos. The syntax highlight he did is just pure awesomeness, but his VSCode's version of the theme is very outdated, also the color saturation is just a bit too high for my own liking, therefore I decided to make my own base on his ideas. Another thing is that I would like to have my own custom theme to go with my Auroral-UI project.

I have used many themes in VSCode, but most of them used way too many colors. After writing hours of codes, my eyes gets very tiring of looking at those colors.

Therefore the main goal of this theme is to use the main colors of Auroral-UI and draw back all the distractions from the colors. Which help us focus on what is important and keep the visual harmony in the theme.

Features

I follow the same goals as Adam Argyle's sublime3 theme, because I couldn't agree more on his design concepts.

Accentuates the signal of your code, while pushing back the noise
Colors, italics, and subdued colors are meaningful to help you understand the code via consistent coloring, there's rhyme and reason for the choices beyond visual harmony
If your code has no errors, then it looks pythonic (noise reduction)
Dark mode that's been tested heavily for day and night use
Leverages cursive if you're using Dank Mono or Operator Mono

Recommended combination

Font: use Dank Mono or Operator Mono
Icon theme: Material Theme Icon Dark

Screenshots

JavaScript

Vue

Vue-HTML

Json

Yaml

How to get this theme?

Go search for the theme from VSCode extension, use the theme name Aurora Future.

Go to the extension's page here

Hope you enjoy!~

Hexo Aurora Theme released today!

Benny Guo — Tue, 30 Mar 2021 16:16:43 +0000

New theme hexo-theme-aurora version 1.0.0 released today!

Aurora is a next generation theme using Aurora colours and UI elements. It give you the feel of smooth flow of colours and a futuristic feeling.

Introduction

My very first theme is hexo-theme-obsidian which got quite a bit of attention. A lot of bloggers loved this theme, but later on the theme gets a bit hard to maintain due to the fact that it's not using a framework like Vue or React.

Ever since than, I start to think about refactoring that theme using Vue, and that's when I started my journey of using Vue to build a Hexo theme.

As I was refactoring Obsidian, I figure some of the user interface isn't as fine tuned as it should be, and as I notice the "Aurora UI" trend in UI designs. (Know as the trend of using gradient colors for UI elements.), at the end I decided to make a brand new theme using the popular UI design elements.

Aurora is a next generation theme using Aurora colours and UI elements. It give you the feel of smooth flow of colours and a futuristic feeling.

Compare to Obsidian

The main significant difference between Obsidian and Aurora, is that Aurora is base on Vue3, which means its performance and user interactions are much better than Obsidian.

Obsidian uses vanilla JavaScript and JQuery to support most of the user interaction and blog features. Of course, I am not saying using vanilla JavaScript and JQuery cannot build a fast and high-quality blog application. It's just using Vue is much more enjoyable for the developer and even more for the readers or viewers.

Why? Because using Vue, we could turn our blog into a Single Page Application (SPA), therefore users will notice fewer distractions during page to page hopping. For an article-reading blog site, SPA is best for this kind of purpose.

Why vue3?

Many asked me that: "Why use Vue3 with Hexo, why not just use the default EJS template system with vuejs."

Of course, I could add in vuejs with the support of the default ejs template system. Honestly, it wasn't easy making the blog fully customizable using fully restful APIs. It could have been much easier with EJS templates.

But in the modern frontend development trend, most web applications had switch to frameworks like Vue, React or Angular, because they are much easier and quicker for development. With the support of SPA for these frameworks, it would improve the user accessibility and interactions of the application.

Thinking through the pros and cons, I had decided to use Vue3 as the framework for developing Aurora.

What's in Aurora?

Let's us take a look at the feature list and plans for Aurora:

⭐️ Features

✔ Featured section - Can add a feature meta to any post, those will be pinned into the feature section on the home page.
✔ Multi-language support - Can dynamically switch the language of the main texts in the application.
✔ Statistics - Statistic of article words, posts, categories and tags counts.
✔ Article reading time - Show the estimated reading time of an article.
✔ Mathjax support - Support mathjax syntax in markdown files.
✔ SPA - No page loading or freezing from page to page.
NEW Comment plugins - Support the using Gitalk or Valine plugins
NEW Comment plugins - Support the using Gitalk or Valine plugins
NEW Blog search - Enable to search all the articles in the blog.
NEW Latest comments - Added a widget showing the latest comments, supported using Gitalk or Valine.

🎨 Theme

✔ Light and dark - Light and dark theme support.
✔ Magazine Layout - Modern magazine grid layout.
✔ Gradient Colours - Modern gradient colour styles.
✔ Timeline styled archives - Archived posts into a timeline format.

🛠 Configuration

✔ Beian Config - This feature is especially for China users, for those that need to put Beian info at the footer of the blog.
✔ Customizable menu - Can customize your menu with internal links, external links, custom pages.
NEW Post navigation - Can navigate to the previous and next article at the end of the article page.
NEW Custom pages - Can add any custom pages, need to use the customized menu to display in the menu.
NEW Page navbar - Can set up a sidebar on the specific page, act as the sub-menu of the page.

What is yet to come?

Everything listed here are currently under development.

WIP Related posts - _Show a list of the related posts at the end of an article. _

What might be coming?

What Aurora might add in the future.

Will host voting events in Discord from time to time. You are welcome to come build Aurora together!

Image Gallery - Adding an image gallery page.
Links of friends - Allow you to list all your friends' blogs.
Tweeting - Allow posting tweets like tweeter for you and your visitors.

Get the theme now!

Github Repo: https://github.com/auroral-ui/hexo-theme-aurora
Usage document: https://aurora.tridiamond.tech/

DEV Community: Benny Guo

Browser Working Principle - HTML Parsing

Introduction

Refactoring file structure

Implementing HTML parser

Initializing the FSM

Parsing HTML tags

Creating Element Tokens

Parsing HTML Attributes

Building DOM Tree with Tokens

Adding Text Node to DOM Tree

Recommended Open Source Projects

Hexo Theme Aurora

Hexo Aurora Theme released today!

Benny Guo for Auroral UI ・ Mar 30 '21

VSCode Aurora Future theme

A futuristic dark theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 17 '21

Firefox Aurora Future

Firefox theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 25 '21

Browser Working Principles - HTTP request and parsing

Introduction

Browser rendering process

Understanding the process of a browser

Use Finite-state Machine to parse character strings

So what is Finite-state Machine (FSM)?

How to implement FSM in JavaScript?

What if we don't want to use FSM?

Parsing characters using a state machine

The basics of HTTP protocol parsing

ISO-OSI 7 layer model

Implement HTTP request

HTTP requests - server-side environment preparation

Understanding HTTP Request Protocol

Implement HTTP Requests

Request Class

Request Method

Implement the send function

Implement HTTP response

Design the ResponseParser

Understanding HTTP Response Protocol

Implement the logic of send request

Implement the RequestParser Class

Parsing the header

Parsing the body content

Finally

Recommended Open Source Projects

Hexo Theme Aurora

Hexo Aurora Theme released today!

Benny Guo for Auroral UI ・ Mar 30 '21

VSCode Aurora Future theme

A futuristic dark theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 17 '21

Firefox Aurora Future

Firefox theme `Aurora Future`

Benny Guo for Auroral UI ・ Apr 25 '21

Font-end learning and knowledge system

「0」Intro

「1」How do you define a good developer?

So how to determine the quality of a developer?

🔭 Knowledge of the field

🧬 Abilities and potentials

🗓 Career planning

📈 How to get promoted?

🏆 Achievements

「2」How to become a good developer?

Career Planning

Advancements in Career

🎡 Achieve Project Objectives

⚙️ Achieve Development Automation

🪄 Resolving technical difficulties

「3」Front-end and learning methods

🔮 A better approach to learning

🏗 Front-end skill model

📚 Learning method

Relating pieces of knowledge

Organizing and sorting

Finding the source

「4」Knowldege System

Design the `ResponseParser`