DEV Community: ayou

Implementing Webpack from Scratch, But in Rust - [5] Support Customized JS Plugin

ayou — Wed, 27 Nov 2024 08:31:57 +0000

Referencing mini-webpack, I implemented a simple webpack from scratch using Rust. This allowed me to gain a deeper understanding of webpack and also improve my Rust skills. It's a win-win situation!

Code repository: https://github.com/ParadeTo/rs-webpack

This article corresponds to the Pull Request: https://github.com/ParadeTo/rs-webpack/pull/6

The previous article implemented the Plugin system on the Rust side, but left a loose end on how to integrate plugins developed by users using JavaScript into rs-webpack. This article will cover the implementation.

For example, if I have developed a plugin in JavaScript, how can I make it work?

module.exports = class MyPlugin {
    apply(compiler) {
        compiler.hooks.beforeRun.tap('myplugin', (compiler) => {
            console.log("before run", compiler)
        })
    }
}

Since we have already referenced rspack earlier, let's continue to follow its lead. After studying it, I found that its approach is roughly as shown in the following diagram:

Our custom JS plugin myPlugin will be passed from rs-webpack-cli to rs-webpack-core. In rs-webpack-core, it uses a library called @rspack/lite-tapable, developed by the rspack team, to create a beforeRun Hook:

export class Compiler {
    bindingRsWebpack: BindingRsWebpack
    hooks: {
        beforeRun: liteTapable.SyncHook<[string]>;
    }
    ...
}

Similar to Rust, when the Compiler is initialized, it iterates through all the plugins and executes their apply methods:

constructor(props: RawConfig) {
    const {plugins} = props
    plugins.forEach(plugin => {
        plugin.apply(this)
    })
}

Then, through a series of operations, it wraps a function called register_before_run_taps and passes it to Rust. register_before_run_taps wraps the call to the beforeRun Hook's call function:

this.registers = {
    registerBeforeRunTaps: this.#createHookRegisterTaps(
        RegisterJsTapKind.BeforeRun,
        () => this.hooks.beforeRun,
        queried => (native: string) => {
            // beforeRun.call 
            queried.call(native);
        }
    ),
}
this.bindingRsWebpack = new BindingRsWebpack(props, this.registers)

After this function is executed, it returns an array, and each element in the array can serve as an interceptor for the before_run Hook in Rust (only the call method is implemented):

#[async_trait]
impl Interceptor<BeforeRunHook> for RegisterBeforeRunTaps {
  async fn call(
    &self,
    hook: &BeforeRunHook,
  ) -> rswebpack_error::Result<Vec<<BeforeRunHook as Hook>::Tap>> {
    if let Some(non_skippable_registers) = &self.inner.non_skippable_registers {
      if !non_skippable_registers.is_non_skippable(&RegisterJsTapKind::BeforeRun) {
        return Ok(Vec::new());
      }
    }
    let js_taps = self.inner.call_register(hook).await?;
    let js_taps = js_taps
      .iter()
      .map(|t| Box::new(BeforeRunTap::new(t.clone())) as <BeforeRunHook as Hook>::Tap)
      .collect();
    Ok(js_taps)
  }
}

In rswebpack_binding, these interceptors are applied through the JsHooksAdapterPlugin:

impl Plugin for JsHooksAdapterPlugin {
  fn name(&self) -> &'static str {
    "rspack.JsHooksAdapterPlugin"
  }

  fn apply(&self, _ctx: PluginContext<&mut ApplyContext>) -> rswebpack_error::Result<()> {
    _ctx
      .context
      .compiler_hooks
      .before_run
      .intercept(self.register_before_run_taps.clone());
  }
}

PS: The call function in the interceptor is executed each time the call function of the Hook is invoked. For example, in the following example:

const hook = new SyncHook(['arg1', 'arg2'])

hook.tap('test', (...args) => {
  console.log('test', ...args)
})

hook.intercept({
  // trigger when execute hook.call
  call: (...args) => {
    console.log('Execute interceptor call', ...args)
  },
})

hook.call('a1', 'a2')

// log
Execute interceptor call a1 a2
test a1 a2

When the before_run in Rust calls call, these interceptors' call functions will also be executed, and then the beforeRun.call wrapped in these interceptors on the JS side will be executed, triggering the execution of the corresponding Tap function in myPlugin.

With these steps, the entire Plugin system is completed. The complete changes can be seen here. I won't go through the code one by one, but by following the order in the diagram, you should be able to understand it.

The original intention of this series of articles was to deepen the understanding of webpack by reimplementing it. However, I found that my Rust skills were limited, and I didn't have the ability to implement a Plugin system. Most of the time was spent on integrating Rspack.

During this process, I realized that there are many areas that I don't understand well. I'll mark them down for future study:

Napi, such as ThreadsafeFunction. Combining this with Node.js can achieve many things. I'll see if I can come up with some examples later.
Asynchronous processing and tokio in Rust.
Rust concurrent programming: multithreading, channels, etc.
Macro programming in Rust, which is difficult to write and debug.

Please kindly give me a star!

Implementing Webpack from Scratch, But in Rust - [4] Implement Plugin System

ayou — Tue, 05 Nov 2024 10:20:24 +0000

Referencing mini-webpack, I implemented a simple webpack from scratch using Rust. This allowed me to gain a deeper understanding of webpack and also improve my Rust skills. It's a win-win situation!

Code repository: https://github.com/ParadeTo/rs-webpack

This article corresponds to the Pull Request: https://github.com/ParadeTo/rs-webpack/pull/5

We know that Webpack uses Tapable to implement its plugin system. So, it seems reasonable to write a Rust version inspired by it. However, implementing it turned out to be not as simple as expected. Let's take SyncHook as an example. A very simple version of it can be implemented in JavaScript like this:

class SyncHook {
  constructor() {
    this.taps = []
  }
  tap(options, fn) {
    this.taps.push(fn)
  }
  call() {
    this.taps.forEach((fn) => fn(...arguments))
  }
}

const hook = new SyncHook(['param1', 'param2']) // Create a hook object
hook.tap('event1', (param) => console.log('event1:', param))
hook.tap('event2', (param) => console.log('event2:', param))
hook.tap('event3', (param, param2) => console.log('event3:', param, param2))
hook.call('hello', 'world')

Let's try to implement it in Rust. We might write code like this:

struct SyncHook<Arg: Copy, R> {
    taps: Vec<Box<dyn Fn(Arg) -> R>>
}

impl<Arg: Copy, R> SyncHook<Arg, R> {
    fn tap(&mut self, f: Box<dyn Fn(Arg) -> R>) {
        self.taps.push(f);
    }

    fn call(&self, a: Arg) {
        for tap in self.taps.iter() {
            tap(a);
        }
    }
}

fn main() {
    let mut sync_hook = &mut SyncHook{taps: vec![]};
    sync_hook.tap(Box::new(|arg| {
        println!("event {}", arg);
    }));
    sync_hook.call("hello")
}

Note that Arg must be constrained to be Copy, otherwise calling tap(a) will result in an error. The above code can run correctly, but it only supports the case where call takes a single argument. However, this can be solved using macros. We can pre-generate a set of structs that support different numbers of arguments:

struct SyncHook1<Arg1: Copy, R> {
    taps: Vec<Box<dyn Fn(Arg1) -> R>>
}

struct SyncHook2<Arg1: Copy, Arg2: Copy, R> {
    taps: Vec<Box<dyn Fn(Arg1, Arg2) -> R>>
}

However, there is still a problem: it does not support passing arguments of type &mut T:


struct Compiler {
    name: String
}

fn main() {
    let mut sync_hook = &mut SyncHook{taps: vec![]};
    sync_hook.tap(Box::new(|arg| {
        println!("event {}", arg);
    }));
    let compiler = &mut Compiler {name: String::from("test")};
    sync_hook.call(compiler)
}

The above code will throw an error: "the trait Copy is not implemented for &mut Compiler".

It seems that implementing a similar functionality is quite challenging for me as a beginner. So, let's take a look at Rspack and see how they approach it.

Upon investigation, I found that Rspack does not have a generic SyncHook. Instead, it defines each hook separately using macros. Let's try to incorporate it into our project.

First, let's copy the code from Rspack's source code, specifically the crates/rspack_macros and crates/rspack_hook directories, into the crates directory of our rs-webpack project. Make sure to rename the directories accordingly.

├── crates
│   ├── rswebpack_hook
│   └── rswebpack_macros

Then, let's add a new module called rswebpack_error as our unified error handling module:

// lib.rs
use anyhow::Result as AnyhowResult;

pub type Result<T> = AnyhowResult<T>;

Finally, we need to modify the dependencies in these libraries accordingly. For example, in rswebpack_hook, change Result from rspack_error::Result to rswebpack_error::Result.

Let's write a demo to test it:

use rswebpack_macros::{define_hook, plugin, plugin_hook};

struct People {
    name: String
}

define_hook!(Test: SyncSeries(people: &mut People));

#[plugin]
struct TestHookTap1;

#[plugin_hook(Test for TestHookTap1)]
fn test1(&self, people: &mut People) -> Result<()> {
    people.name += " tap1";
    Ok(())
}

#[plugin]
struct TestHookTap2;

#[plugin_hook(Test for TestHookTap2)]
fn test2(&self, people: &mut People) -> Result<()> {
    people.name += " tap2";
    Ok(())
}

fn main() {
    let mut test_hook = TestHook::default();
    test_hook.tap(test1::new(&TestHookTap1::new_inner()));
    test_hook.tap(test2::new(&TestHookTap2::new_inner()));
    let people = &mut People { name: "ayou".into() };
    test_hook.call(people);
    println!("{}", people.name); // ayou tap1 tap2
}

Let me explain the code above. First, we define a hook using define_hook!(Test: SyncSeries(people: &mut People));. It expands to something like this:

pub trait Test {
    fn run(&self, people: &mut People) -> rswebpack_hook::__macro_helper::Result<()>;
    fn stage(&self) -> i32 { 0 }
}
pub struct TestHook {
    taps: Vec<Box<dyn Test + Send + Sync>>,
    interceptors: Vec<Box<dyn rswebpack_hook::Interceptor<Self> + Send + Sync>>,
}
impl rswebpack_hook::Hook for TestHook {
    type Tap = Box<dyn Test + Send + Sync>;
    fn used_stages(&self) -> rswebpack_hook::__macro_helper::FxHashSet<i32> { rswebpack_hook::__macro_helper::FxHashSet::from_iter(self.taps.iter().map(|h| h.stage())) }
    fn intercept(&mut self, interceptor: impl rswebpack_hook::Interceptor<Self> + Send + Sync + 'static) { self.interceptors.push(Box::new(interceptor)); }
}
impl std::fmt::Debug for TestHook { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { write!(f, "TestHook") } }
impl Default for TestHook { fn default() -> Self { Self { taps: Default::default(), interceptors: Default::default() } } }
impl TestHook {
    pub fn call(&self, people: &mut People) -> rswebpack_hook::__macro_helper::Result<()> {
        let mut additional_taps = std::vec::Vec::new();
        for interceptor in self.interceptors.iter() { additional_taps.extend(interceptor.call_blocking(self)?); }
        let mut all_taps = std::vec::Vec::new();
        all_taps.extend(&self.taps);
        all_taps.extend(&additional_taps);
        all_taps.sort_by_key(|hook| hook.stage());
        for tap in all_taps { tap.run(people)?; }
        Ok(())
    }
    pub fn tap(&mut self, tap: impl Test + Send + Sync + 'static) { self.taps.push(Box::new(tap)); }
}

You can see that TestHook is similar to our previous implementation, and it also generates the type Test for the tap function of TestHook.

Next, let's use a macro to implement a Tap for Test:

#[plugin]
struct TestHookTap1;

#[plugin_hook(Test for TestHookTap1)]
fn test1(&self, people: &mut People) -> Result<()> {
    people.name += " tap1";
    Ok(())
}

It expands to something like this:

struct TestHookTap1 {
    inner: ::std::sync::Arc<TestHookTap1Inner>,
}
impl TestHookTap1 {
    fn new_inner() -> Self { Self { inner: ::std::sync::Arc::new(TestHookTap1Inner) } }
    fn from_inner(inner: &::std::sync::Arc<TestHookTap1Inner>) -> Self { Self { inner: ::std::sync::Arc::clone(inner) } }
    fn inner(&self) -> &::std::sync::Arc<TestHookTap1Inner> { &self.inner }
}
impl ::std::ops::Deref for TestHookTap1 {
    type Target = TestHookTap1Inner;
    fn deref(&self) -> &Self::Target { &self.inner }
}
#[doc(hidden)]
pub struct TestHookTap1Inner;

#[allow(non_camel_case_types)]
struct test1 {
    inner: ::std::sync::Arc<TestHookTap1Inner>,
}
impl test1 { fn new(plugin: &TestHookTap1) -> Self { test1 { inner: ::std::sync::Arc::clone(plugin.inner()) } } }
impl TestHookTap1 {
    #[allow(clippy::ptr_arg)]
    fn test1(&self, people: &mut People) -> Result<()> {
        people.name += " tap1";
        Ok(())
    }
}
impl ::std::ops::Deref for test1 {
    type Target = TestHookTap1Inner;
    fn deref(&self) -> &Self::Target { &self.inner }
}
impl Test for test1 {
    #[tracing::instrument(name = "TestHookTap1::test1", skip_all)]
    fn run(&self, people: &mut People) -> Result<()> {
        TestHookTap1::test1(&TestHookTap1::from_inner(&self.inner), people )
    }
}

It may seem a bit convoluted at first, but with a few more readings, it becomes understandable.

In this way, we have implemented Tapable-like functionality in Rust. However, we have only demonstrated the usage of the simplest SyncHook for now. We will introduce other hooks in the future.

Next, let's build a plugin system based on this. As we know, implementing a plugin for webpack is typically done like this:

const pluginName = 'ConsoleLogOnBuildWebpackPlugin'

class ConsoleLogOnBuildWebpackPlugin {
  apply(compiler) {
    compiler.hooks.run.tap(pluginName, (compilation) => {
      console.log('The webpack build process is starting!')
    })
  }
}

module.exports = ConsoleLogOnBuildWebpackPlugin

We follow the same pattern and start by defining a Plugin trait to specify the characteristics a plugin should have:

#[async_trait::async_trait]
pub trait Plugin: std::fmt::Debug {
    fn name(&self) -> &'static str {
        "unknown"
    }

    fn apply(&self, _ctx: PluginContext<&mut ApplyContext>) -> Result<()> {
        Ok(())
    }
}

#[derive(Debug, Default)]
pub struct PluginContext<T = ()> {
    pub context: T,
}

#[derive(Debug)]
pub struct ApplyContext<'c> {
    pub compiler_hooks: &'c mut CompilerHooks,
}

define_hook!(BeforeRun: SyncSeries(compiler: &mut Compiler));

#[derive(Default, Debug)]
pub struct CompilerHooks {
    pub before_run: BeforeRunHook,
}

As you can see, we pass the before_run hook as context to the plugins.

Next, we define a PluginDriver to drive the plugins:

pub struct PluginDriver {
    pub plugins: Vec<Box<dyn Plugin>>,
    pub compiler_hooks: CompilerHooks,
}

impl PluginDriver {
    pub fn new(plugins: Vec<Box<dyn Plugin>>) -> Arc<Self> {
        let mut compiler_hooks = CompilerHooks::default();
        let mut apply_context = ApplyContext {
            compiler_hooks: &mut compiler_hooks,
        };

        for plugin in &plugins {
            plugin
                .apply(PluginContext::with_context(&mut apply_context))
                .expect("failed to apply plugin context");
        }

        Arc::new(Self {
            plugins,
            compiler_hooks,
        })
    }
}

Here, we initialize the parameters passed to each plugin, iterate over the plugins, and call their apply method.

Finally, the PluginDriver is used in the Compiler:

impl Compiler {
    pub fn new(mut config: Config, plugins: Vec<BoxPlugin>) -> Compiler {
        let plugin_driver = PluginDriver::new(plugins);

        Compiler {
            root: config.root.clone(),
            entry_id: "".to_string(),
            config,
            modules: HashMap::new(),
            assets: HashMap::new(),
            plugin_driver,
        }
    }
    ...

We temporarily modify the run method of Compiler:

pub fn run(&mut self) {
    self.plugin_driver.clone().compiler_hooks.before_run.call(self);
    // let resolved_entry = Path::new(&self.root).join(&self.config.entry);
    // self.build_module(resolved_entry, true);
    // self.emit_file();
}

Now, let's write a demo to test it:

use rswebpack_core::compiler::Compiler;
use rswebpack_core::config::{Config, Output};
use rswebpack_core::hooks::BeforeRun;
use rswebpack_core::plugin::{ApplyContext, Plugin, PluginContext};
use rswebpack_macros::{plugin, plugin_hook};
use rswebpack_error::Result;

#[plugin]
struct BeforeRunHookTap;

#[plugin_hook(BeforeRun for BeforeRunHookTap)]
fn before_run(&self, compiler: &mut Compiler) -> Result<()> {
    println!("Root is {}", compiler.root);
    Ok(())
}

#[derive(Debug)]
struct TestPlugin;

impl Plugin for TestPlugin {
    fn apply(&self, _ctx: PluginContext<&mut ApplyContext>) -> Result<()> {
        _ctx.context
            .compiler_hooks
            .before_run
            .tap(before_run::new(&BeforeRunHookTap::new_inner()));
        Ok(())
    }
}

fn main() {
    let config = Config::new(
        "test".to_string(),
        "test".to_string(),
        Output {
            path: "out".to_string(),
            filename: "bundle".to_string(),
        },
    );
    let compiler = &mut Compiler::new(config, vec![Box::new(TestPlugin {})]);
    compiler.run(); // Root is test
}

There you have it! However, in actual development, custom plugins are usually developed in JavaScript. How can we integrate these JavaScript plugins? We'll reveal the answer in the next part.

Please kindly give me a star!

Implementing Webpack from Scratch, But in Rust - [3] Using NAPI-RS to Create Node.js Addons

ayou — Thu, 31 Oct 2024 02:51:33 +0000

Referencing mini-webpack, I implemented a simple webpack from scratch using Rust. This allowed me to gain a deeper understanding of webpack and also improve my Rust skills. It's a win-win situation!

Code repository: https://github.com/ParadeTo/rs-webpack

This article corresponds to the Pull Request: https://github.com/ParadeTo/rs-webpack/pull/4

The title seems unrelated to this series, as implementing webpack with Rust involves Node.js plugin development. But don't worry, let me explain. When we use webpack for bundling, don't we often run the following command?

webpack --config webpack.config.js

Similarly, we want our RS Webpack to support such a command. But how can we get the content exported from webpack.config.js in Rust? One solution is to have a JS runtime to execute webpack.config.js. However, this approach seems a bit heavy, and it requires the JS runtime to be able to recreate the content of webpack.config.js in Rust. After searching for tools, I couldn't find anything suitable, so I had to take a different approach.

By examining the source code of Rspack, I found that it uses NAPI-RS to develop Node.js plugins. The specific approach is to write the core code of the packer using Rust, compile it into a plugin using NAPI-RS for use in Node.js, and let Node.js handle the import and parsing of the configuration file, passing it as a parameter to the interface provided by Rust.

To learn how to use NAPI-RS, you can refer to the official website. This article mainly explains how to transform our project into the desired result.

First, let's change our project structure as follows:

.
├── Cargo.lock
├── Cargo.toml
├── crates // Rust crates
│   ├── rswebpack_binding // Generated by NAPI
│   └── rswebpack_core
├── packages // JS packages
│   └── rswebpack-cli
├── pnpm-lock.yaml
├── pnpm-workspace.yaml
└── readme.md

Under crates, we have Rust projects. rswebpack_binding is generated by NAPI and is mainly used for exporting interfaces. rswebpack_core is the core library, where we moved the relevant code from the previous article.
Under packages, we have JS projects. rswebpack-cli will eventually be published as a command-line tool.

In the rswebpack_binding crate, the code is relatively simple. It just wraps the original Compiler:

// lib.rs
#![deny(clippy::all)]

use napi::Result;
use raw_config::RawConfig;
use rs_webpack_core::compiler::Compiler;
#[macro_use]
extern crate napi_derive;

mod raw_config;

#[napi]
pub struct RsWebpack {
  compiler: Box<Compiler>,
}

#[napi]
impl RsWebpack {
  #[napi(constructor)]
  pub fn new(raw_config: RawConfig) -> Result<Self> {
    let config = raw_config.try_into().expect("Config transform error");
    Ok(Self {
      compiler: Box::new(Compiler::new(config)),
    })
  }

  #[napi]
  pub fn run(&mut self) {
    self.compiler.as_mut().run();
  }
}

// raw_config.rs
use rswebpack_core::config::{Config, Output};

#[napi(object)]
pub struct RawOutput {
  pub path: String,
  pub filename: String,
}

impl TryFrom<RawOutput> for Output {
  type Error = ();

  fn try_from(value: RawOutput) -> Result<Self, Self::Error> {
    Ok(Output {
      path: value.path.into(),
      filename: value.filename.into(),
    })
  }
}

#[napi(object)]
pub struct RawConfig {
  pub root: String,
  pub entry: String,
  pub output: RawOutput,
}

impl TryFrom<RawConfig> for Config {
  type Error = ();

  fn try_from(value: RawConfig) -> Result<Self, Self::Error> {
    Ok(Config {
      root: value.root.into(),
      entry: value.entry.into(),
      output: value.output.try_into()?,
    })
  }
}

Here, we define RawConfig to receive the configuration passed from JS, and we also specify how to convert RawConfig into Config. However, the conversion rules are currently very simple.

As for rswebpack-cli, it's even simpler. We just need to parse the command-line arguments, read the configuration, and call the interface exported by the plugin:

#!/usr/bin/env node
const path = require('path')
const {RsWebpack} = require('@rswebpack/binding')

const argv = require('yargs-parser')(process.argv.slice(2))

const config = require(path.resolve(
  process.cwd(),
  argv.config || 'rswebpack.config.js'
))

const rsWebpack = new RsWebpack(config)
rsWebpack.run()

Don't forget to configure the command name in package.json:

{
  "name": "@rswebpack/cli",
  "dependencies": {
    "@rswebpack/binding": "workspace:*",
    "yargs-parser": "^21.1.1"
  },
  "bin": {
    "rswebpack": "./index.js"
  }
}

Then run npm link. After that, create a new directory:

.
├── const.js
├── index.js
└── rswebpack.config.js

The rswebpack.config.js file should contain the following:

const path = require('path')

module.exports = {
  root: path.resolve(__dirname),
  entry: 'index.js',
  output: {
    path: path.resolve(__dirname, 'out'),
    filename: 'bundle.js',
  },
}

Finally, run rswebpack --config rswebpack.config.js. If it outputs bundle.js correctly, it means the refactor was successful.

Please kindly give me a star!

Implementing Webpack from Scratch, But in Rust - [2] MVP Version

ayou — Mon, 28 Oct 2024 09:16:29 +0000

Referencing mini-webpack, I implemented a simple webpack from scratch using Rust. This allowed me to gain a deeper understanding of webpack and also improve my Rust skills. It's a win-win situation!

Code repository: https://github.com/ParadeTo/rs-webpack

This article corresponds to Pull Request

The previous section introduced how to use Oxc to parse and modify JS code, solving the core problem. Now we can implement an MVP version. The goal of this MVP is to bundle the following code and produce the expected output:

// index.js
const b = require('./const.js')
console.log(b)

// const.js
module.exports = 'hello'

We create a Compiler struct with the following properties and methods:

pub struct Compiler {
    config: Config,
    entry_id: String,
    root: String,
    modules: HashMap<String, String>,
    assets: HashMap<String, String>,
}

impl Compiler {
    pub fn new(config: Config) -> Compiler {
        // Implementation
    }

    fn parse(
        &self,
        module_path: PathBuf,
        parent_path: &Path,
    ) -> (String, Rc<RefCell<Vec<String>>>) {
        // Get module's code and dependencies
    }

    fn build_module(&mut self, module_path: PathBuf, is_entry: bool) {
        // Call build_module recursively to get modules (key is module_id, value is the code)
    }

    fn emit_file(&mut self) {
        // Output result
    }

    pub fn run(&mut self) {
        // Entry point
    }
}

In this code, run is the entry point which calls build_module. build_module first calls parse to get the code of the JS module and its dependencies (while also transforming the original code). Then, build_module recursively calls itself to handle these dependencies, resulting in the modules HashMap. modules has module IDs as keys (relative paths from the root) and the transformed module code as values. Finally, run calls emit_file to output the result. You can find the complete code in the Pull Request. Let's discuss some key points.

Transform

In a previous example, we showed how to modify the argument of require by changing it to a fixed value, full_path_of_b:

string_literal.value = Atom::from("full_path_of_b")

However, in actual development, this argument is dynamic and related to the current JS module's path. Let's use the following example to demonstrate how to add a dynamic prefix to the argument of require:

struct MyTransform {
    prefix: String,
}

impl<'a> Traverse<'a> for MyTransform {
    fn enter_call_expression(&mut self, node: &mut CallExpression<'a>, ctx: &mut TraverseCtx<'a>) {
        if node.is_require_call() {
            let argument: &mut Argument<'a> = &mut node.arguments.deref_mut()[0];
            match argument {
                Argument::StringLiteral(string_literal) => {
                    let old_name = string_literal.value.as_str();
                    let new_name = format!("{}{}", self.prefix, old_name);

                    // !!!!!! `new_name` does not live long enough
                    string_literal.value = Atom::from(new_name.as_str());
                }
                _ => {}
            }
        }
    }
}

The above code does not compile and produces an error: new_name does not live long enough. This is because new_name will be destroyed after the function execution, but Atom::from requires a lifetime 'a. The solution is to use:

string_literal.value = ctx.ast.atom(new_name.as_str());

The reason can be explained by examining the source code of ctx.ast.atom:

#[inline]
pub fn atom(self, value: &str) -> Atom<'a> {
    Atom::from(String::from_str_in(value, self.allocator).into_bump_str())
}

We can see that the atom method does not declare a lifetime and ultimately calls Atom::from, but the value inside it is generated by String::from_str_in, where the second parameter self.allocator has a lifetime:

pub struct AstBuilder<'a> {
    pub allocator: &'a Allocator,
}

The Allocator is a memory allocation tool based on bumpalo. It seems that Allocator is commonly used in implementing parsers. You can refer to this tutorial for more information, but for now, we will skip it.

Emit File

When outputting the final bundled file, a template engine called sailfish is used. The template looks like this:

(function(modules) {
    var installedModules = {};
    ...

    // Load entry module and return exports
    return __webpack_require__(__webpack_require__.s = "<%- entry_id %>");
})
({
   <% for (key, value) in modules { %>
     "<%- key %>":
     (function(module, exports, __webpack_require__) {
       eval(`<%- value %>`);
     }),
   <%}%>
});

We only need to output entry_id and the contents of modules into the template.

Running cargo run produces the following output:

// out/bundle.js
(function(modules) {
    var installedModules = {};
    ...
    // Load entry module and return exports
    return __webpack_require__(__webpack_require__.s = "./index.js");
})
({

     "./const.js":
     (function(module, exports, __webpack_require__) {
       eval(`module.exports = "hello";
`);
     }),

     "./index.js":
     (function(module, exports, __webpack_require__) {
       eval(`const b = __webpack_require__("./const.js");
console.log(b);
`);
     }),

});

If running it with Node.js results in the correct output of hello, it means the MVP has been successfully completed.

Please kindly give me a star!

Implementing Webpack from Scratch, But in Rust - [1] Parsing and Modifying JS Code Using Oxc

ayou — Thu, 24 Oct 2024 03:42:38 +0000

Referencing mini-webpack, I implemented a simple webpack from scratch using Rust. This allowed me to gain a deeper understanding of webpack and also improve my Rust skills. It's a win-win situation!

Code repository: https://github.com/ParadeTo/rs-webpack

This article corresponds to Pull Request

To implement a simple webpack, the primary task is to address the issue of JavaScript code parsing. Building a JavaScript parser from scratch is a monumental task, so it's better to choose an existing tool. Here, I chose oxc, which has the endorsement of Evan You.

Although oxc doesn't have as detailed documentation as Babel, the usage patterns are similar. First, we need to use oxc_parser to parse the JS code and generate an AST (Abstract Syntax Tree):

let name = env::args().nth(1).unwrap_or_else(|| "test.js".to_string());
let path = Path::new(&name);
let source_text = Arc::new(std::fs::read_to_string(path)?);
let source_type = SourceType::from_path(path).unwrap();

// Memory arena where Semantic and Parser allocate objects
let allocator = Allocator::default();

// 1 Parse the source text into an AST
let parser_ret = Parser::new(&allocator, &source_text, source_type).parse();
let mut program = parser_ret.program;

println!("Parse result");
println!("{}", serde_json::to_string_pretty(&program).unwrap());

The content of test.js is as follows:

const b = require('./b.js')

The parsed AST looks like this:

{
  "type": "Program",
  "start": 0,
  "end": 28,
  "sourceType": {
    "language": "javascript",
    "moduleKind": "module",
    "variant": "jsx"
  },
  "hashbang": null,
  "directives": [],
  "body": [
    {
      "type": "VariableDeclaration",
      "start": 0,
      "end": 27,
      "kind": "const",
      "declarations": [
        {
          "type": "VariableDeclarator",
          "start": 6,
          "end": 27,
          "id": {
            "type": "Identifier",
            "start": 6,
            "end": 7,
            "name": "b",
            "typeAnnotation": null,
            "optional": false
          },
          "init": {
            "type": "CallExpression",
            "start": 10,
            "end": 27,
            "callee": {
              "type": "Identifier",
              "start": 10,
              "end": 17,
              "name": "require"
            },
            "typeParameters": null,
            "arguments": [
              {
                "type": "StringLiteral",
                "start": 18,
                "end": 26,
                "value": "./b.js"
              }
            ],
            "optional": false
          },
          "definite": false
        }
      ],
      "declare": false
    }
  ]
}

As webpack users know, during bundling, we need to replace require with __webpack_require__ and replace the relative path ./b.js with the full path. To achieve this, we need to modify the original code using oxc_traverse, which allows us to traverse the nodes in the AST and perform operations on the nodes we are interested in.

From the AST result above, we can see that the node of interest is CallExpression. Therefore, we can implement a Transform to modify this node as follows:

struct MyTransform;

impl<'a> Traverse<'a> for MyTransform {
    fn enter_call_expression(&mut self, node: &mut CallExpression<'a>, ctx: &mut TraverseCtx<'a>) {
        if node.is_require_call() {
            match &mut node.callee {
                Expression::Identifier(identifier_reference) => {
                    identifier_reference.name = Atom::from("__webpack_require__")
                }
                _ => {}
            }

            let argument: &mut Argument<'a> = &mut node.arguments.deref_mut()[0];
            match argument {
                Argument::StringLiteral(string_literal) => {
                    string_literal.value = Atom::from("full_path_of_b")
                }
                _ => {}
            }
        }
    }
}

You can use this Transform as follows:

// 2 Semantic Analyze
let semantic = SemanticBuilder::new(&source_text)
    .build_module_record(path, &program)
    // Enable additional syntax checks not performed by the parser
    .with_check_syntax_error(true)
    .build(&program);
let (symbols, scopes) = semantic.semantic.into_symbol_table_and_scope_tree();

// 3 Transform
let t = &mut MyTransform;
traverse_mut(t, &allocator, &mut program, symbols, scopes);

Note that, unlike Babel, we need to use oxc_semantic to perform syntax analysis first and obtain symbols and scopes, which are then passed to traverse_mut.

Finally, we use oxc_codegen to regenerate the code:

// 4 Generate Code
let new_code = CodeGenerator::new()
    .with_options(CodegenOptions {
        ..CodegenOptions::default()
    })
    .build(&program)
    .code;

println!("{}", new_code);

The resulting code will be:

const b = __webpack_require__('full_path_of_b')

Please kindly give me a star!

Implement React v18 from Scratch Using WASM and Rust - [27] Implement useTransition

ayou — Thu, 26 Sep 2024 09:25:37 +0000

Based on big-react，I am going to implement React v18 core features from scratch using WASM and Rust.

Code Repository：https://github.com/ParadeTo/big-react-wasm

The tag related to this article：v27

useTransition is a new hook introduced in React that allows you to update state without blocking the UI. The official React website provides an example that demonstrates the difference before and after using useTransition. There is also an article that analyzes the underlying principles. Now let's implement it. You can find the details of the changes in this link.

First, let's follow the process of adding a new hook and add the relevant code. Eventually, we will come to fiber_hooks.rs:

fn mount_transition() -> Vec<JsValue> {
    let result = mount_state(&JsValue::from(false)).unwrap();
    let is_pending = result[0].as_bool().unwrap();
    let set_pending = result[1].clone().dyn_into::<Function>().unwrap();
    let hook = mount_work_in_progress_hook();
    let set_pending_cloned = set_pending.clone();
    let closure = Closure::wrap(Box::new(move |callback: Function| {
        start_transition(set_pending_cloned.clone(), callback);
    }) as Box<dyn Fn(Function)>);
    let start: Function = closure.as_ref().unchecked_ref::<Function>().clone();
    closure.forget();
    hook.as_ref().unwrap().clone().borrow_mut().memoized_state =
        Some(MemoizedState::MemoizedJsValue(start.clone().into()));
    vec![JsValue::from_bool(is_pending), start.into()]
}

During the mount_transition process, the following data structure is formed:

So when update_transition is called, we can retrieve the values from the hooks:

fn update_transition() -> Vec<JsValue> {
    let result = update_state(&JsValue::undefined()).unwrap();
    let is_pending = result[0].as_bool().unwrap();
    let hook = update_work_in_progress_hook();
    if let MemoizedState::MemoizedJsValue(start) = hook
        .as_ref()
        .unwrap()
        .clone()
        .borrow()
        .memoized_state
        .as_ref()
        .unwrap()
    {
        return vec![JsValue::from_bool(is_pending), start.into()];
    }
    panic!("update_transition")
}

The key lies in the implementation of start_transition:

fn start_transition(set_pending: Function, callback: Function) {
    set_pending.call1(&JsValue::null(), &JsValue::from_bool(true));
    let prev_transition = unsafe { REACT_CURRENT_BATCH_CONFIG.transition };

    // low priority
    unsafe { REACT_CURRENT_BATCH_CONFIG.transition = Lane::TransitionLane.bits() };
    callback.call0(&JsValue::null());
    set_pending.call1(&JsValue::null(), &JsValue::from_bool(false));

    unsafe { REACT_CURRENT_BATCH_CONFIG.transition = prev_transition };
}

According to the analysis in this article, the implementation first updates isPending to true with the current priority. Then it lowers the priority, executes the callback, and updates isPending to false. Finally, it restores the previous priority.

The update process with lowered priority uses Concurrent Mode, which is why it doesn't block the UI:

if cur_priority == Lane::SyncLane {
  ...
} else {
    if is_dev() {
        log!("Schedule in macrotask, priority {:?}", update_lanes);
    }
    let scheduler_priority = lanes_to_scheduler_priority(cur_priority.clone());
    let closure = Closure::wrap(Box::new(move |did_timeout_js_value: JsValue| {
        let did_timeout = did_timeout_js_value.as_bool().unwrap();
        perform_concurrent_work_on_root(root_cloned.clone(), did_timeout)
    }) as Box<dyn Fn(JsValue) -> JsValue>);
    let function = closure.as_ref().unchecked_ref::<Function>().clone();
    closure.forget();
    new_callback_node = Some(unstable_schedule_callback_no_delay(
        scheduler_priority,
        function,
    ))
}

With this, the implementation of useTransition is mostly complete. However, there were a few bugs encountered during the process:

The first bug is in begin_work.rs:

work_in_progress.borrow_mut().lanes = Lane::NoLane;

When a FiberNode has multiple Lanes, this approach causes issues. It should be changed to:

work_in_progress.borrow_mut().lanes -= render_lane;

So that only the currently rendered Lane is removed each time.

The second bug is in work_loop.rs:

log!("render over {:?}", *root.clone().borrow());
WORK_IN_PROGRESS_ROOT_RENDER_LANE = Lane::NoLane;

Originally, this line was in the render_root function, resetting the variable after the Render phase is complete. But in Concurrent Mode, when the Render process is interrupted, this variable should not be reset. Therefore, this line is moved to perform_concurrent_work_on_root:

if exit_status == ROOT_COMPLETED {
    ...
    unsafe { WORK_IN_PROGRESS_ROOT_RENDER_LANE = Lane::NoLane };
}

So that the variable is only reset when the Render process is completed.

The third bug is in update_queue.rs, as shown in the following image:

Additionally, the Scheduler has been refactored. Previously, the min-heap was defined as follows:

static mut TASK_QUEUE: Vec<Task> = vec![];
static mut TIMER_QUEUE: Vec<Task> = vec![];

This required implementing a separate peek_mut function when modifying properties of the Task in the heap:

let mut task = peek_mut(&mut TASK_QUEUE);
task.callback = JsValue::null();

Now it has been changed to:

static mut TASK_QUEUE: Vec<Rc<RefCell<Task>>> = vec![];
static mut TIMER_QUEUE: Vec<Rc<RefCell<Task>>> = vec![];

And the peek function can be used uniformly:

let task = peek(&TASK_QUEUE);
task.borrow_mut().callback = JsValue::null();

Please kindly give me a star!

Implement React v18 from Scratch Using WASM and Rust - [26] Implement React.lazy

ayou — Fri, 20 Sep 2024 01:58:56 +0000

Based on big-react，I am going to implement React v18 core features from scratch using WASM and Rust.

Code Repository：https://github.com/ParadeTo/big-react-wasm

The tag related to this article：v26

Suspense has another useful feature that combines with React.lazy for component lazy loading. Let's continue implementing it. The changes can be found in detail here.

We'll use the following example to illustrate:

import { Suspense, lazy } from 'react';

function delay(promise) {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(promise);
    }, 2000);
  });
}

const Cpn = lazy(() =>
  import('./Cpn').then((res) => delay(res))
);

export default function App() {
  return (
    <Suspense fallback={<div>loading</div>}>
      <Cpn />
    </Suspense>
  );
}

First, we need to export this method in the react library:

#[wasm_bindgen]
pub fn lazy(ctor: &JsValue) -> JsValue {
    let payload = Object::new();
    Reflect::set(&payload, &"_status".into(), &JsValue::from(UNINITIALIZED));
    Reflect::set(&payload, &"_result".into(), ctor);

    let lazy_type = Object::new();

    Reflect::set(
        &lazy_type,
        &"$typeof".into(),
        &JsValue::from_str(REACT_LAZY_TYPE),
    );
    Reflect::set(&lazy_type, &"_payload".into(), &payload);
    let closure = Closure::wrap(
        Box::new(lazy_initializer) as Box<dyn Fn(JsValue) -> Result<JsValue, JsValue>>
    );
    let f = closure.as_ref().unchecked_ref::<Function>().clone();
    closure.forget();
    Reflect::set(&lazy_type, &"_init".into(), &f);
    lazy_type.into()
}

The translation to JavaScript is as follows:

const payload = {
  _status: UNINITIALIZED,
  _result: ctor,
};

const lazy_type = {
  $typeof: REACT_LAZY_TYPE,
  _payload: payload,
  _init: lazy_initializer,
};

Here, the focus is on the lazy_initializer function. Let's explain it using the JavaScript version:

function lazy_initializer(payload) {
  if (payload._status === Uninitialized) {
    const ctor = payload._result;
    const thenable = ctor();
    thenable.then(
      (moduleObject) => {
        payload._status = Resolved;
        payload._result = moduleObject;
      },
      (error) => {
        payload._status = Rejected;
        payload._result = error;
      }
    );

    payload._status = Pending;
    payload._result = thenable;
  }

  if (payload._status === Resolved) {
    const moduleObject = payload._result;
    return moduleObject.default;
  } else {
    throw payload._result;
  }
}

This function is similar to the "use" hook implemented in the previous article. Here, ctor corresponds to () => import('./Cpn').then((res) => delay(res)). When executed, it returns a Promise object. It only returns the result (res) when the object's status is Resolved. In this case, res is a module object, and its default property contains the content exported using export default. For other statuses, it throws _result. When the status is Pending, _result is the Promise object itself, and when the status is Rejected, _result is an error object.

Next, the main file to be modified is begin_work.rs:

....
        WorkTag::LazyComponent => update_lazy_component(work_in_progress.clone(), render_lane),
    };
}

fn update_lazy_component(
    work_in_progress: Rc<RefCell<FiberNode>>,
    render_lane: Lane,
) -> Result<Option<Rc<RefCell<FiberNode>>>, JsValue> {
    let lazy_type = { work_in_progress.borrow()._type.clone() };
    let payload = derive_from_js_value(&lazy_type, "_payload");
    let init_jsvalue = derive_from_js_value(&lazy_type, "_init");
    let init = init_jsvalue.dyn_ref::<Function>().unwrap();
    // return value OR throw
    let Component = init.call1(&JsValue::null(), &payload)?;
    work_in_progress.borrow_mut()._type = Component.clone();
    work_in_progress.borrow_mut().tag = WorkTag::FunctionComponent;
    let child = update_function_component(work_in_progress, Component.clone(), render_lane);
    child
}
....

The key line here is let Component = init.call1(&JsValue::null(), &payload)?;. When init is executed and throws an exception, according to the flow in the previous article, it will find the nearest Suspense component and restart the render process, rendering the fallback of Suspense. When the Promise object is resolved, the update process will be triggered again, and when it reaches this line, the returned value of init will be the exported component, which is Cpn.

In addition, handle_throw in work_loop.rs needs to be modified to handle non-use errors:

fn handle_throw(root: Rc<RefCell<FiberRootNode>>, mut thrown_value: JsValue) {
    /*
        throw possibilities:
            1. use thenable
            2. error (Error Boundary), lazy
    */
    if Object::is(&thrown_value, &SUSPENSE_EXCEPTION) {
        unsafe { WORK_IN_PROGRESS_SUSPENDED_REASON = SUSPENDED_ON_DATA };
        thrown_value = get_suspense_thenable();
    } else {
        let is_wakeable = !thrown_value.is_null()
            && type_of(&thrown_value, "object")
            && derive_from_js_value(&thrown_value, "then").is_function();
        unsafe {
            WORK_IN_PROGRESS_SUSPENDED_REASON = if is_wakeable {
                SUSPENDED_ON_DEPRECATED_THROW_PROMISE
            } else {
                SUSPENDED_ON_ERROR
            };
        };
    }

    unsafe {
        WORK_IN_PROGRESS_THROWN_VALUE = Some(thrown_value);
    }
}

Finally, the previous article mentioned an issue with bailout affecting the normal operation of Suspense. The solution was to move the code responsible for bubbling update priority to fiber_throw.rs:

let closure = Closure::wrap(Box::new(move || {
  ...
  mark_update_lane_from_fiber_to_root(source_fiber.clone(), lane.clone());
  ensure_root_is_scheduled(root.clone());
}) as Box<dyn Fn()>);
...

Additionally, in begin_work.rs, we exclude the Suspense component from the bailout logic:

if !has_scheduled_update_or_context
    && current.borrow().tag != WorkTag::SuspenseComponent
{
  ...
  return Ok(bailout_on_already_finished_work(
      work_in_progress,
      render_lane,
  ));
}

Please kindly give me a star!

Implement React v18 from Scratch Using WASM and Rust - [25] Suspense(2) - Data Fetching with use hook

ayou — Fri, 06 Sep 2024 10:27:56 +0000

Based on big-react，I am going to implement React v18 core features from scratch using WASM and Rust.

Code Repository：https://github.com/ParadeTo/big-react-wasm

The tag related to this article：v25

In the new version of React, Suspense plays a significant role in combining with use to fetch data. Today, let's implement it. You can find the changes in this link.

Let's explain the changes using the following example:

import { Suspense, use } from 'react'

const delay = (t) =>
  new Promise((resolve, reject) => {
    setTimeout(reject, t)
  })

const cachePool = []

function fetchData(id, timeout) {
  const cache = cachePool[id]
  if (cache) {
    return cache
  }
  return (cachePool[id] = delay(timeout).then(() => {
    return { data: Math.random().toFixed(2) * 100 }
  }))
}

export default function App() {
  return (
    <Suspense fallback={<div>loading</div>}>
      <Child />
    </Suspense>
  )
}

function Child() {
  const { data } = use(fetchData(1, 1000))
  return <span>{data}</span>
}

First, we add the relevant code according to the process of adding new hooks. Finally, we arrive at fiber_hooks.rs:

fn _use(usable: JsValue) -> Result<JsValue, JsValue> {
  if !usable.is_null() && type_of(&usable, "object") {
      if derive_from_js_value(&usable, "then").is_function() {
          return track_used_thenable(usable);
      } else if derive_from_js_value(&usable, "$typeof") == REACT_CONTEXT_TYPE {
          return Ok(read_context(usable));
      }
  }
  Err(JsValue::from_str("Not supported use arguments"))
}

From the code, we can see that the use function can accept a Promise object or a Context object. Here, we only discuss the Promise object, so let's take a look at track_used_thenable:

#[wasm_bindgen]
extern "C" {
    pub static SUSPENSE_EXCEPTION: JsValue;
}

pub fn track_used_thenable(thenable: JsValue) -> Result<JsValue, JsValue> {
    ...
    unsafe { SUSPENDED_THENABLE = Some(thenable.clone()) };
    Err(SUSPENSE_EXCEPTION.__inner.with(JsValue::clone))
}

We'll skip the middle part for now. In the end, it returns a variant of Result, Err, with SUSPENSE_EXCEPTION as the payload. This SUSPENSE_EXCEPTION is inserted into the result during the build process:

const SUSPENSE_EXCEPTION = new Error(
  "It's not a true mistake, but part of Suspense's job. If you catch the error, keep throwing it out"
)

Returning SUSPENSE_EXCEPTION instead of thenable directly helps differentiate between exceptions thrown by user code and exceptions thrown by React. The value we are interested in is stored in SUSPENDED_THENABLE.

Next, we come to work_loop.rs:

loop {
    ...
    match if should_time_slice {
        work_loop_concurrent()
    } else {
        work_loop_sync()
    } {
        Ok(_) => {
            break;
        }
        Err(e) => handle_throw(root.clone(), e),
    };
}

The e here is the SUSPENSE_EXCEPTION mentioned earlier. Let's see how handle_throw handles it:

fn handle_throw(root: Rc<RefCell<FiberRootNode>>, mut thrown_value: JsValue) {
    /*
        throw possibilities:
            1. use thenable
            2. error (Error Boundary)
    */
    if Object::is(&thrown_value, &SUSPENSE_EXCEPTION) {
        unsafe { WORK_IN_PROGRESS_SUSPENDED_REASON = SUSPENDED_ON_DATA };
        thrown_value = get_suspense_thenable();
    } else {
        // TODO
    }

    unsafe {
        WORK_IN_PROGRESS_THROWN_VALUE = Some(thrown_value);
    }
}

Here, it checks if the exception is SUSPENSE_EXCEPTION. If it is, the actual value is retrieved, which matches what was mentioned earlier.

This value will eventually be passed to throw_and_unwind_work_loop:

    loop {
        unsafe {
            if WORK_IN_PROGRESS_SUSPENDED_REASON != NOT_SUSPENDED && WORK_IN_PROGRESS.is_some() {
                let thrown_value = WORK_IN_PROGRESS_THROWN_VALUE.clone().unwrap();

                WORK_IN_PROGRESS_SUSPENDED_REASON = NOT_SUSPENDED;
                WORK_IN_PROGRESS_THROWN_VALUE = None;

                // TODO
                mark_update_lane_from_fiber_to_root(
                    WORK_IN_PROGRESS.clone().unwrap(),
                    lane.clone(),
                );

                throw_and_unwind_work_loop(
                    root.clone(),
                    WORK_IN_PROGRESS.clone().unwrap(),
                    thrown_value,
                    lane.clone(),
                );
            }
        }
        ...
    }

We have already discussed this in the previous article, so we won't go into detail here. Let's go back to track_used_thenable:

pub fn track_used_thenable(thenable: JsValue) -> Result<JsValue, JsValue> {
    let status = derive_from_js_value(&thenable, "status");
    if status.is_string() {
      ...
    } else {
        Reflect::set(&thenable, &"status".into(), &"pending".into());
        let v = derive_from_js_value(&thenable, "then");
        let then = v.dyn_ref::<Function>().unwrap();

        let thenable1 = thenable.clone();
        let on_resolve_closure = Closure::wrap(Box::new(move |val: JsValue| {
            if derive_from_js_value(&thenable1, "status") == "pending" {
                Reflect::set(&thenable1, &"status".into(), &"fulfilled".into());
                Reflect::set(&thenable1, &"value".into(), &val);
            }
        }) as Box<dyn Fn(JsValue) -> ()>);
        let on_resolve = on_resolve_closure
            .as_ref()
            .unchecked_ref::<Function>()
            .clone();
        on_resolve_closure.forget();

        let thenable2 = thenable.clone();
        let on_reject_closure = Closure::wrap(Box::new(move |err: JsValue| {
            if derive_from_js_value(&thenable2, "status") == "pending" {
                Reflect::set(&thenable2, &"status".into(), &"rejected".into());
                Reflect::set(&thenable2, &"reason".into(), &err);
            }
        }) as Box<dyn Fn(JsValue) -> ()>);
        let on_reject = on_reject_closure
            .as_ref()
            .unchecked_ref::<Function>()
            .clone();
        on_reject_closure.forget();

        then.call2(&thenable, &on_resolve, &on_reject);
    }
}

When entering this function for the first time, it goes to the else branch. The core logic is to add on_resolve and on_reject methods to thenable and modify its status, value, and reason properties.

When the status of the Promise object is no longer pending, it triggers a re-render. When this function is called again, the status will have a value, and it will enter the if block:

if status.is_string() {
  if status == "fulfilled" {
      return Ok(derive_from_js_value(&thenable, "value"));
  } else if status == "rejected" {
      return Err(derive_from_js_value(&thenable, "reason"));
  }
  ...
}

If the status is fulfilled, it returns the value; otherwise, it throws the exception from the reason field.

This is how Suspense combines with the use hook to fetch data. However, debugging revealed that the bailout logic affects the normal operation of this process. Therefore, for now, we have to temporarily comment out this part of the code and revisit how to solve it later when we have time.

Implement React v18 from Scratch Using WASM and Rust - [24] Suspense(1) - Render Fallback

ayou — Tue, 03 Sep 2024 01:52:50 +0000

Based on big-react，I am going to implement React v18 core features from scratch using WASM and Rust.

Code Repository：https://github.com/ParadeTo/big-react-wasm

The tag related to this article：v24

Suspense is undoubtedly one of the most appealing features in the new version of React, so let's implement it ourselves. This article is the first part, focusing on implementing the Fallback rendering of Suspense.

Consider the following code as an example:

import { Suspense } from 'react';

export default function App() {
  return (
    <Suspense fallback={<div>loading</div>}>
      <Child />
    </Suspense>
  );
}

function Child() {
  throw new Promise((resolve) => setTimeout(resolve, 1000));
}

For the Suspense node, it has two child branches corresponding to Primary and Fallback. The root node of the Primary branch is of type Offscreen, and the root node of the Fallback branch is of type Fragment:

Specifically, in the example above:

During the initial render, the code enters the Primary branch. When it reaches the Child component, an exception is thrown because the component throws a Promise object. This triggers the "unwind" process, which searches for the nearest Suspense node and adds the DidCapture flag to it, and then continues the render process from that node.

Since the Suspense node has the DidCapture flag, the code enters the Fallback branch. The subsequent steps involve the normal render and commit processes, eventually rendering the content within the Fallback.

That's the functionality we want to implement. Now let's briefly go through the code.

First, let's take a look at begin_work.rs where we need to add handling for Suspense:

fn update_suspense_component(
    work_in_progress: Rc<RefCell<FiberNode>>,
) -> Option<Rc<RefCell<FiberNode>>> {
    let current = { work_in_progress.borrow().alternate.clone() };
    let next_props = { work_in_progress.borrow().pending_props.clone() };

    let mut show_fallback = false;
    let did_suspend =
        (work_in_progress.borrow().flags.clone() & Flags::DidCapture) != Flags::NoFlags;

    if did_suspend {
        show_fallback = true;
        work_in_progress.borrow_mut().flags -= Flags::DidCapture;
    }

    let next_primary_children = derive_from_js_value(&next_props, "children");
    let next_fallback_children = derive_from_js_value(&next_props, "fallback");
    push_suspense_handler(work_in_progress.clone());

    if current.is_none() {
        if show_fallback {
            return Some(mount_suspense_fallback_children(
                work_in_progress.clone(),
                next_primary_children.clone(),
                next_fallback_children.clone(),
            ));
        } else {
            return Some(mount_suspense_primary_children(
                work_in_progress.clone(),
                next_primary_children.clone(),
            ));
        }
    } else {
        if show_fallback {
            return Some(update_suspense_fallback_children(
                work_in_progress.clone(),
                next_primary_children.clone(),
                next_fallback_children.clone(),
            ));
        } else {
            return Some(update_suspense_primary_children(
                work_in_progress.clone(),
                next_primary_children.clone(),
            ));
        }
    }
}

Here, we handle four branches based on whether the Fallback is shown and whether it's the first update.

Next, let's look at work_loop.rs:

loop {
  unsafe {
      if WORK_IN_PROGRESS_SUSPENDED_REASON != NOT_SUSPENDED && WORK_IN_PROGRESS.is_some() {
          // unwind process
          ...
      }
  }
  match if should_time_slice {
      work_loop_concurrent()
  } else {
      work_loop_sync()
  } {
      Ok(_) => {
          break;
      }
      Err(e) => handle_throw(root.clone(), e),
  };
}

When an exception is thrown in the component, it enters the Err branch, where we mainly add the handle_throw process, which is currently simple:

fn handle_throw(root: Rc<RefCell<FiberRootNode>>, thrown_value: JsValue) {
    unsafe {
        WORK_IN_PROGRESS_SUSPENDED_REASON = SUSPENDED_ON_DATA;
        WORK_IN_PROGRESS_THROWN_VALUE = Some(thrown_value);
    }
}

The loop continues, entering the unwind process:

loop {
  unsafe {
      if WORK_IN_PROGRESS_SUSPENDED_REASON != NOT_SUSPENDED && WORK_IN_PROGRESS.is_some() {
          let thrown_value = WORK_IN_PROGRESS_THROWN_VALUE.clone().unwrap();

          WORK_IN_PROGRESS_SUSPENDED_REASON = NOT_SUSPENDED;
          WORK_IN_PROGRESS_THROWN_VALUE = None;

          throw_and_unwind_work_loop(
              root.clone(),
              WORK_IN_PROGRESS.clone().unwrap(),
              thrown_value,
              lane.clone(),
          );
      }
  }
  ...
}

fn throw_and_unwind_work_loop(
    root: Rc<RefCell<FiberRootNode>>,
    unit_of_work: Rc<RefCell<FiberNode>>,
    thrown_value: JsValue,
    lane: Lane,
) {
    unwind_unit_of_work(unit_of_work);
}

The task here is to find the nearest Suspense node and mark it with DidCapture.

With this, our task is complete. However, to pave the way for the next article, let's implement a bit more functionality.

Using the same code example, when the initial render reaches the Child component, we should capture the Promise object it throws, call its then method, and trigger the re-rendering logic in the provided function.

This way, when the Promise object's state becomes fulfilled, it will re-enter the render process. At this point, if the Child component still throws an exception, the same process repeats. However, for now, we won't handle this because we haven't implemented the use hook yet. So, this is just a temporary approach for testing.

Let's see how we can capture the Promise object and trigger the re-rendering when it becomes fulfilled:

First, let's add the throw_exception method in throw_and_unwind_work_loop:

fn throw_and_unwind_work_loop(
    root: Rc<RefCell<FiberRootNode>>,
    unit_of_work: Rc<RefCell<FiberNode>>,
    thrown_value: JsValue,
    lane: Lane,
) {
    throw_exception(root.clone(), thrown_value, lane.clone());
}

fn attach_ping_listener(root: Rc<RefCell<FiberRootNode>>, wakeable: JsValue, lane: Lane) {
    let then_value = derive_from_js_value(&wakeable, "then");
    let then = then_value.dyn_ref::<Function>().unwrap();
    let closure = Closure::wrap(Box::new(move || {
        root.clone().borrow_mut().mark_root_updated(lane.clone());
        ensure_root_is_scheduled(root.clone());
    }) as Box<dyn Fn()>);
    let ping = closure.as_ref().unchecked_ref::<Function>().clone();
    closure.forget();
    then.call2(&wakeable, &ping, &ping)
        .expect("failed to call then function");
}

pub fn throw_exception(root: Rc<RefCell<FiberRootNode>>, value: JsValue, lane: Lane) {
    if !value.is_null()
        && type_of(&value, "object")
        && derive_from_js_value(&value, "then").is_function()
    {
        let suspense_boundary = get_suspense_handler();
        if suspense_boundary.is_some() {
            let suspense_boundary = suspense_boundary.unwrap();
            suspense_boundary.borrow_mut().flags |= Flags::ShouldCapture;
        }

        attach_ping_listener(root, value, lane)
    }
}

The ping function is the function passed to then. The core logic is to set the current lane as the priority for the next update and call ensure_root_is_scheduled to start a new update. However, it was found during testing that this is not enough because the performance optimization in begin_work.rs will bail out the update starting from the root node. This issue also exists in big-react (switch to the master branch and run the "suspense-use" example to reproduce it, see the issue for details).

To solve this problem, a temporary solution is to bubble up the update priority one more time before the unwind process. This way, when the update starts again from the root node, it won't enter the bailout process due to the flags on subtree_flags.

loop {
    unsafe {
        if WORK_IN_PROGRESS_SUSPENDED_REASON != NOT_SUSPENDED && WORK_IN_PROGRESS.is_some() {
            let thrown_value = WORK_IN_PROGRESS_THROWN_VALUE.clone().unwrap();

            WORK_IN_PROGRESS_SUSPENDED_REASON = NOT_SUSPENDED;
            WORK_IN_PROGRESS_THROWN_VALUE = None;

            mark_update_lane_from_fiber_to_root(
                WORK_IN_PROGRESS.clone().unwrap(),
                lane.clone(),
            );

            ...
        }
    }
    ...
}

You can find the details of this update here.

Please kindly give me a star!

Implement React v18 from Scratch Using WASM and Rust - [23] Support Fragment

ayou — Tue, 13 Aug 2024 03:02:34 +0000

Based on big-react，I am going to implement React v18 core features from scratch using WASM and Rust.

Code Repository：https://github.com/ParadeTo/big-react-wasm

The tag related to this article：v23

Fragment is also a basic feature in React, so the WASM version needs to support it as well. But first, let's fix a few critical bugs.

Bug 1: In the following example, only the first click has an effect (updates to 1), and subsequent clicks keep it at 1.



function App() {
  const [num, setNum] = useState(0)
  return <div onClick={() => setNum((prev) => prev + 1)}>{num}</div>
}

The issue lies in the update of new_base_state in update_queue.rs. It needs to be modified as follows:



-   new_base_state = result.memoized_state.clone();
+   new_base_state = new_state.clone();

After fixing the above bug, there is still an issue related to useState.

Bug 2: In the following example, only the first click has an effect (updates to 1), and subsequent clicks keep it at 1.



function App() {
  const [num, setNum] = useState(0)
  return <div onClick={() => setNum(num + 1)}>{num}</div>
}

After investigation, it was found that the num inside the onClick function always remains 0, even after the first click when num should be 1. The root cause is that the onClick of the div always references the function passed during the initial rendering, and the captured closure's num is also the initial 0.

Upon reviewing the code, it was discovered that we missed the update logic for props of HostComponent type FiberNode. We only handled the HostText type before. Let's fill in this part.

First, let's redefine HostConfig by removing commit_text_update and adding commit_update:



-    fn commit_text_update(&self, text_instance: Rc<dyn Any>, content: &JsValue);
+    fn commit_update(&self, fiber: Rc<RefCell<FiberNode>>);

Then, implement this trait in the react-dom library as follows:



fn commit_update(&self, fiber: Rc<RefCell<FiberNode>>) {
  let instance = FiberNode::derive_state_node(fiber.clone());
  let memoized_props = fiber.borrow().memoized_props.clone();
  match fiber.borrow().tag {
      WorkTag::HostText => {
          let text = derive_from_js_value(&memoized_props, "content");
          self.commit_text_update(instance.unwrap(), &text);
      }
      WorkTag::HostComponent => {
          update_fiber_props(
              instance
                  .unwrap()
                  .downcast::<Node>()
                  .unwrap()
                  .dyn_ref::<Element>()
                  .unwrap(),
              &memoized_props,
          );
      }
      _ => {
          log!("Unsupported update type")
      }
  };
}

Here, update_fiber_props already exists, and its purpose is to update the latest props onto the corresponding Element of the FiberNode.

Next, add the following code in complete_work.rs:



WorkTag::HostComponent => {
        if current.is_some() && work_in_progress_cloned.borrow().state_node.is_some() {
          // todo: compare props to decide if need to update
+         CompleteWork::mark_update(work_in_progress.clone());

This marks the FiberNode with the Update flag. Further optimization can be done here (comparing the previous and current props to decide whether to mark the update flag), but for simplicity, let's skip it for now.

Finally, modify the handling of Update in commit_work.rs:



if flags.contains(Flags::Update) {
  unsafe {
      HOST_CONFIG
          .as_ref()
          .unwrap()
          .commit_update(finished_work.clone())
  }
  finished_work.borrow_mut().flags -= Flags::Update;
}

The PR for fixing the bugs can be found here. With the bug fixes completed, let's proceed to implement Fragment.

First, Fragment is exported as a constant from react. However, when we try to write it as follows in Rust, an error occurs: "#[wasm_bindgen] can only be applied to a function, struct, enum, impl, or extern block":



#[wasm_bindgen]
pub static Fragment: &str = "react.fragment";

It seems that exporting strings from Rust to JavaScript is not supported. Therefore, we need to continue modifying the compiled output through the build script, specifically by adding the code to export Fragment in the final JS file.



// add Fragment
const reactIndexFilename = `${cwd}/dist/react/jsx-dev-runtime.js`
const reactIndexData = fs.readFileSync(reactIndexFilename)
fs.writeFileSync(
  reactIndexFilename,
  reactIndexData + `export const Fragment='react.fragment';\n`
)
const reactTsIndexFilename = `${cwd}/dist/react/jsx-dev-runtime.d.ts`
const reactTsIndexData = fs.readFileSync(reactTsIndexFilename)
fs.writeFileSync(
  reactTsIndexFilename,
  reactTsIndexData + `export const Fragment: string;\n`
)

Next, we need to add a method create_fiber_from_fragment in fiber.rs:



pub fn create_fiber_from_fragment(elements: JsValue, key: JsValue) -> FiberNode {
  FiberNode::new(WorkTag::Fragment, elements, key, JsValue::null())
}

Here, elements refers to the children of the fragment.

Then, following the workflow, we need to add handling for Fragment in begin_work.rs:



pub fn begin_work(
    work_in_progress: Rc<RefCell<FiberNode>>,
    render_lane: Lane,
) -> Result<Option<Rc<RefCell<FiberNode>>>, JsValue> {
  ...
  return match tag {
    ...
    WorkTag::Fragment => Ok(update_fragment(work_in_progress.clone())),
  };
}

fn update_fragment(work_in_progress: Rc<RefCell<FiberNode>>) -> Option<Rc<RefCell<FiberNode>>> {
    let next_children = work_in_progress.borrow().pending_props.clone();
    reconcile_children(work_in_progress.clone(), Some(next_children));
    work_in_progress.borrow().child.clone()
}

In the reconcile_single_element function, we also need to add handling for Fragment:



-    let mut fiber = FiberNode::create_fiber_from_element(element);
+    let mut fiber ;
+    if derive_from_js_value(&element, "type") == REACT_FRAGMENT_TYPE {
+        let props = derive_from_js_value(&element, "props");
+        let children = derive_from_js_value(&props, "children");
+        fiber = FiberNode::create_fiber_from_fragment(children, key);
+    } else {
+        fiber = FiberNode::create_fiber_from_element(element);
+    }

This way, our React implementation can support Fragment.

However, there is another case that needs to be supported, for example:



function App() {
  const arr = [<span>Hello</span>, <span>World</span>]
  return <div>{arr}</div>
}

In the above example, Fragment is not explicitly used, but we still need to add a layer of handling, as shown below:

This mainly involves modifying the update_from_map function in child_fiber.rs. For more details, please refer to here.

Please kindly give me a star!

Grayscale Release System OOM Troubleshooting

ayou — Mon, 12 Aug 2024 10:58:25 +0000

Previously, in this article, we discussed how to implement a grayscale release system for Node.js services using a process-based approach. The code snippet below demonstrates the concept:

// index.js
const cp = require('child_process')
const url = require('url')
const http = require('http')

const child1 = cp.fork('child.js', [], {
  env: {PORT: 3000},
})
const child2 = cp.fork('child.js', [], {
  env: {PORT: 3001},
})

function afterChildrenReady() {
  let readyN = 0
  let _resolve

  const p = new Promise((resolve) => {
    _resolve = resolve
  })

  const onReady = (msg) => {
    if (msg === 'ready') {
      if (++readyN === 2) {
        _resolve()
      }
    }
  }

  child1.on('message', onReady)
  child2.on('message', onReady)

  return p
}

const httpServer = http.createServer(function (req, res) {
  const query = url.parse(req.url, true).query

  if (query.version === 'v1') {
    http.get('http://localhost:3000', (proxyRes) => {
      proxyRes.pipe(res)
    })
  } else {
    http.get('http://localhost:3001', (proxyRes) => {
      proxyRes.pipe(res)
    })
  }
})

afterChildrenReady().then(() => {
  httpServer.listen(8000, () => console.log('Start http server on 8000'))
})

// child.js
const http = require('http')

const httpServer = http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'})
  setTimeout(() => {
    res.end('handled by child, pid is ' + process.pid + '\n')
  }, 1000)
})

httpServer.listen(process.env.PORT, () => {
  process.send && process.send('ready')
  console.log(`Start http server on ${process.env.PORT}`)
})

In summary, when running index.js, two child processes are forked. The main process determines which child process to proxy based on the request parameters, enabling different users to see different content.

However, due to the additional layer of proxying, the performance of the service is affected. To optimize it, we can consider reusing TCP connections by using the agent parameter when making http.request calls (for more details, see Using HTTP Agent for keep-alive in Node.js). However, this optimization caused issues with the service.

Let's simulate the scenario. First, modify the code snippet above to enable TCP connection reuse and limit it to one connection:

const agent = http.Agent({keepAlive: true, maxSockets: 1})

const httpServer = http.createServer(function (req, res) {
  const query = url.parse(req.url, true).query

  if (query.version === 'v1') {
    http.get('http://localhost:3000', {agent}, (proxyRes) => {
      proxyRes.pipe(res)
    })
  } else {
    http.get('http://localhost:3001', {agent}, (proxyRes) => {
      proxyRes.pipe(res)
    })
  }
})

Next, we use autocannon -c 400 -d 100 http://localhost:8000 to perform load testing.

The test results reveal the following:

During the load test, accessing http://localhost:8000 times out.
During the load test, the memory usage rapidly increases.
After the load test, accessing http://localhost:8000 still times out, but the memory usage gradually decreases. It takes a while before a response is received.

We can think of TCP connections as railway tracks, where each HTTP content is divided into multiple train cars transported on these tracks:

Due to having only one route between the proxy and server, when client requests arrive too quickly, they need to wait in a queue for processing:

This explains why requests time out during the load test.

Furthermore, because the proxy generates many "requests" in the queue, memory usage rapidly increases. This can be further analyzed using Node.js's inspect feature.

To utilize this feature, add the --inspect parameter when starting the Node.js process. For the child processes started with fork, use execArgv to specify the parameter, as shown below:

const child1 = cp.fork('child.js', [], {
  env: {PORT: 3000},
  execArgv: ['--inspect=9999'],
})
const child2 = cp.fork('child.js', [], {
  env: {PORT: 3001},
  execArgv: ['--inspect=9998'],
})

Then, open the Chrome DevTools, click on Node.js's DevTools, and add three connections to observe the following:

Here, we only focus on the master process. First, take a memory snapshot, then start the load testing script, and after running for a while, take another memory snapshot. Compare the two snapshots, and the results are as follows:

We can see that there are indeed many new ClientRequest instances between the two snapshots, confirming our previous speculation.

After the load test, even though no additional requests enter the proxy, due to the backlog of requests and the artificial 1-second delay added to each response in child.js, processing these backlogged requests becomes very slow. This explains why memory usage slowly decreases and it takes a while before a response is received.

The saying "Don't optimize prematurely" is a valuable lesson in software development, and we truly experienced it this time, especially when dealing with optimization techniques that we only partially understood. The root cause of this issue was our assumption that we had some knowledge about Node.js's Agent, which turned out to be unnecessary, and we didn't carefully analyze its impact or conduct detailed performance testing.

Implement React v18 from Scratch Using WASM and Rust - [22] Implement memo

ayou — Fri, 02 Aug 2024 07:07:05 +0000

Based on big-react，I am going to implement React v18 core features from scratch using WASM and Rust.

Code Repository：https://github.com/ParadeTo/big-react-wasm

The tag related to this article：v22

The previous articles were focused on exploring performance optimization features related to React. However, there is still one missing feature: memo. Today, let's implement it. Take the following code as an example:

import {useState, memo} from 'react'

export default function App() {
  const [num, update] = useState(0)
  console.log('App render', num)
  return (
    <div onClick={() => update(num + 1)}>
      <Cpn num={num} name={'cpn1'} />
      <Cpn num={0} name={'cpn2'} />
    </div>
  )
}

const Cpn = memo(function ({num, name}) {
  console.log('render', name)
  return (
    <div>
      {name}: {num}
      <Child />
    </div>
  )
})

function Child() {
  console.log('Child render')
  return <p>i am child</p>
}

When initially rendered, the following will be printed:

App render 0
render cpn1
Child render
render cpn2
Child render

After clicking, only the first Cpn component should be re-rendered, and the console will print:

App render 1
render cpn1
Child render

Now let's see how to implement this.

First, we need to import the memo method from the React library, as shown below:

#[wasm_bindgen]
pub unsafe fn memo(_type: &JsValue, compare: &JsValue) -> JsValue {
    let fiber_type = Object::new();

    Reflect::set(
        &fiber_type,
        &"$$typeof".into(),
        &JsValue::from_str(REACT_MEMO_TYPE),
    );
    Reflect::set(&fiber_type, &"type".into(), _type);

    let null = JsValue::null();
    Reflect::set(
        &fiber_type,
        &"compare".into(),
        if compare.is_undefined() {
            &null
        } else {
            compare
        },
    );
    fiber_type.into()
}

In JavaScript, the translation would be as follows:

export function memo(
    type: FiberNode['type'],
    compare?: (oldProps: Props, newProps: Props) => boolean
) {
    const fiberType = {
        $$typeof: REACT_MEMO_TYPE,
        type,
        compare: compare === undefined ? null : compare
    };
    return fiberType;
}

Similar to the previous context Provider, here we also return an object. The passed-in component is saved in the type field, and the second argument is stored in the compare field. The purpose of the compare field should be clear, so I won't elaborate on it. Clearly, this is a new FiberNode type, and we need to add handling for this type in the begin work phase.


fn update_memo_component(
    work_in_progress: Rc<RefCell<FiberNode>>,
    render_lane: Lane,
) -> Result<Option<Rc<RefCell<FiberNode>>>, JsValue> {
    let current = { work_in_progress.borrow().alternate.clone() };
    let next_props = { work_in_progress.borrow().pending_props.clone() };

    if current.is_some() {
        let current = current.unwrap();
        let prev_props = current.borrow().memoized_props.clone();
        if !check_scheduled_update_or_context(current.clone(), render_lane.clone()) {
            let mut props_equal = false;
            let compare = derive_from_js_value(&work_in_progress.borrow()._type, "compare");
            if compare.is_function() {
                let f = compare.dyn_ref::<Function>().unwrap();
                props_equal = f
                    .call2(&JsValue::null(), &prev_props, &next_props)
                    .unwrap()
                    .as_bool()
                    .unwrap();
            } else {
                props_equal = shallow_equal(&prev_props, &next_props);
            }

            if props_equal && Object::is(&current.borrow()._ref, &work_in_progress.borrow()._ref) {
                unsafe { DID_RECEIVE_UPDATE = false };
                work_in_progress.borrow_mut().pending_props = prev_props;
                work_in_progress.borrow_mut().lanes = current.borrow().lanes.clone();
                return Ok(bailout_on_already_finished_work(
                    work_in_progress.clone(),
                    render_lane,
                ));
            }
        }
    }
    let Component = { derive_from_js_value(&work_in_progress.borrow()._type, "type") };
    update_function_component(work_in_progress.clone(), Component, render_lane)
}

The code here is easy to understand. If there is a current value, it means it's not the initial render, so we can check if there are any nodes in the descendant components that meet the priority of this update, and if not, we can perform performance optimizations related to memoization. Specifically:

We retrieve the compare function, and if it doesn't exist, we use the default shallow_equal function (which compares two objects for equality by comparing their keys and values, performing a shallow comparison).
We pass the new and old props to the function obtained above.
If the compare function returns true, we enter the bailout logic.

Otherwise, we enter the update_function_component logic because memo is just an additional layer outside the FunctionComponent. Note that the parameters for update_function_component are different now. Previously, we only had work_in_progress and render_lane because we only considered the case of a FunctionComponent, where we could retrieve the Component from work_in_progress's _type. Now, with the addition of MemoComponent, we need to retrieve the Component from work_in_progress's _type's type.

I won't go into other minor changes, but you can find more details here.