DEV Community: Martin Kavík

MoonZoon Dev News (5): Chat example, MoonZoon Cloud

Martin Kavík — Sun, 04 Jul 2021 14:46:44 +0000

Let's chat!

Welcome to the MoonZoon Dev News!

MoonZoon is a Rust full-stack framework. If you want to read about new MZ features, architecture and interesting problems & solutions - Dev News is the right place.

Chapters

News
Chat example
MoonZoon Cloud

News

Chat example works! The most of this blog post will cover it - I hope it will be interesting even for people who hear about the MoonZoon for the first time.
The second chapter is dedicated to the MoonZoon Cloud - research & reasons.
I've updated my sponsors page and added "Sponsor" button to MZ repositories. More info in the MZ Cloud chapter.

And I would like to thank:

Agathe from Clever Cloud for excellent support and patience with my batches of annoying questions.
To all my sponsors. Thank you for wanting to improve web development with me. Current sponsors:
- tokcum ($96 / month)
- arn-the-long-beard ($24 / m)
- Tomáš Lauer ($24 / m)
- Jesus Guzman, Jr. ($10 / m)
- Colin Woodbury ($8 / m)
- Craig Mayhew ($8 / m)
- Dustin Bensing ($8 / m)
- Dusty Pomerleau ($8 / m)
- Jay Sistar ($8 / m)
- Ke Ding ($8 / m)
- Kuma Taro ($8 / m)
- sabine ($8 / m)
- Shawn MacIntyre ($8 / m)

Chat example

You've already seen the GIF at the top and probably tried the live demo.
The demo is based on the example in the MoonZoon repo. You can follow these steps to run it on your local machine.

The example's code is split into three crates: frontend, backend and shared.

Shared

chat/shared/src/lib.rs:

use moonlight::serde_lite::{self, Deserialize, Serialize};

// ------ UpMsg ------

#[derive(Serialize, Deserialize, Debug)]
pub enum UpMsg {
    SendMessage(Message),
}

// ------ DownMsg ------

#[derive(Serialize, Deserialize, Debug)]
pub enum DownMsg {
    MessageReceived(Message),
}

// ------ Message ------

#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct Message {
    pub username: String,
    pub text: String,
}

The only purpose of the shared crate is to provide items needed for frontend-backend communication.

UpMsg is sent from the frontend to backend in a Fetch request.
DownMsg is sent from the backend to frontend in a Server-Send Event stream.

All items have to be (de)serializable because they are transferred as JSON. We use serde_lite instead of serde to reduce the Wasm file size of the frontend app.

moonlight is a MoonZoon crate and it works as a "bridge" between the frontend (Zoon) and backend (Moon).

Backend

chat/backend/src/main.rs:

use moon::*;
use shared::{DownMsg, UpMsg};

async fn frontend() -> Frontend {
    Frontend::new().title("Chat example").append_to_head(
        "
        <style>
            html {
                background-color: black;
            }
        </style>",
    )
}

async fn up_msg_handler(req: UpMsgRequest<UpMsg>) {
    println!("{:?}", req);

    let UpMsgRequest { up_msg, cor_id, .. } = req;
    let UpMsg::SendMessage(message) = up_msg;

    sessions::broadcast_down_msg(&DownMsg::MessageReceived(message), cor_id).await;
}

#[moon::main]
async fn main() -> std::io::Result<()> {
    start(frontend, up_msg_handler, |_| {}).await
}

The function frontend is invoked on the the web browser request (if the path doesn't start with _api). The response is HTML that starts the Zoon (the frontend part).
The function up_msg_handler handles message requests from the Zoon. Zoon sends in the UpMsgRequest:
- Your UpMsg.
- New CorId (aka correlation id) generated for each request.
- SessionId generated in the Zoon app before it connects to the Moon.
- Option<AuthToken> containing String defined in your Zoon app.

Sessions

sessions are virtual actors managed by the Moon. Each SessionActor represents a live connection between Zoon and Moon apps.

You can send your DownMsg to all connected Zoon apps by calling sessions::broadcast_down_msg (demonstrated in the code snippet above).

If you want to send the message to only one session (e.g. to simulate a standard request-response mechanism):

let UpMsgRequest { up_msg, cor_id, session_id, .. } = req;
let UpMsg::SendMessage(message) = up_msg;

sessions::by_session_id()
    .get(session_id)
    .unwrap()
    .send_down_msg(&DownMsg::MessageReceived(message), cor_id).await;

Where by_session_id() returns an actor index. Then we try to find the actor and calls its method send_down_msg.

Notes:

All actor methods are asynchronous because the requested actor may live in another server or it doesn't live at all - then the Moon app has to start it and load its state into the main memory before it can process your call. And all those operations and the business logic processing take some time, so asynchronicity allows you to spend the time in better ways than just waiting.
Index API will change a bit during the future development to support server clusters (e.g. get will be probably async).

Actix

I was talking about it in the previous blog post so just one code snippet for newcomers:

use moon::*;
use moon::actix_web::{get, Responder};

async fn frontend() -> Frontend {
    Frontend::new().title("Actix example")
}

async fn up_msg_handler(_: UpMsgRequest<()>) {}

#[get("hello")]
async fn hello() -> impl Responder {
    "Hello!"
}

#[moon::main]
async fn main() -> std::io::Result<()> {
    start(frontend, up_msg_handler, |cfg|{
        cfg.service(hello);
    }).await
}

cfg in the snippet is actix_web::web::ServiceConfig

Frontend

The skeleton of chat/frontend/src/lib.rs:

use shared::{DownMsg, Message, UpMsg};
use zoon::{eprintln, *};

// ------ ------
//    Statics
// ------ ------
// ...

// ------ ------
//   Commands
// ------ ------
// ...

// ------ ------
//     View
// ------ ------
// ...

// ------ ------
//     Start
// ------ ------
// ...

Statics

#[static_ref]
fn username() -> &'static Mutable<String> {
    Mutable::new("John".to_owned())
}

#[static_ref]
fn messages() -> &'static MutableVec<Message> {
    MutableVec::new()
}

#[static_ref]
fn new_message_text() -> &'static Mutable<String> {
    Mutable::new(String::new())
}

#[static_ref]
fn connection() -> &'static Connection<UpMsg, DownMsg> {
    Connection::new(|DownMsg::MessageReceived(message), _| {
        messages().lock_mut().push_cloned(message);
    })
    // .auth_token_getter(|| AuthToken::new("my_auth_token"))
}

Data stored in functions marked by the attribute #[static_ref] are lazily initialized on the first call.
Read the excellent tutorial for Mutable and signals in the futures_signals crate.
A correlation id is automatically generated and sent to the Moon with each request.
A session id is automatically generated when the Connection is created. Then it's sent with each UpMsg.

Commands

fn set_username(name: String) {
    username().set(name);
}

fn set_new_message_text(text: String) {
    new_message_text().set(text);
}

fn send_message() {
    Task::start(async {
        connection()
            .send_up_msg(UpMsg::SendMessage(Message {
                username: username().get_cloned(),
                text: new_message_text().take(),
            }))
            .await
            .unwrap_or_else(|error| eprintln!("Failed to send message: {:?}", error))
    });
}

Task::start spawn the given Future. (Note: Multithreading in Zoon apps isn't supported yet.)

View

The skeleton of the View section:

fn root() -> impl Element {
    Column::new()
        .s(Padding::new().all(30))
        .s(Spacing::new(20))
        .item(received_messages())
        .item(new_message_panel())
        .item(username_panel())
}

// ------ received_messages ------
// ...

// ------ new_message_panel ------
// ...

// ------ username_panel ------
// ...

When the root function is invoked (note: it's invoked only once), all elements are immediately created and rendered to the browser DOM. (It means, for instance, methods Column::new() or .item(..) writes to DOM.)
s is the abbreviation for style. All elements that implement the ability Styleable have the method .s(...). The method accepts all items that implement the trait Style.
Column is a div with CSS properties display: flex and flex-direction: column.
All built-in elements (Column, Row, etc.) have the HTML class similar to the element name, e.g. column or row.

// ------ received_messages ------

fn received_messages() -> impl Element {
    Column::new().items_signal_vec(messages().signal_vec_cloned().map(received_message))
}

fn received_message(message: Message) -> impl Element {
    Column::new()
        .s(Padding::new().all(10))
        .s(Spacing::new(6))
        .item(
            El::new()
                .s(Font::new().bold().color(NamedColor::Gray10).size(17))
                .child(message.username),
        )
        .item(
            El::new()
                .s(Font::new().color(NamedColor::Gray8).size(17))
                .child(message.text),
        )
}

Column::new().items_signal_vec(messages().. means the Column's items are synchronized with messages. I.e. when you add a new item to messages, the new item is rendered in the Column.
Spacing::new(6) sets the CSS property gap to 6px.
NamedColor is a very small and temporary enum of colors. It will be replaced probably with color pallets or a compile-time color generator later. In the meantime, you can create your own enums or other items that implement the trait Color. See also the Color section in the MZ docs.
It's surprisingly difficult to set the font size correctly. See Size and Font Size sections in the MZ docs for more info. Font::size just sets font-size in px until I resolve it properly.
El is a simple div. It must have one child (otherwise you get a compilation error).

// ------ new_message_panel ------

fn new_message_panel() -> impl Element {
    Row::new().item(new_message_input()).item(send_button())
}

fn new_message_input() -> impl Element {
    TextInput::new()
        .s(Padding::new().x(10))
        .s(Font::new().size(17))
        .focus()
        .on_change(set_new_message_text)
        .label_hidden("New message text")
        .placeholder(Placeholder::new("Message"))
        .on_key_down(|event| event.if_key(Key::Enter, send_message))
        .text_signal(new_message_text().signal_cloned())
}

fn send_button() -> impl Element {
    let (hovered, hovered_signal) = Mutable::new_and_signal(false);
    Button::new()
        .s(Padding::new().all(10))
        .s(Background::new()
            .color_signal(hovered_signal.map_bool(|| NamedColor::Green5, || NamedColor::Green2)))
        .s(Font::new().color(NamedColor::Gray10).size(20))
        .on_hovered_change(move |is_hovered| hovered.set(is_hovered))
        .on_press(send_message)
        .label("Send")
}

Row is a div with the CSS property display: flex.
TextInput is an HTML input. It has abilities:
- Styleable
- KeyboardEventAware
- Focusable
- Hoverable
You have to call either .label_hidden(..) or .id(..) to improve accessibility:
- label_hidden sets aria-label to the given value.
- When you set id, then it's expected you create a Label for the input. It's demonstrated in the following snippet.

// ------ username_panel ------

fn username_panel() -> impl Element {
    let id = "username_input";
    Row::new()
        .item(username_input_label(id))
        .item(username_input(id))
}

fn username_input_label(id: &str) -> impl Element {
    Label::new()
        .s(Font::new().color(NamedColor::Gray10))
        .s(Padding::new().all(10))
        .for_input(id)
        .label("Username:")
}

fn username_input(id: &str) -> impl Element {
    TextInput::new()
        .s(Padding::new().x(10))
        .id(id)
        .on_change(set_username)
        .placeholder(Placeholder::new("Joe"))
        .text_signal(username().signal_cloned())
}

Label is an HTML label.

Start

#[wasm_bindgen(start)]
pub fn start() {
    start_app("app", root);
    connection();
}

The function start is invoked automatically from the Javascript code.
Zoon's function start_app appends the element returned from the root function to the element with the id app.
- You can also pass the value None instead of "app" to mount directly to body but it's not recommended.

A question from MZ Discord

JaySistar: "Can a Connection contain app specific state? I thought that i saw that it could, but I don't know which example uses it. In the case of chat, it could be added, so that after a user has "logged in" the username wouldn't need to be sent with each message, but a "username" change message could be sent to change it."

MartinKavik: "Excellent question. It was the first time I was trying to write MZ actors (SessionActor) and to improve its API during the process. I'm not sure yet if the right solution would be to allow to add a custom state directly to SessionActors or developers should create new custom actors like UserActor and create/remove them when "associated" SessionActor is created/removed. So I decided to choose the simplest approach for now (i.e. send username in the Message, basically keep the state in the Zoon app) and eventually revisit the chat example when actors API is stable enough."

MoonZoon Cloud

MoonZoon development goes quite well. Some important Zoon APIs are still missing - like routing, timers or channels. But I've already written them during the Seed development and their MoonZoon alternatives are designed so it's only matter of time. Moon is already usable for writing apps because Up/DownMsg communication works and for other cases you can use Actix directly. The last big missing Moon part is a virtual actor system. However distributable virtual actors are pretty useless without the platform where we can run them and without a storage where we can save their state. Also we would like to deploy simple apps without actors as soon as possible. We need the MoonZoon Cloud.

MoonZoon Cloud is the serverless platform for your MoonZoon apps. Predictable pricing, simplicity and scalability are main goals.

The web administration will be written in MoonZoon.
The CLI part will be implemented as extra mzoon (MoonZoon CLI tool) commands.
The default app URL will be https://[your_app_name].mzoon.app

Current MZ Cloud state

Research.

Clever Cloud looks like the best candidate for the MoonZoon Cloud infrastructure provider. The MZ chat example works without problems: mz-chat-example.mzoon.app (HTTP/2, automatic HTTPS on a custom domain, native support for Rust apps, API for almost everything, etc.). Their support is excellent.

Another possible provider could be Google Cloud Run because it supports scaling to zero and also server-sent events.
But Clever Cloud has more predictable pricing and it's less complex because you don't need to manage Docker containers.

However MZ Cloud may offer multiple providers in the future - it means you would be able to choose Clever or Google Cloud. (Please let me know if you know other suitable providers.)

Why I'm creating MZ Cloud?

There are three reasons why I want to write MZ Cloud:

It's the missing piece to complete my journey to an ideal web development experience.
I need it for my MoonZoon apps.
I hope it will be the main source of income for the MoonZoon development. So if you want to speed up the MZ development, make it sustainable and get early access to the Cloud (once it's ready for it) and other benefits - please visit my Sponsors page.

Note: If the topic "Open-Source & Financial Struggle" is new to you, I've written down one idea how to resolve it and a list of related articles, videos and books: OpenHope.

Note for Seed users: I continue to maintain Seed as usual. Also I can imagine we can write tools for deploying Seed apps to the MZ Cloud when there is demand for it. However I'll focus mainly on my sponsors, because these people and companies rely on Seed and I want to cover their back and watch their successes. Also Seed is now basically feature-complete, I don't remember where somebody reported a bug, community is nice to newcomers and I can't work indefinitely for free.

Don't hesitate to write me (public or private message) if you have some questions about Cloud or ideas or requirements or other things on your mind. You are awesome, thank you.

And that's all for today!
Thank You for reading and I hope you are looking forward to the next episode.

Martin

P.S.
We are waiting for you on Discord.

MoonZoon Dev News (4): Actix, Async CLI, Error handling, Wasm-pack installer

Martin Kavík — Thu, 10 Jun 2021 16:53:58 +0000

Unlimited Actix power!

Welcome to the MoonZoon Dev News!

MoonZoon is a Rust full-stack framework. If you want to read about new MZ features, architecture and interesting problems & solutions - Dev News is the right place.

Chapters

News
Actix
- Why Actix?
- Moon API changes
- MoonZoon.toml changes
- Server-Sent Events
- Moon endpoint changes
mzoon
- Async runtime
- Error handling
- File Watchers
- File Compressors
- File Visitor
- Wasm-pack installer

News

Moon - Warp replaced with Actix. There are API changes to allow you to use Actix directly from your apps.
mzoon - Rewritten with Tokio, implemented --open parameter and wasm-pack installer.
The entire MoonZoon codebase should be clean enough now. Comments are still missing and there should be more tests but if you wanted to know how it really works, you don't have to be afraid to read the code in the MZ repo.
You can select the required mzoon version for heroku-buildpack-moonzoon by adding the file mzoon_commit to your repo with a MZ project.

You'll read about Moon and mzoon improvements mentioned above in the following chapters.

And I would like to thank:

All Rust libraries maintainers. It's tough work but it allows us to write clean code and amazing products.

Actix

Why Actix?

It's fast, async and popular.
Supports HTTP/2 and probably also H3 in the future (related issue).
Actix actor framework could be a good foundation for the first version of virtual actors.
It uses Tokio under the hood. It's the most popular async runtime and we can use it also in mzoon.
The API feels more intuitive than the Warp's one to me. And we were fighting with Warp during the Moon development.
Tide supports only HTTP/1.x.
Why not Rocket (detailed explanation to answer questions on the MZ chat):
- It's too much opinionated with too many batteries included to be used as just a library. Moon would fight with Rocket. Examples: We would need to disable Rocket's console logging, then explain to users they can't use Rocket.toml and hope that Rocket's live reloading won't break Moon's file watchers, etc.
- It's still less popular / downloaded than Actix.
- We would need to find a compatible actor framework to write PoC of virtual actors.
- I've already rewritten Moon to Actix (before Rocket published the version 0.5.0-rc.1). Actix works great so I don't see a real reason to sacrifice dozens hours of my free time and slow down Moon development by another rewriting.
- Every framework has its own problems - there is a chance we would encounter a show-stopper during the Rocket integration.
- It doesn't really matter what framework we choose from the long-term view. The number of reasons why you want to communicate directly with the lower-level framework (Actix/Rocket) in Moon will decrease proportionally to the progress of Moon API development.
- Rocket and Actix (and other frameworks) have pretty similar performance and API in many cases so all Rust web developers should be able to learn the Actix API quickly and even migrate a project from other frameworks in a reasonable time.

Moon API changes

The simplest Moon app:

use moon::*;

async fn frontend() -> Frontend {
    Frontend::new().title("Actix example")
}

async fn up_msg_handler(_: UpMsgRequest) {}

#[moon::main]
async fn main() -> std::io::Result<()> {
    start(frontend, up_msg_handler, |_|{}).await
}

main is now async so we no longer need the init function - you can write your async code directly to the main's body.
start! macro has been rewritten to a simple function start. The interesting is the third argument. See the next example:

use moon::*;
use moon::actix_web::{get, Responder};

async fn frontend() -> Frontend {
    Frontend::new().title("Actix example")
}

async fn up_msg_handler(_: UpMsgRequest) {}

#[get("hello")]
async fn hello() -> impl Responder {
    "Hello!"
}

#[moon::main]
async fn main() -> std::io::Result<()> {
    start(frontend, up_msg_handler, |cfg|{
        cfg.service(hello);
    }).await
}

It's the code used in the GIF at the top.
cfg in the example is actix_web::web::ServiceConfig. It allows you to create custom Actix endpoints and configure the server as you wish.
Multiple crates and items are reexported from moon to mitigate dependency hell caused by incompatible versions and to simplify your Cargo.toml. The current list looks like this:

pub use trait_set::trait_set;
pub use actix_files;
pub use actix_http;
pub use actix_web;
pub use actix_web::main;
pub use futures;
pub use mime;
pub use mime_guess;
pub use parking_lot;
pub use serde;
pub use tokio;
pub use tokio_stream;
pub use uuid;

MoonZoon.toml changes

port = 8080
# port = 8443
https = false
cache_busting = true
backend_log_level = "warn" # "error" / "warn" / "info" / "debug" / "trace"

[redirect]
port = 8081
enabled = false

[watch]
frontend = [
    "frontend/Cargo.toml",
    "frontend/src",
]
backend = [
    "backend/Cargo.toml",
    "backend/src",
]

There is a new property backend_log_level. It sets the env_logger log level.
- info level is useful for debugging because it shows all requests (demonstrated in the GIF at the top).
- Note: There are also independent 404 and 500 error handlers that call eprintl with the error before they pass the response to the client.
- Note: fern looks like a good alternative if we find out env_logger isn't good enough. (Thanks azzamsa for the suggestion.)
[redirect_server] has been renamed to [redirect] because there is no longer a redirection server. The new RedirectMiddleware is activated when you enable the redirect.

Caching has been also improved:

cache_busting = true:
- mzoon generates files like frontend_bg_[uuid].wasm, where uuid is a frontend build id with the type u128.
- Moon serves the files with the header CacheControl set to MaxAge(31536000) (1 year).
cache_busting = false
- mzoon doesn't change the file names at all - e.g. frontend_bg.wasm.
- Moon serves the files with the header ETag with a strong etag set to the frontend build id. (See MDN ETag docs for more info.)

Server-Sent Events

Actix unfortunately doesn't have an official SSE API so I've decided to write a custom one. The current implementation is in the file crates/moon/src/sse.rs.

It sends a ping to all connections every 10 seconds to recognize the disconnected ones.
Integration:
1. let sse = SSE::start();
2. App::new().app_data(sse.clone())

Moon's SSE connector:

async fn sse_responder(
    sse: web::Data<Mutex<SSE>>,
    shared_data: web::Data<SharedData>,
) -> impl Responder {
    let (connection, event_stream) = sse.new_connection();
    let backend_build_id = shared_data.backend_build_id.to_string();

    if connection
        .send("backend_build_id", &backend_build_id)
        .is_err()
    {
        return HttpResponse::InternalServerError()
            .reason("sending backend_build_id failed")
            .finish();
    }

    HttpResponse::Ok()
        .insert_header(ContentType(mime::TEXT_EVENT_STREAM))
        .streaming(event_stream)
}

and the frontend reloader:

async fn reload_responder(sse: web::Data<Mutex<SSE>>) -> impl Responder {
    let _ = sse.broadcast("reload", "");
    HttpResponse::Ok()
}

Warning: Keep in mind that browsers can open only 6 SSE connections over HTTP/1.x to the same domain. It means when you open multiple browser tabs pointing to http://localhost, you may observe infinite loadings or similar problems. The limit for HTTP/2 is 100 connections by default, but can be negotiated between the client and the server.

Moon endpoint changes

App::new()
    // ...
    .service(Files::new("_api/public", "public"))
    .service(
        web::scope("_api")
            .route(
                "up_msg_handler",
                web::post().to(up_msg_handler_responder::<UPH, UPHO>),
            )
            .route("reload", web::post().to(reload_responder))
            .route("pkg/{file:.*}", web::get().to(pkg_responder))
            .route("sse", web::get().to(sse_responder))
            .route("ping", web::to(|| async { "pong" })),
    )
    .route("*", web::get().to(frontend_responder::<FRB, FRBO>))

All backend endpoints are prefixed with _api to prevent conflicts with frontend routes. There are other solutions like hash routing or moving the frontend endpoint to another domain or a prefix for frontend urls but these solutions often lead to many unpredictable problems. Let's keep it simple.

There is a new simple endpoint ping. It's useful for testing if the server is alive. I can imagine we can also implement a heartbeat later (Moon would call a predefined endpoint in a configured interval).

mzoon

Async runtime

mzoon was rewritten with Tokio. The main goal was to remove spaghetti code and boilerplate caused by manual handling of threads, channels and signals. The secondary goal was error handling and improved performance.

There are also other async runtimes like async-std or smol but I've decided to choose the most battle-tested and popular one. Another reason for Tokio is Actix, because Actix is based on Tokio so there should be less context switching during the MoonZoon development.

Error handling

During the mzoon refactor, I've decided to integrate two nice libraries to eliminate boilerplate:

The first one is anyhow. It allows you to write ? wherever you want to return an error early. No need to write error mappers or similar stuff.

anyhow also provides the method context (and its lazy version with_context) and a macro anyhow! for creating errors. An example:

// `anyhow::Result<T>` is an alias 
// for a standard `Result<T, anyhow::Error>`
use anyhow::{anyhow, Context, Result};  

pub async fn build_backend(release: bool, https: bool) -> Result<()> {
    ...
    Command::new("cargo")
        .args(&args)
        .status()
        .await
        .context("Failed to get frontend build status")?
        .success()
        .err(anyhow!("Failed to build backend"))?;
    ...
}

Notes:
- The method .err from the example above is implemented in the crate bool_ext.
- anyhow is useful mostly for apps. If you are writing a library, look at thiserror (written by the same author).

The second error handling library is fehler. I've decided to integrate it into mzoon once I noticed that many functions were returning Ok(()) and their signature was ... -> Result<()>. Ok(()) is a side-effect of anyhow because you want to use ? as much as possible to automatically convert concrete errors to anyhow::Error. The second reason why there were many Ok(())s is the fact that mzoon does many file operations.

I recommend to read these articles about fehler - A brief apology of Ok-Wrapping and From failure to Fehler.

So when we combine both libraries, we can write a clean code without boilerplate:

use anyhow::Error;
use fehler::throws;
// ...
#[throws]
#[tokio::main]
async fn main() {
    // ...
    match opt {
        Opt::New { project_name, here } => command::new(project_name, here).await?,
        Opt::Start { release, open } => command::start(release, open).await?,
        Opt::Build { release } => command::build(release).await?,
    }
}

#[throws] automatically converts the return type from -> () to -> Result<(), Error> and you don't have to write ugly Ok(()) or wrap the entire match into Ok().
All errors before ? are automatically converted to Error and nicely written to the terminal with their contexts thanks to anyhow.

Let's look at another example from mzoon where we integrated the crate apply to help with chaining:

// ...
use anyhow::{Context, Error};
use apply::{Also, Apply};
use fehler::throws;

#[throws]
pub fn run_backend(release: bool) -> Child {
    println!("Run backend");
    MetadataCommand::new()
        .no_deps()
        .exec()?
        .target_directory
        .also(|directory| directory.push(if release { "release" } else { "debug" }))
        .also(|directory| directory.push("backend"))
        .apply(Command::new)
        .spawn()
        .context("Failed to run backend")?
}

Tip: Don't try to write "functional chains" at all costs. It's easy to get lost in long chains, they may be difficult to change and they may increase cognitive load because the reader has to keep intermediate steps/states in his working memory. The example above is very close to the case where clean code is uncomfortable to read.
Note: We have to find the target directory and call the Moon app binary (backend) manually because cargo run always tries to build the project even if the project has been already built. It slows down the build pipeline and writes unnecessary messages to the terminal. Related issue.

File Watchers

While I was rewriting std channels to the tokio ones, I encountered the problem with the notify API. Also its event debouncing wasn't working properly in mzoon. Fortunately notify maintainers are working on a new major version and they've already published 5.0.0-pre.x versions. The API is more flexible but debouncing is still missing in the new notify and in the crate futures-rs. So I had to write a custom debouncer.

The snippets below belong to the current ProjectWatcher implementation in /crates/mzoon/src/watcher/project_watcher.rs.

use notify::{immediate_watcher, Event, RecommendedWatcher, RecursiveMode, Watcher};
use tokio::sync::mpsc::{self, UnboundedReceiver, UnboundedSender};
// ...

pub struct ProjectWatcher {
    watcher: RecommendedWatcher,
    debouncer: JoinHandle<()>,
}

impl ProjectWatcher {
    #[throws]
    pub fn start(paths: &[String], debounce_time: Duration) -> (Self, UnboundedReceiver<()>) {
        let (sender, receiver) = mpsc::unbounded_channel();
        let watcher = start_immediate_watcher(sender, paths)?;
        let (debounced_sender, debounced_receiver) = mpsc::unbounded_channel();

        let this = ProjectWatcher {
            watcher,
            debouncer: spawn(debounced_on_change(
                debounced_sender,
                receiver,
                debounce_time,
            )),
        };
        (this, debounced_receiver)
    }
    // ...

ProjectWatcher is a general watcher based on the notify's watcher. It's used in mzoon's BackendWatcher and FrontendWatcher.
start_immediate_watcher calls the notify's immediate_watcher function to register watched paths and the callback that is invoked when notify observes a file change. The callback sends () (aka unit) through the sender.
The sender's second half - receiver - is passed to the debouncer. It means the debouncer is able to listen for all registered file system events.
The debounced_sender represents the debouncer's output - basically a stream of debounced units (we can replace units with Events if needed in the future).

async fn debounced_on_change(
    debounced_sender: UnboundedSender<()>,
    mut receiver: UnboundedReceiver<()>,
    debounce_time: Duration,
) {
    let mut debounce_task = None::<JoinHandle<()>>;
    let debounced_sender = Arc::new(debounced_sender);

    while receiver.recv().await.is_some() {
        if let Some(debounce_task) = debounce_task {
            debounce_task.abort();
        }
        debounce_task = Some(spawn(debounce(
            Arc::clone(&debounced_sender),
            debounce_time,
        )));
    }

    if let Some(debounce_task) = debounce_task {
        debounce_task.abort();
    }
}

async fn debounce(debounced_sender: Arc<UnboundedSender<()>>, debounce_time: Duration) {
    sleep(debounce_time).await;
    if let Err(error) = debounced_sender.send(()) {
        return eprintln!("Failed to send with the debounced sender: {:#?}", error);
    }
}

When the unit from the notify's callback is received, then a new task is spawned. The task sleeps for the debounce_time and then a unit is sent through the debounced_sender.
When another unit is received, then the sleeping task is aborted and a new one is created. You can understand it as "debounce time reset".

Notice two same code blocks in the previous snippet:

if let Some(debounce_task) = debounce_task {
    debounce_task.abort();
}

The first usage "resets debounce time", but the second one is basically an alternative to drop. Unfortunately neither Rust nor tokio is able to automatically clean all garbage so we have to do it manually - the task handle does nothing when dropped in most cases.

So... how we can stop the watcher?

The ProjectWatcher doesn't have only one method (start) - there is another one:

#[throws]
pub async fn stop(self) {
    let watcher = self.watcher;
    drop(watcher);
    self.debouncer.await?;
}

Drop notify's RecommendedWatcher.
Dropped watcher means that also our sender has been dropped because it was closed by the closure used as a callback / event handler owned by the watcher.
When the sender is dropped, then receiver.recv().await.is_some() returns false to break the while loop in the debouncer.
The debounce task is aborted if there was one running.

Yeah, it's already quite complicated and error prone but we haven't finished yet.

FrontendWatcher and BackendWatcher have the similar relationship to ProjectWatcher as ProjectWatcher to notify's Watcher. Let's look at the FrontendWatcher skeleton:

pub struct FrontendWatcher {
    watcher: ProjectWatcher,
    task: JoinHandle<Result<()>>,
}

impl FrontendWatcher {
    #[throws]
    pub async fn start(config: &Config, release: bool, debounce_time: Duration) -> Self {
        let (watcher, debounced_receiver) =
            ProjectWatcher::start(&config.watch.frontend, debounce_time)
                .context("Failed to start the frontend project watcher")?;
        // ...        
        Self {
            watcher,
            task: spawn(on_change(
                debounced_receiver,
                // ...
            )),
        }
    }

    #[throws]
    pub async fn stop(self) {
        self.watcher.stop().await?;
        self.task.await??;
    }
}

As you can see, there is another stop method that calls the previous stop method and the remaining code is very similar to the ProjectWatcher implementation.

Let's look at the last snippet to know the whole watcher story (/crates/mzoon/src/command/start.rs):

#[throws]
pub async fn start(release: bool, open: bool) {
    // ...
    let frontend_watcher = build_and_watch_frontend(&config, release).await?;
    let backend_watcher = build_run_and_watch_backend(&config, release, open).await?;

    signal::ctrl_c().await?;
    println!("Stopping watchers...");
    let _ = join!(
        frontend_watcher.stop(),
        backend_watcher.stop(),
    );
    println!("Watchers stopped");
}

#[throws]
async fn build_and_watch_frontend(config: &Config, release: bool) -> FrontendWatcher {
    if let Err(error) = build_frontend(release, config.cache_busting).await {
        eprintln!("{}", error);
    }
    FrontendWatcher::start(&config, release, DEBOUNCE_TIME).await?
}

So I can imagine there are some opportunities for another refactor round:

"Hide" loops and debouncer inside Streams.
Use notify's debouncer once it's integrated into the library.
Use async drops once Rust supports them or an alternative.
- See the related article Asynchronous Destructors from the fehler's author.
If you want to investigate the option "Wait until all task done" so we can just abort all tasks in a standard drop and then wait for async runtime to finish, there is the entrance to the rabbit hole.

Feel free to create a PR when you manage to simplify the code.

File Compressors

Frontend files are served compressed to get them quickly to users and to reduce network traffic and server load. Only app files (in the pkg directory) are compressed at the moment but we'll probably compress the entire public folder in the future.

mzoon compresses files when the app has been built in the release mode. The result is three files instead of one: file.xxx (the original), file.xxx.gz and file.xxx.br. Then Moon serves them according to the ACCEPT_ENCODING header sent by clients.

We would use only Brotli algorithm because it produces the smallest files but Firefox supports only Gzip over HTTP. All browsers support Brotli with HTTPS.

Note: If we decide to compress non-cacheable dynamic content - like messages between frontend and backend - then we will probably choose Gzip because it's faster than Brotli.

Let's look at the implementation. The first snippet is from /crates/mzoon/src/helper/file_compressor.rs:

use crate::helper::ReadToVec;
use async_trait::async_trait;
use brotli::{enc::backward_references::BrotliEncoderParams, CompressorReader as BrotliEncoder};
use flate2::{bufread::GzEncoder, Compression as GzCompression};
// ...

#[async_trait]
pub trait FileCompressor {
    async fn compress_file(content: Arc<Vec<u8>>, path: &Path, extension: &str) -> Result<()> {
        let path = compressed_file_path(path, extension);
        let mut file_writer = fs::File::create(&path)
            .await
            .with_context(|| format!("Failed to create the file {:#?}", path))?;

        let compressed_content = spawn_blocking(move || Self::compress(&content)).await??;

        file_writer.write_all(&compressed_content).await?;
        file_writer.flush().await?;
        Ok(())
    }

    fn compress(bytes: &[u8]) -> Result<Vec<u8>>;
}
//...
// ------ Brotli ------

pub struct BrotliFileCompressor;

#[async_trait]
impl FileCompressor for BrotliFileCompressor {
    fn compress(bytes: &[u8]) -> Result<Vec<u8>> {
        BrotliEncoder::with_params(bytes, 0, &BrotliEncoderParams::default()).read_to_vec()
    }
}

// ------ Gzip ------

pub struct GzipFileCompressor;

#[async_trait]
impl FileCompressor for GzipFileCompressor {
    fn compress(bytes: &[u8]) -> Result<Vec<u8>> {
        GzEncoder::new(bytes, GzCompression::best()).read_to_vec()
    }
}

#[async_trait] allows us to write async methods in traits. (The crate async_trait, from the author of anyhow and thiserror.)
The combination of async-trait and fehler is deadly for the Rust compiler. That's why you see Ok(()) + Result<()> instead of #[throws]. I'm not sure if it's async-trait or fehler problem, feel free to investigate it more and let me know.
We need to call spawn_blocking instead of spawn to move compression to a new thread because both encoders / compressors are blocking. I was trying to use async-compression, but there was a bug probably somewhere close to the GzEncoder - the MZ example counter was producing a wasm file that had always only 9KB instead of 16KB. Also I had to use async-compression's futures encoders with the compat layer to resolve the problem with incompatible tokio versions. Feel free to investigate it more and let me know.
Tip: Don't forget to call .flush() after .write_all(). Sometimes it works without .flush(), sometimes it doesn't, so it's difficult to debug.
read_to_vec is a custom helper - see /crates/mzoon/src/helper/read_to_vec.rs.
Both encoders are set to compress in the best quality (i.e. to produce the smallest files at the cost of speed).

The second and the last snippet is from /crates/mzoon/src/build_frontend.rs:

use futures::TryStreamExt;

#[throws]
async fn compress_pkg(wasm_file_path: impl AsRef<Path>, js_file_path: impl AsRef<Path>) {
    try_join!(
        create_compressed_files(wasm_file_path),
        create_compressed_files(js_file_path),
        visit_files("frontend/pkg/snippets")
            .try_for_each_concurrent(None, |file| create_compressed_files(file.path()))
    )?
}

#[throws]
async fn create_compressed_files(file_path: impl AsRef<Path>) {
    let file_path = file_path.as_ref();
    let content = Arc::new(fs::File::open(&file_path).await?.read_to_vec().await?);

    try_join!(
        BrotliFileCompressor::compress_file(Arc::clone(&content), file_path, "br"),
        GzipFileCompressor::compress_file(content, file_path, "gz"),
    )
    .with_context(|| format!("Failed to create compressed files for {:#?}", file_path))?
}

All files are compressed and generated in parallel thanks to spawn_blocking (explained before) and thanks to tokio::fs (we don't block the working thread by waiting for OS file operations).
visit_files is a stream of files (explained in the next section). It works nice with the function try_for_each_concurrent.

File Visitor

When you want to iterate over all files in the given directory and its nested folders, then it's relatively straightforward with the standard Rust library. Just go to the Rust docs for std::fs::read_dir and copy the provided example. Also there is chance we'll see the function fs::read_dir_all in std. Or you can use the crate walkdir from a very experienced maintainer of many Rust libraries.

However the Rust async world is still pretty new and messy. If I chose smol instead of tokio and was brave enough to use the library with only 602 downloads, then I would probably integrate the crate async_walkdir.

Another approach would be to use walkdir to create a list of files and then process the list as needed in parallel. However it doesn't sound as a clean solution and in the case of a large directory tree, you want to return early when the processing fails or when your file search is complete.

I'm not a big fan or recursive functions because:

They often lead to increased cognitive load.
Stack overflow is difficult to catch and debug.
Rust doesn't have a good support for TCO/TCE (tail call optimization / elimination), although there are some libraries like Tailcall and maybe promising news in rust-lang/rfcs.
You often need to use Box in Rust recursive constructs (both functions and types need boxed items). The crate async-recursion basically just wraps the Future into a Box.
Why does NASA not allow recursion?

Fortunately during intensive reading and searching for a better solution, I've found a nice answer on stackoverflow.com compatible with tokio and futures. I've refactored it a little bit and saved to /crates/mzoon/src/helper/visit_files.rs. The code:

pub fn visit_files(path: impl Into<PathBuf>) -> impl Stream<Item = Result<DirEntry>> + Send + 'static {
    #[throws]
    async fn one_level(path: PathBuf, to_visit: &mut Vec<PathBuf>) -> Vec<DirEntry> {
        let mut dir = fs::read_dir(path).await?;
        let mut files = Vec::new();

        while let Some(child) = dir.next_entry().await? {
            if child.metadata().await?.is_dir() {
                to_visit.push(child.path());
            } else {
                files.push(child)
            }
        }
        files
    }

    stream::unfold(vec![path.into()], |mut to_visit| {
        async {
            let path = to_visit.pop()?;
            let file_stream = match one_level(path, &mut to_visit).await {
                Ok(files) => stream::iter(files).map(Ok).left_stream(),
                Err(error) => stream::once(async { Err(error) }).right_stream(),
            };
            Some((file_stream, to_visit))
        }
    })
    .flatten()
}

(Let me know if you know a better solution or a suitable library.)

Wasm-pack installer

I hate complicated installations and configurations, especially if they aren't cross-platform. In an ideal world, we would just write cargo install mzoon, hit enter and done. Unfortunately it isn't so simple even in the Rust + Cargo world.

The Rust compiler is pretty slow so if there is a chance to avoid compilation, we should use it. It applies especially for CI pipelines. So we have to download pre-compiled binaries. But to use binaries, we firstly have to answer these questions:

What are available binary versions / supported platforms?
What is our platform?
Where we should download wasm-pack?
How should we download and unpack wasm-pack?

1) What are available binary versions / supported platforms?

wasm-pack's repo has associated build pipelines for multiple platforms. So we can just look at the release assets.

The current list (version 0.9.1):

wasm-pack-init.exe (7.16 MB)
wasm-pack-v0.9.1-x86_64-apple-darwin.tar.gz (2.97 MB)
wasm-pack-v0.9.1-x86_64-pc-windows-msvc.tar.gz (2.69 MB)
wasm-pack-v0.9.1-x86_64-unknown-linux-musl.tar.gz (5.02 MB)

Note: wasm-pack-init.exe is actually an uncompressed Windows binary with a different name.

2) What is our platform?

There are multiple ways to determine the platform. Two of them are used in mzoon:

There is a build script build.rs in the mzoon crate with this code:

fn main() {
    println!(
        "cargo:rustc-env=TARGET={}",
        std::env::var("TARGET").unwrap()
    );
}

The only purpose is to "forward" the environment variable TARGET (available only during the build process) to the compilation. Then we can read it in the mzoon code (/crates/mzoon/src/wasm_pack.rs):

const TARGET: &str = env!("TARGET");

Unfortunately we can't use it directly because there are cases where it's too strict. For example, Heroku build pipeline is identified as x86_64-unknown-linux-gnu but we have the binary only for x86_64-unknown-linux-musl. However the available binary works even in that Heroku pipeline. So we need more relaxed platform matching in practice:

cfg_if! {
    if #[cfg(target_os = "macos")] {
        const NEAREST_TARGET: &str = "x86_64-apple-darwin";
    } else if #[cfg(target_os = "windows")] {
        const NEAREST_TARGET: &str = "x86_64-pc-windows-msvc";
    } else if #[cfg(target_os = "linux")] {
        const NEAREST_TARGET: &str = "x86_64-unknown-linux-musl";
    } else {
        compile_error!("wasm-pack pre-compiled binary hasn't been found for the target platform '{}'", TARGET);
    }
}

Note: The macro cfg_if belongs to the crate cfg_if.

In the code above I assume mzoon will be compiled only on the most common platforms. When someone wants to compile mzoon on other platforms, we will need to add a fallback to cargo install. There is also a small check to inform you that you may have incompatible platform:

if TARGET != NEAREST_TARGET {
    println!(
        "Pre-compiled wasm-pack binary '{}' will be used for the target platform '{}'",
        NEAREST_TARGET, TARGET
    );
}

The example output from the Heroku build log:

Building frontend...

Installing wasm-pack...

Pre-compiled wasm-pack binary 'x86_64-unknown-linux-musl' will be used for the target platform 'x86_64-unknown-linux-gnu'

wasm-pack installed

3) Where we should download wasm-pack?

wasm-pack contains a self-installer, triggered when the executable name starts with "wasm-pack-init".
The self-installer copy itself next to the rustup executable to make sure it's in PATH.

It isn't a bad idea but there will be problems when multiple MZ projects will need different wasm-pack versions.

wasm-pack uses the crate binary-install to install its binary dependencies like wasm-bindgen or wasm-opt. Those binaries are saved into an OS-specific global cache folder, determined by the function dirs_next::cache_dir() from the crate dirs_next.

It's a better idea, but we still have to manage different wasm-pack versions and I don't like to use global caches too much because it's difficult to remove old files when they are no longer needed.

So the remaining option is to download wasm-pack directly into the user project. We can store it in the target directory. But I think the best option is the frontend directory. Users can use wasm-pack directly if they need it - e.g. to run tests until the test command is implemented in mzoon.

4) How should we download and unpack wasm-pack?

/crates/mzoon/src/helper/download.rs

#[throws]
pub async fn download(url: impl AsRef<str>) -> Vec<u8> {
    reqwest::get(url.as_ref())
        .await?
        .error_for_status()?
        .bytes()
        .await?
        .to_vec()
}

reqwest is popular, universal, async and based on tokio.

/crates/mzoon/src/wasm_pack.rs

const DOWNLOAD_URL: &str = formatcp!(
    "https://github.com/rustwasm/wasm-pack/releases/download/v{version}/wasm-pack-v{version}-{target}.tar.gz",
    version = VERSION,
    target = NEAREST_TARGET,
);

// ...

download(DOWNLOAD_URL)
    .await
    .context(formatcp!(
        "Failed to download wasm-pack from the url '{}'",
        DOWNLOAD_URL
    ))?
    .apply(unpack_wasm_pack)
    .context("Failed to unpack wasm-pack")?;

// ...

#[throws]
fn unpack_wasm_pack(tar_gz: Vec<u8>) {
    let tar = GzDecoder::new(tar_gz.as_slice());
    let mut archive = Archive::new(tar);

    for entry in archive.entries()? {
        let mut entry = entry?;
        let path = entry.path()?;
        let file_stem = path
            .file_stem()
            .ok_or(anyhow!("Entry without a file name"))?;
        if file_stem != "wasm-pack" {
            continue;
        }
        let mut destination = PathBuf::from("frontend");
        destination.push(path.file_name().unwrap());
        entry.unpack(destination)?;
        return;
    }
    Err(anyhow!(
        "Failed to find wasm-pack in the downloaded archive"
    ))?;
}

The formatcp! macro is exported from the nice library const_format.
Decompressing and unpacking is sync (aka blocking), but we still need to wait for wasm-pack installation before we can do anything else (e.g. build frontend). So why complicate our lives with extra wrappers like spawn_blocking? Also we can unblock it later when needed.

And that's all for today!
Thank You for reading and I hope you are looking forward to the next episode.

Martin

P.S.
We are waiting for you on Discord.

MoonZoon Dev News (3): Signals, React-like Hooks, Optimizations

Martin Kavík — Fri, 14 May 2021 16:23:41 +0000

It's small.

It's fast.

What is it?

Rewritten Zoon!

You can try it by yourself: Live demo

Welcome to the MoonZoon Dev News!

MoonZoon is a Rust full-stack framework. If you want to read about new MZ features, architecture and interesting problems & solutions - Dev News is the right place.

Chapters

News
Old Zoon architecture - How React and Rust Hooks work
New Zoon architecture - Signals: You can do it without a Virtual DOM
Builder pattern with rules - Yes, builder pattern can support required parameters
Optimizations - Need for speed. The size matters.

News

Zoon API almost doesn't use macros, it's safer, more expressive and compiler-friendly.
- Working examples rewritten with the new API:
  - counter
  - counters
  - js-framework-benchmark
- Docs updated to reflect changes.
- The trade-off is slightly increased verbosity for Element APIs.
A new article "Rust on the Frontend and Backend" on the blog Always Bet On Rust.
- "An interview with Martin Kavík, creator of the MoonZoon full-stack framework"
The demo project and heroku-buildpack updated. You can use them as a starting point for your experimental MoonZoon apps.
The MoonZoon benchmark is ready to be merged (at the time of writing) to krausest/js-framework-benchmark, however I want to wait until MoonZoon is more mature.
You don't have to be afraid to look at zoon and static_ref_macro crates code.
- They are clean enough thanks to awesome libraries once_cell, futures-signals and dominator.
Some new APIs, configs, features and Brotli / Gzip compression integrated.

I would like to thank:

Pauan for lightning fast resolving of my problems with his libs futures-signals and dominator.
flosse for fighting with Warp in Moon and for MZoon and Moon improvements.
Alexhuszagh for working on lexical and answering my questions.

This blog post is a bit longer but I hope you'll enjoy it!

Old Zoon architecture

How React and Rust Hooks work

First, I would like to write this sentence to sound clever: "The old architecture was based on topologically-aware functions with stable call graph identifiers and local states stored in a heterogenous vector."

Unfortunately, it's not my original idea. It powers React Hooks. Or moxie. Or Crochet.

And now the explanation what is it and why it doesn't work well enough to stay in Zoon (just like all over-engineered stuff).

So let's say we have main and 3 simple functions:

fn main() {
    loop {
        amber()
    }
}

fn amber() {
    mike()
}

fn mike() {
    layla_rose();
    layla_rose()
}

fn layla_rose() { }

Let's add a counter with some helpers and printlns:

use std::sync::atomic::{AtomicUsize, Ordering};
static COUNTER: AtomicUsize = AtomicUsize::new(0);
fn call_id() -> usize { COUNTER.load(Ordering::SeqCst) }
fn increment_call_id() { COUNTER.fetch_add(1, Ordering::SeqCst); }
fn reset_call_id() { COUNTER.store(0, Ordering::SeqCst) }

fn main() {
    for _ in 0..3 { 
        amber();
        reset_call_id()
    }
}

fn amber() {
    increment_call_id();
    println!("amber id: {}", call_id());
    mike()
}

fn mike() {
    increment_call_id();
    println!("mike id: {}", call_id());
    layla_rose();
    layla_rose()
}

fn layla_rose() {
    increment_call_id();
    println!("layla_rose id: {}", call_id());
}

When you run the code (Rust Playground), you should see a loop of the sequence:

amber id: 1
mike id: 2
layla_rose id: 3
layla_rose id: 4

Now you can apply some "magic" to our functions with proc macros like this one to hide unnecessary counter helpers. The result will look like:

fn main() {
    for _ in 0..3 { run(amber) }
}

#[i_am_special]
fn amber() {
    println!("amber id: {}", call_id());
    mike()
}

#[i_am_special]
fn mike() {
    println!("mike id: {}", call_id());
    layla_rose();
    layla_rose()
}

#[i_am_special]
fn layla_rose() {
    println!("layla_rose id: {}", call_id());
}

But let's get back to our non-macro example and improve it by adding STATES and the hook use_age. (Rust Playground)

Note: The code below may look a bit scary but you don't have to understand all implementation details.

use std::sync::atomic::{AtomicUsize, Ordering};
static COUNTER: AtomicUsize = AtomicUsize::new(0);
fn call_id() -> usize { COUNTER.load(Ordering::SeqCst) }
fn increment_call_id() { COUNTER.fetch_add(1, Ordering::SeqCst); }
fn reset_call_id() { COUNTER.store(0, Ordering::SeqCst) }

use std::{sync::Mutex, collections::HashMap};
use once_cell::sync::Lazy;
static STATES: Lazy<Mutex<HashMap<usize, u8>>> = Lazy::new(Mutex::default);

fn use_age(default_value: impl FnOnce() -> u8 + Copy) -> u8 {
    *STATES.lock().unwrap().entry(call_id()).or_insert_with(default_value)
}

fn main() {
    for _ in 0..3 { 
        amber(32);
        println!("{:-<28}", "-");
        reset_call_id()
    }
}

fn amber(age: u8) {
    increment_call_id();
    let age = use_age(|| { println!("Saving amber's state!"); age });
    println!("amber id: {}, age: {}", call_id(), age);
    mike(15)
}

fn mike(age: u8) {
    increment_call_id();
    let age = use_age(|| { println!("Saving  mike's state!"); age });
    println!("mike id: {}, age: {}", call_id(), age);
    layla_rose(26);
    layla_rose(22)
}

fn layla_rose(age: u8) {
    increment_call_id();
    let age = use_age(|| { println!("Saving layla_rose's state!"); age });
    println!("layla_rose id: {}, age: {}", call_id(), age);
}

The output:

Saving amber's state!
amber id: 1, age: 32
Saving  mike's state!
mike id: 2, age: 15
Saving layla_rose's state!
layla_rose id: 3, age: 26
Saving layla_rose's state!
layla_rose id: 4, age: 22
----------------------------
amber id: 1, age: 32
mike id: 2, age: 15
layla_rose id: 3, age: 26
layla_rose id: 4, age: 22
----------------------------
amber id: 1, age: 32
mike id: 2, age: 15
layla_rose id: 3, age: 26
layla_rose id: 4, age: 22
----------------------------

The main fact: Closures passed to the use_age hook are invoked only once. use_age invokes them only if it doesn't find the age from the previous iteration in STATES.

Another important fact: STATES is a key-value storage, where the key is call_id and the value is u8 (aka age).

So.. do we have nice React Hooks and the world is smiling?

Yeah, until a wild condition appears...

mike(30)
if day == "good_day" {
    layla_rose(26)
} else {
    amber(60)
}

The first iteration with a good_day:

mike(30)   // call id == 1  ;  age 30 saved
if day == "good_day" {
    layla_rose(26)   // call id == 2 ; age 26 saved
} else {
    amber(60)    // not called
}

The next iteration with a bad_day:

mike(30)   // call id == 1  ; age 30 loaded
if day == "good_day" {
    layla_rose(26)   // not called
} else {
    amber(60)    // call id == 2  ; age 26 loaded
}

Output (Rust Playground):

Saving mike's state!
mike id: 1, age: 30
Saving layla_rose's state!
layla_rose id: 2, age: 26
----------------------------
mike id: 1, age: 30
amber id: 2, age: 26
----------------------------
mike id: 1, age: 30
layla_rose id: 2, age: 26
----------------------------

You would be really surprised if a Tinder developer accidentally wrapped a React component in a condition and you plan a date with Amber...

That's why there are official React Rules of Hooks:

Don’t call Hooks inside loops, conditions, or nested functions.

Don’t call Hooks from regular JavaScript functions.

Our call ids based on a counter / indices are just not stable enough.

Fortunately, Rust offers more tools to fight with hooks limitations.

We can get the Location of the caller. It means we know where exactly in the source code has been a function called. So we can distinguish different calls by their caller, even if their index is equal.

We can leverage newer Rust built-in attribute #[track_caller] in combination with Location::caller. The code is starting to be pretty complex (Rust Playground).

use std::sync::atomic::{AtomicUsize, Ordering};
use std::panic::Location;
static COUNTER: AtomicUsize = AtomicUsize::new(0);
#[track_caller]
fn call_id() -> (usize, &'static Location<'static>) { 
    (COUNTER.load(Ordering::SeqCst), Location::caller()) 
}
fn increment_call_id() { COUNTER.fetch_add(1, Ordering::SeqCst); }
fn reset_call_id() { COUNTER.store(0, Ordering::SeqCst) }

use std::{sync::Mutex, collections::HashMap};
use once_cell::sync::Lazy;
static STATES: Lazy<Mutex<HashMap<(usize, &'static Location), u8>>> = Lazy::new(Mutex::default);

#[track_caller]
fn use_age(default_value: impl FnOnce() -> u8 + Copy) -> u8 {
    *STATES.lock().unwrap().entry(call_id()).or_insert_with(default_value)
}

fn main() {
    for i in 0..3 { 
        root(if i % 2 == 0 { "good_day" } else { "bad_day" });
        println!("{:-<28}", "-");
        reset_call_id()
    }
}

fn root(day: &str) {
    mike(30);
    if day == "good_day" {
        layla_rose(26)
    } else {
        amber(60)
    }
}

#[track_caller]
fn mike(age: u8) {
    increment_call_id();
    let age = use_age(|| { println!("Saving mike's state!"); age });
    println!("mike id: {:?}, age: {}", call_id(), age);
}

#[track_caller]
fn amber(age: u8) {
    increment_call_id();
    let age = use_age(|| { println!("Saving amber's state!"); age });
    println!("amber id: {:?}, age: {}", call_id(), age);
}

#[track_caller]
fn layla_rose(age: u8) {
    increment_call_id();
    let age = use_age(|| { println!("Saving layla_rose's state!"); age });
    println!("layla_rose id: {:?}, age: {}", call_id(), age);
}

Updated output (notice Amber's age and Saving amber's state!):

Saving mike's state!
mike id: (1, Location { file: "src/main.rs", line: 29, col: 5 }), age: 30
Saving layla_rose's state!
layla_rose id: (2, Location { file: "src/main.rs", line: 31, col: 9 }), age: 26
----------------------------
mike id: (1, Location { file: "src/main.rs", line: 29, col: 5 }), age: 30
Saving amber's state!
amber id: (2, Location { file: "src/main.rs", line: 33, col: 9 }), age: 60
----------------------------
mike id: (1, Location { file: "src/main.rs", line: 29, col: 5 }), age: 30
layla_rose id: (2, Location { file: "src/main.rs", line: 31, col: 9 }), age: 26
----------------------------

To make the code more robust we'll need to track also ancestors. Otherwise we may have calls with equal index and direct callers but different callers of callers... So we need to create a simple blockchain where each call has a hash of the previous call (yeah, another buzzword for SEO..)

However Nemesis for all Javascript and Rust Hooks are loops. Different calls in loops may have equal both index and location. It means we need another factor to correctly distinguish calls - keys. Unfortunately they need to be provided by developer because they depend on application data.

Many frameworks (with or without Hooks) support keys:

When the developer forgets to define keys, the app may be slower or doesn't work as expected (look at the Svelte demonstration above).

So... from the frontend app developer point of view, Hooks (especially Rust ones) may be a useful tool to reduce boilerplate and introduce local state for function-based components, but the developer has to follow some artificial rules.

Let's move to the technical challenges of Hooks.

Complexity. It's pretty hard to implement Hooks correctly, especially due to many edge-cases and macros. It also mean a lot of code bloat if you are not careful enough.
Good luck with Hooks integration into the framework with asynchronous rendering - you may get lost in the Dark caller forest (just a note from MoonZoon trenches).
We were working only with the age in our examples. Hooks have to support as many types as possible (not only u8). It means we need a heterogenous storage for user data, probably based on Any (if you know Typescript any, Rust Any may open similar gates of hell).
It's pretty hard to start a new iteration in a non-root "node" (when you want to invoke only the function in the call graph representing a component with changed data).
However the Hooks Achilles' heel is the storage performance. When you decide to use the simplest solution - a HashMap with Box<dyn Any> as values:
- Box and Any means a lot of type gymnastics and checks and allocations.
- HashMap's default hash function isn't the fastest one, but the replacement with a faster non-secure one didn't help to increase speed in practice.
- HashMap resizing is pretty slow - it has to move all its items to the new location after the reallocation. griddle helps to eliminate resizing spikes, but it doesn't help too much with the overall speed.

As the result, Zoon's code was a slow spaghetti monster. It was working good enough for cca 2_000 elements, but when there were more complex business logic and more elements then the app becomes too slow for comfortable usage.

I've also tried more mature libraries instead of my code but the performance didn't change too much.

Then I remembered the term Sunk cost fallacy from the awesome book Thinking, Fast and Slow and with the words "Don't love your code, no code no bugs" I selected most Zoon files and hit my favorite key: Delete.

New Zoon architecture

Signals: You can do it without a Virtual DOM

So Hooks was a dead end. The Elm architecture has its own problems (explained in the previous post). I don't want to invent another complex component system with templates. What now?

Let's learn from the past and see what works and what doesn't.

Hooks - Simple creation of local states helps to write element/component libraries and don't pollute our business data with GUI-specific variables.
TEA - Single-source of truth (aka Model) eliminates bugs related to state synchronization.
TEA - Asynchronous "pipelines" may be hard to follow in the source code without an await/async mechanism. Imagine a chain of HTTP requests with error handling and some business logic.
Many frameworks / GUI libraries often try to store and manage all objects representing elements/components by themselves and use the target platform only as a "canvas" where they render elements.
- Why write a custom DOM when we still need to use the browser DOM? The custom DOM then basically becomes a cache. And what are the most difficult things in computer science?
- Why to store and manage objects when we only want to render a HTML string for a Google bot?
Passing properties down to child elements/components may lead to boilerplate (TEA) and then to cumbersome abstractions (many frameworks). TEA-like frameworks try to mitigate it with Pub/Sub mechanisms.
There are often problems with keys for element/component lists (explained in the previous chapter).
Virtual DOM + Asynchronous rendering (the render waits for the next animation frame)
- Adds a lot of complexity and causes bugs.
- The typical bug in most frameworks is a "jumping cursor" in text inputs (Elm issue with demonstation, React explanation).
- Text selection is pretty hard to manage in the browser, especially with async rendering.
Many native browser elements behave quite unpredictably and it's very hard to set them correctly. There has to be a layer above them to protect the app developer.
- "Did you know #456: Setting element attributes is order-sensitive?"

Now I'll show you 4 examples with a new Zoon API and explain how they work. Then we'll discuss how the API corresponds with the notes above.

Example 1

use zoon::*;

#[static_ref]
fn counter() -> &'static Mutable<i32> {
    Mutable::new(0)
}

fn increment() {
    counter().update(|counter| counter + 1)
}

fn decrement() {
    counter().update(|counter| counter - 1)
}

fn root() -> impl Element {
    Column::new()
        .item(Button::new().label("-").on_press(decrement))
        .item(Text::with_signal(counter().signal()))
        .item(Button::new().label("+").on_press(increment))
}

#[wasm_bindgen(start)]
pub fn start() {
    // We want to attach our app to the browser element with id "app".
    // Note: `start_app(None, root);` would attach to `body` but it isn't recommended.
    start_app("app", root);
}

The function counter() is marked by the attribute #[static_ref]. It means the function is transformed by a procedural macro into this:

fn counter() -> &'static Mutable<i32> {
    use once_cell::race::OnceBox;
    static INSTANCE: OnceBox<Mutable<i32>> = OnceBox::new();
    INSTANCE.get_or_init(move || Box::new(Mutable::new(0)))
}

The macro is defined in the crate static_ref_macro in the MoonZoon repo.
The macro currently uses OnceBox. It may use OnceCell or probably lazy_static!.
You can deactivate the macro by a Zoon feature flag static_ref.

Mutable is very similar to RwLock. However it has one unique feature - it sends a signal on change. Let's explain it on the Text element.

There are multiple ways to create a new Text element:

.item(counter().get())
.item(Text::new(counter().get())
.item(Text::with_signal(counter().signal()))

Note: The method .item expects the impl IntoElement parameter. Many Rust basic types (&str, Cow<str>, i32, ..) implement IntoElement by creating a new Text.

The first two lines are practically the same. They just creates a Text element with a static value. It means the text doesn't change at all once set. We can only replace the Text element with a new one if we want to change it.

The third line is more interesting. Text created with the method with_signal rerenders its text when it receives a new value from the chosen signal. Mutable transmits its value to all associated signals when the value has been changed. We can say that Text created by with_signal has a dynamic value.

Example 2

use zoon::*;
use std::rc::Rc;

fn root() -> impl Element {
    let counter = Rc::new(Mutable::new(0));
    let on_press = clone!((counter) move |step: i32| *counter.lock_mut() += step);
    Column::new()
        .item(Button::new().label("-").on_press(clone!((on_press) move || on_press(-1))))
        .item_signal(counter.signal())
        .item(Button::new().label("+").on_press(move || on_press(1)))
}

This example works exactly like the previous one but there are some differences in the code.

counter isn't stored in a static reference / global variable, but created as a local variable.
- Soo... where is it stored?? In the browser DOM! Button::new creates immediately a new DOM node and our counter is passed into its on_press handler. It's possible because the root function is invoked only once to build the app / create the DOM.
counter's Mutable is wrapped in Rc.
- We need to pass the same counter into two on_press handlers. Otherwise Rc wouldn't be necessary.
There is a clone! macro.
- It's just an alias for enc! macro in the enclose crate. I hope Rust will support cloning into closures natively.
- The clone! macro is active when the Zoon's feature flag clone is enabled.
counter().update(|counter| counter - 1) has been replaced with *counter.lock_mut() += step.
- You probably wouldn't find the method update in futures-signals docs - there are traits like MutableExt in the Zoon with such helpers.
- Be careful with lock_* methods. There are cases where it's a bit hard to predict in Rust when the lock is unlocked / dropped (you'll find an example in the next chapter). Also futures-signals crate currently uses std::sync::RwLock under the hood that doesn't output a nice error message to the console (especially in Firefox) so it may be hard to track the problem of trying to lock already locked Mutable. (I was talking about it with the futures-signals's author, it should be less confusing in the future.)

Example 3

...
type ID = usize;
struct Row {
    id: ID,
    label: Mutable<String>,
}

#[static_ref]
fn rows() -> &'static MutableVec<Arc<Row>> {
    MutableVec::new()
}

fn remove_row(id: ID) {
    rows().lock_mut().retain(|row| row.id != id);
}
...

fn table() -> RawEl {
    ...
    RawEl::new("tbody")
        .attr("id", "tbody")
        .children_signal_vec(
            rows().signal_vec_cloned().map(row)
        )
    ...
}

fn row(row: Arc<Row>) -> RawEl {
    let id = row.id;
    ...
    row_remove_button(id),
    ...
}

fn row_remove_button(id: ID) -> RawEl {
    ...
    RawEl::new("a")
        .event_handler(move |_: events::Click| remove_row(id))
    ...
}

The most interesting are these two parts:

// from `table()`
.children_signal_vec(
    rows().signal_vec_cloned().map(row)
)
// from `remove_row(id: Id)`
rows().lock_mut().retain(|row| row.id != id)

RawEl::children_signal_vec updates its child elements according to the input signal. The signal comes from a MutableVec returned from rows(). The most important fact is that this signal transmits only differences between the old and the updated vector. It means it's fast because it doesn't have to clone the entire vector on every change and it can transmit only the child index in the case of removing.

Note: RawEl is a "low-level element". It means RawEl is used as a foundation for other Zoon elements like Row and Button. Only the element Text is based on RawText. Both RawEl and RawText implement Element and From for RawElement. There will be probably also a RawSvgEl in the future. The idea is all raw elements can write directly to the browser DOM or to String as needed.

Example 4

// ----- app.rs -----

// ------ ------
//    Statics 
// ------ ------

#[static_ref]
fn columns() -> &'static MutableVec<()> {
    MutableVec::new_with_values(vec![(); 5])
}

#[static_ref]
fn rows() -> &'static MutableVec<()> {
    MutableVec::new_with_values(vec![(); 5])
}

// ------ ------
//    Signals 
// ------ ------

fn column_count() -> impl Signal<Item = usize> {
    columns().signal_vec().len()
}

fn row_count() -> impl Signal<Item = usize> {
    rows().signal_vec().len()
}

pub fn counter_count() -> impl Signal<Item = usize> {
    map_ref!{
        let column_count = column_count(),
        let row_count = row_count() =>
        column_count * row_count
    }
}

// ----- app/view.rs -----

fn counter_count() -> impl Element {
    El::new()
        .child_signal(super::counter_count().map(|count| format!("Counters: {}", count)))
}

This example demonstrates how to combine multiple signals into one.

For more info about signals, mutables, map_ref and other entities I recommend to read the excellent tutorial in the futures-signals crate.

Note: If you remember the old Zoon API: Statics replace SVars ; Signals replace Caches.

You've seen all examples, let's revisit our notes:

Hooks - Simple creation of local states helps to write element/component libraries and don't pollute our business data with GUI-specific variables.
- let counter = Rc::new(Mutable::new(0)); or an equivalent without Rc seems to be a good way to create a local state.
TEA - Single-source of truth (aka Model) eliminates bugs related to state synchronization.
- static refs or Rust atomics and "update functions" (like increment in our counter example) should be a good alternative to Model + update.
TEA - Asynchronous "pipelines" may be hard to follow in the source code without an await/async mechanism. Imagine a chain of HTTP requests with error handling and some business logic.
- futures-signals is based, well, on futures. You can write (according to the official docs) my_state.map_future(|value| do_some_async_calculation(value));. You can create also Streams and much more.
Many frameworks / GUI libraries often try to store and manage all objects representing elements/components by themselves and use the target platform only as a "canvas" where they render elements.
- Raw elements writes directly to the browser DOM and stores the state inside it. They'll be able to write also to String in the future.
Passing properties down to child elements/components may lead to boilerplate (TEA) and then to cumbersome abstractions (many frameworks). TEA-like frameworks try to mitigate it with Pub/Sub mechanisms.
- You'll be able to combine static refs, signals, standard Rust constructs and maybe Zoon's channels to eliminate the boilerplate.
There are often problems with keys for element/component lists (explained in the previous chapter).
- Do you remember RawEl::children_signal_vec from the Example 3? No keys - no problems.
Virtual DOM + Asynchronous rendering (the render waits for the next animation frame)
- No VDOM, no async rendering - no problems. However I can imagine Zoon will need to support async rendering, but ideally it would be used only when the app developer creates animations.
Many native browser elements behave quite unpredictably and it's very hard to set them correctly. There has to be a layer above them to protect the app developer.
- Two layers should shield the app developer. Standard Zoon elements (Button, Row..) is the first layer and raw elements (RawEl, RawText) is the second one.

Builder pattern with rules

Yes, builder pattern can support required parameters

Button::new().label("X").label("Y")
El::new().child("X").child("Y")

Will the button be labeled "Y" or "XY"?
Will the el's children be rendered in a row or in a column?

error[E0277]: the trait bound `LabelFlagSet: FlagNotSet` is not satisfied
  --> frontend\src\lib.rs:17:30
   |
17 |     Button::new().label("X").label("Y");
   |                              ^^^^^ the trait `FlagNotSet` is not implemented for `LabelFlagSet`

error[E0277]: the trait bound `ChildFlagSet: FlagNotSet` is not satisfied
  --> frontend\src\lib.rs:18:26
   |
18 |     El::new().child("X").child("Y");
   |                          ^^^^^ the trait `FlagNotSet` is not implemented for `ChildFlagSet`

The Rust compiler doesn't allow us to write the code that would break Button or El rules. Only one label and one child makes sense for Button and El.

The compilation also fails when you don't set the label or child at all:

fn root() -> impl Element {
    El::new()
}

error[E0277]: the trait bound `zoon::El<ChildFlagNotSet>: zoon::Element` is not satisfied
  --> frontend\src\lib.rs:16:14
   |
16 | fn root() -> impl Element {
   |              ^^^^^^^^^^^^ the trait `zoon::Element` is not implemented for `zoon::El<ChildFlagNotSet>`
   |
   = help: the following implementations were found:
             <zoon::El<ChildFlagSet> as zoon::Element>

Yeah, we may have constructors like El::new(..) with many parameters instead. But then we also need at least El::with_child_signal(..). And other constructors for more complex elements with more required parameters and their combinations. It becomes cumbersome very quickly.

Note: There are exceptions in the Zoon API like RawEl::new("div") and Text::new("text") because it's not possible to even create a builder for these types without the most important input data.

Why we can't just take the last value as the valid one? E.g. Button::new().label("X").label("Y"); would be a button labeled "Y".

All methods (.label(..), .child(..)) modifies the DOM immediately. It means we would need to delete the previous label and it would be pretty inefficient.
It will be confusing - El can have only one child, but Row accepts multiple children.

Why all methods modifies the DOM immediately?

I've tried to store element builder arguments in the builder and render it at once later. However this approach leads to slow and cumbersome elements and it's almost impossible in some cases.

How those rules work?

Let's look at the current Button implementation.

use zoon::*;
use std::marker::PhantomData;

// ------ ------
//    Element 
// ------ ------

make_flags!(Label, OnPress);

pub struct Button<LabelFlag, OnPressFlag> {
    raw_el: RawEl,
    flags: PhantomData<(LabelFlag, OnPressFlag)>
}

impl Button<LabelFlagNotSet, OnPressFlagNotSet> {
    pub fn new() -> Self {
        Self {
            raw_el: RawEl::new("div")
                .attr("class", "button")
                .attr("role", "button")
                .attr("tabindex", "0"),
            flags: PhantomData,
        }
    }
}

impl<OnPressFlag> Element for Button<LabelFlagSet, OnPressFlag> {
    fn into_raw_element(self) -> RawElement {
        self.raw_el.into()
    }
}

// ------ ------
//  Attributes 
// ------ ------

impl<'a, LabelFlag, OnPressFlag> Button<LabelFlag, OnPressFlag> {
    pub fn label(
        self, 
        label: impl IntoElement<'a> + 'a
    ) -> Button<LabelFlagSet, OnPressFlag>
        where LabelFlag: FlagNotSet
    {
        Button {
            raw_el: self.raw_el.child(label),
            flags: PhantomData
        }
    }

    pub fn label_signal(
        self, 
        label: impl Signal<Item = impl IntoElement<'a>> + Unpin + 'static
    ) -> Button<LabelFlagSet, OnPressFlag> 
        where LabelFlag: FlagNotSet
    {
        Button {
            raw_el: self.raw_el.child_signal(label),
            flags: PhantomData
        }
    }

    pub fn on_press(
        self, 
        on_press: impl FnOnce() + Clone + 'static
    ) -> Button<LabelFlag, OnPressFlagSet> 
        where OnPressFlag: FlagNotSet
    {
        Button {
            raw_el: self.raw_el.event_handler(move |_: events::Click| (on_press.clone())()),
            flags: PhantomData
        }
    }
}

make_flags!(Label, OnPress); generates code like:

struct LabelFlagSet;
struct LabelFlagNotSet;
impl zoon::FlagSet for LabelFlagSet {}
impl zoon::FlagNotSet for LabelFlagNotSet {}

struct OnPressFlagSet;
struct OnPressFlagNotSet;
impl zoon::FlagSet for OnPressFlagSet {}
impl zoon::FlagNotSet for OnPressFlagNotSet {}

The only purpose of flags is to enforce rules by the Rust compiler.

The compiler doesn't allow to call label or label_signal if the label is already set. The same rule applies for on_press handler.

The trade-off for compile-time checked rules are generics. However it isn't a big problem in practice because in view you often return elements from a function by impl Element. And when you really need to box them because you want to use them in a collection or in match / if arms, then you can because Element trait is object safe for these purposes.

Another trade-off could be a bit cryptic error messages but I think they aren't too bad and maybe we'll be able to improve them.

What about API with macros?

Column::new()
    .item(Button::new().label("-").on_press(decrement))
    .item(Text::with_signal(counter().signal()))
    .item(Button::new().label("+").on_press(increment))

col![
    button![button::on_press(decrement), "-"],
    text![counter().signal()],
    button![button::on_press(increment), "+"],
]

Macro API advantages:

Less verbosity / boilerplate in most cases.
Can accept more types than standard functions thanks to "tricks" (e.g. implementing different traits with the same methods for different types) to resolve conflicting impls to achieve a simpler specialization. Note: You can see this trick in action in Seed's UpdateEl* traits that power its element macros.
It can protect from locks-related problems. An example:

fn root() -> impl Element {
    Column::new()
        .item(*counter().lock_ref())
        .item(Text::with_signal(counter().signal()))
}

The lock from lock_ref isn't dropped soon enough so the hidden locking in .signal() fails in runtime.

We can resolve it manually:

// By a  closure
.item((|| *counter().lock_ref())())

// By an extra `let` bindings
.item({ let lock = counter().lock_ref(); *lock })

Notes:

I hope Rust compiler will be clever enough to resolve it in the future by itself and also provide a more descriptive error.
Javascript guys come again with many weird names! This time for a self-invoking closure you've seen in the example above.

Macro API disadvantages:

Less compiler friendly (cryptic errors) and less auto-complete / IDE friendly.
May cause code bloat.
Complicates element implementations.
Didn't pass the "girlfriend test" (A non-developer person with a good graphic taste points the finger to the nicer code when two same examples with different APIs are presented for examination.)
Hard to learn for beginners.
Hard to maintain.

Optimizations

Need for speed. The size matters.

Speed

The most Rust tutorials and best practices are focused on speed. It means you can just follow general recommendations and pick the most used libraries and there is a chance everything will be fast.

The simplest way to increase speed is to set your Cargo.toml correctly. Example:

[profile.release]
# Enable link time optimizations - slow compilation, faster & smaller app
lto = true  

# Disable parallel compilation (set to 1 thread) - slow compilation, faster & smaller app
codegen-units = 1  

# Set optimization level - 3 => fast app ; s/z => small app
opt-level = 3 

#  O4 => fast app ; Oz/Os => small app
# [See the explanation below]
[package.metadata.wasm-pack.profile.release]
wasm-opt = ['-O4']

MoonZoon CLI (mzoon) uses wasm-pack to build your frontend app with Zoon. wasm-pack downloads and manages tools like wasm-bindgen CLI and wasm-opt and browser drivers for testing.

wasm-bindgen CLI and the library do the hard work to connect Javascript with Rust / Wasm. wasm-pack / wasm-bindgen CLI generates a Javascript file to "boot" your app stored in the Wasm file. wasm-bindgen also can generate Typescript types or JS files from your JS snippets defined in the Rust code.
wasm-opt is a tool for Wasm file optimizations. It can improve speed, but it's excellent in size reduction. Note: It's written in C++.

wasm-pack can be configured in Cargo.toml. And it automatically installs also the required compilation target wasm32-unknown-unknown.

When you run mzoon start or mzoon build, MZoon checks if the wasm-pack is installed on your system and then runs

wasm-pack --log-level warn build frontend --target web --no-typescript --dev

to compile your app.

Note: mzoon will be able to install wasm-pack automatically in the future.

Ok, we have an idea how the Wasm Rust app compilation work and we set the most important options in Cargo.toml.

However let's look again at our Cargo.toml. I recommend to disable default-features and search for feature flags in docs and source code in your dependencies.

Example from js-framework-benchmark:

[dependencies]
zoon = { path = "../../../../crates/zoon", features = ["static_ref", "fmt"], default-features = false }
rand = { version = "0.8.3", features = ["small_rng", "getrandom"], default-features = false }
getrandom = { version = "0.2", features = ["js"], default-features = false }

When you enable only the needed features, you can reduce compilation speed.
Many creates offer features that optimize the crate itself and its dependencies for a particular platform (embedded, Wasm) or attribute (speed / size).
You often need to look into the source code because many feature flags and conditions aren't documented or visible on docs.rs. Examples:
- rand needs the flag getrandom and getrandom needs the flag js to not fail in runtime in Wasm.
- parking_lot shows wasm-bindgen flag in its docs.rs docs but the README says: "The wasm32-unknown-unknown target is only supported on nightly and requires -C target-feature=+atomics in RUSTFLAGS"
- wasm-bindgen shows std flag in docs, but it doesn't work without it. On the other hand, you can enable the feature flag enable-interning (will be explained later)

Zoon's current features are:

[features]
default = ["static_ref", "panic_hook", "small_alloc", "clone"]
static_ref = ["static_ref_macro", "once_cell"]
panic_hook = ["console_error_panic_hook"]
small_alloc = ["wee_alloc"]
fast_alloc = []
# tracing_alloc = ["wasm-tracing-allocator"]
clone = ["enclose"]
fmt = ["ufmt", "lexical"]

The features related to performance:

`small_alloc` or `fast_alloc` or `tracing_alloc`

The default allocator for Rust Wasm is currently dlmalloc. You can choose a different allocator (see the list of allocators), but only a couple of them are compatible with Wasm.

I haven't found a Wasm-compatible allocator faster then the default dlmalloc. So when you enable the flag fast_alloc, compilation fails with the message ""Do you know a fast allocator working in Wasm?"".

The flag small_alloc is enabled by default. It means your app will use a bit slower but small wee_alloc.

The flag tracing_alloc would switch to the wasm-tracing-allocator.

"wasm-tracing-allocator enables you to better debug and analyze memory leaks and invalid frees in an environment where we don't have access to the conventional tools like Valgrind."
I tried it, it works but I didn't find it very useful for me. I'll integrate it into Zoon if we find some reasons in the future.

`fmt`

This feature enables dependencies ufmt and lexical. It could replace std::fmt machinery (Debug, format!, println!) however I'll probably focus on it in another MoonZoon dev iteration. You'll see some other fmt-related notes later, in the Size section.

Now we can finally talk about your application code.

There are some recommendation for Wasm + JS:

Wrap &str in intern where it makes sense. It caches strings in JS to mitigate slow string passing through the Rust-JS "bridge" created by wasm-bindgen. Zoon (more accurately dominator) interns automatically many element arguments so you don't need to do it by yourself in most cases.
Be careful with sending strings and more complex items from and to JS world. It may be slow because of encoding and serialization. And it may cause some boilerplate in the app because it's often needed to tell wasm-bindgen how your items should be serialized for export to JS. Note: This problem should be mitigated in the future by richer Wasm API that allows faster Wasm-JS communication.
Use unchecked_* alternatives where it makes sense - see, for instance, wasm_bindgen::JsCast.

And some general recommendations:

Reduce memory allocations as much as possible. It means generics instead of Box, arrays instead of vectors and similar stuff.
Reduce the number of expensive .clone, .to_owned, .to_string, .collect, .into .., calls.
Reduce reallocations. Try to call, for instance, Vec::with_capacity instead of Vec::new / vec![] where possible.
Pick the most suitable algorithms / structures - e.g. HashMap vs IndexMap vs HashMap with a non-secure hash function vs BTreeMap vs SlotMap, etc. It makes sense only when you've prepared benchmarks - results could be quite surprising. Tip: Watch out for println! calls in your benchmarks, console operations could be pretty slow.
There are libraries like smallvec or im or fst which helps A LOT if you know how and where to use them.
Don't use Rc, RefCell, Mutex and similar stuff if you don't have to.
Create errors or default values lazily where it makes sense - e.g. call Result::unwrap_or_else instead of Result::unwrap_or or Option::map_or_else instead of Option::map_or.

Recommendations for all web apps:

Preloading

Moon generates index.html very similar to this one:

<head>
    ...
    <link rel="preload" href="/pkg/frontend_bg_{id}.wasm" as="fetch" type="application/wasm" crossorigin>
    <link rel="modulepreload" href="/pkg/frontend_{id}.js" crossorigin>
    {head_extra} 
</head>

<body>
    ...
    <script type="module">
    import init from '/pkg/frontend_{id}.js';
    init('/pkg/frontend_bg_{id}.wasm');
    </script>
</body>

Notice preload and modulepreload (don't ask me why they are two distinct names, HTML and browser APIs is one big mystery to me).

The browser will be downloading files marked with preload / moduleprelod even if it needs to resolve scripts and styles hidden under the placeholder {head_extra} before it can move to process body.

When the browser starts to import frontend_{id}.js or init frontend_bg_{id}.wasm, it doesn't have to download these files because they have been already preloaded (or they are loading).

Note: There shouldn't be a large time span between preloading and using the files (it may happen when there is a script in {head_extra} hosted on a slow server). Otherwise the browser may show a warning or maybe don't match the files at all.

HTTP/2

There are many reasons why to use HTTP/2 instead of HTTP/1.1. See the basic list of improvements and benchmarks in the article HTTP/2 vs HTTP/1 - Performance Comparison. HTTP/2 is also important for MoonZoon because it allows more SSE connections than HTTP/1.

When you start your MoonZoon app or a MoonZoon example (see Development.md), it runs on HTTP/1.1 by default. You can check it in the browser developer tools, in the tab Network when you add the column Protocol.

To enable HTTP/2 you have to enable HTTPS. Modify the file MoonZoon.toml in your project or in a MoonZoon example:

# port = 8080
port = 8443
https = true
# ...

then start the server (makers mzoon start for examples) and go to https://localhost:8443. Accept a potential security risk caused be a self-signed certificate and you should see HTTP/2 (Firefox) or h2 (Chrome) in the dev tools.

Aaaand how can we measure performance?

Learn to use the browser tools. Chrome dev tools are probably the best - tutorial: Analyze runtime performance.
You can use web_sys::Performance to measure individual functions and your business logic speed. See related MDN docs.
I'm sure you'll be able to find some Rust or Javascript benchmark libraries suitable for Wasm. (Don't hesitate to share your experience.)

Tips:

Keep in mind that debug build is often MUCH slower than the release and optimized one, but it contains debug info needed to show function names and other data by profilers / benchmarks.
There are cases when optimization for size results in higher speed - always test and measure your changes and avoid premature optimization.

Size

There are 2 things that very likely increase the size A LOT - dependencies and macros.

Dependencies

Optimized counter example (makers mzoon build -r):

Without additional deps
- 33 KB (GZip: 16 KB, Brotli: 14 KB, debug: 445 KB)
With one formatting call 1.2.to_string()
- 52 KB (GZip: 24 KB, Brotli: 21 KB, debug: 468 KB)
With the url crate
- 338 KB (GZip: 144 KB, Brotli: 113 KB, debug: 1239 KB)
With the url and regex crate
- 928 KB (GZip: 326 KB, Brotli: 236 KB, debug: 3601 KB)

So if one of your dependency calls format! or .to_string on a float number, expect cca 20 KB larger Wasm file. If you want to use libraries like reqwest, expect more than 300 KB of extra binary size because it uses url crate.

So I recommend to first look at your dependencies and try to find popular, but large libraries like url, regex and serde. Also some parts of std contributes to the code bloat, especially std::fmt.

Try to find alternatives - e.g. ufmt for std::fmt or serde-lite for serde.

Zoon hides ufmt and lexical for float number formatting behind a non-default feature flag. I'll probably add also serde-lite and integrate them properly in the future.

Or you can try to use the browser API instead of Rust libs - e.g. js_sys::RegExp instead of regex or web_sys::Url instead of url.

Macros

Macros are basically code generators. Be prepared for larger binaries if you use them. Don't forget that attributes like #[derive(Debug)] could generate a lot of code.

If you need to write your custom macros, try to extract as much code as possible to new functions.

Panics / errors

The most panic-related code is fortunately removed by wasm-opt, but not all of them. However we can help it:

Call expect_throw and unwrap_throw instead of standard expect and unwrap. See wasm_bindgen::UnwrapThrowExt.

In the Zoon code, there is registered a panic hook:

pub fn start_app ... {
    #[cfg(feature = "panic_hook")]
    #[cfg(debug_assertions)]
    console_error_panic_hook::set_once();

This panic hook is useful for debugging because it shows panic errors in console log. However we don't need it in the release build. wasm-opt can't remove it by itself so we should mark all debug helpers that should be omitted in the release build by #[cfg(debug_assertions)] or a similar compile-time condition.

Note: We need console_error_panic_hook because panics aren't automatically redirected to the console log. There are many std APIs that just do nothing in Wasm. That's why you need to, for instance, add use zoon::{*, println}; if you want to call println in your app.

Allocators

We were already talking about allocators. When you enable Zoon's feature small_alloc (it's enabled by default), then wee_alloc allocator is used.

Related Zoon's code:

#[cfg(feature = "small_alloc")]
#[global_allocator]
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;

Cargo.toml config

It's very similar to optimization for speed:

[profile.release]
lto = true  
codegen-units = 1  
opt-level = 's'

[package.metadata.wasm-pack.profile.release]
wasm-opt = ['-Os']

You need to experiment with values 's' / ['-Os'] and 'z' / ['-Oz']. Sometimes s makes the app smaller than z and even faster then 3. It depends on your app and maybe on the weather. Who knows.

Generics

There is a nice popular word in the Rust world - monomorphization.

An excerpt from the Rust book, section Performance of Code Using Generics:

"You might be wondering whether there is a runtime cost when you’re using generic type parameters. The good news is that Rust implements generics in such a way that your code doesn’t run any slower using generic types than it would with concrete types."

Well, it's a bad news for us. Explained in the docs of a very useful code size profiler for Wasm Twiggy:

"Generic functions with type parameters in Rust and template functions in C++ can lead to code bloat if you aren't careful. Every time you instantiate these generic functions with a concrete set of types, the compiler will monomorphize the function, creating a copy of its body replacing its generic placeholders with the specific operations that apply to the concrete types."

So it basically says we shouldn't use generics if we want to optimize for size because of inlining. However there are two problems:

Generics are often used in your dependencies - out of your control. E.g. Twiggy says that most code bloat because of generics is caused by the crate futures-signals in Zoon's js-framework-benchmark example.
When I was trying to optimize size by replacing generics with other constructs, the app was becoming slower and paradoxically also bigger. So I wouldn't recommend to focus too much on this optimization if you aren't sure it'll really reduce the Wasm file size.

Compression

Browsers support multiple kinds of compression, always at least Gzip or Brotli.

MoonZoon CLI (mzoon) automatically compresses wasm and other files with both algorithms during the release build. And then Moon serves them according to the header Accept-Encoding extracted from incoming requests.

You've already saw examples above, but let's look again:

A small optimized app: 33 KB - GZip: 16 KB - Brotli: 14 KB
A large optimized app: 928 KB - GZip: 326 KB - Brotli: 236 KB

This way you can significantly reduce traffic between frontend and backend.

Note: Firefox and probably other browsers support Brotli only on HTTPS. Chrome supports both Gzip and Brotli also on HTTP. It means you can't use only Brotli for all cases.

Dev Note: It's difficult to serve files according to a header from Warp.

Why are MoonZoon apps optimized for size by default?

Storing small files is cheaper. For you and for hostings / CDNs - it means there is also a higher probability the files will be cached longer on such services.
Sending small files are cheaper. It means you pay less for bandwidth and there will be lower traffic so you'll save money on servers.
Users using a pay-per-use internet connection are happier.
Users with slow internet are happier.
Better SEO thanks to faster page load. (Applies if the bot can run Wasm/JS and prerendering/SSR is disabled.)
Rust / Wasm is fast enough for almost all cases even when optimized for size.
My WiFi signal is weak in the kitchen.

And that's all for today!
Thank You for reading and I hope you are looking forward to the next episode.

Martin

P.S.
We are waiting for you on Discord.

MoonZoon Dev News (2): Live demo, Zoon, Examples, Architectures

Martin Kavík — Thu, 01 Apr 2021 13:21:40 +0000

The Zoon dev marathon is over. And it does something! And it does something on Heroku!

You can try it by yourself: Live demo

Welcome to the MoonZoon Dev News!

MoonZoon is a Rust full-stack framework. If you want to read about new MZ features, architecture and interesting problems & solutions - Dev News is the right place.

News

There are again two big news. I've written the Zoon core (GitHub PR) and created a new GitHub repo with the demo.

However the main goal for this iteration was to write working examples:

You can follow the updated Development.md to run those examples on your local machine.

Other significant changes:

The dev server runs on HTTP by default and the Moon server(s) are configurable through environment variables. See updated Cli.md.
The Zoon API has been changed a bit - see updated Frontend.md.
I've written a super simple Heroku buildpack. Feel free to improve it (e.g. to use cache).

And I would like to thank:

gilescope for MoonZoon improvements in PRs 3 and 4.
sagiegurari for very quick integration of my PRs into his awesome cargo-make. This task runner is very useful for both MoonZoon and Seed.

Frontend framework architectures

The basic purpose of all frontend frameworks is to present some useful data to users and handle user reactions to those presented data.

The Elm architecture

The Elm architecture (TEA) is the most accurate representation of that purpose definition.

The official Elm guide explains TEA this way:

Credits: elm-lang.org

The Elm program produces HTML to show on screen, and then the computer sends back messages of what is going on. "They clicked a button!"

What happens within the Elm program though? It always breaks into three parts:

Model — the state of your application
View — a way to turn your state into HTML
Update — a way to update your state based on messages

These three concepts are the core of The Elm Architecture.

However every architecture shows its trade-offs when apps become larger.

In the case of TEA, you'll soon find out:

Your code consists of 5 big isolated trees - Model, init (to setup Model), Msg, update and view.
Many parts of these trees are used only by one module (aka app part / component).
Many modules are spread across all trees.

A typical example of such module is a page. A contact page and a login page use different data and business logic and render different elements. However both pages probably want to know what user is currently logged in.

Then, app developers cut the trees into pieces and group them together to somehow express the relation between modules and trees.

And that's how components are born in TEA.

However the pieces are grouped mostly logically (by a name or location in the source code), not by a language construct, so developers tend to start looking for a stronger grouping mechanism due to the fear of broken components. In the end, they sacrifice flexibility for cumbersome abstraction. And then they add an extra abstraction layer for shared data (e.g. the logged user in our example).

Credits: Game Layers of Fear, downloaded from WallpaperCave

With those groups (aka components) comes another problem - inter-component communication. There are some patterns for parent-child communication but for other relations there are basically none. Elm has subscriptions to alleviate the problem but you can't create custom subscriptions. Seed has a similar concept subscribe/notify that helps a lot with the app architecture from my experience.

The great advantage of TEA is Model (aka global state). It guarantees a single source of truth for all your data and you can easily inspect the entire app state when needed. However one big chunk of data has also some disadvantages.

There are two data features that are difficult to take into account in TEA:

A) You can't use most "raw" data directly for rendering.
- You often need formatting functions - e.g. you need to round real numbers to show nice values in your e-shop orders.
- There are also much more complex transformations - e.g. you need to interpolate values and compute geometric data to render your charts. (Note: We often escape from TEA and use custom elements and Javascript libraries for such cases.)
B) Not all data are equally important for your business and for all app parts.
- User's mail will be used by many pages and you really don't want to accidentally modify it or use its old value.
- The size of an animated button will be important probably only for the associated button.

In TEA we have only one Model. It means our business data will be mixed with non-business values and precomputed data (or we have to generate temporary data for each rendering). Also relations between "raw" and "derived" data won't be obvious on the first look.

Credits: Photo by Joanna Kosinska on Unsplash

I was focusing mostly on TEA drawbacks in this article, but I think TEA is one of the best architectures, beginner friendly and allows to write very reliable apps with readable code. However there are some trade-offs: It makes writing reusable app parts difficult and it lacks some mechanisms for data management and communication within larger apps. Another problem is difficult optimization of apps based on TEA in many programming languages.

Component-based architecture

Imagine an apple tree. Where tree is a DOM tree. And where apples have their own properties and life-cycles like grow, change color, drop.

Tada! New Javascript framework is ready. Just sprinkle a template engine, CSS-in-JS and a state management on it and we have the version 1.0. A router will be included in the next major version.

Credits: Downloaded from WallpaperCave

However I don't think components are evil. I think many frontend frameworks have 4 problems:

They are too modular. "No, I don't feel like I play with Lego when I write frontend apps. I sew a Frankenstein's monster."
Abstraction at wrong places. "Why we have things like Higher-Order Components but I still have to keep in mind that multi-line input (aka textarea) doesn't have the attribute value and z-index doesn't work with the default position?"
Javascript. It's trendy to write compile-to-JS languages or at least framework-specific compilers and template engines to fight with Javascript (and HTML / CSS).
Everybody wants components but almost nobody really knows how to work with them effectively. There are no strict rules defined like in TEA.

As the result, there are million of frontend frameworks and libraries because they try to heal only some symptoms and be compatible with all other layers.

There are also many "kinds" of web developers. From veterans: "Your website should run with no Javascript!" or "CSS is the best, you just need to learn it for years to become expert like me!" to clever beginners: "Javascript is easy and the best language ever, let's publish some libraries!". It means there are relatively too few people who have time and knowledge to really resolve the problems and define best practices and patterns. Their works are also difficult to find among other ones.

TEA vs Components architecture (CA):

TEA is simple and clearly defined. CA is a pretty general concept, each framework uses different patterns.
TEA advantage is a single source of truth. CA needs a state manager.
TEA requires to write some boilerplate to integrate a new component. Writing component libraries and component usage is easy in CA (its best selling point).

Zoon architecture

Let's pick the best parts from both previous architectures and add HTML and CSS abstraction.

#[s_var]
fn counter() -> SVar<i32> {
    0
}

#[update]
fn increment() {
    counter().update(|counter| counter + 1);
}

#[update]
fn decrement() {
    counter().update(|counter| counter - 1);
}

#[cmp]
fn root() -> Cmp {
    col![
        button![button::on_press(decrement), "-"],
        counter().inner(),
        button![button::on_press(increment), "+"],
    ]
}

The snippet above is very similar to TEA

counter would be Model
increment and decrement would be Msg handlers in update
root would be view

However TEA trees have been naturally split into standalone blocks (#[s_var], #[update], #[cmp], ...). And the root component is recomputed only on counter change.

Instead of divs with explicitly defined CSS properties like float / flexbox / position and aria-* attributes to improve accessibility, we just use col! and button! (they can be also created by Column::new() and Button::new()). It allows us to write more accessible apps faster and also to write reusable element libraries.

Another Zoon example:

    #[s_var]
    fn column_count() -> SVar<i32> {
        5
    }

    #[s_var]
    fn row_count() -> SVar<i32> {
        5
    }

    #[cache]
    fn counter_count() -> Cache<i32> {
        column_count().inner() * row_count().inner()
    }

counter_count is automatically recomputed on column_count or row_count change. It allows us to pre-compute required values outside of the render loop. And developers are able to easily distinguish between "raw" data and "derived" data.

And the last Zoon example:

#[cmp]
fn click_me_button() -> Cmp {
    let title = cmp_var(|| "Click me!".to_owned());
    let click_count = cmp_var(|| 0);
    row![
        button![
            title.inner(),
            button::on_press(move || {
                click_count.update(|count| count + 1);
                title.set(format!("Clicked {}x", click_count.inner()));
            }),
        ],
    ]
}

title and click_count are local variables for the component click_me_button. It means each click_me_button component instance will have own local state and we don't have to mix those variables with our important business data in other parts of the app.

Only the related click_me_button component instance is recomputed and effectively rerendered on click_count or title change.

Let's admire my second professional schema with basic Zoon blocks:

Notes:

Zoon will have also listen/notify mechanism, but I'll write about it once it's implemented.
I'll write more about Zoon implementation when I think the code is refactored enough.

And that's all for today!
Thank You for reading and I hope you are looking forward to the next episode.

Martin

P.S.
We are waiting for you on Discord.

MoonZoon Dev News (1): CLI, Build pipeline, Live-reload, HTTPS

Martin Kavík — Mon, 01 Mar 2021 15:07:25 +0000

It's alive! It runs!

It... doesn't do anything useful yet. Just like an average kitten. We have to wait until they grow up - but who doesn't like to watch progress?

Welcome to the MoonZoon Dev News!

MoonZoon is a Rust full-stack framework. If you want to read about new MZ features, architecture and interesting problems & solutions - Dev News is the right place.

There are two big news. I've written my first tweet ever! And also a couple MoonZoon lines of code - a build pipeline, live-reload, certificate generator and servers (GitHub PR).

Awesome Discord friends tested it on Fedora, Ubuntu, Windows and macOS with Chrome, Firefox and Safari. Live-reload works also on my older iPhone SE. Thanks @adsick, @UberIntuiter and @EvenWei!

Follow these steps to try it by yourself.

How the build process works?

When you run in examples/counter the command

cargo run --manifest-path "../../crates/mzoon/Cargo.toml" start
# or in the future:
mzoon start

then:

MZoon (aka MoonZoon CLI) loads the project's MoonZoon.toml. It contains only configuration for file watchers atm:

[watch]
frontend = [
        "frontend/Cargo.toml",
        "frontend/src",
]
backend = [
        "backend/Cargo.toml",
        "backend/src",
]

MZoon checks if wasm-pack exists and panics if doesn't.
- Note: MZoon will automatically install the required wasm-pack version defined in MoonZoon.toml and check the compatible Rust version in the future.
MZoon generates a certificate for the localhost domain using rcgen. The result is two files - private.pem and public.pem - saved in the backend/private directory.
- Note: Git ignores the private directory content.
- Warning: I recommend to set the certificate's serial number explicitly to a unique value (you can use the current unix time). Otherwise Firefox may fail to load your app with the error code SEC_ERROR_REUSED_ISSUER_AND_SERIAL.
wasm-pack builds the frontend part. If you used the parameter -r / --release together with the command start, then it builds in the release mode and also optimizes the output (.wasm file) for size.
A unique frontend_build_id is generated and saved to the examples/pkg/build_id. The build id is added as a name suffix to some files in the pkg directory. It's a cache busting mechanism because pkg files are served by the backend.
- Note: pkg is generated by wasm-pack and its content is ignored by Git.
The frontend file watcher is set according to the paths in MoonZoon.toml. It sends an empty POST request to Moon (https://127.0.0.1:8443/api/reload) on a file change.
- Warning: Browsers treat unregistered self-signed certificates as invalid so we must allow the acceptance of such certificates before we fire the request:
```
  reqwest::blocking::ClientBuilder::new()
      .danger_accept_invalid_certs(true)
```
cargo run builds and starts the backend. MZoons sets the backend file watcher and saves a generated backend_build_id to backend/private/build_id.
- Note: If you like async spaghetti, you won't be disappointed by looking at the related code. Why?
  - We can't easily split cargo run to standalone "build" and "run" parts. We ideally need something like cargo run --no-build.
  - We need to handle "Ctrl+C signal".
  - We need to somehow find out when the backend has been started or turned off (from the MZoon's point of view).
  - (Don't worry, I'll refactor it later and probably rewrite with an async runtime.)

How the backend works?

The backend part consists two Warp servers.

The HTTPS one runs on the port 8443. It uses generated certificates.
The HTTP one runs on the port 8080 and redirects to the HTTPS one.
- Question: What's the best way of HTTP -> HTTPS redirection? I don't like the current code.

We need HTTPS server because otherwise browsers can't use HTTP/2. And we need HTTP/2 to eliminate SSE limitations. Also it's better to use HTTPS on the local machine to make the dev environment similar to the production one.

Both servers binds to 0.0.0.0 (instead of 127.0.0.1) to make servers accessible outside of your development machine. It means you can connect to your dev servers with your phone on the address like https://192.168.0.1:8443.

I assume some people will need to use custom dev domains, sub-domains, ports, etc. Let me know when it happens.

Tip: I recommend to test server performance through an IP address and not a domain (localhost) because DNS resolving could be slow.

I've chosen Warp because I wanted a simpler server with HTTP/2 support. Also I have a relatively good experience with hyper (Warp's HTTP implementation) from writing a proxy server for my client.

How live-reload works?

When you go to https://127.0.0.1:8443, the frontend route is selected by Warp in the Moon. Moon responds with a generated HTML + Javascript code.

HTML and JS for app initialization aren't very interesting so let's focus on the live-reload code (Note: I know, the code needs refactor, but it should be good enough for explanation):

<script type="text/javascript">
    {reconnecting_event_source}
    var uri = location.protocol + '//' + location.host + '/sse';
    var sse = new ReconnectingEventSource(uri);
    ...
    sse.addEventListener("reload", function(msg) {
        sse.close();
        location.reload();
    });
</script>

What's {reconnecting_event_source}? And why I see ReconnectingEventSource instead of EventSource?

Well, welcome to the world of browsers where nothing works as expected, specs are unreadable and jQuery and polyfills are still needed.

{reconnecting_event_source} is a placeholder for the ReconnectingEventSource code. The library description:

This is a small wrapper library around the JavaScript EventSource API to ensure it maintains a connection to the server. Normally, EventSource will reconnect on its own, however there are some cases where it may not. This library ensures a reconnect always happens.

I've already found such "edge-case" - just run the app in Firefox and restart backend. Firefox permanently closes the connection. Chrome (and I hope other browsers) try to reconnect as expected. Question: Do you know a better solution?

Let's move forward and look at this snippet:

sse.addEventListener("reload", function(msg) {
    sse.close();
    location.reload();
});

It means we listen for messages with the event name reload. Moon creates them in the POST /api/reload endpoint this way:

Event::default().event("reload").data(""))

Warning: Notice the empty string in data(""). It doesn't work without it.

We should call sse.close() if we don't want to see an ugly error message in some console logs when the browser kills the connection on reload.

The last part, hidden under the ... mark in the first code snippet, is:

var backendBuildId = null;
sse.addEventListener("backend_build_id", function(msg) {
    var newBackendBuildId = msg.data;
    if(backendBuildId === null) {
        backendBuildId = newBackendBuildId;
    } else if(backendBuildId !== newBackendBuildId) {
        sse.close();
        location.reload();
    }
});

The only purpose of this code is to reload the frontend when the backend has been changed. The backend sends the message backend_build_id automatically when the client connects to /sse endpoint - i.e. when the SSE connection has been opened.

And that's all for today!
Thank YOU for reading and I hope you look forward to the next episode.

Martin

P.S.
We are waiting for you on Discord.

Open-Source Roads

Martin Kavík — Fri, 19 Feb 2021 20:01:51 +0000

You wake up. Drink your morning coffee. Get in the car to drive to work. But then you realize you hit many potholes on the last trip. And you think: "Today is an ideal day for catching bronze in the sun and breath liquid asphalt!". You run to to the nearest store instead of going to work. To buy a pickaxe and rent an asphalt paving machine.

Your family and friends think you are going crazy.

You are damn good in fixing potholes after some weeks of failing and learning. You are proud of yourself when you are driving to your old work on the fixed road (you still need a day job to afford to buy new asphalt and tools).

"Most people just weren't born for asphalt paving, no matter how hard they try."

Sometimes, wild passersby appear. They grab shovels and help a bit. You need to teach them and often check and fix their work but it's a nice change and you are mostly glad for the help. However, most people just weren't born for asphalt paving, no matter how hard they try. Or they have a very different opinion about paving.

Your road is the best in the city. Everyone loves you. The world is beautiful. A couple of drivers even toss a coin to you! It doesn't help you to buy an excavator but a scoop of ice cream is very welcome after ten-hour shift in the heating sun.

However the most of interactions with the drivers represents shouting requests from the passing vehicles. An example: "I saw a pothole several kilometers back. My company has to pay too much money for new tires! Can you fix it?". And then the same driver on the way home: "Ping @road_worker. What's the progress?".

Does it sound absurd? Welcome to the world of open-source!

How can we ensure quality roads and not create voluntary slavery during the process?

Paid maps. Drivers would be able to use roads without changes, but they would pay road workers for access to maps. Then, the workers would be able to provide better services and they would be much happier and motivated.

The result: Navigation systems would be better and road would be safer and maintained. It means less overall expenses for drivers and companies because they would pay less for car maintenance and operation.

How would it work in practice?

The driver would pay at least a minimum map access fee. Then he would divide the paid fee among chosen road workers.

Where would be maps stored? Who would provide map management tools?

A new map platform. Drivers and workers would be able to use a part of their access fee to support the platform development and expenses.

What about side roads that only a couple of people use?

"There are no main roads without side roads, no rivers without brooks and no projects without dependencies."

The most of money would probably receive the maintainers of the main roads - just because they are the most used and visible. However there are no main roads without side roads, no rivers without brooks and no projects without dependencies. That's why the maintainers would have to divide the half of their income among other workers.

How does that all relate to the open-source development?

Imagine maps replaced by project documentation and tutorials. Imagine a library, tool or framework maintainer instead of the road worker. Imagine company employees instead of drivers. Imagine a new platform. OpenHope.

Images

Truck by Milovan Vudrag
Pothole by Matt Hoffman

Cure for Web Development

Martin Kavík — Fri, 19 Feb 2021 19:51:05 +0000

Imagine you have a small business. You sell flowers in your brick-and-mortar store. Everything goes well, but you would like to attract more customers and provide some extra services - e.g. deliver flowers to a given address.

So you need a website. Just some basic stuff - a home page, flower catalog, order form and maybe a simple chat and blog.

You don't have enough money to hire a professional developer. You can't find a suitable website builder. Fortunately, you know something about computers so you decide to create the website by yourself.

The gates of hell have been opened.

You don't know where to start, but internet knows everything. After a while, you find some ridiculously complicated diagrams of the necessary knowledge.

Then the torment continues, but you find out you need to learn at least these things:

HTML
CSS
Javascript
JSON
Git
DNS
HTTP
REST
WebSockets
SQL
Cookies / JWT

... to have an idea how to choose the right languages, libraries, frameworks and hostings.

You fall into the rabbit hole and then analysis-paralysis kicks in. Should you use:

Javascript or Typescript?
Bootstrap or Tailwind?
React or Vue?
Mongo or Postgre?
Serverless or VPS?
Monolith or Micro-services?
Webpack or Parcel?
Less or Sass?
...

Who knows? Nobody. Flame wars are very popular among web developers.

It works, until you try to do the first major refactor - you'll feel like you are going through a minefield.

When you look at most frontend or backend frameworks and squint your eyes a bit, you'll see just a bunch of helpers for HTML / CSS / JSON manipulation and HTTP wrappers.

But what if HTML and CSS are not suitable for web apps? They were designed for simple web pages a long time ago and basically cannot be improved due to backward compatibility. As the result, the number of CSS and HTML features is growing and almost no one is able to use them properly.

Then combine HTML with CSS and a language designed for writing short scripts - such as Javascript - and write a web app. It works, until you try to do the first major refactor - you'll feel like you are going through a minefield. Anything you touch may explode in the runtime. And no one is brave enough to even enter a CSS minefield and remove old CSS code.

So maybe there are so many frameworks because they do too clever HTML manipulations and want to be as composable and flexible and universal as possible but they fail to achieve the most important goal - to make the web development easier.

I can imagine many developers pray for a small number of users

Another source of fun for an entire day is choosing database and eliminating single points of failure. There are basically two popular approaches:

Use one database (ideally a managed cluster) and one or more stateless apps in containers. Then hope it doesn't become a DevOps nightmare with random database deadlocks.
Use serverless functions and databases and hope you don't have cold starts, don't receive a surprising invoice and don't need real-time communication (e.g. WebSockets or Server-Sent Events).

I can imagine many developers pray for a small number of users so the infrastructure doesn't fall like a house of cards or burn all the money.

There is a cure for this madness:

A statically typed language without footguns like null, undefined or inheritance. Fast and pragmatic. Rust.
A frontend framework with a good API for page elements (HTML and CSS abstraction). It should motivate you to write accessible and SEO content. You shouldn't need to deal with low-level stuff (e.g. with communication protocols).
A backend framework that manages data directly without a database. It can automatically join multiple server nodes to a cluster and use them transparently as one large server. With built-in authentication.
A hosting / cloud with a reasonable and predictable pricing. With monitoring, logging, autoscaling and one-command deployment.

Let me introduce to you the Rust fullstack framework: MoonZoon.

Images:

Code by Markus Spiske
Pills by Kate Hliznitsova

DEV Community: Martin Kavík

MoonZoon Dev News (5): Chat example, MoonZoon Cloud

Chapters

News

Chat example

Shared

Backend

Sessions

Actix

Frontend

Statics

Commands

View

Start

A question from MZ Discord

MoonZoon Cloud

Current MZ Cloud state

Why I'm creating MZ Cloud?

MoonZoon Dev News (4): Actix, Async CLI, Error handling, Wasm-pack installer

Chapters

News

Actix

Why Actix?

Moon API changes

MoonZoon.toml changes

Server-Sent Events

Moon endpoint changes

mzoon

Async runtime

Error handling

File Watchers

File Compressors

File Visitor

Wasm-pack installer

MoonZoon Dev News (3): Signals, React-like Hooks, Optimizations

Chapters

News

Old Zoon architecture

New Zoon architecture

Example 1

Example 2

Example 3

Example 4

Builder pattern with rules

Macro API advantages:

Macro API disadvantages:

Optimizations

Speed

small_alloc or fast_alloc or tracing_alloc

fmt

Preloading

HTTP/2

Size

Dependencies

Macros

Panics / errors

Allocators

Cargo.toml config

Generics

Compression

MoonZoon Dev News (2): Live demo, Zoon, Examples, Architectures

News

Frontend framework architectures

The Elm architecture

Component-based architecture

Zoon architecture

MoonZoon Dev News (1): CLI, Build pipeline, Live-reload, HTTPS

How the build process works?

How the backend works?

How live-reload works?

Open-Source Roads

Cure for Web Development

`small_alloc` or `fast_alloc` or `tracing_alloc`

`fmt`