DEV Community: Wren [Undefined]

The Unstandardized Standardized Implementation Limititations of WebAssembly

Wren [Undefined] — Sun, 19 Nov 2023 03:02:21 +0000

This post will be pretty short. I came across these very arbitrary limits originally in wasmparser, a Rust library for parsing WASM binaries. Turns out they're in most, if not all, of the major JS engines.

wasmparser, used by wasmtime and wasmer:

/* Copyright 2017 Mozilla Foundation
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

// The following limits are imposed by wasmparser on WebAssembly modules.
// The limits are agreed upon with other engines for consistency.
pub const MAX_WASM_TYPES: usize = 1_000_000;
pub const MAX_WASM_FUNCTIONS: usize = 1_000_000;
pub const MAX_WASM_EXPORTS: usize = 100_000;
pub const MAX_WASM_GLOBALS: usize = 1_000_000;
pub const MAX_WASM_ELEMENT_SEGMENTS: usize = 100_000;
pub const MAX_WASM_DATA_SEGMENTS: usize = 100_000;
pub const MAX_WASM_MEMORY32_PAGES: u64 = 65536;
pub const MAX_WASM_MEMORY64_PAGES: u64 = 1 << 48;
pub const MAX_WASM_STRING_SIZE: usize = 100_000;
pub const MAX_WASM_FUNCTION_SIZE: usize = 128 * 1024;
pub const MAX_WASM_FUNCTION_LOCALS: usize = 50000;
pub const MAX_WASM_FUNCTION_PARAMS: usize = 1000;
pub const MAX_WASM_FUNCTION_RETURNS: usize = 1000;
pub const _MAX_WASM_TABLE_SIZE: usize = 10_000_000;
pub const MAX_WASM_TABLE_ENTRIES: usize = 10_000_000;
pub const MAX_WASM_TABLES: usize = 100;
pub const MAX_WASM_MEMORIES: usize = 100;
pub const MAX_WASM_TAGS: usize = 1_000_000;
pub const MAX_WASM_BR_TABLE_SIZE: usize = MAX_WASM_FUNCTION_SIZE;

// Component-related limits
pub const MAX_WASM_MODULE_SIZE: usize = 1024 * 1024 * 1024; //= 1 GiB
pub const MAX_WASM_MODULE_TYPE_DECLS: usize = 1000;
pub const MAX_WASM_COMPONENT_TYPE_DECLS: usize = 1000;
pub const MAX_WASM_INSTANCE_TYPE_DECLS: usize = 1000;
pub const MAX_WASM_RECORD_FIELDS: usize = 1000;
pub const MAX_WASM_VARIANT_CASES: usize = 1000;
pub const MAX_WASM_TUPLE_TYPES: usize = 1000;
pub const MAX_WASM_FLAG_NAMES: usize = 1000;
pub const MAX_WASM_ENUM_CASES: usize = 1000;
pub const MAX_WASM_UNION_TYPES: usize = 1000;
pub const MAX_WASM_INSTANTIATION_EXPORTS: usize = 1000;
pub const MAX_WASM_CANONICAL_OPTIONS: usize = 10;
pub const MAX_WASM_INSTANTIATION_ARGS: usize = 1000;
pub const MAX_WASM_START_ARGS: usize = 1000;
pub const MAX_WASM_TYPE_SIZE: usize = 100_000;
pub const MAX_WASM_MODULES: usize = 1_000;
pub const MAX_WASM_COMPONENTS: usize = 1_000;
pub const MAX_WASM_INSTANCES: usize = 1_000;
pub const MAX_WASM_VALUES: usize = 1_000;

Spidermonkey, used by Firefox and Servo:

// These limits are agreed upon with other engines for consistency.

static const unsigned MaxRecGroups = 1000000;
static const unsigned MaxTypes = 1000000;
static const unsigned MaxSubTypingDepth = 63;
static const unsigned MaxTags = 1000000;
static const unsigned MaxFuncs = 1000000;
static const unsigned MaxTables = 100000;
static const unsigned MaxMemories = 100000;
static const unsigned MaxImports = 100000;
static const unsigned MaxExports = 100000;
static const unsigned MaxGlobals = 1000000;
static const unsigned MaxDataSegments = 100000;
static const unsigned MaxDataSegmentLengthPages = 16384;
static const unsigned MaxElemSegments = 10000000;
static const unsigned MaxElemSegmentLength = 10000000;
static const unsigned MaxTableLimitField = UINT32_MAX;
static const unsigned MaxTableLength = 10000000;
static const unsigned MaxLocals = 50000;
static const unsigned MaxParams = 1000;
static const unsigned MaxResults = 1000;
static const unsigned MaxStructFields = 10000;
static const uint64_t MaxMemory32LimitField = uint64_t(1) << 16;
static const uint64_t MaxMemory64LimitField = uint64_t(1) << 48;
static const unsigned MaxStringBytes = 100000;
static const unsigned MaxModuleBytes = 1024 * 1024 * 1024;
static const unsigned MaxFunctionBytes = 7654321;
static const unsigned MaxArrayNewFixedElements = 10000;

WebKit, used by Safari and all iOS browsers:

constexpr size_t maxTypes = 1000000;
constexpr size_t maxFunctions = 1000000;
constexpr size_t maxImports = 100000;
constexpr size_t maxExports = 100000;
constexpr size_t maxExceptions = 100000;
constexpr size_t maxGlobals = 1000000;
constexpr size_t maxDataSegments = 100000;
constexpr size_t maxStructFieldCount = 10000;
constexpr size_t maxArrayNewFixedArgs = 10000;
constexpr size_t maxRecursionGroupCount = 10000;
constexpr size_t maxSubtypeSupertypeCount = 1;

constexpr size_t maxStringSize = 100000;
constexpr size_t maxModuleSize = 1024 * 1024 * 1024;
constexpr size_t maxFunctionSize = 7654321;
constexpr size_t maxFunctionLocals = 50000;
constexpr size_t maxFunctionParams = 1000;
constexpr size_t maxFunctionReturns = 1000;

constexpr size_t maxTableEntries = 10000000;
constexpr unsigned maxTables = 1000000;

V8, used by NodeJS, Electron, and Chromium derivatives:

// These constants limit the amount of *declared* memory. At runtime, memory can
// only grow up to kV8MaxWasmMemory{32,64}Pages.
constexpr size_t kSpecMaxMemory32Pages = 65'536;  // 4GB
// TODO(clemensb): Adapt once the spec defines a limit here. For now, use 16GB.
constexpr size_t kSpecMaxMemory64Pages = 262'144;  // 16GB

// The following limits are imposed by V8 on WebAssembly modules.
// The limits are agreed upon with other engines for consistency.
constexpr size_t kV8MaxWasmTypes = 1'000'000;
constexpr size_t kV8MaxWasmFunctions = 1'000'000;
constexpr size_t kV8MaxWasmImports = 100'000;
constexpr size_t kV8MaxWasmExports = 100'000;
constexpr size_t kV8MaxWasmGlobals = 1'000'000;
constexpr size_t kV8MaxWasmTags = 1'000'000;
constexpr size_t kV8MaxWasmExceptionTypes = 1'000'000;
constexpr size_t kV8MaxWasmDataSegments = 100'000;
// This indicates the maximum memory size our implementation supports.
// Do not use this limit directly; use {max_mem{32,64}_pages()} instead to take
// the spec'ed limit as well as command line flag into account.
// Also, do not use this limit to validate declared memory, use
// kSpecMaxMemory{32,64}Pages for that.
constexpr size_t kV8MaxWasmMemory32Pages = kSystemPointerSize == 4
                                               ? 32'767   // = 2 GiB - 64Kib
                                               : 65'536;  // = 4 GiB
constexpr size_t kV8MaxWasmMemory64Pages = kSystemPointerSize == 4
                                               ? 32'767    // = 2 GiB - 64Kib
                                               : 262'144;  // = 16 GiB
constexpr size_t kV8MaxWasmStringSize = 100'000;
constexpr size_t kV8MaxWasmModuleSize = 1024 * 1024 * 1024;  // = 1 GiB
constexpr size_t kV8MaxWasmFunctionSize = 7'654'321;
constexpr size_t kV8MaxWasmFunctionLocals = 50'000;
constexpr size_t kV8MaxWasmFunctionParams = 1'000;
constexpr size_t kV8MaxWasmFunctionReturns = 1'000;
constexpr size_t kV8MaxWasmFunctionBrTableSize = 65'520;
// Don't use this limit directly, but use the value of
// v8_flags.wasm_max_table_size.
constexpr size_t kV8MaxWasmTableSize = 10'000'000;
constexpr size_t kV8MaxWasmTableInitEntries = 10'000'000;
constexpr size_t kV8MaxWasmTables = 100'000;
constexpr size_t kV8MaxWasmMemories = 100'000;

// GC proposal.
constexpr size_t kV8MaxWasmStructFields = 10'000;
constexpr uint32_t kV8MaxRttSubtypingDepth = 63;
constexpr size_t kV8MaxWasmArrayNewFixedLength = 10'000;

// Stringref proposal. This limit is not standardized yet.
constexpr size_t kV8MaxWasmStringLiterals = 1'000'000;

#![no_std] with WASI is more complicated than I thought it would be

Wren [Undefined] — Thu, 03 Jun 2021 05:11:01 +0000

This post is meant to document my experience using #![no_std] in Rust with WASI, which was way more complicated than I thought it would. Perhaps this can help to make the documentation better, who knows? :P

So I was wanting to learn more about WASI (WebAssembly System Interface), which is basically like POSIX interfaces for WASI, with capabilities for better security. I decided that perhaps making a "Hello, world!" example in Rust with #[no_std] would be a good idea.

It was... difficult. There were a ton of roadblocks.

So, let's start this.

I started with a basic shell of a #![no_std] program:

#![no_std]
#![no_main]

use core::panic::PanicInfo;

#[panic_handler]
fn panic_handler(_info: &PanicInfo) -> ! {
    loop {}
}

#[no_mangle]
pub extern "C" fn _start() -> ! {
    loop {}
}

In the Cargo.toml I set panic = "abort":

[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"

I use cargo-wasi to build it:

$ cargo wasi build
   Compiling wasi-testing v0.1.0 (/home/james/Code/Rust/wasi-testing)
error: linking with `rust-lld` failed: exit status: 1
  |
  = note: ... really long command line invocation ...
  = note: rust-lld: error: duplicate symbol: _start
          >>> defined in /home/james/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-wasi/lib/self-contained/crt1-command.o
          >>> defined in /home/james/Code/Rust/wasi-testing/target/wasm32-wasi/debug/deps/wasi_testing.5awycwmt43f1nxf9.rcgu.o


error: aborting due to previous error

error: could not compile `wasi-testing`

To learn more, run the command again with --verbose.

Uh... what?

So what happened is that _start is already defined by this mysterious crt1-command.o. I didn't learn until much after that this is from wasi-libc:

#include <wasi/api.h>
#include <stdlib.h>
extern void __wasm_call_ctors(void);
extern int __original_main(void);
extern void __wasm_call_dtors(void);

__attribute__((export_name("_start")))
void _start(void) {
    // Call `__original_main` which will either be the application's zero-argument
    // `__original_main` function or a libc routine which calls `__main_void`.
    // TODO: Call `main` directly once we no longer have to support old compilers.
    int r = __original_main();

    // If main exited successfully, just return, otherwise call `exit`.
    if (r != 0) {
        exit(r);
    }
}

// TODO: Call `main` directly once we no longer have to support old compilers.

This one comment is what caused 99% of my pain with starting the actual program.

At the time, I didn't know about this, so I just tried changing the name of _start to __original_main:

#[no_mangle]
pub extern "C" fn __original_main() -> ! {
    loop {}
}

I build it, and I get this error:

error: linking with `rust-lld` failed: exit status: 1
  |
  = note: ... really long command line invocation ...
  = note: rust-lld: error: function signature mismatch: __original_main
          >>> defined as () -> i32 in /home/james/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-wasi/lib/self-contained/crt1-command.o
          >>> defined as () -> void in /home/james/Code/Rust/wasi-testing/target/wasm32-wasi/debug/deps/wasi_testing.5awycwmt43f1nxf9.rcgu.o

Ok, looks like this function must return an i32. That's fine, I guess. I assumed that this function's return value is the exit code, which I eventually learn is true.

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    0
}

I compile it, and it works.

Now I run it:

Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::exit` has not been defined

Uh... ok then.
At this point I decide that I should probably use WASI to define this exit function, instead of just endlessly looping. So I link in proc_exit. According to the WASI docs, proc_exit takes in an exitcode (a u32) and returns nothing. Perfect!

So I link it in. Notice the deadly mistake? I didn't, yet.

extern "C" {
    fn proc_exit(code: u32);
}

Then I define fn exit() -> !:

#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe { proc_exit(code); }
    loop {} // (hopefully) never reached
}

It builds, but it doesn't work:

Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::proc_exit` has not been defined

Hm.

So I try defining proc_exit as such:

pub mod internal {
    #[no_mangle]
    #[export_name = "proc_exit"]
    pub unsafe extern "C" fn proc_exit(code: u32) {
        super::proc_exit(code);
    }
}

I run it, and... uh what?

Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to invoke command default
    1: wasm trap: call stack exhausted
       wasm backtrace:
           0: 0xffffffff - <unknown>!proc_exit
           1:   0xd8 - <unknown>!proc_exit
           2:   0xd8 - <unknown>!proc_exit
           3:   0xd8 - <unknown>!proc_exit
           4:   0xd8 - <unknown>!proc_exit
           5:   0xd8 - <unknown>!proc_exit
           6:   0xd8 - <unknown>!proc_exit
           7:   0xd8 - <unknown>!proc_exit
           ... a TON of lines cut ...
           32682:   0xd8 - <unknown>!proc_exit
           32683:  0x10b - <unknown>!exit
           32684:  0x122 - <unknown>!_start
         note: run with `WASMTIME_BACKTRACE_DETAILS=1` environment variable to display more information

Uh ok. At this point I realized I was probably making it call into itself endlessly.

At this point, I look into Bytecode Alliance's WASI rust bindings, used by Rust's std.

I look multiple times, and only about the third time do I notice this fatal line:

    #[link(wasm_import_module = "wasi_snapshot_preview1")]
    extern "C" {
        // ... //
    }

I add that to my extern "C" block and remove the weird internal::proc_exit function. Here's the full code at this point:

#![no_std]
#![no_main]

use core::panic::PanicInfo;

#[panic_handler]
fn panic_handler(_info: &PanicInfo) -> ! {
    loop {}
}

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    0
}

#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe {
        proc_exit(code);
    }
    loop {} // (hopefully) never reached
}

#[link(wasm_import_module = "wasi_snapshot_preview1")]
extern "C" {
    fn proc_exit(code: u32);
}

I build and run it, and it works. Since it prints nothing to the screen, I change the 0 in __original_main to a different value that I can observe from the terminal. The specific number doesn't matter, just that it's not zero.

$ wasmtime target/wasm32-wasi/debug/wasi-testing.wasm
$ echo $status # I use fish, with POSIX-compliant shells it's `$!`.
42

And it works! Ok, time to do something a bit more complex. I'm going to work on a print function.

I use fd_write, and since I don't want to look into how interface types work, I just look at the wasi crate again, with some help from the wasmtime WASI WAT tutorial.

I link it in (adding some helper structs, aliases, and enums which I've omitted for brevity, but can be seen in the full code at the end of the post):

// extern "C" block from before
extern "C" {
    pub fn fd_write(
        fd: Fd,
        ciovecs: *const Ciovec,
        n_ciovecs: Size,
        n_written: *mut Size,
    ) -> Errno;
}

I then create a print function:

fn print(s: &str) -> Result<Size, Errno> {
    assert!(
        s.len() <= u32::MAX as usize,
        "please don't store >4GB of text in a string, thanks"
    );
    let ciovec = Ciovec {
        buf: s.as_ptr(),
        buf_len: s.len() as u32,
    };
    let ciovec_ptr = &ciovec as *const Ciovec;
    let mut n_written: Size = 0;
    let errno = unsafe { fd_write(1, ciovec_ptr, 1, &mut n_written as *mut Size) };
    if errno == Errno::Esuccess {
        return Ok(n_written);
    }
    Err(errno)

Here's how this works:

I first assert that the string is not larger than the 32-bit limit, which would be logically incorrect and potentially even unsound to try and print.
I then construct the ciovec (const IO vector) with the buffer (pointer to the string as a *const u8), and the length as a u32.
I then get a pointer to that ciovec on the stack.
I create a variable on the stack, n_written.
I call fd_write with the file descriptor (1 = stdout), the pointer to the ciovecs, the length of the ciovec array (1), as you can print multiple ciovec arrays at once as an optimization for buffered writes, I think, and finally a mutable reference to the n_written variable on the stack.
If the errno returned did not indicate an error (0 = success), then I return Ok(n_written).
Otherwise I return the errno.

I call the function within the __original_main function (changing the return value to 0, again):

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    print("Hello, world!\n").unwrap();
    0
}

At this point, since I'm calling .unwrap(), I'm now realizing I should probably have a better #[panic_handler]. So I change it now:

#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    loop {}
}

I then again realize that format! can't be used in #![no_std] contexts without alloc.

I add this to the top of the file to indicate that it should use alloc:

extern crate alloc;

Then I import format! from alloc:

use alloc::format;

I build it:

error: no global memory allocator found but one is required; link to std or add `#[global_allocator]` to a static item that implements the GlobalAlloc trait.

error: `#[alloc_error_handler]` function required, but not found.

note: Use `#![feature(default_alloc_error_handler)]` for a default error handler.

Ok. Now I need to add in a global allocator. I use wee_alloc since I know it works pretty well with WASM:
Cargo.toml

[dependencies]
wee_alloc = "0.4.5"

main.rs

use wee_alloc::WeeAlloc;

#[global_allocator]
pub static ALLOC: WeeAlloc = WeeAlloc::INIT;

And then I need to create an allocator error handler:

#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    loop {}
}

This means now that I have to add the feature flag for #[alloc_error_handler], meaning that I now have to use the nightly branch:

#![feature(alloc_error_handler)]

I add a panic!() in the main function to test it.

I run it:

Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::memcpy` has not been defined

Um. Ok, I remember this kind of thing. I do some searching and then rediscover compiler_builtins.

I add it to the program:
Cargo.toml

[dependencies]
# ...
compiler_builtins = "0.1.44"

main.rs

extern crate compiler_builtins;

It still doesn't work, however. It still complains that memcpy is not defined. I then realize that I have to enable the mem feature:

[dependencies]
# ...
compiler_builtins = { version = "0.1.44", features = ["mem"] }

And it works! Well, kind of:

Hello, world!
panicked at 'explicit panic', src/main.rs:31:5

It unfortunately, however, infinitely hangs thanks to the infinite loop {}. I decide to use proc_raise to send an abort signal:

extern "C" {
    // ...
    fn proc_raise(sig: Signal) -> Errno;
}

You can see the definition of Signal and Errno in the full code at the end of this post.

I then define a function, abort(), because I'll be using this same logic within the alloc_error_handler so that when allocation fails, it doesn't silently hang.

fn abort() -> ! {
    unsafe { proc_raise(Signal::Abrt); }
    loop {} // should not ever be reached, but required for the `!` return type
}

I then also change the two error/panic handlers to use this now:

#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    abort()
}

#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    abort()
}

And it works! Well, kind of.

Hello, world!
panicked at 'explicit panic', src/main.rs:31:5
Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to invoke command default
    1: proc_raise unsupported
       wasm backtrace:
           0: 0xa692 - <unknown>!wasi_testing::abort::h7ce8efa81892bb51
           1: 0x4b6a - <unknown>!rust_begin_unwind
           2: 0xacf8 - <unknown>!core::panicking::panic_fmt::h3e7a769e079b4878
           3: 0xa3d6 - <unknown>!core::panicking::panic::h0c30bb50c6755362
           4: 0x8431 - <unknown>!__original_main
           5: 0xb136 - <unknown>!_start
       note: run with `WASMTIME_BACKTRACE_DETAILS=1` environment variable to display more information

proc_raise isn't supported by wasmtime yet, but it is part of the WASI API, so I should probably use it as it's the best method at the moment, and may be supported at some point.

And that is the hellish journey that I took to get WASI working with #![no_std] in Rust.

Here is the full code at this point:

#![feature(alloc_error_handler)]
#![allow(dead_code)]
#![no_std]
#![no_main]

extern crate alloc;
extern crate compiler_builtins;

use alloc::format;
use core::panic::PanicInfo;
use wee_alloc::WeeAlloc;

#[global_allocator]
pub static ALLOC: WeeAlloc = WeeAlloc::INIT;

#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    abort()
}

#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    abort()
}

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    print("Hello, world!\n").unwrap();
    0
}

#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe {
        proc_exit(code);
    }
    loop {} // (hopefully) never reached
}

fn abort() -> ! {
    unsafe {
        proc_raise(Signal::Abrt);
    }
    loop {} // should not ever be reached, but required for the `!` return type
}

#[repr(u16)]
#[derive(Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub enum Errno {
    Esuccess,
    E2big,
    Eacces,
    Eaddrinuse,
    Eaddrnotavail,
    Eafnosupport,
    Eagain,
    Ealready,
    Ebadf,
    Ebadmsg,
    Ebusy,
    Ecanceled,
    Echild,
    Econnaborted,
    Econnrefused,
    Econnreset,
    Edeadlk,
    Edestaddrreq,
    Edom,
    Edquot,
    Eexist,
    Efault,
    Efbig,
    Ehostunreach,
    Eidrm,
    Eilseq,
    Einprogress,
    Eintr,
    Einval,
    Eio,
    Eisconn,
    Eisdir,
    Eloop,
    Emfile,
    Emlink,
    Emsgsize,
    Emultihop,
    Enametoolong,
    Enetdown,
    Enetreset,
    Enetunreach,
    Enfile,
    Enobufs,
    Enodev,
    Enoent,
    Enoexec,
    Enolck,
    Enolink,
    Enomem,
    Enomsg,
    Enoprotoopt,
    Enospc,
    Enosys,
    Enotconn,
    Enotdir,
    Enotempty,
    Enotrecoverable,
    Enotsock,
    Enotsup,
    Enotty,
    Enxio,
    Eoverflow,
    Eownerdead,
    Eperm,
    Epipe,
    Eproto,
    Eprotonosupport,
    Eprototype,
    Erange,
    Erofs,
    Espipe,
    Esrch,
    Estale,
    Etimedout,
    Etxtbsy,
    Exdev,
    Enotcapable,
}

pub type Size = u32;
pub type Fd = u32;
pub type ExitCode = u32;

#[repr(C)]
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct Ciovec {
    pub buf: *const u8,
    pub buf_len: Size,
}

#[repr(u8)]
pub enum Signal {
    None,
    Hup,
    Int,
    Quit,
    Ill,
    Trap,
    Abrt,
    Bus,
    Fpe,
    Kill,
    Usr1,
    Segv,
    Usr2,
    Pipe,
    Alrm,
    Term,
    Chld,
    Cont,
    Stop,
    Tstp,
    Ttin,
    Ttou,
    Urg,
    Xcpu,
    Xfsz,
    Vtalrm,
    Prof,
    Winch,
    Poll,
    Pwr,
    Sys,
}

#[link(wasm_import_module = "wasi_snapshot_preview1")]
extern "C" {
    pub fn fd_write(fd: Fd, ciovecs: *const Ciovec, n_ciovecs: Size, n_written: *mut Size)
        -> Errno;

    pub fn proc_raise(sig: Signal) -> Errno;

    pub fn proc_exit(rval: ExitCode);
}

fn print(s: &str) -> Result<Size, Errno> {
    assert!(
        s.len() <= u32::MAX as usize,
        "please don't store >4GB of text in a string, thanks"
    );
    let ciovec = Ciovec {
        buf: s.as_ptr(),
        buf_len: s.len() as u32,
    };
    let ciovec_ptr = &ciovec as *const Ciovec;
    let mut n_written: Size = 0;
    let errno = unsafe { fd_write(1, ciovec_ptr, 1, &mut n_written as *mut Size) };
    if errno == Errno::Esuccess {
        return Ok(n_written);
    }
    Err(errno)
}

And the Cargo.toml:

[package]
name = "wasi-testing"
version = "0.1.0"
authors = ["ThePuzzlemaker <tpzker@thepuzzlemaker.info>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
wee_alloc = "0.4.5"
compiler_builtins = { version = "=0.1.43", features = ["mem"] }

[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"

The most underrated but useful Rust standard library type

Wren [Undefined] — Tue, 04 May 2021 05:07:01 +0000

The Rust standard library is full of many useful types, traits, and abstractions. Today I'll be talking about one that I think is underrated, but is quite useful. That type is Cow.

What is `Cow`?

According to the standard library docs, std::borrow::Cow is:

a smart pointer providing clone-on-write functionality: it can enclose and provide immutable access to borrowed data, and clone the data lazily when mutation or ownership is required. The type is designed to work with general borrowed data via the Borrow trait.

I feel like this description is accurate, but it just doesn't give it justice.

Cow is actually just an enum:

pub enum Cow<'a, B: ?Sized + 'a>
where
    B: ToOwned,
{
    /// Borrowed data.
    Borrowed(&'a B),

    /// Owned data.
    Owned(<B as ToOwned>::Owned),
}

This definition not only makes it possible to have clone-on-write semantics (where if data is borrowed, it is turned into owned data before being accessible mutably), but it also allows you to store data that is potentially owned. The abstraction with ToOwned allows you to use Cow<'_, str>, storing either a &str or a String. Without this abstraction, Cow wouldn't be as helpful, because you would only be able to hold e.g. a str and a Box<str> (which is definitely not as useful as String).

You could of course just write your own definition of a PotentiallyOwned<'a, T> type, but if it already exists in the standard library, you might as well use it! Plus, the clone-on-write semantics are a helpful addition to a "potentially owned" type.

Why is it helpful?

The "potentially owned" semantics of Cow make it helpful for some cases where you may have a large string that you need to modify, but don't want to clone unnecessarily. This is an example that's not brought up in the standard library docs (I argue that it should definitely be).

Here is a simple (albeit, somewhat contrived) example of how this is helpful:

fn foo(s: &str, some_condition: bool) -> &str {
    if some_condition {
        &s.replace("foo", "bar")
    } else {
        s
    }
}

Let me explain this code. This function may replace all instances of "foo" with "bar" in the string s, if the condition some_condition is true. But there's a problem!

The function returns a &str, but str.replace returns a String, and you can't return a reference to data owned by a function!

Okay, so let's make the function return String.

fn foo(s: &str, some_condition: bool) -> String {
    if some_condition {
        s.replace("foo", "bar")
    } else {
        s.to_string()
    }
}

This function works, but notice that it always clones the string. This is generally bad, because if the string is long, this might be super expensive! So what do we do if we don't want to unnecessarily clone?

Well, we can use Cow! It allows us to return data that is potentially owned.

Here is a version of the function that doesn't unnecessarily clone:

use std::borrow::Cow;

fn foo<'a>(s: &'a str, some_condition: bool) -> Cow<'a, str> {
    if some_condition {
        Cow::from(s.replace("foo", "bar"))
    } else {
        Cow::from(s)
    }
}

And it works! Our code passes the borrow checker, and it doesn't waste memory!

Why is it confusing?

Note: This section is mostly just my speculations so don't take it as fact.

I personally feel that Cow is a lot more confusing than it needs to be, thanks to the borrow checker. Don't get me wrong, the borrow checker is amazing. It definitely prevents memory safety bugs that would otherwise be present without it. But lifetimes are quite a papercut for new Rust users, and I think this is one reason why Cow is kind of underrated--it's kind of hard to use.

Anything producing or using a Cow needs to have a lifetime attached, and that makes it just a bit harder to use. It's also potentially hard for new users to reason about unsized types, so using a Cow<'_, str> might not make sense to a new user, as they may think something on the lines of "but wait, I thought you would only use &str, not just str?"

These aren't really problems that can be solved with some "quick fix", so this will continue to be a potential papercut.

Another reason why I find that Cow is confusing is because the naming and documentation clouds the potential usages of it for potentially owned data. A new user wondering "can I have borrowed data returned by this part of a function and owned data by this part" will probably not immediately think "oh, I need a clone-on-write smart pointer" (and they might end up trying to implement their owned "partially owned" data type, and may reach some papercuts with lifetimes or lack of ToOwned). I don't think that there's really a way to fix the naming part, without a lot of backwards-incompatibility, but there is definitely a way to fix the documentation.

Conclusion

Cow is a very useful type with somewhat hidden capabilities. I argue that these capabilities should be more documented in Rust documentation and books. As far as I know, there's no mention of Cow in the Rust Book or Rust By Example.

Thanks for reading this, and have a good morning/afternoon/evening/night/etc.
-Wren

If you liked this article, you may like some of my writings on my journey through writing Calypso, a programming language that I'm implementing in Rust:

Introducing The Calypso Chronicles

Wren [Undefined] ・ Mar 2 '21

#calypso #rust #devjournal #langdev

(I will be writing another post in that series soon(tm))

Tips for Productivity that I've Learned from Calypso

Wren [Undefined] — Wed, 24 Mar 2021 05:35:20 +0000

In this post, I'd like to share some tips for productivity that I've already learned from working on Calypso.

I'll start with some basic tips:

Try to keep your code maintainable. It's generally better to overengineer something maintainable than to underengineer something that's impossible to change once you find a bug.
Don't overwhelm yourself. Start small, don't try to finish everything at once.

Now, here's the largest lesson I've learned from working on Calypso:

Don't get bogged down in the details. This is a HUGE issue for me. Basically, if you have a project with multiple interconnected parts, don't start by trying to finish one of those parts and then working on another part. Start small, and work on them individually. Get the whole system working with a subset of your expected input, before expanding to a full, working system.

When starting Calypso, I started with the lexer (as this is usually the first part you'd start with in a programming language). But I got bogged down in the details. I kept trying to finish the lexer before finishing the parser, and other parts of the implementation of the programming language.

This led to what I've had explained to me as "productive procrastination". Basically, I was so bogged down in the details that I felt that I had to get the lexer "perfect" before I could work on the next part in the implementation. So I started to work on unrelated parts that didn't require the lexer at all (or at least didn't require it to be "perfect"), and I did get some work done, but ultimately at the moment it hasn't been very helpful.

And I wasn't able to get the lexer perfect without finishing the language design. And then that got me bogged down in writing documentation (which I guess is ok, but...), and then that got me stuck writing syntax highlighting, which then led to writing a EBNF grammar for the language, which then got me back to language design. It was unchecked recursion and it went from productive procrastination to unproductive procrastination.

Recently, I decided to just stop. I commented out most of the code paths for the lexer, and just worked on the part that I hadn't worked on yet because I had failed to implement it a few times before and I was scared that I wasn't going to get it to work. But I did. I found a few bugs later, and that was fine. I got those sorted, out, then I started on a basic parser. Not the whole language, just simple arithmetic. Then I moved on to boolean operators and bitwise operators. Then I kept building and building until now I have a basic parser for expressions (barring things like lists and maps).

Sometimes you have to just stop yourself. You sometimes just need to say to yourself (possibly in a comment), "This code is purposefully incomplete. I'll come back to it later, once I'm done with the other parts and I have a better perspective on the design of the system overall".

Comment out code. Erase it, rewrite it if you need to! Then work on other parts in parallel. The only way to truly have a better perspective of the design of the overall system is not to design the whole thing in advance, it's to get stuck. You've probably heard the mantra, "If at first you do not succeed, try, try again." This is sometimes true, and this is a case. In fact, I don't want you to succeed at first. You shouldn't want that. 99% of the time, I've found that if something seems to work on the first try, it probably doesn't. And through this failure and suspicion, you can learn new things. You can find new perspectives and new ways to design things better and more correctly.

It's fine to fail. It's fine for something to not be perfect. And that's the most important thing to keep productivity. If you always succeed, there is no triumph in the end. Failure is a natural part of engineering software, and it's better that you have failure while you're developing your software, than while you're using your software in production and it's interconnected with a million other things that will break on one error, caused in code that you thought would work because it seemed to work on the first try. Failure is important, and while it's not always fun, it can help.

If you need to, take a break if you experience failure that you're not sure how to fix. Whether you're just pivoting from coding to another real-life activity, or just pivoting from one spot of the code to another, breaks are important. They can help you to get a new perspective on issues that you didn't have a good perspective on before.

Thanks for reading this. I hope this helps someone out there. Have a good morning/day/afternoon/evening/night/etc!
-James

A Bunch of Odysseus Pseudocode I Guess

Wren [Undefined] — Wed, 17 Mar 2021 20:22:52 +0000

Hey there! Sorry for the small hiatus.

Calypso

Notes and Thoughts

I've mostly finished the redesign of the syntax. I'm working on writing a EBNF grammar to both (semi-)specify the language and to make it easier to write the parser once it comes to that part.
I've also got a bit of syntax highlighting working. There's a VSCode extension, but be aware it's pretty buggy and I'm going to rewrite it anyway once I'm done writing the EBNF grammar.

TODOs

Finish EBNF grammar
Redo syntax highlighting
Finish lexer rewrite

Odysseus

Notes and Thoughts

I've been working a little bit on the design of Odysseus. It's probably going to be somewhat inspired by the BEAM (cause it's pretty good, to be honest, and I have no dang clue how to write a VM). Here's an example of the code so far:

// Factorial (recursive) and hello world
// PSEUDOCODE -- Subject to change

/*
fn factorial(0);
fn factorial(1) ->
    1
end

fn factorial(n) where n > 1 ->
    n * factorial(n - 1)
end

fn hello() ->
    println("Hello, world!")
end
*/

fn factorial/1 ->
label @0:
    register %n, const fnarg:0
    select %n, @2, [(uint(1), @1), (uint(2), @1)]
label @1:
    ret uint(1)
label @2:
    test_guard is_gt, [%n, uint(1)]
    $0 = call_ext_dispatch 1, &:"gaia.ops.Sub.sub", %n, [uint(1)]
    $1 = call 1, &:"factorial", @0, [$0]
    $2 = call_ext_dispatch 1, &:"gaia.ops.Mul.mul", %n, [$1]
    ret $2
end

data ->
    0: string("Hello, world!")
end

fn hello/0 ->
label @0:
    call_ext_tail 1, &:"atlas.io.println", [data:0]
end

Let's go through this (roughly) line-by-line. Ignore the comments at the start, those are just info about the file and the source code. Let's start with hello/0. This is the code that corresponds to it:

data ->
    0: string("Hello, world!")
end

fn hello/0 ->
label @0:
    call_ext_tail 1, &:"atlas.io.println", [data:0]
end

The first section there is the data section. It defines a single entry in the data section of the binary, data:0, which is a string, "Hello, world!".

Then, the function.

fn hello/0 -> defines the function hello/0 (hello with arity 0, i.e. 0 arguments).

Then, we define label @0, the first label to be executed when calling the function.

Inside that label, we have one instruction:

call_ext_tail 1, &:"atlas.io.println", [data:0]

call_ext_tail means to do a tail call of an external function. Tail calls are helpful because they can allow tail call optimization, which is where a call instruction (which requires a stack frame) can be turned into a simple jump.

The next argument to that instruction, 1, is the arity of the function. Then we have the name of the function, atlas.io.println. The syntax &:"atlas.io.println" is just a function reference to the name of the function as an atom. Note that this is just pseudocode, so this syntax will probably look different in the future.

Then, we have our last argument, the list of arguments to the function: [data:0]. data:0 is simply a reference to the 0th entry in the data section of the binary. Thus, data:0 is a reference to the "Hello, world!" string. Thus, this simply calls println("Hello, world!").

Next, let's go through the factorial function. This one is a bit more complex.

fn factorial/1 ->
label @0:
    register %n, const fnarg:0
    select %n, @2, [(uint(1), @1), (uint(2), @1)]
label @1:
    ret uint(1)
label @2:
    test_guard is_gt, [%n, uint(1)]
    $0 = call_ext_dispatch 1, &:"gaia.ops.Sub.sub", %n, [uint(1)]
    $1 = call 1, &:"factorial", @0, [$0]
    $2 = call_ext_dispatch 1, &:"gaia.ops.Mul.mul", %n, [$1]
    ret $2
end

First, we define the function:

fn factorial/1 ->

Then, we define the label @0. Inside it, we initialize the register %n as a register corresponding to the first function argument (which is a constant). Then, we do a pattern match over %n:

select %n, @2, [(uint(1), @1), (uint(2), @1)]

select is an instruction that matches over the register provided and jumps to the labels provided, or a label for failure to match.

In this case, it matches over %n. If it's uint(1) (the unsigned integer 1), it jumps to @1. If it's uint(2), it also jumps to @1. Otherwise, in failure, it jumps to @2.

The label @1 just contains one instruction: ret uint(1). This simply returns the unsigned integer 1.

The label @2 is a bit more complicated. This is the case where we run n * factorial(n - 1).

label @2:
    test_guard is_gt, [%n, uint(1)]
    $0 = call_ext_dispatch 1, &:"gaia.ops.Sub.sub", %n, [uint(1)]
    $1 = call 1, &:"factorial", @0, [$0]
    $2 = call_ext_dispatch 1, &:"gaia.ops.Mul.mul", %n, [$1]
    ret $2

We first test the function guard, which is n > 1. To do this, we test is_gt (is greater than), with the arguments %n and uint(1). In the case of failure, it will cause an exception as it is the only remaining guard.

Then, we set the temporary register $0 (prefixed by $ rather than %) to the value of call_ext_dispatch 1, &:"gaia.ops.Sub.sub", %n, [uint(1)]

This function dispatches an external trait function. The first argument is the arity of the function, excluding the self argument. The second argument is the trait function, in this case gaia.ops.Sub.sub. The third argument is the self object, in this case, %n. The last argument is the list of arguments, in this case simply [uint(1)].

This is equivalent to n - 1, just with operator overloading via traits.

Then, we set the temporary register $1 to the value of call 1, &:"factorial", @0, [$0]. call is a non-tail call in the local module. The first argument is, again, the arity. The second is the function name, in this case factorial. The third argument is the label of that function, which can help to enable optimizations. In this case, we just use the entry label (@0). The final argument is the list of arguments, in this case, just $0 which was set to n - 1.

We then do a similar thing to the subtraction for the multiplication, just switching out the registers and trait function. Then, finally, we return $2, which is now equal to n * factorial(n - 1).

I hope this made some semblance of sense, but if it doesn't, feel free to comment or talk to me on Discord. Note that this is all pseudocode and is subject to change.

TODOs

Solidify ABI and ISA
Pretty much everything

Calypso Build Tool

I need a name for the Calypso build tool (kinda like cargo for Rust or mix for Elixir), but I'm bad at naming things. If you have any ideas, leave a comment or DM me on Discord!

Conclusion

Basically, it's going well! :)

If you find anything that I might've missed or you'd like more information on, feel free to comment or otherwise contact me!

Otherwise, have a good morning/afternoon/evening/night/etc.!
-James

What is Calypso?

Wren [Undefined] — Fri, 05 Mar 2021 03:53:35 +0000

This post was partially adapted from The Calypso Manifesto, which I wrote on my "original" blog a bit over a week ago.

I saw that quite a few people read my last post, which you can see here:

Introducing The Calypso Chronicles

Wren [Undefined] ・ Mar 2 '21

#calypso #rust #devjournal #langdev

I thought that it might be helpful to some people if I introduce them to Calypso, as I'm sure that 99.9% of the people that read it (or at least looked at it enough for dev.to to count it as a view) probably did not know what Calypso is.

Calypso is a mostly imperative language with some functional influences that is focused on flexibility and simplicity.

This is the tagline for Calypso on its GitHub page, which you can see here:

calypso-lang / calypso

Calypso is a mostly imperative language with some functional influences that is focused on flexibility and simplicity.

Calypso

Calypso is a mostly imperative language with some functional influences that is focused on flexibility and simplicity.

Note that this code is very work-in-progress. Contributions are welcome (and encouraged!), but it's not recommended to use this in production unless you're ready for some serious pain. Or code that just doesn't work.

Example

The following example is an implementation of FizzBuzz that goes until a number specified in the CLI arguments of the program, or 100 if that is not present. Note that this is currently psuedocode and may change.

import standard.process.Args
fn main(args: Args) ->
    args
    |> _.get(0)
    |> _.unwrap_or("100")
    |> uint.from_string
    |> _.unwrap_or(100)
    |> fizzbuzz
    |> _.each(&println("{}", &

…

View on GitHub

But what does this tagline mean, exactly? In this post, I'm going to describe why I created Calypso, what it is, what it will hopefully become, and what it will not become.

So, what is Calypso?

Calypso is a programming language that uses mostly imperative programming features, but also has some influences from functional programming. It was inspired by Elixir and Rust. I'm writing the compiler and virtual machine in Rust as I'm fairly comfortable with it, and it's arguably a pretty good language to do so.

In terms of the overall architecture, Calypso is mostly inspired by Rust. However, a lot of the functional aspects are inspired by Elixir or Erlang.

A lot of times, Elixir and Erlang are described as functional languages that break free to the imperative world at points (for example, with side effects). I would describe Calypso as the opposite an imperative language that breaks free to the functional world at points. While these seem very similar in nature, they're pretty different.

I'm aiming for Calypso to be familiar but introduce some potentially unfamiliar concepts from the functional world that can help to keep code short, simple, flexible, and maintainable.

What does the tagline mean?

Flexibility and simplicity are two of Calypso's guiding principles. Here's what exactly they are:

Flexibility. This means that you shouldn't have to "hack" Calypso to do more complex things.
Simplicity. This means that code is expressive but is not too verbose to the point where it's hard to understand.
Performance. This means that code shouldn't have to be micro-optimized (or otherwise manually optimized) to run well, without sacrificing flexibility or simplicity. As I'm not that experienced with optimization, this may take a while to implement effectively, but you're obviously welcome to contribute! :)

These guiding principles should not necessarily be mutually exclusive, which is Calypso's goal. Thus, you shouldn't have to sacrifice one of them to have the other, so you can have code that is not only simple but is also flexible and performant.

Also, they should also apply to the standard library/ies of Calypso, which should be expressive, flexible, and performant.

So why did I create Calypso?

In case you didn't already know, I love Rust and Elixir. However, there are just some things that Elixir isn't suited for. And this isn't exclusive to Elixir--there are also some things that Rust isn't the best for.

Sometimes, Rust seems kind of overkill or seems just too slow of a development process due to its incredibly static nature. While this makes Rust easy to work with, in the long run, it makes short development cycles and compilation times long. This draws me towards more dynamic languages like Elixir.

However, as Elixir is inherently tied to the BEAM and Erlang/OTP, it means that it has features like immutable variables (though you can rebind them and have state via process loops, e.g. Agents or GenServers). This also means that OTP is the main "application framework" for Elixir/Erlang. In my opinion, OTP is a pretty good design philosophy for applications, but sometimes it's just kind of hard to set up or wrap your head around. It can cause you to think about a lot of things when designing even simple applications, which means that it's sometimes much harder to develop applications incrementally.

I have also found that already available scripting-like languages such as JavaScript and Python are not really that easy to work with when scaling up; for example, JavaScript, with its many design flaws (which I'm not going to explain in this post because it's not a post about JS), and Python. While both work somewhat well when you start and when scaling up to an extent, they just get hard to maintain.

And these reasons are some of the reasons why I created Calypso. I enjoy Rust and Elixir, but for some applications, including scripts that are not quite full applications but not quite worthy of being just a Bash script, they're just too much. Also, sometimes I have applications that are not really the best suited for either and are better suited for a more generalized dynamic environment.

What would I like Calypso to become?

Personally, I'd like to see Calypso become a language that could be used for applications, simple and complex, offline and offline, command-line and graphical.

I have no way of knowing if this will happen or not, but I have a goal to make Calypso approachable enough that people will be able to bring it to this point.

What am I not expecting Calypso to become?

I'm not expecting Calypso to replace Rust, Elixir, JavaScript, Python, or really, well, anything. It's not created with that in mind--it definitely isn't a systems programming language and probably won't have as good concurrency as Elixir, being on the Erlang stack. (Though I do hope that I can make concurrency work pretty well, though I haven't really thought that far ahead yet.)

I'm hoping Calypso can be used to make applications that could otherwise be made in Rust or Elixir, but I'm not expecting it to singlehandedly replace the entirety of those languages. That's just not a goal for it.

Calypso will probably not be as fast as Rust or Elixir either. It's intended to be simple, so it will definitely compile faster than Rust, but I'm not expecting it to particularly rival the performance of either language. It may be about as fast as Elixir at some point, but as a non-systems language, it's definitely not going to rival Rust's performance without JIT or other heavy optimizations (including native code, both of which I don't currently have plans for).

What are the parts of the Calypso ecosystem?

There are a few main parts of the Calypso ecosystem. In summary, these are:

The language itself
The compiler and tooling
The standard library/ies, Atlas and Gaia
The bytecode virtual machine, Odysseus
Officially developed and maintained libraries

These parts should ideally work together to make ensuring Calypso's guarantees are enforced is easy and approachable.

You might not be familiar with some of these aspects, so I'll explain them briefly.

Calypso Compiler and Tooling

The Calypso compiler and tooling (including the calypso binary) are used to integrate development and testing of Calypso projects.

The standard library/ies

There are technically two standard libraries for Calypso: Atlas and Gaia.

Atlas is the kind of standard library that you'd be familiar with from a language like Rust or Elixir.

Gaia is more complicated. It's basically the minimum required to integrate Calypso and Odysseus. It will include things like FFI types, primitive types, language items, etc.

If you're wondering why they're named this way, it's because I'm bad at naming things. Congratulations me!

The bytecode virtual machine

Odysseus is a bytecode virtual machine. Basically, it's a system in which an abstract representation of a program that's not quite machine code but not quite source code or an AST is run on a "virtual machine" (hence the name) that is compatible with pretty much any platform the language it's programmed in supports (so, in this case, any platform Rust supports). I'm not able to test multiple platforms at this point, so please contribute or report issues if you have issues on another platform!

Officially developed and maintained libraries

In order to further the Calypso ecosystem, but not bloat the standard library, I will most likely develop some libraries that are helpful for things such as cryptography, HTTP, etc.

One library that I may eventually make which will probably not be an officially maintained library but which I will maintain myself will be a partial or full port of pwntools (a library written in Python for CTFs) to Calypso.
For more information on CTFs, atan made a pretty cool post about them:

What is CTF and how to get started!

atan ・ Mar 28 '19

#security #ctf #beginners

Conclusion

Well, that was a long post. If you notice any mistakes, have any questions, or would just like to discuss something about Calypso, feel free to make a comment or hit me up on Discord (my Discord account is Paradoxical#2936). There's also a #calypso channel on the pretty popular programming language design Discord server.

Anyway, have a good morning/afternoon/evening/night/etc!
-James

Introducing The Calypso Chronicles

Wren [Undefined] — Tue, 02 Mar 2021 02:43:52 +0000

Edit: Apparently this got featured in TWIR! Thanks editors! :)
If you don't already know what Calypso is, I'd recommend reading the post I wrote about it a little bit after when I wrote this:

What is Calypso?

Wren [Undefined] ・ Mar 5 '21

#calypso

I recently found this site and actually have started to consider using it. I'm probably going to use this as a new version of my blog as I've been meaning to use something else for a little while.

Anyway, I'm going to start a new, (semi-)weekly series called "The Calypso Chronicles" about my status on Calypso, both how it's going and my plans for the future. This is mostly just for myself (though feel free to discuss anything here if you want to!).

It will be split into two sections: one for Calypso and one for Odysseus.

Onto the content!

Calypso

Notes and Thoughts

I finally have a stable part of my API! Unfortunately, this is currently just calypso_error and calypso_filety but it's progress! (Especially because calypso_filety was SUPER messy, so I ended up rewriting it completely. It's a lot better now.)
I wrote a blog post called "The Calypso Manifesto" about why I made Calypso, what I want it to become, and what it's not going to become. If you'd like to read that, it's available here.

TODOs

Since pretty much everything is a TODO, I'm just going to write immediate TODOs here for now. :P

Finish redesign of the language, including error handling.
Rewrite/fix lexer according to new design
- see https://github.com/calypso-lang/calypso/tree/wip-lexer-rewrite (WIP lexer rewrite branch, blocked on language redesign)
- see
  track: Lexer tracking issue #17
  ThePuzzlemaker posted on Dec 09, 2020
  This is a tracking issue for the lexer, contained within calypso_parsing::lexer. This is marked as blocking as it is impossible for Calypso to function without the lexer fully functioning.
  
  Updates
  
  12/12/20: The lexer has been split into multiple files (and clippy::pedantic has been enabled for calypso_parsing) as of d9d988c.
  
  Checklist
  
  [x] Lexer debug interface (int lex)
  
  [x] Basic tokens (e.g. <, +=, ...)
  
  [x] Keywords
  
  [x] Identifiers
  
  [ ] Literals
  
  [x] Character literals
  
  [x] String literals
  
  [ ] Number literals
  
  [ ] Signed integer literals
  
  [ ] Unsigned integer literals
  
  [ ] Float literals
  View on GitHub

Odysseus

Notes and Thoughts

Oh goodness generics can be cursed. In order to reduce boilerplate, I created these traits for the Odysseus bytecode builder:

//! Traits used to make construction of bytecode builders much easier. These
//! may use complex or generally irritating generics, but it is much better
//! than repeating a lot of boilerplate. It is recommended that you read the
//! documentation before implementing any of these traits.

/// A trait used for "entries" of a bytecode [`Parent`], similar to a
/// [`std::collections::hash_map::Entry`].
pub trait Entry {
    /// The parent type of this entry.
    type Parent: Parent<<Self as Entry>::Element>;
    /// The element type of this entry.
    type Element: Element;

    /// Get the bytecode element behind this entry, if finished.
    ///
    /// # Implementors
    ///
    /// Do not re-implement this function. It has an implementation that is
    /// already sufficient for generalized functionality.
    fn get<'p>(&self, parent: &'p <Self as Entry>::Parent) -> Option<&'p <Self as Entry>::Element> {
        if !self.is_finished(parent) {
            panic!("cannot get an unfinished bytecode element");
        }
        parent.get(self.id())
    }

    /// Check if the bytecode element has been finished.
    ///
    /// # Implementors
    ///
    /// Do not re-implement this function. It has an implementation that is
    /// already sufficient for generalized functionality.
    fn is_finished(&self, parent: &<Self as Entry>::Parent) -> bool {
        parent.is_finished(self.id())
    }

    /// Enter the builder context and use the closure to build the bytecode
    /// element.
    ///
    /// # Implementors
    ///
    /// Do not re-implement this function. It has an implementation that is
    /// already sufficient for generalized functionality.
    fn enter(
        &mut self,
        parent: &mut <Self as Entry>::Parent,
        f: impl FnOnce(
            <<Self as Entry>::Element as Element>::Builder,
        ) -> <<Self as Entry>::Element as Element>::Builder,
    ) -> &mut Self {
        if self.is_finished(parent) {
            panic!("cannot enter a finished bytecode element");
        }
        let builder = f(parent.create_builder(self.id()));
        self.internal_build(parent, builder);
        self
    }

    /// Finish the bytecode element.
    ///
    /// # Implementors
    ///
    /// Do not re-implement this function. It has an implementation that is
    /// already sufficient for generalized functionality.
    fn finish<'p>(
        &mut self,
        parent: &'p mut <Self as Entry>::Parent,
    ) -> &'p <Self as Entry>::Element {
        if self.is_finished(parent) {
            panic!("cannot build a finished bytecode element")
        }
        parent.finish(self.id())
    }

    /// Get the ID of the bytecode element behind the entry.
    fn id(&self) -> <<Self as Entry>::Element as Element>::Id;

    /// An internal function to register changes caused by a builder. Do not
    /// call this function as it may cause errors.
    fn internal_build(
        &mut self,
        parent: &mut <Self as Entry>::Parent,
        builder: <<Self as Entry>::Element as Element>::Builder,
    );
}

/// A bytecode parent, i.e. a structure that contains one or more types of
/// bytecode elements.
pub trait Parent<Child>
where
    Child: Element,
{
    /// Check if a bytecode element is finished. This function should return
    /// `false` if the ID was not present within the parent. This function is
    /// called by [`Entry::is_finished`].
    fn is_finished(&self, id: Child::Id) -> bool;

    /// Get a reference to a bytecode element. This function is called by
    /// [`Entry::get`].
    fn get(&self, id: Child::Id) -> Option<&Child>;

    /// Create a builder from the specified bytecode element ID.
    ///
    /// # Panics
    ///
    /// This function may panic if the ID was not present within the parent,
    /// depending on the implementor.
    fn create_builder(&mut self, id: Child::Id) -> Child::Builder;

    /// Finish the bytecode element with the specified ID.
    ///
    /// # Panics
    ///
    /// This function may panic if the ID was not present within the parent,
    /// depending on the implementor.
    fn finish(&mut self, id: Child::Id) -> &Child;
}

/// A bytecode element, i.e. a single element in the bytecode, whether
/// high-level or low-level.
///
/// For example, all these are bytecode elements:
/// - Modules
/// - Functions
/// - Blocks
pub trait Element {
    /// The entry type of this bytecode element.
    type Entry: Entry;
    /// The builder type of this bytecode element, i.e. a structure that can
    /// construct a bytecode element. It should store changes within its own
    /// structure, then its corresponding [`Entry`] type should implement the
    /// [`Entry::internal_build`] function that writes these changes to the
    /// [`Parent`].
    type Builder;
    /// The ID type of this bytecode element. It is required to be [`Copy`] to
    /// ensure that IDs are small and also because of limitations with
    /// borrowing and traits.
    type Id: PartialEq + Eq + Copy;
}

I might just be lucky but these are the most cursed generics I've ever had to use. I'm not sure why I have to explicitly disambiguate (e.g. <<Self as Entry>::Element as Element>::Builder) but rustc complains at me if I don't. :'(

TODOs

After the redesign of Calypso is done, solidify ABI and ISA for Odysseus.
After that's done, I can complete the bytecode builder and then eventually the runtime. (And at that point, Odysseus is basically done! The runtime is going to take a while though.)

Conclusion

Basically, it's going well! :)

If you find anything that I might've missed or you'd like more information on, feel free to comment or otherwise contact me!

Otherwise, have a good morning/afternoon/evening/night/etc.!
-James

DEV Community: Wren [Undefined]

The Unstandardized Standardized Implementation Limititations of WebAssembly

#![no_std] with WASI is more complicated than I thought it would be

The most underrated but useful Rust standard library type

What is Cow?

Why is it helpful?

Why is it confusing?

Conclusion

Introducing The Calypso Chronicles

Wren [Undefined] ・ Mar 2 '21

Tips for Productivity that I've Learned from Calypso

A Bunch of Odysseus Pseudocode I Guess

Calypso

Notes and Thoughts

TODOs

Odysseus

Notes and Thoughts

TODOs

Calypso Build Tool

Conclusion

What is Calypso?

Introducing The Calypso Chronicles

Wren [Undefined] ・ Mar 2 '21

calypso-lang / calypso

Calypso is a mostly imperative language with some functional influences that is focused on flexibility and simplicity.

Calypso

Example

So, what is Calypso?

What does the tagline mean?

So why did I create Calypso?

What would I like Calypso to become?

What am I not expecting Calypso to become?

What are the parts of the Calypso ecosystem?

Calypso Compiler and Tooling

The standard library/ies

The bytecode virtual machine

Officially developed and maintained libraries

What is CTF and how to get started!

atan ・ Mar 28 '19

Conclusion

Introducing The Calypso Chronicles

What is Calypso?

Wren [Undefined] ・ Mar 5 '21

Calypso

Notes and Thoughts

TODOs

track: Lexer tracking issue #17

Updates

Checklist

Odysseus

Notes and Thoughts

TODOs

Conclusion

What is `Cow`?