How to seamlessly exchange data between JavaScript environments

JSON limitations

Wouldn't you find it strange if adults who are fluent in the same language spoke to each other using the vocabulary of a 3-year-old? Well, something analogous is happening when browsers and JavaScript servers exchange data using JSON, the de facto serialization format on the internet.

For example, if we wanted to send a Date object from a JavaScript server to a browser, we would have to:

1. Convert the Date object to a number.
2. Convert the number to a JSON string.
3. Send the JSON string to the browser.
4. Revert the JSON string to a number.
5. Realize the number represents a date.
6. Revert the number to a Date object.

This roundabout route seems ludicrous, because the browser and server both support the Date object, but is necessary, because JSON does not support the Date object.

In fact, JSON does not support most of the data types and data structures intrinsic to JavaScript.

JOSS as a solution

The aforementioned limitations of JSON motivated us to create the JS Open Serialization Scheme (JOSS), a new binary serialization format that supports almost all data types and data structures intrinsic to JavaScript.

JOSS also supports some often overlooked features of JavaScript, such as primitive wrapper objects, circular references, sparse arrays, and negative zeros. Please read the official specification for all the gory details.

JOSS serializations come with the textbook advantages that binary formats have over text formats, such as efficient storage of numeric data and ability to be consumed as streams. The latter allows for JOSS serializations to be handled asynchronously, which we shall see in the next section.

Reference implementation

The reference implementation of JOSS is available to be downloaded as an ES module (for browsers and Deno), CommonJS module (for Node.js), and IIFE (for older browsers). It provides the following methods:

• serialize() and deserialize() to handle serializations in the form of static data.
• serializable(), deserializable(), and deserializing() to handle serializations in the form of readable streams.

To illustrate the syntax of the methods, allow us to guide you through an example in Node.js.

First, we import the CommonJS module into a variable called JOSS.

// Change the path accordingly
const JOSS = require("/path/to/joss.node.min.js");

Next, we create some dummy data.

const data = {
  simples: [null, undefined, true, false],
  numbers: [0, -0, Math.PI, Infinity, -Infinity, NaN],
  strings: ["", "Hello world", "I \u2661 JavaScript"],
  bigints: [72057594037927935n, 1152921504606846975n],
  sparse: ["a", , , , , ,"g"],
  object: {foo: {bar: "baz"}},
  map: new Map([[new String("foo"), new String("bar")]]),
  set: new Set([new Number(123), new Number(456)]),
  date: new Date(),
  regexp: /ab+c/gi,
};

To serialize the data, we use the JOSS.serialize() method, which returns the serialized bytes as a Uint8Array or Buffer object.

const bytes = JOSS.serialize(data);

To deserialize, we use the JOSS.deserialize() method, which simply returns the deserialized data.

const copy = JOSS.deserialize(bytes);

If we inspect the original data and deserialized data, we will find they look exactly the same.

console.log(data, copy);

It should be evident by now that you can migrate from JSON to JOSS by replacing all occurrences of JSON.stringify/parse in your code with JOSS.serialize/deserialize.

Readable Streams

If the data to be serialized is large, it is better to work with readable streams to avoid blocking the JavaScript event loop.

To serialize the data, we use the JOSS.serializable() method, which returns a readable stream from which the serialized bytes can be read.

const readable = JOSS.serializable(data);

To deserialize, we use the JOSS.deserializable() method, which returns a writable stream to which the readable stream can be piped.

const writable = JOSS.deserializable();
readable.pipe(writable).on("finish", () => {
  const copy = writable.result;
  console.log(data, copy);
});

To access the deserialized data, we wait for the piping process to complete and read the result property of the writable stream.

Whilst writable streams are well supported in Deno and Node.js, they are either not supported or not enabled by default in browsers at the present time.

To deserialize when we do not have recourse to writable streams, we use the JOSS.deserializing() method, which returns a Promise that resolves to the deserialized data.

const readable2 = JOSS.serializable(data);
const promise = JOSS.deserializing(readable2);
promise.then((result) => {
  const copy = result;
  console.log(data, copy);
});

Servers

In practice, we would serialize data to be sent in an outgoing HTTP request or response, and deserialize data received from an incoming HTTP request or response.

The reference implementation page contains examples on how to use JOSS in the context of the Fetch API, Deno HTTP server, and Node.js HTTP server.

Closing remarks

JOSS will evolve with the JavaScript specification. To keep track of changes to JOSS, please star or watch the GitHub repository.

Comments

[𒎏Wii 🏳️‍⚧️]

How does this compare to MessagePack?

[Wynn Tee]

MessagePack is more general-purpose covering more languages. JOSS is specific to JavaScript and its idiosyncrasies.

For example, in JavaScript both Object and Map are associative arrays. So if you used a Map on the client-side, JOSS will ensure you get a Map (as opposed to Object) on the server-side.

There are also the little things like primitive wrapper objects, sparse arrays, and signed zeros. They might not be important to some, but the point is JOSS will ensure they're reproduced.

When Temporal becomes an official part of JavaScript, JOSS will also support that in addition to Date. Hope that answers your question.