DEV Community: Ben Lesh

BigInt and JSON.stringify/JSON.parse

Ben Lesh — Mon, 04 Jan 2021 18:25:10 +0000

As of this writing, JavaScript's JSON.parse cannot serialize the new JavaScript type BigInt.

Imagine you have the following:

const data = {
  value1: BigInt('1231231231231231213'),
  deep: {
    // NOTE the "n" at the end -- also a BigInt!
    value2: 848484848484848484884n,
  }
}

If you try to just JSON.stringify(data) you will get the error TypeError: Do not know how to serialize a BigInt.

Serialization and Deserialization

It should be noted that how you choose to serialize your BigInts affects how you deserialize your BigInts. Generally, I serialize them by doing appending the "n" suffix to the end, similar to how we can declare a BigInt inline. (BigInt(0) and 0n yield the same result).

Serialization

Here we use JSON.stringify's second argument (It's not always null!!! haha.) which is the replacer. The job of this function, if provided, is to determine how to serialize something based off of it's key and value. If the typeof the value is "bigint", we're going to convert it to a string, and tack an "n" to the end.

// Serialization
const json = JSON.stringify(data, (key, value) =>
  typeof value === "bigint" ? value.toString() + "n" : value
);

The result: json is:

{
  "value1": "1231231231231231213n",
  "deep": {
    "value2": "848484848484848484884n",
  }
}

Deserialization

In order to deserialize what we have above, we can use the second argument to JSON.parse(). (I bet most people didn't know it has a second argument) This is called the reviver, and it's job is to do basically the opposite of the replacer above.

Here we'll test for the type and shape of the value to see that it matches a bunch of numbers followed by an "n".

// Deserialize
const backAgain = JSON.parse(json, (key, value) => {
  if (typeof value === "string" && /^\d+n$/.test(value)) {
    return BigInt(value.substr(0, value.length - 1));
  }
  return value;
});

Alternative serializations

This is all a little tricky, because you have to be sure that none of your other data is in a format where it's a bunch of numbers and an "n" at the end. If it is, you need to change your serialization strategy. For example, perhaps you serialize to BigInt::1231232123 and deserialize the same at the other side, such as the example below:

// Serialize
const json = JSON.stringify(data, (key, value) =>
  typeof value === "bigint" ? `BIGINT::${value}` : value
);

// Deserialize
const backAgain = JSON.parse(json, (key, value) => {
  if (typeof value === "string" && value.startsWith('BIGINT::')) {
    return BigInt(value.substr(8));
  }
  return value;
});

The choice is really up to you, just as long as you have the tools to do it.

Observables, Reactive Programming, and Regret

Ben Lesh — Mon, 29 Jun 2020 21:16:20 +0000

As of this writing, I've been working on the RxJS project for almost 6 years, I think. When I started out, I really had no idea what I was getting into (And I wouldn't have been able to ship those first versions without Paul Taylor and others, for sure). I can remember looking at the number of weekly downloads on npm, and being able to figure out how exactly how many of them were mine. Fast forward to today, RxJS and observables have become extremely popular... Loved by many, hated by a few, and my fear is, misunderstood by most.

Observables vs Reactive Programming vs RxJS

A big problem I see nowadays is how observables have now been intimately linked, for better or worse, to RxJS. Looking back, I sort of wish we had published the Observable primitive as a separate package, and the operators in another package.

When this effort started, I was naively optimistic that Observable would land in the ECMAScript standard, and RxJS would just "become a collection of helper functions" as I think I put it. But years passed, and the TC39 proposal stalled. And the way the world came to know Observable was through RxJS.

Observables are not RxJS. Observables do not require "operators". They are a primitive. The "dual" of the Iterable. A simple push-based type. Nothing more.

Reactive programming isn't necessarily observables. Reactive programming is a paradigm or a practice. It can be done with functions, Promises, etc. In essence, if you can compartmentalize your code into functions that will "react" to incoming events without knowing anything about the source, congrats, you're "reactive".

RxJS is a library of functions built around observables, not the other way around. Observables can, and do, exist in the wild without RxJS. They show up in other libraries, often times in slightly different shapes, but the overall concept is the same. Facebook's Relay has an internal Observable implementation that is eerily similar to RxJS's implementation. In fact, I've lost count of the number of times I've seen an abstraction that amounts to an interface that accepts a callback to handle multiple values, an error, or a completion, and returns or otherwise uses some sort of cancellation semantic.

Regrets

1. The huge API

RxJS 5 inherited its HUGE API surface area from RxJS 4 and under. RxJS 4 and under, in turn, inherited it's API from RxNET, many, many years ago. So much of the API that some might deem "unnecessary" exists because "it always has been, and always must be". RxJS 5 might have been our only chance in the history of the library to truly ween that down. Which we did a bit, but probably not enough. The large API surface leads to confusion and loathing in the community. All of which is understandable, IMO.

2. RxJS out-shined Observable

Observables never had a chance to shine on their own. The real win, IMO, to RxJS is the Observable type itself. Not the operators. Those are just fluff that allow you to do some cool things. Having a lazy type with guarantees like Observable is actually a bigger deal.

With Observable you're guaranteed:

Once it's complete, errored, or unsubscribed, you will get no more messages
Registered teardown WILL occur. If you complete, error, or unsubscribe, you are guaranteed to clean up resources.
A unified API that can represent a wide variety of things: Events, multiple values, single values, user interactions, streaming data, synchronous values, asynchronous values, etc. etc.

There are other great advantages to its design. But IMO, those are the biggest.

RxJS and all of its operators are inseparable in some people's heads from observables. And that's a real shame. Observable is a simple thing. A very simple type. RxJS is complicated with it's huge API and odd names.

3. We never really outlined where RxJS would best serve people

Disclaimer: These are MY opinions on RxJS/Observable use, and not really the RxJS core team's. Feel free to use RxJS or whatever library in whatever way you see fit. If it works, you can maintain it, and you can test it, IMO it's good code. The end.

To put it simply, once people get into RxJS, it's an exciting technology. It suddenly gets used for everything. It's fair to say this mentality exists in tech for a lot of libraries and frameworks. But I think with RxJS it becomes insidious to the detriment of the RxJS community.

Examples:

You have a button that, when clicked, fetches the latest data and displays it. Do you need full-on RxJS? No, probably not. "But what about cancellation???" .. You wanted an observable. Not operators. You can use RxJS here for the Observable implementation, but I would caution against jumping into concatMap et al. Especially if your team isn't used to RxJS. But that doesn't mean you shouldn't use Observable. In fact, you probably should.
You have streaming data over a web socket, and you need to split it into a couple of different streams and update two parts of your UI. Yes! This is what RxJS is for. You're a filter operator away from a solid use case.
You have complex async coordination and/or race conditions, even with APIs that return promises? Honestly, you might want to use RxJS here as well, because of guarantees provided by Observable, and useful operators like concatMap that can guarantee ordering, etc, and have complete interop with async/await and Promise.

4. We never taught people how to write readable code with RxJS

We handed people powerful tools and let them go at it. No guidance or experienced wisdom provided with how to effectively use the library so that you didn't drive your coworkers crazy. This is sort of like getting a power tool set with no manuals. How do you maintain it? How do you resolve issues? Where do you store the tools? etc.

The result of this is people write code they don't understand when they revisit it. Most amazingly, some engineers, who are usually a rational bunch, then declare RxJS to be "unreadable", as in, no matter what they did, they could never make the code readable. Seems defeatist to me. Like anything else, good practices and strategies around reading and organizing rxjs code can be learned and taught. But I know that I personally haven't done enough to spread this know-how.

Consequences

For the most part, I think the response to RxJS has been overwhelmingly positive. The community has organized a conference. I've seen a lot of discussion about it across many communities (beyond just Angular). And usage has been steadily growing.

But on the back swing, there is a trail of destruction to the reputation of RxJS and Observable that has been wrought by misunderstandings about Observable and RxJS, and misuse of the library in general, IMO. There have been well-known tech personalities who have called out "wishing RxJS didn't exist". And my fear is that sort of thinking, if it spreads, will spell doom for the Observable type itself. That would be the biggest shame to this, honestly.

The Observable itself is huge win. It's a primitive that, like I said above, shows up in many forms in many places, and I think it deserves a spot in the language as much as Iterable and Promise. People having a distaste for RxJS's API and/or abuse and misuse is completely understandable, IMO.

There are parts of RxJS I don't like, and here I am unable to pivot the library quickly because it's so popular we'd simply break too many people. But the parts I like the most, the Observable itself, and the guarantees it provides, are in jeopardy of being thrown out with the bath water by some folks. And that's tragic, IMO.

The road forward

For my part, I plan on trying to continue to champion promoting understanding of the when/where/why of RxJS and Observable. And I want to do better to disambiguate Observable from RxJS. I also want to work very hard to simplify the RxJS API: Tighten the API, remove what does not need to be there, improve documentation and readability, add more guidance for folks on how to make their code more maintainable, etc.

Don't get me wrong, I have other regrets with regards to RxJS as it stands, but I'm confident we can remedy all of those things over time. My deepest concern is that there are a huge numbers of people that still don't understand the Observable primitive and its benefits, because they associate it with RxJS and are standoffish about getting involved there because of the learning curve.

Understanding Array reduce by building it from scratch

Ben Lesh — Tue, 31 Dec 2019 18:24:03 +0000

Recently, I was involved in a thread on Twitter, where I mentioned that I also, at one time, found Array reduce challenging to wrap my head around. At first glance, it doesn't make any sense. The name, the signature of the thing, it's a bit alien in comparison to methods like map, filter, and forEach.

But what if I told you that each of the above methods are really just specializations of reduce?

Why is it called "reduce"? I'm not actually sure here. But how I remember what it does is that, generally, you're taking the array and you're "reducing it down" to something else. Now, this is a bit of a misnomer still, because you could use reduce to make a new, larger, array. But when I say "reduce", I mean more like "reduction" in cooking. You're taking your array, and you're making something else with it by running it through a process.

Starting with a basic loop

We already have ways to do this without any methods, of course. Considering the following:

const numbers = [1, 2, 3];
const plus1 = [];

for (let i = 0; i < numbers.length; i++) {
  const item = numbers[i];
  plus1.push(item + 1);
}

console.log(plus1); // [2, 3, 4]

Here, we have a source array, numbers, and we're looping over it, then we're updating a bit of existing state, the array plus1, by pushing values onto it derived from the items in our source array. It's efficient, and pretty simple, overall.

Now imagine we wanted to refactor this process into a few pieces so we could test it or reuse it in other ways. We could do the work inside of the loop in its own function:

function add1AndAppend(arr, item) {
  arr.push(item + 1);
}

const numbers = [1, 2, 3];
const plus1 = [];

for (let i = 0; i < numbers.length; i++) {
  add1AndAppend(plus1, numbers[i]);
}

console.log(plus1); // [2, 3, 4]

Now we have this function add1AndAppend we're calling on each loop. It's fine, but it's not great. For one thing, it's not a "pure" function, it's actually mutating the array we are passing to it. This means it could behave in undesirable ways, or be a pain to deal with later, as there's more to reason about. (There's been volumes written on the dangers of shared mutable state).

So we can refactor it to return a new array every time, making it "pure", in fact, I'll even rename it to add1AndConcat:

function add1AndConcat(arr, item) {
  return [...arr, item + 1];
}

const numbers = [1, 2, 3];
let plus1 = [];

for (let i = 0; i < numbers.length; i++) {
  plus1 = add1AndConcat(plus1, numbers[i]);
}

console.log(plus1); // [2, 3, 4]

And now we have this method, that we can easily test, that will take an array, and an item, and add 1 to the item, then create a new array that contains the items from the old array and the new item plus 1. We can reuse it, and we can test it:

expect(add1AndConcat([1, 2], 4)).toEqual([1, 2, 5]);

Creating a primitive reduce method

Wouldn't it be great if we had a method that could do these things for us (yes, yes, there's map, but that's not what we're learning about here yet).

function add1AndConcat(arr, item) {
  return [...arr, item + 1];
}

// This isn't the "real reduce" yet.
// Also, don't augment types like this in JavaScript. It's bad.
Array.prototype.reduce = function (callback) {
  let result = [];

  for (let i = 0; i < this.length; i++) {
    result = callback(result, this[i]);
  }

  return result;
};

const numbers = [1, 2, 3];

const plus1 = numbers.reduce(add1AndConcat);

console.log(plus1); // [2, 3, 4]

Now, wouldn't it be nice if we could use this method for more things? Like what if we don't always want the result to be an array? What if we want an object? or a number? We need to be able to change what result is initialized to:

Array.prototype.reduce = function (callback, initialState) {
  let result = initialState;

  for (let i = 0; i < this.length; i++) {
    // We can pass the index to the callback too, because why not?
    result = callback(result, this[i], i);
  }

  return result;
}

// and we'd call it like so:
const plus1 = numbers.reduce(add1AndConcat, []);

So this is pretty useful! We can use it to do all sorts of things now. Maybe we can take an array of values and turn it into an object:

const keysAndValues = ['x', 20, 'y', 30, 'z': 3, 'name', 'Emma' ];

function toAnObject(obj, item, i) {
  if (i % 2 === 0) {
    // keys
    obj[item] = undefined;
  } else {
    // values
    obj[keysAndValues[i - 1]] = item;
  }

  return obj;
}

const obj = keysAndValues.reduce(toAnObject, {});
console.log(obj); // { x: 20, y: 30, z: 3, name: "Emma" }

BUT WAIT! That sucks! We can't really test that function because it's not "pure", as it's closing over keysAndValues as shared state. So what if we added one more argument to our callback, which is the source array:

Array.prototype.reduce = function (callback, initialState) {
  let result = initialState;

  for (let i = 0; i < this.length; i++) {
    result = callback(result, this[i], i, this);
  }

  return result;
}

function toAnObject(obj, item, i, source) {
  if (i % 2 === 0) {
    // keys
    obj[item] = undefined;
  } else {
    // values
    obj[source[i - 1]] = item;
  }

  return obj;
}

const obj = keysAndValues.reduce(toAnObject, {});
console.log(obj); // { x: 20, y: 30, z: 3, name: "Emma" }

And now we can test it:

const source = ['a', 1, 'b', 2];
expect(toAnObject({}, 'a', 0, source)).toEqual({ a: undefined });
expect(toAnObject({ a: undefined }, 1, 1, source)).toEqual({ a: 1 });
expect(toAnObject({ a: 1 }, 'b', 2, source)).toEqual({ a: 1, b: undefined, });
expect(toAnObject({ a: 1, b: undefined }, 2, 2, source)).toEqual({ a: 1, b: 2 });

No second argument

Probably the most confusing behavior of reduce

There's one quirk people don't often get, which is: What happens when you don't pass an initial state to reduce? The second argument is actually optional.

In the event that an initial state is NOT provided, the first value from the array will be "skipped" by the reducer function (the callback) and used as the initial state. These two things are equivalent:

[a, b, c].reduce(fn, INIT);

// is the same as

[INIT, a, b, c].reduce(fn);

This makes our fake reduce method above a lot more complicated:

Array.prototype.reduce = function (callback, initialState) {
  const hasInitialState = arguments.length > 1;

  let result = initialState;

  for (let i = 0; i < this.length; i++) {
    if (i === 0 && !hasInitialState) {
      result = this[i];
    } else {
      result = callback(result, this[i], i, this);
    }
  }

  return result;
}

DIY map and filter from reduce:

Well, we already sorta did a "map" above with the add1AndConcat, but let's just make a fake map right here:

map

Array.prototype.map = function (callback) {
  return this.reduce(
    (result, item, i, source) =>
      [...result, callback(item, i, source)],
    []
  );
}

Filter is more of the same, but this time we're asserting on a predicate before deciding to append to the result:

filter

Array.prototype.filter = function (callback) {
  return this.reduce(
    (result, item, i, source) =>
      callback(item, i, source) ? [...result, item] : result,
    []
  );
}

Reduce and reducer functions in the world at large

The callback to Array reduce is called a "reducer", and in recent years, its shape has been popularized by libraries like Redux, NgRx, and RxJS. It's a function signature for creating a pure function that is able to handle passing of some pre-existing state, as well as some value (such as an action, or other array item), then return a new state. In TypeScript that could be declared (very loosely, like so):

type ReducerFunction<T, S> = (currentState: S, item: T, index: number) => S; // returns new state

While Redux, RxJS and NgRx, are all doing things to state "asynchronously", as opposed to the synchronous behavior we see in Array reduce, the principles are exactly the same. An underlying state is initialized and maintained, and passed to the callback at each turn. In the cases of RxJS, Redux, and NgRx, the resulting state is something that requires subscription to observe.

In RxJS can can be expressed with scan:

import { of } from 'rxjs';
import { scan } from 'rxjs/operators';

function toSquares(result, number) {
  return [...result, number * number];
}

of(1, 2, 3).pipe(
  scan(toSquares, []);
).subscribe(x => console.log(x));

/**
 * [1]
 * [1, 4]
 * [1, 4, 9]
 */

But notice, we could reuse the same exact reducer with Array reduce:

[1, 2, 3].reduce(toSquares, []); // [1, 4, 9]

Special thanks to @EmmaBostian for inspiring me to write this article. It's knowledge that I've had for a long time and I take for granted. Hopefully others find this useful.

A simple explanation of functional pipe in JavaScript

Ben Lesh — Thu, 27 Jun 2019 22:11:43 +0000

Sometimes I'm asked why we don't have "dot-chaining" in RxJS anymore, or why RxJS made the switch to use pipe. There are a lot of reasons, but this is really something that needs to be looked at from a higher level than just RxJS.

The need for piping functions comes from two problems butting heads: The desire to have a broad set of available development APIs for simple types (like Array, Observable, Promise, etc), and the desire to ship smaller apps.

The size problem

JavaScript is a very unique language with a problem that most other programming languages do not have: Usually, JavaScript is shipped over a network, parsed, and executed at the exact moment the user wants to use the app the JavaScript is powering. The more JavaScript shipped, the longer it will take to download and parse, thus slowing down your app's responsiveness. Something that can have a HUGE impact on the user experience.

This means that trying to keep JavaScript apps small is critically important. Fortunately, we have a lot of great tools to do this nowadays. We have a lot of "build time" bundlers and optimizers that can do things like tree-shaking in order to get rid of unused code prior at build time, so we can ship the least amount of JavaScript possible to the users.

Unfortunately, tree-shaking doesn't remove code if it can't statically be sure that the code isn't being used somewhere.

Providing broad APIs

For types to be as useful as possible, it is nice to have a well-groomed set of known functionality attached to the type. Especially in such a way that it can be "chained" by making calls left-to-right on that type.

The "built-in" way for JavaScript to provide broad APIs for a given type is prototype augmentation. This means that you would add methods to any given type's prototype object. So if we wanted to add a custom odds filter to array, we could do it like this:

Array.prototype.odds = function() {
  return this.filter(x => x % 2 === 1)
}

Array.prototype.double = function () {
  return this.map(x => x + x);
}

Array.prototype.log = function () {
  this.forEach(x => console.log(x));
  return this;
}

Prototype augmentation is problematic

Mutating global variables. You're now manipulating something that everyone else can touch. This means other code could start depending on this odds method being on Array, without knowing that it actually came from a third party. It also means that another bit of code could come through and trample odds with it's own definition of odds. There are solutions to this, like using Symbol, but it's still not ideal.

Prototype methods cannot be tree-shaken. Bundlers will not currently attempt to remove unused methods that have been patched onto the prototype. For reasoning, see above. The bundler has no way of knowing whether or not a third party is depending on using that prototype method.

Functional programming FTW!

Once you realize that the this context is really just a fancy way to pass another argument to a function, you realize you can rewrite the methods above like so:

function odds(array) {
  return array.filter(x => x % 2 === 0);
}

function double(array) {
  return array.map(x => x + x);
}

function log(array) {
  array.forEach(x => console.log(x));
  return array;
}

The problem now is that you have to read what's happening to your array from right-to-left, rather than from left-to-right:

// Yuck!
log(double(odds([1, 2, 3, 4, 5])))

The advantage, though, is that if we don't use double, let's say, a bundler will be able to tree-shake and remove the double function from the end result that is shipped to users, making your app smaller and faster.

Piping for Left-to-right readability

In order to get better left-to-right readability, we can use a pipe function. This is a common functional pattern that can be done with a simple function:

function pipe(...fns) {
  return (arg) => fns.reduce((prev, fn) => fn(prev), arg);
}

What this does is return a new higher-order function that takes a single argument. The function that that returns will pass the argument to the first function in the list of functions, fns, then take the result of that, and pass it to the next function in the list, and so on.

This means that we can now compose this stuff from left-to-right, which is a little more readable:

pipe(odds, double, log)([1, 2, 3, 4, 5])

You could also make a helper that allowed you to provide the argument as the first argument to make it even more readable (if a bit less reusable) like so:

function pipeWith(arg, ...fns) {
  return pipe(...fns)(arg);
}

pipeWith([1, 2, 3, 4, 5], odds, double, log);

In the case of pipeWith, now it's going to take the first argument, and pass it to the function that came right after it in the arguments list, then it will take the result of that and pass it to the next function in the arguments list, and so on.

"Pipeable" functions with arguments

To create a function that can be piped, but has arguments, look no further than a higher order function. For example, if we wanted to make a multiplyBy function instead of double:

pipeWith([1, 2, 3, 4, 5], odds, multiplyBy(2), log);

function multiplyBy(x) {
  return (array) => array.map(n => n * x);
}

Composition

Because it's all just functions, you can simplify code and make it more readable by using pipe to create other reusable, and pipeable, functions!

const tripleTheOdds = pipe(odds, multiplyBy(3));


pipeWith([1, 2, 3, 4, 5], tripleTheOdds, log)

The larger JS ecosystem and the Pipeline Operator

This is roughly the same pattern that is used by RxJS operators via Observable pipe method. This was done to get around all of the issues listed with prototype above. But this will clearly work with any type.

While prototype augmentation may be the "blessed" way to add methods to types in JavaScript, in my opinion, it is a bit of an antipattern. JavaScript needs to start embracing this pattern more, and ideally we can get a simple version of the pipeline operator proposal to land in JavaScript.

With the pipeline operator, the above code could look like this, but be functionally the same, and there wouldn't be a need to declare the pipe helper.

pipeWith([1, 2, 3, 4, 5], odds, double, log);

// becomes

[1, 2, 3, 4, 5] |> odds |> double |> log