DEV Community: EmNudge

Understanding Loose Equality In JavaScript

EmNudge — Wed, 06 Jan 2021 17:54:52 +0000

For those who prefer a more audiovisual form, a video almost identical to the article can be seen over here:

Abstract Equality, or as I've titled this article "Loose Equality" is (I think) one of the most misunderstood topics in JavaScript. People know loose equality, the double equals (==), to check if its operands are roughly equal to each other. The string "55" and the number 55 are kind of the same thing, but not strictly the same thing, with triple equals (===).

People usually advise against using loose equality. Personally? Well if JavaScript came out with a **strict* strict mode* that removed loose equality, I wouldn't be too bothered.

But there's a lot of misinformation out there and I thought it would be helpful to clean some of that up. Which is why I've been working on this topic for so long.



// loose equality vs strict equality
"55" == 55   // -> true
"55" === 55  // -> false

Loose equality, in reality, is a process that tries to implicitly coerce its operands to be the same type before passing it off to strict equal to give you the real result. Implicit coercion by itself actually isn't too bad. It's used in many other languages and JavaScript programmers use it pretty often.

In this example, we take advantage of falsy and truthy values to check whether we should print out an array to the console. If the array exists and has a length property greater than 0, print it out.



// example of implicit coercion
const myArr = [1, 2, 3, 4, 5];
if (myArr && myArr.length) {
  console.log("My arr is: " + myArr);
}

Falsy values include all the JavaScript values that will evaluate to false when converted into a boolean.



Boolean('')         // -> false
Boolean(0)          // -> false
Boolean(0n)         // -> false
Boolean(NaN)        // -> false
Boolean(null)       // -> false
Boolean(undefined)  // -> false

Do not confuse this with abstract equality, however. Double equals often does not rely on this system whatsoever. While using the exact same values, we get true for half only. I'm no statistician, but 50-50 looks like zero correlation to me.



false == ''         // -> true
false == 0          // -> true
false == 0n         // -> true
false == NaN        // -> false
false == null       // -> false
false == undefined  // -> false

In fact, I would go so far as to say the concept of falsy values never comes up within abstract equality in the spec? What's the spec?

The JavaScript specification is an esoteric document that instructs browsers on how JavaScript should work. Browsers all can code up the implementation themselves, but if you want to know how JavaScript works without digging through C++ code, this is the best place to look.

The spec can often be pretty confusing, but this particular section is actually kind of readable. It defines abstract equality as a list of steps and I think it's pretty cool. If you're ever wondering why null is loosely equal to undefined, this is why. Because it says so. There is no low-level reason why it must be that way - the discussion stops here. It works that way because the document says it should.

While I can go through the document, I'm going to instead use a tool I've been working on to explain it a bit more simply - The Abstract Equality Stepper. I've written up the steps to roughly match spec. There are some minor changes in formatting to help with how my tool works, but it's essentially the same.

Let's punch in some examples we've just shown to explore how this works. false and 0 perhaps.

(View it here)

We can see that it declares either of the operands are a boolean, we convert the boolean to a number. Always. No matter what the other value is.

Notice that it tells us to perform an abstract equality comparison, but these are the steps that define what an abstract equality comparison is. That's right, this is recursion. We restart with new values. Since the types are now equal, we chuck it off to a strict equality comparison which returns true since they're the same value.

Notice that abstract equality uses strict equality.

So technically abstract equality must be less performant if the implementation matches the spec exactly. This is way too minor to matter in practice, but I thought it was interesting.

Let's try false and ''. We convert the boolean to a number like last time, but now we're left with a number versus a string.

(View it here)

We convert the string to a number and then go to strict equality. We're converting to numbers a lot here. It's for good reason. Numbers can be thought of as the most primitive type. It's easy to compare number to number and it's essentially what we're doing when we compare anything else. Even when we compare using reference equality (like with 2 objects) we're comparing memory locations, which, as you might have guessed, are numbers.

We can substitute 0 for false for all of the other examples.



0 == NaN        // -> false
0 == null       // -> false
0 == undefined  // -> false

0 isn't NaN so that's gonna be false. And then there is no step to define 0 and null or undefined, so we get false by default.

Nothing to do with falsy values here. Just looking at steps and following the rules.

With that out of the way, let's look at a common example of abstract equality weirdness - a real headscratcher.

WTFJS - The Headscratcher



![] == []  // -> true

This looks paradoxical, but it actually makes sense. First, we convert the left array to a boolean. This does involve the concept of falsy, but we haven't touched abstract equality yet, just expression evaluation. Since arrays aren't falsy, we would get true, but we're using an exclamation mark, so we flip that and get false.



false == []

Since booleans always turn to numbers in this system, our operands are 0 and []. Now what?

Well, now we find ourselves face to face with the magical ToPrimitive. This one is interesting. We can't just compare a primitive value and an object, we need 2 primitive values or 2 objects. We try turning our array into a primitive and out pops an empty string.

(Note: a function is just a callable object. When we use the term object, we include functions)

0 and '' means we turn the string into a number, which leads us to 0 and 0 which are equal.

But how does ToPrimitive work? What does it do?

We can look at the spec again, but this time it's a little more difficult, so I've taken the liberty of converting it to plain JavaScript.

If we're passed a primitive value, just return that. No need to convert a primitive to a primitive.

Then we check for a Symbol.toPrimitive property. This is a rather recent addition to JavaScript which allows you to define the ToPrimitive behavior a bit more easily.

If such a method exists, we try to convert it to a number. How? We check for a .valueOf property, which is what Number calls. If you try adding your object to a number, it will try to look for this property and call it.

If this property doesn't exist on your object or it itself returns an object, we try converting it to a string. Using, of course, the .toString property. This is actually defined on all object by default, including arrays. If you don't touch your object then ToPrimitive will return a string. For arrays, this means returning all its values as a comma-separated list. If it's empty, that's an empty string.



const obj = {
    valueOf() {
        console.log('calling valueOf');
        return 100;
    },
    toString() {
        console.log('calling toString');
        return '👀';
    }
};

console.log(obj + 43);

console.log(`I see you ${obj}`);

(Note: string concatenation itself doesn't always call .toString)

And there's your explanation!

But if you look a bit closer, you'll notice a few errors being thrown. Wait, does that mean...

Yup! There are often times where just using double equals will throw an error instead of returning false. Let's create such a scenario right now.

Throwing Errors With Equality Checks



const obj1 = {
    [Symbol.toPrimitive]: 45
};

console.log(obj1 == 45);
// Uncaught TypeError: number 45 is not a function

We can also just make it a function, but return an object.



const obj2 = {
    [Symbol.toPrimitive]: () => Object()
};

console.log(obj2 == 45);
// Uncaught TypeError: Cannot convert object to primitive value

Or do the same with the other methods



const obj3 = {
    toString: () => Object(),
    valueOf: () => Object()
};

console.log(obj3 == 45);
// Uncaught TypeError: Cannot convert object to primitive value

Now, we can't actually delete these methods on most objects. I mentioned earlier that all objects implement this by default. All objects of course inherit this method from the object prototype and we can't really delete that.

However, it's possible to make an object with no prototype using Object.create(null). Since it has no prototype, it has no valueOf() and no toString() and thus it will throw an error if we compare it to a primitive. Magical!



Object.create(null) == 45
// Uncaught TypeError: Cannot convert object to primitive value

With that detour, let's close with the essence of this article - how to understand loose equality.

Conclusion

When comparing 2 things of different types, it'll help to convert the more complex type to a simpler representation. If we can convert to a number, do that. If we're adding an object to the mix, get the primitive value and again try squeezing a number out of it.

null and undefined are loosely equal and that's that.

If we get something like Symbol() or we compare null or undefined with anything else by each other, we get false by default. Symbol() actually has a .toString() method, but it doesn't really matter. The spec says we get false, so we get false.

If we want to describe the steps in a bit of a simpler form, it looks something like this:

null equals undefined
Number(string) == number
BigInt(string) == bigint
Number(boolean) == anything
ToPrimitive(object) == anything
BigInt(number) == bigint
false

Stay curious!

Deno - An Unfinished Beauty

EmNudge — Mon, 26 Oct 2020 00:05:53 +0000

2022 Update:

My concerns around immaturity have long since been addressed since the 1.0 release. While issues around arm32 builds might still exist, my concerns around instability, ICU, npm-interop, and permissions have all been addressed. You may read this article as a lens into the time period when it was released as opposed to a modern look on the tooling.

This is going to be a bit dense. I've been using Deno for a while now and I've been told it might be useful to publish my thoughts on it so far. I'll be jumping around a lot, going over its features, so don't expect a super philosophically consistent article. This is mostly an opinion piece based on my experiences.

For those unaware, Deno is a JavaScript runtime.

A What?

The differences between a JavaScript runtime and an engine aren't super clear, but colloquially I've found it to be:

JavaScript Runtime = JavaScript Engine + User-Facing Layer

The user-facing layer would be both the included APIs and how you run your code. Chrome, Node.js, and Deno all use Google's V8 engine to actually run the code, but they have varying APIs and ways with which you set up your code to be executed.

With Chrome you must include the script in an executed HTML file, with Node.js you use the node binary and perhaps a package.json, and with Deno you use the deno binary with a bunch of flags to define security and features.

As for their APIs, some things stay the same, while others might differ. All platforms implement the URL API the same. All platforms allow for WebAssembly, but the way your run WASM in the web differs from Deno and Node. Node and Deno both include (almost identical) path utilities, whereas the web does not. Deno and the web often include very similar APIs such as WebSockets, FormData, and ArrayBuffer, whereas Node might have something vaguely similar or requires third-party libraries. The web also includes some very useful APIs such as Canvas and DOMParser which Deno and Node are forced to use third-party libraries for.

But of course at the end of the day, they all run JavaScript. Deno is targeted at being a better version of Node, using the web and Node's faults as lessons to build from. It generally tries to match Web APIs where it can, but being a stand-alone binary, it can leverage a lot of power that a regular website cannot.

The Great

With that general introduction, Deno has a lot of really nice Quality Of Life features. I'm going to separate them into categories because they'd be too much to tackle individually.

The Standard Library

This one isn't mentioned all that often, but Deno has a really nice standard library. It has a lot of helpful functions that really do feel like extras rather than necessities, but I don't mean that in a bad way. Where the web and node will be forced to use 3rd party libraries for many things, Deno has some great packages out of the box.

datetime - replacement for moment or day.js
encoding - replacement for hand-picked yaml, toml, and other parsers
log and colors - replacement for chalk
testing - replacement for Jest and benchmark.js
uuid - replacement for uuid (the npm package)

When I say "replacement" I don't mean they perform the same thing, but you might not need the entire 3rd party library when you can instead use Deno's versions.

And then of course you've got your standard filesystem utilities

path - like Node's path
fs - like Node's fs
hash - like Node's Crypto library
http - like Node's http

and so on.

A recurring theme in Deno is going to be removing the need for 3rd party tooling, a popular complaint of the modern JS ecosystem. I think this is a much nicer benefit than people give it credit for.

TypeScript

I use TypeScript for just all my JavaScript projects these days. It's more about type completion and IDE intellisense than anything else. Refactoring code is much much easier and I don't have to keep tabbing between documentation and my code editor.

Deno includes TS compilation out of the box. It claims to be a runtime for TypeScript, but it's more like a compiler with some Deno integration. You can import a TypeScript file directly, it's rather quick at compiling, and it doesn't produce any JavaScript files as output, as it's unnecessary.

Not needing to set up your whole project with ts-node is a big win for people upset with boilerplate and configuration.

It does feel like an unusual choice though, since TypeScript still feels "unofficial". It feels like as much of a necessity as something like Flow. I much prefer TypeScript and TypeScript does see much more use these days, but I do understand why some might see this as a bit controversial.

Deno's API

This is not the same as its standard library as these don't need to be imported and are generally considered stable (more on that later). Here are 2 fantastic ones:

FormData - a web API for sending forms instead of JSON-encoded data
- Required for Discord Bots when uploading files, usually implemented with third party libraries in Node
fetch - a web API for sending server requests.
- Node requires 3rd party libraries. There are several to choose from with different strengths.

I've done a fair amount of web testing and scraping with Deno. Having these 2 built in has been so much nicer than Node.js where the implementations are up to the library creators and may not follow how the web APIs work.

Deno also defaults to promise-based operations instead of callbacks. In Node you will often need to promisify libraries or APIs in order to not slip into callback hell. With Deno, Iterables and promises are defaults, so there's no need for glue code anymore!

The Good

Those are my favorites. Now onto the "good" - the things that are really nice to have, but not as important to me as the aforementioned materials.

Deno's Multi-Purpose Binary

Deno's binary doesn't just run code. It also provides the ability to install Deno projects, inspect a Deno project, bundle code, generate documentation, and format.

installer - install Deno projects as CLI tools, like npm i -g
formatter - format code like Prettier according to predefined rules
bundler - collects all code into a single file for use, even inlines imports for use in web environments
doc generator - emits documentation using JSDoc to stdout or JSON to be used with a documentation displayer.
info - shows dependencies
watcher - mentioned in passing, it is an argument that arguably replaces nodemon

A lot of these were again possible with third-party tools, but having an opinionated, built-in way to do all this is super useful. The doc generator could be better, but is much better than nothing for all the people creating Deno libraries.

The formatter is opinionated, so it lets many Deno projects maintain a consistent style. There are some aspects to it that I would change if I could, but I'm sure others think similarly about parts I would hate to be changed, so it's a compromise.

The installer really cements my opinion of Deno's use case which I will again get into later.

No Package.json

This can also be seen as a negative, but it's again one of the usual complaints against Node. The package.json file is often unwieldy and hard to read for some. It includes metadata about the package, custom scripts, and dependencies split into different categories. It does a lot at once, so Deno just gets rid of it.

There is no one file to view all the dependencies, but that's what deno info is for. The less configuration a project has, often the easier it is to read, so that's a nice plus. With no package.json, package-lock.json, or node_modules/, the root directory is often a bit cleaner than most Node projects.

Import Maps

One of the "bad" parts of Deno is that imports are always URLs or paths, never referencing some locally installed package directory, like node_modules. Deno caches imports, so this isn't extraordinarily problematic, but it does make imports a big ole mess.

We've gotten rid of both the package.json and package.lock files, but to remedy this, people usually end up making somewhat of an exports.ts file. This file imports everything from URLs and then re-exports it for the rest of the package to use. This feels like yet another package.json, so Deno also includes import maps.

Import maps let you alias imports and are something seen usually only with build tools like Rollup and Webpack. So now instead of looking at:

import { readText } from 'http://packages.example.org/deno/cool-coder55/CoolLib/src/utils';

We can instead have our imports looking like:

import { readText } from 'CoolLib/src/utils';

I've taken advantage of this a lot. Sometimes URLs are also versioned, so this allows us to update the version of a library by changing it in one place instead of every URL in our project one-by-one.

Security

It might seem a bit weird to list this one last, but it's not something I think is as effective as it seems.

Deno forces users to also list the permissions they grant a program when installing projects or running files. This can be things like network ability, write access, read access, and environment variable access.

This is usually one of the first things about Deno mentioned and I do think it makes sense as a default. It's sort of like immutability as a default in a language, you'd rather choose times when things should have access then try to restrict things after. I just don't think it has quite the effect that it feels it would in practice.

The OSS scene is usually vetted for malicious code, so people are unlikely to check exactly why a program requires certain permissions. In fact, I often see people run files with -A which automatically grants all permissions, rendering this precaution null and void. However, I do see cases where this is useful.

The Bad

With all the good over, we can now get into the frustrations. This section is not as large as the previous 2 combined, but it arguably has greater weight when deciding whether or not to adapt Deno for your projects. You may have been able to pick out some of this section from how I phrased things in earlier sections.

Immaturity

What's fairly obvious is Deno is new. like really new. v1.0.0 was released in May of this year (2020) and it still feels far from 1.0.0 when looked at as a whole.

Deno still has no Arm32 builds which means that hosting scripts on a Raspberry pi Zero, 1, or 2 is not yet possible. I unfortunately produced a Deno Discord bot using Discordeno before finding this out and I have since rented a VPS.

Deno has not had ICU support for a while, which means all JavaScript unicode-aware functions and properties will not work. This includes the u RegExp flag and String.prototype.normalize(). It was initially rejected due to the proportionally massive bump in file size this would force, but recent discussion has shown that they're making progress with its integration.

There is clearly a problem with npm interop. There has been some recent work towards that goal, but it's nowhere near ready. There are ways to import web-libraries or node libraries that don't use any Node APIs, but if your png-to-jpeg Node library even makes a single call to Node's API, that package isn't getting in your Deno project without a repository fork.

There are still a few web APIs that Deno lacks. Node has some third party libraries to fill in the gaps, but these cannot be imported by Deno, so we're stuck without any option in that area. Sometimes the problem is even more fundamental than it just using Node APIs, like with node-canvas using Cairo. I am watching this issue at the moment, though.

I don't know of any build tooling, so if you have a particular feature you'd like to add onto JavaScript, like glob imports, good luck. This is theoretically possible to implement, but I've yet to see it in practice and I think many users of Deno would be against it philosophically.

Deno has some support in code editors and debugging environments, but because it's a small player and it piggybacks off of the work done for Node.js, its debugger in VS Code has broke due to some update, which forced me to roll back my VS Code version.

Additionally, I've found bugs in Deno itself. This isn't much for a library, but when it affects crucial things like text rendering in console and leads to non-actionable error states, it surely presents a hurdle for anyone with less experience who wouldn't know where to turn.

In general, I don't think I would have faired so well if I didn't have years of experience with other frustrating dev environments and the know-how of who and where to ask for help.

Breaking Changes

Oh, and on the topic of it not feeling like 1.0.0? The standard library seems to be far from 1.0.0 these days.

The standard library of course being the thing containing many very important utility functions, like basic file reading and parsing. Really interesting new libraries get added often and this means the entire library's stability is held back by new content. Since, according to the semver spec, semantic versioning doesn't need to be respected before 1.0.0, the entire standard library is unstable. Since most Deno scripts use the standard library, you cannot guarantee that you're able to run an older Deno script without rolling back your Deno version. The version of Deno isn't listed anywhere in a project, so you just have to guess.

A lot of very important features are locked behind an --unstable flag. With Deno, unlike some other languages/projects, this often means breaking changes are more likely than not. An awful lot is still behind stable, so it's rare to find a project that doesn't require it if it ever interacts with the file system.

--unstable also doesn't let you opt into specific features. Recently, Deno forced projects to use import type when importing types or your code would not run. This also affects libraries you import, so you cannot use modern Deno features anymore with older libraries. This requirement is theoretically going to be pushed into stable Deno, breaking its stability (unless they push the version to 2.0.0 by then).

The tsConfig.json can be edited, but most features cannot. Once you edit one, you must provide the defaults for everything else as well. Some features cannot be edited at all and this aspect wasn't extraordinarily clear, making the breaking change a very frustrating experience.

The Lack of Scripts

With the removal of package.json, we can no longer bake a bunch of custom scripts into the project with short aliases. With Deno, the command to run a program can be rather long. Assuming the program has yet to be installed globally, you might be using:

 deno run --allow-net --allow-read=/usr --allow-write=/usr mod.ts -q Stephen Fry -n funny -r 10 -p 10

And yes, this is very similar to a script from a project of mine. Everything after mod.ts is from my own project and so mostly my own doing. Luckily I can just press UP on my keyboard to re-execute it, but this makes managing multiple scripts a bit messy.

Some projects recommend including some sort of run.sh file when the same commands need to be run again fairly often. This ends up defeating the benefit of a cleaner root directory, but it does provide a simplification over package.json since the file only does one thing. This does require another file per script. We could probably add a scripts.toml file and then add a make a nushell or bash script to read it and execute the desired script, but these are again things not native to Deno.

Admittedly, the most common use cases for differing scripts are testing, transpiling TS, bundling, and file watchers. Deno includes all of those natively. One might use a bundler for more than what Deno provides, however, such as supporting custom JS features through a second stage of traspilation.

Conclusion - The Use Case

Deno is not production ready. It has passed 1.0.0, but it has not "passed 1.0". By "production ready" I mean don't build something if you're hoping for stability.

I'm a big fan of "new software" in general. Maybe that's a naive quality. I find that newer software has the advantage of creating much more sensible defaults based on past experiences. It is able to learn from other frustrations and break convention. Languages like Rust and Nim and now Deno show this well.

Even Deno libraries are often "better" versions of Node libraries that had some clear faults. It often takes some getting used to, but you end up with a better product (either in DX or performance).

Deno's main use case seems to be with small scripts. As long as these scripts are bundled using deno bundle and a Deno compatibility version is listed, it should be fine. It will be rather frustrating to switch Deno versions each time you want to run a script, however.

I don't live in the CLI myself, but due to Deno's match with Web APIs, I've found web scraping and http requests to be much nicer with Deno. There should be some advancement on that end as well if Deno ever decides to add DOMParser.

I'm going to continue using Deno for basic scripts and web scraping. I've worked through enough bugs and frustrations that I think I'll be able to avoid them to some extent. It's much easier to set up a Deno project than a Node.js one which makes prototyping much nicer. I think this is a use case most could get behind.

Decoding UTF-8

EmNudge — Wed, 09 Sep 2020 14:41:44 +0000

I'm a JavaScript guy, so this is a bit of a weird article. I recently went through a month's worth of research into unicode for an article and then subsequent video on variable names. JavaScript uses UTF-16 and I put some more specific info into the extra video, rather than the main video.

I'd advise watching it if you're curious. I was recently asked about UTF-8 by another JavaScript developer and so I decided to look into it. Resources online are scarce or not friendly enough that I was asked to turn my personal explanation into an article for future reference.

First off, why UTF-8?

Encoding Unicode

Every unicode encoding is tasked with a similar problem:

How do we represent all 1.1M possible code points?

The actual number is 1_111_998. In binary this takes up 21 bits. That means it's possible to have a single character that must require us to use 21 bits at a minimum to encode.

We stick to powers of 2 for binary encoding, so our first thought might be exactly what UTF-32 is today. Let's use 32 bits!

This works, but is very inefficient. If we approach a character that requires 21 bits, we'll need to use 4 bytes (32 bits) anyway, but most characters do not have a codepoint that large. This means we are wasting the first 11 bits of every codepoint, at a minimum. In fact, this article only uses 3 characters larger than 122 (the 3 example characters later on in the article)! That's (for the most part) only 7 bits per character!

In comes UTF-16 with a clever trick.

UTF-16

It recognizes that characters with codepoints that require more than 16 bits are even more rare! 16 bits gives us roughly 65k possible codepoints. What we do is reserve 2 separate ranges of 1024 characters out of our initial 65k codepoints available. These are used for "surrogate pairs" (more info my unicode extras video). Using a bit of math we find that if we combine them, we're able to get 20 more bits (enabling 1_048_576 more characters).

If we go by character by character, as computers usually do, we're able to tell whether the codepoint we are looking at is a lone one or a surrogate pair just by checking what range it is in!

This way we can shave off 16-bits for most characters! The surrogate pair ranges are purposely empty anyway, so we're not kicking any characters out.

While JavaScript uses UTF-16, HTML traditionally uses UTF-8. Only 8 bits? It looks like we barely got by with 16 bits! Are we going to reserve another few ranges? How? Out of 256 possible codepoints, which ranges do we reserve?!

We could use multiple bytes for larger codepoints, but how would we ever know if we were currently looking at a lone character or one that is meant to be read together with others? UTF-16 and UTF-32 gives us the guarantee that we'll be able to start from anywhere in a file and, provided we know our index, we can regain our footing.

UTF-8

And now the point of the article. UTF-8 takes a different approach. Instead of reserving a specific range, each starting byte will tell us how many more bytes to expect. The answer to this can be either 0, 1, 2, or 3, since UTF-8 characters can be composed of anywhere from 1 to 4 characters.

Our base case is 0.

If we're given a byte that begins with a 0, then this is the only codepoint required to parse this into a character. Using the codepoint of 103 as an example (the character g), its UTF-8 encoding would be 01100111. Notice the leading 0. We can express this codepoint with 7 bits, so we use the leading bit to tell whomever is asking that this is the only number in this sequence. Only one byte!

This works fine up until codepoints over 127. In such a case we are forced to use 2 bytes.

We need to signal this somehow. UTF-8 tells us to start the first byte with 110. The second byte must start with 10. Since each byte is 8 bits, we're left with (8 - 3) + (8 - 2) = 11 total bits! This allows us to express codepoints in the range 128-2047.

Therefore the British Pound (£), with a codepoint of 163, requires 2 bytes. Its direct binary representation is 10100011. Expanding it to the full 11 bits turns it to 00010100011. Combining the required 110 with the first 5 bits gives us 11000010 for our first bit. Our second bit then combines 10 with our remaining 6 bits (100011) gives us 10100011.

But again, this only covers the first 2047 characters. For larger codepoints (up until 65,536) we need 3 bytes.

This time we use 1110 (notice an extra 1) at the start and follow it up by 2 numbers which both start with 10. This is useful for codepoints such as ￫ which have a codepoint of 65_515. Since this time it leaves us with (8 - 4) + 2(8 - 2) = 16 bits to express 63,488 more characters!

And finally we get to the remaining characters.

For characters such as 😀, with a codepoint of 128_512, we can use a leading 11110 (another 1) to signal that this is a part of a 4 byte codepoint. The next 3 characters again start with 10. Applying our basic math expression, we find this leaves us with (8 - 5) + 3(8 - 2) = 21 bits! Exactly enough to express all unicode codepoints!

The Algorithm

Now that we've gone through the explanation, let's make it concise with some steps. Let's say we're placed at a random byte in a file which we know to be UTF-8 encoded.

Does this byte begin with a 0? If so this can be parsed by itself and we can move on to the next byte.
Does this byte begin with 10? Whoops! If so, we're in the middle of a character! We need to move forward until this is no longer the case or move backwards to find out what sort of character we were trying to parse.
Does this byte begin with 2-4 1s followed up by a 0? If so, provided n refers to the number of 1s, we've approached an n-byte codepoint and should parse the next n - 1 bytes and this one as a single codepoint.
Does this byte begin with 5 or more 1s? This is invalid UTF-8! Well, maybe. Perhaps in the future we have somehow expanded to ~69 Billion codepoints.

The Code

Because we're in binary-land, we also have quite a few operators at our disposal to make things more efficient - bitwise operators. They're a bit unintuitive at first, so I suggest going through them at first to fully understand what they do before seeing how they're used.

I've placed the code for my UTF-8 encoder/decoder into a Svelte REPL over here: https://svelte.dev/repl/9d89e50badbd458599fc62cde67fc9b3?version=3.24.1

I won't be going over the majority of the code, since it'd be a bit too much for this article (and a bit too boring, frankly), so let's just clarify 2 common operations that use bitwise operators in the code.

Retrieving Part of a Binary
- To isolate a binary, we create another binary with the bits we want to preserve and use the & operator - make every bit you want to keep a 1. If it starts in the middle, we can use the >> operator to move it back to the start.
- 0b1111 & myBinary = isolate the first 4 bits
- (0b111100 & myBinary) >> 2 = isolate the first 4 bits after the first 2 bits
Adding on Bits to a Binary
- We cannot add on without knowing the binary length. If we do, just create a binary with n digits + however many the binary is and use the | operator. Make the rest of the bits (first n digits) all 0.
- 0b10000 | myBinary = if binary is 3 digits long, adds on 10 to the end.
- 0b10000000 | ((myBinary >> 6) & 0b111111) = take the first 6 bits and add on 10 to the end.

Conclusion - Efficiency

From the looks of it, it seems like UTF-8 is the most efficient encoding so far. However, there are clearly some edge cases where it would be much less efficient than something like UTF-16.

As a closing, here are 2 quick tests:

On something like Herman Melville's Moby Dick in English, UTF-16 increases the file size just about twofold, from 1.19MB to 2.39MB as the majority of its characters (from the text file I found) would be within the first 127 codepoints. That's one byte per character in UTF-8 and 2 in UTF-16). UTF-32 would no doubt have doubled it yet again.
On Dan Abramov's My Decade In Review in Simplified Chinese, UTF-16 is slightly smaller than the UTF-8 variant (28.8kb vs 36kb). Many of the codepoints are in the ~25k range. This would be 3 bytes in UTF-8 and only 2 in UTF-16 - a roughly 1.5x increase. The markdown file also includes many links and single byte UTF-8 characters, which is why it isn't entirely 1.5x.

Without looking anything up, we can assert that UTF-8 would be less efficient than UTF-16 for characters with codepoints between 2048-65534. This would favor UTF-16 when such codepoints dominate the medium, like a book rather than a blogpost, where the benefit is only marginal due to the quantity of single-byte characters that UTF-16 forces to be 2 bytes.

UTF-16 would almost always be more space-efficient than UTF-32 unless the Unicode standard opened up codepoints that could be expressed with 22-32 bits. In which case such codepoints would be better expressed with UTF-32.

For most purposes, particularly code in just about any programming language, UTF-8 is more efficient.

In regards to computational efficiency, the efficiency order would go in reverse - UTF-32, UTF-16, and then UTF-8.

Lossless compression formats such as .zip and .72 would also likely make the size differences much thinner (or perhaps even reverse the current order). Specially tailored compression formats even more so. While these encoders work on character to character, other compression formats may be able to work on text blobs as a whole.

Critique? Let me know in the comments!

From Scratch - Reactive Values

EmNudge — Mon, 10 Aug 2020 17:58:08 +0000

Note: This was originally a script for a video. As it takes months at times to release a video, I've decided to turn what I have into an article as well.

I'm a fairly big proponent of the idea that certain programming topics are best taught by discussing the low level fundamentals rather than the high level API. In the same way that Dan Abramov teaches Redux or Francis Stokes teaches just about anything.
In this article we're going to discuss a fairly hip JavaScript topic: Reactive Data Structures. Let's first get into a use case.

The Problem

For the most basic of use cases, let's try to share data between modules. Perhaps we're creating a game and we want our score to be able to be changed via multiple different modules containing their own functions.

For this purpose we usually create somewhat of a function hierarchy (see React's Data Flow), but this may require us to change our main function when we want a change in a smaller function. It also leads to highly nested code with data being passed through multiple levels for simple updates (known in React as Prop Drilling). So we're not going to go with that method.

Frameworks like Solid.js and Svelte.js solve this problem using Reactive Data Structures, often called Stores or Signals. Other frameworks may have slightly differing approaches, like React's Context and Vue's Vuex. We're going to implement the Solid/Svelte approach without using the framework.

Let's set up our code. We'll store all data, such as our score, in a file called data.js. Our main file, index.js, will be responsible for taking the score and displaying it, as well as importing the buttons.js file which contains the code for our buttons.

We could just create another script tag instead of an import, but I prefer this method.

Code below available at: https://codesandbox.io/s/reactor-p1-nu3ik

├── index.html
├── index.js
├── buttons.js
└── data.js

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Reactor Example</title>
    <meta charset="UTF-8" />
    <script type="module" src="index.js"></script>
  </head>
  <body>
    <h1 class="score">0</h1>
    <button class="score-increase">Increase Score</button>
    <button class="score-decrease">Decrease Score</button>
  </body>
</html>

// index.js
import './buttons.js';
import { score } from './data.js';

const h1 = document.querySelector('h1.score');
h1.textContent = `score: ${score}`;

Our first instinct here is just to export a variable called score that points to a number.

// data.js
export const score = 0;

// buttons.js
import { score } from './data.js';

const b1 = document.querySelector('button.score-increase');
b1.addEventListener('click', () => score++);

const b2 = document.querySelector('button.score-decrease');
b2.addEventListener('click', () => score--);

We're unfortunately going to run into a problem immediately. We cannot assign to any imported variables. They're defined as constant binding values when imported. Changing it to let won't help either, as it will only be mutable to the module it's exported from.

One option might be to use export let and also export a changeScore function which should have edit access. There's a simpler solution, however.

Using Objects

As with all constant variables in JavaScript, we actually can change its properties if it's an object. Moving score to an object with a value property is an easy fix there.

Code below available at: https://codesandbox.io/s/reactor-p2-5obug

// data.js
export const score = { value: 0 };

// buttons.js

// ...
b1.addEventListener('click', () => score.value++);
// ...
b2.addEventListener('click', () => score.value--);
// ...

// index.js

// ...
h1.textContent = `score: ${score.value}`;

Now this actually works. Our value is changed and the changes carries over from module to module. We're not seeing any change visually, however. When we click our buttons, the h1 does not update.

This is because our code in index.js is only ran once. It has no idea when our data has changed. We can probably start an interval which sets our value ever few milliseconds, but this really isn't a viable option for everywhere that we end up using our score.

A better alternative is to have our score tell everyone when its value changes. Like a newspaper, we can give people the option to subscribe and we'll notify them when we get a new issue... or value.

Subscribers

This requires us to know when we've been mutated. We usually use functions for this thing, but we can preserve using .value by turning our object into a class and creating getters and setters.

Note that, with the exception of Vue.js and a few others, this isn't often how reactivity libs work - we often just use functions for updating. For this article, I prefer the OOP method as it cuts down on some code complexity. We don't need a separate read, set, and update method (update takes a function, whereas set only takes a value). I advise you to look up getters and setters in JS, however, if you're unfamiliar.

Code below available at: https://codesandbox.io/s/reactor-p3-e8dxg

// reactor.js

export class Reactor {
  constructor(value) {
    // private value for where it's really stored
    this._val = value;
    // private list of functions to be notified
    this._subscribers = [];
  }

  // return value when requested
  get value() {
    return this._val;
  }
  // set value and then notify everyone
  set value(newVal) {
    this._val = newVal;
    for (const subscribeFunc of this._subscribers) {
      subscribeFunc(newVal);
    }
  }

  // add function to subscriber list and immediately invoke
  subscribe(func) {
    this._subscribers.push(func);
    func(this._val);
  }
}

One way that we differ from a newspaper is that subscribers get a value instantly upon subscription. This lets our score counter work without having to set it an additional time right before subscribing, but it's also important to keep this in mind for a feature we're going to add later.

// data.js
import { Reactor } from "./reactor.js";

export const score = new Reactor(0);

// index.js

// ...
score.subscribe(val => {
    h1.textContent = `score: ${val}`;
});

At this point we've already created a reactive data structure. The fact that this reacts to changes and updates its subscribers is the reactivity we've been looking for. We can have one reactive value update another reactive value and create chains of reactivity.

const score = new Reactor(0);
const halfScore = new Reactor(0);
score.subscribe(val => halfScore.value = val/2);

One thing we can't really do as easily though is have one value change in response to any of multiple values changing. What if we want to generate a high score out of multiple reactive scores? We might do something like this:

// example.js
import { Reactor } from './reactor.js';

const scores = new Reactor([]);
const highScore = new Reactor(0);

// finds highest reactive score and changes highScore to it
function setHighScore(val) {
    // we use this for scores as well, so check if it's a number
    let highestNum = typeof val === "number" ? val : 0;

    for (const score of scores.value) {
        if (score.value <= highestNum) continue;
        highestNum = score.value;
    }

    highScore.value = highestNum;
}

// adds new score and makes it reactive when changed
function addScore(num = 0) {
    const score = new Reactor(num);
    score.subscribe(setHighScore);
    // we cannot use .push() - we need to use = for it to react
    scores.value = [...scores.value, score];
}

addScore(0);
addScore(45);
addScore(26);

This looks a bit messier than I'd like it to. We're forced to have our addScore also subscribe each score individually. Since our subscribe function is called immediately, we're also updating the highScore when add add a new one, but if we added one any other way, it wouldn't update the high score.

Computed Values

There's a cleaner way - computed values. At the cost of more complex library code, we get a cleaner user experience. Here's what a computed version of that code might look like.

import { Reactor, computed } from './reactor.js';

const scores = new Reactor([]);
const highScore = computed(() => {
    let highestVal = 0;

    for (const score of scores.value) {
        if (score.value <= highestVal) continue;
        highestVal = score.value;
    }

    return highestVal;
});
highsScore.subscribe(num => console.log('high score: ' + num));
// high score: 0

scores.value = [new Reactor(0)];
// high score: 0

scores.value = [...scores.value, new Reactor(45)];
// high score: 45

scores.value = [...scores.value, new Reactor(26)];
// high score: 45

const firstScore = scores.value[0];
firstScore.value = 103;
// high score: 103

I'm not sure if we're all looking at the same code here, but this looks like magic to me.

Our high score will change whenever a new value is added or when any value inside of it changes its own value.

...how?

We're not subscribing to anything. How does the computed function know about which variables are inside of it? We're not stringifying anything and we're not doing static analysis. We're using an array, so there aren't any unique variable names. Is this something specifically with arrays?

Nope! Here's a sample with some other values:

import { Reactor, computed } from './reactor.js';

const num1 = new Reactor(45);
const num2 = new Reactor(92);
const unusedVal = new Reactor(34);

const num4 = computed(() => num1.value + num2.value);
num4.subscribe(num => console.log('num4: ' + num));
// num4: 137

num1.value = 8;
// num4: 100

num2.value = 2;
// num4: 10

unusedVal.value = 17;
// num4 is unchanged and doesn't console.log since we never used unusedVal for num4

A computed value is like a regular subscription, but it allows us to subscribe, dynamically, to multiple values. It knows exactly which reactive variables are inside it and only has them specifically subscribed.

This seems impossible unless computed and Reactor are communicating in some way. They're separate, but they must share some sort of local state or else there's no way this is possible.

And that's right on the mark. The trick to all of this working is the following:

We automatically run subscriptions once after subscribing.
There is a single (non-exported, but top-level) variable in the same module as both computed and Reactor that may or may not have a value at any given time.

The Trick

So computed is able to communicate with Reactor by the following method:

Set our local variable (computeFunc) to the function passed to computed.
Run the function passed to computed once.
Have Reactor values automatically subscribe to computeFunc when they're read from and computeFunc is not empty.
Set computeFunc back to what it was before.

This way, we're able to communicate with all reactive values in the function without knowing specifically what they are, since it's the job of the reactive values themselves to check this variable.

To reiterate, since this is perhaps the most complex part of this article - both computed and Reactor have computeFunc in scope. computeFunc is usually empty. As JS, in this context, is single threaded, the only time it ever contains a value is exactly when computed initially runs. This way we're ensuring that every Reactor inside the function passed to computed subscribes to this function. If we did not set computeFunc back to what it was before (usually undefined), then every reactive value would subscribe to it - even ones not related to any computed.

We set it back to "what it was before" and not undefined because computed values can contain computed values. This means we may be getting deep into some stack and since every computed uses the same variable, computeFunc, we need to set it back to was before, as it may have not been undefined, but just some other function.

That was a lot of talk and perhaps it may be clearer in code. A computed value is just a regular Reactor, so let's set that up first.

// reactor.js

export function computed(func) {
    // we can give it anything, since we're changing it momentarily
    const reactor = new Reactor(null);

    // run it immediately to get a new value
    reactor.value = func();

    return reactor;
}

// ...

This doesn't look like much yet. Let's add our local variable and change Reactor to check for it.

Code below available at: https://codesandbox.io/s/reactor-p4-1tcij?file=/reactor.js

// reactor.js

// initially undefined. We can set it to null instead. 
let computeFunc;

export function computed(func) {
    const reactor = new Reactor(null);

    // THIS is the function we subscribe to, which updates the reactor
    const fn = () => reactor.value = func();

    // set computeFunc to fn and store previous value for later
    const prevVal = computeFunc;
    computeFunc = fn;

    fn();

    // set computeFunc back to previous value
    computeFunc = prevVal;

    return reactor;
}

export class Reactor {
    // ...

    get value() {
        // If it exists, we add it to the subscribers.
        // Do not call it, unlike a regular subscriber.
        if (computeFunc) this._subscribers.push(computeFunc);

        return this._val;
    }

    // ...
}

And now computed works! We can create new reactive values from other ones.

We're not quite done yet, however. We'll find that our array example does not work yet. This is because our computed function does not account for dynamically added values.

Accounting For Arrays & Cleanup

We're only setting computeFunc on the initial function creation, so only the Reactors that are inside the computeFunc on initial creation will subscribe to fn. With our array example, we're adding reactive values even after computed is initially called. We need to change fn to account for that.

Code below available at: https://codesandbox.io/s/reactor-p5-cdx10?file=/reactor.js

export function computed(func) {
    const reactor = new Reactor(null);

    // move the local variable assignment into the subcribed function
    const fn = () => {
        const prevVal = computeFunc;
        computeFunc = fn;

        reactor.value = func();

        computeFunc = prevVal;
    };

    fn();

    return reactor;
}

The problem with this is that we're now going to run into an infinite loop. Whenever a reactive value in the computed is changed, we loop through our subscribed functions and call them.

Then the function we're subscribing to is setting ComputeFunc and calling our get value method. Doing that forces us to add a subscriber to ourself. We're adding a subscriber while looping through subscribers, so we always have another subscriber to loop over. Thus, an infinite loop.

A quick solution is making sure we have no duplicates of any functions in our array. Move our array to a new Set().

export class Reactor {
  constructor(value) {
    // ...
    this._subscribers = new Set();
  }

  get value() {
        // change from .push() to .add()
    if (computeFunc) this._subscribers.add(computeFunc);
    // ...
  }

  subscribe(func) {
    this._subscribers.add(func);
    // ...
  }
}

At this point we may want to add some more cleanup code. Different reactive libs have different sort of safe guards and differing ways to do similar things. We may want to firstly add an unsubscribe function, which is usually just returned from the subscribe function.

subscribe(func) {
  this._subscribers.add(func);
  func(this._val);

  // remove the subscriber
  return () => this._subscribers.delete(func);
}

Using Set makes this process super clean.

We also may want to add some infinite loop protection. That can be done by checking if the function we're in (fn) is equal to computeFunc.

if (fn === computeFunc) {
  throw Error("Circular computation detcted");
}

Now doing the following throws an error instead of lagging the page until your tab crashes:

import { Reactor, computed } from './reactor.js';

const num1 = new Reactor(0);

// ERROR: Circular computation detected
const num2 = computed(() => {
    num1.value++;
    return num1.value + 1;
});

Practical Application - Mini Framework

At this point I was going to see if I could describe how RxJs's approach differs from ours. Instead I think I'm going to show how we can turn our library into a mini framework, to illustrate the effectiveness of this approach.

We often want frameworks to be fairly reactive - where changes to variables are reflected in the DOM and vice versa. Our reactive system is perfect for this.

Code below available at: https://codesandbox.io/s/reactor-p6-ynq3h

import { Reactor, computed } from './reactor.js';
import { get, create } from './framework.js';

const num1 = new Reactor(0);
const num2 = new Reactor(0);
const total = computed(() => num1.value + num2.value);

const inputOptions = {
  rejectOn: isNaN,
  mutator: Number, 
};

const input1 = create('input')
  .bind('value', num1, inputOptions);

const input2 = create('input')
  .bind('value', num2, inputOptions);

const span = create('span')
  .bind('textContent', total);

get('body')
  .append(input1)
  .append(' + ')
  .append(input2)
  .append(' = ')
  .append(span);

Our framework exposes 2 functions - get and create which wrap HTMLElements in a class called El. This class exposes the methods bind, append, and on. With simple rules, we can create a 2-way binding between our reactive values and input elements.

get simply uses document.querySelector(). create is a simple call to document.createElement(). on is .addEventListener() and append is .appendChild().

bind is the interesting one here.

bind(name, funcOrReactor, options = {}) {
    // grab reactor from function, if it isn't a reactor
  const reactor = funcOrReactor instanceof Reactor ? funcOrReactor : computed(funcOrReactor);

    // if editing value, apply 2-way  binding
  if (name === 'value') {
    this.on('input', e => {
      const val = options.mutator ? options.mutator(e.target.value) : e.target.value;
      if (options.rejectOn && options.rejectOn(val)) return;
      reactor.value = val; 
    });

        // change property when reactive value changes
    reactor.subscribe(val => this._el[name] = val);
  } else if (name === 'textContent') {
    reactor.subscribe(val => this._el[name] = val);
  } else {
        // if not textContent or value, it's probably an attribute
    reactor.subscribe(val => this._el.setAttribute(name, val));
  }

    // allow method to be chained
  return this;
}

bind just adds a subscription unless the name is value in which case it also tries to change the reactive value with an eventListener. In such a case, we can mutate the value and possibly prevent updates with rejectOn. Here we're using it to prevent non-valid numbers from getting in our reactive values.

Conclusion

I hope you learned a bit from this walk through. Special thanks to Ryan Carniato and Jimmy Breck-McKye who were instrumental in my understanding of all of this. I ended up rewriting Jimmy's library to fully understand some concepts. You can see that here if you'd like to improve your understanding of some concepts.

If you're up to it, let me know what you liked and didn't, so that I can improve my technical writing for future publications!

Invisible Unicode - Coding Deep Dive Episode #1

EmNudge — Tue, 04 Aug 2020 00:52:49 +0000

This is an article I first published on Dev.to in March of this year over here: https://dev.to/emnudge/adventures-in-horrible-no-good-very-bad-variable-names-20ii

I figured that the way the article was formatted might work better as a video, so I wrote a script, did some more research, recorded it, and made it into a video. Took about 5 months, but here it is!

Adventures in Horrible No Good Very Bad Variable Names

EmNudge — Mon, 16 Mar 2020 18:21:02 +0000

Recently I got a DM on Discord. This person did not have much knowledge of JavaScript, but they had seen this rather interesting snippet of JS which affected tweets on Twitter (now deleted). It changes a couple of very specific tweets, revealing text that wasn't previously there.

They had run this JavaScript snippet in their dev console and wanted me to explain how it worked. For future reference, if you do not fully understand a JavaScript snippet, please do not do this. They can be malicious.

How did this work? The tweets contained text that wasn't viewable by most font-sets. Many times these icons will show up as missing symbol boxes (▯ or □). These characters simply do not show at all.

The JS snippet got the UTF code point for each character using String.prototype.codePointAt() and then converted it into an english character using String.fromCodePoint().

These special characters are ones returned from String.fromCodePoint() when passed the first 4096 numbers starting from 0xe0000 (917504). You can "see" all of them by running the following:

for (let i = 0; i < 4096; i++) {
    const char = String.fromCodePoint(i + 0xe0000);
    console.log(char);
}

Most dev tools will combine console logs into one output if they contain the same text. As these are distinct symbols, they will appear as 4096 separate logs.

As they are distinct symbols, they do indeed contain length. In fact, we could probably artifically increase this article's "reading length" by filling it with these symbols. In between these two arrows are 100 characters. You can copy/paste it into dev tools and check its length to confirm.

→󠀀󠀁󠀂󠀃󠀄󠀅󠀆󠀇󠀈󠀉󠀊󠀋󠀌󠀍󠀎󠀏󠀐󠀑󠀒󠀓󠀔󠀕󠀖󠀗󠀘󠀙󠀚󠀛󠀜󠀝󠀞󠀟󠀠󠀡󠀢󠀣󠀤󠀥󠀦󠀧󠀨󠀩󠀪󠀫󠀬󠀭󠀮󠀯󠀰󠀱󠀲󠀳󠀴󠀵󠀶󠀷󠀸󠀹󠀺󠀻󠀼󠀽󠀾󠀿󠁀󠁁󠁂󠁃󠁄󠁅󠁆󠁇󠁈󠁉󠁊󠁋󠁌󠁍󠁎󠁏󠁐󠁑󠁒󠁓󠁔󠁕󠁖󠁗󠁘󠁙󠁚󠁛󠁜󠁝󠁞󠁟󠁠󠁡󠁢󠁣←

Note that using String.prototype.length will actually print a length of 202 instead of the expected 102 (almost double) because every character after 0xFFFF (65,535) (called the BMP or Basic Multilingual Plane) exceeds the limit of JavaScript's UTF-16 strings. The arrows (in order to be displayed on smaller font sets) has a code point of 0x2192 (8,594). To actually retrieve the number of characters in a string, use a for...of loop and take advantage of JS iterables!

function getNumChars(str) {
    let length = 0;
    for (const char of str) length++;
    return length;
}

or, since the spread operator also works on iterables, a bit of a simpler method:

[...str].length

In general, the intricacies of all of this is a bit more than what I'd like to get into. Mathias Bynens has a fantastic article on all this, which I highly advise you read for more information.

You can quickly view a character's code point via the following function:

function getCharHexCode(char) {
    const hex = char.codePointAt(0).toString(16).toUpperCase();
    return `0x${hex}`;
}

"Astral code points" (ones after 0xFFFF, such as 🡆) also contain a second index. It will return a code point that is relational to the actual code point defined by the following expression:

codePoint + 1024 * Math.floor((codePoint - 0xFFFF) / 1024) + 9216

or the following function

function getAssociatedCodePoint(codePoint) {
    if (codePoint <= 0xFFFF) return null;

    // This is just here. No clue.
    const BASE_DIFF = 9216;

    // how many sets of 1024 numbers this is above 0xFFFF
    const index = Math.floor((codePoint - 0xFFFF) / 1024);
    // for every 1024 numbers, the difference jumps 1024
    const levelNum = 1024 * index;

    return BASE_DIFF + levelNum + codePoint;
}

I honestly do not know why this is so. Drop a comment if you have an explanation.

6/12/2020 EDIT: It turns out it's just the right surrogate pair.
One would get the same result doing '🡆'.codePointAt(1) as one would doing '🡆'[1].codePointAt(0). codePointAt does not remove the right surrogate pair when retrieving the codepoint, just the left one.
Read more about this stuff at: https://dmitripavlutin.com/what-every-javascript-developer-should-know-about-unicode/#24-surrogate-pairs

While all this might be interesting to some, that wasn't why I wrote this article. I wanted to investigate variable names (hence the title). Could these special characters be used as variable names?

Variable Names And You

Most people stick to standard conventions when making variable names in JS.

Use English characters (no umlauts or diacritics).
Start with $ for jQuery or querySelector-based libraries.
Start with _ for lodash/underscore or unused variable names.

Although these aren't physical limitations, people tend to stick to them. If one developer used diacritics, it would be difficult for developers without specific keyboard layouts to replicate them.

What I'm interested in is what are we physically bound by. Could we use a number literal as a variable name, for instance? No. We are physically bound from doing that.

const 32 = 24;
// Uncaught SyntaxError: Unexpected number
const .32 = 24;
// Uncaught SyntaxError: Unexpected number

Some other things we can't use:

reserved keywords
- if, while, let, const, etc
immutable global object properties in the global scope
- NaN, Infinity, and undefined
variable names starting with unicode outside of the Unicode derived core property ID_Start (excluding $ and _).

Thanks again to Mathias Bynens for this info

Mathias also provided an online JavaScript variable name validator for if you would like to test things out yourself.

One thing to note is that there is a difference in valid variable names for ES5, ES5-era engines, and ES6. We are using ES6.
Mathias (yet again) has an article for this.

What I was interested in was the odd stuff. A theoretical prank.

The Theoretical Prank

Every now and again this "meme" floats around where it advises pranking a coworker by replacing their semicolons with greek question marks (; or 0x037E).

These days, we have pretty good linters (in most languages) which will catch these. This prank can be found out very quickly. Let's try spicing things up a bit.

What information from our knowledge of valid variable names can we use for our prank?

Well firstly, Zalgo text is fine. Zalgo text is the result of combining a bunch of diacritics to extend text outside of its vertical container. It tends to look like ṱ̶͇̭̖̩̯͚̋͛͗̋h̶̳̳̫͕̄͆̈̍̓̀̉ͅi̷̥̩̼̒̏s̷̰̣̽̇̀͆̀͠ and it's both valid unicode and a valid identifier.

const ṱ̶͇̭̖̩̯͚̋͛͗̋h̶̳̳̫͕̄͆̈̍̓̀̉ͅi̷̥̩̼̒̏s̷̰̣̽̇̀͆̀͠ = 32;

console.log(ṱ̶͇̭̖̩̯͚̋͛͗̋h̶̳̳̫͕̄͆̈̍̓̀̉ͅi̷̥̩̼̒̏s̷̰̣̽̇̀͆̀͠); // > 32

Since diacritics are valid in variable names, there's nothing really stopping us from combining them ad infinitum. This isn't very pleasant to look at, but it's still not what I had in mind for a prank.

We previously discussed invisible characters. What if we could create invisible variable names? Are these valid?

const 󠀀= 42;
// Uncaught SyntaxError: Invalid or unexpected token

It doesn't seem so. And in case you were wondering, there is indeed a character there between const and =. If there wasn't, we would get a separate error.

const = 42;
// Uncaught SyntaxError: Unexpected token '='

We could use the aforementioned tool to check valid variable names, but we'd be entering characters one by one. I need a way to automate this. I can copy Mathias's code, using a ton of regex and all that, or....

function isValidVariableName(str) {
    try {
        eval(`const ${str} = 42;`);
    } catch {
        return false;
    }
    return true;
}

-"eval is evil" but we can make an exception for personal testing. Note that I'm specifically not using let since passing a space to isValidVariableName will return a false-positive if let were used. After all, the following is valid:

let = 42;
console.log(let + 8); // 50

As let along with 8 other words were not considered reserved keywords outside of strict mode.

With that in mind, let's get into a bit of width testing.

Width Testing

I want to find valid variable names with thin, weird characters. The easiest way to do this is via your eyes. Looking at characters is a pretty good way to tell how they look. Unfortunately, this is time-consuming. Especially for possibly over 1 million characters.

Let's set up some test code

function getSmallestChars() {
    const chars = [];
    const min = { width: Infinity, height: Infinity };

    for (let i = 0; i < 0xFFFF; i++) {
        const char = String.fromCodePoint(i);

        if (!isValidVariableName(char)) continue;

        const { width, height  } = getStrSize(char);

        if (width > min.width || height > min.height) continue;

        if (width < min.width && height < min.height) {
            min.width = width;
            min.height = height;
            chars.length = 0;
        }

        chars.push(char);
    }

    return chars;
}

The upper bound of i is just small for the initial test. The important question is how do we find out how much space a character takes up? The question is font-specific and the DOM generally will not give the specific character size, but rather the space the parent TextNode takes up.

For this, we need to use Canvas.

const cnvs = document.createElement('canvas');
const ctx = cnvs.getContext("2d");

function getStrSize(str) {
    const textMetric = ctx.measureText(str);
    const width = textMetric.actualBoundingBoxRight - textMetric.actualBoundingBoxLeft;
    const height = textMetric.actualBoundingBoxAscent - textMetric.actualBoundingBoxDescent;

    return { 
        width: Math.abs(width), 
        height: Math.abs(height) 
    };
}

What you might notice is that we're declaring 2 variables outside of the scope of the function. This is generally bad practice, but this function will be called thousands of times and I want to self-optimize a bit, just in case.

If you have worked with ctx.measureText before, you might also realize I'm not using its returned width property, which should be exactly what I want. Some diacritics actually contain a negative width and the returned width will only go as low as 0. I am calculating it myself to avoid such cases.

You can view the resulting code on JS Fiddle.

The code takes a while to run, but we (at least on my machine) get an array of 3 characters.

Yup. 3 spaces of varying widths. The canvas must have calculated these to be of zero width. Using these spaces, we can make some funky valid code.

const ﾠ= 42;
const ㅤ= 58;
console.log(ﾠ+ㅤ); // 100

I'm excluding one of the spaces as it doesn't show up on some devices (such as Android phones or Windows 10 PC's). The other 2 spaces are known as hangul filler characters. One is a half-width, which is why it's thinner.

As an aside, while this test only ran through UTF-16 characters, I have done a test involving all unicode characters and gotten the same results.

At this point, we've gotten the 2 characters that ES6 will allow us to start a variable name with, but we haven't explored all the valid variable-naming characters.

As discussed before, a number cannot be at the beginning of a variable name, although it can be anywhere after the first character.

const 1 = 42;
// Uncaught SyntaxError: Unexpected number
const num = 42;
// good
const 1num = 42;
// Uncaught SyntaxError: Invalid or unexpected token
const num1 = 42; 
// good

Our isValidVariableName fails to check for this. We can use the same function, but pass in a valid character as the first symbol to fully test this out. In our code, let's change the following code:

function getSmallestChars() {
    // snip...

    if (!isValidVariableName(char)) continue;

    // snip...
}

function getSmallestChars() {
    // snip...

    if (isValidVariableName(char)) continue;
    if (!isValidVariableName(`h${char}`)) continue;

    // snip...
}

With this code we are automatically skipping over super valid symbols and only keeping ones that are "kinda valid". We are prepending h to the symbol. This way, if it passes, it is valid only after the first character.

Using this change, we get 51 symbols (vs the 3 we originally got).

The newline (↵ or 0x21B5) character is a false-positive. It is not that the newline character is a part of the variable, it is simply getting skipped over. It reads similar to the following:

const h
= 42;

Which, due to how ASI works, is valid code. Although, only h (not h↵) has been set to 42. We need to modify isValidVariableName a bit for this checking.

function isValidVariableName(str, kinda = false) {
    try {
        if (kinda) {
            eval(`
                const h = 42;
                const h${str} = 42;
            `);
        } else {
            eval(`const ${str} = 42;`);
        }
    } catch {
        return false;
    }
    return true;
}

By already defining h before we use the passed string, we can guarantee an error will be thrown if the ASI simply interprets this as whitespace.

Let's also change the previous code to

function getSmallestChars() {
    // snip...

    if (isValidVariableName(char)) continue;
    if (!isValidVariableName(char, true)) continue;

    // snip...
}

Running it we get 27 array elements.

That means 24 of our previously returned symbols were whitespace characters. Here are the 27 hex codes:

const symbols = ["0x34F", "0x5A2", "0x5BA", "0x5C5", "0x17B4", "0x17B5", "0x180B", "0x180C", "0x180D", "0x200C", "0x200D", "0xFE00", "0xFE01", "0xFE02", "0xFE03", "0xFE04", "0xFE05", "0xFE06", "0xFE07", "0xFE08", "0xFE09", "0xFE0A", "0xFE0B", "0xFE0C", "0xFE0D", "0xFE0E", "0xFE0F"]

It's at this point that I might as well mention that I have been doing most of these tests on a MacBook. I switch off between a MacBook and a Windows 10 Desktop PC depending on where I am. Windows 10 comes with a font containing many more unicode characters than other devices (aside for a few Linux distros).

We want our "prank" to affect the majority of users, so we won't be using the larger 119 characters that my Windows machine gave me and only sticking to the 27 that both machines seem to share.

The first 9 characters are viewable on Windows' default font, so we're going to skip to the following 18.

The first 2 characters (0x200C and 0x200D) are zero width joiner/non-joiners. 0x200B, the zero width space (and the one right behind the other 2) was not included. Probably because it's whitespace and not a valid variable name.

The following 16 (from 0xFE00 to 0xFE0F) are variation selectors. There are many more than 16, but the rest are passed 0xFFFF and thus would not come up in our search.

Here are all those characters: →‌‍︀︁︂︃︄︅︆︇︈︉︊︋︌︍︎️←

Running this code with the full extent of unicode doesn't genarate vastly different results. This means our aforementioned invisible tweet characters are not valid variable names. However, Our new characters are.

Put Into Action

We went over a lot. We have 18 non-starting variable characters and 2 starting blank characters. All within UTF-16 (not that it's strictly needed).

Now for the "prank". Let's create a Babel transformer plugin.

module.exports = () => {
  // counts the number of variables we have transformed
  let index = 0;

  return {
    visitor: {
      Identifier({ node }) {
        // convert number to hexidecimal and split into array
        const indexes = index.toString(16).split('');

        // turn each hex into an invisible char
        const chars = indexes.map((hex) => {
          const codePoint = 0xfe00 + parseInt(hex, 16);
          return String.fromCodePoint(codePoint);
        });

        // add these chars onto the node name
        node.name += chars.join('');

        // increase the number of variables we have transformed
        index++;
      }
    }
  };
};

This plugin will add invisible characters onto every variable name, making every variable unique. Passing this plugin to a babel transformation will render the code broken. The error messages will be even more cryptic, as nothing will seem to have changed.

Of course fixing this code manually will be extraordinarily difficult, which is why I've produced the cure as well!

module.exports = () => {
  return {
    visitor: {
      Identifier({ node }) {
        const newName = node.name.split('').filter(char => {
          const codePoint = char.codePointAt(0);
          // if it's within this range, it's invisible.
          const isInvisible = codePoint >= 0xfe00 && codePoint <= 0xfe0f;
          // only allow visible characters
          return !isInvisible
        }).join('');

        node.name = newName;
      }
    }
  };
};

Conclusion

I thought ending with a somewhat "practical" application of what we've found through researching unicode might be interesting.

It goes without saying, but please don't actually use the aforementioned babel transformation on an unsuspecting participant's code. This was all in good fun and learning. The resulting output can be extraordinarily aggravating to debug.

June 4th Edit:

When discussing this post with a friend, we found it was possible to check valid variable characters using regex. This brings with it a significant speed improvement, so I'd advise using it over try{}catch{}.
One can find if a character is a valid starting character with /\p{ID_Start}/u.test(char) and if it's a valid "continuation" character with /\p{ID_Continue}/u.test(char).

Good Blogs Outside of Dev.to

EmNudge — Sun, 08 Mar 2020 00:22:11 +0000

Heyo! I like to write about seemingly difficult coding topics. This blog is specifically focused on JavaScript.

There are some posts, however, that I don't end up writing. Not for lack of experience with the topic, but simply because they've been written much better than I could ever write them.

The main problem with this is that my audience might not ever read these posts because they're unaware of them. In order to remedy that (and because I haven't written much in a while), I created this post.

note: The ordering of people is arbitrary

Tan Li Hau

Fascinating articles on compilers, parsing, code-splitting, and bundlers.

Some of my favorite articles:

Cliff L. Biffle

More recently, Cliffle's blog has been about Rust and Wasm. This is the one non-JS blogger, but I'm including him here because of how much I value his posts.

Some of my favorite articles:

Keith Cirkel

While he hasn't written much as of late, his 2015-2016 articles on ES6 are still useful today.

Some of my favorite articles:

Tom Dale

His articles are often linked to, and for good reason! He only has a few articles and they're fairly general, but all a good read.

Some of my favorite articles:

Dan Abramov

Dan is probably the most well known of the bunch. He works on react and blogs about it too, in addition to reflections of general dev "best" practices and stereotypes.

Some of my favorite articles:

Allen Wirfs-Brock

TC39 member and programming language expert. Most of his JS posts are rather old, but still just as fascinating. He delves deep into some areas that you wouldn't think relevant.

Some of my favorite articles:

Others

Do you have some others that you particularly like? Drop a comment to let others know!

Lazy Iterators From Scratch

EmNudge — Fri, 31 Jan 2020 18:05:01 +0000

I really like functional programming paradigms. Not necessarily functional programming. I've never quite got into it.

But things such as higher-order functions, pattern matching, immutable data structures, pure functions, and so on are really nice to use and reason about. These aspects allow for cleaner and readable code, but may come at the expense of performance if not implemented properly.

One of the easiest ways to sacrifice performance for readable code in javascript is with the higher-order functions. They're fantastic, but you can land yourself in some situations that could have been avoided with a less functional approach.

Let's create a bit of a contrived but somewhat practical example. We need to do some shopping. To simplify things, we won't include any named for the products, just the price. We will try to calculate which items we can afford to buy via filtering out the ones that go over the total.

// constants
const TAX_RATE = 1.08875;
const BASE_SHIPPING = 8;
const BANK_BALANCE = 40; //

// all the items we want to buy
const itemPrices = [2, 4, 5, 9, 10, 13];


let currTotal = BASE_SHIPPING;

const newPrices = itemPrices
    .map(n => n * TAX_RATE)
    .filter(n => {
        if (currTotal + n > BANK_BALANCE) return false;
        currTotal += n;
        return true;
    });

console.log(newPrices)

Did you catch the problem? No? Let's pretend our array had a thousand elements. A million elements, maybe. Let's also keep our bank balance the same. We're a child with a piggy bank and big dreams.

Each method call takes in a higher-order function and loops through the entire array. Is there any way to stop looping prematurely? Yes, but not with this approach.

We are checking if the current total is greater than our bank balance. Once the total exceeds the balance, there isn't really a need to continue. We know the rest of the items are not within our budget.

(This wouldn't necessarily be the case if the items weren't sorted. They are in this snippet.)

Let's now write the same code with a for-loop:

// snip...

const newPrices = [];

for (const n of itemPrices) {
    const priceWithTax = n * TAX_RATE;

    if (currTotal + priceWithTax > BANK_BALANCE) break;

    currTotal += priceWithTax;

    newPrices.push(priceWithTax);
}

// snip...

Our object-oriented code, aside for the keyword, is faster by virtue of not creating a new array each time. We combined both map and filter into statements in our for loop. Only one array is created.

But did you notice that keyword?

break

It lets us prematurely exit the loop. Not only are we no longer checking if we reached our total, but we also aren't even adding the tax! We have skipped 2 operations that we otherwise couldn't!

The functional approach using Array.prototype.map and Array.prototype.filter are just less performant due to the very nature of the methods themselves.

One thing you also may have noticed is that our less functional approach is almost objectively less readable. It's harder to scan and realize what's going on. Yes, it's less performant, but it may need to be a sacrifice taken when writing clean code for smaller arrays where a couple of extra operations is insignificant.

However, there is a way to satisfy the performance problem while still applying a clean-code/imperative paradigm. This is with lazy iterators.

Lazy Iterators

One thing that may seem obvious to use about these higher-order functions is that they do something when you call them. When you tell it to map, it maps. When you tell it to filter, it filters.

What other way can we make these work? We can probably envision a system where the filter method is provided another parameter - one that tells it when to stop iterating. This would involve moving the method provided to filter into a for loop.

We can also probably envision a system where the map and filter are combined as we did in our object-oriented approach.

This is all possible with lazy iterators. We can take in methods such as map and filter, but not execute them until we're told to. We take the functions passed into these methods and execute them in a for loop in order to break iteration early.

A lazy iterator library in JS might looks something like:

const arr = [1, 2, 3, 4, 5, 6, 7, 8, 9];
const arrIter = new LazyIter(arr);

const newArr = arrIter
    .map(n => n ** 2)
    .filter(n => n < 30)
    .collect();

Although the code looks very similar to the functional version, it uses a for loop under the hood where all functions are executed on each element, one by one. This also provides some other benefits.

// snip...
const arrIter = new LazyIter(itemPrices); // use lazy iter library

const newPrices = arrIter
    .map(n => n * TAX_RATE)
    .takeWhile(n => {
        if (currTotal + n > BANK_BALANCE) return false;
        currTotal += n;
        return true;
    })
    .collect();

// snip...

takeWhile is a method that stops the iteration when it returns false on a given element. Because every function gets executed once per element instead of each one iterating over the entire set, we can also ensure the map is only getting executed for the elements returned.

As it is an iteratable, we can also use it in a for loop without collecting and then stop prematurely using break, saving on function calls yet again, since the functions are only called when each element is retrieved.

const arr = new LazyIter(
    [1, 2, 3, 4, 5, 6, 7, 8, 9]
).map(expensiveOperation);

for (const item of arr)  {
    break; // only called expensiveOperation once
}

Let's Make It

Well, this wouldn't be a "from scratch" article if we didn't go over how to make one. It's surprisingly simple.

Let's first create our class.

class LazyIter {
    constructor(arr) {
        this.arr = arr;
        this.funcs = [];
    }
}

Nothing especially important here. We are storing the array provided to us and then creating an array to store all the functions that users will add via the methods provided.

class LazyIter {
    // snip..

    map(func) {
        this.funcs.push({ type: 'map', func })
        return this;
    }

    filter(func) {
        this.funcs.push({ type: 'filter', func })
        return this;
    }

    takeWhile(func) {
        this.funcs.push({ type: 'take_while', func })
        return this;
    }
}

Here we have functions that add the parameter to the funcs array, with a string identifying what kind of function via the type property. You may also notice the return this at the end of each function. This is to allow method chaining, but is not strictly necessary.

These are the only 3 function methods we're going to provide. Other ones should be simply as trivial, but I'll leave the minutia up to you.

class LazyIter {
    // snip...

    *[Symbol.iterator]() {
        for (const item of this.arr) {
            yield item;
        }
    }
}

So this might look a bit odd. It's not finished, don't worry.

This here is a [Symbol.iterator] method. If there exists a Symbol.iterator method that returns an iterator, the class/object is known as an iterable, which lets us use it in for loops and other areas where iterables can be used.

We can alternatively create a generator instead of the weird mess hand-implementing the iterator-protocol would require. That's what the * means. Any expression we yield will be an item in our iterator.

That means our class currently can be shoved into a for loop and give us the elements in our array. Since we could have done that without shoving the array into this class, this isn't especially helpful.

class LazyIter {
    // snip...

    *[Symbol.iterator]() {
        outer:
        for (const item of this.arr) {
            let val = item;

            for (const { type, func } of this.funcs) {
                if (type === 'map') {
                    val = func(val);
                    continue;
                }

                if (type === 'filter') {
                    if (!func(val)) continue outer;
                    continue;
                }

                if (!func(val)) break outer;
            }

            yield val;
        }
    }
}

You'll find a bit of an odd coding style here, like how I use continue instead of else, but it's easy if you take it slowly.

Essentially, we have 2 loops - one to loop over the array and an inner one to apply all the functions to each item.

We are labeling the outer loop with outer: in order to break out of both loops from the innermost one without making things a little too complicated.

Take note of continue outer and break outer. This is how we skip out of the inner loop and perform some action continue/break on the outer loop. A filter would skip the outer loop from yielding the value, essentially filtering out the item. A takeWhile would break the outer loop, removing all subsequent items.

We are going to use this iterator protocol to create our collect method, finishing up our entire class.

class LazyIter {
    // snip...

    collect() { 
        return [...this];
    }

    // snip...
}

Yup. Simple as that. Since we are now an iterable, we can spread ourselves into an array. This lets us keep our code nice and simple.

We can create other methods similar to collect, such as take(num) which accepts a number of elements to retrieve. It's simple enough to code, so I'll leave that up to you.

Here is the class in its entirety:

class LazyIter {
    constructor(arr) {
        this.arr = arr;
        this.funcs = [];
    }

    map(func) {
        this.funcs.push({ type: 'map', func })
        return this;
    }

    filter(func) {
        this.funcs.push({ type: 'filter', func })
        return this;
    }

    takeWhile(func) {
        this.funcs.push({ type: 'take_while', func })
        return this;
    }

    collect() { 
        return [...this];
    }

    *[Symbol.iterator]() {
        outer:
        for (const item of this.arr) {
            let val = item;

            for (const { type, func } of this.funcs) {
                if (type === 'map') {
                    val = func(val);
                    continue;
                }

                if (type === 'filter') {
                    if (!func(val)) continue outer;
                    continue;
                }

                if (!func(val)) break outer;
            }

            yield val;
        }
    }
}

Closing

I don't usually make tutorials. My articles are more conceptual than anything.

I wanted to write this one to outline the type of performance improvements that developers may want to focus on. While micro-optimizations and language-specific optimizations are never a very wise choice in JS, algorithmic improvements work across languages and are very difficult for the engine to optimize.

A developer's primary concern should be code clarity, with performance coming at a close second, depending on the situation. When performance benefits can be achieved without sacrificing code clarity, there often isn't much of an excuse.

If your goal is more important in your situation, the object-oriented approach will always be faster than using our abstraction class. It is simply much more difficult to read and reason about.

EDIT: After writing this (and yes, after) I decided to go ahead and put a lazy iterable class on github. This one is made with TS, so there are some code changes and additions.

Identifying Negative Zero

EmNudge — Sat, 25 Jan 2020 01:19:40 +0000

As usual, I like to delve into some of the weird or complicated bits of JS. The other week I was thinking about a way to possibly identify a negative zero.

Some of you may be taken aback. What is a negative zero? Can zero even be negative? It can according to the IEEE! This is a standard used for almost all languages in their number systems. Thusly, a lot of the "weirdness" in JS number systems is actually standard across other languages, too!

Although, how would we know if the input we were receiving was -0 or 0? Does it matter? Not really, but it's a fun exercise.

Let's first create the rules. We have a function called isZeroNegative which accepts a single parameter. We must return true if the zero is negative and false if it is a positive zero. We can write whatever code we want inside the function.

function isZeroNegative(zero) {
  // some code
  return false; // default return
}

Our Options

It seems pretty simple, but you will find it a bit challenging. You might first think to check if it's below 0, as that's usually the definition of a negative number, but this won't do. The check will return false even though it is negative. -0 < 0 === false.

Math.sign() might be your next guess. Its very purpose, after all, is to let us know whether a number is negative or not. If you pass a negative number, it will return -1 and 1 if it is positive. Unfortunately, it will return the same zero we passed in if the number is a zero, whether negative or positive. Math.sign(-0) === -0.

The next option might be to just check if it's strictly equal to -0. Easy enough! Unfortunately, not even strict equality is strict enough! -0 === 0.

We're running out of options quickly. We need to run our -0 through a system that won't spit out a zero. It needs to give us back a number lower or higher than that or just a different data-type altogether.

What about stringification? We can call .toString() on any number to get its string equivalent. We can then check for the negative sign. We can get the negative sign in -4 by doing (-4).toString()[0]. Unfortunately, yet again, negative zero is one step ahead of us. Darn that negative zero!

Doing (-0).toString() will simply result in "0". Negative zero stringifies to just zero.

Enough of the cat-and-mouse games. Let's review some actual answers. Try to think of one your own. Remember what we said 3 paragraphs back.

Actual Answers

There are a couple of methods. These might not even be an exhaustive list, so feel free to try out other ideas!

I first came across this blog post when researching this problem. It's from Allen Wirfs-Brock, a person currently on tc39, but it was was written in 2011, so there may be more methods made available recently. I will be taking the first 2 solutions from there.

The first solution we're going to explore is possibly the most performant as we don't change any data types. We work entirely within the number system.

Mathematics and the IEEE

We should first think about what kind of mathematical application can we involve -0 in to affect the result? Addition or subtraction? No, those would both act similarly to 0. Multiplication or Division? Multiplication would give us a zero (either negative or positive) and so that leaves us right where we started. Division with the zero as the numerator brings the same problem as multiplication!

What if we involve -0 as the denominator? Any number aside for another zero on the numerator would result in -Infinity! We can check for -Infinity pretty easily!

We have to be sure it is only -0 on the denominator that can result in -Infinity, however. Jeff Waldon provides -Math.pow(2, -1074) as a counterexample. We can thusly add a strictly equals check to ensure we're dealing with a zero.

Via this solution, our winning code is now.

function isZeroNegative(zero) {
  const isZero = zero === 0;
  const isNegative = 1 / zero === -Infinity;
  return isNegative && isZero;
}

Interestingly enough, we can now create a more full-proof Math.sign() using the same sort of method. This one works the same as the old one, but now works well with zeros. As a side-effect, Math.sign('') now returns 1 instead of 0.

Math.sign = function(num) {
    if (Number(num) !== Number(num)) return NaN;
    if (num === -Infinity) return -1;
    return 1 / num < 0 ? -1 : 1;
}

The Strictest Equality

Let's go back to a previous attempt at a solution. We found that strict equality (===) wasn't strict enough. What kind of equality is more strict than that? What about constants? If the engine can identify -0 as being different than 0, we may be able to use that somehow.

Unfortunately, const is too strict. We can't reassign a constant variable in the same scope, no matter what. Even if we reassign it to what it already was. The mere presence of the = operator is enough to trigger an error.

What we're trying to do is something like the following.

function isZeroNegative(zero) {
  if (zero !== 0) return false;

  const posZero = 0;
  try {
    posZero = num;
  } catch(e) {
    return true;
  }

  return false;
}

This will, unfortunately, trip even if we receive a positive zero. As mentioned before, the presence of = is enough to set things off.

Are there any other constants in JS? Well there in fact are! Using Object.freeze, we can make an object immutable.

It is important to note that mutating a property on an object that has been frozen will not throw an error. We need that to happen. To do so, we will use the more direct Object.defineProperty.

function isZeroNegative(zero) {
  if (zero !== 0) return false;

  const posZero = Object.freeze({ val: 0 });
  try {
    Object.defineProperty(posZero, 'val', { value: num });
  } catch(e) {
    return true;
  }

  return false;
}

This will throw an error if the new value is anything other than 0!

Modern String Conversion

Let's yet again approach a solution we dismissed earlier. While it is true that (-0).toString() returns "0", there is a more modern stringifier - .toLocaleString(). It's pretty powerful, but recently I came across a tweet regarding how calling it on Infinity will return the symbol ("∞"), not the normal stringified version ("Infinity").

Calling .toLocaleString() on -0 actually returns "-0"!
Using this finding, we can modify the code as follows:

function isZeroNegative(zero) {
  if (zero !== 0) return false;

  return zero.toLocaleString()[0] === "-";
}

The last one is the quickest and simplest, but possibly not the most performant, which may be important in situations where finding a negative zero is important.

Conclusion

Why would one even need to find a negative zero? I can't think of one off hand. The reason isn't necessary.

This article was more so an investigation into thought processes. Solving problems is an extremely important skill as a software engineer. The way you investigate solutions can be more important than the solution itself.

let, const, and my view

EmNudge — Thu, 26 Dec 2019 07:38:05 +0000

"Well this isn't going to be a controversial post" he said, wrongly.

For those unaware, the title is in reference to a little discussion that spanned across web dev twitter and reddit on the 21st of December.

This tweet was in reference to a reddit post of his, which linked to an article he wrote from his fantastic blog, https://overreacted.io/

Many comments on the post critiqued a part of his article which mentioned, in reference to variable bindings, favoring let and only using const when needed. As people usually act in the opposite way, favoring const and only let when a mutation is needed, a lot of the comments on the post where in regards to that snippet.

Dan defended his views on twitter and had some people from TC39 backing him up - either presently or with links in the replies.

A post which seemed to echo his views by a TC39 member is one by Jamie Builds: https://jamie.build/const

The post uses quite a bit of foul language, for anyone who is sensitive to that, but it's a fun read.

My take? I'm a const advocate. I hadn't left any reply on twitter since I tend not to use any social media (at least for social media purposes. I just post jokes on twitter).

With that said, I wasn't going to make a post, but this has been banging around in my head for the past few days and it has gotten to the point where I have to write something on the matter. There's a key point that I feel a lot of people are missing and I'd like to provide it.

My posts usually tackle some of the trickier bits on JS, so this is a bit out of my field, but still something I'm going to try and cover.

`const` vs `let`

Right off the bat, what do they do? Why were they provided in ES6? Didn't we have var?

var is almost entirely out of use due to an aspect that many find to look much more like a bug. var allows for semi-hoisting. I elaborate a bit on this in my article about for loops, but essentially, the variable declaration is hoisted to the top of the current function/file scope.

Additionally, var is not constrained to block scopes, allowing it to hoist itself out of for loops. These 2 annoying aspects brought us the 2 alternatives - let and const. let can be seen as var, but without those 2 "bugs", and const can be seen as let, but with constant variable bindings.

The "constant variable bindings" and not "creates constant values" distinction is important. If you create a variable with const that is assigned to some non-primitive data structure, the structure can be mutated, but the variable cannot be re-assigned. This distinction can be conveyed with a single sentence and it would be much easier to teach if people stopped teaching const as something to create immutable data structures, but rather immutable bindings.

As a quick example:

const myObj = {};       // {} is a non-primitive
myObj.name = 'EmNudge'; // You can still mutate the object
myObj = 3               // ERROR - you can't reassign it.

And that's all there is to it. It's good practice to not use var unless you are Kyle Simpson (all jokes, he does make some nice observations on this very topic, though) and just use const when you need to, right?

Not exactly. Coming from var, the first instinct is to use let, but many javascript developers will advise you differently. First let's discuss why people like Dan Abramov and others are pro-let preference.

prefer-`let`

Dan Abramov actually wrote an article shortly after this debacle unfolded on his justifications. He also compared both sides, but again missed the point I see missing from most posts.

Let's go through his points. His views seem to be inline with most people who share his perspective, so I won't be including anything from other sources.

Loss of intent

Dan claims that prefer-const means we won't be able to know if something has some importance in being constant. Perhaps we have some value for the amount of seconds in a day? This is a value that we'd want to notify developers about being constant.

Confusion

Dan also claims that he sees many beginners confuse const with immutability. As mentioned before, const only serves to make a constant binding, not data structure, but that is not immediately obvious to a beginner.

Pressure to avoid redeclaring

This is actually an interesting one and a reason why I am an advocate for do-expressions or at least using an IIFE in place occasionally. This can also be solved by separating some things into a function.

Essentially, what Dan is saying is that people will often use the ternary operator instead of a mutable value and using an if statement to redefine it.

`const` ain't all it's cracked up to be

The next 2 reasons are essentially that const doesn't provide any real benefit. It doesn't prevent bugs and there isn't any performance benefit. Any benefit is already made by the engine, even with let.

So what's my defense?

prefer-`const`

I'm going to tackle these a bit out of order, so bear with me.

Performance benefits

Yes, there is no engine benefit that I am directly aware of. That's not why we advocate for const, but I'll toss it up as a win for the let people for now. The people who write browser engines happen to be very good at what they do. It leads to a lot of specific ways to write JS code, like "no js-specific micro optimizations".

Confusion

I think this is a bit of an annoying point. JS has many weird parts to it and I would love if none of them existed, but we can't ignore them. Using let in most cases will only prevent people from seeing const right now and this aspect is pointless while plenty of codebases already do prefer-const.

const needs to be taught as constant bindings and not constant values. It's unfortunate that it's not immutable values, but "tant pis".

If someone is used to var and declaration hoisting, let may be confusing to them. Closures are confusing. Implicit coercion is confusing.
Yes, the less confusing aspects introduced the better, but this is not one that we should try to avoid for reasons explained a bit more later.

No benefit

I'm going to take an interesting stance and not argue wholly with this. const is useful to me when used as a default. If you use const occasionally, you should only really use it when you need to. It's when you use it always that the real benefit comes in.

This will play a lot into the next and final point, which will need its own section.

Intention Has Been Moved

Intent is not lost, just moved. A big issue I have is with people assuming that developers are using const to declare intent. In reality, people are using let to declare intent.

Let's backtrack a bit.

Readability

One of the favored aspects of a particular paradigm of programming is how easy it is to read someone else's code, as that's what you do most as a developer in many jobs.

If the code is shorter, that's usually easier to read. If the code uses good variable names, that's usually easier to read. If it is declarative rather than specifically imperative, that's usually easier to read.

One of the biggest time-savers in skimming code is reading intent. If I see someone's code using declarative function names, I know at least what their code is trying to do. If it isn't doing what it says it's doing, I additionally know it's a bug.

Intent through code

This extends to other parts as well - not just function names. Comments show intent. The type of HOF you might use can show intent. Using backticks vs quotes for strings shows intent. A lot of things show intent and allow me to skim your code and better understand what's going on.

In regards to variable declarations, rust has immutability be default. I much prefer this way because mutation is actually much more rare than variables made to clear up code.

In rust, variables are declared with let. If they're to be mutated late on, they're declared with let mut. When I see mut, I am expecting to see this variable change. I like this. mut shows me intent to mutate. const has a separate meaning, but the point is that variable immutability isn't something we need to bother showing intent behind. It's the natural way of things.

When I changed to prefer-const, I was first fairly reluctant. I had no idea how many of my variables were truly never changing. These days, my code contains so few lets, you might not notice any outside of a for loop unless you're really searching for them.

In reality, it is let that shows intent, not the other way around (as every let-advocate seems to be insinuating or declaring outright). If I see let, a fairly unusual keyword, I know to expect something to change soon.

You can still use `let`

If someone does not want to use ternaries for a new variable, they can very well use let and I will know to expect a mutation soon. If you use prefer-let, I will not know to expect a change with an if statement. You can also take this opportunity to use an intermediate function and still use const without involving ternaries.

prefer-const does not mean leaving let. You can still use let, but only when you find it advantageous for the scenario.

Want to specify something as const? Use screaming case. THIS_VARIABLE_NAME already shows me something is specifically constant. I don't need another keyword to indicate this. Using prefer-const does not remove const intent and instead provides us with a new let intent, which turns out to be much more useful.

Conclusion

As Dan Abramov says, it's usually preferable to stick with the conventions of a specific project rather than pioneer your own. Consistent code is cleaner code.

Happy holidays!

The Comma Operator and Operator Precedence in JavaScript

EmNudge — Wed, 27 Nov 2019 20:05:00 +0000

Let's take a look at an expression. Give you best guess for what it evaluates to:

    40 + 36,000

Some might say 76000, others might say it'll throw an error. In actuality, this expression is completely valid in JS and it evaluates to 0. How? The comma operator.

The comma operator is the operator with the weakest "binding" in JS. Remember PEMDAS from math class? Operators have their own order of operations, but it's instead referred to as bindings or "Operator Precedence".

MDN describes precedence as "Operators with higher precedence become the operands of operators with lower precedence". What this means is that if an operator (which has 2 operands) has a higher precedence, it is as if it is surrounded by parentheses; it is more strongly bound to the values to its right and/or left.

    40 + 36,000                          // 0
    (40 + 36) , 000                      // 0

    new Date().getDay() + 3 * 2          // some number  6 - 12
    (((new Date()).getDay)()) + (3 * 2)  // the same number

    ! void false && true || false        // true
    !((void (false && true)) || false)   // true

    true ^ ! 100 < 56 ? 2 : 1             // 3
    true ^ (((! 100) < 56) ? 2 : 1)       // 3

The levels of precedence for each is listed on the MDN page near the bottom, right after the example. If 2 operators have the same precedence, their associativity tells us whether to go right-to-left or left-to-right when evaluating them.

The comma operator has the lowest precedence. It binds last. If you ever see a comma, you know it won't accidentally bind to some code that it shouldn't.

So what does the comma do? It takes 2 expressions as operands and returns the rightmost one. It is left-to-right associative and we can therefore chain them. Here are some examples

    5 , 2               // 2
    3 , 4 * 2           // 8
    40 , 5 , 1 , false  // false

Note that it specifically takes in expressions, not statements. That means we can't put things like let x = 4 as one of the operands, but we can put things like x = 4. Declarations are statements, while assignments are expressions.

What else is an expression? decrements, increments, and Function calls!

This means the following is valid JS:

    let x;
    const myFunc = num => {
        for (let i = 0; i < num; i++) console.log(i);
    } 

    const result = (x = 3, x++, myFunc(x), x * 2);
    console.log(`result: ${result}`);

    // logs:
    // 0
    // 1
    // 2
    // 3
    // result: 8

Parentheses are required as the comma operator's low precedence would make us accidentally combine the assignment and the first expression. Parentheses have the highest precedence, so they contrast nicely to the comma's lowest precedence.

The one case where parentheses are not required is when one approaches an "operator" of even higher precedence, the semicolon.

This allows us to leave the parentheses behind where semicolons are involved, such as in loop headers.

    let i;
    for (i = 0, i++, i--; i < 10, i < 5; i += 2, i-= 1) {
        console.log(i)
    } 

    // logs:
    // 0
    // 1
    // 2
    // 3
    // 4

How is any of this useful? One of my favorite way is code golfing. As long as we don't involve any statements, we can stuff an entire loop body into the header.

    for (let i = 0; i < 3; console.log('hey!'), i++);

    // logs:
    // hey
    // hey
    // hey

If we have a variable that is already defined and initialized, we can also just use a while loop.

    let i = 0;
    while(console.log('hey!'), ++i < 3);

    // logs:
    // hey
    // hey
    // hey

One of the situations which the comma operator would have been especially useful is for a replacement for a function body when it comes to arrow functions. Unfortunately, the comma operator has an even lower precedence than => and so we require parentheses to separate them. This means we're not saving any characters, as we're using () and , instead of {} and ;.

It is only useful in arrow functions if you intend on returning a value, in which you are missing out on a return statement due to the arrow function's implicit return when no function body is present.

    const myFunc1 = () => (console.log("func1 called"), 3);
    const myFunc2 = () => {console.log("func2 called"); return 3}

    console.log(myFunc1());
    console.log(myFunc2());

    // logs:
    // func1 called
    // 3
    // func 2 called
    // 3

Consclusion

I hope you learnt a bit more about operator precedence and the power of manipulating it. I often see people grouping operators where they otherwise wouldn't have to. Grouping operators can often make code more readable, but too many can just as well bring about expression and operator soup.

Use your new found power wisely.

(a == 1 && a == 2 && a == 3) === true - Wait, hold on...

EmNudge — Tue, 26 Nov 2019 18:44:10 +0000

Some of you may recognize the problem in the title. It's a bit of a famous sample of wtfJS, explained very well by Brandon Morelli in 2018.

The code sample is as follows:

    if (a == 1 && a == 2 && a == 3) {
        console.log("What?!");
    }
    // logs: What?!

Why does it work? Well the trick is in realizing that a here is not a primitive. It's an object with a getter - pretending to be a primitive.

So what happens when we even try to compare an object and a primitive? If we take a look at the spec it says (by rules 8 & 9) that we try to coerce the object into a primitive. How? Via ToPrimitive, a function defined in the spec as well.

In short, it tries to coerce the object into a number. If that doesn't work, it tries to coerce it into a string. Let's try to coerce an object to a string and number.

    const num = Number({});
    console.log(num); // NaN

    const str = String({});
    console.log(str); // [object Object]

Okay, so neither of those are helpful. How exactly is it coercing them?

According to the spec, it's calling .valueOf to get the number and .toString to get the string. If .valueOf returns an object, it moves on to .toString. If .toString doesn't return a primitive, it will actually throw an error: Uncaught TypeError: Cannot convert object to primitive value.

We can override them ourselves like so:

    const a = {
        valueOf()  {
            return 55;
        },
        toString() {
            return 100;
        }
    };

    if (55 == a) console.log("we got valueOf()!");
    if (100 == a) console.log("we got toString()!");
    // logs: we got valueOf()!

    // returning an object, so it will be skipped
    a.valueOf = function() { return {} };

    if (55 == a) console.log("we got valueOf()!");
    if (100 == a) console.log("we got toString()!");
    // logs: we got toString()!

You see, we don't actually have to return a string or number in either.

So how do we use this to solve our problem? We make one of the getters return a value and increment it.

    const a = {
        val: 0,
        valueOf()  {
            this.val++;
            console.log("value incremented!");
            return this.val;
        }
    };

    if (a == 1 && a == 2 && a == 3) {
        console.log("We got it!");
    }

    // logs:
    // value incremented!
    // value incremented!
    // value incremented!
    // We got it!

We can do something similarly with the Proxy class, but taking advantage of the same concept.

const a = new Proxy({ value: 1 }, {
    get(obj, prop) {
        if (prop !== 'valueOf') return obj[prop];
        return () => obj.value++;
    }
})

I won't go too into Proxy in this article, as Keith Cirkel made a much better article on the subject over here.

In essence, we are defining a new object with a getter "trap" that returns the current value property and increments it if its .valueOf() method is called. This is just a fancier way of doing something that we did much simpler just before.

Regardless, is this impossible using strict equality? What if we were presented with the same example, but with triple equals?

Strict Equality

Well actually, it's possible. But first, we have to nail down a few fundamentals.

The first being the window object. Any property on this object is automatically given to us as if it were defined in some global scope. As such, window.parseInt is the same as just parseInt, window.alert is the same as just alert, and so on.

We can also define our own properties and seemingly create variables dynamically.

    function makeVariables() {
        window.foo = 55;
        window.bar = "hello";
    }
    makeVariables()

    if (foo) console.log(foo);
    if (bar) console.log(bar);
    if (baz) console.log(baz);

    // logs:
    // 55
    // "hello"
    // Uncaught ReferenceError: baz is not defined

Side note - this is a bad idea. Don't do this. But we're going to need this for our own example.

Next, we need to go over Object.defineProperty. This function lets us define properties on objects with unique qualities. It feels new, but it actually works on IE9.

This cool method lets us make a property really constant, so people don't change it. It also lets us define a custom getter method! Things are starting to feel a bit familiar!

    const myObj = {}
    Object.defineProperty(myObj, 'val', {
        get() {
            return Math.random();
        }
    })

    console.log(myObj.val);
    console.log(myObj.val);
    console.log(myObj.val);

    // logs:
    // 0.6492479252057994
    // 0.6033118630593071
    // 0.6033118630593071

Why is this better than the previous method? Well this time we don't have to rely on coercion!

Let's combine the 2 things we just discussed to finalize our second example:

    let value = 0;
    Object.defineProperty(window, 'a', {
        get() {
            value++;
            console.log("value incremented!");
            return value;
        }
    })

    if (a === 1 && a === 2 && a === 3) {
        console.log("We got it!");
    }

    // logs:
    // value incremented!
    // value incremented!
    // value incremented!
    // We got it!

Nice! Now we have it working with strict equality!

We unfortunately can't define a variable in the object itself (and then access it in the getter), but if we really don't want to pollute the scope, we can use closures and IIFEs in a really tricky way (credit goes to P35 from the SpeakJS discord server).

    Object.defineProperty(window, 'a', (function(){
        let value = 0;
        return {
            get() {
                value++;
                console.log("value incremented!");
                return value;
            } 
        }
    })());

but this obviously is a pretty messy example.

What about Proxy? Could we use it here? Unfortunately, Proxy will not work with the window object, so it doesn't help us in this instance.

Wrap Up

So when is this useful? Hardly ever.

Well, there are some occasions. Have you ever encountered a really weird error when using a JS framework? Something like Uncaught TypeError: Invalid property descriptor. Cannot both specify accessors and a value or writable attribute?

Your framework is probably using Proxies and getter methods under the hood. They're useful, but only when things get complicated and you want to hide that underlying complexity.

DEV Community: EmNudge

Understanding Loose Equality In JavaScript

WTFJS - The Headscratcher

Throwing Errors With Equality Checks

Conclusion

Deno - An Unfinished Beauty

2022 Update:

A What?

The Great

The Standard Library

TypeScript

Deno's API

The Good

Deno's Multi-Purpose Binary

No Package.json

Import Maps

Security

The Bad

Immaturity

Breaking Changes

The Lack of Scripts

Conclusion - The Use Case

Decoding UTF-8

Encoding Unicode

UTF-16

UTF-8

The Algorithm

The Code

Conclusion - Efficiency

From Scratch - Reactive Values

The Problem

Using Objects

Subscribers

Computed Values

The Trick

Accounting For Arrays & Cleanup

Practical Application - Mini Framework

Conclusion

Invisible Unicode - Coding Deep Dive Episode #1

Adventures in Horrible No Good Very Bad Variable Names

Variable Names And You

The Theoretical Prank

Width Testing

Put Into Action

Conclusion

June 4th Edit:

Good Blogs Outside of Dev.to

Tan Li Hau

Cliff L. Biffle

Keith Cirkel

Tom Dale

Dan Abramov

Allen Wirfs-Brock

Others

Lazy Iterators From Scratch

Lazy Iterators

Let's Make It

Closing

Identifying Negative Zero

Our Options

Actual Answers

Mathematics and the IEEE

The Strictest Equality

Modern String Conversion

Conclusion

let, const, and my view

const vs let

prefer-let

Loss of intent

Confusion

Pressure to avoid redeclaring

const ain't all it's cracked up to be

prefer-const

Performance benefits

Confusion

No benefit

Intention Has Been Moved

Readability

Intent through code

You can still use let

`const` vs `let`

prefer-`let`

`const` ain't all it's cracked up to be

prefer-`const`

You can still use `let`