DEV Community: Sergey Shandar

My JavaScript Set of Best Practices

Sergey Shandar — Mon, 20 Nov 2023 04:35:29 +0000

The article shares my set of rules to follow when coding JavaScript. But, before we go deep into them, I must let you know that the way how I use JavaScript is a little bit unusual. My main purpose in using JavaScript is to use only one programming language for high-level development, instead of C#, F#, Java, Scala, Python, etc. So, I'm trying to avoid non-generic JavaScript frameworks and platform-specific tools. For low-level and system programming, I'm using Rust, and considering Zig programming language.

TypeScript as a linter

While I'm a big fan of static typing and am impressed by the TypeScript project, I no longer use TypeScript .ts files in my projects. The main reason is to avoid building steps. A project should be ready to use without any build steps. However, I use the TypeScript compiler as a linter and JSDoc as type definitions. To use the TypeScript compiler as a linter, you will need to set these properties in your tsconfig.json file:

{
  "compilerOptions": {
    ...
    "allowJs": true,
    "checkJs": true,
    ...
    "noEmit": true,
    ...
  }
}

Hopefully, one day, we will see the Type Annotation proposal in ECMAScript and supported by popular JavaScript engines.

No classes or symbols

I think that user's defined nominal types have no future in deterministic distributed systems. There are two main problems with JavaScript classes:

Classes are part of the JavaScript nominal type system, in contrast to the TypeScript structural type system. Nominal typing is location-based instead of content-based, like structural typing. It means a class is identified by where or when it's defined instead of its content. Location-based identifiers are very difficult to scale and create dependency hell.
Serialization and deserialization of classes require additional code. This code, usually, doesn't have any semantic meaning and is an anti-pattern. Use standard JavaScript objects and arrays instead; they can be serialized and deserialized with just one function call.

JavaScript Symbols have the same scalability problems as classes.

Functional programming

Use functional programming and pure functions as much as possible. For example:

Don't use direct I/O. Direct I/O is the primary source of side effects. You can use dependency injection. Instead of writing to a file directly, use a passed function that should write to a file. It will allow you to test your code much more effortlessly.
Avoid mutability or try to localize data mutations.
Use arrow functions and currying instead of multiple parameters.

You can read more about functional programming in JavaScript in this article.

Use ECMAScript modules

ECMAScript modules are supported by most of the modern JavaScript engines and browsers. To avoid confusion with other module systems, use .mjs file extension instead .js. Currently, I'm using only export default because it's consistent with loading JSON files and the CommonJS system. I want to use only one export method and prefer simplicity over syntax sugar.

import my from './export-default.mjs'
const { a, b } = my

Instead of

import { a, b } from './export.mjs'

Avoid third-party dependencies without a good reason

Development dependencies such as TypeScript or ESLint are okay if they don't require additional build steps. Try to avoid libraries and frameworks that use direct I/O or platform specific. They will make your code very hard to test.

Endnotes

This set of rules is not for everyone. If you work in a big company with a big existing codebase, this may not work for you. However, if you are working on a new project, or you are a startup, or you are working on an open-source project, you may consider these points. Don't rush to infect your codebase with complex things that are difficult to maintain. Please keep it simple as long as possible and add complexity only when needed.

BLOCKSET v0.2

Sergey Shandar — Thu, 12 Oct 2023 20:34:18 +0000

blockset v0.2

I'm pleased to announce that blockset v0.2 has been released. It's the first working version.

What's the `blockset`?

The blockset application is a command line program that can store and retrieve data blocks using a content-dependent tree (CDT) hash function as a universal address of the blocks. The CDT hash function splits data into small connected parts of various sizes. The algorithm allows the detection of the same parts in blocks, even if they are located in different positions of the files. In essence, storage and network systems based on a CDT hash function should save space and traffic by detecting the same duplicate parts in data blocks. For example, it may save significant space if we store build artifacts of CI in such storage.

CDT function in the `blockset`

There are a lot of possible CDT hash functions. As a community, we should select only a few, to make communication and storage more efficient. After multiple attempts, I selected one, which I call CDT0. It uses SHA224 as a compress function and Crockford's base32 (45 characters) as a printable address, suitable for URLs and file names. I would like to publish an RFC for the function when I have more time.

The CDT storage

There are different ways we can build storage based on the CDT0 function. The blockset stores parts as a set of relatively small files located in a cdt0/ folder. Keeping block parts as files in the cdt0/ folder storage has its pros and cons, which are outlined below:

Advantages of the `blockset` storage

A simple file copy command can synchronize multiple storages. When files in different storages have matching names, they contain identical content, eliminating user dilemmas over potential overwrites.
The files can be stored statically on CDN, and a relatively simple script can download parts and restore a requested data block. Each blockset file is relatively small (about several kilobytes), so the script can use a simple fetch function. There is no need for fancy P2P network protocols, nodes, and custom servers.

Disadvantages of the `blockset` storage

As mentioned before, the blockset maintains many small files. Keeping a lot of small files is not space-efficient.
Each blockset file represents only one hash. However, the CDT hash function offers a superior better resolution. This higher resolution can increase the likelihood of detecting identical parts within data blocks.

There are multiple solutions to how these problems can be solved. We can use multiple different internal storage formats and synchronize multiple storages using different protocols as long as we use the same CDT function.

Installation of `blockset`

The blockset can be installed on any computer and platform that supports Rust. To install Rust, see this page.

Installing the blockset:

cargo install blockset

Uninstalling the blockset:

cargo uninstall blockset

Commands

Address validation:

blockset validate 3v1d4j94scaseqgcyzr0ha5dxa9rx6ppnfbndck971ack

Calculate address:

blockset address ./README.md

Add to the local storage cdt0/:

blockset add ./LICENSE

Get a file by address:

blockset get ngd7zembwj6f2tsh4gyxrcyx26h221e3f2wdgfbtq87nd ./old.md

Internals

The blockset is an open-source project under GPL-3 license. You can find its source code here. The project is written in Rust, and we've made a deliberate choice to minimize the use of macros. This enhances code readability and reduces hidden control flows, ensuring a more transparent and developer-friendly experience. Currently, the blockset code has no third-party dependencies. All source files except main.rs don't use I/O directly, which allows us to achieve and maintain 100% code coverage.

Don't hesitate to contact me if you would like to know more, would like to build on either CDT0 or blockset, or need another license:

Content-Dependent Hash Tree

Sergey Shandar — Sat, 30 Sep 2023 05:06:10 +0000

In our early discussion, we highlighted the advantages of using a cryptographic hash function based on a Merkle tree for data transferring in CAN. However, the Merkle tree is not shift-resistant, leading to potential redundancies in our block set or CAS. Here, we delve deeper to tackle this challenge.

The main reason I publish the algorithm is that I firmly believe that we can't solve the data vendor lock-in problem if we keep our data in proprietary formats.

Deciphering a Sequence of Numbers

Imagine intercepting messages from extraterrestrials. We don't know their language, but we assume that they use a sequential language unless they are from the Arrival film. The messages manifest as a sequence of numbers. We only know that each number is a finite number between 0 and N-1. How can we structure this stream without any linguistic reference points? How do we identify repetitive segments?

Using N=8 as an example, consider these similar sequences:

s0 = [7, 0, 5, 1, 2, 4, 6, 1, 7, 0, 4, 7, 3, 4, 6, 2] and
s1 = [7, 0, 5, 1, 2, 5, 4, 6, 1, 7, 0, 4, 7, 3, 4, 6].

At a glance, shared groupings [7, 0, 5, 1, 2] and [6, 1, 7, 0, 4, 7, 3, 4, 6] are evident. But what about handling mammoth data streams in the range of gigabytes or terabytes? Some algorithms require O(n^2) operations, where n is the length of the sequences. We need something close to O(n). To achieve this, we should split our sequences into groups, without considering any knowledge of other sequences.

If we group the numbers by pairs, like Merkle Tree does, we will have

s0: [[7, 0], [5, 1], [2, 4], [6, 1], [7, 0], [4, 7], [3, 4], [6, 2]],
s1: [[7, 0], [5, 1], [2, 5], [4, 6], [1, 7], [0, 4], [7, 3], [4, 6]].

As you can see, initially, we have the same groups [7, 0], [5, 1], but after that our groups are all different. The problem with such an approach is that the group size is fixed and independent of the group's content.

One simple idea to have content-dependant grouping is to group numbers until they stop descending:

s0: [[7, 0, 5], [1, 2], [4, 6], [1, 7], [0, 4], [7, 3, 4], [6, 2]],
s1: [[7, 0, 5], [1, 2], [5, 4, 6], [1, 7], [0, 4], [7, 3, 4], [6]].

One important rule is the autonomy of each group, akin context-free grammar. Grouping should only depend on items within the group. Extracting an element from a sequence mandates its inclusion in the current group. There's no going back. Sorry, "No Return Policy". An item may end the current group, and we will create a new one for the following numbers. Context-free grouping is suitable for validation because we can always validate a group without knowledge of surrounding groups.

This algorithm can create the same groups of numbers in both sequences despite a shift in the middle: [7, 0, 5] - 2 times, [1, 2] - 2 times, [1, 7] - 2 times, [0, 4] - 2 times, [7, 3, 4] - 2 times. The next step is to convert each group into a number and repeat the process.

Let's try a simple approach: our function will convert numbers to strings and then concatenate them:

s0 = ['705', '12', '46', '17', '04', '734', '62'],
s1 = ['705', '12', '546', '17', '04', '734', '6'].

The next level of grouping will have the same group 1704:

s0 = ['7051246', '1704', '73462'],
s1 = ['70512546', '1704', '7346'],

And the roots of the trees:

s0 = ['7051246170473462']
s1 = ['7051254617047346']

The string concatenation function works well for small sequences, but to create a good function for big data blocks, we need to research the properties of the groups.

Group Properties

If N is a number of possible items, then one group can have a maximum of N+1 items. For example, the longest groups for N equals 8 will be [7, 6, 5, 4, 3, 2, 1, 0, x], where x is any number from 0 to 7. A group can be partitioned into:

Body, all items in the group except the last one.
Tail, the last item in the group.

Properties of a body:

a body has at least one item,
numbers in the body never repeat,
numbers are arranged in ascending order,
the total number of unique bodies equals 2^N - 1.

Properties of a tail:

it's always one item,
the item is less than or equal to the last number of the body.

So the total group count M equals N*2^(N-1) + (N-1)*2^(N-2) + ... + 3*2^2 + 2*2 + 1 = (N-1)*2^N+1.

The number of internal states during group assembling S equals (N-1)*2^(N+1).

N	M	S
2	5	8
3	17	32
4	49	96
5	129	256
6	321	640
7	769	1536
8	1793	3584

The average length of the group is e, ~2.71828 or less for small N.

Bit Stream

We can start building our tree from any N. The smallest possible N is 2, and a leaf is either 0 or 1. A good thing about a bit stream is that it doesn't rely on any artificial structure like a byte.

Level 1

The level 1 has 5 groups. Each group forms a new number id for the next level.

group	id
00	0
01	1
11	2
100	3
101	4

N = 5
S = 8
Min size: 2 bits. For example: 00.
Max size: 3 bits. For example: 101.

Level 2

We use 5 numbers from the previous level as input for this level.

N = 4*2^5 + 1 = 129
S = 256
Min size: 4 bits. For example, 00 of level 1 numbers or 0000 as a bit sequence.
Max size: 15 bits. For example, 432104 of level 1 numbers or 101_100_11_01_00_101 as a bit sequence.

Level 3

N = 128*2^129 + 1 = 2^136+1.
S = 2^137. We need 137 bits to store an internal state on the level 3.
Min size: 8 bits.
Max size: 0x480 + 0xF = 0x48F = 1152 bits.

Level 4 and up

For levels four and up, we use hashes instead of actual data due to the explosive growth in N. As we discussed before, the average length of the group is e, but the maximal length of the groups for a big N could be very long. I would like to discuss splitting the groups into smaller parts in one of the following articles.

Conclusion

In essence, the content-dependent hash tree optimizes the identification of identical groups, catering well to both CAS and CAN applications. However, crafting a CAS on such a tree requires its own in-depth exploration.

Some aspects of Merkle Tree

Sergey Shandar — Wed, 13 Sep 2023 20:53:02 +0000

Previously, I shared my thoughts about how we can solve data vendor lock-in problem using content-addressable network (CAN) and content-addressable storage (CAS) for personal use. As mentioned in the article, we don't care how our network works as long as we can download a data block for a specific hash because we can always validate the downloaded data against the hash value. There is one problem with validation. If the hash validation fails and our file is quite big (say several gigabytes), we waste our network time and resources for nothing.

We wouldn't have such a problem if we could validate small parts of the data block while downloading the entire data block. One of the solutions is to use Merkle Tree for our hash function. There are a lot of good articles and videos about it, and I don't want to repeat them. I want to focus on some of the essential aspects of the Merkle tree that you should know if you would like to implement one.

Definitions

f is a cryptographic compress hash function that accepts two digests and returns one. For example, h01 = f(h0, h1).

We split a data block into a list of digests [d0, d1, ..., dN]. Let's assume that N is 2^P, where P is the height of the corresponding Merkle tree. Our tree will look like this:

Second Preimage Attack

If we know data for a specific hash, we can create a collision because the tree doesn't distinguish between hash and data digests. So, it's possible to send hash digests instead of actual data. In this case, the validation will succeed, but the data will differ.

It's called second preimage attack. To prevent such attacks, we need to separate data from hashes.

We can use another hash function for the data blocks to convert them into hash digests. Let's call it g.

The g function should differ from f.

Another way to fix the issue is to extend a digest with one bit. If the bit is
set to 0, then it's data; otherwise, it's a result of the f hash function.

Note that such extended digest should not be used to transfer secure data because short messages can be easily restored.

Tails

Actual data usually is not aligned to 2^P. One way to fix the problem is to add one bit 1 and then fill the rest with zeros.

Length Extension Attack

If we have one Merkle Tree hash, we can use it as a node in another tree and create a valid hash for a new data block. This is called length extension attack.
To solve it, we can transform a root hash into another hash before publishing it.

No Shift Resistance

Merkle Tree is a very good idea to split data blocks into parts. We can even reduce traffic if some parts are the same. Or, we don't need to download some parts if we already have them in our hash table. However, it only works if the same data parts are aligned with Merkle Tree parts. This is a big limitation of Merkle Tree.

Data Vendor Lock-In and Web3

Sergey Shandar — Sat, 09 Sep 2023 00:49:30 +0000

What is the biggest problem with Web2?

Most would say the biggest problem with Web2 is centralization. For example, we host our data using central services, like Web2 social networks, Web2 cloud providers, Web2 email providers, and others. We don’t have much control over our data using Web2 services. At any time, the service can refuse access to all or part of our data. People are tired of switching platforms and losing their data.

Does Web3 solve the problem of Web2?

According to Wikipedia, one of the main properties of Web3 is decentralization. There have been multiple attempts to provide decentralized services. For example,

social networks and messengers: Mastodon, Matrix;
cloud providers, like IPFS;
finances: Bitcoin, Ethereum;
and source control systems: Git, Mercurial.

However, these services don't really solve the fundamental problem that a user's data should belong to the user. Even if it is decentralized, we need different applications or services to access and store our data. Some decentralized services (like blockchain) might shut down or change their protocol, and we will most likely lose access to our data again. It could also be hard to reference the data from a new storage. On top of that, storing data in a blockchain could be pretty expensive. I don't want to pay for every message that I send to a friend.

Another problem with Web3 technologies, like blockchain, is that they are not designed for sub-networks. For example, often, a blockchain doesn't allow forking. How will it work if we would like to run the same internet on another planet while only having occasional synchronization? For example, a signal to Mars takes from 5 to 20 minutes. I know we are probably far away from this, but still, there can be other examples of isolated sub-networks that need to be synchronized occasionally.

So, instead of centralized data vendor lock-in, we now have decentralized data vendor lock-in. It's better but doesn't completely solve the problem at hand and it's not future-proof.

What's a solution?

I think the solution is to switch our focus from protocols (how we access the data) to focusing on the structure of data and data formats.

Let's say I create some information, like a block of data. It can be an article, an image, a short text message, a document, or something else. Imagine we have a storage that keeps such blocks. Then, how can we uniquely identify the blocks and avoid duplications in the storage? Well, there is a family of functions/algorithms that can produce a unique identifier for any data block; they are called cryptographic hash functions. For example: SHA2, and SHA3. This family of functions is already used by many decentralized systems. A storage that keeps data blocks and accesses them by a hash function is called a hash table.

Note: A block of data is not a file. A file has additional information, such as a name and a file extension. A name of a file doesn't uniquely identify the data it holds. You may have multiple files with different names/paths that hold the same data.

Data synchronization

If we keep our data in multiple file storages, we may face synchronization problems. Have you seen that message that repeats when you try to synchronize your files to a cloud file storage? "File already exists, do you want to replace it?". Honestly, the message freaks me out. I have no idea what I should answer. I just don't want to lose my data and avoid duplications. However, if we keep our data in a big hash table, it's not a big deal to compare two hash tables and synchronize them. There are no merge conflicts, no prompts, no data loss, and no duplications.

What happens if our hash algorithm is compromised?

In this case, we would freeze all our data, which is using the old hash algorithm, and make a kind of immutable registry of allowed/known hashes. This means we can't add new data using the old hash algorithm, but we can access it. For new data blocks, we can use a new hash algorithm that is not compromised yet.

For example, we have a SHA1 hash table:

hash	value
sha1(A)	A
sha1(B)	B

After we find that the SHA1 is compromised, we create two new tables. The first table is an immutable mapping from SHA1 to SHA256. No new blocks can be added to this table.

hash	value
sha1(A)	sha256(A)
sha1(B)	sha256(B)

The second table is a new table with SHA256. We can add new blocks to this table.

hash	value
sha256(A)	A
sha256(B)	B
sha256(C)	C

How can we reference such data blocks from other blocks?

Most URL schemas specify a protocol, for example, https://. Even Web3 has URL-specific protocols, like bitcoin://. To reference a data block or a file, we don't have to use a specific protocol. Some URL schemas are protocol agnostic, such as the URN family. Centralized registration authorities usually assign a URN for each resource. For example, ISBN is used to assign unique numbers to commercial books.

To reference a data block, we shouldn't have to use a registry authority, even if it's decentralized. The URN has no sub-schema for hash-based names, but there is a URL schema that is designed specifically for this being RFC6920. This RFC describes two schemas: ni and nih. To put it briefly, the ni schema is for a data block hash, like ni://sha256;.... The nih is for a short form of the hash; it works similarly to a tiny URL.

Directed acyclic graph

If a data block can reference another data block, then such storage can be presented as a DAG. Assuming that our cryptographic hash is strong, it is almost impossible to create cycle references.

Note: A tree is a special case of a DAG, and a blockchain is a special case of such a tree. When I want to create a new data block, I don't want to rely on a public blockchain like Bitcoin. Public blockchains are too expensive for such simple tasks.

Source of truth and cache

A hash table storage should be considered a source of truth. It also can be used as a source of events. In hash table storages, we should only add new blocks but never delete old ones. If we want to delete an old block, we should add a new block that says the old one is deleted.

The storage may contain a vast number of blocks, and if we would like to understand the current state of the storage, we would need to traverse all blocks. Although, it would not be an efficient algorithm for a big hash table, especially for a large amount of users. However, we can use a cache to store the current state of the storage. One of the good properties of a cache is that it can be easily recreated from the source of truth. A good cache-creating algorithm can also be scalable and distributed between multiple nodes using advanced techniques such as MapReduce.

The arrows show the direction of the data flow. SourceOfTruth is read-only for the Cache, and the Cache is read-only for View in Application.

If we would like to edit information, then we should add new blocks to the source of truth:

User identity

A user should not rely on centralized authorities to create an identity. Instead, they should be able to create a unique digital signature, publish its public key, and sign messages using the signature. A user may also publish associated centralized identities (for example, email addresses), but these identities should be considered temporary.

Time stamping

Sometimes, signing a data block with a digital signature is not enough. A user may need to prove that they are the first who created the data block. Otherwise, another user can sign the same data block with their own digital signature.

To sign a data block with a time stamp, we need to publish a new data block that will reference the original one. The new data block would contain a signed time stamp using trusted timestamping services. There are also decentralized solutions, for example, using a blockchain. The timestamp can be used to prove that the data block was created before an event.

Building data block formats for different applications

Here are some examples and thoughts about how we can make applications and services based on hash tables.

A file catalog

Building a file catalog is very simple.

{
    "files": {
        "a.jpg": "sha256:1234567890...",
        "f/a.jpg": "sha256:1234567890..."
    }
}

You may extend this format to include additional file attributes, such as executables.

A source control system

A source control system is built similarly to a file, but it references previous file systems.

{
    "previous": ["sha256:1234567890...", ""],
    "message": "commit message",
    "files": {
        "a.jpg": "sha256:1234567890...",
        "f/a.jpg": "sha256:1234567890..."
    }
}

Branches can be implemented either inside the data block (like in Mercurial) or outside (like in Git).

Building a private group chat (social network)

Making a public group chat is quite easy. A private group chat may require additional thoughts.

We assume that our network protocol is encrypted, and we will only focus on the data that users receive and store. We cannot control what users are doing with our messages. They can store it, and some of them can even share it with others.

That leads us to the problem of how can we make sure that everyone receives the same information and if the information is leaked how can we identify the source of the leak? When Alice sends a new message into a group chat, she sends it to all members of the group chat. After that, Alice sends different digital signatures for the message to each chat member. Each member of the group chat can verify that the message is signed by Alice. However, if the message is leaked, to prove that the message was sent by Alice, a whistleblower needs to uncover his identity.

What else?

Task manager, calendar, resume, etc. They all should have very simple formats and should be easy to edit. Note, that the performance should be provided by the cache, not by the source of truth.

How do we know what's the format of the file?

Unfortunately, there is no universal solution for this. We can try to parse the data with different readers/parsers and see if it's valid. Some applications may parse only a specific set of formats. We can also create a hint/container data block format, which will have an extension or MIME-type and a reference/contain to the data block.

{
    "format": "hint",
    "extension": "jpg",
    "mime-type": "image/jpeg",
    "data": "sha256:1234567890..."
}

{
    "format": "container",
    "extension": "txt",
    "mime-type": "text/plain",
    "data": "Hello world!"
}

However, because we store different types of blocks in one storage, the data can be shared between applications of different vendors and types, for example between social networks, task managers, calendars, etc.

What happens if a user's private key is stolen?

In this case, a user can publish a new public key and notify their friends. The friends can publish a block that confirms that they do not accept new blocks signed by the old user's key and accept only blocks that are signed with the new one.

What I hope for

I would like to store data in such hash-table storage. I may have multiple storages, like public, private, and shared. Some of them may be in zero-knowledge encrypted storages, and some of them offline. Data blocks in the storage should still be able to reference each other. Then I can control access to the different storages by different applications.

Applications and services may have their own rules, and it's their right. If it is a social network, they may decide not to show some of my posts, they may even decline service to me. However, they can never delete my data and I can still access my data using different applications and services.

Also, I don't want to split my data between different services and applications. I would like to split my data by access, who can access the data.

The current state of the internet reminds me of email hosting from internet providers. Then, you are stuck with this internet provider. It's a data vendor lock-in. Our identity and data should not depend on the service provider, on a big or small company, not even on a decentralized specific blockchain. Social network applications/services should use my data, not store it in proprietary storage. I should be able to switch between different social networks without losing my data. I should be able to create my own social network application that will browse my data. How many times we should start it from scratch and lose our data?

Hotmail, Gmail, ProtonMail,
ICQ, Skype, Viber, WhatsUp, Telegram, Signal,
Teams, Slack, Discord,
LiveJournal, Facebook, Twitter, Instagram, Threads, Mastodon,
SourceForge, BitBucket, GitHub, GitLab,
Wikipedia, Quora, Reddit, StackOverflow,
Google Drive, Dropbox, OneDrive, iCloud.
Google Calendar, Outlook Calendar, Apple Calendar.
Asana, Jira,
LinkedIn, Hired, Upwork, Freelancer,

Please, stop! Enough! Really, it's enough! I like innovations, I like competitions, I like new start-ups, and I'm happy to change my internet/storage provider and applications. I'm okay with changing a communication protocol, however, I hate when I have to lose my data, my contacts, my communication, and other things. What is the point of sign-in to a new social network if your friends are not there? Or do I have to post my blogs on every social network? The data vendor lock-in problem is the biggest problem in Web2.
Let's not carry it to Web3.

In the end, I would like to highlight four main principles that we should follow when building a decentralized internet for people:

Protocol agnostic storages. It doesn't matter which protocol we use to transfer our data to storage. As soon as the storage receives new data, it is able to accept the data without conflicts.
Source of truth. The user's data is a source of truth. Some services may store derived data, such as a cache, but it should be possible to recreate it from the source of truth if the derived data is lost.
No data vendor lock-in. Applications and services should store user data in the storage of the user's choice.
Identity is decentralized. Anyone can create new identities. The process doesn't require any registration authority. The identity is not tied to a specific service or application.

Purely Functional Programming in JavaScript

Sergey Shandar — Sat, 09 Sep 2023 00:36:12 +0000

There are a lot of articles, videos, and blog posts about functional programming using different programming languages, including JavaScript.

Usually, the main topic of these articles is how to use various functional programming paradigms, such as first-class functions, immutable objects, and currying.

Nevertheless, the primary value of purely functional programming languages is an absence of side effects. Partial applications of different functional paradigms in impure languages, such as JavaScript, may reduce the number of side effects but don't guarantee their complete elimination.

Side effects reduce scalability and the ability to replace components and platforms. So, it is preferable to reduce the number of side effects to a bare minimum.

There are dozens of purely functional programming languages. Some of them are pretty successful in the software development industry - for example, Haskell, Elm, and PureScript. However, the most popular programming language is JavaScript, and it is not purely functional.

The main reason to use JavaScript, besides its popularity, is that almost any web browser can run it. Also, one of the most popular data interchange and file formats is JSON, a subset of JavaScript. Because of this JSON/JavaScript relation, serialization in JavaScript is more straightforward than in other programming languages. In my experience, object-oriented programming languages usually have the biggest challenges in serialization.

Any working program has side effects such as input/output, functions that return the current time, or random numbers.

But it is possible to write a big part of a program without using impure functions. An impure function can be rewritten as a pure function.

For example:

const addAndPrint = a => b => {
    const result = a + b
    console.log(result)
    return result
}

const pureAddAndPrint = log => a => b => {
    const result = a + b
    log(result)
    return result
}

Pure functions are much more flexible. A developer may use the pureAddAndPrint function with either pure or impure arguments, such as console.log. Some platforms may not have console.log, and in that case, a developer could provide a replacement for it.

Another use case is unit testing, and a developer may create a mock function and pass it as an argument.

Currying

You may notice function declarations in this article use currying. In most purely functional programming languages, a function can accept only one argument, and currying is a way to provide multiple arguments to a function.

Another way is to use a tuple as an argument:

const tupleAddAndPrint = ([log, a, b]) => {
    const result = a + b
    log(result)
    return result
}

However, currying can simplify partial function applications:

// using currying
const consoleLogAddCurry = pureAddAndPrint(console.log)
// using tuples
const consoleLogAddTuple = ([a, b]) => 
    tupleAddAndPrint([console.log, a, b])

Safety

Usually, purely functional languages provide better safety. A pure function can’t access data outside passed arguments. On the contrary, an impure function can access almost anything, which increases the probability of vulnerabilities.

One such example is the famous Log4Shell. Log4j is written in an impure language (Java), and users were not aware it uses HTTPS to download and run code. A pure implementation of Log4j would require an HTTPS protocol as an argument.

In this case, users have some level of control, and, most likely, they would provide a stub instead of an actual HTTPS protocol. Pure functions do not provide absolute protection, but they can significantly reduce the probability of vulnerabilities.

FunctionalScript

It is possible to write purely functional code in an impure language. FunctionalScript is an attempt to create a purely functional subset of JavaScript, and the subset should not have the ability to create a function with side effects.

Because FunctionalScript is a subset of JavaScript, we do not need to develop compilers, transpilers, debuggers, IDEs, and other development tools for the language.

Also, developers do not need to learn an entirely new programming language and how it interacts with other systems and languages. FunctionalScript is an open specification and has no risk of vendor lock-in.

Even if the FunctionalScript specification disappears completely, any FunctionalScript code will still work like any other JavaScript code.

Recursion Problem

Most purely functional programming languages have no loops because all data is immutable.

Instead, developers use recursion:

const factorial = n => n <= 0 ? 1 : n * factorial(n - 1)

Recursion consumes stack, and it can cause a stack overflow in case of too many recursive calls. Functional languages solve this problem by tail call elimination. Note that a compiler can only eliminate a call if it’s the last call or operation.

For example, a tail call elimination can not be applied to our factorial function because the last operation is multiplication instead of factorial. However, we can change the function so that the tail call elimination can be applied.

const factorialTail = result => n =>
    n <= 0 ? result : factorialTail(result * n)(n - 1)
const factorial = factorialTail(1)

The JavaScript standard (ECMAScript 6) supports the tail call elimination (aka a proper tail call), but V8 and SpiderMonkey do not. That means that Google Chrome, Microsoft Edge, Node.js, and Firefox do not support PTC. So, de facto, JavaScript has no PTC.

Loops in FunctionalScript

The problem is that FunctionalScript objects are immutable, and, as shown above, we can’t use recursion for iterations.

FunctionalScript allows reassigning of local variables declared with let as a workaround for this problem, and such variables can only be used inside a function where the variables are declared.

const factorial = n => {
    let i = n
    let result = 1
    while (i > 1) {
        result = result * i
        i = i - 1
    }
    return result
}

WebAssembly

WebAssembly allows developers to create web applications using almost any programming language. It is derived from asm.js, which is also a subset of JavaScript.

Advantages:

near-native code execution speed,
different programming languages support compilation to WebAssembly.

Disadvantages:

requires additional build steps and tools,
WebAssembly programs should interact with DOM and other JavaScript API using a language interoperability layer.

asm.js inspired FunctionalScript as a subset of JavaScript. Compared to asm.js and WebAssembly, FunctionalScript is a high-level programming language. Theoretically, it is possible to create JIT and AOT compilers from FunctionalScript to WebAssembly, or any other assembly language.

Compared to JavaScript, a compiler from FunctionalScript may generate more optimal code because similar FunctionalScript code is more deterministic.

For example, FunctionalScript can use a reference counter instead of a proper garbage collector because immutable data can not have circular references.

Also, other purely functional programming languages, such as Elm, can use FunctionalScript as a compilation target.

FunctionalScript API Limitations

As was mentioned earlier, FunctionalScript can not directly call functions with side effects. Because JavaScript API has many impure functions, only a limited subset of JavaScript API is available to FunctionalScript.

However, a JavaScript program can pass impure functions to FunctionalScript modules.

Typing

FunctionalScript derived a dynamic type system from JavaScript. Nevertheless, it is possible to use JSDoc type annotations and a TypeScript compiler as a validator. For example

/** @type {(a: number) => (b: number) => number} */
const add = a => b => a + b

See TypeScript JSDoc Reference for more details.

TypeScript uses a structural type system instead of a nominal type system. Languages with a nominal type system may cause typecasting problems in big projects with many third-party modules. For example, two definitions of Vector3D are not compatible, and adapters are required. Because of this, structural type systems enhance modularization and code reuse.

Modules and Packages

FunctionalScript uses a Node.js package manager (npm) and CommonJS as a module system. CommonJS is easy to implement even without a FunctionalScript parser.

Because FunctionalScript is a purely functional language, a FunctionalScript module can only reference another FunctionalScript module. But, a JavaScript module can reference any FunctionalScript module.

Currently, FunctionalScript does not support ECMAScript Modules and asynchronous modules.

JSON Modules

CommonJS supports loading JSON files as JavaScript modules. Because JSON contains only data, any JSON file is also a FunctionalScript module.

Note that the loading procedure differs for JSON and JavaScript files, even if JSON is a subset of JavaScript.

A JSON module declares all public exports in the first expression.

{
   "a": "Hello",
   "b": 42
}

A JavaScript Common.js module declares all public exports in module.exports.

module.exports = {
   a: "Hello",
   b: 42
}

Applications

FunctionalScript code can be used in any JavaScript/TypeScript application. Because FunctionalScript code has no direct access to IO, the same code can be used on different platforms, for example, web-browser, Node.js.

FunctionalScript is a superset of JSON. Because it has no side effects, it can be used as a JSON with pure functions and expressions, for example, in configuration files.

Another application is a query language as an alternative to SQL and LINQ.