DEV Community: Hyeseong Kim

Dealing with Unicode text, done right and better.

Hyeseong Kim — Sun, 16 Jun 2024 02:59:00 +0000

TL;DR

I created a library called "unicode-segmenter" to handle Unicode grapheme clusters with good performance and reasonable bundle size. Check it out!

This article is not just about promoting my library but also about the process of creating a high-quality library, including optimization and testing.

Updated: unicode-segmenter is now e18e recommendation!

Intro

Have you ever heard of the term "grapheme cluster"? If you're using non-English languages or supporting emoji in your service, you probably have.

One key lesson from Unicode is that strings often require more space than they appear to.

If we used UCS-4 (or UTF-32), it would be sufficient to represent every character ever invented, making it easy to count each character. However, this would waste a lot of space and isn't even enough.

Therefore, most programming languages use more efficient encodings like UTF-8 or UTF-16. JavaScript uses UTF-16 to represent characters like "👋" as surrogate pairs (two characters). So in JavaScript, its .length property is 2.

Some characters can be even longer by using special characters called "joiners".

"🇰🇷" displays as 1, but has a .length of 4
"👩‍👩‍👦‍👦" displays as 1, but has a .length of 11
"अनुच्छेद" displays as 4, but has a .length of 8
Some complex characters displays as 6, but has a .length of 75! 🤯

A unit internally expressed as multiple characters but logically treated as one character is called a "Grapheme Cluster."

Unicode® standardizes how to handle these grapheme clusters in UAX#29, the "Unicode Segmentation" rules.

The problem

Handling graphemes is crucial when creating custom inputs, text editors, or code analyzers that count characters.

There is the Web's Intl.Segmenter API that supports the Unicode Segmentation standard. It's available not only in Web browsers like Chrome and Safari, but also in JS runtimes like Node.js, Deno, and Bun.

However, it's so new that I couldn't use it for my company's app, Karrot. For more context, Karrot is a massive app with over 18M monthly active users in South Korea, so supporting users on older versions is essential. Given the update frequency of mobile users, versions like Chrome 87 and Safari 14.5 are still not sufficient.

One day, my colleague was analyzing the bundle size of their production and reported an issue with our design system library as it seemed unexpectedly large.

It was the "graphemer" library inside, more than 95% of the size of the problematic input component, and is a large enough portion of the overall app that it's even visually noticeable.

Today, graphemer is the most popular way to handle grapheme clusters in JavaScript. (20M weekly downloads on NPM !!)

But graphemer is an old library and is no longer actively maintained. This may result in differences with the latest Unicode version (mostly in Hindi), and no ESM support.

Custom implementations for Unicode are not trivial in bundle size because they include Unicode data. Also, graphemer generates tons of if-else statements for performance reasons. (spoiler: it's not performant)

Well, isn't there a better alternative?

Can WebAssembly (and Rust) save us?

Probably YES!

I already knew that there was a good quality library unicode-segmentation in Rust, and Rust has a great WebAssembly toolchain called wasm-bindgen.

So I created a simple binding using wasm-bindgen.

use unicode_segmentation::UnicodeSegmentation;
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn count(val: &str) -> usize {
    val.graphemes(true).count()
}

#[wasm_bindgen]
pub fn collect(val: &str) -> Vec<String> {
    val.graphemes(true).map(|s| s.to_string()).collect()
}

The above code generates a WASM binary with JavaScript binding. It was surprisingly easy.

After a few manual optimizations, I got a WASM binary in 52KB size. When combined with the bindings, it totals around 55KB. Also, WASM binaries compress very well, coming in at a total of 23KB when compressed by gzip!

This is already smaller than the parsed size of the graphemer library, and since WASM supports streaming compilation, it appeared to be a better option to me.

However, feedback from my colleagues wasn’t positive. The reason was that the size was still too large, and using WebAssembly was not yet feasible for us.

RIIJS (Rewrite It In JavaScript)

Rust people have a cultural idiom called RIIR (Rewrite It In Rust).

Although WebAssembly is known for its performance, its size can sometimes be unacceptable in mobile apps, or the performance might not meet expectations due to binding overhead.

A high-quality JavaScript library is still valuable, so I decided to do the opposite: Rewrite It in JavaScript.

First, I reviewed the Rust library’s license. It's dual-licensed under MIT and Apache 2.0, I chose MIT.

Then I began by modifying its code generator, which was written in Python, to produce JavaScript instead of Rust. Since I’m not familiar with Python code, this was the most time-consuming task.

By utilizing my knowledge of Rust and JavaScript, I ported the implementation of its Iterator trait into JavaScript’s iteration protocol.

Since JavaScript doesn't have a pattern-matching like Rust, it could be hard to replicate the same logic. I used the ReScript compiler to maintain the original logic as much as possible. It made me able to port it confidently. Specifically check_pair function can be converted into this.

After some test-fix loops, I got the first working version and the result was way better than expected. It was 3x smaller in size and 2.5x faster than the graphemer 😲

(It was the initial result and even better now!)

Reducing bundle size

The major win of the library comes from choosing the simplest structure to store the Unicode data table. This wins in both size and performance. Not only that but there are also various techniques and efforts to achieve an even smaller size.

Using JavaScript generator

JavaScript has a syntax for stateful logic known as "Generator". Generator is highly expressive, making them 40% smaller (1682 bytes into 1011 bytes) than an equivalent class with the same functionality!

You can see both class version and generator version in old commits.

Considering compression

JavaScript code is mostly transmitted with compression like gzip or brotli (zstd soon).

The minified graphemer library is 89.08 KB, but after gzip compression, it reduces to 13.15 KB. This is because the library size is significantly inflated by a bunch of if-else statements, and compression algorithms excel at such repetitive stuff.

unicode-segmenter aims to be small even before compression, but it also carefully considers the size after compression.

For instance, when dealing with character data like “\u{1F680}”, a minifier like Terser tries to unescape it to "🚀". Unescaping can reduce code size since escape sequences require additional characters. However, in the case of data tables with a significant number of escaped characters, gzip compression is highly effective, making it better not to unescape them. The results are summarized in this pull request.

Afterward, It was switched to use int32 codepoint directly for performance reasons. And it compresses better.

Even further, I compressed the number strings to Base36 (0~9, a~z), which is easily reversible with toString/parseInt, and still benefits from additional compression from gzip.

initUnicodeRangeBuffer(
  new Uint32Array(bufferSize),
  ',9,a,,b,1,d,,e,h,3j,w,4p,,4t,, ...', // encoded Unicode ranges
);

function initUnicodeRangeBuffer(buffer, value) {
  let nums = value.split(',').map(s => s ? parseInt(s, 36) : 0);
  for (let i = 0, n = 0; i < nums.length; i++)
    buffer[i] = i % 2 ? n + nums[i] : (n = nums[i]);
  return buffer;
};

This technique is inspired by Marijin's code, the author of CodeMirror

Enabling tree-shaking

JavaScript production apps use bundlers. These tools can exclude unused modules from the final bundle or completely remove unused code through static analysis.

The one goal of unicode-segmenter is to provide a complete polyfill for Intl.Segmenter. However, if only grapheme segmentation is needed, it adopts a well-considered modular structure to ensure that unnecessary parts are not included.

The exposed APIs have independent entries based on their respective topics.

unicode-segmenter/grapheme (5.3 KB): Provides text segmentation by extended grapheme cluster
unicode-segmenter/intl-polyfill (5.7 KB): Provide Intl.Segmenter (currently grapheme only) polyfill
unicode-segmenter/emoji (825 B): Provides emoji matchers
unicode-segmenter/general (4.3 KB): Provides alphanumeric matchers
unicode-segmenter/utils (353 B): Provide utilities for UTF-16 and Unicode code points.

Optimizing performance

Thanks to the unicode-rs' excellent implementation I referenced, it was already 2.5x better than graphemer in its first version. It's mostly a hand-written binary search in a compressed Unicode data table. It's simple but very efficient.

However, for even better results, it is important to fully consider the characteristics of the language.

Leverage the compiler

As mentioned above, the ReScript compiler was very helpful in the initial porting process.

I ended up rewriting it by hand for the best performance, but until then it allowed me to focus on other areas first, as it always ensures accuracy and sufficient efficiency.

Using the optimal data type

JavaScript engines also have a compilation process called JIT(Just-In-Time compilation). One well-known thing is that they are very efficient when dealing with 32-bit integers (aka SMI; small integers).

A Unicode data is a list of ranges of Unicode code points, which are 32-bit integers. Therefore, I was able to make all operations internally based on SMI.

Unlike Rust, where the compiler checks data types, in JavaScript, this responsibility falls entirely on the developer. I managed to do it with careful attention.

As a result, performance more than doubled in the second release.

Explicit bindings

When using generators or regular functions instead of classes, it is natural to implicitly capture external variables through "closures". However, implicit bindings can negatively impact optimization and garbage collection.

To avoid this, simply hoisting functions resulted in an additional 10% improvement. (See the change)

Avoiding copying and constructing structs

When dealing with more than one piece of data, there’s a desire to handle it as a structure.

However, JavaScript lacks efficient records/tuples, and both Object and Array always come with a cost. While this isn’t an issue for most applications, it can be critical in the micro-benchmarks of libraries.

It is also very important to avoid implicit copying and unnecessary GCs. Pass references directly and perform the copy explicitly where it really needed.

// retained `cache` reference
function cat(cp, cache) {
  ...
    if (cp < cache[0] || cp > cache[1]) {
      let result = searchGraphemeCategory(cp);
      // update its values by explicit copying
      cache[0] = result[0];
      cache[1] = result[1];
      cache[2] = result[2];
    }
    return cache[2];
  }
};

Continuous benchmarking and efforts!

What helped getting faster mostly was the benchmark tool that was set up from the very beginning and the continuous measurements.

I use mitata (pretiously tinybench). If you care about performance, I highly recommend you do it.

However, poorly designed benchmarks can be as dangerous as having none. They might be completely invalidated by the compiler, too small to reflect real-world scenarios, or fail to identify scalability issues.

To avoid these pitfalls, it is crucial to design benchmarks to be as close to the real world as possible. unicode-segmenter gets help from ChatGPT to construct tests that resemble real code!

Benchmarks can yield very different results depending on the environment. Therefore, I run it not only in my local environment but also across various versions of Node.js and Bun, on various OS and CPU architectures, and I also configure them to run in browsers to verify performance in both Chrome and Safari. Through this process, I confirmed that the performance of Intl.Segmenter has improved significantly in recent versions of Node.js, Chrome, and Safari. This is very exciting news!

But efforts are still worth it since not everyone is using the latest version or ideal environment :)

Testing strategy

Testing is crucial. Since Unicode involves characters that might be unfamiliar, extensive testing was necessary.

I set up initial test suites with Node.js test runner (It's great!) and manually wrote a few known cases, but it was never enough.

Property-based testing

I tried a more precise method called "Property-based testing".

fast-check is the most popular PBT tool in the JavaScript ecosystem. With fast-check, I can easily write test code that defines properties and verifies them based on automated inputs.

Then, how is the verification done? The idea is to check if the results match those of Intl.Segmenter, which is already available at the Node.js runtime!

test('graphemeSegments', async t => {
  await t.test('unicode string', () => {
    // Intl.Segmenter is available on Node.js!
    const intlSegmenter = new Intl.Segmenter();

    fc.assert(
      fc.property(fc.fullUnicodeString(), input => {
        // assert through automated input
        assertObjectContaining(
          [...graphemeSegments(input)],
          [...intlSegmenter.segment(input)],
        );
      }),
    );
  });
});

Through PBT, I found and fixed numerous bugs in the initial implementation. The first bug PBT identified was an empty string (""). 😅 This is a surprisingly common mistake.

I completely switched my primary testing method to PBT and set up additional tests to debug the counterexamples found.

Generating test suites

Unicode provides official test data. I used the data to generate test suites and check for spec compliance.

During this process, I discovered that the Rust library I referenced had not yet implemented the “GB9c” rule, so I implemented it myself. (It seems they implemented it recently)

Production testing

I understand that despite all these efforts, it might still be insufficient. Reality can sometimes be more extreme than theory.

I tried to create migration PRs directly to major dependents of graphemer. I found a few issues related to implementation, and many others related to TypeScript configuration, sourcemap setup, etc.

By supporting both large and small-scale projects, I was able to update the library and gain confidence in its final quality.

Conclusion

As a result, my app achieved a smaller bundle size and better performance.

Updated: It's now only 6.7 KB (parsed) and 3.4 KB (gzipped) in v0.14.4

I also learned that what it does is not that simple. This should be a temporary alternative to Intl.Segmenter.

I recommend to use Intl.Segmenter where possible. The unicode-segmenter library might become another graphemer after 10 years. Who knows!

But if you are in a special environment where Intl.Segmenter is not available, try unicode-segmenter. It provides good performance in a reasonable size.

Especially if you build something on the React Native and the Hermes engine. unicode-segmenter supports it pretty well!

Also, there are more coming shortly

It will soon provide a full Intl.Segmenter polyfill, including word/sentence support. (issue #25)
It will support more advanced patterns that Intl.Segmenter doesn't, like backward iteration. (issue #26)

Are you struggling with the quality of the library in another area? Consider taking some time to investigate. We still have many opportunities for further improvement in the JavaScript ecosystem.

When and Where to use ReScript? The ReScript happy path

Hyeseong Kim — Fri, 24 Nov 2023 16:35:20 +0000

Disclaimer: This article is opinionated, and it's assuming you've already looked into ReScript a bit. I wrote this to defend my judgment on using or not using ReScript in some projects. If you are curious more, visit the ReScript Forum and see more opinions.

Are you ever hesitant about adopting ReScript, or have you tried it and been frustrated? I will give you a realistic guide for adopting ReScript in projects.

What's ReScript?

ReScript is an language, that compiles to JavaScript, providing a superior type system, fast compilation, and type-checking. It offers best-in-class JavaScript interoperability compared to many similar languages.

What's the difference from TypeScript?

According to its official docs, ReScript and TypeScript have pretty different approaches to code.

TypeScript's design aims to express code that is already written in JS. In a sense, it goes beyond the scope of a type system. As a result, TypeScript's type system is more powerful and flexible than any other type systems. It has the power to express even messy and ambiguous models.

ReScript, on the other hand, aims to support clean data models. It has first-class ADT(Algebraic Data Type) support, useful for modeling, and pattern matching for expressing declarative logic, all of those based on its sound type system.

Let me demonstrate a simple state machine in ReScript.

// Counter.res

type data = { count: int }

type state =
  | Paused(data)
  | Running(data)

type event =
  | Start
  | Pause
  | Resume
  | Increase
  | Decrease
  | Reset

let init = () => Paused({ count: 0 })

let next = (state, event) => {
  // Compiler understand what type you want here, without additional annotations
  switch (state, event) {
    | (Paused(data), Start | Resume) => Running(data)
    | (Running({ count }), Increase) => Running({ count: count + 1 })
    | (Running({ count }), Decrease) => Running({ count: count - 1 })
    | (Running({ count }), Pause) => Paused({ count: count })
    | (_, Reset) => init()
  }
}

Beautiful, right?

And the compiler is super smart so you can immediately noticed actually some unhandled patterns there:

You forgot to handle a possible case here, for example: 
  (Running({count: _, _}), Start | Resume)
| (Paused(_), Pause | Increase | Decrease)

However, in some cases, a sound type system may also imply sacrificing some expressiveness to the model. In return for the features, you can't express everything you can express in JavaScript/TypeScript with ReScript.

When ReScript hurts

If you use lodash/fp or Ramda.js to compose functions, or like using ts-pattern to simulate pattern matching in TypeScript, you might also be interested in languages like ReScript.

But adopting is often difficult.

Because of the differences mentioned above, you may encounter challenges when trying to express your intentions, or you may feel overwhelmed when trying to bind frequently used external types.

For example:

@send external beginPath: t => unit = "beginPath"
@send external closePath: t => unit = "closePath"
@send external fill: t => unit = "fill"
@send external stroke: t => unit = "stroke"
@send external clip: t => unit = "clip"
@send external moveTo: (t, ~x: float, ~y: float) => unit = "moveTo"
@send external lineTo: (t, ~x: float, ~y: float) => unit = "lineTo"
@send
external quadraticCurveTo: (t, ~cp1x: float, ~cp1y: float, ~x: float, ~y: float) => unit =
  "quadraticCurveTo"
@send
external bezierCurveTo: (
  t,
  ~cp1x: float,
  ~cp1y: float,
  ~cp2x: float,
  ~cp2y: float,
  ~x: float,
  ~y: float,
) => unit = "bezierCurveTo"
@send
external arcTo: (t, ~x1: float, ~y1: float, ~x2: float, ~y2: float, ~r: float) => unit = "arcTo"
@send
external arc: (
  t,
  ~x: float,
  ~y: float,
  ~r: float,
  ~startAngle: float,
  ~endAngle: float,
  ~anticw: bool,
) => unit = "arc"
@send external rect: (t, ~x: float, ~y: float, ~w: float, ~h: float) => unit = "rect"
@send external isPointInPath: (t, ~x: float, ~y: float) => bool = "isPointInPath"

/* Path2D */
type path2d
@new external newPath2D: string => path2d = "Path2D"
@send external fillPath2D: (t, path2d) => unit = "fill"
@send external strokePath2D: (t, path2d) => unit = "stroke"

/* Text */
@set external font: (t, string) => unit = "font"
@set external textAlign: (t, string) => unit = "textAlign"
@set external textBaseline: (t, string) => unit = "textBaseline"
@send
external fillText: (t, string, ~x: float, ~y: float, ~maxWidth: float=?, @ignore unit) => unit =
  "fillText"
@send
external strokeText: (t, string, ~x: float, ~y: float, ~maxWidth: float=?, @ignore unit) => unit =
  "strokeText"
@send external measureText: (t, string) => measureText = "measureText"
@get external width: measureText => float = "width"

/* Rectangles */
@send external fillRect: (t, ~x: float, ~y: float, ~w: float, ~h: float) => unit = "fillRect"
@send external strokeRect: (t, ~x: float, ~y: float, ~w: float, ~h: float) => unit = "strokeRect"
@send external clearRect: (t, ~x: float, ~y: float, ~w: float, ~h: float) => unit = "clearRect"

This is part of the rescript-webapi binding code to use the Canvas API.

If your initial experience with ReScript involved dealing with a graphics library interface rather than your own application logic, you may have abandoned it quickly.

The major concern

You end up in a dilemma. Which is better: ReScript or TypeScript?

You might consider the choice of language to be a significant concern; You might be concerned that opting for ReScript could lead to an irreversible decision, making it challenging for newcomers to understand the codebase or even hindering the hiring process.

But for all your worries, language doesn't have as major an impact as you might think. What really matters is what you express in the language, your business.

If your business is complicated, it will look complicated even if you express it in the most appropriate language, and if your business is niche, it will be difficult to hire new people even if you use the most popular language.

Enter the "clean room"

When a business and the software represents it become complex, developers have a hard time keeping up with it.

Even if TypeScript’s type system tolerates the situation, it doesn't help much. This is not because it is essentially a problem caused by language, but because the mixing different models and its expressions.

You have to orchestrate models and dependencies well for deealing with growing complexity.

Worth paying attention to the famous approach "Clean Architecture". I'm not to say it's the answer, but it's worth noting that it isolates the business model and puts it to the "core" of the entire dependencies.

In this approach, by definition, there are no external dependencies on the "core" module. Without worrying about the outside world, you can just model your business and build logic from scratch. This is one of the few places where you can adopt DSL(Domain Specific Language) with confidence.

You may still use TypeScript here, but this is where ReScript really shines. In my experience, ReScript shows a few times better expressiveness than TypeScript when dealing with pure business logic.

Let me show examples:

// Entity_Organization.res

// ...

@genType
type data = {
  name: string,
  label: string,
  owner: memberId,
  members: array<memberId>,
}

@genType
type state =
  | Active(data)
  | Archived(data)

@genType
type event =
  | Created({date: Date.t, by: memberId, name: string, label: string})
  | MemberAdded({date: Date.t, by: option<memberId>, member: memberId})
  | MemberRemoved({date: Date.t, by: option<memberId>, member: memberId})

@genType
type error =
  | MemberAlreadyJoined({by: option<memberId>, member: memberId})
  | CannotRemoveOwner({by: option<memberId>, ownerMember: memberId})
  | CannotRemoveSelf({member: memberId})
  | CannotModifyArchived({by: option<memberId>})

@genType
type t = {
  id: id,
  seq: int,
  events: array<event>,
  state: option<state>,
}

@genType
let make = (id, ~state=?, ~seq=0, ()) => {
  id,
  seq,
  events: [],
  state,
}

module Logic = Abstract.Logic.Make({
  type id = id
  type state = state
  type t = t
  type event = event
  type error = error
  let make = make
})

let logic: Logic.t = (t, event) => {
  switch (t, event) {
  // bunch of patterns
  | ({state: Some(Archived(_))}, MemberAdded({by})) =>
    Error(CannotModifyArchived({by}))

  | ({state: Some(Active({members})}, MemberAdded({member}))
    if members->Array.some(existing => existing == member) =>
    Error(MemberAlreadyJoined({by, member}))

  | ({state: Some(Acvite(state))}, MemberAdded({member})) =>
    Ok({
      ...t,
      events: t.events->Array.concat([event]),
      state: Some(Active({
        ...state,
        members: state.members->Array.concat([member]),
      })),
    })
  }
  // ...
}

@genType
let addMember = (t, ~date, ~by, ~member) => {
  let event = MemberAdded({
    date,
    by,
    member,
  })
  logic->Logic.run(t, event)
}

Bonus, features as labeled arguments and HM type inference save you the hassle of declaring signatures of dependencies.

// Service_Organization.res

@genType
let addMemberToOrganization = async (
  // Just name of dependencies, let the compiler work for you.
  ~findMember,
  ~findOrganization,
  ~memberId,
  ~organizationId,
  ~by,
  ~date,
) => {
  switch await Promise.all2((findMember(memberId), findOrganization(organizationId))) {
  | (Some(member), Some(organization)) =>
    switch (
      organization->Organization.addMember(~member=member.Member.id, ~by, ~date),
      member->Member.joinToOrganization(~organization=organization.id, ~date),
    ) {
    | (Ok(organization), Ok(member)) => Ok({"organization": organization, "member": member})
    | (organizationResult, memberResult) =>
      Error(
        #AggregatedError({
          "member": memberResult->Util.someError,
          "organization": organizationResult->Util.someError,
        }),
      )
    }
  | (member, organization) =>
    Error(
      #InvalidParameter({
        "member": member->Option.map(member => member.id),
        "organization": organization->Option.map(organization => organization.id),
      }),
    )
  | exception Js.Exn.Error(exn) => Error(#IOError({"exn": exn}))
  }
}

See? There is no need to declare types such as interface AddMemberToOrganizationDeps { ... }.

This is very convinient on React components too

@genType
@react.component
let make = (~onClick) => {
  // ... useSomthing

  <button onClick> {message} </button>
}

By just adding labels findOrganization or onClick and using it, the ReScript compiler infers actual types from context.

Then add @genType. The ReScript compiler will generate correct TypeScript definitions for you. So you can use the Service_Organization module written in ReScript on your GraphQL server written in TypeScript, using Pothos, Prisma, whatever you want.

import * as OrganizationService from 'core/Service_Organization.gen.ts';

builder.mutationFields((t) => ({
  addMemberToOrganization: t.field({
    type: AddMemberToOrganizationOutputSchema,
    args: {
      input: t.arg({ type: AddMemberToOrganizationInputSchema, required: true }),
    },
    authScopes: {
      activeMember: true,
    },
    async resolve(_root, args, ctx) {
      const result = await OrganizationService.addMemberToOrganization({
        findMember: ctx.app.repo.findMember,
        findOrganization: ctx.app.repo.findOrganization,
        memberId: args.input.memberId,
        organizationId: args.input.organizationId,
        by: ctx.req.currentMember?.id,
        date: Date.now(),
      });

      if (result.tag === 'Error') {
        // handle errors
      }

      // ... other works

      return ctx.app.eventStore.publishAny(result.value);
    },
  }),
}));

This is a very basic inversion of control.

It's nice that you don't need additonal decorators or container like NestJS for dependency injection. Everything is plain function and it can be checked at compile time by ReScript and TypeScript.

These advantages can be applied regardless of whether it is backend or frontend.

There are several architectural patterns for UI applications. It's not an exact classification, but I categorize it into Elm style (aka the Elm Architecture) and Cycle.js (aka Reactive programming or Data-flow graph) style.

Elm is actually a language very similar to ReScript (as the same ML family), and the background to its success was that it provided this architectural pattern from the very beginning. Elm sandboxes the core UI logic, and interaction with the system (e.g. DOM APIs) is provided through the Elm runtime.

You can choose a similar architectural pattern for your UI application. Use ReScript at its core, and use TypeScript to build your own Elm runtime.

Or you may be more familiar with Redux. In Redux, all dirty works is handled by middleware. Then the rest of the UI can be written in ReScript without any inconvenience.

Choosing a general-purpose language

Counterintuitively, languages are more convenient when they represent a limited surface of a model. Although you may fear that it will be difficult to read and difficult to understand if the language varies. However, TOML code is actually more concise and easier to understand than TypeScript code for the "configuration" model

Another example of how convenient a language can be is Mint. It represents declarative, component-based UI, styling, and state management in a very clean way.

Some of the special-purpose languages are not that simple, some of them restrict which interpreter or runtime (VM) you can use. From the perspective of an expression and interpretation system, frameworks like Svelte and Vue are also able to be categorized as languages.

Here, the advantages of TypeScript and ReScript are similar; "bring your own interpreter". Both have no restrictions other than specifying JavaScript as the target. And the JavaScript is universal.

They can also be used for special purposes, like DSL by deliberately limiting their expressiveness.

ReScript can be used to express React components with formatted type signature.

ts-pattern can be called a (kinda) DSL and its interpreter for declarative logic (ReScript is obviously better for this). And so on.

They may not be the languages with the best representation of the model, but they can provide a level of representation that the developer is satisfied with.

In my experience, the closer to the core, the more powerful ReScript is, and the further out, the more convenient TypeScript is. But why don't pick both?

When need quick and dirty

Not all software has this kind of code architecture. Code architectures are difficult to maintain. If critical parts are not properly isolated, they can easily become messy.

However, actually, in most cases, messily wired monolithic code is good enough. In these codebases, flexible languages win.

Is ReScript flexible enough? Perhaps not (yet)

But even lack flexibility, it can be sufficiently productive with a proper "framework" (it's like your own interpreter) for its purpose.

IMO this is one of the missing pieces in the current ReScript ecosystem. Something like ReScript-first full-stack webapp framework (binding to Next.js is not sufficient)

However, the ReScript language and ecosystem are still growing steadily, and I'm sure we will get there in the near future.

Conclusion

You don't need to use ReScript for the whole of your project. The same goes for TypeScript. Where the language makes more sense, use it. The use of multiple languages in any complex project is more common than you might think.

Before embracing a language like ReScript, first evaluate your domain model and the appropriate architecture. Try using ReScript in spaces that have as few dependencies as possible and deal with pure business logic.

Not necessarily, but it makes you happier.

As you become more comfortable and confident using ReScript, you can try ReScript with other external models as well. Help ReScript cover more by sharing it with the community.

Compressing GraphQL Global Node ID

Hyeseong Kim — Mon, 10 Apr 2023 07:40:31 +0000

You may be familiar with Global Object Identification(GOI), especially if you've used Relay.

GOI is one of best practices to build a good GraphQL API, makes possible to interact with clients more efficiently.

It only requires to implement this interface:

interface Node {
  id: ID!
}

type Query {
  node(id: ID!): Node
}

But how?

Typical implementation

const query = {
  node(root, args) {
    // root => {}
    // args => { id: string }
    // 
    // You need a way to specifing which type of node you should load.
    // ... But you only have args.id here.
  },
};

const node = {
  __resolveType(node) {
    // node is the return type of the `query.node` resolver above.
    // How can you determine the typename?
  },
};

You should implement resolvers for make GOI work, but as you can see, there is only id argument in the whole context.

Generally, to load an entity, you need at least the typename and id (unless you're using a Graph DB).

So you need to find out the entity type with only the ID, but the database does not support this (again, unless you're using a Graph DB).

Instead, you can convert all ids in the GraphQL application layer to include typenames.

base64(`${typename}:${id}`)

As the result, when client request with that ID, your API can determine its typename by extracting from ID back.

This is very typical GOI implementaion, as same as of graphql-relay's one.

However, this implementation has several drawbacks.

It will be much longer than the original ID.
Using Base64 string is not URL-safe.

Those drawbacks make it difficult to be used in permalink URLs (e.g. /posts/{node.id}).

But it's just an implementation, not a constraint by definition. How to make a better implementation?

Compressing it!

Technically, the previous ID compression can be represented as

[typename, id] -> CSV(':') -> Base64

You can change it to a better compression algorithm. Here is my suggestion:

[version, dict(version, typename), id] -> CBOR -> Base64URL

dict(version, typename): since the typenames are well-known strings, it can be compressed to a short integer using a dictionary.
version: Using ID for permalink URLs means that can be permanently exposed. You have to maintain all versions of dictionaries you make.
CBOR (Concise Binary Object Representation) is a codec for small JSON object, similar to MessagePack but in the internet standard.
Base64 URL is an alternative to Base64, use only URL-safe character set.

How well does it compress than previous one?

import { toGlobalId } from 'graphql-relay';
import { encode } from 'cbor-x';

const node = {
  typename: 'User',
  id: 1234567,
};

function toCompressedGlobalIdV1(typename, id) {
  const dict = {
    User: 1,
    Post: 2,
    Category: 3,
  };
  return encode([1, dict[typename], id]).toString('base64url');
}

console.log('graphql-relay toGlobalId :', toGlobalId(node.typename, node.id));
console.log('toCompressedGlobalIdV1   :', toCompressedGlobalIdV1(node.typename, node.id));

// graphql-relay toGlobalId : VXNlcjoxMjM0NTY3
// toCompressedGlobalIdV1   : gwEBGgAS1oc

It does pretty well if the original ID format is numeric. If the original ID format is a string like cuid or UUID, this has little effect, but it won't make it any longer than the old way, and still guarantee URL-safety.

Your implementation is not necessarily identical to this one. To summarize the key points:

Compress typenames using a dictionary
Use Base64 URL, Base62, Base58 to get URL-safe string
Keep original ID short

(Or, you can use a Graph DB in the first place 😉)