DEV Community: Sergey Samokhov

be-good: Simple and Flexible Data Decoders for TypeScript

Sergey Samokhov — Wed, 24 Jun 2020 00:00:00 +0000

Intro

This post introduces both JSON decoders and my humble library of decoder building blocks, called be-good. Let’s start with the first.

What Are Decoders And Why Do We Need Them

JSON decoders are well known in statically typed languages like Elm or ReasonML (and are probably older than those two). Static types are a great way to catch many bugs at compilation time, which is sweet when the alternative is chasing those bugs in production. And even when it comes to preventing bugs, static types have some advantages over other techniques. I won’t discuss types vs. tests here, tempted as I am, but defensive programming is obviously wordier (thus obscuring business logic and creates more places for bugs to hide) and has some runtime cost.

But of course, even the most sound type system keeps you safe only if it controls the whole data lifecycle, cradle to grave. If the data comes from the outside world (say, across the network), what can the compiler say? If your types suggest a request returns data of type { status: Status, data: Data } but instead you get { status: -1, error: {}}, not only can your code break in runtime; it can also bring down your whole app exactly because you thought you didn’t need no stinking defensive programming any more, and now your code has no immunity.

Sure, there are situations where you can trust your types for external data, e.g., whey you generate them from a Swagger or a GraphQL schema. If you have something like that, and your DevOps scenarios are sound, you probably don’t need decoders—congrats, godspeed, carry on.

But maybe you don’t have that luxury. Perhaps you have a lot of legacy back-end, or it’s some tool developed by a single intern with no notion of what REST is, or the API is third-party. There are plenty of reasons you might have zero control over the API, and so typegen may be out of the question. Or maybe your app is just too small to bother, or you hate modern tooling and generally can’t be arsed.

Whatever your reasons, decoders may be of use. Let’s start with a (very loose) definition:

A decoder is a function that takes some external data that could be anything and makes sure it is of the type it’s supposed to be.

Enter `be-good`

But enough of theory, let’s see some code:

import { be, beObjectOf, or } from 'be-good';
import { isString } from 'lodash';

const beString = be(isString);
const optional = or(null);

type User = {
    name: string,
    surname: string,
    middleName?: string,
}

const userDecoder = optional(
    beObjectOf<User>({
        name: beString,
        surname: beString,
        middleName: optional(beString)
    })
);
// typeof userDecoder === (x: unknown) => User | null

Here’s how it treats various inputs:

userDecoder(false) // null: not an object
userDecoder({}) // null: missing some properties
userDecoder({ name: 0, surname: true }) // wrong property types
userDecoder({ name: 'Jackie', surname: 'Chan' })
// { name: 'Jackie', surname: 'Chan' } (middleName is optional)
userDecoder({ name: 'Samuel', middleName: 'L.', surname: 'Jackson' })
// { name: 'Samuel', middleName: 'L.' surname: 'Jackson' }

As you see, its runtime behavior is true to its signature: whatever the input, it returns either a value of type User, or null.

Basic building block: `be`

Immoral though it might sound, the most important part of be-good is be. be is a decoder factory, i.e., it creates decoders. Its signature looks like this:

function be<T>(predicate: (a: any) => a is T): Decoder<T>

The a is T part is crucial: the factory be takes a user-defined type guard. Happily, functions like lodash/isString or those in check-types are typed exactly like that. And later in this post, we’ll see how to roll your own.

What does be return? Given a predicate asserting that a is T, be returns Decoder<T>, i.e., (x: unknown) => T. So, whenever the predicate returns true, the resulting decoder returns a value of type T.

“But what if the predicate returns false?!”—an acute reader might cry in alarm. In that case, since the return type only allows values of type T and the input is quite obviously not of type T, the decoder has literally no other recourse but to throw a bloody exception. E.g.:

const beString = be(isString);
beString('foo'); // 'foo'
beString(12); // throws

Catching decorators: `or`

So, decoders can throw. One way to deal with that is to let whole parts of your application fail and show some fallback UI, e.g., using React’s error boundaries. This approach makes sense, but there are cases to be made for catching decoding exceptions closer to its source.

One of those cases is some data can be quite optional, as we’ve seen with middleName above:

const optional = or(null);

const beUser = beObjectOf({
    name: beString,
    surname: beString,
    middleName: optional(beString)    
});

No need to invalidate a whole user because she doesn’t have a middle name.

Another case is when you validate/invalidate your data rather far from the code that displays any UI (e.g., you put the decoded data in a Redux store and then update your components with it). Then it makes sense to express invalidity of

the data with… well, data.

const optional = or(null);
const beUser = optional(beObjectOf(/*...*/)); // unknown -> (User | null)

// somewhere in JSX
{user ?? (
<div className="section">
    <UserInfo {...user} />
</div>)}

A third case is a default value.

const orZero = or(0);
const beSum = orZero(beNumber); // unknown -> (number)

As you’ve probably divined from the examples above, or accepts a fallback

value and returns a decorator function. Decorator is then applied to a decoder

and, being a decorator, returns a new decoder. That new decoder never throws:

when the input data is invalid, it returns the fallback value instead. In a

bastardized semi-haskellish type notation, its signature is:

Fb => (In => Out) => In => (Out | Fb)

Put another way, when the original decoder returns values of type A, and the fallback value you give to or is of type B, the resulting (decorated)

decoder returns values of type A | B. A and B can differ (e.g., you can use nullish—or any other values to signal invalid/missing data) or they can coincide (e.g., when a fallback is just a default value). This (as many other things with be-good) is totally up to the consumer.

Decoding objects, arrays, and dictionaries

You have already seen beOjectOf in action: it checks if the input is an object, and then it validates the object properties, it’s rather self-explanatory. Let’s move on to collections, e.g., arrays and dictionaries. For that, we have two other factories.

beArrayOf and beDictOf are similar to beObjectOf, but their parameters are a bit different. First, they take a single element decoder—meaning all the elements are supposed to be of the same type. Second, the factory can optionally take a config:

type BeCollectionOptions = {
  /** What to invalidate on errors: one element or the whole collection */
  invalidate?: 'single' | 'all'
  /** Minimum count of (valid) collection elements */
  minSize?: number
}

Some examples:

const beNumber = be(isNumber)

beArrayOf(beNumber)([3, 25.4, false, -7])
// [3, 25.4, -7], because by default, `invalidate` option is 'singe',
// and that means simply omitting invalid elements

beArrayOf(beNumber, { invalidate: 'all' })([3, 25.4, false, -7])
// throws on reaching the first bad element

const orFallback = or('<fallback>') 
beArrayOf(orFallback(beNumber))([3, 25.4, false, -7])
// [3, 25.4, '<fallback>', -7], compare to the first example

beArrayOf(beNumber, { minSize: 4 })([3, 25.4, false, -7])
// throws: only 3 valid elements

/* beDictOf works pretty similarly, and in fact, takes the same options */

beDictOf(beNumber)({ a: 3, b: 25.4, c: false, d: -7 })
// { a: 3, b: 25.4, d: -7 }
// etc...

Decoding nested structures

Since beObjectOf, beArrayOf, and beDictOf take other decoders, they can also take the object/collection decoders, meaning you can nest them if necessary:

const beNestedData = beObjectOf({
    items: beArrayOf(beObjectOf({
        id: beNumber,
        dict1: beDictOf(beNumber),
        dict2: or({})(beDictOf(beString))
    }))
})

Mind where you catch though: if you don’t use or—and you use invalidate: ‘all’ a lot, a single wrong property can invalidate your whole structure.

A note on generics and type inference

beObjectOf and it’s brethern are generic, but you don’t have to spell out beObjectOf<Foo> all the time. The decoder return type will be inferred from the property decoders you gave to beObjectOf: e.g.

beObjectOf({ a: beString }) will have type (x: unknown) => { a : string }.

And since TypeScript types are structural, it doesn’t matter how the type is

called as long as the shape is right.

But if you make a mistake in a property name or a kind of decoder you give to beObjectOf, TypeScript will fail—somewhere—and the error message might point to a place far from where the actual error is, and you’ll spend more time fixing it. It might be better to specify the expected type right inside a decoder (like above), or maybe right outside of it, like this:

import { beOjbectOf, Decoder } from 'be-good'
// ...
const objDecoder: Decoder<Type> = optional(beObjectOf(/* ... */))

Fail early, I say.

Custom predicates and custom decoders

I think that’s quite enough for one post already (finish early, I say), but I must mention an essential design goal of be-good: being flexible. Indeed, it’s more flexible than the above examples might suggest. For one thing, the type guards from Lodash are handy, but what if you want to check for more?

const isEven = (n: unknown): n is number => isNumber(n) && n % 2 === 0;

const beEven = be(isEven) // unknown => number

If you dig opaque types, you can use those too (the enum trick is from

an article by Patrick Bacon).

enum PriceBrand {}
type Price = number & PriceBrand
const isPrice = (n: unknown): n is Price => isNumber(n) && n > 0
const bePrice = be(isPrice) // unknown => Price

You can also write whole custom decoders, be it because you cannot validate

fields separately, or because you need to ~~mess with~~ manipulate your data

structure. Enter two low-level helpers: beObject (not to be confused with

beObjectOf) and fail:

import { be, beObject, fail, or } from 'be-good'

type Range = {
  min: number;
  max: number;
}

type rangeDecoder = or(null)((input: unknown): Range => {
  // note that `start` and `end` are the properties of the input, not of the output
  const { start, end } = beObject(input) // typeof start === typeof end === unknown

  if (!isNumber(start) || !isNumber(end) || end > start) fail('Invalid range')

  return { min: start, max: end }
})

Note how the earlier examples mostly compose functions, but as you see here,

that’s not the only resource you have. Sure, here we still use a catching

decorator™ (i.e., the result of or(null)), which counts as functional

composition. But you can also create variables, fail the decoder imperatively

and do all the stuff you typically do in JavaScript—even though I don’t

recommend side effects in your decoders. And sure, you could write this

particular decoder in a more functional fashion, but the point is you don’t have

too. be-good is not hellbent on forcing one specific programming style.

Another important thing is using fail to… well, fail the decoder. Notice

how, like beObject, fail is used inside a function wrapped in a catching

decorator. You don’t want unchecked exceptions everywhere. And while on the one

hand, you have to remember that decoders use exceptions under the hood (and may

blow that hood if you don’t pay attention), on the other hand, it’s not a good

idea to throw exceptions manually. If you want to fail your decoder, call

fail. So far, it doesn’t do much besides throwing, but it might in the future,

so don’t break the abstraction.

Left unsaid

I don’t believe be-good is complete: it misses mechanisms for decoding sum types (unions), proper docs, type tests, and maybe it needs better error messages (file an issue if you think it does). I also probably haven’t done writing about it. What’s left unsaid is:

design goals
comparison to known alternatives (see gcanti/io-ts & nvie/decoders)
comparing null(ish) | Entity with monadic types like Result
advanced composability of decoders
why the hell the functions are not curried
and so probably on

In the meantime, try be-good, play with it, file some issues, maybe even contribute if you like, and stay safe.

Posted first at hoichi.io, narcissist that I am.

Simple Sum Types In TypeScript Without Common Fields

Sergey Samokhov — Fri, 05 Jun 2020 11:53:00 +0000

Discriminated unions are well known in TypeScript. Their only (?) downside is they need a common property usually named kind or tag, e.g.:

type RenderProps<T> =
| { kind: 'children',
    children: (x: T) => ReactElement | null;
  }
| { kind: 'render',
    render: (x: T) => ReactElement | null;
  }

Which makes it a tad too wordy to use:

const RPComponent = (props: RenderProps<number>) => {
    switch(props.kind) {
    case ('children'):
        return props.children(42);
    case ('render')
        return props.render(42);
    }
}

// elsewhere
<RPComponent kind="render" render={n => <i>{n}</i>} />

Now, I’m fine with JS in templates (if you still call of JSX as of templates, and why not), but that unnecessary kind prop gets my goat.

So here’s a more compact solution:

type RenderProps<T> =
| { children: (x: T) => ReactElement | null;
    render?: never;
  }
| { children?: never;
    render: (x: T) => ReactElement | null;
  }

const RPComponent = (props: RenderProps<number>) =>
    (props.children || props.render)(42);

// and
<RPComponent render={n => <i>{n}</i>} />

It’s still a sum type (you can neither omit both children and render nor provide them both), but now you don’t need no stinking kind anywhere.

Mind that for some reason it’s not enough to declare the union variants as { chidlren: SomeType, render: undefined }. At least for JSX, TypeScript will want you to still specify a prop and give it a value of undefined. But render?: undefined (or never, which, I think, better conveys your intention) does the trick.

Posted first at hoichi.io, because you can’t be too paranoid.

Real-World Composable Error Handling in Reason

Sergey Samokhov — Sat, 21 Dec 2019 11:53:00 +0000

The Question

If you’ve ever considered handling errors in Reason or OCaml, chances are, you’ve happened upon Vladimir Keleshev’s article. Everybody links to it, so it seems like the way to handle errors.

But how does it look in practice?

The Experiment

Let’s try and write a simple useFetch, a custom ReasonReact hook that wraps bs-fetch. Then we’ll use another dependency, for decoding. And — the whole purpose for this experiment — we’ll try to compose their errors.

Using bs-fetch

Let’s start with the basics:

// UseFetch.re

type fetchError = [| `FetchError(Js.Promise.error)];

type t =
  | Fetching
  | Complete(Belt.Result.t(Js.Json.t, fetchError));

let useFetch = url => {
  let (state, setState) = React.useState(_ => Fetching);

  React.useEffect1(
    () => {
      Js.Promise.(
        Fetch.fetch(url)
        |> then_(Fetch.Response.json)
        |> then_(json => setState(_ => Complete(Ok(json))) |> resolve)
        |> catch(error =>
             `FetchError(error)
             |> (error => setState(_ => Complete(Error(error))) |> resolve)
           )
        |> ignore
      );

      None;
    },
    [|url|],
  );

  state;
};

First, let’s check if it works:

[@react.component]
let make = () => {
  let reposJson =
    UseFetch.useFetch(
      "https://api.github.com/search/repositories?q=language:reason&sort=stars&order=desc",
    );

  Js.log(reposJson);

  ReasonReact.string("Fetching...");
};

It does work, as in, it logs some JSON.

bs-fetch Error

Now let’s talk about error handling. Note how fetchError has only one variant because bs-fetch uses Js.Promise.error for everything, and Js.Promise.error is opaque by design. The thing is, you can technically throw anything in JS, not just an exception. If that worries you, run before it gets worse: the upcoming React Suspense for Data Fetching in React is officially supposed to throw promises by way of algebraic effects).

So here’s one spanner in the works: opaque types in underlying libs. We sort of know that what bs-fetch happens to throw are exceptions, so we could use magic to convert Js.Promise.error, and then extract the message, but even if we were to parse error messages, what good is that? Parsing messages is heuristic and, therefore, brittle, and using them raw is only any good for technical purposes, like logging, not for anything user-facing.

But speaking of logging, we didn’t fetch JSON just to dump it it to console. Let’s decode.

Decoding

Of all the Reason json decoders I know, decco is by far the most sugary, using ppx to create decoders from annotated type definitions. Here’s how it looks.

// GhRepo.re
[@decco]
type repo = {
  [@decco.key "full_name"]
  fullName: string,
  [@decco.key "html_url"]
  htmlUrl: string,
};
[@decco]
type t = {items: array(repo)};

// Decode.re
type decodeError = [| `DecodeError(Decco.decodeError)];

// wrapping Decco.decodeError in a polymorphic variant
let mapDecodingError =
  fun
  | Ok(x) => Ok(x)
  | Error(e) => Error(`DecodeError(e));

Decco.decodeError is not as opaque: it’s defined as a record, holding path, error message, and JSON value. Still, to discern various errors, we’d have to resort to parsing messages just the same; let’s not.

To make composing with decoders easier, let’s also create a helper in the UseFetch module.

let mapOk = (t, f) =>
  switch (t) {
  | Fetching => Fetching
  | Complete(Ok(r)) => Complete(f(r))
  | Complete(Error(_)) as e => e
  };

UseFetch.mapOk is similar to Result.flatMap: the function parameter that it takes can return either Ok(data) or Error(error)—exactly the signature of Decco decoders.

All together now:

// App.re
[@react.component]
let make = () => {
  let repos =
    UseFetch.(
      useFetch(
        "https://api.github.com/search/repositories?q=language:reason&sort=stars&order=desc",
      )
      ->mapOk(r => GhRepo.t_decode(r)->Decode.mapDecodingError)
    );

  ReasonReact.string("Fetching...");
};

But lo, an error.

A Decoding Error, or, Parametrizing UseFetch.mapOk

Immediately we get an error somewhere inside that mapOk call:

  This has type:
    result(Js.Array.t(GhRepo.t), ([> `DecodeError(Decco.decodeError)] as 'a))
      (defined as Belt_Result.t(Js.Array.t(GhRepo.t), 'a))
  But somewhere wanted:
    Belt.Result.t(Js.Json.t, UseFetch.fetchError) (defined as
      Belt_Result.t(Js.Json.t, UseFetch.fetchError))

  The incompatible parts:
    Js.Array.t(GhRepo.t) (defined as array(GhRepo.t))
    vs
    Js.Json.t (defined as Js.Json.t)

I won’t pretend I understood this error right off the bat, but neither will I burden you with the story of my googling. The culprit is the way that UseFetch.mapOk is defined: our decoding process yields decoded data and a decoding error (typed as a superset of the same error), but mapOk expects strictly Js.Json.t and fetchError. So we need to parametrize both, and we start by parametrizing UseFetch.t.

type t('d, 'e) =
  | Fetching
  | Complete(result('d, [> fetchError] as 'e));

And luckily, that’s the only place you have to introduce type parameters. You don’t have to provide 'd or 'e when consuming those types, nor do you have to specify the mapOk signature, because OCaml/Reason type inference is pretty great. If you had to provide the signature, it’d look like this:

let mapOk: 'a 'b 'e. (t('a, 'e), 'a => result('b, 'e)) => t('b, 'e) =
  (t, f) =>
    switch (t) {
    | Fetching => Fetching
    | Complete(Ok(r)) => Complete(f(r))
    | Complete(Error(_)) as e => e
    };

As you can see, there are two type parameters for data, 'a & 'b, because obviously, the mapping function can, and usually will, convert data from one type to another (e.g., from JSON to a record). But the error type stays the same because unifying an error type is precisely what we’re after. And again, let me remind you we don’t actually have to spell out the mapOk type.

Anyway, here we are, consuming the decoded result and the possible errors:

[@react.component]
let make = () => {
  UseFetch.(
    useFetch(
      "https://api.github.com/search/repositories?q=language:reason&sort=stars&order=desc",
    )
    ->mapOk(r => GhRepo.t_decode(r)->Decode.mapDecodingError)
    ->(
        fun
        | Fetching => ReasonReact.string("Fetching...")
        | Complete(Ok(({items}: GhRepo.t))) =>
          <ul>
            {Belt.Array.map(items, ({fullName, htmlUrl}: GhRepo.repo) =>
               <li key=fullName>
                 <a href=htmlUrl> {ReasonReact.string(fullName)} </a>
               </li>
             )
             ->React.array}
          </ul>
        | Complete(Error(`FetchError(_))) =>
          ReasonReact.string("Fetch error!")
        | Complete(Error(`DecodeError(_))) =>
          ReasonReact.string("Decode error!")
      )
  );
};

Conclusion

So, all in all, composable error handling with real-world ReasonML libraries is quite possible, with a few caveats.

Firstly, out of 2 libs we’ve tried, neither used polymorphic variants, so we’ve taken their own (non-discernible) errors and wrapped them in our polymorphic variants. Not very granular, but then the consuming code didn’t have to branch over a whole lot of variants. I imagine having a few dozens possible variants would bring its inconveniences as well.

Secondly, to make polymorphic variants extensible, you’re going to have to parametrize a few things here and there. But OCaml type inference being seriously impressive, that problem is very manageable; at least, you don’t have to be verbose to solve it.

Is that all? Maybe that’s all. And oh, here’s the repo with the code.

Posted first at hoichi.io, because home sweet home.

Type-And-Test Driven Development

Sergey Samokhov — Mon, 04 Nov 2019 11:53:00 +0000

The Question

I know what type-driven development is (didn’t say I’ve mastered it). I sort of get what is test-driven development, or even red-green-refactor. But can you do both at the same time? If you start with types, is there anything left of the green-red-refactor cycle? Or does that mean you write your tests post factum?

The typed language I use for the experiment is, by the way, ReasonML, which is basically another syntax for OCaml. Not that I think it matters much: I have more experience with TypeScript, but OCaml is seemingly a smaller language and its type system feels slightly simpler. Overall, whether a cognitive load of doing OCaml types is bigger or smaller compared to TS, it’s probably not by much. Or something.

Spoiler: turns out doing both iteratively is easier than I thought, but it’s not always such an even-going process as the basic TDD examples would make you believe.

The Task

Let’s suppose that we need collections of blog posts to use in all kinds of post lists, like a blog feed, tag pages, and RSS. For simplicity’s sake, though, let’s not think about pagination or filtering, and instead create a simple API that allows:

Creating a collection;
Adding posts to a collection;
- adding a post with the same path should update the existing record;
Getting an array of posts;
- should be a sorted array.

And without any further design (some people would make you think that tests are all the design you’ll ever need, and others advise you design with types), we set off.

The Most Minimal Array

A real true honest-to-god TDD aficionado might start with expect([||]) |> toBe([||]). Coming from static types, that feels like too little code, but hey, maybe that’s a good exercise, so let’s roll with it.

But let’s actually have some code to test:

// Collection_test.re
expect(Collection.toArray()) |> toEqual([||]);

// Collection.re
let toArray = () => [||];

That API is not quite right. Shouldn’t we create a collection first? Shouldn’t toArray accept Collection.t? Let’s start by updating the test:

expect(Collection.make()->Collection.toArray) |> toEqual([||]);

That breaks our types. Let’s fix them:

let make = () => ();

let toArray = _ => [||];

Or, if you feel farsighted,

let make = () => [||];

let toArray = a => a;

Again, we try to keep up the red-green spirit and write the least amount of code to pass a test.

Adding values

An empty collection is more or less covered; let’s add values.

// collection_test.re
expect(Collection.(make()->add(1)->toArray)) |> toEqual([|1|]);

// collection.re
let add = (arr, el) => Js.Array2.concat(arr, [|el|]);

Now two values.

expect(Collection.(make()->add(1)->add(2)->toArray)) |> toEqual([|1, 2|])

Three values!

expect(Collection.(
  make()->add(1)->add(2)->add(3)->toArray))
|> toEqual([|1, 2, 3|])

Hey, I could do this all day.

And How Is It Type-Driven, Exactly?

We’re interrupting our flow to ask ourselves: isn’t all of the above just the plain old TDD (as in, test-driven). Weren’t we supposed to start our design like this:

type t;
let make: unit => t;
let add: (t, Post.t) => t;
let toArray: t => array(Post.t);

Maybe we wouldn’t even end up with those useless int tests. Because I already smell trouble ahead. But let’s ignore the smell for a minute and see where does this precarious path lead.

A Stab at Sorting

Here’s a test for sorting order. Look out below.

expect(Collection.(make()->add(3)->add(5)->add(2)->add(-4)->toArray))
|> toEqual([|(-4), 2, 3, 5|])

I just knew it. Broken.

By the way, that test exemplifies a possible bug that types won’t catch. Not the OCaml/Reason types, at least.

But let’s implement sorting.

Types Head Their Rear Again

This is when I’m beginning to have second thoughts about the red-green minimalism. Obviously (to me), the next minimal step is to sort the ints to fix the test. But our actual goal isn’t to sort ints. We need to sort articles. By dates. And if we sort ints now, we’ll have more tests to refactor later.

Shall we skip to the next step then, and sort dates? The quickest would be to use polymorphic comparison, which feels a little flaky, and anyway, all this thinking got me thinking: do I want to create a collection of anything, or do I merely need a collection of posts? Because the former would require a functor with a few parameters.

Of course, functors are cool and make you feel like a real programmer. They also probably make property testing easier, and property tests make you feel safer. But then again, a functor would make you have to write more tests to make sure parametrization works correctly. So, no, can’t be bothered with functors for now. Not until I need to collect something other than posts.

So, answering the question of what to sort, yes, I could go and change Collection.t to Js.Date.t and rewrite the int tests accordingly. I’d even be able to reuse the dates sorting logic and the date values in the tests. But since I’ve already taken myself out of the busy mood, I say let’s bite the bullet and switch to the actual posts already.

Enter The Actual Posts

One more thing before we dive back into the red-green rush. We’re probably going to create more mock posts that is healthy to read in full form. Better add some helpers for readability.

// some mocks from our stash
module Any = { module Old = Mock.AnyOld; }

let postWithDate =
  dateStr => Post.{
    meta: {
      ...Any.Old.meta,
      date: Js.Date.fromString(dateStr),
    },
    title: "",
    content: Markup.Markdown(""),
    excerpt: Markup.Markdown(""),
    source: Any.Old.source,
  };

let postsArray = Belt.Array.map(_, postWithDate);
let addPosts = (c: Collection.t, posts) =>
  postsArray(posts)->Belt.Array.reduce(c, Collection.add);

And with that, let’s rewrite our singleton collection test:

test("single value", () =>
  expect(
    Collection.(make()->addPosts([|"2019-06-01T20:38:01.155Z"|])->toArray),
  )
  |> toEqual(postsArray([|"2019-06-01T20:38:01.155Z"|]))
);

Green. All right.

Sorting, Continued

Feel like breaking something?

test("several unsorted values", () =>
  expect(
    Collection.(
      make()->addPosts([|"2019-09-01", "2019-07-01", "2019-05-02"|])->toArray
    ),
  )
  |> toEqual(postsArray([|"2019-05-02", "2019-07-01", "2019-09-01"|]))
);

And we’re in the red again. Time to do the sorting.

let toArray =
  Belt.SortArray.stableSortBy(_, (p1: Post.t, p2: Post.t) =>
    Js.Date.(compare(p1.meta.date->getTime, p2.meta.date->getTime))
  );

That wasn’t too hard. Of course, it’s not a final implementation, as we’ll see in a minute, but we’re getting close.

Updating by Path

There’s one part of spec left to implement. If we add a post with the same full path for a second or third or whichever time, our collection should only hold the latest value.

First, let’s add a parameter to our test helpers to accept different paths.

let postWithDate = ((fullPath, dateStr)) =>
  Post.{
    meta: {
      ...Any.Old.meta,
      date: Js.Date.fromString(dateStr),
    },
    title: "",
    content: Markup.Markdown(""),
    excerpt: Markup.Markdown(""),
    source: {
      ...Any.Old.source,
      path: {
        ...Any.Old.source.path,
        full: fullPath,
      },
    },
  };

Digression: did you notice how I didn’t have to update postArray and addPosts? Point-free programming in OCaml is pretty neat, even without the composeRight operator.

Anyway, here’s how tests should look now:

test("several unsorted values", () =>
  expect(
    Collection.(
      make()
      ->addPosts([|
          ("a/x.md", "2019-09-01"),
          ("a/y.md", "2019-07-01"),
          ("a/z.md", "2019-05-02"),
        |])
      ->toArray
    ),
  )
  |> toEqual(
       postsArray([|
         ("a/z.md", "2019-05-02"),
         ("a/y.md", "2019-07-01"),
         ("a/x.md", "2019-09-01"),
       |]),
     )
);

Now let’s try to repeat some paths:

test("same paths get updated", () =>
  expect(
    Collection.(
      make()
      ->addPosts([|
          ("a/x.md", "2019-09-01"),
          ("a/y.md", "2019-07-01"),
          ("a/z.md", "2019-05-02"),
          ("a/y.md", "2019-04-01"),
          ("a/x.md", "2019-01-09"),
        |])
      ->toArray
    ),
  )
  |> toEqual(
       postsArray([|
         ("a/x.md", "2019-01-09"),
         ("a/y.md", "2019-04-01"),
         ("a/z.md", "2019-05-02"),
       |]),
     )
);

Red, as expected. Let’s update the implementation. Our requirements sound like a job for a Map, which means we should update the t implementation and pretty much all the functions.

open Belt;

module PostCmp =
  Id.MakeComparable({
    type t = string;
    let cmp = Pervasives.compare;
  });

type t = Map.t(string, Post.t, PostCmp.identity);

let make = () => Map.make(~id=(module PostCmp));

let add = (m, p: Post.t) => Belt.Map.set(m, p.source.path.full, p);

let toArray = m =>
  Map.valuesToArray(m)
  ->SortArray.stableSortBy(_, (p1: Post.t, p2: Post.t) =>
      Js.Date.(compare(p1.meta.date->getTime, p2.meta.date->getTime))
    );

And with that, we’re green once again and more or less done.

Conclusion

In the beginning, we’ve asked ourselves if it’s possible to combine type-driven development and red-green-refactor-flavored test-driven development. And the short answer is that it is indeed possible to do both at once and do it iteratively.

Of course, the above is just one data point, and neither the types nor the implementation was too complicated. Still, in this particular example, it seems the combined complexity of types and test cases is very manageable.

Here are a few finer points:

Even with types, you still need some unit tests. E.g., you can’t encode sorting order in types (at least not in ML).
It’s nice to be able to keep your red-green cycles short, but sometimes it makes more sense to slow down. That breaks your stride, yes, but I think the ability to stop and think when called for is as crucial for a programmer as the ability to pivot for an agile team.
It pays to have a written spec, however brief, before you start coding. Were we to start writing my “specs” in tests, or even in types, it might have taken noticeably more cycles.
And by the way, with the helpers we wrote, it should be relatively easy to write property tests.
But that is a story for another day.

Posted first at hoichi.io, but you’re welcome to read it anywhere.

How Not to Introduce ReasonReact to Your Project

Sergey Samokhov — Sat, 19 Jan 2019 11:53:00 +0000

Well, actually the reality is not as harsh as the clickbaitey title would make you think. Nothing particularly wrong came out of my introduction of ReasonReact and, by extension, ReasonML to the static build of this very site. The claim that you can introduce Reason & ReasonReact to an existing application one component at a time is entirely true.

Moreover, ReasonML is pretty sweet, and BuckleScript is the bestest, fastest, most optimizing -to-JS compiler I know of.

It’s just that my particular implementation was just a little bit inefficient and probably not a good management decision. If I could go back in time, I’d do it differently—or maybe wouldn’t do at all in the foreseeable future.

That disclaimer aside, let me start from the start.

What Is Wrong With TypeScript

The reason I was eyeing a language like ReasonML is that TypeScript seems a tad suboptimal for me (and a little boring). It’s still wordy, all the ESNext advances notwithstanding. You can emulate both sum types and exhaustive cases in it, but they look somewhat monstrous. Its types give you way fewer guarantees than, say, types in ML languages. And don’t get me started on FP patterns in JS: pipe(foo, bar, baz)(x) reads infinitely less natural than x |> foo |> bar |> baz.

Probably as important as the guarantees, it feels like the TypeScript types allow you a little too much. Extending object types, for one. Yes, Object.assign() is totally a thing in JavaScript that those types are modeling. Yes, often it does feel convenient to go and add some more properties to an existing object.

It also leads to lazy thinking. Take my static site build scripts. They have:

Pages (posts) as I read them off a file system, with path information and raw content data
Same, with YAML frontmatter extracted and markdown parsed to HTML
Same, put into lists for blog feed, tag feeds, and RSS
Both individual pages and feed pages, wrapped in the appropriate page/post layout

The types for all of the above are way under-designed because every single time I thought: Nah, I’ll just copy the old object with some extra prop, and maybe override the content property, nevermind it’s now HTML instead of Markdown. Yes, I know that with proper discipline you can sit down and think those specs through, but discipline is depletable, so you tend to wing it—and then waste time setting breakpoints and logging stuff.

So I hoped that OCaml’s restrictive records would nudge me towards designing with types.

There was also some nice-to-haves, like maybe processing more data with the Reason standard library in a more concise way, or maybe turning my blog into semi-SPA with ReasonReact. Not that the page load speed is a problem I have with a Netlify-hosted static site.

Why, Then, Have I Started With Templating

But all of a sudden I’ve decided that I need to introduce comments. And maybe a new theme. And all of it to improve the chances of getting a better job. For that, one felt, the lack of ReasonReact was a showstopper. For if I were to use it later at all, doing work on my current Jade templates felt a little wasteful.

So, instead of solving an interesting problem of designing with types, I’ve found myself doing something closer to a chore. Also, was I sure it was necessary? Do I have actual plans to evolve the client part of my blog? All for fear of not finding a good job? Is blogging on random tech bits something I want more than just hacking or learning stuff?

Lousy management, like I said. False urgency over importance.

So, How It Went?

Well, it went ok. I had to resort to a little hack because the BuckleScript compiler has fewer options regarding input/output dirs. I also had to use the rss package, because React’s idea of how you use the link tag is not compatible with how it is used in RSS (which is not HTML, of course, but try telling it to React). But then maybe my feed is now more standard-compliant for it, and converting some random Atom example to Jade was a worse hack, to begin with.

However, the whole result is also mildly useless. The scheme is now like this: I’ve got a templates.re file that returns a dictionary of template functions to be consumed in JS-land. So, it uses JS interop to:

construct the dictionary itself
wrap the individual templates and convert the JS param objects to ML labeled parameters
also, for templates that get arrays of posts and tags, get individual props off those posts and tags, because I’ve got too lazy to convert those objects to proper ML records

So, the bulk of my work went into the interop that I’m going to have to throw away when (or if) I convert everything to Reason/OCaml. Mind you, BuckleScript JS interop is pretty terrific, and ReasonML sugar for it is great too, But the process is just a bit tedious and doesn’t feel like making something important, especially since it’s all makeshift.

Conversely, there isn’t a whole lot of ML to ML API: the wrappers pass ML params to the templates, and those pass them on to the ReasonReact components. Not that many types to help me grok my state management, or yell at me if I do something wrong.

Summary?

All in all, I did add a few smallish ReasonReact components to a TypeScript module, and they work. As far as learning projects go, it went well. What I’ve learned wasn’t particularly fundamental, but then you can’t do much practical stuff without some non-fundamental knowledge. The bane of project-based learning, I guess.

Next time, though, I’m going to think twice whether it’s worth it, and where that new technology is going to have the most impact. Using Reason or OCaml for the build logic itself would take longer, but might bring more clarity. Converting everything would take even longer, but would free me from having to write that interop code—or rather, I’d have to write it for some other modules, like @most/core.Provided I’d managed to debug the whole thing at once.

Especially since I have minimal experience with tests. Maybe I should write more of those in 2019.

Posted first at hoichi.io, but you’re welcome to read it anywhere.

Tail-optimized functions in ReasonML

Sergey Samokhov — Sat, 15 Sep 2018 11:30:00 +0000

All the examples are in ReasonML, which, very roughly speaking, is a cross of OCaml & JS I’m very interested in at the moment.

With that intro aside, let’s write a function that generates a range of ints, i.e., takes a tuple of (beginning, end) and returns a list of all ints in between, including the boundaries. The naive approach:

let rec genRange = ((min, max)) =>
  min > max
    ? []
    : [min, ...genRange((min + 1, max))];

The problem with it is if the range is rather long, it may cause a stack overflow. See this expression: [min, ...genRange((min + 1, max))]? To evaluate it the machine has to wait for the recursive call to return and then use the return value to build the longer list.

(Would be even worse with arrays, by the way. You’d have as many arrays as there would be elements in a final result, and they wouldn’t share any of the memory between them, effectively allocating about (len+1) * len/2 elements, none of which could be garbage collected or otherwise freed until the most deeply nested calls start to return.)

Now, both the native OCaml compiler and BuckleScript (the latter compiles OCaml/ReasonML to JS) support tail call optimization. Meaning that instead of adding another frame to the stack, the compiler makes the generated code to reuse the current one. (It’s a bit different in JS code, but we’ll get to it later.)

For that mechanism to work, you have to be sure you don’t do anything with the result of the recursive call other than return it as is. That way, you don’t have to return at all. The function instance that is called last evaluates the final result and returns it straight to the original callee.

However, that means that the instance you call last should have all the data to evaluate that final value. Passed as parameters (since we’re talking FP). So, our old naive gen_range doesn’t cut it.

gen_range already has min and max as its parameters. What does it lack from that expression, [min, ...gen_range((min + 1, max))], to evaluate full list on the final step?

min? We already have it. The result of gen_range(min + 1, max)? That’s more like it. We need to pass the current version of the list to build the next one. Let’s add list as a parameter:

let rec genRangeTco = (list, (min, max)) =>
  max < min
    ? list
    : genRangeTco([max, ...list], (min, max - 1));

You might have noticed that now we do not increase min, we decrease max instead. Looks a bit unnatural, especially if your ‘natural’ is the good old i++, but it’s closer to how lists are built: what was the head yesterday, today is part of the tail.

Another quite noticeable thing is that the signature is less convenient now: you have to provide an empty list as a parameter. This is why with TCO functions, the recursive one is usually a helper function, and users get a dumbed down version:

let genRangeTco = {
  let rec helper = (list, (min, max)) =>
    max < min
      ? list
      : helper([max, ...list], (min, max - 1));

  helper([]);
};

helper([]) is a partial application, by the way, pre-applying an empty list to helper and returning a function that expects a tuple of (min, max), just like our first version.

One last thing. The future of tail call optimization proposal is, as I’m writing this, quite murky. But the BuckleScript compiler (that, again, kind of powers ReasonML), hellbent as it is on performance of the JS it generates, won’t be denied. See what it does with the naive version first: the good ole recursive call, nothing special, amirite? And now, behold, the tail-optimized version. It’s got compiled to a loop!

See why I’m interested in Reason?

Posted first at hoichi.io, because home sweet home.

TIL: Why ReasonML Has No Nullary Functions

Sergey Samokhov — Fri, 17 Aug 2018 00:00:00 +0000

Why doesn’t ReasonML have nullary functions? That is due to ReasonML always performing partial application (explained in detail later): If you don’t provide all of a function’s parameters, you get a new function from the remaining parameters to the result. As a consequence, if you could actually provide no parameters at all, then func() would be the same as func and neither would actually call func.

– Exploring ReasonML and functional programming by Dr. Axel Rauschmayer

Posted first at hoichi.io, because you can’t be too paranoid.

DEV Community: Sergey Samokhov

be-good: Simple and Flexible Data Decoders for TypeScript

Intro

What Are Decoders And Why Do We Need Them

Enter be-good

Basic building block: be

Catching decorators: or

Decoding objects, arrays, and dictionaries

Decoding nested structures

A note on generics and type inference

Custom predicates and custom decoders

Left unsaid

Simple Sum Types In TypeScript Without Common Fields

Real-World Composable Error Handling in Reason

The Question

The Experiment

Using bs-fetch

bs-fetch Error

Decoding

A Decoding Error, or, Parametrizing UseFetch.mapOk

Conclusion

Type-And-Test Driven Development

The Question

The Task

The Most Minimal Array

Adding values

And How Is It Type-Driven, Exactly?

A Stab at Sorting

Types Head Their Rear Again

Enter The Actual Posts

Sorting, Continued

Updating by Path

Conclusion

How Not to Introduce ReasonReact to Your Project

What Is Wrong With TypeScript

Why, Then, Have I Started With Templating

So, How It Went?

Summary?

Tail-optimized functions in ReasonML

TIL: Why ReasonML Has No Nullary Functions

Enter `be-good`

Basic building block: `be`

Catching decorators: `or`